OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.
Go to file
Mo Li f2af49337d
[Feature] Add ATC Choice Version (#1019)
* Squashed commit of the following:

commit c48ad194c3976dc63d1b60d8c8ab2d5ff9e1cbfe
Author: DseidLi <2568818204@qq.com>
Date:   Tue Apr 2 16:57:43 2024 +0800

    add atc_choice

commit 3ac6efea29619573e6fac8fa3cce464853dcead0
Merge: 2d4e559 8e3a9c3
Author: DseidLi <2568818204@qq.com>
Date:   Tue Apr 2 16:41:38 2024 +0800

    Merge branch 'atc_choice' into atc_add_choice

commit 8e3a9c396a3e5546d3faf584183f6fd60b974d5e
Merge: 150a036 0a6a03f
Author: DseidLi <2568818204@qq.com>
Date:   Tue Mar 26 04:47:07 2024 +0800

    Merge branch 'main' into atc_choice

    Conflicts:
    	configs/summarizers/needlebench.py
    	opencompass/datasets/needlebench/multi.py
    	opencompass/datasets/needlebench/origin.py
    	opencompass/datasets/needlebench/parallel.py

commit 150a036d6d990f26a57c974d1af83d88c31a0f9d
Merge: 8d6ac9a 940dd18
Author: DseidLi <2568818204@qq.com>
Date:   Wed Mar 20 03:49:08 2024 +0800

    Merge branch 'needlebench_fix' into atc_choice

commit 8d6ac9a1a43b1c9d0f0ea27e7d58968a203ea898
Author: DseidLi <2568818204@qq.com>
Date:   Wed Mar 20 03:41:49 2024 +0800

    optimize needlebench code

commit 940dd18a4270f24bc69edd2a780182c68918e1a9
Author: DseidLi <2568818204@qq.com>
Date:   Wed Mar 20 03:39:46 2024 +0800

    fix vllm

commit d8be6877bc41051f3edcc0421c462c834c0f1c9a
Merge: ecad78a 2527fda
Author: DseidLi <2568818204@qq.com>
Date:   Tue Mar 19 21:07:08 2024 +0800

    Merge remote-tracking branch 'origin/add_1M_dataset' into atc_choice

commit 2527fda8a5
Author: DseidLi <2568818204@qq.com>
Date:   Tue Mar 19 16:03:40 2024 +0800

    add model configs

commit 75425acdf8
Author: DseidLi <2568818204@qq.com>
Date:   Tue Mar 19 16:02:15 2024 +0800

    add prompt postion args

commit 367ba1ba61
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 21:40:00 2024 +0800

    add Needlebench-1000K configs

commit ecad78af14c4bb00fe325779114b384c57ab30bf
Author: DseidLi <2568818204@qq.com>
Date:   Thu Mar 14 22:08:32 2024 +0800

    fix atc

commit 08772c0787b18872abadc9ffec3223941a5ee0c2
Merge: 9f3f8cf caf1cf8
Author: DseidLi <2568818204@qq.com>
Date:   Thu Mar 14 22:07:28 2024 +0800

    Merge branch 'main' into atc_choice

    Conflicts:
    	configs/datasets/needlebench/readme.md
    	configs/datasets/needlebench/readme_zh-CN.md
    	configs/summarizers/needlebench.py
    	opencompass/datasets/needlebench/atc.py
    	opencompass/summarizers/needlebench.py

commit 9f3f8cfb4452722734d334114ac1d14110e57406
Author: DseidLi <2568818204@qq.com>
Date:   Thu Mar 14 21:35:53 2024 +0800

    add atc-choice test

commit 52be7c1202376b4e09821188b826f1a805328129
Author: DseidLi <2568818204@qq.com>
Date:   Wed Mar 6 02:54:15 2024 +0800

    update needlebench randomseed and add vllm qwen14b

commit fc1effce596ae2e5ece4933e8cd34aef8e64a6f9
Merge: 4e747ed caf1cf8
Author: DseidLi <2568818204@qq.com>
Date:   Wed Mar 6 02:51:14 2024 +0800

    Merge branch 'main' into add_model_configs

commit 31834f9b23af3354ac3581ec86d693d0f05cdd1c
Merge: 7dabc82 120bf8b
Author: DseidLi <2568818204@qq.com>
Date:   Sun Mar 3 23:29:42 2024 +0800

    Merge branch 'main' of https://github.com/open-compass/opencompass into atc_choice

commit 4e747ed1988ddbcfcc7fff334601259ade72d363
Author: DseidLi <2568818204@qq.com>
Date:   Sun Mar 3 22:15:25 2024 +0800

    add internlm2-lmdeploy model and gemma configs

commit 7dabc828123d711c8cf834d6aab4137bb55e85ed
Author: DseidLi <2568818204@qq.com>
Date:   Sat Mar 2 17:26:15 2024 +0800

    add atc choice version -ZH

commit 996f8ae43d
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 16:58:56 2024 +0800

    update readme for needlebench

commit f7266e873c
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 16:44:53 2024 +0800

    move readme.md

commit 1c7375681d
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 16:38:31 2024 +0800

    fix linting error

commit b6524f3ebf
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 16:33:51 2024 +0800

    lint summarizer

commit c0d1190e39
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 16:29:03 2024 +0800

    add needlebench intro, fix summarizer

commit 0965baf785
Author: DseidLi <2568818204@qq.com>
Date:   Mon Feb 26 13:31:26 2024 +0800

    fix bug in needlebench summarizer

commit 5d32b31eb8
Author: DseidLi <2568818204@qq.com>
Date:   Sat Feb 24 03:19:08 2024 +0800

    update act prompt

commit af82a7f085
Merge: 32bf9fe 53fe788
Author: DseidLi <2568818204@qq.com>
Date:   Fri Feb 23 17:50:32 2024 +0800

    Merge remote-tracking branch 'upstream/main' into needlebench

commit 32bf9fe802
Author: DseidLi <2568818204@qq.com>
Date:   Fri Feb 23 17:31:32 2024 +0800

    simplify needlebench 32k, 128k, 200k for eval

commit a7cb025e05
Author: DseidLi <2568818204@qq.com>
Date:   Fri Feb 23 14:48:58 2024 +0800

    add needlebench

* fix summarizer

* remove repeated code

* remove chinese comments
2024-04-07 15:46:20 +08:00
.github update links and checkers (#890) 2024-03-13 11:01:35 +08:00
configs [Feature] Add ATC Choice Version (#1019) 2024-04-07 15:46:20 +08:00
docs updates docs (#1015) 2024-04-02 10:30:04 +08:00
opencompass [Feature] Add ATC Choice Version (#1019) 2024-04-07 15:46:20 +08:00
requirements [Feature] Support AlpacaEval_V2 (#1006) 2024-03-28 16:49:04 +08:00
tests [Fix] meta template & unit tests (#170) 2023-08-10 16:49:13 +08:00
tools [Sync] Sync Internal (#941) 2024-03-04 14:42:36 +08:00
.codespellrc [Sync] update github token (#475) 2023-10-13 06:50:54 -05:00
.gitignore [Sync] Sync Internal (#941) 2024-03-04 14:42:36 +08:00
.owners.yml [Misc] Update owners (#961) 2024-03-12 11:51:25 +08:00
.pre-commit-config-zh-cn.yaml [Sync] Merge branch 'dev' into zfz/update-keyset-demo (#876) 2024-02-05 23:29:10 +08:00
.pre-commit-config.yaml [Sync] Merge branch 'dev' into zfz/update-keyset-demo (#876) 2024-02-05 23:29:10 +08:00
LICENSE [Docs] add en docs (#15) 2023-07-06 12:58:44 +08:00
README_zh-CN.md [Docs] Update README (#956) 2024-03-12 11:40:34 +08:00
README.md [Docs] Update README (#956) 2024-03-12 11:40:34 +08:00
requirements.txt [Fix] requirements (#229) 2023-08-18 14:34:20 +08:00
run.py [Feature] Add multi-model judge and fix some problems (#1016) 2024-04-02 11:52:06 +08:00
setup.py Add author as: author='OpenCompass Contributors' (#578) 2023-11-13 17:45:13 +08:00

👋 join us on Discord and WeChat

📣 OpenCompass 2.0

We are thrilled to introduce OpenCompass 2.0, an advanced suite featuring three key components: CompassKit, CompassHub, and CompassRank. oc20

CompassRank has been significantly enhanced into the leaderboards that now incorporates both open-source benchmarks and proprietary benchmarks. This upgrade allows for a more comprehensive evaluation of models across the industry.

CompassHub presents a pioneering benchmark browser interface, designed to simplify and expedite the exploration and utilization of an extensive array of benchmarks for researchers and practitioners alike. To enhance the visibility of your own benchmark within the community, we warmly invite you to contribute it to CompassHub. You may initiate the submission process by clicking here.

CompassKit is a powerful collection of evaluation toolkits specifically tailored for Large Language Models and Large Vision-language Models. It provides an extensive set of tools to assess and measure the performance of these complex models effectively. Welcome to try our toolkits for in your research and products.

🧭 Welcome

to OpenCompass!

Just like a compass guides us on our journey, OpenCompass will guide you through the complex landscape of evaluating large language models. With its powerful algorithms and intuitive interface, OpenCompass makes it easy to assess the quality and effectiveness of your NLP models.

🚩🚩🚩 Explore opportunities at OpenCompass! We're currently hiring full-time researchers/engineers and interns. If you're passionate about LLM and OpenCompass, don't hesitate to reach out to us via email. We'd love to hear from you!

🔥🔥🔥 We are delighted to announce that the OpenCompass has been recommended by the Meta AI, click Get Started of Llama for more information.

Attention
We launch the OpenCompass Collaboration project, welcome to support diverse evaluation benchmarks into OpenCompass! Clike Issue for more information. Let's work together to build a more powerful OpenCompass toolkit!

🚀 What's New

  • [2024.02.29] We supported the MT-Bench, AlpacalEval and AlignBench, more information can be found here 🔥🔥🔥.
  • [2024.01.30] We release OpenCompass 2.0. Click CompassKit, CompassHub, and CompassRank for more information ! 🔥🔥🔥.
  • [2024.01.17] We supported the evaluation of InternLM2 and InternLM2-Chat, InternLM2 showed extremely strong performance in these tests, welcome to try! 🔥🔥🔥.
  • [2024.01.17] We supported the needle in a haystack test with multiple needles, more information can be found here 🔥🔥🔥.
  • [2023.12.28] We have enabled seamless evaluation of all models developed using LLaMA2-Accessory, a powerful toolkit for comprehensive LLM development.
  • [2023.12.22] We have released T-Eval, a step-by-step evaluation benchmark to gauge your LLMs on tool utilization. Welcome to our Leaderboard for more details!

More

Introduction

image

OpenCompass is a one-stop platform for large model evaluation, aiming to provide a fair, open, and reproducible benchmark for large model evaluation. Its main features include:

  • Comprehensive support for models and datasets: Pre-support for 20+ HuggingFace and API models, a model evaluation scheme of 70+ datasets with about 400,000 questions, comprehensively evaluating the capabilities of the models in five dimensions.

  • Efficient distributed evaluation: One line command to implement task division and distributed evaluation, completing the full evaluation of billion-scale models in just a few hours.

  • Diversified evaluation paradigms: Support for zero-shot, few-shot, and chain-of-thought evaluations, combined with standard or dialogue-type prompt templates, to easily stimulate the maximum performance of various models.

  • Modular design with high extensibility: Want to add new models or datasets, customize an advanced task division strategy, or even support a new cluster management system? Everything about OpenCompass can be easily expanded!

  • Experiment management and reporting mechanism: Use config files to fully record each experiment, and support real-time reporting of results.

📊 Leaderboard

We provide OpenCompass Leaderboard for the community to rank all public models and API models. If you would like to join the evaluation, please provide the model repository URL or a standard API interface to the email address opencompass@pjlab.org.cn.

🔝Back to top

🛠️ Installation

Below are the steps for quick installation and datasets preparation.

💻 Environment Setup

Open-source Models with GPU

conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y
conda activate opencompass
git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .

API Models with CPU-only

conda create -n opencompass python=3.10 pytorch torchvision torchaudio cpuonly -c pytorch -y
conda activate opencompass
git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .
# also please install requiresments packages via `pip install -r requirements/api.txt` for API models if needed.

📂 Data Preparation

# Download dataset to data/ folder
wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-core-20240207.zip
unzip OpenCompassData-core-20240207.zip

Some third-party features, like Humaneval and Llama, may require additional steps to work properly, for detailed steps please refer to the Installation Guide.

🔝Back to top

🏗️ Evaluation

After ensuring that OpenCompass is installed correctly according to the above steps and the datasets are prepared, you can evaluate the performance of the LLaMA-7b model on the MMLU and C-Eval datasets using the following command:

python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl

OpenCompass has predefined configurations for many models and datasets. You can list all available model and dataset configurations using the tools.

# List all configurations
python tools/list_configs.py
# List all configurations related to llama and mmlu
python tools/list_configs.py llama mmlu

You can also evaluate other HuggingFace models via command line. Taking LLaMA-7b as an example:

python run.py --datasets ceval_ppl mmlu_ppl \
--hf-path huggyllama/llama-7b \  # HuggingFace model path
--model-kwargs device_map='auto' \  # Arguments for model construction
--tokenizer-kwargs padding_side='left' truncation='left' use_fast=False \  # Arguments for tokenizer construction
--max-out-len 100 \  # Maximum number of tokens generated
--max-seq-len 2048 \  # Maximum sequence length the model can accept
--batch-size 8 \  # Batch size
--no-batch-padding \  # Don't enable batch padding, infer through for loop to avoid performance loss
--num-gpus 1  # Number of minimum required GPUs

Note


To run the command above, you will need to remove the comments starting from # first.

Through the command line or configuration files, OpenCompass also supports evaluating APIs or custom models, as well as more diversified evaluation strategies. Please read the Quick Start to learn how to run an evaluation task.

🔝Back to top

📖 Dataset Support

Language Knowledge Reasoning Examination
Word Definition
  • WiC
  • SummEdits
Idiom Learning
  • CHID
Semantic Similarity
  • AFQMC
  • BUSTM
Coreference Resolution
  • CLUEWSC
  • WSC
  • WinoGrande
Translation
  • Flores
  • IWSLT2017
Multi-language Question Answering
  • TyDi-QA
  • XCOPA
Multi-language Summary
  • XLSum
Knowledge Question Answering
  • BoolQ
  • CommonSenseQA
  • NaturalQuestions
  • TriviaQA
Textual Entailment
  • CMNLI
  • OCNLI
  • OCNLI_FC
  • AX-b
  • AX-g
  • CB
  • RTE
  • ANLI
Commonsense Reasoning
  • StoryCloze
  • COPA
  • ReCoRD
  • HellaSwag
  • PIQA
  • SIQA
Mathematical Reasoning
  • MATH
  • GSM8K
Theorem Application
  • TheoremQA
  • StrategyQA
  • SciBench
Comprehensive Reasoning
  • BBH
Junior High, High School, University, Professional Examinations
  • C-Eval
  • AGIEval
  • MMLU
  • GAOKAO-Bench
  • CMMLU
  • ARC
  • Xiezhi
Medical Examinations
  • CMB
Understanding Long Context Safety Code
Reading Comprehension
  • C3
  • CMRC
  • DRCD
  • MultiRC
  • RACE
  • DROP
  • OpenBookQA
  • SQuAD2.0
Content Summary
  • CSL
  • LCSTS
  • XSum
  • SummScreen
Content Analysis
  • EPRSTMT
  • LAMBADA
  • TNEWS
Long Context Understanding
  • LEval
  • LongBench
  • GovReports
  • NarrativeQA
  • Qasper
Safety
  • CivilComments
  • CrowsPairs
  • CValues
  • JigsawMultilingual
  • TruthfulQA
Robustness
  • AdvGLUE
Code
  • HumanEval
  • HumanEvalX
  • MBPP
  • APPs
  • DS1000

📖 Model Support

Open-source Models API Models
  • OpenAI
  • Gemini
  • Claude
  • ZhipuAI(ChatGLM)
  • Baichuan
  • ByteDance(YunQue)
  • Huawei(PanGu)
  • 360
  • Baidu(ERNIEBot)
  • MiniMax(ABAB-Chat)
  • SenseTime(nova)
  • Xunfei(Spark)
  • ……

🔝Back to top

🔜 Roadmap

  • Subjective Evaluation
    • Release CompassAreana
    • Subjective evaluation.
  • Long-context
    • Long-context evaluation with extensive datasets.
    • Long-context leaderboard.
  • Coding
    • Coding evaluation leaderboard.
    • Non-python language evaluation service.
  • Agent
    • Support various agenet framework.
    • Evaluation of tool use of the LLMs.
  • Robustness
    • Support various attack method

👷‍♂️ Contributing

We appreciate all contributions to improving OpenCompass. Please refer to the contributing guideline for the best practice.

🤝 Acknowledgements

Some code in this project is cited and modified from OpenICL.

Some datasets and prompt implementations are modified from chain-of-thought-hub and instruct-eval.

🖊️ Citation

@misc{2023opencompass,
    title={OpenCompass: A Universal Evaluation Platform for Foundation Models},
    author={OpenCompass Contributors},
    howpublished = {\url{https://github.com/open-compass/opencompass}},
    year={2023}
}

🔝Back to top