diff --git a/README.md b/README.md new file mode 100644 index 00000000..f8766154 --- /dev/null +++ b/README.md @@ -0,0 +1,313 @@ +
+ +
+
+ +[![docs](https://readthedocs.org/projects/opencompass/badge/?version=dev-1.x)](https://opencompass.readthedocs.io/en/dev-1.x/?badge=dev-1.x) +[![license](https://img.shields.io/github/license/IntenLM/opencompass.svg)](https://github.com/InternLM/opencompass/blob/main/LICENSE) +[![PyPI](https://badge.fury.io/py/opencompass.svg)](https://pypi.org/project/opencompass/) + +[📘Documentation](https://opencompass.readthedocs.io/en/latest/) | +[🛠️Installation](https://opencompass.readthedocs.io/en/latest/get_started/install.html) | +[🤔Reporting Issues](https://github.com/InternLM/opencompass/issues/new/choose) + +English | [简体中文](README_zh-CN.md) + +
+ +Welcome to **OpenCompass**! + +Just like a compass guides us on our journey, OpenCompass will guide you through the complex landscape of evaluating large language models. With its powerful algorithms and intuitive interface, OpenCompass makes it easy to assess the quality and effectiveness of your NLP models. + +## Introduction + +OpenCompass is a one-stop platform for large model evaluation, aiming to provide a fair, open, and reproducible benchmark for large model evaluation. Its main features includes: + +- **Comprehensive support for models and datasets**: Pre-support for 20+ HuggingFace and API models, a model evaluation scheme of 50+ datasets with about 300,000 questions, comprehensively evaluating the capabilities of the models in five dimensions. + +- **Efficient distributed evaluation**: One line command to implement task division and distributed evaluation, completing the full evaluation of billion-scale models in just a few hours. + +- **Diversified evaluation paradigms**: Support for zero-shot, few-shot, and chain-of-thought evaluations, combined with standard or dialogue type prompt templates, to easily stimulate the maximum performance of various models. + +- **Modular design with high extensibility**: Want to add new models or datasets, customize an advanced task division strategy, or even support a new cluster management system? Everything about OpenCompass can be easily expanded! + +- **Experiment management and reporting mechanism**: Use config files to fully record each experiment, support real-time reporting of results. + +## Leaderboard + +We provide [OpenCompass Leaderbaord](https://opencompass.org.cn/rank) for community to rank all public models and API models. If you would like to join the evaluation, please provide the model repository URL or a standard API interface to the email address `opencompass@pjlab.org.cn`. + +[![image](https://github.com/InternLM/OpenCompass/assets/7881589/6b56c297-77c0-4e1a-9acc-24a45c5a734a)](https://opencompass.org.cn/rank) + + + +## Dataset Support + + + + + + + + + + + + + + + + + + + + +
+ Language + + Knowledge + + Reasoning + + Exam + + Understanding +
+
+Word Definition + +- WiC +- SummEdits + +
+ +
+Idiom Learning + +- CHID + +
+ +
+Semantic Similarity + +- AFQMC +- BUSTM + +
+ +
+Coreference Resolution + +- CLUEWSC +- WSC +- WinoGrande + +
+ +
+Translation + +- Flores + +
+ 		+
+Knowledge Question Answering + +- BoolQ +- CommonSenseQA +- NaturalQuestion +- TrivialQA + +
+ +
+Multi-language Question Answering + +- TyDi-QA + +
+ 		+
+Textual Entailment + +- CMNLI +- OCNLI +- OCNLI_FC +- AX-b +- AX-g +- CB +- RTE + +
+ +
+Commonsense Reasoning + +- StoryCloze +- StoryCloze-CN (coming soon) +- COPA +- ReCoRD +- HellaSwag +- PIQA +- SIQA + +
+ +
+Mathematical Reasoning + +- MATH +- GSM8K + +
+ +
+Theorem Application + +- TheoremQA + +
+ +
+Code + +- HumanEval +- MBPP + +
+ +
+Comprehensive Reasoning + +- BBH + +
+ 		+
+Junior High, High School, University, Professional Examinations + +- GAOKAO-2023 +- CEval +- AGIEval +- MMLU +- GAOKAO-Bench +- MMLU-CN (coming soon) +- ARC + +
+ 		+
+Reading Comprehension + +- C3 +- CMRC +- DRCD +- MultiRC +- RACE + +
+ +
+Content Summary + +- CSL +- LCSTS +- XSum + +
+ +
+Content Analysis + +- EPRSTMT +- LAMBADA +- TNEWS + +
+
+ +## Model Support + + + + + + + + + + + + + + +
+ HuggingFace Models + + API Models + + Custom Models +
+ +- InternLM +- LLaMA +- Vicuna +- Alpaca +- Baichuan +- WizardLM +- ChatGLM-6B +- ChatGLM2-6B +- MPT +- Falcon +- TigerBot +- MOSS +- …… + + + +- OpenAI +- Claude (coming soon) +- PaLM (coming soon) +- …… + + + +- GLM +- …… + +
+ +## Installation + +Below are the steps for quick installation. Some third-party features may require additional steps to work properly, for detailed steps please refer to the [Installation Guide](https://opencompass.readthedocs.io/en/latest/get_started.html). + +```Python +conda create --name opencompass python=3.8 pytorch torchvision -c pytorch -y +conda activate opencompass +git clone https://github.com/InternLM/opencompass opencompass +cd opencompass +pip install -e . +# Download dataset to data/ folder +# TODO: .... +``` + +## Evaluation + +Please read the [Quick Start](https://opencompass.readthedocs.io/en/latest/get_started.html) to learn how to run an evaluation task. + +## Acknowledgements + +Some code in this project is cited and modified from [OpenICL](https://github.com/Shark-NLP/OpenICL). + +## Citation + +```bibtex +@misc{2023opencompass, + title={OpenCompass: A Universal Evaluation Platform for Foundation Models}, + author={OpenCompass Contributors}, + howpublished = {\url{https://github.com/InternLM/OpenCompass}}, + year={2023} +} +``` diff --git a/README_zh-CN.md b/README_zh-CN.md index 48e8ef12..fb97d1ed 100644 --- a/README_zh-CN.md +++ b/README_zh-CN.md @@ -1,10 +1,13 @@
- + +
+
[![docs](https://readthedocs.org/projects/opencompass/badge/?version=dev-1.x)](https://opencompass.readthedocs.io/en/dev-1.x/?badge=dev-1.x) [![license](https://img.shields.io/github/license/IntenLM/opencompass.svg)](https://github.com/InternLM/opencompass/blob/main/LICENSE) [![PyPI](https://badge.fury.io/py/opencompass.svg)](https://pypi.org/project/opencompass/) +[🌐Website](https://opencompass.org.cn/) | [📘Documentation](https://opencompass.readthedocs.io/en/latest/) | [🛠️Installation](https://opencompass.readthedocs.io/en/latest/get_started/install.html) | [🤔Reporting Issues](https://github.com/InternLM/opencompass/issues/new/choose) @@ -13,25 +16,272 @@
+欢迎来到OpenCompass！ + +就像指南针在我们的旅程中为我们导航一样，我们希望OpenCompass能够帮助你穿越评估大型语言模型的重重迷雾。OpenCompass提供丰富的算法和功能支持，期待OpenCompass能够帮助社区更便捷地对NLP模型的性能进行公平全面的评估。 + ## 介绍 -OpenCompass 是面向大模型评测的一站式平台，旨在提供一套公平、公开、可复现的大模型评测基准方案。其主要特点如下： +OpenCompass 是面向大模型评测的一站式平台。其主要特点如下： -- **模型及数据集的全方位支持**：预支持 20+ HuggingFace 及 API 模型，并提供 50+ 个数据集约 30 万题的的模型评测方案，6 大维度的能力全面评测。 +- **开源可复现**：提供公平、公开、可复现的大模型评测方案 -- **高效分布式评测**：一行命令实现任务分割和分布式评测，数小时即可完成千亿模型全量评测\*。 +- **全面的能力维度**：五大维度设计，提供 50+ 个数据集约 30 万题的的模型评测方案，全面评估模型能力 -- **多样化评测范式**：支持零样本、小样本及思维链评测，结合标准型或对话型提示词模板，轻松激发各种模型最大性能。 +- **丰富的模型支持**：已支持 20+ HuggingFace 及 API 模型 -- **易于扩展的模块化设计**：想增加新模型或数据集？想要自定义更高级的任务分割策略，甚至接入新的集群管理系统？OpenCompass 的一切均可轻松扩展！ +- **分布式高效评测**：一行命令实现任务分割和分布式评测，数小时即可完成千亿模型全量评测 -- **完善的实验记录及上报机制**：使用配置文件完整记录每一次实验，关键信息有迹可循；结果实时上报飞书机器人，第一时间知晓实验情况。 +- **多样化评测范式**：支持零样本、小样本及思维链评测，结合标准型或对话型提示词模板，轻松激发各种模型最大性能 -## 模型能力排名 +- **灵活化拓展**：想增加新模型或数据集？想要自定义更高级的任务分割策略，甚至接入新的集群管理系统？OpenCompass 的一切均可轻松扩展！ -## 能力维度 & 模型支持 +## 性能榜单 -## 安装 +我们将陆续提供开源模型和API模型的具体性能榜单，请见 [OpenCompass Leaderbaord](https://opencompass.org.cn/rank) 。如需加入评测，请提供模型仓库地址或标准的 API 接口至邮箱 `opencompass@pjlab.org.cn`. + + +![image](https://github.com/InternLM/OpenCompass/assets/7881589/fddc8ab4-d2bd-429d-89f0-4ca90606599a) + + +## 数据集支持 + + + + + + + + + + + + + + + + + + + + +
+ 语言 + + 知识 + + 推理 + + 考试 + + 理解 +
+
+字词释义 + +- WiC +- SummEdits + +
+ +
+成语习语 + +- CHID + +
+ +
+语义相似度 + +- AFQMC +- BUSTM + +
+ +
+指代消解 + +- CLUEWSC +- WSC +- WinoGrande + +
+ +
+翻译 + +- Flores + +
+ 		+
+知识问答 + +- BoolQ +- CommonSenseQA +- NaturalQuestion +- TrivialQA + +
+ +
+多语种问答 + +- TyDi-QA + +
+ 		+
+文本蕴含 + +- CMNLI +- OCNLI +- OCNLI_FC +- AX-b +- AX-g +- CB +- RTE + +
+ +
+常识推理 + +- StoryCloze +- StoryCloze-CN（即将上线） +- COPA +- ReCoRD +- HellaSwag +- PIQA +- SIQA + +
+ +
+数学推理 + +- MATH +- GSM8K + +
+ +
+定理应用 + +- TheoremQA + +
+ +
+代码 + +- HumanEval +- MBPP + +
+ +
+综合推理 + +- BBH + +
+ 		+
+初中/高中/大学/职业考试 + +- GAOKAO-2023 +- CEval +- AGIEval +- MMLU +- GAOKAO-Bench +- MMLU-CN (即将上线) +- ARC + +
+ 		+
+阅读理解 + +- C3 +- CMRC +- DRCD +- MultiRC +- RACE + +
+ +
+内容总结 + +- CSL +- LCSTS +- XSum + +
+ +
+内容分析 + +- EPRSTMT +- LAMBADA +- TNEWS + +
+
+ +## 模型支持 + + + + + + + + + + + + + + +
+ HuggingFace 模型 + + API 模型 + + 自定义模型 +
+ +- LLaMA +- Vicuna +- Alpaca +- Baichuan +- WizardLM +- ChatGLM-6B +- ChatGLM2-6B +- MPT +- Falcon +- TigerBot +- MOSS +- …… + + + +- OpenAI +- Claude (即将推出) +- PaLM (即将推出) +- …… + + + +- GLM +- …… + +
+ +# 安装 下面展示了快速安装的步骤。有部分第三方功能可能需要额外步骤才能正常运行，详细步骤请参考[安装指南](https://opencompass.readthedocs.io/zh_cn/latest/get_started.html)。 @@ -40,7 +290,6 @@ conda create --name opencompass python=3.8 pytorch torchvision -c pytorch -y conda activate opencompass git clone https://github.com/InternLM/opencompass opencompass cd opencompass -pip install -r requirements/runtime.txt pip install -e . # 下载数据集到 data/ 处 # TODO: .... @@ -50,7 +299,7 @@ pip install -e . 请阅读[快速上手](https://opencompass.readthedocs.io/zh_cn/latest/get_started.html)了解如何运行一个评测任务。 -## 致谢 +## 致谢 该项目部分的代码引用并修改自 [OpenICL](https://github.com/Shark-NLP/OpenICL)。 diff --git a/docs/en/_static/image/logo.svg b/docs/en/_static/image/logo.svg index 80648e23..fe19a0c7 100644 --- a/docs/en/_static/image/logo.svg +++ b/docs/en/_static/image/logo.svg @@ -1,30 +1,79 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + diff --git a/docs/zh_cn/_static/image/logo.svg b/docs/zh_cn/_static/image/logo.svg index 80648e23..fe19a0c7 100644 --- a/docs/zh_cn/_static/image/logo.svg +++ b/docs/zh_cn/_static/image/logo.svg @@ -1,30 +1,79 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + diff --git a/requirements.txt b/requirements.txt index 77b80bbe..e2950dbf 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,26 +1,26 @@ accelerate>=0.19.0 +boto3 +colossalai +cpm_kernels datasets>=2.12.0 evaluate>=0.3.0 +fairscale faiss_gpu==1.7.2 +jieba +mmengine>0.8.0 nltk==3.8 numpy==1.23.4 openai==0.27.1 +openai +pandas<2.0.0 rank_bm25==0.2.2 requests==2.28.1 scikit_learn==1.2.1 sentence_transformers==2.2.2 +tabulate +tabulate +tiktoken +tokenizers>=0.13.3 torch>=1.13.1 tqdm==4.64.1 transformers>=4.29.1 -openai -mmengine -jieba -pandas<2.0.0 -cpm_kernels -tokenizers>=0.13.3 -tabulate -fairscale -colossalai -tabulate -boto3 -tiktoken \ No newline at end of file