Update readme (#6)

* update readme * update readme * update readme * update readme * update readme * update readme * update readme * update readme * update readme * update readme * update readme * Update README.md Add description for name * Update README_zh-CN.md Update introduction * Update README_zh-CN.md * Update README_zh-CN.md * update readme * Update README.md Add Leaderboard * Update README.md * Update README_zh-CN.md --------- Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2025-05-30 16:03:24 +08:00 · 2023-07-06 12:14:23 +08:00 · 2023-07-06 12:14:23 +08:00 · 2d0b184bb6
commit 2d0b184bb6
parent c7ba1f70b6
5 changed files with 742 additions and 82 deletions
--- a/README.md
+++ b/README.md
@ -0,0 +1,313 @@
+<div align="center">
+  <img src="docs/en/_static/image/logo.svg" width="500px"/>
+  <br />
+  <br />
+
+[![docs](https://readthedocs.org/projects/opencompass/badge/?version=dev-1.x)](https://opencompass.readthedocs.io/en/dev-1.x/?badge=dev-1.x)
+[![license](https://img.shields.io/github/license/IntenLM/opencompass.svg)](https://github.com/InternLM/opencompass/blob/main/LICENSE)
+[![PyPI](https://badge.fury.io/py/opencompass.svg)](https://pypi.org/project/opencompass/)
+
+[📘Documentation](https://opencompass.readthedocs.io/en/latest/) |
+[🛠️Installation](https://opencompass.readthedocs.io/en/latest/get_started/install.html) |
+[🤔Reporting Issues](https://github.com/InternLM/opencompass/issues/new/choose)
+
+English | [简体中文](README_zh-CN.md)
+
+</div>
+
+Welcome to **OpenCompass**! 
+
+Just like a compass guides us on our journey, OpenCompass will guide you through the complex landscape of evaluating large language models. With its powerful algorithms and intuitive interface, OpenCompass makes it easy to assess the quality and effectiveness of your NLP models.
+
+## Introduction
+
+OpenCompass is a one-stop platform for large model evaluation, aiming to provide a fair, open, and reproducible benchmark for large model evaluation. Its main features includes:
+
+- **Comprehensive support for models and datasets**: Pre-support for 20+ HuggingFace and API models, a model evaluation scheme of 50+ datasets with about 300,000 questions, comprehensively evaluating the capabilities of the models in five dimensions.
+
+- **Efficient distributed evaluation**: One line command to implement task division and distributed evaluation, completing the full evaluation of billion-scale models in just a few hours.
+
+- **Diversified evaluation paradigms**: Support for zero-shot, few-shot, and chain-of-thought evaluations, combined with standard or dialogue type prompt templates, to easily stimulate the maximum performance of various models.
+
+- **Modular design with high extensibility**: Want to add new models or datasets, customize an advanced task division strategy, or even support a new cluster management system? Everything about OpenCompass can be easily expanded!
+
+- **Experiment management and reporting mechanism**: Use config files to fully record each experiment, support real-time reporting of results.
+
+## Leaderboard
+
+We provide [OpenCompass Leaderbaord](https://opencompass.org.cn/rank) for community to rank all public models and API models. If you would like to join the evaluation, please provide the model repository URL or a standard API interface to the email address `opencompass@pjlab.org.cn`.
+
+[![image](https://github.com/InternLM/OpenCompass/assets/7881589/6b56c297-77c0-4e1a-9acc-24a45c5a734a)](https://opencompass.org.cn/rank)
+
+
+
+## Dataset Support
+
+<table align="center">
+  <tbody>
+    <tr align="center" valign="bottom">
+      <td>
+        <b>Language</b>
+      </td>
+      <td>
+        <b>Knowledge</b>
+      </td>
+      <td>
+        <b>Reasoning</b>
+      </td>
+      <td>
+        <b>Exam</b>
+      </td>
+      <td>
+        <b>Understanding</b>
+      </td>
+    </tr>
+    <tr valign="top">
+      <td>
+<details open>
+<summary><b>Word Definition</b></summary>
+
+- WiC
+- SummEdits
+
+</details>
+
+<details open>
+<summary><b>Idiom Learning</b></summary>
+
+- CHID
+
+</details>
+
+<details open>
+<summary><b>Semantic Similarity</b></summary>
+
+- AFQMC
+- BUSTM
+
+</details>
+
+<details open>
+<summary><b>Coreference Resolution</b></summary>
+
+- CLUEWSC
+- WSC
+- WinoGrande
+
+</details>
+
+<details open>
+<summary><b>Translation</b></summary>
+
+- Flores
+
+</details>
+      </td>
+      <td>
+<details open>
+<summary><b>Knowledge Question Answering</b></summary>
+
+- BoolQ
+- CommonSenseQA
+- NaturalQuestion
+- TrivialQA
+
+</details>
+
+<details open>
+<summary><b>Multi-language Question Answering</b></summary>
+
+- TyDi-QA
+
+</details>
+      </td>
+      <td>
+<details open>
+<summary><b>Textual Entailment</b></summary>
+
+- CMNLI
+- OCNLI
+- OCNLI_FC
+- AX-b
+- AX-g
+- CB
+- RTE
+
+</details>
+
+<details open>
+<summary><b>Commonsense Reasoning</b></summary>
+
+- StoryCloze
+- StoryCloze-CN (coming soon)
+- COPA
+- ReCoRD
+- HellaSwag
+- PIQA
+- SIQA
+
+</details>
+
+<details open>
+<summary><b>Mathematical Reasoning</b></summary>
+
+- MATH
+- GSM8K
+
+</details>
+
+<details open>
+<summary><b>Theorem Application</b></summary>
+
+- TheoremQA
+
+</details>
+
+<details open>
+<summary><b>Code</b></summary>
+
+- HumanEval
+- MBPP
+
+</details>
+
+<details open>
+<summary><b>Comprehensive Reasoning</b></summary>
+
+- BBH
+
+</details>
+      </td>
+      <td>
+<details open>
+<summary><b>Junior High, High School, University, Professional Examinations</b></summary>
+
+- GAOKAO-2023
+- CEval
+- AGIEval
+- MMLU
+- GAOKAO-Bench
+- MMLU-CN (coming soon)
+- ARC
+
+</details>
+      </td>
+      <td>
+<details open>
+<summary><b>Reading Comprehension</b></summary>
+
+- C3
+- CMRC
+- DRCD
+- MultiRC
+- RACE
+
+</details>
+
+<details open>
+<summary><b>Content Summary</b></summary>
+
+- CSL
+- LCSTS
+- XSum
+
+</details>
+
+<details open>
+<summary><b>Content Analysis</b></summary>
+
+- EPRSTMT
+- LAMBADA
+- TNEWS
+
+</details>
+      </td>
+    </tr>
+</td>
+    </tr>
+  </tbody>
+</table>
+
+## Model Support
+
+<table align="center">
+  <tbody>
+    <tr align="center" valign="bottom">
+      <td>
+        <b>HuggingFace Models</b>
+      </td>
+      <td>
+        <b>API Models</b>
+      </td>
+      <td>
+        <b>Custom Models</b>
+      </td>
+    </tr>
+    <tr valign="top">
+      <td>
+        
+- InternLM
+- LLaMA
+- Vicuna
+- Alpaca
+- Baichuan
+- WizardLM
+- ChatGLM-6B
+- ChatGLM2-6B
+- MPT
+- Falcon
+- TigerBot
+- MOSS
+- ……
+
+</td>
+<td>
+
+- OpenAI
+- Claude (coming soon)
+- PaLM (coming soon)
+- ……
+
+</td>
+<td>
+
+- GLM
+- ……
+
+</td>
+</tr>
+  </tbody>
+</table>
+
+## Installation
+
+Below are the steps for quick installation. Some third-party features may require additional steps to work properly, for detailed steps please refer to the [Installation Guide](https://opencompass.readthedocs.io/en/latest/get_started.html).
+
+```Python
+conda create --name opencompass python=3.8 pytorch torchvision -c pytorch -y
+conda activate opencompass
+git clone https://github.com/InternLM/opencompass opencompass
+cd opencompass
+pip install -e .
+# Download dataset to data/ folder
+# TODO: ....
+```
+
+## Evaluation
+
+Please read the [Quick Start](https://opencompass.readthedocs.io/en/latest/get_started.html) to learn how to run an evaluation task.
+
+## Acknowledgements
+
+Some code in this project is cited and modified from [OpenICL](https://github.com/Shark-NLP/OpenICL).
+
+## Citation
+
+```bibtex
+@misc{2023opencompass,
+    title={OpenCompass: A Universal Evaluation Platform for Foundation Models},
+    author={OpenCompass Contributors},
+    howpublished = {\url{https://github.com/InternLM/OpenCompass}},
+    year={2023}
+}
+```
--- a/README_zh-CN.md
+++ b/README_zh-CN.md
@ -1,10 +1,13 @@
 <div align="center">
-  <img src="https://user-images.githubusercontent.com/22607038/250798681-b52045d2-cedd-4070-84e2-410903ac404f.png" width="500px"/>
+  <img src="docs/zh_cn/_static/image/logo.svg" width="500px"/>
+  <br />
+  <br />

 [![docs](https://readthedocs.org/projects/opencompass/badge/?version=dev-1.x)](https://opencompass.readthedocs.io/en/dev-1.x/?badge=dev-1.x)
 [![license](https://img.shields.io/github/license/IntenLM/opencompass.svg)](https://github.com/InternLM/opencompass/blob/main/LICENSE)
 [![PyPI](https://badge.fury.io/py/opencompass.svg)](https://pypi.org/project/opencompass/)

+[🌐Website](https://opencompass.org.cn/) |
 [📘Documentation](https://opencompass.readthedocs.io/en/latest/) |
 [🛠️Installation](https://opencompass.readthedocs.io/en/latest/get_started/install.html) |
 [🤔Reporting Issues](https://github.com/InternLM/opencompass/issues/new/choose)
@ -13,25 +16,272 @@

 </div>

+欢迎来到OpenCompass！
+
+就像指南针在我们的旅程中为我们导航一样，我们希望OpenCompass能够帮助你穿越评估大型语言模型的重重迷雾。OpenCompass提供丰富的算法和功能支持，期待OpenCompass能够帮助社区更便捷地对NLP模型的性能进行公平全面的评估。
+
 ## 介绍

-OpenCompass 是面向大模型评测的一站式平台，旨在提供一套公平、公开、可复现的大模型评测基准方案。其主要特点如下：
+OpenCompass 是面向大模型评测的一站式平台。其主要特点如下：

- **模型及数据集的全方位支持**：预支持 20+ HuggingFace 及 API 模型，并提供 50+ 个数据集约 30 万题的的模型评测方案，6 大维度的能力全面评测。
+- **开源可复现**：提供公平、公开、可复现的大模型评测方案

- **高效分布式评测**：一行命令实现任务分割和分布式评测，数小时即可完成千亿模型全量评测\*。
+- **全面的能力维度**：五大维度设计，提供 50+ 个数据集约 30 万题的的模型评测方案，全面评估模型能力

- **多样化评测范式**：支持零样本、小样本及思维链评测，结合标准型或对话型提示词模板，轻松激发各种模型最大性能。
+- **丰富的模型支持**：已支持 20+ HuggingFace 及 API 模型

- **易于扩展的模块化设计**：想增加新模型或数据集？想要自定义更高级的任务分割策略，甚至接入新的集群管理系统？OpenCompass 的一切均可轻松扩展！
+- **分布式高效评测**：一行命令实现任务分割和分布式评测，数小时即可完成千亿模型全量评测

- **完善的实验记录及上报机制**：使用配置文件完整记录每一次实验，关键信息有迹可循；结果实时上报飞书机器人，第一时间知晓实验情况。
+- **多样化评测范式**：支持零样本、小样本及思维链评测，结合标准型或对话型提示词模板，轻松激发各种模型最大性能

-## 模型能力排名
+- **灵活化拓展**：想增加新模型或数据集？想要自定义更高级的任务分割策略，甚至接入新的集群管理系统？OpenCompass 的一切均可轻松扩展！

-## 能力维度 & 模型支持
+## 性能榜单

-## 安装
+我们将陆续提供开源模型和API模型的具体性能榜单，请见 [OpenCompass Leaderbaord](https://opencompass.org.cn/rank) 。如需加入评测，请提供模型仓库地址或标准的 API 接口至邮箱  `opencompass@pjlab.org.cn`.
+
+
+![image](https://github.com/InternLM/OpenCompass/assets/7881589/fddc8ab4-d2bd-429d-89f0-4ca90606599a)
+
+
+## 数据集支持
+
+<table align="center">
+  <tbody>
+    <tr align="center" valign="bottom">
+      <td>
+        <b>语言</b>
+      </td>
+      <td>
+        <b>知识</b>
+      </td>
+      <td>
+        <b>推理</b>
+      </td>
+      <td>
+        <b>考试</b>
+      </td>
+      <td>
+        <b>理解</b>
+      </td>
+    </tr>
+    <tr valign="top">
+      <td>
+<details open>
+<summary><b>字词释义</b></summary>
+
+- WiC
+- SummEdits
+
+</details>
+
+<details open>
+<summary><b>成语习语</b></summary>
+
+- CHID
+
+</details>
+
+<details open>
+<summary><b>语义相似度</b></summary>
+
+- AFQMC
+- BUSTM
+
+</details>
+
+<details open>
+<summary><b>指代消解</b></summary>
+
+- CLUEWSC
+- WSC
+- WinoGrande
+
+</details>
+
+<details open>
+<summary><b>翻译</b></summary>
+
+- Flores
+
+</details>
+      </td>
+      <td>
+<details open>
+<summary><b>知识问答</b></summary>
+
+- BoolQ
+- CommonSenseQA
+- NaturalQuestion
+- TrivialQA
+
+</details>
+
+<details open>
+<summary><b>多语种问答</b></summary>
+
+- TyDi-QA
+
+</details>
+      </td>
+      <td>
+<details open>
+<summary><b>文本蕴含</b></summary>
+
+- CMNLI
+- OCNLI
+- OCNLI_FC
+- AX-b
+- AX-g
+- CB
+- RTE
+
+</details>
+
+<details open>
+<summary><b>常识推理</b></summary>
+
+- StoryCloze
+- StoryCloze-CN（即将上线）
+- COPA
+- ReCoRD
+- HellaSwag
+- PIQA
+- SIQA
+
+</details>
+
+<details open>
+<summary><b>数学推理</b></summary>
+
+- MATH
+- GSM8K
+
+</details>
+
+<details open>
+<summary><b>定理应用</b></summary>
+
+- TheoremQA
+
+</details>
+
+<details open>
+<summary><b>代码</b></summary>
+
+- HumanEval
+- MBPP
+
+</details>
+
+<details open>
+<summary><b>综合推理</b></summary>
+
+- BBH
+
+</details>
+      </td>
+      <td>
+<details open>
+<summary><b>初中/高中/大学/职业考试</b></summary>
+
+- GAOKAO-2023
+- CEval
+- AGIEval
+- MMLU
+- GAOKAO-Bench
+- MMLU-CN (即将上线)
+- ARC
+
+</details>
+      </td>
+      <td>
+<details open>
+<summary><b>阅读理解</b></summary>
+
+- C3
+- CMRC
+- DRCD
+- MultiRC
+- RACE
+
+</details>
+
+<details open>
+<summary><b>内容总结</b></summary>
+
+- CSL
+- LCSTS
+- XSum
+
+</details>
+
+<details open>
+<summary><b>内容分析</b></summary>
+
+- EPRSTMT
+- LAMBADA
+- TNEWS
+
+</details>
+      </td>
+    </tr>
+</td>
+    </tr>
+  </tbody>
+</table>
+
+## 模型支持
+
+<table align="center">
+  <tbody>
+    <tr align="center" valign="bottom">
+      <td>
+        <b>HuggingFace 模型</b>
+      </td>
+      <td>
+        <b>API 模型</b>
+      </td>
+      <td>
+        <b>自定义模型</b>
+      </td>
+    </tr>
+    <tr valign="top">
+      <td>
+
+- LLaMA
+- Vicuna
+- Alpaca
+- Baichuan
+- WizardLM
+- ChatGLM-6B
+- ChatGLM2-6B
+- MPT
+- Falcon
+- TigerBot
+- MOSS
+- ……
+
+</td>
+<td>
+
+- OpenAI
+- Claude (即将推出)
+- PaLM (即将推出)
+- ……
+
+</td>
+<td>
+
+- GLM
+- ……
+
+</td>
+</tr>
+  </tbody>
+</table>
+
+# 安装

 下面展示了快速安装的步骤。有部分第三方功能可能需要额外步骤才能正常运行，详细步骤请参考[安装指南](https://opencompass.readthedocs.io/zh_cn/latest/get_started.html)。

@ -40,7 +290,6 @@ conda create --name opencompass python=3.8 pytorch torchvision -c pytorch -y
 conda activate opencompass
 git clone https://github.com/InternLM/opencompass opencompass
 cd opencompass
-pip install -r requirements/runtime.txt
 pip install -e .
 # 下载数据集到 data/ 处
 # TODO: ....
@ -50,7 +299,7 @@ pip install -e .

 请阅读[快速上手](https://opencompass.readthedocs.io/zh_cn/latest/get_started.html)了解如何运行一个评测任务。

-## 致谢
+## 致谢

 该项目部分的代码引用并修改自 [OpenICL](https://github.com/Shark-NLP/OpenICL)。

--- a/docs/en/_static/image/logo.svg
+++ b/docs/en/_static/image/logo.svg
--- a/docs/zh_cn/_static/image/logo.svg
+++ b/docs/zh_cn/_static/image/logo.svg
--- a/requirements.txt
+++ b/requirements.txt
@ -1,26 +1,26 @@
 accelerate>=0.19.0
+boto3
+colossalai
+cpm_kernels
 datasets>=2.12.0
 evaluate>=0.3.0
+fairscale
 faiss_gpu==1.7.2
+jieba
+mmengine>0.8.0
 nltk==3.8
 numpy==1.23.4
 openai==0.27.1
+openai
+pandas<2.0.0
 rank_bm25==0.2.2
 requests==2.28.1
 scikit_learn==1.2.1
 sentence_transformers==2.2.2
+tabulate
+tabulate
+tiktoken
+tokenizers>=0.13.3
 torch>=1.13.1
 tqdm==4.64.1
 transformers>=4.29.1
-openai
-mmengine
-jieba
-pandas<2.0.0
-cpm_kernels
-tokenizers>=0.13.3
-tabulate
-fairscale
-colossalai
-tabulate
-boto3
-tiktoken