diff --git a/README.md b/README.md index 5858acc7..656a2365 100644 --- a/README.md +++ b/README.md @@ -324,6 +324,10 @@ Make sure you have installed OpenCompass correctly and prepared your datasets ac For more tutorials, please check our [Documentation](https://opencompass.readthedocs.io/en/latest/index.html). +## 👷‍♂️ Contributing + +We appreciate all contributions to improve OpenCompass. Please refer to the [contributing guideline](https://opencompass.readthedocs.io/en/latest/notes/contribution_guide.html) for the best practice. + ## 🤝 Acknowledgements Some code in this project is cited and modified from [OpenICL](https://github.com/Shark-NLP/OpenICL). diff --git a/README_zh-CN.md b/README_zh-CN.md index b36e8a40..0aeab87a 100644 --- a/README_zh-CN.md +++ b/README_zh-CN.md @@ -325,6 +325,10 @@ unzip OpenCompassData.zip 更多教程请查看我们的[文档](https://opencompass.readthedocs.io/zh_CN/latest/index.html)。 +## 👷‍♂️ 贡献 + +我们感谢所有的贡献者为改进和提升 OpenCompass 所作出的努力。请参考[贡献指南](https://opencompass.readthedocs.io/zh_CN/latest/notes/contribution_guide.html)来了解参与项目贡献的相关指引。 + ## 🤝 致谢 该项目部分的代码引用并修改自 [OpenICL](https://github.com/Shark-NLP/OpenICL)。 diff --git a/docs/en/_static/js/custom.js b/docs/en/_static/js/custom.js index 01293c8b..84da69d4 100644 --- a/docs/en/_static/js/custom.js +++ b/docs/en/_static/js/custom.js @@ -1,4 +1,4 @@ -var collapsedSections = ['Advanced Guides', 'Tools', 'User Guides', 'Notes']; +var collapsedSections = []; $(document).ready(function () { $('.model-summary').DataTable({ diff --git a/docs/en/index.rst b/docs/en/index.rst index f50497b2..a1c2dedc 100644 --- a/docs/en/index.rst +++ b/docs/en/index.rst @@ -51,10 +51,10 @@ We always welcome *PRs* and *Issues* for the betterment of OpenCompass. :maxdepth: 1 :caption: Prompt - prompt/few_shot.md - prompt/chain_of_thought.md + prompt/overview.md prompt/prompt_template.md prompt/meta_template.md + prompt/chain_of_thought.md .. _Tools: .. toctree:: diff --git a/docs/en/notes/contribution_guide.md b/docs/en/notes/contribution_guide.md index 431d3dbb..0787894b 100644 --- a/docs/en/notes/contribution_guide.md +++ b/docs/en/notes/contribution_guide.md @@ -1,10 +1,17 @@ # Contributing to OpenCompass - [Contributing to OpenCompass](#contributing-to-opencompass) - - [Workflow](#workflow) + - [What is PR](#what-is-pr) + - [Basic Workflow](#basic-workflow) + - [Procedures in detail](#procedures-in-detail) + - [1. Get the most recent codebase](#1-get-the-most-recent-codebase) + - [2. Checkout a new branch from `main` branch](#2-checkout-a-new-branch-from-main-branch) + - [3. Commit your changes](#3-commit-your-changes) + - [4. Push your changes to the forked repository and create a PR](#4-push-your-changes-to-the-forked-repository-and-create-a-pr) + - [5. Discuss and review your code](#5-discuss-and-review-your-code) + - [6. Merge your branch to `main` branch and delete the branch](#6--merge-your-branch-to-main-branch-and-delete-the-branch) - [Code style](#code-style) - [Python](#python) - - [Pre-commit Hook](#pre-commit-hook) Thanks for your interest in contributing to OpenCompass! All kinds of contributions are welcome, including but not limited to the following. @@ -12,21 +19,107 @@ Thanks for your interest in contributing to OpenCompass! All kinds of contributi - Add documentation or translate the documentation into other languages - Add new features and components -## Workflow +## What is PR -We recommend the potential contributors follow this workflow for contribution. +`PR` is the abbreviation of `Pull Request`. Here's the definition of `PR` in the [official document](https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests) of Github. -1. Fork and pull the latest OpenCompass repository, follow [get started](https://OpenCompass.readthedocs.io/en/latest/get_started.html) to setup the environment. -2. Checkout a new branch (**do not use the master or dev branch** for PRs) - -```bash -git checkout -b xxxx # xxxx is the name of new branch +``` +Pull requests let you tell others about changes you have pushed to a branch in a repository on GitHub. Once a pull request is opened, you can discuss and review the potential changes with collaborators and add follow-up commits before your changes are merged into the base branch. ``` -3. Edit the related files follow the code style mentioned below -4. Use [pre-commit hook](https://pre-commit.com/) to check and format your changes. -5. Commit your changes -6. Create a PR with related information +## Basic Workflow + +1. Get the most recent codebase +2. Checkout a new branch from `main` branch. +3. Commit your changes ([Don't forget to use pre-commit hooks!](#3-commit-your-changes)) +4. Push your changes and create a PR +5. Discuss and review your code +6. Merge your branch to `main` branch + +## Procedures in detail + +### 1. Get the most recent codebase + +- When you work on your first PR + + Fork the OpenCompass repository: click the **fork** button at the top right corner of Github page + ![avatar](https://github.com/InternLM/opencompass/assets/22607038/851ed33d-02db-49c9-bf94-7c62eee89eb2) + + Clone forked repository to local + + ```bash + git clone git@github.com:XXX/opencompass.git + ``` + + Add source repository to upstream + + ```bash + git remote add upstream git@github.com:InternLM/opencompass.git + ``` + +- After your first PR + + Checkout the latest branch of the local repository and pull the latest branch of the source repository. + + ```bash + git checkout main + git pull upstream main + ``` + +### 2. Checkout a new branch from `main` branch + +```bash +git checkout main -b branchname +``` + +### 3. Commit your changes + +- If you are a first-time contributor, please install and initialize pre-commit hooks from the repository root directory first. + + ```bash + pip install -U pre-commit + pre-commit install + ``` + +- Commit your changes as usual. Pre-commit hooks will be triggered to stylize your code before each commit. + + ```bash + # coding + git add [files] + git commit -m 'messages' + ``` + + ```{note} + Sometimes your code may be changed by pre-commit hooks. In this case, please remember to re-stage the modified files and commit again. + ``` + +### 4. Push your changes to the forked repository and create a PR + +- Push the branch to your forked remote repository + + ```bash + git push origin branchname + ``` + +- Create a PR + ![avatar](https://github.com/InternLM/opencompass/assets/22607038/08feb221-b145-4ea8-8e20-05f143081604) + +- Revise PR message template to describe your motivation and modifications made in this PR. You can also link the related issue to the PR manually in the PR message (For more information, checkout the [official guidance](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)). + +- You can also ask a specific person to review the changes you've proposed. + +### 5. Discuss and review your code + +- Modify your codes according to reviewers' suggestions and then push your changes. + +### 6. Merge your branch to `main` branch and delete the branch + +- After the PR is merged by the maintainer, you can delete the branch you created in your forked repository. + + ```bash + git branch -d branchname # delete local branch + git push origin --delete branchname # delete remote branch + ``` ## Code style @@ -44,25 +137,3 @@ We use the following tools for linting and formatting: - [docformatter](https://github.com/myint/docformatter): A formatter to format docstring. Style configurations of yapf and isort can be found in [setup.cfg](https://github.com/open-mmlab/OpenCompass/blob/main/setup.cfg). - -## Pre-commit Hook - -We use [pre-commit hook](https://pre-commit.com/) that checks and formats for `flake8`, `yapf`, `isort`, `trailing whitespaces`, `markdown files`, -fixes `end-of-files`, `double-quoted-strings`, `python-encoding-pragma`, `mixed-line-ending`, sorts `requirements.txt` automatically on every commit. -The config for a pre-commit hook is stored in [.pre-commit-config](xxxxxxx). - -After you clone the repository, you will need to install initialize pre-commit hook. - -```shell -pip install -U pre-commit -``` - -From the repository folder - -```shell -pre-commit install -``` - -After this on every commit check code linters and formatter will be enforced. - -> Before you create a PR, make sure that your code lints and is formatted by yapf. diff --git a/docs/en/prompt/overview.md b/docs/en/prompt/overview.md index 91170dae..eb0d3447 100644 --- a/docs/en/prompt/overview.md +++ b/docs/en/prompt/overview.md @@ -7,3 +7,5 @@ In OpenCompass, we split the template into two parts: the data-side template and The data-side template is referred to as [prompt_template](./prompt_template.md), which represents the process of converting the fields in the dataset into prompts. The model-side template is referred to as [meta_template](./meta_template.md), which represents how the model transforms these prompts into its expected input. + +We also offer some prompting examples regarding [Chain of Thought](./chain_of_thought.md). diff --git a/docs/zh_cn/_static/js/custom.js b/docs/zh_cn/_static/js/custom.js index 01293c8b..84da69d4 100644 --- a/docs/zh_cn/_static/js/custom.js +++ b/docs/zh_cn/_static/js/custom.js @@ -1,4 +1,4 @@ -var collapsedSections = ['Advanced Guides', 'Tools', 'User Guides', 'Notes']; +var collapsedSections = []; $(document).ready(function () { $('.model-summary').DataTable({ diff --git a/docs/zh_cn/advanced_guides/evaluation_turbomind.md b/docs/zh_cn/advanced_guides/evaluation_turbomind.md index 750b4717..ee1d790c 100644 --- a/docs/zh_cn/advanced_guides/evaluation_turbomind.md +++ b/docs/zh_cn/advanced_guides/evaluation_turbomind.md @@ -1,16 +1,16 @@ -# 评测LMDeploy模型 +# 评测 LMDeploy 模型 -我们支持评测使用[LMDeploy](https://github.com/InternLM/lmdeploy)加速过的大语言模型。LMDeploy 由 MMDeploy 和 MMRazor 团队联合开发，是涵盖了 LLM 任务的全套轻量化、部署和服务解决方案。 **TurboMind** 是 LMDeploy 推出的高效推理引擎。OpenCompass 对 TurboMind 进行了适配，本教程将介绍如何使用 OpenCompass 来对 TurboMind 加速后的模型进行评测。 +我们支持评测使用 [LMDeploy](https://github.com/InternLM/lmdeploy) 加速过的大语言模型。LMDeploy 由 MMDeploy 和 MMRazor 团队联合开发，是涵盖了 LLM 任务的全套轻量化、部署和服务解决方案。 **TurboMind** 是 LMDeploy 推出的高效推理引擎。OpenCompass 对 TurboMind 进行了适配，本教程将介绍如何使用 OpenCompass 来对 TurboMind 加速后的模型进行评测。 ## 环境配置 -### 安装OpenCompass +### 安装 OpenCompass -请根据OpenCompass[安装指南](https://opencompass.readthedocs.io/en/latest/get_started.html) 来安装算法库和准备数据集。 +请根据 OpenCompass [安装指南](https://opencompass.readthedocs.io/en/latest/get_started.html) 来安装算法库和准备数据集。 -### 安装LMDeploy +### 安装 LMDeploy -使用pip安装LMDeploy( python 3.8+) +使用 pip 安装 LMDeploy (python 3.8+)： ```shell pip install lmdeploy @@ -18,9 +18,9 @@ pip install lmdeploy ## 评测 -我们使用InternLM作为例子来介绍如何评测 +我们使用 InternLM 作为例子来介绍如何评测。 -### 第一步: 获取InternLM模型 +### 第一步: 获取 InternLM 模型 ```shell # 1. Download InternLM model(or use the cached model's checkpoint) @@ -46,10 +46,10 @@ python -m lmdeploy.turbomind.chat ./workspace ### 第三步: 评测转换后的模型 -在OpenCompass项目文件执行： +在 OpenCompass 项目目录执行： ```shell python run.py configs/eval_internlm_chat_7b_turbomind.py -w outputs/turbomind ``` -当模型完成推理和指标计算后，我们便可获得模型的评测结果 +当模型完成推理和指标计算后，我们便可获得模型的评测结果。 diff --git a/docs/zh_cn/index.rst b/docs/zh_cn/index.rst index c7cd9592..408f0d36 100644 --- a/docs/zh_cn/index.rst +++ b/docs/zh_cn/index.rst @@ -43,10 +43,10 @@ OpenCompass 上手路线 :maxdepth: 1 :caption: 提示词 - prompt/few_shot.md - prompt/chain_of_thought.md + prompt/overview.md prompt/prompt_template.md prompt/meta_template.md + prompt/chain_of_thought.md .. _进阶教程: .. toctree:: diff --git a/docs/zh_cn/notes/contribution_guide.md b/docs/zh_cn/notes/contribution_guide.md index ac7f7d1a..2ccc3702 100644 --- a/docs/zh_cn/notes/contribution_guide.md +++ b/docs/zh_cn/notes/contribution_guide.md @@ -1,10 +1,17 @@ # 为 OpenCompass 做贡献 -- [为OpenCompass做贡献](#为opencompass做贡献) - - [工作流程](#工作流程) +- [为 OpenCompass 做贡献](#为-opencompass-做贡献) + - [什么是拉取请求？](#什么是拉取请求) + - [基本的工作流：](#基本的工作流) + - [具体步骤](#具体步骤) + - [1. 获取最新的代码库](#1-获取最新的代码库) + - [2. 从 `main` 分支创建一个新的开发分支](#2-从-main-分支创建一个新的开发分支) + - [3. 提交你的修改](#3-提交你的修改) + - [4. 推送你的修改到复刻的代码库，并创建一个拉取请求](#4-推送你的修改到复刻的代码库并创建一个拉取请求) + - [5. 讨论并评审你的代码](#5-讨论并评审你的代码) + - [6. `拉取请求`合并之后删除该分支](#6-拉取请求合并之后删除该分支) - [代码风格](#代码风格) - [Python](#python) - - [预提交钩子 (Pre-commit Hook)](#预提交钩子-pre-commit-hook) 感谢你对于OpenCompass的贡献！我们欢迎各种形式的贡献，包括但不限于以下几点。 @@ -12,21 +19,116 @@ - 添加文档或将文档翻译成其它语言 - 添加新功能和组件 -## 工作流程 +## 什么是拉取请求？ -我们建议潜在的贡献者遵循以下的贡献工作流程。 +`拉取请求` (Pull Request), [GitHub 官方文档](https://docs.github.com/en/github/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests)定义如下。 -1. Fork并拉取最新的OpenCompass仓库，按照[开始使用](https://OpenCompass.readthedocs.io/en/latest/get_started.html)来设置环境。 -2. 检出一个新的分支（**不要使用master或dev分支来创建PR**） - -```bash -git checkout -b xxxx # xxxx 是新分支的名称 +``` +拉取请求是一种通知机制。你修改了他人的代码，将你的修改通知原来作者，希望他合并你的修改。 ``` -3. 编辑相关文件，并且遵循下面提到的代码风格 -4. 使用[预提交钩子](https://pre-commit.com/)来检查和格式化你的更改。 -5. 提交你的更改 -6. 创建一个带有相关信息的PR +## 基本的工作流： + +1. 获取最新的代码库 +2. 从最新的 `main` 分支创建分支进行开发 +3. 提交修改 ([不要忘记使用 pre-commit hooks!](#3-提交你的修改)) +4. 推送你的修改并创建一个 `拉取请求` +5. 讨论、审核代码 +6. 将开发分支合并到 `main` 分支 + +## 具体步骤 + +### 1. 获取最新的代码库 + +- 当你第一次提 PR 时 + + 复刻 OpenCompass 原代码库，点击 GitHub 页面右上角的 **Fork** 按钮即可 + ![avatar](https://github.com/InternLM/opencompass/assets/22607038/851ed33d-02db-49c9-bf94-7c62eee89eb2) + + 克隆复刻的代码库到本地 + + ```bash + git clone git@github.com:XXX/opencompass.git + ``` + + 添加原代码库为上游代码库 + + ```bash + git remote add upstream git@github.com:InternLM/opencompass.git + ``` + +- 从第二个 PR 起 + + 检出本地代码库的主分支，然后从最新的原代码库的主分支拉取更新。 + + ```bash + git checkout main + git pull upstream main + ``` + +### 2. 从 `main` 分支创建一个新的开发分支 + +```bash +git checkout main -b branchname +``` + +### 3. 提交你的修改 + +- 如果你是第一次尝试贡献，请在 OpenCompass 的目录下安装并初始化 pre-commit hooks。 + + ```bash + pip install -U pre-commit + pre-commit install + ``` + + ````{tip} + 对于中国地区的用户，由于网络原因，安装 pre-commit hook 可能会失败。可以尝试以下命令切换为国内镜像源： + ```bash + pre-commit install -c .pre-commit-config-zh-cn.yaml + pre-commit run –all-files -c .pre-commit-config-zh-cn.yaml + ``` + ```` + +- 提交修改。在每次提交前，pre-commit hooks 都会被触发并规范化你的代码格式。 + + ```bash + # coding + git add [files] + git commit -m 'messages' + ``` + + ```{note} + 有时你的文件可能会在提交时被 pre-commit hooks 自动修改。这时请重新添加并提交修改后的文件。 + ``` + +### 4. 推送你的修改到复刻的代码库，并创建一个拉取请求 + +- 推送当前分支到远端复刻的代码库 + + ```bash + git push origin branchname + ``` + +- 创建一个拉取请求 + + ![avatar](https://github.com/InternLM/opencompass/assets/22607038/08feb221-b145-4ea8-8e20-05f143081604) + +- 修改拉取请求信息模板，描述修改原因和修改内容。还可以在 PR 描述中，手动关联到相关的议题 (issue),（更多细节，请参考[官方文档](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)）。 + +- 你同样可以把 PR 关联给相关人员进行评审。 + +### 5. 讨论并评审你的代码 + +- 根据评审人员的意见修改代码，并推送修改 + +### 6. `拉取请求`合并之后删除该分支 + +- 在 PR 合并之后，你就可以删除该分支了。 + + ```bash + git branch -d branchname # 删除本地分支 + git push origin --delete branchname # 删除远程分支 + ``` ## 代码风格 @@ -44,24 +146,3 @@ git checkout -b xxxx # xxxx 是新分支的名称 - [docformatter](https://github.com/myint/docformatter): 一个格式化docstring的工具。 yapf和isort的样式配置可以在[setup.cfg](https://github.com/OpenCompass/blob/main/setup.cfg)中找到。 - -## 预提交钩子 (Pre-commit Hook) - -我们使用[预提交钩子](https://pre-commit.com/)用于在每次提交时自动检查与格式化`flake8`、`yapf`、`isort`、`trailing whitespaces`、`markdown files`， -修复`end-of-files`、`double-quoted-strings`、`python-encoding-pragma`、`mixed-line-ending`，并自动排序`requirements.txt`。预提交钩子的配置存储在[.pre-commit-config](<>)中。 - -在你克隆仓库后，你需要安装并初始化预提交钩子。 - -```shell -pip install -U pre-commit -``` - -从仓库文件夹运行 - -```shell -pre-commit install -``` - -之后，在每次提交时都会强制执行代码 linters 和格式化器。 - -> 在你创建PR前，确保你的代码通过了 lint 检查并被 yapf 格式化。 diff --git a/docs/zh_cn/prompt/overview.md b/docs/zh_cn/prompt/overview.md index c91ec950..5786b520 100644 --- a/docs/zh_cn/prompt/overview.md +++ b/docs/zh_cn/prompt/overview.md @@ -7,3 +7,5 @@ 数据侧的 template 被称为 [prompt_template](./prompt_template.md)，它表示了把数据集的字段转化成提示词的过程。 模型侧的 template 被称为 [meta_template](./meta_template.md)，它表示了模型将这些提示词转化为自身期望的输入的过程。 + +我们另外还提供了一些 [思维链](./chain_of_thought.md) 的 prompt 示例。