jiangzhs/OpenCompass
1
0
Fork 0
mirror of https://github.com/open-compass/opencompass.git synced 2025-05-30 16:03:24 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
445 Commits 11 Branches 35 Tags 66 MiB
af82a7f085
Commit Graph

120 Commits

Author SHA1 Message Date
DseidLi
af82a7f085 Merge remote-tracking branch 'upstream/main' into needlebench 2024-02-23 17:50:32 +08:00
Jingming
53fe788d27
[Fix] fix ifeval (#909) 2024-02-23 16:52:03 +08:00
DseidLi
a7cb025e05 add needlebench 2024-02-23 14:48:58 +08:00
bittersweet1999
45c606bcd0
[Fix] Fix IFEval (#906) 
* fix ifeval

* fix ifeval

* fix ifeval

* fix ifeval
 2024-02-22 16:51:34 +08:00
hailsham
dd444685bb
fix bug of gsm8k_postprocess (#863) 
* fix bug of gsm8k_postprocess

* update postprocess

---------

Co-authored-by: Lei Fei <SENSETIME\leifei1@cn3114002087l.domain.sensetime.com>
Co-authored-by: Leymore <zfz-960727@163.com>
 2024-02-06 23:52:47 +08:00
Connor-Shen
444d8d9507
[feat] support multipl-e (#846) 
* [feat] support humaneval_multipl-e

* format

---------

Co-authored-by: Leymore <zfz-960727@163.com>
 2024-02-06 23:30:28 +08:00
Yggdrasill7D6
a6c49f15ce
fix lawbench 2-1 f0.5 score calculation bug (#795) 
* fix lawbench 2-1 f0.5 score calculation bug

* use path in overall datasets folder

---------

Co-authored-by: Leymore <zfz-960727@163.com>
 2024-02-06 22:20:11 +08:00
Fengzhe Zhou
d34ba11106
[Sync] Merge branch 'dev' into zfz/update-keyset-demo (#876) 2024-02-05 23:29:10 +08:00
Skyfall-xzz
7ad1168062
Support NPHardEval (#835) 
* support NPHardEval

* add .md file and fix minor bugs

* refactor and minor fix

---------

Co-authored-by: Leymore <zfz-960727@163.com>
 2024-02-05 15:52:28 +08:00
Yuchen Yan
fed7d800c6
[Fix] Fix error in gsm8k evaluator (#782) 
Co-authored-by: jiangjin1999 <1261842974@qq.com>
 2024-02-04 22:55:11 +08:00
Jingming
2801883351
[Fix] Fix acc of IFEval (#849) 
* [Feature] Add IFEval

* [Fix] Changing the Score Rule.
 2024-01-27 22:27:07 +08:00
Xiaoming Shi
35aace776a
[Fix] Update MedBench (#845) 2024-01-26 17:56:13 +08:00
Hubert
4aa74565e2
[Feat] minor update agent related (#839) 
* [Feat] update cibench

* [Feat] Support CIBench

* [Feat] Support CIBench

* [Feat] Support CIBench

* [Feat] Support CIBench
 2024-01-26 14:15:51 +08:00
Fengzhe Zhou
0991dd33a0
[Sync] Updata dataset cfg for internMath (#837) 
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
 2024-01-24 16:30:32 +08:00
bittersweet1999
2ee8e8a1a1
[Feature] add mtbench (#829) 
* add mtbench

* add mtbench

* Update configs/datasets/subjective/multiround/mtbench_judgeby_gpt4.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/multiround/mtbench_judgeby_gpt4.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/__init__.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/mtbench.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* fix mtbench

---------

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
 2024-01-24 12:11:47 +08:00
Jingming
e059a5c2bf
[Feature] Add IFEval (#813) 
* [Feature] Add IFEval

* [Doc] add introduction of IFEval
 2024-01-23 20:07:49 +08:00
bittersweet1999
2d4da8dd02
[Feature] Add CompassArena (#828) 
* add compass arena

* add compass_arena

* add compass arena

* Update opencompass/summarizers/subjective/compass_arena.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/summarizers/subjective/__init__.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/compass_arena.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/__init__.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/eval_subjective_compassarena.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/compassarena/compassarena_compare.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/eval_subjective_compassarena.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/compassarena/compassarena_compare.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* fix check position bias

---------

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
 2024-01-23 15:12:46 +08:00
Guo Qipeng
e975a96fa1
Update cdme config and evaluator (#812) 
* update cdme config and evaluator

* fix cdme prompt

* move CDME trim post-processor as a separate evaluator

---------

Co-authored-by: 郭琦鹏 <guoqipeng@pjlab.org.cn>
 2024-01-19 11:29:27 +08:00
Fengzhe Zhou
b4afe3e7c1
[Sync] Add InternLM2 Keyset Evaluation Demo (#807) 
Co-authored-by: zhangyifan1 <zhangyifan1@pjlab.org.cn>
 2024-01-17 13:48:12 +08:00
Mo Li
acae560911
Added support for multi-needle testing in needle-in-a-haystack test (#802) 
* Add NeedleInAHaystack Test

* Apply pre-commit formatting

* Update configs/eval_hf_internlm_chat_20b_cdme.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* add needle in haystack test

* update needle in haystack test

* update plot function in tools_needleinahaystack.py

* optimizing needleinahaystack dataset generation strategy

* modify minor formatting issues

* add English version support

* change NeedleInAHaystackDataset to dynamic loading

* change NeedleInAHaystackDataset to dynamic loading

* fix needleinahaystack test eval bug

* fix needleinahaystack config bug

* Added support for multi-needle testing in needle-in-a-haystack test

* Optimize the code for plotting in the needle-in-a-haystack test.

* Correct the typo in the dataset parameters.

* update needleinahaystack test docs

---------

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
 2024-01-17 13:47:34 +08:00
bittersweet1999
814b3f73bd
reorganize subject files (#801) 2024-01-16 18:03:11 +08:00
bittersweet1999
83d6c48378
[Feature] Add configs for creationbench (#791) 
* add creationv2_zh

* add creationv2_zh

* add eng config for creationbench

* add eng config for creationbench

* add eng config for creationbench
 2024-01-12 14:20:21 +08:00
Xiaoming Shi
ad872a5dc2
[Feature] Update MedBench (#779) 
* update medbench

* medbench update

* format medbench

* format

* Update

* update

* update

* update suffix

---------

Co-authored-by: 施晓明 <PJLAB\shixiaoming@pjnl104220118l.pjlab.org>
Co-authored-by: Leymore <zfz-960727@163.com>
 2024-01-09 11:42:44 +08:00
Fengzhe Zhou
32f40a8f83
[Sync] Sync with internal codes 2023.01.08 (#777) 2024-01-08 14:07:24 +00:00
liyucheng09
0b2863039e
[Feature] Contamination analysis for MMLU, Hellaswag, and ARC_c (#699) 
* Contamination analysis for ARC_c, mmlu, and Hellaswag

* update `eval_contamination.py`

* update `contamination.py` summarizer

* fix `eval_contamination.py`

* add mmlu groups for contamination analysis
 2024-01-08 15:51:48 +08:00
Connor-Shen
30a90d8dd8
Support Mbpp_plus dataset (#770) 
* support mbpp+

* support mbpp+

* minor fix

* [Feat] minor fix

---------

Co-authored-by: yingfhu <yingfhu@gmail.com>
 2024-01-05 22:01:57 +08:00
bittersweet1999
2163f9398f
[Feature] add subject ir dataset (#755) 
* add subject ir

* Add ir dataset

* Add ir dataset
 2024-01-05 12:00:57 +00:00
bittersweet1999
be369c3e06
[Feature] Add multi_round dataset evaluation (#766) 
* multi_round dataset

* add multi_round evaluation
 2024-01-04 10:37:52 +00:00
Mo Li
33f8df1ca3
[Update] Change NeedleInAHaystackDataset to dynamic dataset loading (#754) 
* Add NeedleInAHaystack Test

* Apply pre-commit formatting

* Update configs/eval_hf_internlm_chat_20b_cdme.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* add needle in haystack test

* update needle in haystack test

* update plot function in tools_needleinahaystack.py

* optimizing needleinahaystack dataset generation strategy

* modify minor formatting issues

* add English version support

* change NeedleInAHaystackDataset to dynamic loading

* change NeedleInAHaystackDataset to dynamic loading

* fix needleinahaystack test eval bug

* fix needleinahaystack config bug

---------

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
 2024-01-02 17:22:56 +08:00
Francis-llgg
b69fe2343b
[Feature] Add GPQA Dataset (#729) 
* check

* message

* add

* change prompt

* change a para nameq

* modify name of the file

* delete an useless file
 2024-01-01 15:54:40 +08:00
Francis-llgg
ef3ae63539
[Feature] Add new dataset mastermath2024v1 (#744) 
* add new dataset mastermath2024v1

* change it to simplified chinese prompt

* change file name
 2024-01-01 15:53:24 +08:00
Mo Li
17b8e929dd
[Feature] Update plot function in tools_needleinahaystack.py (#747) 
* Add NeedleInAHaystack Test

* Apply pre-commit formatting

* Update configs/eval_hf_internlm_chat_20b_cdme.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* add needle in haystack test

* update needle in haystack test

* update plot function in tools_needleinahaystack.py

* optimizing needleinahaystack dataset generation strategy

* modify minor formatting issues

---------

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
 2023-12-29 18:51:09 +08:00
bittersweet1999
fe0b717033
add creationbench (#753) 2023-12-29 10:03:44 +00:00
Connor-Shen
81098722d2
add chinese version of humaneval, mbpp (#743) 
* add chinese_version of humaneval,mbpp

* add humaneval&mbpp gen.py

* minor fix

* minor add

---------

Co-authored-by: yingfhu <yingfhu@gmail.com>
 2023-12-28 14:47:56 +08:00
Hubert
0a525985e8
[Feature] Support sanitized MBPP dataset (#745) 2023-12-27 22:17:23 +08:00
bittersweet1999
dfd9ac0fd9
[Feature] Add other judgelm prompts for Alignbench (#731) 
* add judgellm prompts

* add judgelm prompts

* update import info

* fix situation that no abbr in config

* fix situation that no abbr in config

* add summarizer for other judgellm

* change config name

* add maxlen

* add maxlen

* dict assert

* dict assert

* fix strings

* fix strings
 2023-12-27 17:54:53 +08:00
philipwangOvO
34561ececb
[Feature] Add InfiniteBench (#739) 
* add InfiniteBench

* add InfiniteBench

---------

Co-authored-by: wangchonghua <wangchonghua@pjlab.org.cn>
 2023-12-26 15:36:27 +08:00
Fengzhe Zhou
3a68083ecc
[Sync] update configs (#734) 2023-12-25 21:59:16 +08:00
Mo Li
0e24f4213e
[Feature] Add NeedleInAHaystack Test Support (#714) 
* Add NeedleInAHaystack Test

* Apply pre-commit formatting

* Update configs/eval_hf_internlm_chat_20b_cdme.py

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* add needle in haystack test

* update needle in haystack test

---------

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
 2023-12-23 12:00:51 +08:00
Skyfall-xzz
b35d991786
[Feature] Add ReasonBench(Internal) dataset (#577) 
* [Feature] Add reasonbench dataset

* add configs for supporting generative inference & merge datasets in the same category

* modify config filename to prompt version

* fix codes to meet pre-commit requirements

* lint the code to meet pre-commit requirements

* Align Load_data Sourcecode Briefly

* fix bugs

* reduce code redundancy
 2023-12-20 17:57:42 +08:00
Jingming
76a95e9e81
[Feature] Support the use of humaneval_plus. (#720) 
* [Feature] Support the use of humaneval_plus.

* [Feature] Add humaneval_plus_gen.py

* minor check

* [Fix] Fix bug

---------

Co-authored-by: yingfhu <yingfhu@gmail.com>
 2023-12-20 17:25:17 +08:00
DseidLi
db2920326a
[Fix] remove redundant in gsm8k.py (#700) 
Removed redundant code in GSM8KDataset.load method.
 2023-12-14 19:55:58 +08:00
bittersweet1999
1fe152b3e8
[Feature] Support AlignmentBench infer and judge (#697) 
* alignmentbench infer and judge

* alignmentbench

* alignmentbench done

* alignment all done

* alignment all done
 2023-12-13 19:59:30 +08:00
bittersweet1999
465308e430
[Feature] Add Subjective Evaluation (#680) 
* new version of subject

* fixed draw

* fixed draw

* fixed draw

* done

* done

* done

* done

* fixed lint
 2023-12-11 22:22:11 +08:00
Hubert
e78857ac36
[Sync] minor test (#683) 2023-12-11 17:42:53 +08:00
Jingming
dd4318f6ab
[Feature] enhance the ability of humaneval_postprocess (#676) 
* [Feature] enhance the ability of humaneval_postprocess

* refactor

* [Feature] Keep the old version of the function and realize the new function in humaneval_postprocess_v2.

* Update opencompass/datasets/humaneval.py

---------

Co-authored-by: Leymore <zfz-960727@163.com>
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
 2023-12-11 14:39:56 +08:00
Xiaoming Shi
1bf85949ef
[Feature] Add medbench (#678) 
* update medbench

* medbench update

* format medbench

* format

---------

Co-authored-by: 施晓明 <PJLAB\shixiaoming@pjnl104220118l.pjlab.org>
Co-authored-by: Leymore <zfz-960727@163.com>
 2023-12-09 16:05:46 +08:00
liyucheng09
05bbce8b08
[Feature] Add Data Contamination Analysis (#639) 
* add contamination analysis to ceval

* fix bugs

* add contamination docs

* to pass CI check

* update

---------

Co-authored-by: zhangyifan1 <zhangyifan1@pjlab.org.cn>
Co-authored-by: Leymore <zfz-960727@163.com>
 2023-12-08 10:00:11 +08:00
bittersweet1999
1c95790fdd
New subjective judgement (#660) 
* TabMWP

* TabMWP

* fixed

* fixed

* fixed

* done

* done

* done

* add new subjective judgement

* add new subjective judgement

* add new subjective judgement

* add new subjective judgement

* add new subjective judgement

* modified to a more general way

* modified to a more general way

* final

* final

* add summarizer

* add new summarize

* fixed

* fixed

* fixed

---------

Co-authored-by: caomaosong <caomaosong@pjlab.org.cn>
 2023-12-06 13:28:33 +08:00
rolellm
e10f1c9139
added rolebench dataset. (#633) 
* added rolebench

* 修改了不合理的变量名

* 修改了评论中的变量名
 2023-12-01 22:54:42 +08:00