Fengzhe Zhou
004ed79593
[Feature] Add TheoremQA with 5-shot ( #1048 )
...
* add TheoremQA with 5-shot
* cherry pick from add-huggingface-above-v4.33, good TheoremQA results
2024-04-22 15:22:04 +08:00
Fengzhe Zhou
8c85edd1cd
[Sync] deprecate old mbpps ( #1064 )
2024-04-19 20:49:46 +08:00
liuwei130
a00e57296f
[Feature] Add ChemBench ( #1032 )
...
* add ChemBench
* update results
* molbench -> ChemBench
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-04-12 08:46:26 +08:00
Fengzhe Zhou
b39f501563
[Sync] update taco ( #1030 )
2024-04-09 17:50:23 +08:00
Mo Li
f2af49337d
[Feature] Add ATC Choice Version ( #1019 )
...
* Squashed commit of the following:
commit c48ad194c3976dc63d1b60d8c8ab2d5ff9e1cbfe
Author: DseidLi <2568818204@qq.com>
Date: Tue Apr 2 16:57:43 2024 +0800
add atc_choice
commit 3ac6efea29619573e6fac8fa3cce464853dcead0
Merge: 2d4e5590a6a03f2527fda2527fda8a575425acdf8367ba1ba61caf1cf8caf1cf8120bf8bhttps://github.com/open-compass/opencompass into atc_choice
commit 4e747ed1988ddbcfcc7fff334601259ade72d363
Author: DseidLi <2568818204@qq.com>
Date: Sun Mar 3 22:15:25 2024 +0800
add internlm2-lmdeploy model and gemma configs
commit 7dabc828123d711c8cf834d6aab4137bb55e85ed
Author: DseidLi <2568818204@qq.com>
Date: Sat Mar 2 17:26:15 2024 +0800
add atc choice version -ZH
commit 996f8ae43df7266e873c1c7375681db6524f3ebfc0d1190e390965baf7855d32b31eb8af82a7f08532bf9fe53fe78832bf9fe802a7cb025e05
2024-04-07 15:46:20 +08:00
Mo Li
0a6a03fe1a
[Feature] update needlebench and configs ( #986 )
...
* add Needlebench-1000K configs
* add prompt postion args
* add model configs
* Update parallel.py
* fix lint
2024-03-25 18:05:01 +08:00
Connor-Shen
0221d30877
[Fix] Update APPS/TACO ( #988 )
...
* [Feature] update apps/taco
* [Feature] update apps/taco
2024-03-19 20:21:39 +08:00
Connor-Shen
8a3c6e51ed
[Feature] Update APPS ( #985 )
...
* update post process
* update post process
2024-03-19 15:47:05 +08:00
Connor-Shen
d92595b671
[Feat] Support TACO ( #966 )
...
* [Feat] Support TACO
* update README
* update README
2024-03-19 15:39:16 +08:00
Jingming
89a8a8917b
[Feature] Add the implement of QuALITY datasets ( #976 )
...
#976
2024-03-15 21:22:38 +08:00
Connor-Shen
3098d78845
[Bench] Support APPS ( #963 )
...
* [Feat] support apps
* [Feat] support apps
* [Feat] support apps
* update README
2024-03-13 16:09:23 +08:00
Fengzhe Zhou
bdd85358cc
[Sync] update 20240308 ( #953 )
2024-03-11 22:34:19 +08:00
Fengzhe Zhou
b03d5dc531
[Sync] Sync Internal ( #941 )
2024-03-04 14:42:36 +08:00
yuantao2108
bbec7d8733
[Feature] add lveval benchmark ( #914 )
...
* add lveval benchmark
* add LVEval readme file
* update LVEval readme file
* Update configs/eval_bluelm_32k_lveval.py
* Update configs/eval_llama2_7b_lveval.py
---------
Co-authored-by: yuantao <yuantao@infini-ai.com>
Co-authored-by: Mo Li <82895469+DseidLi@users.noreply.github.com>
2024-03-04 11:22:03 +08:00
Mo Li
8142f399a8
[Feature] Upgrade the needle-in-a-haystack experiment to Needlebench ( #913 )
...
* add needlebench
* simplify needlebench 32k, 128k, 200k for eval
* update act prompt
* fix bug in needlebench summarizer
* add needlebench intro, fix summarizer
* lint summarizer
* fix linting error
* move readme.md
* update readme for needlebench
* update docs of needlebench
* simplify needlebench summarizers
2024-03-04 11:10:52 +08:00
Skyfall-xzz
4c45a71bbc
[Feature] Support OpenFinData ( #896 )
...
* [Feature] Support OpenFinData
* add README for OpenFinData
* update README
2024-02-29 12:55:07 +08:00
Fengzhe Zhou
5ce8e0450e
[Fix] Fix type hint in IFEval ( #915 )
2024-02-28 10:53:40 +08:00
Jingming
53fe788d27
[Fix] fix ifeval ( #909 )
2024-02-23 16:52:03 +08:00
bittersweet1999
45c606bcd0
[Fix] Fix IFEval ( #906 )
...
* fix ifeval
* fix ifeval
* fix ifeval
* fix ifeval
2024-02-22 16:51:34 +08:00
hailsham
dd444685bb
fix bug of gsm8k_postprocess ( #863 )
...
* fix bug of gsm8k_postprocess
* update postprocess
---------
Co-authored-by: Lei Fei <SENSETIME\leifei1@cn3114002087l.domain.sensetime.com>
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-06 23:52:47 +08:00
Connor-Shen
444d8d9507
[feat] support multipl-e ( #846 )
...
* [feat] support humaneval_multipl-e
* format
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-06 23:30:28 +08:00
Yggdrasill7D6
a6c49f15ce
fix lawbench 2-1 f0.5 score calculation bug ( #795 )
...
* fix lawbench 2-1 f0.5 score calculation bug
* use path in overall datasets folder
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-06 22:20:11 +08:00
Fengzhe Zhou
d34ba11106
[Sync] Merge branch 'dev' into zfz/update-keyset-demo ( #876 )
2024-02-05 23:29:10 +08:00
Skyfall-xzz
7ad1168062
Support NPHardEval ( #835 )
...
* support NPHardEval
* add .md file and fix minor bugs
* refactor and minor fix
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-05 15:52:28 +08:00
Yuchen Yan
fed7d800c6
[Fix] Fix error in gsm8k evaluator ( #782 )
...
Co-authored-by: jiangjin1999 <1261842974@qq.com>
2024-02-04 22:55:11 +08:00
Jingming
2801883351
[Fix] Fix acc of IFEval ( #849 )
...
* [Feature] Add IFEval
* [Fix] Changing the Score Rule.
2024-01-27 22:27:07 +08:00
Xiaoming Shi
35aace776a
[Fix] Update MedBench ( #845 )
2024-01-26 17:56:13 +08:00
Hubert
4aa74565e2
[Feat] minor update agent related ( #839 )
...
* [Feat] update cibench
* [Feat] Support CIBench
* [Feat] Support CIBench
* [Feat] Support CIBench
* [Feat] Support CIBench
2024-01-26 14:15:51 +08:00
Fengzhe Zhou
0991dd33a0
[Sync] Updata dataset cfg for internMath ( #837 )
...
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
2024-01-24 16:30:32 +08:00
bittersweet1999
2ee8e8a1a1
[Feature] add mtbench ( #829 )
...
* add mtbench
* add mtbench
* Update configs/datasets/subjective/multiround/mtbench_judgeby_gpt4.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update configs/datasets/subjective/multiround/mtbench_judgeby_gpt4.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/datasets/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/datasets/subjective/mtbench.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* fix mtbench
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2024-01-24 12:11:47 +08:00
Jingming
e059a5c2bf
[Feature] Add IFEval ( #813 )
...
* [Feature] Add IFEval
* [Doc] add introduction of IFEval
2024-01-23 20:07:49 +08:00
bittersweet1999
2d4da8dd02
[Feature] Add CompassArena ( #828 )
...
* add compass arena
* add compass_arena
* add compass arena
* Update opencompass/summarizers/subjective/compass_arena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/summarizers/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/datasets/subjective/compass_arena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/datasets/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update configs/eval_subjective_compassarena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update configs/datasets/subjective/compassarena/compassarena_compare.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update configs/eval_subjective_compassarena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update configs/datasets/subjective/compassarena/compassarena_compare.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* fix check position bias
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2024-01-23 15:12:46 +08:00
Guo Qipeng
e975a96fa1
Update cdme config and evaluator ( #812 )
...
* update cdme config and evaluator
* fix cdme prompt
* move CDME trim post-processor as a separate evaluator
---------
Co-authored-by: 郭琦鹏 <guoqipeng@pjlab.org.cn>
2024-01-19 11:29:27 +08:00
Fengzhe Zhou
b4afe3e7c1
[Sync] Add InternLM2 Keyset Evaluation Demo ( #807 )
...
Co-authored-by: zhangyifan1 <zhangyifan1@pjlab.org.cn>
2024-01-17 13:48:12 +08:00
Mo Li
acae560911
Added support for multi-needle testing in needle-in-a-haystack test ( #802 )
...
* Add NeedleInAHaystack Test
* Apply pre-commit formatting
* Update configs/eval_hf_internlm_chat_20b_cdme.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* add needle in haystack test
* update needle in haystack test
* update plot function in tools_needleinahaystack.py
* optimizing needleinahaystack dataset generation strategy
* modify minor formatting issues
* add English version support
* change NeedleInAHaystackDataset to dynamic loading
* change NeedleInAHaystackDataset to dynamic loading
* fix needleinahaystack test eval bug
* fix needleinahaystack config bug
* Added support for multi-needle testing in needle-in-a-haystack test
* Optimize the code for plotting in the needle-in-a-haystack test.
* Correct the typo in the dataset parameters.
* update needleinahaystack test docs
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2024-01-17 13:47:34 +08:00
bittersweet1999
814b3f73bd
reorganize subject files ( #801 )
2024-01-16 18:03:11 +08:00
bittersweet1999
83d6c48378
[Feature] Add configs for creationbench ( #791 )
...
* add creationv2_zh
* add creationv2_zh
* add eng config for creationbench
* add eng config for creationbench
* add eng config for creationbench
2024-01-12 14:20:21 +08:00
Xiaoming Shi
ad872a5dc2
[Feature] Update MedBench ( #779 )
...
* update medbench
* medbench update
* format medbench
* format
* Update
* update
* update
* update suffix
---------
Co-authored-by: 施晓明 <PJLAB\shixiaoming@pjnl104220118l.pjlab.org>
Co-authored-by: Leymore <zfz-960727@163.com>
2024-01-09 11:42:44 +08:00
Fengzhe Zhou
32f40a8f83
[Sync] Sync with internal codes 2023.01.08 ( #777 )
2024-01-08 14:07:24 +00:00
liyucheng09
0b2863039e
[Feature] Contamination analysis for MMLU, Hellaswag, and ARC_c ( #699 )
...
* Contamination analysis for ARC_c, mmlu, and Hellaswag
* update `eval_contamination.py`
* update `contamination.py` summarizer
* fix `eval_contamination.py`
* add mmlu groups for contamination analysis
2024-01-08 15:51:48 +08:00
Connor-Shen
30a90d8dd8
Support Mbpp_plus dataset ( #770 )
...
* support mbpp+
* support mbpp+
* minor fix
* [Feat] minor fix
---------
Co-authored-by: yingfhu <yingfhu@gmail.com>
2024-01-05 22:01:57 +08:00
bittersweet1999
2163f9398f
[Feature] add subject ir dataset ( #755 )
...
* add subject ir
* Add ir dataset
* Add ir dataset
2024-01-05 12:00:57 +00:00
bittersweet1999
be369c3e06
[Feature] Add multi_round dataset evaluation ( #766 )
...
* multi_round dataset
* add multi_round evaluation
2024-01-04 10:37:52 +00:00
Mo Li
33f8df1ca3
[Update] Change NeedleInAHaystackDataset to dynamic dataset loading ( #754 )
...
* Add NeedleInAHaystack Test
* Apply pre-commit formatting
* Update configs/eval_hf_internlm_chat_20b_cdme.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* add needle in haystack test
* update needle in haystack test
* update plot function in tools_needleinahaystack.py
* optimizing needleinahaystack dataset generation strategy
* modify minor formatting issues
* add English version support
* change NeedleInAHaystackDataset to dynamic loading
* change NeedleInAHaystackDataset to dynamic loading
* fix needleinahaystack test eval bug
* fix needleinahaystack config bug
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2024-01-02 17:22:56 +08:00
Francis-llgg
b69fe2343b
[Feature] Add GPQA Dataset ( #729 )
...
* check
* message
* add
* change prompt
* change a para nameq
* modify name of the file
* delete an useless file
2024-01-01 15:54:40 +08:00
Francis-llgg
ef3ae63539
[Feature] Add new dataset mastermath2024v1 ( #744 )
...
* add new dataset mastermath2024v1
* change it to simplified chinese prompt
* change file name
2024-01-01 15:53:24 +08:00
Mo Li
17b8e929dd
[Feature] Update plot function in tools_needleinahaystack.py ( #747 )
...
* Add NeedleInAHaystack Test
* Apply pre-commit formatting
* Update configs/eval_hf_internlm_chat_20b_cdme.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* add needle in haystack test
* update needle in haystack test
* update plot function in tools_needleinahaystack.py
* optimizing needleinahaystack dataset generation strategy
* modify minor formatting issues
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2023-12-29 18:51:09 +08:00
bittersweet1999
fe0b717033
add creationbench ( #753 )
2023-12-29 10:03:44 +00:00
Connor-Shen
81098722d2
add chinese version of humaneval, mbpp ( #743 )
...
* add chinese_version of humaneval,mbpp
* add humaneval&mbpp gen.py
* minor fix
* minor add
---------
Co-authored-by: yingfhu <yingfhu@gmail.com>
2023-12-28 14:47:56 +08:00
Hubert
0a525985e8
[Feature] Support sanitized MBPP dataset ( #745 )
2023-12-27 22:17:23 +08:00
