bittersweet1999
68ca48496b
[Refactor] Reorganize subjective eval ( #1284 )
...
* fix pip version
* fix pip version
* reorganize subjective eval
* reorg sub
* reorg subeval
* reorg subeval
* update subjective doc
* reorg subeval
* reorg subeval
2024-07-05 22:11:37 +08:00
Songyang Zhang
409a042d93
[Feature] Add InternLM2.5 ( #1286 )
...
* [Feature] Add InternLM2.5
* Update
* update readme
---------
Co-authored-by: zhangsongyang <zhangsongyang@pjlab.org.cn>
Co-authored-by: Leymore <zfz-960727@163.com>
2024-07-04 20:10:31 +08:00
zhulinJulia24
167cfdcca3
[ci] update daily testcase ( #1285 )
...
* Update daily-run-test.yml
* Create eval_regression_chat.py
* Delete .github/scripts/.github/scripts/eval_regression_chat.py
* Create eval_regression_chat.py
* Update pr-run-test.yml
* Update daily-run-test.yml
* Update daily-run-test.yml
* Update daily-run-test.yml
* Update oc_score_baseline.yaml
* Update oc_score_assert.py
* Update daily-run-test.yml
* Update daily-run-test.yml
* Update oc_score_baseline.yaml
* Update oc_score_assert.py
* Update oc_score_assert.py
* fix lint
* update
* update
* update
* update
* update
* update
* update
* update
* update
* Update daily-run-test.yml
* update
---------
Co-authored-by: zhulin1 <zhulin1@pjlab.org.cn>
2024-07-03 18:56:09 +08:00
liushz
fc2c9dea8c
Update MathBench summarizer & fix cot setting ( #1282 )
...
* Update MathBench
* Update MathBench
* Update MathBench
---------
Co-authored-by: liushz <liuhongwei@pjlab.rog.cn>
2024-07-01 21:51:17 +08:00
Fengzhe Zhou
a32f21a356
[Sync] Sync with internal codes 2024.06.28 ( #1279 )
2024-06-28 14:16:34 +08:00
klein
1fa62c4a42
Support wildbench ( #1266 )
...
Co-authored-by: Leymore <zfz-960727@163.com>
2024-06-24 13:16:27 +08:00
bittersweet1999
e0d7808b4e
[Fix] fix pip version ( #1228 )
...
* fix pip version
* fix pip version
2024-06-06 11:48:07 +08:00
bittersweet1999
982e024540
[Feature] add dataset Fofo ( #1224 )
...
* add fofo dataset
* add dataset fofo
2024-06-06 11:40:48 +08:00
Xingyuan Bu
02a0a4e857
MT-Bench-101 ( #1215 )
...
* add mt-bench-101
* add readme and requirements
* add mt-bench-101 data
* Update readme_mtbench101.md
* update readme
* update leaderboard
* fix typo
* Update readme_mtbench101.md
* fit newest opencompass
* update readme.md
* mtbench101 to opencompass
* mtbench101 to opencompass
* for code review
* for code review
* for code review
* hook
* hook
---------
Co-authored-by: liujie <ljie@buaa.edu.cn>
2024-06-03 14:52:12 +08:00
bittersweet1999
7c381e5be8
[Fix] fix summarizer ( #1217 )
...
* fix summarizer
* fix summarizer
2024-05-31 11:40:47 +08:00
Fengzhe Zhou
a77b8a5cec
[Sync] format ( #1214 )
2024-05-30 00:21:58 +08:00
Fengzhe Zhou
d59189b87f
[Doc] Update running command in README ( #1206 )
2024-05-30 00:06:39 +08:00
Fengzhe Zhou
0b50112dc1
[Fix] Rollback opt model configs ( #1213 )
2024-05-30 00:03:22 +08:00
Xu Song
808582d952
Fix VLLM argument error ( #1207 )
2024-05-29 10:14:08 +08:00
Fengzhe Zhou
2954913d9b
[Sync] bump version ( #1204 )
2024-05-28 23:09:59 +08:00
Fengzhe Zhou
9fa80b0f93
[Feat] Update charm summary ( #1194 )
2024-05-27 16:17:01 +08:00
jxd
608ff5810d
support CHARM ( https://github.com/opendatalab/CHARM ) reasoning tasks ( #1190 )
...
* support CHARM (https://github.com/opendatalab/CHARM ) reasoning tasks
* fix lint error
* add dataset card for CHARM
* minor refactor
* add txt
---------
Co-authored-by: wujiang <wujiang@pjlab.org.cn>
Co-authored-by: Leymore <zfz-960727@163.com>
2024-05-27 13:48:22 +08:00
bittersweet1999
07a6dacf33
fix length ( #1180 )
2024-05-24 23:30:01 +08:00
klein
5eb8f14d97
[Fix] Fix drop_gen.py ( #1191 )
...
Fix the bug in drop_gen: wrong import
2024-05-24 23:17:50 +08:00
bittersweet1999
31afe87026
fix yi-chat template ( #1178 )
2024-05-21 18:14:12 +08:00
liushz
1448be00e2
Update MathBench ( #1176 )
...
* Add Math Evaluation with Judge Model Evaluator
* Add Math Evaluation with Judge Model Evaluator
* Add Math Evaluation with Judge Model Evaluator
* Add Math Evaluation with Judge Model Evaluator
* Fix Llama-3 meta template
* Fix MATH with JudgeLM Evaluation
* Fix MATH with JudgeLM Evaluation
* Fix MATH with JudgeLM Evaluation
* Fix MATH with JudgeLM Evaluation
* Update acclerator
* Update MathBench
---------
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
2024-05-21 14:45:43 +08:00
Fengzhe Zhou
2b3d4150f3
[Sync] update evaluator ( #1175 )
2024-05-21 14:22:46 +08:00
Fengzhe Zhou
5de85406ce
[Sync] add OC16 entry ( #1171 )
2024-05-17 16:50:58 +08:00
Fengzhe Zhou
8ea2c404d7
[Feat] enable HuggingFacewithChatTemplate with --accelerator via cli ( #1163 )
...
* enable HuggingFacewithChatTemplate with --accelerator via cli
* rm vllm_internlm2_chat_7b
2024-05-15 21:51:07 +08:00
Fengzhe Zhou
62dbf04708
[Sync] update github workflow ( #1156 )
2024-05-14 22:42:23 +08:00
Fengzhe Zhou
aa2dd2b58c
[Format] Add config lints ( #892 )
2024-05-14 15:35:58 +08:00
Xu Song
3dbba11945
[Feat] Support dataset_suffix check for mixed configs ( #973 )
...
* [Feat] Support dataset_suffix check for mixed configs
* update mixed suffix
* update suffix
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-05-14 15:03:28 +08:00
Fengzhe Zhou
7505b3cadf
[Feature] Add huggingface apply_chat_template ( #1098 )
...
* add TheoremQA with 5-shot
* add huggingface_above_v4_33 classes
* use num_worker partitioner in cli
* update theoremqa
* update TheoremQA
* add TheoremQA
* rename theoremqa -> TheoremQA
* update TheoremQA output path
* rewrite many model configs
* update huggingface
* further update
* refine configs
* update configs
* update configs
* add configs/eval_llama3_instruct.py
* add summarizer multi faceted
* update bbh datasets
* update configs/models/hf_llama/lmdeploy_llama3_8b_instruct.py
* rename class
* update readme
* update hf above v4.33
2024-05-14 14:50:16 +08:00
Mo Li
6c711cb262
[Fix] Fix Needlebench Summarizer ( #1143 )
...
* update few-shot example
* add 128k
2024-05-13 15:59:34 +08:00
bittersweet1999
5432dfc1ff
fix multiround ( #1146 )
2024-05-13 15:58:39 +08:00
bittersweet1999
833a35140b
[Fix] fix alpacaeval while add caching path ( #1139 )
...
* fix alpacaeval
* fix alpacaeval
2024-05-11 14:02:26 +08:00
Alexander Lam
a71122ee18
[Feature] Add Qwen1.5 MoE 7b and Mixtral 8x22b model configs ( #1123 )
...
* added qwen moe and mixtral 8x22 model configs
* updated README files news section
2024-05-09 11:04:26 +08:00
Mo Li
cb080fa7de
[Fix] Fix NeedleBench Summarizer Typo ( #1125 )
...
* update needleinahaystack eval docs
* update needlebench summarizer
* fix english docs typo
2024-05-08 20:00:15 +08:00
JuhaoLiang
d2c40e5648
[Feature] Add AceGPT-MMLUArabic benchmark ( #1099 )
...
* add AceGPT-MMLUArabic benchmark
* update readme and fix lint issue
* remove unused package
* add MMLUArabic zero-shot settings
* rename filename and update readme
2024-05-08 15:00:26 +08:00
Fangyu Lei
862044fb7d
[Feature] Add S3Eval Dataset ( #916 )
...
* s3eval_branch
* update s3eval
2024-05-06 19:41:52 +08:00
Xu Song
d501710155
[Fix] Fix AGIEval chinese sets ( #972 )
...
* [Fix] Fix AGIEval chinese sets
* Create agieval_gen_617738.py
* [Fix] Fix AGIEval chinese sets
* Restore agieval_gen_64afd3.py
* Update agieval_gen.py
* Create agieval_mixed_0fa998.py
* Update agieval_mixed.py
2024-05-06 15:31:42 +08:00
Yggdrasill7D6
af10ecc272
add mgsm datasets ( #1081 )
...
* add mgsm datasets
* fix lint
* fix lint
* update mgsm
* update mgsm
* ease code spell
* update
* update
* update
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-05-06 15:29:34 +08:00
klein
153c4fc988
[Feature] update drop dataset from openai simple eval ( #1092 )
...
* [Feature] update drop dataset from openai simple eval
* update drop template presentation
* update
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-05-06 13:37:08 +08:00
Fengzhe Zhou
d43392a3bb
[Feature] Add mmlu prompt from simple_evals, openai ( #1074 )
...
* add mmlu prompt from simple_evals, openai
* return empty str on failure
2024-05-06 13:26:26 +08:00
Yang Yong
53fe390454
fix LightllmApi workers bug ( #1113 )
2024-04-30 22:09:22 +08:00
Alexander Lam
35c94d0cde
[Feature] Adding support for LLM Compression Evaluation ( #1108 )
...
* fixed formatting based on pre-commit tests
* fixed typo in comments; reduced the number of models in the eval config
* fixed a bug in LLMCompressionDataset, where setting samples=None would result in passing test[:None] to load_dataset
* removed unnecessary variable in _format_table_pivot; changed lark_reporter message to English
2024-04-30 10:51:01 +08:00
liushz
a6f67e1a65
[Fix] Fix Math Evaluation with Judge Model Evaluator & Add README ( #1103 )
...
* Add Math Evaluation with Judge Model Evaluator
* Add Math Evaluation with Judge Model Evaluator
* Add Math Evaluation with Judge Model Evaluator
* Add Math Evaluation with Judge Model Evaluator
* Fix Llama-3 meta template
* Fix MATH with JudgeLM Evaluation
* Fix MATH with JudgeLM Evaluation
* Fix MATH with JudgeLM Evaluation
* Fix MATH with JudgeLM Evaluation
---------
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
2024-04-28 21:58:58 +08:00
bittersweet1999
0b7de67c4a
fix prompt template ( #1104 )
2024-04-28 21:54:30 +08:00
Yggdrasill7D6
58a57a4c45
[Feature] add support for Flames datasets ( #1093 )
...
* add flames datasets
* fix lint
* rm quota
* add judgemodel info and fix os path
* support flames dataset
* support flames dataset
---------
Co-authored-by: bittersweet1999 <1487910649@qq.com>
2024-04-28 18:56:24 +08:00
Mo Li
76dd814c4d
[Doc] Update NeedleInAHaystack Docs ( #1102 )
...
* update NeedleInAHaystack Test Docs
* update docs
2024-04-28 18:51:47 +08:00
Haodong Duan
3a232db471
[Deperecate] Remove multi-modal related stuff ( #1072 )
...
* Remove MultiModal
* update index.rst
* update README
* remove mmbench codes
* update news
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-04-26 21:20:14 +08:00
Francis-llgg
f1ee11de14
[Feature] Add gpqa prompt from simple_evals, openai ( #1080 )
...
* add gpqa_openai_simple_eval
* 触发CI构建
* reorg
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-04-26 20:13:00 +08:00
klein
e4830a6926
Update CIBench ( #1089 )
...
* modify the requirements/runtime.txt: numpy==1.23.4 --> numpy>=1.23.4
* update cibench: dataset and evluation
* cibench summarizer bug
* update cibench
* move extract_code import
---------
Co-authored-by: zhangchuyu@pjlab.org.cn <zhangchuyu@pjlab.org.cn>
Co-authored-by: Leymore <zfz-960727@163.com>
2024-04-26 18:46:02 +08:00
bittersweet1999
e404b72c52
[Feature] support arenahard evaluation ( #1096 )
...
* support arenahard
* support arenahard
* support arenahard
2024-04-26 15:42:00 +08:00
bittersweet1999
6ba1c4937d
[Feature] Support Math evaluation via judgemodel ( #1094 )
...
* support openai math evaluation
* support openai math evaluation
* support openai math evaluation
* support math llm judge
* support math llm judge
2024-04-26 14:56:23 +08:00
Jingming Zhuo
41196c48ae
Add humaneval prompt from simple_evals, openai ( #1076 )
...
* [Feature] Add IFEval
* add humaneval prompt from simple_evals, openai
2024-04-24 17:40:50 +08:00
liushz
17735f0c13
Fix Llama-3 meta template ( #1079 )
...
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
2024-04-24 16:46:25 +08:00
Ke Bao
81d0e4d793
[Feature] Add lmdeploy tis python backend model ( #1014 )
...
* add lmdeploy tis python backend model
* fix pr check
* update
2024-04-23 14:27:11 +08:00
Fengzhe Zhou
004ed79593
[Feature] Add TheoremQA with 5-shot ( #1048 )
...
* add TheoremQA with 5-shot
* cherry pick from add-huggingface-above-v4.33, good TheoremQA results
2024-04-22 15:22:04 +08:00
Fengzhe Zhou
a256753221
[Feature] Add LLaMA-3 Series Configs ( #1065 )
...
* add LLaMA-3 Series configs
* update readme
2024-04-22 14:39:31 +08:00
bittersweet1999
6f98c8d9ab
[Fix] Fix MultiRound Subjective Evaluation( #1043 )
...
* fix multiround
* fix
2024-04-22 12:06:03 +08:00
Fengzhe Zhou
8c85edd1cd
[Sync] deprecate old mbpps ( #1064 )
2024-04-19 20:49:46 +08:00
liuwei130
a00e57296f
[Feature] Add ChemBench ( #1032 )
...
* add ChemBench
* update results
* molbench -> ChemBench
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-04-12 08:46:26 +08:00
Fengzhe Zhou
b39f501563
[Sync] update taco ( #1030 )
2024-04-09 17:50:23 +08:00
Mo Li
16f29b25f1
[Fix] Simplify needlebench summarizer ( #1024 )
...
* Conflicts:
configs/summarizers/needlebench.py
* fix lint problems
2024-04-07 17:51:13 +08:00
Mo Li
f2af49337d
[Feature] Add ATC Choice Version ( #1019 )
...
* Squashed commit of the following:
commit c48ad194c3976dc63d1b60d8c8ab2d5ff9e1cbfe
Author: DseidLi <2568818204@qq.com>
Date: Tue Apr 2 16:57:43 2024 +0800
add atc_choice
commit 3ac6efea29619573e6fac8fa3cce464853dcead0
Merge: 2d4e5590a6a03f2527fda2527fda8a575425acdf8367ba1ba61caf1cf8caf1cf8120bf8bhttps://github.com/open-compass/opencompass into atc_choice
commit 4e747ed1988ddbcfcc7fff334601259ade72d363
Author: DseidLi <2568818204@qq.com>
Date: Sun Mar 3 22:15:25 2024 +0800
add internlm2-lmdeploy model and gemma configs
commit 7dabc828123d711c8cf834d6aab4137bb55e85ed
Author: DseidLi <2568818204@qq.com>
Date: Sat Mar 2 17:26:15 2024 +0800
add atc choice version -ZH
commit 996f8ae43df7266e873c1c7375681db6524f3ebfc0d1190e390965baf7855d32b31eb8af82a7f08532bf9fe53fe78832bf9fe802a7cb025e05
2024-04-07 15:46:20 +08:00
Mo Li
b50d163265
[Fix] Refactor Needlebench Configs for CLI Testing Support ( #1020 )
...
* add needlebench datasets suffix
* fix import
* update run.py args for summarizer key and dataset suffix
* update utils/run.py
2024-04-07 15:12:56 +08:00
bittersweet1999
2d4e559763
[Feature] Add multi-model judge and fix some problems ( #1016 )
...
* support multi-model judge and moe judge
* test_moe
* test_moe
* test
* add moe judge
* support multi-judge-model
2024-04-02 11:52:06 +08:00
bittersweet1999
02e7eec911
[Feature] Support AlpacaEval_V2 ( #1006 )
...
* support alpacaeval_v2
* support alpacaeval
* update docs
* update docs
2024-03-28 16:49:04 +08:00
Mo Li
0a6a03fe1a
[Feature] update needlebench and configs ( #986 )
...
* add Needlebench-1000K configs
* add prompt postion args
* add model configs
* Update parallel.py
* fix lint
2024-03-25 18:05:01 +08:00
bittersweet1999
0665bb91a8
[Fix] Quick fix ( #995 )
2024-03-22 19:54:19 +08:00
Ke Bao
e415ddf96a
[Fix] Fix turbomind_tis ( #992 )
2024-03-22 15:50:12 +08:00
bittersweet1999
054e9fa7e5
[Feature] add one script for subjective ( #993 )
...
* add one script for subjective
* add one script for subjective
* add one script for subjective
* add one script for subjective
---------
Co-authored-by: thebestannie <1290646445@qq.com>
2024-03-20 23:20:41 +08:00
Connor-Shen
0221d30877
[Fix] Update APPS/TACO ( #988 )
...
* [Feature] update apps/taco
* [Feature] update apps/taco
2024-03-19 20:21:39 +08:00
Connor-Shen
8a3c6e51ed
[Feature] Update APPS ( #985 )
...
* update post process
* update post process
2024-03-19 15:47:05 +08:00
Connor-Shen
d92595b671
[Feat] Support TACO ( #966 )
...
* [Feat] Support TACO
* update README
* update README
2024-03-19 15:39:16 +08:00
bittersweet1999
c78a4df923
add support for set prediction path ( #984 )
2024-03-19 14:32:15 +08:00
Jingming
89a8a8917b
[Feature] Add the implement of QuALITY datasets ( #976 )
...
#976
2024-03-15 21:22:38 +08:00
Jingming
c2d4717be2
[Fix] Fix a bug in internlm2 series configs ( #977 )
2024-03-15 15:21:35 +08:00
Connor-Shen
3098d78845
[Bench] Support APPS ( #963 )
...
* [Feat] support apps
* [Feat] support apps
* [Feat] support apps
* update README
2024-03-13 16:09:23 +08:00
Jingming
4c1533e59e
[Fix] fix the config's name of deepseek-coder ( #964 )
2024-03-12 19:36:52 +08:00
Fengzhe Zhou
bdd85358cc
[Sync] update 20240308 ( #953 )
2024-03-11 22:34:19 +08:00
bittersweet1999
848e7c8a76
[fix] add different temp for different question in mtbench ( #954 )
...
* add temp for mtbench
* add document for mtbench
* add document for mtbench
2024-03-11 17:24:39 +08:00
Yang Yong
107e022cf4
Support prompt template for LightllmApi. Update LightllmApi token bucket. ( #945 )
2024-03-06 15:33:53 +08:00
RunningLeon
c54a5d3b0f
Support get_ppl for TurbomindModel ( #878 )
...
* update ppl for turbomindmodel
* update api_server
* rename config and set thread_safe for pytorch engine if possible
2024-03-06 11:44:19 +08:00
Xu Song
2e993989a6
[Fix] FinanceIQ_datasets import error ( #939 )
...
* [Fix] Fix KeyError: 'FinanceIQ_datasets'
* [Fix] Fix KeyError: 'FinanceIQ_datasets'
2024-03-05 20:32:24 +08:00
Jingming
66d3aa4c01
[Feature] Add configs of deepseek-coder ( #943 )
2024-03-05 11:38:28 +08:00
Jingming
d0550268f3
[Fix] fix a bug of humanevalplus config ( #944 )
2024-03-05 11:37:17 +08:00
Fengzhe Zhou
b03d5dc531
[Sync] Sync Internal ( #941 )
2024-03-04 14:42:36 +08:00
yuantao2108
bbec7d8733
[Feature] add lveval benchmark ( #914 )
...
* add lveval benchmark
* add LVEval readme file
* update LVEval readme file
* Update configs/eval_bluelm_32k_lveval.py
* Update configs/eval_llama2_7b_lveval.py
---------
Co-authored-by: yuantao <yuantao@infini-ai.com>
Co-authored-by: Mo Li <82895469+DseidLi@users.noreply.github.com>
2024-03-04 11:22:03 +08:00
Mo Li
8142f399a8
[Feature] Upgrade the needle-in-a-haystack experiment to Needlebench ( #913 )
...
* add needlebench
* simplify needlebench 32k, 128k, 200k for eval
* update act prompt
* fix bug in needlebench summarizer
* add needlebench intro, fix summarizer
* lint summarizer
* fix linting error
* move readme.md
* update readme for needlebench
* update docs of needlebench
* simplify needlebench summarizers
2024-03-04 11:10:52 +08:00
Mo Li
120bf8b399
add vllm model configs ( #938 )
2024-03-01 17:31:51 +08:00
Skyfall-xzz
4c45a71bbc
[Feature] Support OpenFinData ( #896 )
...
* [Feature] Support OpenFinData
* add README for OpenFinData
* update README
2024-02-29 12:55:07 +08:00
bittersweet1999
001e77fea2
[Feature] add support for gemini ( #931 )
...
* add gemini
* add gemini
* add gemini
2024-02-28 19:38:34 +08:00
Jingming
53fe788d27
[Fix] fix ifeval ( #909 )
2024-02-23 16:52:03 +08:00
bittersweet1999
45c606bcd0
[Fix] Fix IFEval ( #906 )
...
* fix ifeval
* fix ifeval
* fix ifeval
* fix ifeval
2024-02-22 16:51:34 +08:00
RunningLeon
32ba0b074e
Support lmdeploy pytorch engine ( #875 )
...
* add lmdeploy pytorch model
* fix
* speed up encoding and decoding
* fix
* change tokenizer
2024-02-22 03:46:07 -03:00
Xu Song
6d04decab4
[Fix] Fix moss template config ( #897 )
2024-02-21 11:19:24 +08:00
Fengzhe Zhou
2b7d376e3d
[Fix] Fix chatglm2 config ( #893 )
2024-02-19 14:55:53 +08:00
Fengzhe Zhou
9119e2ac39
[Fix] rename qwen2-beta -> qwen1.5 ( #894 )
2024-02-19 14:55:35 +08:00
Yang Yong
b6e21ece38
Support LightllmApi input_format ( #888 )
2024-02-19 10:02:59 +08:00
hailsham
e257254b00
[Feature] add global retriever config ( #842 )
...
* add global retriever config
* give zero shot overwrite example
* give zero shot overwrite example
---------
Co-authored-by: Lei Fei <SENSETIME\leifei1@cn3114002087l.domain.sensetime.com>
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-07 00:30:20 +08:00
hailsham
dd444685bb
fix bug of gsm8k_postprocess ( #863 )
...
* fix bug of gsm8k_postprocess
* update postprocess
---------
Co-authored-by: Lei Fei <SENSETIME\leifei1@cn3114002087l.domain.sensetime.com>
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-06 23:52:47 +08:00
Connor-Shen
444d8d9507
[feat] support multipl-e ( #846 )
...
* [feat] support humaneval_multipl-e
* format
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-06 23:30:28 +08:00
bittersweet1999
1c8e193de8
[Fix] hotfix for mtbench ( #877 )
...
* hotfix for mtbench
* hotfix
2024-02-06 21:26:47 +08:00
Fengzhe Zhou
d34ba11106
[Sync] Merge branch 'dev' into zfz/update-keyset-demo ( #876 )
2024-02-05 23:29:10 +08:00
bittersweet1999
32b5948f4e
[Fix] add do sample demo for subjective dataset ( #873 )
...
* add do sample demo for subjective dataset
* fix strings
* format
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-05 15:55:58 +08:00
Skyfall-xzz
7ad1168062
Support NPHardEval ( #835 )
...
* support NPHardEval
* add .md file and fix minor bugs
* refactor and minor fix
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-05 15:52:28 +08:00
bittersweet1999
7806cd0f64
[Feature] support alpacaeval ( #809 )
...
* support alpacaeval_v1
* Update opencompass/summarizers/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/summarizers/subjective/alpacaeval_v1.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* fix conflict
* support alpacaeval v2
* support alpacav2
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2024-02-04 14:18:36 +08:00
RunningLeon
4c87e777d8
[Feature] Add end_str for turbomind ( #859 )
...
* fix
* update
* fix internlm1
* fix docs
* remove sys
2024-02-01 22:31:14 +08:00
bittersweet1999
5c6dc908cd
fix compass arena ( #854 )
2024-01-30 16:34:38 +08:00
Songyang Zhang
cdca59ff49
[Fix] Update Zhipu API and Fix issue min_out_len issue of API models ( #847 )
...
* Update zhipu api and fix min_out_len issue of API class
* Update example
* Update example
2024-01-28 14:52:43 +08:00
Jingming
2801883351
[Fix] Fix acc of IFEval ( #849 )
...
* [Feature] Add IFEval
* [Fix] Changing the Score Rule.
2024-01-27 22:27:07 +08:00
Xiaoming Shi
35aace776a
[Fix] Update MedBench ( #845 )
2024-01-26 17:56:13 +08:00
Songyang Zhang
8ed022b4c4
Update Sensetime API ( #844 )
2024-01-26 16:40:49 +08:00
Hubert
4aa74565e2
[Feat] minor update agent related ( #839 )
...
* [Feat] update cibench
* [Feat] Support CIBench
* [Feat] Support CIBench
* [Feat] Support CIBench
* [Feat] Support CIBench
2024-01-26 14:15:51 +08:00
bittersweet1999
77be07dbb5
[Fix] fix corev2 ( #838 )
...
* fix corev2
* fix corev2
2024-01-24 18:15:29 +08:00
Fengzhe Zhou
0991dd33a0
[Sync] Updata dataset cfg for internMath ( #837 )
...
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
2024-01-24 16:30:32 +08:00
bittersweet1999
2ee8e8a1a1
[Feature] add mtbench ( #829 )
...
* add mtbench
* add mtbench
* Update configs/datasets/subjective/multiround/mtbench_judgeby_gpt4.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update configs/datasets/subjective/multiround/mtbench_judgeby_gpt4.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/datasets/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/datasets/subjective/mtbench.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* fix mtbench
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2024-01-24 12:11:47 +08:00
Jingming
e059a5c2bf
[Feature] Add IFEval ( #813 )
...
* [Feature] Add IFEval
* [Doc] add introduction of IFEval
2024-01-23 20:07:49 +08:00
bittersweet1999
2d4da8dd02
[Feature] Add CompassArena ( #828 )
...
* add compass arena
* add compass_arena
* add compass arena
* Update opencompass/summarizers/subjective/compass_arena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/summarizers/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/datasets/subjective/compass_arena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update opencompass/datasets/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update configs/eval_subjective_compassarena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update configs/datasets/subjective/compassarena/compassarena_compare.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update configs/eval_subjective_compassarena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* Update configs/datasets/subjective/compassarena/compassarena_compare.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* fix check position bias
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2024-01-23 15:12:46 +08:00
RangiLyu
40a2441deb
Update hf_internlm2_chat template ( #823 )
...
* Update hf_internlm2_chat template
* Update 20B
2024-01-19 18:21:47 +08:00
Guo Qipeng
e975a96fa1
Update cdme config and evaluator ( #812 )
...
* update cdme config and evaluator
* fix cdme prompt
* move CDME trim post-processor as a separate evaluator
---------
Co-authored-by: 郭琦鹏 <guoqipeng@pjlab.org.cn>
2024-01-19 11:29:27 +08:00
Mo Li
dcc32ed856
[Fix] Update yi 200k config ( #815 )
2024-01-18 20:54:24 +08:00
RunningLeon
61fe873c89
[Fix] Fix turbomind and update docs ( #808 )
...
* update
* update docs
* add engine_config and gen_config in eval_config
* update
* fix
* fix
* fix
* fix docstr
* fix url
2024-01-18 14:41:35 +08:00
Fengzhe Zhou
b4afe3e7c1
[Sync] Add InternLM2 Keyset Evaluation Demo ( #807 )
...
Co-authored-by: zhangyifan1 <zhangyifan1@pjlab.org.cn>
2024-01-17 13:48:12 +08:00
Mo Li
acae560911
Added support for multi-needle testing in needle-in-a-haystack test ( #802 )
...
* Add NeedleInAHaystack Test
* Apply pre-commit formatting
* Update configs/eval_hf_internlm_chat_20b_cdme.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* add needle in haystack test
* update needle in haystack test
* update plot function in tools_needleinahaystack.py
* optimizing needleinahaystack dataset generation strategy
* modify minor formatting issues
* add English version support
* change NeedleInAHaystackDataset to dynamic loading
* change NeedleInAHaystackDataset to dynamic loading
* fix needleinahaystack test eval bug
* fix needleinahaystack config bug
* Added support for multi-needle testing in needle-in-a-haystack test
* Optimize the code for plotting in the needle-in-a-haystack test.
* Correct the typo in the dataset parameters.
* update needleinahaystack test docs
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2024-01-17 13:47:34 +08:00
RunningLeon
0836aec67b
[Feature] Update evaluate turbomind ( #804 )
...
* update
* fix
* fix
* fix
2024-01-17 11:09:50 +08:00
bittersweet1999
814b3f73bd
reorganize subject files ( #801 )
2024-01-16 18:03:11 +08:00
bittersweet1999
83d6c48378
[Feature] Add configs for creationbench ( #791 )
...
* add creationv2_zh
* add creationv2_zh
* add eng config for creationbench
* add eng config for creationbench
* add eng config for creationbench
2024-01-12 14:20:21 +08:00
Songyang Zhang
467ad0ac21
Update gsm8k agent prompt ( #788 )
2024-01-11 14:07:36 +08:00
notoschord
d3a0ddc3ef
[Feature] Add support for Nanbeige API ( #786 )
...
Co-authored-by: notoschord <wangzekai@kanzhun.com>
2024-01-11 13:54:27 +08:00
Xiaoming Shi
ad872a5dc2
[Feature] Update MedBench ( #779 )
...
* update medbench
* medbench update
* format medbench
* format
* Update
* update
* update
* update suffix
---------
Co-authored-by: 施晓明 <PJLAB\shixiaoming@pjnl104220118l.pjlab.org>
Co-authored-by: Leymore <zfz-960727@163.com>
2024-01-09 11:42:44 +08:00
Fengzhe Zhou
32f40a8f83
[Sync] Sync with internal codes 2023.01.08 ( #777 )
2024-01-08 14:07:24 +00:00
Fengzhe Zhou
f78fcf6eeb
[Docs] Update contamination docs ( #775 )
2024-01-08 16:37:28 +08:00
liyucheng09
0b2863039e
[Feature] Contamination analysis for MMLU, Hellaswag, and ARC_c ( #699 )
...
* Contamination analysis for ARC_c, mmlu, and Hellaswag
* update `eval_contamination.py`
* update `contamination.py` summarizer
* fix `eval_contamination.py`
* add mmlu groups for contamination analysis
2024-01-08 15:51:48 +08:00
Yuchen Yan
11f3b91e78
[Fix] fix typos in drop prompt ( #773 )
...
Co-authored-by: yanyuchen04 <yanyuchen04@meituan.com>
2024-01-08 14:22:35 +08:00
Connor-Shen
30a90d8dd8
Support Mbpp_plus dataset ( #770 )
...
* support mbpp+
* support mbpp+
* minor fix
* [Feat] minor fix
---------
Co-authored-by: yingfhu <yingfhu@gmail.com>
2024-01-05 22:01:57 +08:00
bittersweet1999
2163f9398f
[Feature] add subject ir dataset ( #755 )
...
* add subject ir
* Add ir dataset
* Add ir dataset
2024-01-05 12:00:57 +00:00
bittersweet1999
be369c3e06
[Feature] Add multi_round dataset evaluation ( #766 )
...
* multi_round dataset
* add multi_round evaluation
2024-01-04 10:37:52 +00:00
Chris Liu
3eb225a5e6
[Feature] Support LLaMA2-Accessory ( #732 )
...
* Support LLaMA2-Accessory
* remove strip
* clear imports
* reformat
* fix lint
* fix lint
* update readme
* update readme
* update readme
* update readme
2024-01-02 20:48:51 +08:00
HUANG Fei
ba027eeeac
[Feature] Add support of qwen api ( #735 )
2024-01-02 20:47:12 +08:00
Mo Li
33f8df1ca3
[Update] Change NeedleInAHaystackDataset to dynamic dataset loading ( #754 )
...
* Add NeedleInAHaystack Test
* Apply pre-commit formatting
* Update configs/eval_hf_internlm_chat_20b_cdme.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* add needle in haystack test
* update needle in haystack test
* update plot function in tools_needleinahaystack.py
* optimizing needleinahaystack dataset generation strategy
* modify minor formatting issues
* add English version support
* change NeedleInAHaystackDataset to dynamic loading
* change NeedleInAHaystackDataset to dynamic loading
* fix needleinahaystack test eval bug
* fix needleinahaystack config bug
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2024-01-02 17:22:56 +08:00
Francis-llgg
b69fe2343b
[Feature] Add GPQA Dataset ( #729 )
...
* check
* message
* add
* change prompt
* change a para nameq
* modify name of the file
* delete an useless file
2024-01-01 15:54:40 +08:00
Francis-llgg
ef3ae63539
[Feature] Add new dataset mastermath2024v1 ( #744 )
...
* add new dataset mastermath2024v1
* change it to simplified chinese prompt
* change file name
2024-01-01 15:53:24 +08:00
Mo Li
17b8e929dd
[Feature] Update plot function in tools_needleinahaystack.py ( #747 )
...
* Add NeedleInAHaystack Test
* Apply pre-commit formatting
* Update configs/eval_hf_internlm_chat_20b_cdme.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* add needle in haystack test
* update needle in haystack test
* update plot function in tools_needleinahaystack.py
* optimizing needleinahaystack dataset generation strategy
* modify minor formatting issues
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2023-12-29 18:51:09 +08:00
Hubert
327951087f
[Feat] update code config ( #749 )
...
* [Feat] update code dataset
* [Feat] update code dataset
* [Feat] update code dataset
2023-12-29 18:46:34 +08:00
bittersweet1999
fe0b717033
add creationbench ( #753 )
2023-12-29 10:03:44 +00:00
bittersweet1999
8728287a55
fix erro in configs ( #750 )
2023-12-28 11:53:07 +00:00
Connor-Shen
81098722d2
add chinese version of humaneval, mbpp ( #743 )
...
* add chinese_version of humaneval,mbpp
* add humaneval&mbpp gen.py
* minor fix
* minor add
---------
Co-authored-by: yingfhu <yingfhu@gmail.com>
2023-12-28 14:47:56 +08:00
Hubert
0a525985e8
[Feature] Support sanitized MBPP dataset ( #745 )
2023-12-27 22:17:23 +08:00
bittersweet1999
dfd9ac0fd9
[Feature] Add other judgelm prompts for Alignbench ( #731 )
...
* add judgellm prompts
* add judgelm prompts
* update import info
* fix situation that no abbr in config
* fix situation that no abbr in config
* add summarizer for other judgellm
* change config name
* add maxlen
* add maxlen
* dict assert
* dict assert
* fix strings
* fix strings
2023-12-27 17:54:53 +08:00
Yang Yong
54345c56b7
Update LightllmApi and Fix mmlu bug ( #738 )
...
* Update LightllmApi and Fix mmlu bug
* checkout mmlu_gen_a484b3.py
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2023-12-27 13:49:08 +08:00
philipwangOvO
34561ececb
[Feature] Add InfiniteBench ( #739 )
...
* add InfiniteBench
* add InfiniteBench
---------
Co-authored-by: wangchonghua <wangchonghua@pjlab.org.cn>
2023-12-26 15:36:27 +08:00
Fengzhe Zhou
3a68083ecc
[Sync] update configs ( #734 )
2023-12-25 21:59:16 +08:00
