Arshdeep Singh
fc95a851a7
Adding support for fireworks and vllm models and sample for wiki model eval
2024-04-22 21:15:33 -04:00
Fengzhe Zhou
004ed79593
[Feature] Add TheoremQA with 5-shot ( #1048 )
...
* add TheoremQA with 5-shot
* cherry pick from add-huggingface-above-v4.33, good TheoremQA results
2024-04-22 15:22:04 +08:00
bittersweet1999
6f98c8d9ab
[Fix] Fix MultiRound Subjective Evaluation( #1043 )
...
* fix multiround
* fix
2024-04-22 12:06:03 +08:00
Fengzhe Zhou
8c85edd1cd
[Sync] deprecate old mbpps ( #1064 )
2024-04-19 20:49:46 +08:00
Robin Chen
c172401323
[Fix] Fixed repeated loading of VLLM ( #1051 )
...
* [fix]Fixed the issue caused by the repeated loading of VLLM model during task segmentation.
* [fix] avoid TypeError: VLLM.__init__() got an unexpected keyword argument 'tokenizer_only'
* restore .pre-commit-config.yaml
* restore opencompass/tasks/openicl_infer.py
---------
Co-authored-by: IcyFeather <mengzhuo.happy@gmail.com>
Co-authored-by: Leymore <zfz-960727@163.com>
2024-04-17 20:36:08 +08:00
Fengzhe Zhou
881bdbf6bd
[Sync] Bump version to 0.2.4 ( #1052 )
...
(cherry picked from commit 16ac6306c72fa202173289b55eaefe85e0fcb73c)
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
2024-04-16 18:09:46 +08:00
Fengzhe Zhou
7a41951dda
[Fix] logger.error -> logger.debug in OpenAI wrapper ( #1050 )
...
* logger.error -> logger.info in OpenAI
* logger.info -> logger.debug in OpenAI
2024-04-15 21:08:13 +08:00
liuwei130
a00e57296f
[Feature] Add ChemBench ( #1032 )
...
* add ChemBench
* update results
* molbench -> ChemBench
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-04-12 08:46:26 +08:00
Fengzhe Zhou
b39f501563
[Sync] update taco ( #1030 )
2024-04-09 17:50:23 +08:00
Mo Li
16f29b25f1
[Fix] Simplify needlebench summarizer ( #1024 )
...
* Conflicts:
configs/summarizers/needlebench.py
* fix lint problems
2024-04-07 17:51:13 +08:00
Mo Li
f2af49337d
[Feature] Add ATC Choice Version ( #1019 )
...
* Squashed commit of the following:
commit c48ad194c3976dc63d1b60d8c8ab2d5ff9e1cbfe
Author: DseidLi <2568818204@qq.com>
Date: Tue Apr 2 16:57:43 2024 +0800
add atc_choice
commit 3ac6efea29619573e6fac8fa3cce464853dcead0
Merge: 2d4e5590a6a03f2527fda2527fda8a575425acdf8367ba1ba61caf1cf8caf1cf8120bf8bhttps://github.com/open-compass/opencompass into atc_choice
commit 4e747ed1988ddbcfcc7fff334601259ade72d363
Author: DseidLi <2568818204@qq.com>
Date: Sun Mar 3 22:15:25 2024 +0800
add internlm2-lmdeploy model and gemma configs
commit 7dabc828123d711c8cf834d6aab4137bb55e85ed
Author: DseidLi <2568818204@qq.com>
Date: Sat Mar 2 17:26:15 2024 +0800
add atc choice version -ZH
commit 996f8ae43df7266e873c1c7375681db6524f3ebfc0d1190e390965baf7855d32b31eb8af82a7f08532bf9fe53fe78832bf9fe802a7cb025e05
2024-04-07 15:46:20 +08:00
Mo Li
b50d163265
[Fix] Refactor Needlebench Configs for CLI Testing Support ( #1020 )
...
* add needlebench datasets suffix
* fix import
* update run.py args for summarizer key and dataset suffix
* update utils/run.py
2024-04-07 15:12:56 +08:00
bittersweet1999
2d4e559763
[Feature] Add multi-model judge and fix some problems ( #1016 )
...
* support multi-model judge and moe judge
* test_moe
* test_moe
* test
* add moe judge
* support multi-judge-model
2024-04-02 11:52:06 +08:00
bittersweet1999
02e7eec911
[Feature] Support AlpacaEval_V2 ( #1006 )
...
* support alpacaeval_v2
* support alpacaeval
* update docs
* update docs
2024-03-28 16:49:04 +08:00
Mo Li
0a6a03fe1a
[Feature] update needlebench and configs ( #986 )
...
* add Needlebench-1000K configs
* add prompt postion args
* add model configs
* Update parallel.py
* fix lint
2024-03-25 18:05:01 +08:00
Chaseldot
1d3198554b
[Fix] base.py change status into list ( #994 )
2024-03-22 17:06:34 +08:00
Ke Bao
e415ddf96a
[Fix] Fix turbomind_tis ( #992 )
2024-03-22 15:50:12 +08:00
Connor-Shen
0221d30877
[Fix] Update APPS/TACO ( #988 )
...
* [Feature] update apps/taco
* [Feature] update apps/taco
2024-03-19 20:21:39 +08:00
Connor-Shen
8a3c6e51ed
[Feature] Update APPS ( #985 )
...
* update post process
* update post process
2024-03-19 15:47:05 +08:00
Connor-Shen
d92595b671
[Feat] Support TACO ( #966 )
...
* [Feat] Support TACO
* update README
* update README
2024-03-19 15:39:16 +08:00
bittersweet1999
c78a4df923
add support for set prediction path ( #984 )
2024-03-19 14:32:15 +08:00
Jingming
89a8a8917b
[Feature] Add the implement of QuALITY datasets ( #976 )
...
#976
2024-03-15 21:22:38 +08:00
Connor-Shen
3098d78845
[Bench] Support APPS ( #963 )
...
* [Feat] support apps
* [Feat] support apps
* [Feat] support apps
* update README
2024-03-13 16:09:23 +08:00
Fengzhe Zhou
ab6cdb2be8
[Sync] Bump version 0.2.3 ( #957 )
2024-03-12 11:51:56 +08:00
Fengzhe Zhou
64fde73b15
[Fix] Use logger.error on failure ( #960 )
2024-03-12 11:51:39 +08:00
Fengzhe Zhou
bdd85358cc
[Sync] update 20240308 ( #953 )
2024-03-11 22:34:19 +08:00
bittersweet1999
848e7c8a76
[fix] add different temp for different question in mtbench ( #954 )
...
* add temp for mtbench
* add document for mtbench
* add document for mtbench
2024-03-11 17:24:39 +08:00
Yang Yong
3829be87b1
Fix LightllmApi ppl test ( #951 )
2024-03-08 12:04:44 +08:00
Yang Yong
107e022cf4
Support prompt template for LightllmApi. Update LightllmApi token bucket. ( #945 )
2024-03-06 15:33:53 +08:00
RunningLeon
c54a5d3b0f
Support get_ppl for TurbomindModel ( #878 )
...
* update ppl for turbomindmodel
* update api_server
* rename config and set thread_safe for pytorch engine if possible
2024-03-06 11:44:19 +08:00
Fengzhe Zhou
b03d5dc531
[Sync] Sync Internal ( #941 )
2024-03-04 14:42:36 +08:00
yuantao2108
bbec7d8733
[Feature] add lveval benchmark ( #914 )
...
* add lveval benchmark
* add LVEval readme file
* update LVEval readme file
* Update configs/eval_bluelm_32k_lveval.py
* Update configs/eval_llama2_7b_lveval.py
---------
Co-authored-by: yuantao <yuantao@infini-ai.com>
Co-authored-by: Mo Li <82895469+DseidLi@users.noreply.github.com>
2024-03-04 11:22:03 +08:00
Mo Li
8142f399a8
[Feature] Upgrade the needle-in-a-haystack experiment to Needlebench ( #913 )
...
* add needlebench
* simplify needlebench 32k, 128k, 200k for eval
* update act prompt
* fix bug in needlebench summarizer
* add needlebench intro, fix summarizer
* lint summarizer
* fix linting error
* move readme.md
* update readme for needlebench
* update docs of needlebench
* simplify needlebench summarizers
2024-03-04 11:10:52 +08:00
Kdump
3e9844ed33
[Fix]Fixed the problem of never entering task.run() mode in local scheduling mode. ( #930 )
...
* Fixed the problem of never entering task.run() mode in local scheduling mode.
get_command_template方法中为命令行前缀添加了CUDA_VISIBLE_DEVICES=或set CUDA_VISIBLE_DEVICES=。导致task.run()分支失效。
---------
CUDA_VISIBLE_DEVICES= or set CUDA_VISIBLE_DEVICES= is added to the command line prefix in the get_command_template method. Causes the task.run() branch to fail.
* [Fix]Fixed the problem of never entering task.run() mode in local scheduling mode.
get_command_template方法中为命令行前缀添加了CUDA_VISIBLE_DEVICES=或set CUDA_VISIBLE_DEVICES=。导致task.run()分支失效。
---
CUDA_VISIBLE_DEVICES= or set CUDA_VISIBLE_DEVICES= is added to the command line prefix in the get_command_template method. Causes the task.run() branch to fail.
* [Fix]Fixed the problem of never entering task.run() mode in local scheduling mode.
get_command_template方法中为命令行前缀添加了CUDA_VISIBLE_DEVICES=或set CUDA_VISIBLE_DEVICES=。导致task.run()分支失效。
CUDA_VISIBLE_DEVICES= or set CUDA_VISIBLE_DEVICES= is added to the command line prefix in the get_command_template method. Causes the task.run() branch to fail.
2024-02-29 14:35:45 +08:00
Skyfall-xzz
4c45a71bbc
[Feature] Support OpenFinData ( #896 )
...
* [Feature] Support OpenFinData
* add README for OpenFinData
* update README
2024-02-29 12:55:07 +08:00
bittersweet1999
001e77fea2
[Feature] add support for gemini ( #931 )
...
* add gemini
* add gemini
* add gemini
2024-02-28 19:38:34 +08:00
Fengzhe Zhou
9afbfa3639
[Sync] Fix TEvalEvaluator ( #929 )
2024-02-28 16:05:30 +08:00
Fengzhe Zhou
5ce8e0450e
[Fix] Fix type hint in IFEval ( #915 )
2024-02-28 10:53:40 +08:00
Jingming
53fe788d27
[Fix] fix ifeval ( #909 )
2024-02-23 16:52:03 +08:00
bittersweet1999
45c606bcd0
[Fix] Fix IFEval ( #906 )
...
* fix ifeval
* fix ifeval
* fix ifeval
* fix ifeval
2024-02-22 16:51:34 +08:00
RunningLeon
32ba0b074e
Support lmdeploy pytorch engine ( #875 )
...
* add lmdeploy pytorch model
* fix
* speed up encoding and decoding
* fix
* change tokenizer
2024-02-22 03:46:07 -03:00
Yang Yong
b6e21ece38
Support LightllmApi input_format ( #888 )
2024-02-19 10:02:59 +08:00
Fengzhe Zhou
08133e060a
[Sync] Bump version to 0.2.2 ( #880 )
2024-02-07 10:45:48 +08:00
hailsham
dd444685bb
fix bug of gsm8k_postprocess ( #863 )
...
* fix bug of gsm8k_postprocess
* update postprocess
---------
Co-authored-by: Lei Fei <SENSETIME\leifei1@cn3114002087l.domain.sensetime.com>
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-06 23:52:47 +08:00
Connor-Shen
444d8d9507
[feat] support multipl-e ( #846 )
...
* [feat] support humaneval_multipl-e
* format
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-06 23:30:28 +08:00
Yggdrasill7D6
a6c49f15ce
fix lawbench 2-1 f0.5 score calculation bug ( #795 )
...
* fix lawbench 2-1 f0.5 score calculation bug
* use path in overall datasets folder
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-06 22:20:11 +08:00
bittersweet1999
1c8e193de8
[Fix] hotfix for mtbench ( #877 )
...
* hotfix for mtbench
* hotfix
2024-02-06 21:26:47 +08:00
Fengzhe Zhou
d34ba11106
[Sync] Merge branch 'dev' into zfz/update-keyset-demo ( #876 )
2024-02-05 23:29:10 +08:00
Skyfall-xzz
7ad1168062
Support NPHardEval ( #835 )
...
* support NPHardEval
* add .md file and fix minor bugs
* refactor and minor fix
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2024-02-05 15:52:28 +08:00
Yuchen Yan
fed7d800c6
[Fix] Fix error in gsm8k evaluator ( #782 )
...
Co-authored-by: jiangjin1999 <1261842974@qq.com>
2024-02-04 22:55:11 +08:00
