Yixiao Fang
fada77a31c
[Feature] Add open source dataset eval config of instruct-blip ( #370 )
...
* add configs
* refactor model
* add post processor and prompt constructor
2023-09-08 15:07:09 +08:00
Leymore
49c467458f
[Feature] Update llama2 ( #372 )
2023-09-08 12:47:56 +08:00
Tong Gao
b11838f80a
[Feature] Update claude2 postprocessor ( #365 )
...
* [Feature] Update claude2 config
* [Feature] Update claude2 postprocessor
2023-09-07 11:26:26 +08:00
Yike Yuan
b885ec84df
[Feat] Support Qwen-VL-Chat on MMBench. ( #312 )
...
* [Feat] Support Qwen-VL base.
* [Feat] Support Qwen-VL-Chat on MMBench.
* [Fix] Add postprocessor and fix format.
* [Fix] Add type hint and remove redundant codes.
* [Fix] fix bugs in postprocessor.
* [Fix] Use given commit id.
2023-09-06 18:42:19 +08:00
Hubert
ddb8197212
[Feat] support wizardcoder series ( #344 )
...
* [Feat] support wizardcoder series
* minor fix
2023-09-06 17:52:35 +08:00
Leymore
880b34e759
[Fix] Quick lint fix ( #362 )
...
* add default value
* lint fix
* use None
2023-09-06 14:33:13 +08:00
Tong Gao
5d75c1bbb9
[Enhancement] Increase default task size ( #360 )
2023-09-05 10:38:13 +08:00
Leymore
b8bf16e81c
[Fix] zero retriever add default value ( #361 )
2023-09-05 10:37:42 +08:00
Mashiro
ab21f3be66
[Enhance] Supress warning raised by get_logger ( #353 )
2023-09-04 15:27:08 +08:00
Leymore
a1782f9a08
[Fix] triviaqa & nq postprocess ( #350 )
2023-09-04 15:24:52 +08:00
Tong Gao
ce65d3393b
[Sync] Use finally to clean up temp files ( #337 )
2023-09-04 15:20:16 +08:00
Yixiao Fang
2cd994c3d1
[Fix] add import check of multimodal ( #352 )
2023-09-04 14:41:07 +08:00
Leymore
8774465a8f
[Enhancement] ignore ZeroRetriever error when id_list provided ( #340 )
2023-09-04 11:12:16 +08:00
Yuanhan Zhang
f2dd98ca7a
[Feat] Support LLaVA and mPLUG-Owl ( #331 )
...
* refine gitignore
* [Feature]: Add minigpt-4
* [Feature]: Add mm local runner
* [Feature]: Add instructblip
* add otter and llama-adapter
* add owl
* add llama2-adapter and owl
* lint
* [Feature]: Add minigpt-4
* [Feature]: Add instructblip
* add otter and llama-adapter
* add owl
* add llama2-adapter and owl
* lint
* lint
* update
* lint
* lint
* add __init__.py
* update
* update
* update
---------
Co-authored-by: liuyuan <3463423099@qq.com>
2023-09-01 23:32:05 +08:00
Leymore
e810974068
[Fix] Fix when missing both pad and eos token ( #287 )
...
* fix when missing both pad and eos token
* update pad_token_id impl
2023-08-31 16:53:39 +08:00
Li Bo
a4d6840739
[Feat] Add Otter to OpenCompass MMBench Evaluation ( #232 )
...
* add otter model for opencompass mmbench
* add docs
* add readme docs
* debug for otter opencomass eval
* delete unused folders
* change to default data path
* remove unused files
* remove unused files
* update
* update config file
* flake8 lint formated and add prompt generator
* add prompt generator to config
* add a specific postproecss
* add post processor
* add post processor
* add post processor
* update according to suggestions
* remove unused redefinition
2023-08-31 12:55:53 +08:00
Leymore
7ca6ba625e
[Feature] Add qwen & qwen-chat support ( #286 )
...
* add and apply update suffix tool
* add tool doc
* add qwen configs
* add cmmlu
* rename bbh
* update datasets
* delete
* update hf_qwen_7b.py
2023-08-31 11:29:05 +08:00
Hubert
fd389e2d78
[Feat] support codellama and preds collection tools ( #335 )
2023-08-31 11:14:42 +08:00
Tong Gao
9058be07b8
[Feature] Simplify entry script ( #204 )
...
* [Feature] Simply entry script
* update
2023-08-25 17:36:30 +08:00
Tong Gao
f480b72703
[Feature] Support model-bound prediction postprocessor, use it in Claude ( #268 )
...
* [Feature] Support model-bound text postprocessor, add claude as an example
* update
* update
* minor fix
---------
Co-authored-by: zhoufengzhe <zhoufengzhe@pjlab.org.cn>
2023-08-25 16:12:21 +08:00
Yike Yuan
3f601f420b
[Feat] Support public dataset of visualglm and llava. ( #265 )
...
* [Feat] Add public dataset support of VisualGLM.
* [Feat] Refactor LLaVA.
* [Feat] Add public dataset support of LlaVA.
* [Fix] Add arg.
2023-08-25 15:44:32 +08:00
Yuan Liu
dc6e54f6f4
[Feature]: Verify the acc of these public datasets ( #269 )
...
* [Feature]: Refactor public dataset eval
* [Feature]: Verify public dataset acc
2023-08-25 15:01:58 +08:00
philipwangOvO
3f37c40aa3
[Dataset] Refactor LEval
2023-08-25 11:46:23 +08:00
Tong Gao
60c2d3d76b
[Feature] Add Claude support ( #253 )
...
* [Feature] Add Claude support
* [Feature] Add Claude support
* Update opencompass/models/claude_api.py
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
* raise import erorr
---------
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
2023-08-24 14:29:45 +08:00
Yuan Liu
343f785b07
[Feature]: Add Flamingo ( #258 )
...
* [Feature]: Add Openflamingo MMBench
* [Fix]: Fix import error
* [Fix]: Revert task config
* [Fix]: Fix path bug
2023-08-24 14:11:29 +08:00
LZHgrla
77745a84ea
[Fix] Fix bugs for PeftModel generate ( #252 )
...
* fix bugs
* fix typo
2023-08-24 14:07:33 +08:00
Tong Gao
bd47a00f27
[Fix] use sympy only when necessary ( #255 )
2023-08-24 10:15:20 +08:00
Tong Gao
01372a4806
update ( #251 )
2023-08-23 16:25:23 +08:00
Yixiao Fang
1034c487ef
[Refactor] Refactor instructblip ( #227 )
...
* refactor instructblip
* add post processor
* add forward
* fix lint
* update
* update
2023-08-23 15:33:59 +08:00
liushz
02ce139bc6
[Feature] Add Tree-of-Thought method ( #173 )
...
* Add ToT method
* Update ToT
* Update ToT
* Update ToT
* Update ToT
* Update ToT
* Update ToT
* Update ToT
* Update chain_of_thought.md
* Update icl_tot_inferencer.py
---------
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
2023-08-23 12:23:05 +08:00
Leymore
ff5ab92331
[Feature] Add llama2 native implements ( #235 )
...
* add llama2 native implements
* rename configs/eval_llama_7b.py
---------
Co-authored-by: zhoufengzhe <zhoufengzhe@pjlab.org.cn>
2023-08-23 11:33:25 +08:00
Leymore
fdc69f9d58
[Fix] local runner debug ( #238 )
2023-08-21 16:58:36 +08:00
Yike Yuan
8d368d1cd6
[Feat] Support visualglm and llava for MMBench evaluation. ( #211 )
...
* [Feat] Support visualglm inference on MMBench.
* [Feat] Support llava inference on MMBench.
* [Fix] Fix pre-commit format.
* [Fix] Add docstring for llava
* [Fix] Fix multi-process inference error of LlaVA and add comments.
1. Set `low_cpu_mem_usage` to False to address device issue.
2. Add docstring and type hints.
3. Rename class and remove registry.
* [Fix] Pre-commit fix.
* [Fix] add forward entry, add dynamic import to seedbench
* [Fix] Fix pre-commit.
* [Fix] Fix missing context.
* [Fix] Fix docstring.
2023-08-21 15:57:30 +08:00
Yike Yuan
a6552224cb
[Feat] Support multi-modal evaluation on MME benchmark. ( #197 )
...
* [Feat] Support multi-modal evaluation on MME benchmark.
* [Fix] Remove debug code.
* [Fix] Remove redundant codes and add type hints.
* [Fix] Rename in config.
* [Fix] Rebase main.
* [Fix] Fix isort and yapf conflict.
2023-08-21 15:53:20 +08:00
philipwangOvO
3b29aaee2b
[Fix] bin_trim ( #237 )
...
Co-authored-by: wangchonghua <wangchonghua@pjlab.org.cn>
2023-08-21 15:44:49 +08:00
philipwangOvO
655a807f4b
[Dataset] LongBench ( #236 )
...
Co-authored-by: wangchonghua <wangchonghua@pjlab.org.cn>
2023-08-21 14:15:20 +08:00
Yixiao Fang
0fa2482661
[Feature] Support SEED-Bench ( #203 )
...
* support seedbench
* update docstrings
* update
* update
* update
* update according to review
* rebase
* fix lint
* update
2023-08-17 17:24:02 +08:00
Ezra-Yu
17ccaa5980
[Feat] Add codegeex2 and Humanevalx ( #210 )
...
* add codegeex2
* add humanevalx dataset
* add evaluator
* update evaluator
* update configs
* update clean code
* update configs
* fix lint
* remove sleep
* fix lint
* update docs
* fix lint
2023-08-17 11:03:16 +08:00
Hubert
0fe2366a72
[Feat] support adv_glue dataset for adversarial robustness ( #205 )
...
* [Feat] support adv_glue dataset for adversarial robustness
* reorg files
* minor fix
* minor fix
2023-08-16 18:42:06 +08:00
Yuan Liu
78df9bd0cb
[Feature]: Add other public datasets ( #206 )
...
* [Feature]: Refactor class name
* [Feature]: Add minigpt-4 coco caption
* [Feature]: Update minigpt-4 coco caption
* [Feature]: Add MiniGPT-4 ScienceQA
* [Feature]: Add minigpt-4 vqav2
* [Feature]: Add VSR
* [Feature]: Revert task to previous version
2023-08-16 11:37:26 +08:00
Yike Yuan
3a46b6c64f
[Fix] Fix bugs of multiple rounds of inference when using mm_eval ( #201 )
2023-08-16 11:15:11 +08:00
Hubert
7c393192af
[Fix] fix bug for postprocessor ( #195 )
...
* [Fix] fix bug for postprocessor
* minor fix
2023-08-11 18:41:12 +08:00
Tong Gao
10cbc2b175
Bump version to 0.1.2 ( #190 )
2023-08-11 17:43:14 +08:00
Tong Gao
bf79ff1c6d
[Feature] Add LEval datasets
...
Co-authored-by: kennymckormick <dhd@pku.edu.cn>
2023-08-11 17:38:31 +08:00
Hubert
8d9cee060f
[Feat] update postprocessor to get first option more accurately ( #193 )
...
* [Feat] update postprocessor to get first option
* minor fix
* minor fix
2023-08-11 17:33:00 +08:00
Leymore
14332e08fd
[Feature] add llama-oriented dataset configs ( #82 )
...
* add llama-oriented dataset configs
* update
* revert cvalues & update llama_example
2023-08-11 12:48:05 +08:00
Hubert
5a9539f375
[Feat] add safety to collections ( #185 )
...
* [Feat] add safety to collections
* minor fix
2023-08-11 11:19:26 +08:00
Zaida Zhou
f4c70ba6c3
[Feature] Support filtering specified levels message ( #187 )
...
* Support filtering message
* minor fix
2023-08-11 10:46:46 +08:00
Zaida Zhou
f256abffd3
[Enhancement] Skip invalid keys to avoid requesting API ( #184 )
...
* Skip invalid keys to avoid requesting API
* get expected key
* print warning info
2023-08-10 18:41:43 +08:00
Ma Zerun
59bf56349c
[Feature] Support CUDA_VISIBLE_DEVICES and multiple tasks on one GPU ( #148 )
...
* [Feature] Support CUDA_VISIBLE_DEVICES and multiple tasks on one GPU
* Fix UT
* Update according to comments
2023-08-10 16:53:03 +08:00
