Tong Gao
f480b72703
[Feature] Support model-bound prediction postprocessor, use it in Claude ( #268 )
...
* [Feature] Support model-bound text postprocessor, add claude as an example
* update
* update
* minor fix
---------
Co-authored-by: zhoufengzhe <zhoufengzhe@pjlab.org.cn>
2023-08-25 16:12:21 +08:00
cdpath
6df124d40b
[Docs] update descriptions for tools ( #270 )
2023-08-25 16:00:26 +08:00
Tong Gao
fda42fd5fd
[Fix] wrong path in dataset collections ( #272 )
2023-08-25 15:50:30 +08:00
Yike Yuan
3f601f420b
[Feat] Support public dataset of visualglm and llava. ( #265 )
...
* [Feat] Add public dataset support of VisualGLM.
* [Feat] Refactor LLaVA.
* [Feat] Add public dataset support of LlaVA.
* [Fix] Add arg.
2023-08-25 15:44:32 +08:00
Yuan Liu
dc6e54f6f4
[Feature]: Verify the acc of these public datasets ( #269 )
...
* [Feature]: Refactor public dataset eval
* [Feature]: Verify public dataset acc
2023-08-25 15:01:58 +08:00
philipwangOvO
3f37c40aa3
[Dataset] Refactor LEval
2023-08-25 11:46:23 +08:00
Tong Gao
60c2d3d76b
[Feature] Add Claude support ( #253 )
...
* [Feature] Add Claude support
* [Feature] Add Claude support
* Update opencompass/models/claude_api.py
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
* raise import erorr
---------
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
2023-08-24 14:29:45 +08:00
Yuan Liu
343f785b07
[Feature]: Add Flamingo ( #258 )
...
* [Feature]: Add Openflamingo MMBench
* [Fix]: Fix import error
* [Fix]: Revert task config
* [Fix]: Fix path bug
2023-08-24 14:11:29 +08:00
LZHgrla
77745a84ea
[Fix] Fix bugs for PeftModel generate ( #252 )
...
* fix bugs
* fix typo
2023-08-24 14:07:33 +08:00
Songyang Zhang
2a5cef2914
Update .owners.yml ( #261 )
2023-08-24 13:41:14 +08:00
Tong Gao
bd47a00f27
[Fix] use sympy only when necessary ( #255 )
2023-08-24 10:15:20 +08:00
Tong Gao
01372a4806
update ( #251 )
2023-08-23 16:25:23 +08:00
Yixiao Fang
1034c487ef
[Refactor] Refactor instructblip ( #227 )
...
* refactor instructblip
* add post processor
* add forward
* fix lint
* update
* update
2023-08-23 15:33:59 +08:00
liushz
02ce139bc6
[Feature] Add Tree-of-Thought method ( #173 )
...
* Add ToT method
* Update ToT
* Update ToT
* Update ToT
* Update ToT
* Update ToT
* Update ToT
* Update ToT
* Update chain_of_thought.md
* Update icl_tot_inferencer.py
---------
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
2023-08-23 12:23:05 +08:00
Leymore
ff5ab92331
[Feature] Add llama2 native implements ( #235 )
...
* add llama2 native implements
* rename configs/eval_llama_7b.py
---------
Co-authored-by: zhoufengzhe <zhoufengzhe@pjlab.org.cn>
2023-08-23 11:33:25 +08:00
Leymore
c0e58632ca
[Doc] Add summarizer doc ( #231 )
...
* add summarizer doc
* update
* update doc
* Apply suggestions from code review
---------
Co-authored-by: Tong Gao <gaotongxiao@gmail.com>
2023-08-23 11:18:01 +08:00
liushz
a85634a32a
[Enhancement] Update run.py ( #247 )
...
* Update run.py
* Update run.py
2023-08-23 10:56:21 +08:00
Songyang Zhang
0d574c036f
update news ( #243 )
...
Co-authored-by: zhangsongyang <zhangsongyang@pjlab.org.cn>
2023-08-22 11:31:14 +08:00
Songyang Zhang
8f7bdb4b36
Update news ( #241 )
2023-08-21 23:03:53 +08:00
Leymore
fdc69f9d58
[Fix] local runner debug ( #238 )
2023-08-21 16:58:36 +08:00
Yike Yuan
8d368d1cd6
[Feat] Support visualglm and llava for MMBench evaluation. ( #211 )
...
* [Feat] Support visualglm inference on MMBench.
* [Feat] Support llava inference on MMBench.
* [Fix] Fix pre-commit format.
* [Fix] Add docstring for llava
* [Fix] Fix multi-process inference error of LlaVA and add comments.
1. Set `low_cpu_mem_usage` to False to address device issue.
2. Add docstring and type hints.
3. Rename class and remove registry.
* [Fix] Pre-commit fix.
* [Fix] add forward entry, add dynamic import to seedbench
* [Fix] Fix pre-commit.
* [Fix] Fix missing context.
* [Fix] Fix docstring.
2023-08-21 15:57:30 +08:00
Yike Yuan
a6552224cb
[Feat] Support multi-modal evaluation on MME benchmark. ( #197 )
...
* [Feat] Support multi-modal evaluation on MME benchmark.
* [Fix] Remove debug code.
* [Fix] Remove redundant codes and add type hints.
* [Fix] Rename in config.
* [Fix] Rebase main.
* [Fix] Fix isort and yapf conflict.
2023-08-21 15:53:20 +08:00
philipwangOvO
3b29aaee2b
[Fix] bin_trim ( #237 )
...
Co-authored-by: wangchonghua <wangchonghua@pjlab.org.cn>
2023-08-21 15:44:49 +08:00
philipwangOvO
655a807f4b
[Dataset] LongBench ( #236 )
...
Co-authored-by: wangchonghua <wangchonghua@pjlab.org.cn>
2023-08-21 14:15:20 +08:00
Tong Gao
c6a3494993
[Fix] requirements ( #229 )
2023-08-18 14:34:20 +08:00
Yuan Liu
90c07a3dfd
[Fix]: Fix name ( #223 )
2023-08-17 18:30:48 +08:00
Yuan Liu
3d49a20b95
[Feature]: Add launch script ( #222 )
2023-08-17 18:26:01 +08:00
Yixiao Fang
0fa2482661
[Feature] Support SEED-Bench ( #203 )
...
* support seedbench
* update docstrings
* update
* update
* update
* update according to review
* rebase
* fix lint
* update
2023-08-17 17:24:02 +08:00
Yuan Liu
ae3c1869da
[Feature]: Add other public datasets config ( #214 )
...
* [Feature]: Add flickr30k
* [Feature]: Add GQA
* [Feature]: Add OCR VQA
* [Feature]: Add OK VQA
* [Feature]: Add text vqa
* [Feature]: Add other vqa
2023-08-17 11:11:26 +08:00
Ezra-Yu
17ccaa5980
[Feat] Add codegeex2 and Humanevalx ( #210 )
...
* add codegeex2
* add humanevalx dataset
* add evaluator
* update evaluator
* update configs
* update clean code
* update configs
* fix lint
* remove sleep
* fix lint
* update docs
* fix lint
2023-08-17 11:03:16 +08:00
Hubert
0fe2366a72
[Feat] support adv_glue dataset for adversarial robustness ( #205 )
...
* [Feat] support adv_glue dataset for adversarial robustness
* reorg files
* minor fix
* minor fix
2023-08-16 18:42:06 +08:00
Ezra-Yu
d7cb39581a
update conf ( #212 )
2023-08-16 15:22:14 +08:00
Yuan Liu
78df9bd0cb
[Feature]: Add other public datasets ( #206 )
...
* [Feature]: Refactor class name
* [Feature]: Add minigpt-4 coco caption
* [Feature]: Update minigpt-4 coco caption
* [Feature]: Add MiniGPT-4 ScienceQA
* [Feature]: Add minigpt-4 vqav2
* [Feature]: Add VSR
* [Feature]: Revert task to previous version
2023-08-16 11:37:26 +08:00
Yike Yuan
3a46b6c64f
[Fix] Fix bugs of multiple rounds of inference when using mm_eval ( #201 )
2023-08-16 11:15:11 +08:00
Leymore
4fc1701209
[Doc] update readme ( #196 )
...
* update readme
* Apply suggestions from code review
---------
Co-authored-by: Tong Gao <gaotongxiao@gmail.com>
2023-08-11 18:43:41 +08:00
Hubert
7c393192af
[Fix] fix bug for postprocessor ( #195 )
...
* [Fix] fix bug for postprocessor
* minor fix
2023-08-11 18:41:12 +08:00
Tong Gao
10cbc2b175
Bump version to 0.1.2 ( #190 )
2023-08-11 17:43:14 +08:00
Tong Gao
bf79ff1c6d
[Feature] Add LEval datasets
...
Co-authored-by: kennymckormick <dhd@pku.edu.cn>
2023-08-11 17:38:31 +08:00
Hubert
8d9cee060f
[Feat] update postprocessor to get first option more accurately ( #193 )
...
* [Feat] update postprocessor to get first option
* minor fix
* minor fix
2023-08-11 17:33:00 +08:00
Leymore
14332e08fd
[Feature] add llama-oriented dataset configs ( #82 )
...
* add llama-oriented dataset configs
* update
* revert cvalues & update llama_example
2023-08-11 12:48:05 +08:00
Tong Gao
e464265cf8
[Docs] Update contribution guide & toc, improve user experience ( #188 )
...
* [Docs] Update contribution guide & toc
* update
* Update docs/en/notes/contribution_guide.md
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
* update
* update
---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
2023-08-11 11:36:09 +08:00
Hubert
5a9539f375
[Feat] add safety to collections ( #185 )
...
* [Feat] add safety to collections
* minor fix
2023-08-11 11:19:26 +08:00
Zaida Zhou
f4c70ba6c3
[Feature] Support filtering specified levels message ( #187 )
...
* Support filtering message
* minor fix
2023-08-11 10:46:46 +08:00
Songyang Zhang
99ae786598
[Feature] update news ( #186 )
...
* update news
* update
---------
Co-authored-by: gaotongxiao <gaotongxiao@gmail.com>
2023-08-10 18:52:09 +08:00
Zaida Zhou
f256abffd3
[Enhancement] Skip invalid keys to avoid requesting API ( #184 )
...
* Skip invalid keys to avoid requesting API
* get expected key
* print warning info
2023-08-10 18:41:43 +08:00
Tong Gao
0406e4e7ed
[Docs] Enhance issue template ( #183 )
2023-08-10 17:02:58 +08:00
Ma Zerun
59bf56349c
[Feature] Support CUDA_VISIBLE_DEVICES and multiple tasks on one GPU ( #148 )
...
* [Feature] Support CUDA_VISIBLE_DEVICES and multiple tasks on one GPU
* Fix UT
* Update according to comments
2023-08-10 16:53:03 +08:00
Tong Gao
312095de9d
[Fix] meta template & unit tests ( #170 )
2023-08-10 16:49:13 +08:00
liushz
ed248af136
[Fix] Fix some sc errors ( #177 )
...
* Update sc
* Update sc doc
* Apply suggestions from code review
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
---------
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
2023-08-10 16:40:32 +08:00
Tong Gao
2931f3dcb8
[Enhancement] Add humaneval postprocessor for GPT models & eval config for GPT4, enhance the original humaneval postprocessor ( #129 )
...
* [Enhancement] Enhance humaneval postprocessor
* add human-eval testcase
* update
* update
---------
Co-authored-by: Leymore <zfz-960727@163.com>
2023-08-10 16:31:12 +08:00
