jiangzhs/OpenCompass
1
0
Fork 0
mirror of https://github.com/open-compass/opencompass.git synced 2025-05-30 16:03:24 +08:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
944 Commits 11 Branches 35 Tags 66 MiB
b84518c656
Commit Graph

176 Commits

Author SHA1 Message Date
Songyang Zhang
cdca59ff49
[Fix] Update Zhipu API and Fix issue min_out_len issue of API models (#847) 
* Update zhipu api and fix min_out_len issue of API class

* Update example

* Update example
 2024-01-28 14:52:43 +08:00
Songyang Zhang
8ed022b4c4
Update Sensetime API (#844) 2024-01-26 16:40:49 +08:00
Hubert
4aa74565e2
[Feat] minor update agent related (#839) 
* [Feat] update cibench

* [Feat] Support CIBench

* [Feat] Support CIBench

* [Feat] Support CIBench

* [Feat] Support CIBench
 2024-01-26 14:15:51 +08:00
Songyang Zhang
793e32c9cc
[Feature] Update API implementation (#834) 2024-01-24 13:35:21 +08:00
Yang Yong
f09a2ff418
Add LightllmApi KeyError log & Update doc (#816) 
* Add LightllmApi KeyError log

* Update LightllmApi doc
 2024-01-18 22:23:38 +08:00
RunningLeon
61fe873c89
[Fix] Fix turbomind and update docs (#808) 
* update

* update docs

* add engine_config and gen_config in eval_config

* update

* fix

* fix

* fix

* fix docstr

* fix url
 2024-01-18 14:41:35 +08:00
Fengzhe Zhou
b4afe3e7c1
[Sync] Add InternLM2 Keyset Evaluation Demo (#807) 
Co-authored-by: zhangyifan1 <zhangyifan1@pjlab.org.cn>
 2024-01-17 13:48:12 +08:00
RunningLeon
0836aec67b
[Feature] Update evaluate turbomind (#804) 
* update

* fix

* fix

* fix
 2024-01-17 11:09:50 +08:00
notoschord
d3a0ddc3ef
[Feature] Add support for Nanbeige API (#786) 
Co-authored-by: notoschord <wangzekai@kanzhun.com>
 2024-01-11 13:54:27 +08:00
bittersweet1999
5679edb490
add temperature in alles (#787) 2024-01-11 03:57:24 +00:00
Fengzhe Zhou
32f40a8f83
[Sync] Sync with internal codes 2023.01.08 (#777) 2024-01-08 14:07:24 +00:00
jiangjin1999
8194199d79
[Feature] *_batch_generate* function, add the MultiTokenEOSCriteria (#772) 
* jiangjin1999: in the _batch_generate function, add the MultiTokenEOSCriteria feature to speed up inference.

* jiangjin1999: in the _batch_generate function, add the MultiTokenEOSCriteria feature to speed up inference.

---------

Co-authored-by: jiangjin08 <jiangjin08@MBP-2F32S5MD6P-0029.local>
Co-authored-by: jiangjin08 <jiangjin08@a.sh.vip.dianping.com>
 2024-01-08 16:40:02 +08:00
Chris Liu
3eb225a5e6
[Feature] Support LLaMA2-Accessory (#732) 
* Support LLaMA2-Accessory

* remove strip

* clear imports

* reformat

* fix lint

* fix lint

* update readme

* update readme

* update readme

* update readme
 2024-01-02 20:48:51 +08:00
HUANG Fei
ba027eeeac
[Feature] Add support of qwen api (#735) 2024-01-02 20:47:12 +08:00
Hubert
327951087f
[Feat] update code config (#749) 
* [Feat] update code dataset

* [Feat] update code dataset

* [Feat] update code dataset
 2023-12-29 18:46:34 +08:00
Yang Yong
54345c56b7
Update LightllmApi and Fix mmlu bug (#738) 
* Update LightllmApi and Fix mmlu bug

* checkout mmlu_gen_a484b3.py

---------

Co-authored-by: Leymore <zfz-960727@163.com>
 2023-12-27 13:49:08 +08:00
AllentDan
336d8d76ff
add turbomind restful api support (#693) 
* add turbomind restful api support

* config

* top_p 0.8

* top_k = 1
 2023-12-24 01:40:00 +08:00
RunningLeon
e34c552282
[Feature] Update configs for evaluating chat models like qwen, baichuan, llama2 using turbomind backend (#721) 
* add llama2 test

* fix

* test qwen chat-7b

* test w4

* add baichuan2

* update

* update

* update configs and docs

* update
 2023-12-21 18:22:17 +08:00
Hubert
eda72e756e
[Fix] minor fix openai (#711) 2023-12-18 15:45:31 +08:00
Songyang Zhang
637628a70f
[Doc] Update Doc for Alignbench (#707) 
* update alignmentbench

* update alignmentbench

* update doc

* update

* update
 2023-12-15 15:07:25 +08:00
Songyang Zhang
bfe4aa2af5
[Fix] Update alignmentbench (#704) 
* update alignmentbench

* update alignmentbench

* update alignmentbench
 2023-12-14 18:24:21 +08:00
Hubert
e78857ac36
[Sync] minor test (#683) 2023-12-11 17:42:53 +08:00
Songyang Zhang
e25c5f9525
[Enhancement] Update API Interface and Mixtral (#681) 
* [Enhancement] Update API interface

* [Enhancement] Update API interface

* Update mixtral

* Update readme
 2023-12-10 13:29:26 +08:00
Hubert
9eb5cadcac
[Feat] update gsm8k and math agent config (#652) 
* [Feat] update gsm8k and math agent config

* minor fix
 2023-12-01 15:08:38 +08:00
Ma Zerun
6aaf3b91ec
[Feature] Support chat style inferencer. (#643) 
* [Feature] Support chat style inferencer.

* [Fix] use new prompt

* [Fix] use new prompt

---------

Co-authored-by: yingfhu <yingfhu@gmail.com>
 2023-11-30 14:00:06 +08:00
Yang Yong
522241a8c9
[Fix] Fix lightllmapi list bug (#635) 2023-11-24 14:24:13 +08:00
Songyang Zhang
5202456b4c
[API] Update API (#624) 
* update api

* update generation_kwargs impl

* update api

* refactor

---------

Co-authored-by: Leymore <zfz-960727@163.com>
 2023-11-23 15:06:20 +08:00
Fengzhe Zhou
d4d1330a5a
[Sync] Fix cmnli, fix vicuna meta template, fix longbench postprocess and other minor fixes (#625) 2023-11-23 14:05:59 +08:00
Kevin Wang
c0785e53d8
[Feature] support download from modelscope (#534) 
* [Feature] download from modelscope

* [Feature] download from modelscope

* minor fix

---------

Co-authored-by: yingfhu <yingfhu@gmail.com>
 2023-11-22 15:32:21 +08:00
Songyang Zhang
721a45c68f
[Bug] Update api with generation_kargs (#614) 
* update api

* update generation_kwargs impl

---------

Co-authored-by: Leymore <zfz-960727@163.com>
 2023-11-22 10:02:57 +08:00
Lyu Han
eb56fd6d16
Integrate turbomind python api (#484) 
* integrate turbomind python api

* update

* update user guide

* update

* fix according to reviewer's comments

* fix error

* fix linting

* update user guide

* remove debug log

---------

Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>
 2023-11-21 22:34:46 +08:00
Songyang Zhang
d925748266
[Feature] Support 360API and FixKRetriever for CSQA dataset (#601) 
* [Feature] Support 360API and FixKRetriever for CSQA dataset

* Update API

* Update API

* [Feature] Support 360API and FixKRetriever for CSQA dataset

* Update API

* Update API

* rm mathbench

* fix_lint

* Update opencompass/models/bytedance_api.py

Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>

* update

* update

* update

---------

Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
 2023-11-21 20:25:47 +08:00
Yang Yong
d3b0d5c4ce
[Feature] Support Lightllm API (#613) 
* [Feature] Support Lightllm api

* formatting & renaming

---------

Co-authored-by: Leymore <zfz-960727@163.com>
 2023-11-21 19:18:40 +08:00
Yuan Feng
7199acc25d
Add support for DataCanvas Alaya LM (#612) 
* Support for Alaya

* Remove useless requirements
 2023-11-21 17:51:30 +08:00
Hubert
91fba2c2e9
[Feat] support humaneval and mbpp pass@k (#598) 
* [Feat] support pass@ k

* [Feat] support pass@k

* [Feat] support pass@k

* [Feat] support pass@k

* [Feat] support pass@k

* [Feat] support pass@k docs

* update naming

---------

Co-authored-by: Leymore <zfz-960727@163.com>
 2023-11-16 21:22:06 +08:00
Fengzhe Zhou
d3de5c41fb
[Sync] update model configs (#574) 2023-11-13 15:15:34 +08:00
Songyang Zhang
9e42cb163b
[Feature] Update xunfei api (#572) 
* update xunfei api

* fix lint

* avoid warning
 2023-11-10 22:46:06 +08:00
Hubert
9f8a721313
[Fix] fix registry error with internal (#551) 
* [Fix] fix conflict with internal

* [Fix] fix conflict with internal
 2023-11-07 20:01:23 +08:00
Hubert
bb2ecf416e
[Feat] Support cibench (#538) 
* [Feat] support cidataset

* [Feat] support cidataset

* [Feat] support cidataset

* [Feat] support cidataset

* minor fix

* minor fix

* minor fix

* minor fix

* minor fix

* minor fix

* rename cibench

* rename cibench

* rename cibench

* rename cibench

* minor fix

* minor fix

* minor fix
 2023-11-07 19:11:44 +08:00
Songyang Zhang
239c2a346e
[Feature] Add support for MiniMax API (#548) 
* update requirement

* update requirement

* update with minimax

* update api model

* Update readme

* fix error

---------

Co-authored-by: zhangsongyang <zhangsongyang@pjlab.org.cn>
 2023-11-06 21:57:32 +08:00
Hubert
1ccdfaa623
[Feat] support xunfei api (#547) 2023-11-06 19:29:26 +08:00
Fengzhe Zhou
dbb20b8270
[Sync] update (#517) 2023-10-27 20:31:22 +08:00
Fengzhe Zhou
df07391ed8
[Fix] Enforce do_sample=False in HF model (#506) 
* update hf model wrapper

* patch llama

---------

Co-authored-by: bot <bot@bot.com>
 2023-10-27 16:54:19 +08:00
Fengzhe Zhou
e3d4901bed
[Feat] Add _set_model_kwargs_torch_dtype for HF model (#507) 
* add _set_model_kwargs_torch_dtype for hf models

* add logger
 2023-10-27 11:45:41 +08:00
Hubert
b3f5d9e421
[Feat] support math/gms8k agent config (#494) 
* support math agent

* support gsm8k agent

* support gsm8k agent

* minor fix

* minor fix

* minor fix

* Update configs/eval_codeagent.py
 2023-10-25 23:05:15 +08:00
Hubert
ac3a2c4501
[Feat] local api speed up with fixed concurrent users (#497) 
* [Feat] local api speed up

* fix lint

* fix lint

* minor fix

* add example api
 2023-10-25 21:12:20 +08:00
Lyu Han
6738247142
Integrate turbomind inference via its RPC API instead of its python API (#414) 
* support tis

* integrate turbomind inference via its RPC API instead of its python API

* update guide

* update ip address spec

* update according to reviewer's comments
 2023-10-07 10:27:48 +08:00
philipwangOvO
3bb3d330eb
[Sync] Update LongEval (#443) 2023-09-27 16:32:40 +08:00
Ma Zerun
0f2c388280
Support GSM8k evaluation with tools by Lagent and LangChain (#277) 
* Support GSM8k evaluation with tools by Lagent and LangChain

* Avoid to use MMEngine new feature

* update document

---------

Co-authored-by: Leymore <zfz-960727@163.com>
 2023-09-22 15:28:22 +08:00
Tong Gao
681d3013de
[Feature] Log gold answer in prediction output (#419) 
* [Feature] Log gold answer in prediction output

* support clp golden ans

* minor fix

---------

Co-authored-by: yingfhu <yingfhu@gmail.com>
 2023-09-22 12:44:40 +08:00
Leymore
49c467458f
[Feature] Update llama2 (#372) 2023-09-08 12:47:56 +08:00
Tong Gao
b11838f80a
[Feature] Update claude2 postprocessor (#365) 
* [Feature] Update claude2 config

* [Feature] Update claude2 postprocessor
 2023-09-07 11:26:26 +08:00
Leymore
e810974068
[Fix] Fix when missing both pad and eos token (#287) 
* fix when missing both pad and eos token

* update pad_token_id impl
 2023-08-31 16:53:39 +08:00
Tong Gao
9058be07b8
[Feature] Simplify entry script (#204) 
* [Feature] Simply entry script

* update
 2023-08-25 17:36:30 +08:00
Tong Gao
f480b72703
[Feature] Support model-bound prediction postprocessor, use it in Claude (#268) 
* [Feature] Support model-bound text postprocessor, add claude as an example

* update

* update

* minor fix

---------

Co-authored-by: zhoufengzhe <zhoufengzhe@pjlab.org.cn>
 2023-08-25 16:12:21 +08:00
Tong Gao
60c2d3d76b
[Feature] Add Claude support (#253) 
* [Feature] Add Claude support

* [Feature] Add Claude support

* Update opencompass/models/claude_api.py

Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>

* raise import erorr

---------

Co-authored-by: Hubert <42952108+yingfhu@users.noreply.github.com>
 2023-08-24 14:29:45 +08:00
LZHgrla
77745a84ea
[Fix] Fix bugs for PeftModel generate (#252) 
* fix bugs

* fix typo
 2023-08-24 14:07:33 +08:00
Leymore
ff5ab92331
[Feature] Add llama2 native implements (#235) 
* add llama2 native implements

* rename configs/eval_llama_7b.py

---------

Co-authored-by: zhoufengzhe <zhoufengzhe@pjlab.org.cn>
 2023-08-23 11:33:25 +08:00
philipwangOvO
3b29aaee2b
[Fix] bin_trim (#237) 
Co-authored-by: wangchonghua <wangchonghua@pjlab.org.cn>
 2023-08-21 15:44:49 +08:00
philipwangOvO
655a807f4b
[Dataset] LongBench (#236) 
Co-authored-by: wangchonghua <wangchonghua@pjlab.org.cn>
 2023-08-21 14:15:20 +08:00
Zaida Zhou
f256abffd3
[Enhancement] Skip invalid keys to avoid requesting API (#184) 
* Skip invalid keys to avoid requesting API

* get expected key

* print warning info
 2023-08-10 18:41:43 +08:00
Tong Gao
312095de9d
[Fix] meta template & unit tests (#170) 2023-08-10 16:49:13 +08:00
Songyang Zhang
3f36db3b06
[Feature] Support turbomind (#166) 
* support turbomind

* update doc

* Update docs/en/advanced_guides/evaluation_turbomind.md

Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

* Update docs/zh_cn/advanced_guides/evaluation_turbomind.md

Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

* Update docs/zh_cn/advanced_guides/evaluation_turbomind.md

Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

* Update docs/en/advanced_guides/evaluation_turbomind.md

Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

* update

---------

Co-authored-by: Tong Gao <gaotongxiao@gmail.com>
 2023-08-10 16:25:11 +08:00
Haodong Duan
d5d4f47371
[API] Refine OpenAI (#175) 2023-08-09 12:38:57 +08:00
Zaida Zhou
af436f5951
[Feature] Calculate max_out_len without hard code for OpenAI model (#158) 
* calulate max_out_len without hard code

* set default value

* update configs

* Update configs/eval_gpt3.5.py

Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

---------

Co-authored-by: Tong Gao <gaotongxiao@gmail.com>
 2023-08-08 15:16:56 +08:00
Tong Gao
bbdedc6c95
[Enhancement] Optimize OpenAI models (#128) 
* [Feature] Enhance OpenAI API, add example config for GPT evaluation
 2023-08-03 14:55:16 +08:00
Leymore
d862f570aa
[Feature] Add SC (#126) 
* add self-consistency

* add CoT method Self-Consistency

* fix typo error and update openicl_eval

* add tydiQA-GoldP task

* fix sc

* rename gsm8k_sc

* fix sc

* add self-consistency doc

* refine sc

---------

Authored-by: liushz <qq1791167085@163.com>
 2023-07-28 17:29:37 +08:00
Haodong Duan
46c9645753
[Feature] Allow explicitly setting the temperature for API model (#121) 
* allow explicitly setting the temperature

* update
 2023-07-28 11:28:15 +08:00
gowithme
57fcfc975a
[Feature] Support intern lanuage model (#51) 
* support internLM

* support internLM

* simplify intern model files

* update storage_manager

* support internLM

* Modify the file organization structure

* support internLM

* support internLM

* support internLM

* support internLM

* change some details
 2023-07-27 18:49:36 +08:00
Leymore
eea8b04417
[Feature] Add llama-2 models (#81) 
* add llama-2 models

* update docs

---------

Co-authored-by: gaotongxiao <gaotongxiao@gmail.com>
 2023-07-19 19:51:29 +08:00
LZH
26e2f171f4
[Feature] Support load PEFT adapter for HuggingFace model (#74) 
* support peft for HuggingFace model

* add docstring
 2023-07-18 16:21:43 +08:00
Tong Gao
1e44541730
[Enhancement] Test linting in CI and fix existing linting errors (#69) 
* [Enhancement] Test linting in CI

* fix linting
 2023-07-17 15:59:10 +08:00
Tong Gao
7ee5a86fee
[Feature] Enhance OpenAI API, add example config for GPT evaluation (#53) 
* [Feature] Enhance OpenAI API, add example config for GPT evaluation

* fix
 2023-07-12 16:43:46 +08:00
mzr1996
04dd01a235 Update configs and code 2023-07-05 11:45:08 +08:00
Leymore
c94cc94348 Add release contribution 2023-07-05 03:15:31 +00:00
gaotongxiao
7d346000bb initial commit 2023-07-04 21:34:55 +08:00