mirror of
https://github.com/open-compass/opencompass.git
synced 2025-05-30 16:03:24 +08:00
046b6f75c6
* support omni-math * update config * upload README * Delete opencompass/configs/datasets/omni_math/__init__.py * update greedy config & README of LiveMathBench * update intro for max_out_len * rename livemathbench greedy confi * delete greedy config --------- Co-authored-by: liushz <qq1791167085@163.com>
75 lines
1.9 KiB
Markdown
75 lines
1.9 KiB
Markdown
# LiveMathBench
|
|
|
|
## v202412
|
|
|
|
### Details of Datsets
|
|
|
|
| dataset | language | #single-choice | #multiple-choice | #fill-in-the-blank | #problem-solving |
|
|
| -- | -- | -- | -- | -- | -- |
|
|
| AMC | cn | 0 | 0 | 0 | 46 |
|
|
| AMC | en | 0 | 0 | 0 | 46 |
|
|
| CCEE | cn | 0 | 0 | 13 | 31 |
|
|
| CCEE | en | 0 | 0 | 13 | 31 |
|
|
| CNMO | cn | 0 | 0 | 0 | 18 |
|
|
| CNMO | en | 0 | 0 | 0 | 18 |
|
|
| WLPMC | cn | 0 | 0 | 0 | 11 |
|
|
| WLPMC | en | 0 | 0 | 0 | 11 |
|
|
|
|
|
|
### How to use
|
|
|
|
#### G-Pass@k
|
|
```python
|
|
from mmengine.config import read_base
|
|
|
|
with read_base():
|
|
from opencompass.datasets.livemathbench_gen import livemathbench_datasets
|
|
|
|
livemathbench_datasets[0]['eval_cfg']['evaluator'].update(
|
|
{
|
|
'model_name': 'Qwen/Qwen2.5-72B-Instruct',
|
|
'url': [
|
|
'http://0.0.0.0:23333/v1',
|
|
'...'
|
|
] # set url of evaluation models
|
|
}
|
|
)
|
|
livemathbench_dataset['infer_cfg']['inferencer'].update(dict(
|
|
max_out_len=32768 # for o1-like models you need to update max_out_len
|
|
))
|
|
|
|
```
|
|
|
|
#### Greedy
|
|
```python
|
|
from mmengine.config import read_base
|
|
|
|
with read_base():
|
|
from opencompass.datasets.livemathbench_greedy_gen import livemathbench_datasets
|
|
|
|
livemathbench_datasets[0]['eval_cfg']['evaluator'].update(
|
|
{
|
|
'model_name': 'Qwen/Qwen2.5-72B-Instruct',
|
|
'url': [
|
|
'http://0.0.0.0:23333/v1',
|
|
'...'
|
|
] # set url of evaluation models
|
|
}
|
|
)
|
|
livemathbench_dataset['infer_cfg']['inferencer'].update(dict(
|
|
max_out_len=32768 # for o1-like models you need to update max_out_len
|
|
))
|
|
|
|
```
|
|
|
|
### Output Samples
|
|
|
|
| dataset | version | metric | mode | Qwen2.5-72B-Instruct |
|
|
|----- | ----- | ----- | ----- | -----|
|
|
| LiveMathBench | 9befbf | G-Pass@16_0.0 | gen | xx.xx |
|
|
| LiveMathBench | caed8f | G-Pass@16_0.25 | gen | xx.xx |
|
|
| LiveMathBench | caed8f | G-Pass@16_0.5 | gen | xx.xx |
|
|
| LiveMathBench | caed8f | G-Pass@16_0.75 | gen | xx.xx |
|
|
| LiveMathBench | caed8f | G-Pass@16_1.0 | gen | xx.xx |
|
|
|