OpenCompass/configs/datasets/apps
Connor-Shen 3098d78845
[Bench] Support APPS (#963)
* [Feat] support apps

* [Feat] support apps

* [Feat] support apps

* update README
2024-03-13 16:09:23 +08:00
..
apps_gen_5b4254.py Update configs (#9) 2023-07-06 12:27:41 +08:00
apps_gen_7fbb95.py Update configs (#9) 2023-07-06 12:27:41 +08:00
apps_gen_b4dee3.py Update configs (#9) 2023-07-06 12:27:41 +08:00
apps_gen_d82929.py [Bench] Support APPS (#963) 2024-03-13 16:09:23 +08:00
apps_gen.py [Bench] Support APPS (#963) 2024-03-13 16:09:23 +08:00
README.md [Bench] Support APPS (#963) 2024-03-13 16:09:23 +08:00

APPS

Dataset Description

APPS is a benchmark for code generation with 10000 problems. It can be used to evaluate the ability of language models to generate code from natural language specifications.

Dataset Structure

DatasetDict({
    train: Dataset({
        features: ['problem_id', 'question', 'solutions', 'input_output', 'difficulty', 'url', 'starter_code'],
        num_rows: 5000
    })
    test: Dataset({
        features: ['problem_id', 'question', 'solutions', 'input_output', 'difficulty', 'url', 'starter_code'],
        num_rows: 5000
    })
})

How to Use

You can also filter the dataset based on difficulty level: introductory, interview and competition. Just pass a list of difficulty levels to the filter. For example, if you want the most challenging questions, you need to select the competition level:

ds = load_dataset("codeparrot/apps", split="train", difficulties=["competition"])
print(next(iter(ds))["question"])

Evaluation results

Dataset Metric Baichuan2-7B Baichuan2-13B InternLM2-7B InternLM2-20B
APPS(testset) pass@1 0.0 0.06 0.0 0.0

Please refer to Table 3 of code llama for original results if needed.

Citation

@article{hendrycksapps2021,
  title={Measuring Coding Challenge Competence With APPS},
  author={Dan Hendrycks and Steven Basart and Saurav Kadavath and Mantas Mazeika and Akul Arora and Ethan Guo and Collin Burns and Samir Puranik and Horace He and Dawn Song and Jacob Steinhardt},
  journal={NeurIPS},
  year={2021}
}