Merge 56fc5748d8 into 07930b854a

examples/eval_benchmax.py

@@ -0,0 +1,20 @@
+from mmengine.config import read_base
+
+with read_base():
+    from opencompass.configs.datasets.xlivecodebench.xlivecodebench_gen import \
+        LCB_datasets
+    from opencompass.configs.datasets.xgpqa.xgpqa_gen import \
+        gpqa_datasets
+    from opencompass.configs.datasets.xIFEval.xIFeval_gen import \
+        xifeval_datasets
+    from opencompass.configs.models.hf_llama.hf_llama3_8b_instruct import \
+        models as hf_llama3_8b_instruct_models
+    
+datasets = [
+    *LCB_datasets,
+    *gpqa_datasets,
+    *xifeval_datasets
+]
+models = [
+    *hf_llama3_8b_instruct_models
+]

opencompass/configs/datasets/xIFEval/IFEval.md

@@ -0,0 +1,55 @@
+# IFEval
+## Introduction
+The following introduction comes from the abstract in [Instruction Following Evaluation for Large Language Models](arxiv.org/abs/2311.07911)
+
+```
+IFEval is a straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword of AI at least 3 times". We identified 25 types of those verifiable instructions and constructed around 500 prompts, with each prompt containing one or more verifiable instructions.
+```
+
+## Official link
+
+### Paper
+
+[Instruction Following Evaluation for Large Language Models](arxiv.org/abs/2311.07911)
+
+### Repository
+
+[instruction_following_eval](github.com/google-research/google-research/tree/master/instruction_following_eval)
+
+## Examples
+Input example I:
+```
+Write a 300+ word summary of the wikipedia page \"https://en.wikipedia.org/wiki/Raymond_III,_Count_of_Tripoli\". Do not use any commas and highlight at least 3 sections that has titles in markdown format, for example *highlighted section part 1*, *highlighted section part 2*, *highlighted section part 3*.
+```
+Output example I (from GPT-4):
+```
+Raymond III was the Count of Tripoli from 1152 to 1187 and Prince of Galilee and Tiberias in the Kingdom of Jerusalem. He was born in 1140 to Raymond II of Tripoli and Hodierna of Jerusalem. His mother was the daughter of Baldwin II of Jerusalem. \n\n*Early Life and Succession*\n\nRaymond III was only a child when his father was murdered. His mother Hodierna was regent until Raymond came of age. In 1155 Raymond married Eschiva the daughter of Walter I of Beirut. They had three children: Raymond IV Bertrand and a daughter who married Guy of Lusignan. \n\n*Reign*\n\nRaymond III's reign was marked by a series of conflicts and alliances with the Muslim world. He was involved in the defense of the Kingdom of Jerusalem against Nur ad-Din and later Saladin. He was also a key figure in the internal politics of the kingdom. He was a supporter of the queen mother Amalric of Jerusalem and opposed the succession of Guy of Lusignan. \n\n*Later Life and Death*\n\nIn 1187 Raymond III was part of the disastrous Battle of Hattin where the Christian forces were defeated by Saladin. He was one of the few to escape the battlefield but died later that year. His son Raymond IV succeeded him as Count of Tripoli. \n\nRaymond III's life and reign were marked by the complex politics and warfare of the Crusader states. He was a key figure in the defense of the Kingdom of Jerusalem and his death marked a significant loss for the Christian forces in the region. His legacy is a testament to the turbulent times in which he lived and the challenges faced by the Crusader states in their relations with the Muslim world.
+```
+Input example II:
+```
+"I am planning a trip to Japan, and I would like thee to write an itinerary for my journey in a Shakespearean style. You are not allowed to use any commas in your response."
+```
+Output example II (from GPT-4):
+```
+Hark! Hearken to the tale of thy journey to the land of the rising sun, Japan. \n\nUpon the morrow of thy arrival, thou shalt find thyself in Tokyo, the city of lights and towers. In the morn, thou shalt explore the grandeur of the Imperial Palace, a sight to behold. As the sun reaches its zenith, thou shalt partake in a feast of sushi, a delicacy of the land. As the day wanes, thou shalt wander through the electric streets of Akihabara, a realm of technology and animation.\n\nOn the second day, thou shalt journey to the ancient city of Kyoto. In the soft light of dawn, thou shalt visit the golden pavilion of Kinkaku-ji, a vision of beauty and tranquility. As the day unfolds, thou shalt wander through the bamboo grove of Arashiyama, a place of peace and serenity. As the sun sets, thou shalt partake in a traditional tea ceremony, a ritual of grace and harmony.\n\nOn the third day, thou shalt travel to the city of Hiroshima. In the morning, thou shalt pay thy respects at the Peace Memorial Park, a place of remembrance and hope. As the day progresses, thou shalt journey to the island of Miyajima, home to the floating torii gate, a sight of wonder and awe. As the day ends, thou shalt feast on okonomiyaki, a dish of delight and satisfaction.\n\nOn the fourth day, thou shalt return to Tokyo. In the morning, thou shalt ascend the Tokyo Skytree, a tower of height and grandeur. As the day unfolds, thou shalt explore the bustling streets of Shibuya, a realm of fashion and culture. As the sun sets, thou shalt dine in the vibrant district of Roppongi, a place of gastronomy and nightlife.\n\nOn the fifth and final day, thou shalt bid farewell to the land of the rising sun. With memories of beauty and wonder, thou shalt embark on thy journey home, carrying with thee the spirit of Japan. \n\nThus ends the tale of thy journey, a tale of exploration and discovery. May thy travels be filled with joy and adventure, and may the land of the rising sun forever hold a place in thy heart.
+```
+## Evaluation results
+
+```
+dataset    version    metric                        mode      baichuan2-7b-chat-hf    baichuan2-13b-chat-hf    internlm2-chat-7b-hf    internlm2-chat-20b-hf    llama-2-7b-chat-hf    llama-2-13b-chat-hf
+---------  ---------  ----------------------------  ------  ----------------------  -----------------------  ----------------------  -----------------------  --------------------  ---------------------
+IFEval     3321a3     Prompt-level-strict-accuracy  gen                      36.04                    35.49                   38.26                    33.09                 33.46                  33.64
+IFEval     3321a3     Inst-level-strict-accuracy    gen                      46.76                    46.76                   49.16                    45.32                 45.68                  45.44
+IFEval     3321a3     Prompt-level-loose-accuracy   gen                      37.52                    37.71                   42.51                    39.37                 43.81                  47.32
+IFEval     3321a3     Inst-level-loose-accuracy     gen                      48.44                    49.16                   53.72                    51.08                 55.64                  58.03
+```
+
+## Reference
+```
+@article{zhou2023instruction,
+  title={Instruction-Following Evaluation for Large Language Models},
+  author={Zhou, Jeffrey and Lu, Tianjian and Mishra, Swaroop and Brahma, Siddhartha and Basu, Sujoy and Luan, Yi and Zhou, Denny and Hou, Le},
+  journal={arXiv preprint arXiv:2311.07911},
+  year={2023}
+}
+```

opencompass/configs/datasets/xIFEval/IFEval_gen.py

@@ -0,0 +1,4 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .IFEval_gen_3321a3 import ifeval_datasets  # noqa: F401, F403

opencompass/configs/datasets/xIFEval/IFEval_gen_3321a3.py

@@ -0,0 +1,34 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import xIFEvalDataset, xIFEvaluator
+
+xifeval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='reference')
+
+xifeval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=2048))
+
+xifeval_eval_cfg = dict(
+    evaluator=dict(type=xIFEvaluator),
+    pred_role='BOT',
+)
+LANG = 'de'
+
+xifeval_datasets = [
+    dict(
+        abbr='xIFEval',
+        type=xIFEvalDataset,
+        path=f'data/ifeval/input_data_google_{LANG}.jsonl',
+        reader_cfg=xifeval_reader_cfg,
+        infer_cfg=xifeval_infer_cfg,
+        eval_cfg=xifeval_eval_cfg)
+]

opencompass/configs/datasets/xIFEval/README.md

@@ -0,0 +1,31 @@
+# IFEval
+
+```bash
+python3 run.py --models hf_internlm2_chat_7b --datasets IFEval_gen_3321a3 --debug
+```
+
+## Chat Models
+
+|             model             |   Prompt-level-strict-accuracy |   Inst-level-strict-accuracy |   Prompt-level-loose-accuracy |   Inst-level-loose-accuracy |
+|:-----------------------------:|-------------------------------:|-----------------------------:|------------------------------:|----------------------------:|
+|     qwen1.5-0.5b-chat-hf      |                          13.12 |                        23.26 |                         15.71 |                       26.38 |
+|     qwen1.5-1.8b-chat-hf      |                          16.08 |                        26.26 |                         18.30 |                       29.02 |
+|      qwen1.5-4b-chat-hf       |                          25.51 |                        35.97 |                         28.84 |                       39.81 |
+|      qwen1.5-7b-chat-hf       |                          38.82 |                        50.00 |                         42.70 |                       53.48 |
+|      qwen1.5-14b-chat-hf      |                          42.51 |                        54.20 |                         49.17 |                       59.95 |
+|      qwen1.5-32b-chat-hf      |                          49.54 |                        60.43 |                         53.97 |                       64.39 |
+|      qwen1.5-72b-chat-hf      |                          51.02 |                        61.99 |                         57.12 |                       67.27 |
+|     qwen1.5-110b-chat-hf      |                          55.08 |                        65.59 |                         61.18 |                       70.86 |
+|    internlm2-chat-1.8b-hf     |                          18.30 |                        28.78 |                         21.44 |                       32.01 |
+|  internlm2-chat-1.8b-sft-hf   |                          18.67 |                        31.18 |                         19.78 |                       32.85 |
+|     internlm2-chat-7b-hf      |                          34.75 |                        46.28 |                         40.48 |                       51.44 |
+|   internlm2-chat-7b-sft-hf    |                          39.19 |                        50.12 |                         42.33 |                       52.76 |
+|     internlm2-chat-20b-hf     |                          36.41 |                        48.68 |                         40.67 |                       53.24 |
+|   internlm2-chat-20b-sft-hf   |                          44.55 |                        55.64 |                         46.77 |                       58.03 |
+|    llama-3-8b-instruct-hf     |                          68.02 |                        76.74 |                         75.42 |                       82.85 |
+|    llama-3-70b-instruct-hf    |                          78.00 |                        84.65 |                         84.29 |                       89.21 |
+| llama-3-8b-instruct-lmdeploy  |                          69.13 |                        77.46 |                         77.26 |                       83.93 |
+| llama-3-70b-instruct-lmdeploy |                          75.97 |                        82.97 |                         83.18 |                       88.37 |
+|  mistral-7b-instruct-v0.1-hf  |                          40.30 |                        50.96 |                         41.96 |                       53.48 |
+|  mistral-7b-instruct-v0.2-hf  |                          49.17 |                        60.43 |                         51.94 |                       64.03 |
+| mixtral-8x7b-instruct-v0.1-hf |                          50.09 |                        60.67 |                         55.64 |                       65.83 |

opencompass/configs/datasets/xIFEval/xIFeval_gen.py

@@ -0,0 +1,37 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import xIFEvalDataset, xIFEvaluator
+
+xifeval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='reference')
+
+xifeval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=4096))
+
+xifeval_eval_cfg = dict(
+    evaluator=dict(type=xIFEvaluator),
+    pred_role='BOT',
+)
+
+xifeval_datasets = []
+
+LANGS = ['ar', 'bn', 'cs', 'de', 'es', 'fr', 'hu', 'ja', 'ko', 'ru', 'sr', 'sw', 'te', 'th', 'vi', 'zh']
+for LANG in LANGS:
+    path = f'data/xifeval/input_data_google_{LANG}.jsonl'
+    xifeval_datasets.append(dict(
+        type=xIFEvalDataset,
+        abbr=f'xIFEval_{LANG}',
+        path=path,
+        reader_cfg=xifeval_reader_cfg,
+        infer_cfg=xifeval_infer_cfg,
+        eval_cfg=xifeval_eval_cfg,
+    ))

opencompass/configs/datasets/xgpqa/README.md

@@ -0,0 +1,69 @@
+# GPQA
+
+```bash
+python3 run.py --models hf_internlm2_7b --datasets gpqa_ppl_6bf57a --debug
+python3 run.py --models hf_internlm2_chat_7b --datasets gpqa_gen_4baadb --debug
+```
+
+## Base Models
+
+|          model           |   GPQA_diamond |
+|:------------------------:|---------------:|
+|    llama-7b-turbomind    |          24.24 |
+|   llama-13b-turbomind    |          25.25 |
+|   llama-30b-turbomind    |          22.73 |
+|   llama-65b-turbomind    |          21.72 |
+|   llama-2-7b-turbomind   |          25.25 |
+|  llama-2-13b-turbomind   |          23.74 |
+|  llama-2-70b-turbomind   |          28.28 |
+|   llama-3-8b-turbomind   |          31.82 |
+|  llama-3-70b-turbomind   |          40.91 |
+| internlm2-1.8b-turbomind |          24.24 |
+|  internlm2-7b-turbomind  |          28.28 |
+| internlm2-20b-turbomind  |          31.31 |
+|   qwen-1.8b-turbomind    |          28.79 |
+|    qwen-7b-turbomind     |          24.75 |
+|    qwen-14b-turbomind    |          27.78 |
+|    qwen-72b-turbomind    |          31.31 |
+|     qwen1.5-0.5b-hf      |          23.74 |
+|     qwen1.5-1.8b-hf      |          28.79 |
+|      qwen1.5-4b-hf       |          23.23 |
+|      qwen1.5-7b-hf       |          20.71 |
+|      qwen1.5-14b-hf      |          32.32 |
+|      qwen1.5-32b-hf      |          30.81 |
+|      qwen1.5-72b-hf      |          31.82 |
+|   qwen1.5-moe-a2-7b-hf   |          28.79 |
+|    mistral-7b-v0.1-hf    |          24.75 |
+|    mistral-7b-v0.2-hf    |          23.74 |
+|   mixtral-8x7b-v0.1-hf   |          28.79 |
+|  mixtral-8x22b-v0.1-hf   |          36.36 |
+|         yi-6b-hf         |          28.28 |
+|        yi-34b-hf         |          35.86 |
+|   deepseek-7b-base-hf    |          20.71 |
+|   deepseek-67b-base-hf   |          25.25 |
+
+## Chat Models
+
+|             model             |   GPQA_diamond |
+|:-----------------------------:|---------------:|
+|     qwen1.5-0.5b-chat-hf      |          19.70 |
+|     qwen1.5-1.8b-chat-hf      |          29.80 |
+|      qwen1.5-4b-chat-hf       |          25.25 |
+|      qwen1.5-7b-chat-hf       |          31.82 |
+|      qwen1.5-14b-chat-hf      |          30.30 |
+|      qwen1.5-32b-chat-hf      |          31.31 |
+|      qwen1.5-72b-chat-hf      |          32.83 |
+|     qwen1.5-110b-chat-hf      |          35.86 |
+|    internlm2-chat-1.8b-hf     |          25.76 |
+|  internlm2-chat-1.8b-sft-hf   |          26.26 |
+|     internlm2-chat-7b-hf      |          28.28 |
+|   internlm2-chat-7b-sft-hf    |          27.27 |
+|     internlm2-chat-20b-hf     |          30.30 |
+|   internlm2-chat-20b-sft-hf   |          29.29 |
+|    llama-3-8b-instruct-hf     |          25.76 |
+|    llama-3-70b-instruct-hf    |          37.88 |
+| llama-3-8b-instruct-lmdeploy  |          25.76 |
+| llama-3-70b-instruct-lmdeploy |          37.88 |
+|  mistral-7b-instruct-v0.1-hf  |          30.30 |
+|  mistral-7b-instruct-v0.2-hf  |          25.25 |
+| mixtral-8x7b-instruct-v0.1-hf |          30.30 |

opencompass/configs/datasets/xgpqa/gpqa_0shot_nocot_gen_772ea0.py

@@ -0,0 +1,52 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import GPQADataset, GPQA_Simple_Eval_postprocess, GPQAEvaluator
+
+# openai_simple_eval prompt
+align_prompt = """
+Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of ABCD.
+
+{question}
+
+A) {A}
+B) {B}
+C) {C}
+D) {D}
+""".strip()
+
+gpqa_reader_cfg = dict(
+    input_columns=['question', 'A', 'B', 'C', 'D'],
+    output_column='answer')
+
+gpqa_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            round=[
+                dict(role='HUMAN', prompt=align_prompt),
+            ], )),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer))
+
+gpqa_eval_cfg = dict(evaluator=dict(type=GPQAEvaluator),
+                     pred_postprocessor=dict(type=GPQA_Simple_Eval_postprocess))
+
+gpqa_datasets = []
+gpqa_subsets = {
+    # 'extended': 'gpqa_extended.csv',
+    # 'main': 'gpqa_main.csv',
+    'diamond': 'gpqa_diamond.csv'
+}
+
+for split in list(gpqa_subsets.keys()):
+    gpqa_datasets.append(
+        dict(
+            abbr='GPQA_' + split,
+            type=GPQADataset,
+            path='./data/gpqa/',
+            name=gpqa_subsets[split],
+            reader_cfg=gpqa_reader_cfg,
+            infer_cfg=gpqa_infer_cfg,
+            eval_cfg=gpqa_eval_cfg)
+    )

opencompass/configs/datasets/xgpqa/gpqa_few_shot_ppl_4b5a83.py

@@ -0,0 +1,53 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import FixKRetriever
+from opencompass.openicl.icl_inferencer import PPLInferencer
+from opencompass.openicl.icl_evaluator import AccwithDetailsEvaluator
+from opencompass.datasets import GPQADataset, GPQAEvaluator
+from opencompass.utils import first_option_postprocess
+
+gpqa_reader_cfg = dict(
+    input_columns=['question', 'A', 'B', 'C', 'D'],
+    output_column='answer')
+
+hint = f'For the multiple choice question below, please provide the correct answer option directly.'
+question_and_options = 'Question: {question}\n(A){A}\n(B){B}\n(C){C}\n(D){D}\n'
+gpqa_infer_cfg = dict(
+    ice_template=dict(
+        type=PromptTemplate,
+        template={
+        opt: f'{question_and_options}\nAnswer: {opt}' for opt in ['A', 'B', 'C', 'D']},
+        ),
+    prompt_template=dict(
+        type=PromptTemplate,
+        template={
+        opt: f'{hint}\n</E>{question_and_options}\nAnswer: {opt}' for opt in ['A', 'B', 'C', 'D']
+        },
+        ice_token='</E>'
+        ),
+    retriever=dict(type=FixKRetriever, fix_id_list=[0, 1, 2, 3, 4]),
+    inferencer=dict(type=PPLInferencer))
+
+gpqa_eval_cfg = dict(evaluator=dict(type=AccwithDetailsEvaluator))
+
+gpqa_datasets = []
+LANG = ['de']
+gpqa_subsets = {
+    lang: f'gpqa_main_{lang}.csv' for lang in LANG
+}
+# gpqa_subsets = {
+#     # 'extended': 'gpqa_extended.csv',
+#     # 'main': 'gpqa_main.csv',
+#     'diamond': 'gpqa_main_{}.csv'
+# }
+
+for split in list(gpqa_subsets.keys()):
+    gpqa_datasets.append(
+        dict(
+            abbr='GPQA_' + split,
+            type=GPQADataset,
+            path='./data/gpqa/',
+            name=gpqa_subsets[split],
+            reader_cfg=gpqa_reader_cfg,
+            infer_cfg=gpqa_infer_cfg,
+            eval_cfg=gpqa_eval_cfg)
+    )

opencompass/configs/datasets/xgpqa/gpqa_gen_015262.py

@@ -0,0 +1,46 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import GPQADataset, GPQAEvaluator
+from opencompass.utils import first_option_postprocess
+
+gpqa_reader_cfg = dict(
+    input_columns=['question', 'A', 'B', 'C', 'D'],
+    output_column='answer')
+
+gpqa_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            round=[
+                dict(role='HUMAN', prompt='What is the correct answer to this question: {question}\nChoices:\n'
+                                          '(A){A}\n'
+                                          '(B){B}\n'
+                                          '(C){C}\n'
+                                          '(D){D}\n'
+                                          'Format your response as follows: "The correct answer is (insert answer here)"'),
+            ], )),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer))
+
+gpqa_eval_cfg = dict(evaluator=dict(type=GPQAEvaluator),
+                     pred_postprocessor=dict(type=first_option_postprocess, options='ABCD'))
+
+gpqa_datasets = []
+gpqa_subsets = {
+    'extended': 'gpqa_extended.csv',
+    'main': 'gpqa_main.csv',
+    'diamond': 'gpqa_diamond.csv'
+}
+
+for split in list(gpqa_subsets.keys()):
+    gpqa_datasets.append(
+        dict(
+            abbr='GPQA_' + split,
+            type=GPQADataset,
+            path='./data/gpqa/',
+            name=gpqa_subsets[split],
+            reader_cfg=gpqa_reader_cfg,
+            infer_cfg=gpqa_infer_cfg,
+            eval_cfg=gpqa_eval_cfg)
+    )

opencompass/configs/datasets/xgpqa/gpqa_gen_4baadb.py

@@ -0,0 +1,46 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import GPQADataset, GPQAEvaluator
+from opencompass.utils import first_option_postprocess
+
+gpqa_reader_cfg = dict(
+    input_columns=['question', 'A', 'B', 'C', 'D'],
+    output_column='answer')
+
+gpqa_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            round=[
+                dict(role='HUMAN', prompt='What is the correct answer to this question: {question}\nChoices:\n'
+                                          '(A){A}\n'
+                                          '(B){B}\n'
+                                          '(C){C}\n'
+                                          '(D){D}\n'
+                                          'Format your response as follows: "The correct answer is (insert answer here)"'),
+            ], )),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer))
+
+gpqa_eval_cfg = dict(evaluator=dict(type=GPQAEvaluator),
+                     pred_postprocessor=dict(type=first_option_postprocess, options='ABCD'))
+
+gpqa_datasets = []
+gpqa_subsets = {
+    # 'extended': 'gpqa_extended.csv',
+    # 'main': 'gpqa_main.csv',
+    'diamond': 'gpqa_diamond.csv'
+}
+
+for split in list(gpqa_subsets.keys()):
+    gpqa_datasets.append(
+        dict(
+            abbr='GPQA_' + split,
+            type=GPQADataset,
+            path='./data/gpqa/',
+            name=gpqa_subsets[split],
+            reader_cfg=gpqa_reader_cfg,
+            infer_cfg=gpqa_infer_cfg,
+            eval_cfg=gpqa_eval_cfg)
+    )

opencompass/configs/datasets/xgpqa/gpqa_openai_simple_evals_gen_5aeece.py

@@ -0,0 +1,57 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import GPQADataset, GPQA_Simple_Eval_postprocess, GPQAEvaluator
+
+# openai_simple_eval prompt
+align_prompt = """
+Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering.
+
+{question}
+
+A) {A}
+B) {B}
+C) {C}
+D) {D}
+""".strip()
+
+gpqa_reader_cfg = dict(
+    input_columns=['question', 'A', 'B', 'C', 'D'],
+    output_column='answer')
+
+gpqa_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            round=[
+                dict(role='HUMAN', prompt=align_prompt),
+            ], )),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer))
+
+gpqa_eval_cfg = dict(evaluator=dict(type=GPQAEvaluator),
+                     pred_postprocessor=dict(type=GPQA_Simple_Eval_postprocess))
+
+gpqa_datasets = []
+# gpqa_subsets = {
+#     # 'extended': 'gpqa_extended.csv',
+#     # 'main': 'gpqa_main.csv',
+#     'diamond': 'gpqa_diamond.csv'
+# }
+gpqa_datasets = []
+LANGS = ['en','ar', 'bn', 'cs', 'es', 'fr', 'hu', 'ja', 'ko', 'ru', 'sr', 'sw', 'te', 'th', 'vi', 'zh']
+gpqa_subsets = {
+    lang: f'gpqa_main_{lang}.csv' for lang in LANGS
+}
+
+for split in list(gpqa_subsets.keys()):
+    gpqa_datasets.append(
+        dict(
+            abbr='GPQA_' + split,
+            type=GPQADataset,
+            path='./data/gpqa/', # need to change
+            name=gpqa_subsets[split],
+            reader_cfg=gpqa_reader_cfg,
+            infer_cfg=gpqa_infer_cfg,
+            eval_cfg=gpqa_eval_cfg)
+    )

opencompass/configs/datasets/xgpqa/gpqa_ppl_6bf57a.py

@@ -0,0 +1,40 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import PPLInferencer
+from opencompass.datasets import GPQADataset, GPQAEvaluator
+from opencompass.utils import first_option_postprocess
+
+gpqa_reader_cfg = dict(
+    input_columns=['question', 'A', 'B', 'C', 'D'],
+    output_column='answer')
+
+gpqa_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template={
+            opt: f'Question: {{question}}\n(A){{A}}\n(B){{B}}\n(C){{C}}\n(D){{D}}\nAnswer: {opt}' for opt in ['A', 'B', 'C', 'D']
+        }),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=PPLInferencer))
+
+gpqa_eval_cfg = dict(evaluator=dict(type=GPQAEvaluator),
+                     pred_postprocessor=dict(type=first_option_postprocess, options='ABCD'))
+
+gpqa_datasets = []
+gpqa_subsets = {
+    # 'extended': 'gpqa_extended.csv',
+    # 'main': 'gpqa_main.csv',
+    'diamond': 'gpqa_diamond.csv'
+}
+
+for split in list(gpqa_subsets.keys()):
+    gpqa_datasets.append(
+        dict(
+            abbr='GPQA_' + split,
+            type=GPQADataset,
+            path='./data/gpqa/',
+            name=gpqa_subsets[split],
+            reader_cfg=gpqa_reader_cfg,
+            infer_cfg=gpqa_infer_cfg,
+            eval_cfg=gpqa_eval_cfg)
+    )

opencompass/configs/datasets/xgpqa/xgpqa_gen.py

@@ -0,0 +1,7 @@
+from mmengine.config import read_base
+
+# with read_base():
+#     from .gpqa_openai_simple_evals_gen_5aeece import gpqa_datasets
+
+with read_base():
+    from .gpqa_openai_simple_evals_gen_5aeece import gpqa_datasets

opencompass/configs/datasets/xhumaneval/README.md

@@ -0,0 +1,69 @@
+# HumanEval
+
+```bash
+python3 run.py --models hf_internlm2_7b --datasets deprecated_humaneval_gen_d2537e --debug
+python3 run.py --models hf_internlm2_chat_7b --datasets humaneval_gen_8e312c --debug
+```
+
+## Base Models
+
+|          model           |   pass@1 |
+|:------------------------:|---------:|
+|    llama-7b-turbomind    |    12.80 |
+|   llama-13b-turbomind    |    15.24 |
+|   llama-30b-turbomind    |     9.15 |
+|   llama-65b-turbomind    |     7.32 |
+|   llama-2-7b-turbomind   |    14.02 |
+|  llama-2-13b-turbomind   |    15.24 |
+|  llama-2-70b-turbomind   |    15.24 |
+|   llama-3-8b-turbomind   |    28.05 |
+|  llama-3-70b-turbomind   |    28.05 |
+| internlm2-1.8b-turbomind |    30.49 |
+|  internlm2-7b-turbomind  |    48.17 |
+| internlm2-20b-turbomind  |    51.83 |
+|   qwen-1.8b-turbomind    |    16.46 |
+|    qwen-7b-turbomind     |    23.78 |
+|    qwen-14b-turbomind    |    23.78 |
+|    qwen-72b-turbomind    |    66.46 |
+|     qwen1.5-0.5b-hf      |     8.54 |
+|     qwen1.5-1.8b-hf      |    23.17 |
+|      qwen1.5-4b-hf       |    41.46 |
+|      qwen1.5-7b-hf       |    53.05 |
+|      qwen1.5-14b-hf      |    57.32 |
+|      qwen1.5-32b-hf      |    70.12 |
+|      qwen1.5-72b-hf      |    65.85 |
+|   qwen1.5-moe-a2-7b-hf   |    45.73 |
+|    mistral-7b-v0.1-hf    |    14.02 |
+|    mistral-7b-v0.2-hf    |     9.15 |
+|   mixtral-8x7b-v0.1-hf   |    24.39 |
+|  mixtral-8x22b-v0.1-hf   |    16.46 |
+|         yi-6b-hf         |    14.63 |
+|        yi-34b-hf         |    17.07 |
+|   deepseek-7b-base-hf    |    18.29 |
+|   deepseek-67b-base-hf   |    23.17 |
+
+## Chat Models
+
+|             model             |   pass@1 |
+|:-----------------------------:|---------:|
+|     qwen1.5-0.5b-chat-hf      |     9.15 |
+|     qwen1.5-1.8b-chat-hf      |    15.85 |
+|      qwen1.5-4b-chat-hf       |    30.49 |
+|      qwen1.5-7b-chat-hf       |    40.85 |
+|      qwen1.5-14b-chat-hf      |    50.00 |
+|      qwen1.5-32b-chat-hf      |    57.93 |
+|      qwen1.5-72b-chat-hf      |    60.37 |
+|     qwen1.5-110b-chat-hf      |    65.24 |
+|    internlm2-chat-1.8b-hf     |    33.54 |
+|  internlm2-chat-1.8b-sft-hf   |    34.15 |
+|     internlm2-chat-7b-hf      |    56.71 |
+|   internlm2-chat-7b-sft-hf    |    61.59 |
+|     internlm2-chat-20b-hf     |    67.68 |
+|   internlm2-chat-20b-sft-hf   |    67.68 |
+|    llama-3-8b-instruct-hf     |    55.49 |
+|    llama-3-70b-instruct-hf    |    70.73 |
+| llama-3-8b-instruct-lmdeploy  |    57.93 |
+| llama-3-70b-instruct-lmdeploy |    70.73 |
+|  mistral-7b-instruct-v0.1-hf  |    32.32 |
+|  mistral-7b-instruct-v0.2-hf  |    29.27 |
+| mixtral-8x7b-instruct-v0.1-hf |    34.15 |

opencompass/configs/datasets/xhumaneval/deprecated_humaneval_gen_4a6eef.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCreate a Python script for this problem:\n{prompt}\n\n### Response:\n'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/deprecated_humaneval_gen_6d1cc2.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nComplete the following python function.:\n{prompt}\n\n### Response:\n'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/deprecated_humaneval_gen_a82cae.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/deprecated_humaneval_gen_d2537e.py

@@ -0,0 +1,33 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template='Complete the following python code:\n{prompt}',
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/deprecated_humaneval_gen_fd5822.py

@@ -0,0 +1,31 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template='{prompt}'),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/deprecated_humaneval_gen_ff7054.py

@@ -0,0 +1,41 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            begin=[
+                dict(
+                    role='SYSTEM',
+                    fallback_role='HUMAN',
+                    prompt='Complete the following python code:'),
+            ],
+            round=[
+                dict(role='HUMAN', prompt='{prompt}'),
+            ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/README.md

@@ -0,0 +1,69 @@
+# HumanEval
+
+```bash
+python3 run.py --models hf_internlm2_7b --datasets deprecated_humaneval_gen_d2537e --debug
+python3 run.py --models hf_internlm2_chat_7b --datasets humaneval_gen_8e312c --debug
+```
+
+## Base Models
+
+|          model           |   pass@1 |
+|:------------------------:|---------:|
+|    llama-7b-turbomind    |    12.80 |
+|   llama-13b-turbomind    |    15.24 |
+|   llama-30b-turbomind    |     9.15 |
+|   llama-65b-turbomind    |     7.32 |
+|   llama-2-7b-turbomind   |    14.02 |
+|  llama-2-13b-turbomind   |    15.24 |
+|  llama-2-70b-turbomind   |    15.24 |
+|   llama-3-8b-turbomind   |    28.05 |
+|  llama-3-70b-turbomind   |    28.05 |
+| internlm2-1.8b-turbomind |    30.49 |
+|  internlm2-7b-turbomind  |    48.17 |
+| internlm2-20b-turbomind  |    51.83 |
+|   qwen-1.8b-turbomind    |    16.46 |
+|    qwen-7b-turbomind     |    23.78 |
+|    qwen-14b-turbomind    |    23.78 |
+|    qwen-72b-turbomind    |    66.46 |
+|     qwen1.5-0.5b-hf      |     8.54 |
+|     qwen1.5-1.8b-hf      |    23.17 |
+|      qwen1.5-4b-hf       |    41.46 |
+|      qwen1.5-7b-hf       |    53.05 |
+|      qwen1.5-14b-hf      |    57.32 |
+|      qwen1.5-32b-hf      |    70.12 |
+|      qwen1.5-72b-hf      |    65.85 |
+|   qwen1.5-moe-a2-7b-hf   |    45.73 |
+|    mistral-7b-v0.1-hf    |    14.02 |
+|    mistral-7b-v0.2-hf    |     9.15 |
+|   mixtral-8x7b-v0.1-hf   |    24.39 |
+|  mixtral-8x22b-v0.1-hf   |    16.46 |
+|         yi-6b-hf         |    14.63 |
+|        yi-34b-hf         |    17.07 |
+|   deepseek-7b-base-hf    |    18.29 |
+|   deepseek-67b-base-hf   |    23.17 |
+
+## Chat Models
+
+|             model             |   pass@1 |
+|:-----------------------------:|---------:|
+|     qwen1.5-0.5b-chat-hf      |     9.15 |
+|     qwen1.5-1.8b-chat-hf      |    15.85 |
+|      qwen1.5-4b-chat-hf       |    30.49 |
+|      qwen1.5-7b-chat-hf       |    40.85 |
+|      qwen1.5-14b-chat-hf      |    50.00 |
+|      qwen1.5-32b-chat-hf      |    57.93 |
+|      qwen1.5-72b-chat-hf      |    60.37 |
+|     qwen1.5-110b-chat-hf      |    65.24 |
+|    internlm2-chat-1.8b-hf     |    33.54 |
+|  internlm2-chat-1.8b-sft-hf   |    34.15 |
+|     internlm2-chat-7b-hf      |    56.71 |
+|   internlm2-chat-7b-sft-hf    |    61.59 |
+|     internlm2-chat-20b-hf     |    67.68 |
+|   internlm2-chat-20b-sft-hf   |    67.68 |
+|    llama-3-8b-instruct-hf     |    55.49 |
+|    llama-3-70b-instruct-hf    |    70.73 |
+| llama-3-8b-instruct-lmdeploy  |    57.93 |
+| llama-3-70b-instruct-lmdeploy |    70.73 |
+|  mistral-7b-instruct-v0.1-hf  |    32.32 |
+|  mistral-7b-instruct-v0.2-hf  |    29.27 |
+| mixtral-8x7b-instruct-v0.1-hf |    34.15 |

opencompass/configs/datasets/xhumaneval/humaneval/deprecated_humaneval_gen_4a6eef.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCreate a Python script for this problem:\n{prompt}\n\n### Response:\n'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/deprecated_humaneval_gen_6d1cc2.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nComplete the following python function.:\n{prompt}\n\n### Response:\n'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/deprecated_humaneval_gen_a82cae.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/deprecated_humaneval_gen_d2537e.py

@@ -0,0 +1,33 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template='Complete the following python code:\n{prompt}',
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/deprecated_humaneval_gen_fd5822.py

@@ -0,0 +1,31 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template='{prompt}'),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/deprecated_humaneval_gen_ff7054.py

@@ -0,0 +1,41 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            begin=[
+                dict(
+                    role='SYSTEM',
+                    fallback_role='HUMAN',
+                    prompt='Complete the following python code:'),
+            ],
+            round=[
+                dict(role='HUMAN', prompt='{prompt}'),
+            ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/humaneval_gen.py

@@ -0,0 +1,4 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .humaneval_gen_8e312c import humaneval_datasets  # noqa: F401, F403

opencompass/configs/datasets/xhumaneval/humaneval/humaneval_gen_66a7f4.py

@@ -0,0 +1,35 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(input_columns=['prompt'], output_column='task_id', train_split='test')
+
+HUMANEVAL_TEMPLATE = dict(
+    round=[
+        dict(role='HUMAN', prompt='You are an intelligent programming assistant to produce Python algorithmic solutions.\nCan you complete the following Python function?\n```python\n{prompt}\n```'),
+    ]
+)
+
+humaneval_infer_cfg = dict(
+    prompt_template=dict(type=PromptTemplate, template=HUMANEVAL_TEMPLATE),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024),
+)
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    k=[1, 10, 100],
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg,
+    )
+]

opencompass/configs/datasets/xhumaneval/humaneval/humaneval_gen_8e312c.py

@@ -0,0 +1,37 @@
+# THIS SHALL ALSO BE DEPRECATED
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Complete the following python code:\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/humaneval_openai_sample_evals_gen_159614.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Read the following function signature and docstring, and fully implement the function described. Your response should only contain the code for this function.\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/humaneval_passk_gen_8e312c.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Complete the following python code:\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval_passk',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/humaneval_repeat10_gen_8e312c.py

@@ -0,0 +1,37 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Complete the following python code:\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval_repeat10',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        num_repeats=10,
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/internal_humaneval_gen_ce6b06.py

@@ -0,0 +1,33 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_internal_v2_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template='# Complete the following python code:\n{prompt}',
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_internal_v2_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval/internal_humaneval_gen_d2537e.py

@@ -0,0 +1,33 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_internal_v1_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template='Complete the following python code:\n{prompt}',
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_internal_v1_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval_gen_66a7f4.py

@@ -0,0 +1,35 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(input_columns=['prompt'], output_column='task_id', train_split='test')
+
+HUMANEVAL_TEMPLATE = dict(
+    round=[
+        dict(role='HUMAN', prompt='You are an intelligent programming assistant to produce Python algorithmic solutions.\nCan you complete the following Python function?\n```python\n{prompt}\n```'),
+    ]
+)
+
+humaneval_infer_cfg = dict(
+    prompt_template=dict(type=PromptTemplate, template=HUMANEVAL_TEMPLATE),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024),
+)
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    k=[1, 10, 100],
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg,
+    )
+]

opencompass/configs/datasets/xhumaneval/humaneval_gen_8e312c.py

@@ -0,0 +1,49 @@
+# THIS SHALL ALSO BE DEPRECATED
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import xHumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2, HumanEvalPlusEvaluator
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Complete the following python code:\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+# humaneval_datasets = [
+#     dict(
+#         abbr='openai_humaneval',
+#         type=xHumanevalDataset,
+#         path='opencompass/humaneval',
+#         reader_cfg=humaneval_reader_cfg,
+#         infer_cfg=humaneval_infer_cfg,
+#         eval_cfg=humaneval_eval_cfg)
+# ]
+LANGS = ['ar']
+humaneval_datasets = []
+for lang in LANGS:
+    humaneval_datasets.append(
+        dict(
+            abbr=f'humaneval_{lang}',
+            type=xHumanevalDataset,
+            path=f'data/xhumaneval_plus/humaneval_plus_gpt4o_{lang}.jsonl',
+            reader_cfg=humaneval_reader_cfg,
+            infer_cfg=humaneval_infer_cfg,
+            eval_cfg=humaneval_eval_cfg)
+    )

opencompass/configs/datasets/xhumaneval/humaneval_openai_sample_evals_gen_159614.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Read the following function signature and docstring, and fully implement the function described. Your response should only contain the code for this function.\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval_passk_gen_8e312c.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Complete the following python code:\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval_passk',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/humaneval_repeat10_gen_8e312c.py

@@ -0,0 +1,37 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Complete the following python code:\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval_repeat10',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        num_repeats=10,
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/internal_humaneval_gen_ce6b06.py

@@ -0,0 +1,33 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_internal_v2_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template='# Complete the following python code:\n{prompt}',
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_internal_v2_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/internal_humaneval_gen_d2537e.py

@@ -0,0 +1,33 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_internal_v1_postprocess
+
+humaneval_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template='Complete the following python code:\n{prompt}',
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_internal_v1_postprocess),
+)
+
+humaneval_datasets = [
+    dict(
+        abbr='openai_humaneval',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_reader_cfg,
+        infer_cfg=humaneval_infer_cfg,
+        eval_cfg=humaneval_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval/xhumaneval_gen.py

@@ -0,0 +1,4 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .humaneval_gen_8e312c import humaneval_datasets  # noqa: F401, F403

opencompass/configs/datasets/xhumaneval_plus/humaneval_plus_gen.py

@@ -0,0 +1,4 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .humaneval_plus_gen_8e312c import humaneval_plus_datasets  # noqa: F401, F403

opencompass/configs/datasets/xhumaneval_plus/humaneval_plus_gen_66a7f4.py

@@ -0,0 +1,35 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalPlusEvaluator, humaneval_postprocess_v2
+
+humaneval_plus_reader_cfg = dict(input_columns=['prompt'], output_column='task_id', train_split='test')
+
+HUMANEVAL_TEMPLATE = dict(
+    round=[
+        dict(role='HUMAN', prompt='You are an intelligent programming assistant to produce Python algorithmic solutions.\nCan you complete the following Python function?\n```python\n{prompt}\n```'),
+    ]
+)
+
+humaneval_plus_infer_cfg = dict(
+    prompt_template=dict(type=PromptTemplate, template=HUMANEVAL_TEMPLATE),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024),
+)
+
+humaneval_plus_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalPlusEvaluator),
+    k=[1, 10, 100],
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_plus_datasets = [
+    dict(
+        abbr='humaneval_plus',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_plus_reader_cfg,
+        infer_cfg=humaneval_plus_infer_cfg,
+        eval_cfg=humaneval_plus_eval_cfg,
+    )
+]

opencompass/configs/datasets/xhumaneval_plus/humaneval_plus_gen_8e312c.py

@@ -0,0 +1,57 @@
+# THIS SHALL ALSO BE DEPRECATED
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import xHumanevalDataset, HumanEvalPlusEvaluator, humaneval_postprocess_v2
+
+humaneval_plus_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_plus_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Complete the following python code:\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=2048))
+
+
+
+
+# print('humaneval_plus_datasets ru')
+# humaneval_plus_datasets = [
+#     dict(
+#         abbr='xhumaneval_plus_ru',
+#         type=xHumanevalDataset,
+#         path='xhumaneval_plus',
+#         name='humaneval_plus_gpt4o_ru.jsonl',
+#         local_mode=True,
+#         reader_cfg=humaneval_plus_reader_cfg,
+#         infer_cfg=humaneval_plus_infer_cfg,
+#         eval_cfg=humaneval_plus_eval_cfg)
+# ]
+
+LANGS = ['de']
+
+humaneval_plus_datasets = []
+for lang in LANGS:
+    humaneval_plus_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalPlusEvaluator),
+        pred_role='BOT',
+        k=[1, 10],  # the parameter only for humaneval
+        lang=lang,
+        pred_postprocessor=dict(type=humaneval_postprocess_v2),
+    )
+    humaneval_plus_datasets.append(
+        dict(
+            abbr=f'xhumaneval_plus_{lang}',
+            type=xHumanevalDataset,
+            path=f'data/xhumaneval_plus/humaneval_{lang}.jsonl',
+            reader_cfg=humaneval_plus_reader_cfg,
+            infer_cfg=humaneval_plus_infer_cfg,
+            eval_cfg=humaneval_plus_eval_cfg)
+    )

opencompass/configs/datasets/xhumaneval_plus/humaneval_plus_openai_simple_evals_gen_159614.py

@@ -0,0 +1,38 @@
+# THIS SHALL ALSO BE DEPRECATED
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalPlusEvaluator, humaneval_postprocess_v2
+
+humaneval_plus_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_plus_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Read the following function signature and docstring, and fully implement the function described. Your response should only contain the code for this function.\n{prompt}'
+            ),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_plus_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalPlusEvaluator),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_plus_datasets = [
+    dict(
+        abbr='humaneval_plus',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_plus_reader_cfg,
+        infer_cfg=humaneval_plus_infer_cfg,
+        eval_cfg=humaneval_plus_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval_plus/humaneval_plus_passk_gen_8e312c.py

@@ -0,0 +1,36 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_plus_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_plus_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Complete the following python code:\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_plus_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator, metric='EvalPlus'),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_plus_datasets = [
+    dict(
+        abbr='humaneval_plus_passk',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        reader_cfg=humaneval_plus_reader_cfg,
+        infer_cfg=humaneval_plus_infer_cfg,
+        eval_cfg=humaneval_plus_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval_plus/humaneval_plus_repeat10_gen_8e312c.py

@@ -0,0 +1,37 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import HumanevalDataset, HumanEvalEvaluator, humaneval_postprocess_v2
+
+humaneval_plus_reader_cfg = dict(
+    input_columns=['prompt'], output_column='task_id', train_split='test')
+
+# TODO: allow empty output-column
+humaneval_plus_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(round=[
+            dict(
+                role='HUMAN',
+                prompt='Complete the following python code:\n{prompt}'),
+        ])),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=512))
+
+humaneval_plus_eval_cfg = dict(
+    evaluator=dict(type=HumanEvalEvaluator, metric='EvalPlus'),
+    pred_role='BOT',
+    k=[1, 10, 100],  # the parameter only for humaneval
+    pred_postprocessor=dict(type=humaneval_postprocess_v2),
+)
+
+humaneval_plus_datasets = [
+    dict(
+        abbr='humaneval_plus_repeat10',
+        type=HumanevalDataset,
+        path='opencompass/humaneval',
+        num_repeats=10,
+        reader_cfg=humaneval_plus_reader_cfg,
+        infer_cfg=humaneval_plus_infer_cfg,
+        eval_cfg=humaneval_plus_eval_cfg)
+]

opencompass/configs/datasets/xhumaneval_plus/xhumaneval_plus_gen.py

@@ -0,0 +1,4 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .humaneval_plus_gen_8e312c import humaneval_plus_datasets  # noqa: F401, F403

opencompass/configs/datasets/xlivecodebench/README.md

@@ -0,0 +1,42 @@
+# LiveCodeBench
+
+## Dataset
+
+LiveCodeBench provides holistic and contamination-free evaluation of coding capabilities of LLMs. Particularly, LiveCodeBench continuously collects new problems over time from contests across three competition platforms -- LeetCode, AtCoder, and CodeForces. Next, LiveCodeBench also focuses on a broader range of code-related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts four hundred high-quality coding problems that were published between May 2023 and March 2024.
+
+- Origin Project: https://livecodebench.github.io/leaderboard.html
+
+## Setting
+
+| Model Type | Code Generation | Test Output Prediction | Code Execution |
+|------------|--------|--------|--------|
+| Base Model    |  ❌      | ❌          |  ❌         |
+| Chat Model    | ✅        |    ✅      |  ✅       |
+
+
+
+## Baseline Performance
+
+
+| Model Type | Code Generation(pass@1) | Test Output Prediction(pass@1) | Code Execution(pass@1) |
+|------------|--------|--------|--------|
+|  Qwen2.5-7B-Instruct(HF)    |  39.25      | 48.64          | 41.96         |
+|  Meta-Llama-3.1-8B-Instruct(HF)  | 20.25       |   24.66     |  17.12      |
+
+
+## Citation
+
+```bibtex
+@article{jain2024livecodebench,
+  author    = {Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, Ion Stoica},
+  title     = {LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code},
+  year      = {2024},
+  journal   = {arXiv preprint},
+}
+@misc{2023opencompass,
+    title={OpenCompass: A Universal Evaluation Platform for Foundation Models},
+    author={OpenCompass Contributors},
+    howpublished = {\url{https://github.com/open-compass/opencompass}},
+    year={2023}
+}
+```

opencompass/configs/datasets/xlivecodebench/livecodebench_gen_6966bc.py

@@ -0,0 +1,166 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets.xlivecodebench import (
+    LCBCodeExecutionDataset,
+    LCBTestOutputPredictionDataset,
+    LCBCodeExecutionEvaluator,
+    LCBTestOutputEvaluator
+)
+from opencompass.datasets.xlivecodebench import xLCBCodeGenerationDataset, xLCBCodeGenerationEvaluator
+from opencompass.datasets.livecodebench import TestOutputPromptConstants
+
+
+lcb_code_generation_reader_cfg = dict(
+    input_columns=[
+        'question_content',
+        'format_prompt',
+    ],
+    # output_column='evaluation_sample',
+    output_column='question_id',
+)
+
+SYSTEM_MESSAGE_GENERIC = f'You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.'
+
+prompt_template = '### Question:\n{question_content}\n\n{format_prompt}' + \
+                    '### Answer: (use the provided format with backticks)\n\n'
+
+
+# Code Generation Tasks
+lcb_code_generation_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt=prompt_template
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_code_generation_eval_cfg = dict(
+    evaluator=dict(
+        type=xLCBCodeGenerationEvaluator,
+        path='data/code_generation_lite/lcb_gpt4o_ar.jsonl',
+        num_process_evaluate=4,
+        timeout=6,
+    ),
+    pred_role='BOT',
+)
+
+LCB_datasets = []
+LANGS = ['ar', 'bn', 'cs', 'de', 'es', 'fr', 'hu', 'ja', 'ko', 'ru', 'sr', 'sw', 'te', 'th', 'vi', 'zh']
+for LANG in LANGS:
+    LCBCodeGeneration_dataset = dict(
+        type=xLCBCodeGenerationDataset,
+        abbr=f'lcb_code_generation_{LANG}',
+        path=f'data/code_generation_lite/lcb_gpt4o_{LANG}.jsonl',
+        local_mode=True,
+        reader_cfg=lcb_code_generation_reader_cfg,
+        infer_cfg=lcb_code_generation_infer_cfg,
+        eval_cfg=lcb_code_generation_eval_cfg
+    )
+    # Code Execution Dataset
+    lcb_code_execution_reader_cfg = dict(
+        input_columns=[
+            'prompt',
+        ],
+        output_column='evaluation_sample',
+    )
+
+    lcb_code_execution_infer_cfg = dict(
+        prompt_template=dict(
+            type=PromptTemplate,
+            template=dict(
+                begin=[
+                    dict(
+                        role='SYSTEM',
+                        fallback_role='HUMAN',
+                        prompt='You are an expert at Python programming, code execution, test case generation, and fuzzing.'
+                    ),
+                ],
+                round=[
+                    dict(
+                        role='HUMAN',
+                        prompt='{prompt}'
+                    )
+                ]
+            )
+        ),
+        retriever=dict(type=ZeroRetriever),
+        inferencer=dict(type=GenInferencer, max_out_len=1024)
+    )
+
+    lcb_code_execution_eval_cfg = dict(
+        evaluator=dict(
+            type=LCBCodeExecutionEvaluator,
+        ),
+        pred_role='BOT',
+    )
+
+    LCBCodeExecution_dataset = dict(
+        type=LCBCodeExecutionDataset,
+        abbr=f'lcb_code_execution_{LANG}',
+        path='opencompass/execution-v2',
+        reader_cfg=lcb_code_execution_reader_cfg,
+        infer_cfg=lcb_code_execution_infer_cfg,
+        eval_cfg=lcb_code_execution_eval_cfg,
+    )
+
+    # TestOuputput Dataset
+    lcb_test_output_reader_cfg = dict(
+        input_columns=[
+            'prompt',
+        ],
+        output_column='evaluation_sample',
+    )
+
+    system_prompt = 'You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.'
+
+    lcb_test_output_infer_cfg = dict(
+        prompt_template=dict(
+            type=PromptTemplate,
+            template=dict(
+            # begin=[
+            #     dict(
+            #         role='SYSTEM',
+            #         prompt=system_prompt
+            #     ),
+            # ],
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt='{prompt}'
+                )
+                ]
+            )
+        ),
+        retriever=dict(type=ZeroRetriever),
+        inferencer=dict(type=GenInferencer, max_out_len=1024)
+    )
+
+    lcb_test_output_eval_cfg = dict(
+        evaluator=dict(
+            type=LCBTestOutputEvaluator,
+        ),
+        pred_role='BOT',
+    )
+
+    LCBTestOutput_dataset = dict(
+        type=LCBTestOutputPredictionDataset,
+        abbr=f'lcb_test_output_{LANG}',
+        path='opencompass/test_generation',
+        reader_cfg=lcb_test_output_reader_cfg,
+        infer_cfg=lcb_test_output_infer_cfg,
+        eval_cfg=lcb_test_output_eval_cfg,
+    )
+    LCB_datasets += [
+        LCBCodeGeneration_dataset,
+        LCBCodeExecution_dataset,
+        LCBTestOutput_dataset,
+    ]

opencompass/configs/datasets/xlivecodebench/livecodebench_gen_b2b0fd.py

@@ -0,0 +1,163 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import (
+    LCBCodeGenerationDataset,
+    LCBCodeExecutionDataset,
+    LCBTestOutputPredictionDataset,
+    LCBCodeGenerationEvaluator,
+    LCBCodeExecutionEvaluator,
+    LCBTestOutputEvaluator
+)
+from opencompass.datasets.livecodebench import TestOutputPromptConstants
+
+
+lcb_code_generation_reader_cfg = dict(
+    input_columns=[
+        'question_content',
+        'format_prompt',
+    ],
+    # output_column='evaluation_sample',
+    output_column='question_id',
+)
+
+SYSTEM_MESSAGE_GENERIC = f'You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.'
+
+prompt_template = '### Question:\n{question_content}\n\n{format_prompt}' + \
+                    '### Answer: (use the provided format with backticks)\n\n'
+
+
+# Code Generation Tasks
+lcb_code_generation_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt=prompt_template
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_code_generation_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBCodeGenerationEvaluator,
+        num_process_evaluate=4,
+        timeout=6,
+    ),
+    pred_role='BOT',
+)
+
+LCBCodeGeneration_dataset = dict(
+    type=LCBCodeGenerationDataset,
+    abbr='lcb_code_generation',
+    path='opencompass/code_generation_lite',
+    reader_cfg=lcb_code_generation_reader_cfg,
+    infer_cfg=lcb_code_generation_infer_cfg,
+    eval_cfg=lcb_code_generation_eval_cfg
+)
+
+# Code Execution Dataset
+lcb_code_execution_reader_cfg = dict(
+    input_columns=[
+        'prompt',
+    ],
+    output_column='evaluation_sample',
+)
+
+lcb_code_execution_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            begin=[
+                dict(
+                    role='SYSTEM',
+                    prompt='You are an expert at Python programming, code execution, test case generation, and fuzzing.'
+                ),
+            ],
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt='{prompt}'
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_code_execution_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBCodeExecutionEvaluator,
+    ),
+    pred_role='BOT',
+)
+
+LCBCodeExecution_dataset = dict(
+    type=LCBCodeExecutionDataset,
+    abbr='lcb_code_execution',
+    path='opencompass/execution-v2',
+    reader_cfg=lcb_code_execution_reader_cfg,
+    infer_cfg=lcb_code_execution_infer_cfg,
+    eval_cfg=lcb_code_execution_eval_cfg,
+)
+
+# TestOuputput Dataset
+lcb_test_output_reader_cfg = dict(
+    input_columns=[
+        'prompt',
+    ],
+    output_column='evaluation_sample',
+)
+
+system_prompt = 'You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.'
+
+lcb_test_output_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            # begin=[
+            #     dict(
+            #         role='SYSTEM',
+            #         prompt=system_prompt
+            #     ),
+            # ],
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt='{prompt}'
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_test_output_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBTestOutputEvaluator,
+    ),
+    pred_role='BOT',
+)
+
+LCBTestOutput_dataset = dict(
+    type=LCBTestOutputPredictionDataset,
+    abbr='lcb_test_output',
+    path='opencompass/test_generation',
+    reader_cfg=lcb_test_output_reader_cfg,
+    infer_cfg=lcb_test_output_infer_cfg,
+    eval_cfg=lcb_test_output_eval_cfg,
+)
+
+LCB_datasets = [
+    LCBCodeGeneration_dataset,
+    LCBCodeExecution_dataset,
+    LCBTestOutput_dataset,
+]

opencompass/configs/datasets/xlivecodebench/livecodebench_o1_gen_f0ed6c.py

@@ -0,0 +1,165 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import (
+    LCBCodeGenerationDataset,
+    LCBCodeExecutionDataset,
+    LCBTestOutputPredictionDataset,
+    LCBCodeGenerationEvaluator,
+    LCBCodeExecutionEvaluator,
+    LCBTestOutputEvaluator
+)
+from opencompass.datasets.livecodebench import TestOutputPromptConstants
+
+
+lcb_code_generation_reader_cfg = dict(
+    input_columns=[
+        'question_content',
+        'format_prompt',
+    ],
+    # output_column='evaluation_sample',
+    output_column='question_id',
+)
+
+SYSTEM_MESSAGE_GENERIC = f'You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.'
+
+prompt_template = '### Question:\n{question_content}\n\n{format_prompt}' + \
+                    '### Answer: (use the provided format with backticks)\n\n'
+
+
+# Code Generation Tasks
+lcb_code_generation_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt=prompt_template
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_code_generation_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBCodeGenerationEvaluator,
+        num_process_evaluate=4,
+        timeout=6,
+        release_version='release_v4',
+    ),
+    pred_role='BOT',
+)
+
+LCBCodeGeneration_dataset = dict(
+    type=LCBCodeGenerationDataset,
+    abbr='lcb_code_generation_v4',
+    path='opencompass/code_generation_lite',
+    reader_cfg=lcb_code_generation_reader_cfg,
+    infer_cfg=lcb_code_generation_infer_cfg,
+    eval_cfg=lcb_code_generation_eval_cfg,
+    release_version='release_v4',
+)
+
+# Code Execution Dataset
+lcb_code_execution_reader_cfg = dict(
+    input_columns=[
+        'prompt',
+    ],
+    output_column='evaluation_sample',
+)
+
+lcb_code_execution_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            begin=[
+                dict(
+                    role='SYSTEM',
+                    prompt='You are an expert at Python programming, code execution, test case generation, and fuzzing.'
+                ),
+            ],
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt='{prompt}'
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_code_execution_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBCodeExecutionEvaluator,
+    ),
+    pred_role='BOT',
+)
+
+LCBCodeExecution_dataset = dict(
+    type=LCBCodeExecutionDataset,
+    abbr='lcb_code_execution',
+    path='opencompass/execution-v2',
+    reader_cfg=lcb_code_execution_reader_cfg,
+    infer_cfg=lcb_code_execution_infer_cfg,
+    eval_cfg=lcb_code_execution_eval_cfg,
+)
+
+# TestOuputput Dataset
+lcb_test_output_reader_cfg = dict(
+    input_columns=[
+        'prompt',
+    ],
+    output_column='evaluation_sample',
+)
+
+system_prompt = 'You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.'
+
+lcb_test_output_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            # begin=[
+            #     dict(
+            #         role='SYSTEM',
+            #         prompt=system_prompt
+            #     ),
+            # ],
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt='{prompt}'
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_test_output_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBTestOutputEvaluator,
+    ),
+    pred_role='BOT',
+)
+
+LCBTestOutput_dataset = dict(
+    type=LCBTestOutputPredictionDataset,
+    abbr='lcb_test_output',
+    path='opencompass/test_generation',
+    reader_cfg=lcb_test_output_reader_cfg,
+    infer_cfg=lcb_test_output_infer_cfg,
+    eval_cfg=lcb_test_output_eval_cfg,
+)
+
+LCB_datasets = [
+    LCBCodeGeneration_dataset,
+    # LCBCodeExecution_dataset,
+    # LCBTestOutput_dataset,
+]

opencompass/configs/datasets/xlivecodebench/livecodebench_split_v4_o1_gen_f0ed6c.py

@@ -0,0 +1,165 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import (
+    LCBCodeGenerationDataset,
+    LCBCodeExecutionDataset,
+    LCBTestOutputPredictionDataset,
+    LCBCodeGenerationEvaluator,
+    LCBCodeExecutionEvaluator,
+    LCBTestOutputEvaluator
+)
+from opencompass.datasets.livecodebench import TestOutputPromptConstants
+
+
+lcb_code_generation_reader_cfg = dict(
+    input_columns=[
+        'question_content',
+        'format_prompt',
+    ],
+    # output_column='evaluation_sample',
+    output_column='question_id',
+)
+
+SYSTEM_MESSAGE_GENERIC = f'You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.'
+
+prompt_template = '### Question:\n{question_content}\n\n{format_prompt}' + \
+                    '### Answer: (use the provided format with backticks)\n\n'
+
+
+# Code Generation Tasks
+lcb_code_generation_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt=prompt_template
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_code_generation_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBCodeGenerationEvaluator,
+        num_process_evaluate=4,
+        timeout=6,
+        release_version='release_split_v4',
+    ),
+    pred_role='BOT',
+)
+
+LCBCodeGeneration_dataset = dict(
+    type=LCBCodeGenerationDataset,
+    abbr='lcb_code_generation_split_v4',
+    path='opencompass/code_generation_lite',
+    reader_cfg=lcb_code_generation_reader_cfg,
+    infer_cfg=lcb_code_generation_infer_cfg,
+    eval_cfg=lcb_code_generation_eval_cfg,
+    release_version='release_split_v4',
+)
+
+# Code Execution Dataset
+lcb_code_execution_reader_cfg = dict(
+    input_columns=[
+        'prompt',
+    ],
+    output_column='evaluation_sample',
+)
+
+lcb_code_execution_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            begin=[
+                dict(
+                    role='SYSTEM',
+                    prompt='You are an expert at Python programming, code execution, test case generation, and fuzzing.'
+                ),
+            ],
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt='{prompt}'
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_code_execution_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBCodeExecutionEvaluator,
+    ),
+    pred_role='BOT',
+)
+
+LCBCodeExecution_dataset = dict(
+    type=LCBCodeExecutionDataset,
+    abbr='lcb_code_execution',
+    path='opencompass/execution-v2',
+    reader_cfg=lcb_code_execution_reader_cfg,
+    infer_cfg=lcb_code_execution_infer_cfg,
+    eval_cfg=lcb_code_execution_eval_cfg,
+)
+
+# TestOuputput Dataset
+lcb_test_output_reader_cfg = dict(
+    input_columns=[
+        'prompt',
+    ],
+    output_column='evaluation_sample',
+)
+
+system_prompt = 'You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.'
+
+lcb_test_output_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            # begin=[
+            #     dict(
+            #         role='SYSTEM',
+            #         prompt=system_prompt
+            #     ),
+            # ],
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt='{prompt}'
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_test_output_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBTestOutputEvaluator,
+    ),
+    pred_role='BOT',
+)
+
+LCBTestOutput_dataset = dict(
+    type=LCBTestOutputPredictionDataset,
+    abbr='lcb_test_output',
+    path='opencompass/test_generation',
+    reader_cfg=lcb_test_output_reader_cfg,
+    infer_cfg=lcb_test_output_infer_cfg,
+    eval_cfg=lcb_test_output_eval_cfg,
+)
+
+LCB_datasets = [
+    LCBCodeGeneration_dataset,
+    # LCBCodeExecution_dataset,
+    # LCBTestOutput_dataset,
+]

opencompass/configs/datasets/xlivecodebench/livecodebench_v1_o1_gen_f0ed6c.py

@@ -0,0 +1,164 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets import (
+    LCBCodeGenerationDataset,
+    LCBCodeExecutionDataset,
+    LCBTestOutputPredictionDataset,
+    LCBCodeGenerationEvaluator,
+    LCBCodeExecutionEvaluator,
+    LCBTestOutputEvaluator
+)
+from opencompass.datasets.livecodebench import TestOutputPromptConstants
+
+
+lcb_code_generation_reader_cfg = dict(
+    input_columns=[
+        'question_content',
+        'format_prompt',
+    ],
+    # output_column='evaluation_sample',
+    output_column='question_id',
+)
+
+SYSTEM_MESSAGE_GENERIC = f'You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.'
+
+prompt_template = '### Question:\n{question_content}\n\n{format_prompt}' + \
+                    '### Answer: (use the provided format with backticks)\n\n'
+
+
+# Code Generation Tasks
+lcb_code_generation_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt=prompt_template
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_code_generation_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBCodeGenerationEvaluator,
+        num_process_evaluate=4,
+        timeout=6,
+    ),
+    pred_role='BOT',
+)
+
+LCBCodeGeneration_dataset = dict(
+    type=LCBCodeGenerationDataset,
+    abbr='lcb_code_generation_v1',
+    path='opencompass/code_generation_lite',
+    reader_cfg=lcb_code_generation_reader_cfg,
+    infer_cfg=lcb_code_generation_infer_cfg,
+    eval_cfg=lcb_code_generation_eval_cfg,
+    release_version='release_v1',
+)
+
+# Code Execution Dataset
+lcb_code_execution_reader_cfg = dict(
+    input_columns=[
+        'prompt',
+    ],
+    output_column='evaluation_sample',
+)
+
+lcb_code_execution_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            begin=[
+                dict(
+                    role='SYSTEM',
+                    prompt='You are an expert at Python programming, code execution, test case generation, and fuzzing.'
+                ),
+            ],
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt='{prompt}'
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_code_execution_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBCodeExecutionEvaluator,
+    ),
+    pred_role='BOT',
+)
+
+LCBCodeExecution_dataset = dict(
+    type=LCBCodeExecutionDataset,
+    abbr='lcb_code_execution',
+    path='opencompass/execution-v2',
+    reader_cfg=lcb_code_execution_reader_cfg,
+    infer_cfg=lcb_code_execution_infer_cfg,
+    eval_cfg=lcb_code_execution_eval_cfg,
+)
+
+# TestOuputput Dataset
+lcb_test_output_reader_cfg = dict(
+    input_columns=[
+        'prompt',
+    ],
+    output_column='evaluation_sample',
+)
+
+system_prompt = 'You are an expert Python programmer. You will be given a question (problem specification) and will generate a correct Python program that matches the specification and passes all tests. You will NOT return anything except for the program.'
+
+lcb_test_output_infer_cfg = dict(
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            # begin=[
+            #     dict(
+            #         role='SYSTEM',
+            #         prompt=system_prompt
+            #     ),
+            # ],
+            round=[
+                dict(
+                    role='HUMAN',
+                    prompt='{prompt}'
+                )
+            ]
+        )
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer, max_out_len=1024)
+)
+
+lcb_test_output_eval_cfg = dict(
+    evaluator=dict(
+        type=LCBTestOutputEvaluator,
+    ),
+    pred_role='BOT',
+)
+
+LCBTestOutput_dataset = dict(
+    type=LCBTestOutputPredictionDataset,
+    abbr='lcb_test_output',
+    path='opencompass/test_generation',
+    reader_cfg=lcb_test_output_reader_cfg,
+    infer_cfg=lcb_test_output_infer_cfg,
+    eval_cfg=lcb_test_output_eval_cfg,
+)
+
+LCB_datasets = [
+    LCBCodeGeneration_dataset,
+    # LCBCodeExecution_dataset,
+    # LCBTestOutput_dataset,
+]

opencompass/configs/datasets/xlivecodebench/xlivecodebench_gen.py

@@ -0,0 +1,4 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .livecodebench_gen_6966bc import LCB_datasets  # noqa: F401, F403

opencompass/configs/datasets/xmgsm/README.md

@@ -0,0 +1,67 @@
+# MGSM
+
+## Introduction
+
+The following introduction comes from the abstract in [Language models are multilingual chain-of-thought reasoners](https://arxiv.org/abs/2210.03057)
+
+```
+We introduce the Multilingual Grade School Math (MGSM) benchmark, by manually translating 250 grade-school math problems from the GSM8K dataset into ten typologically diverse languages.
+```
+
+## Official link
+
+### Paper
+
+[Language models are multilingual chain-of-thought reasoners](https://arxiv.org/abs/2210.03057)
+
+### Repository
+
+[MGSM](https://github.com/google-research/url-nlp)
+
+## Examples
+
+Input example I:
+
+```
+Solve this math problem. Give the reasoning steps before giving the final answer on the last line by itself in the format of "Answer:". Do not add anything other than the integer answer after "Answer:".
+
+Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
+```
+
+Output example I (from GPT-4):
+
+```
+Answer: 18
+```
+
+## Evaluation results
+
+```
+dataset         version    metric         mode      llama-3-8b-instruct-hf
+--------------  ---------  -------------  ------  ------------------------
+mgsm_bn         b65151     accuracy       gen                        14.4
+mgsm_de         2cc8ae     accuracy       gen                        60
+mgsm_en         5de71e     accuracy       gen                        76
+mgsm_es         d6b459     accuracy       gen                        61.6
+mgsm_fr         813e3c     accuracy       gen                        54.4
+mgsm_ja         04424f     accuracy       gen                        42.8
+mgsm_ru         400469     accuracy       gen                        62.8
+mgsm_sw         9e41ed     accuracy       gen                         0.8
+mgsm_te         346d97     accuracy       gen                         0
+mgsm_th         e70bee     accuracy       gen                        44
+mgsm_zh         d5cf30     accuracy       gen                        28.4
+mgsm_latin      -          naive_average  gen                        50.56
+mgsm_non_latin  -          naive_average  gen                        32.07
+mgsm            -          naive_average  gen                        40.47
+```
+
+## Reference
+
+```
+@article{shi2022language,
+  title={Language models are multilingual chain-of-thought reasoners},
+  author={Shi, Freda and Suzgun, Mirac and Freitag, Markus and Wang, Xuezhi and Srivats, Suraj and Vosoughi, Soroush and Chung, Hyung Won and Tay, Yi and Ruder, Sebastian and Zhou, Denny and others},
+  journal={arXiv preprint arXiv:2210.03057},
+  year={2022}
+}
+```

opencompass/configs/datasets/xmgsm/mgsm_gen.py

@@ -0,0 +1,4 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .mgsm_gen_d967bc import mgsm_datasets

opencompass/configs/datasets/xmgsm/mgsm_gen_d967bc.py

@@ -0,0 +1,62 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.openicl.icl_evaluator import JiebaRougeEvaluator
+from opencompass.datasets import xMGSMSDataset, xMGSM_Evaluator, mgsm_postprocess
+
+
+ALL_LANGUAGES = ['bn', 'de', 'en', 'es', 'fr', 'ja', 'ru', 'sw', 'te', 'th', 'zh', 'ar', 'sr', 'hu', 'cs', 'vi', 'ko']
+
+LANG_TO_INSTRUCTIONS = {
+    'en': """Solve this math problem. Give the reasoning steps before giving the final answer on the last line by itself in the format of "Answer:". Do not add anything other than the integer answer after "Answer:".\n\n{question}""",
+    'bn': """এই গণিতের সমস্যাটি সমাধান করুন। চূড়ান্ত উত্তর দেওয়ার আগে যুক্তিসম্পন্ন পদক্ষেপ প্রদান করুন। চূড়ান্ত উত্তরটি একক সংখ্যা হিসাবে "উত্তর:" এর পরে শেষ লাইনে দিন। "উত্তর:" এর পরে অন্য কিছু যুক্ত করবেন না।.\n\n{question}""",
+    'de': """Löse dieses Mathematikproblem. Gib die Schritte zur Begründung an, bevor du die endgültige Antwort in der letzten Zeile alleine im Format "Antwort:" gibst. Füge nichts anderes als die ganzzahlige Antwort nach "Antwort:" hinzu.\n\n{question}""",
+    'es': """Resuelve este problema matemático. Proporciona los pasos de razonamiento antes de dar la respuesta final en la última línea por sí misma en el formato de "Respuesta:". No añadas nada más que la respuesta entera después de "Respuesta:".\n\n{question}""",
+    'fr': """Résolvez ce problème de mathématiques. Donnez les étapes de raisonnement avant de fournir la réponse finale sur la dernière ligne elle-même dans le format de "Réponse:". N'ajoutez rien d'autre que la réponse entière après "Réponse:".\n\n{question}""",
+    'ja': """の数学の問題を解いてください。最終的な答えを出す前に、解答の推論過程を記述してください。そして最後の行には "答え:" の形式で答えを記述し、その後には整数の答え以外何も追加しないでください。\n\n{question}""",
+    'ru': """Решите эту математическую задачу. Объясните шаги рассуждения перед тем, как дать окончательный ответ в последней строке сам по себе в формате "Ответ:". Не добавляйте ничего, кроме целочисленного ответа после "Ответ:".\n\n{question}""",
+    'sw': """Suluhisha tatizo hili la hesabu. Toa hatua za mantiki kabla ya kutoa jibu la mwisho kwenye mstari wa mwisho peke yake katika muundo wa "Jibu:". Usiongeze chochote kingine isipokuwa jibu la integer baada ya "Jibu:".\n\n{question}""",
+    'te': """ఈ గణిత సమస్యను పరిష్కరించండి. చివరి సమాధానాన్ని ఇవ్వదానికి ముందు తర్కాత్మక అదుగులను ఇవ్వండి. చివరి పంక్తిలో మాత్రమే 'సమాధానం:' అనే ఆకారంలో చివరి సమాధానాద్ని ఇవ్వండి సమాధానం: తర్వాత పూర్ణాంక సమాధానానికి తప్పించి ఎదేనా చేర్చవద్దు.\n\n{question}""",
+    'th': """แก้ปัญหาคณิตศาสตร์นี้ ให้ให้ขั้นตอนการใช้เหตุผลก่อนที่จะให้คำตอบสุดท้ายในบรรทัดสุดท้ายโดยอยู่ในรูปแบบ "คำตอบ:" ไม่ควรเพิ่มอะไรนอกจากคำตอบที่เป็นจำนวนเต็มหลังจาก "คำตอบ:"\n\n{question}""",
+    'zh': """解决这个数学问题。在最后一行给出答案前，请提供推理步骤。最后一行应该以 "答案: " 的形式独立给出答案。在 "答案：" 后不要添加除整数答案之外的任何内容。\n\n{question}""",
+    'ar': """حل هذه المسألة الرياضية. قبل أن تعطي الإجابة النهائية، يرجى إعطاء الخطوات التفكيرية. يجب أن تكون الإجابة النهائية في السطر الأخير بالشكل "الإجابة:". لا تضيف أي شيء آخر غير الإجابة العددية النهائية بعد "الإجابة:".\n\n{question}""",
+    'sr': """Решење овог математичког проблема. Пре него што дате коначни одговор, пошаљите логички кораци. Коначни одговор мора бити на последњој линији у формату "Одговор:". Не додавајте ништа друго осим коначног целобројног одговора после "Одговор:".\n\n{question}""",
+    'hu': """Megoldás ezt a matematikai problémát. Mielőtt megadná a végső választ, adjon meg logikai lépéseket. A végső válasz a végső sorban kell legyen a "Válasz:" formátumban. Ne adjon hozzá semmit mást, csak a végső egész szám választ a "Válasz:" után.\n\n{question}""",
+    'cs': """Řešení tohoto matematického problému. Předtím, než dáte finální odpověď, přidejte logické kroky. Finální odpověď musí být na posledním řádku ve formátu "Odpověď:". Nezahrněte nic jiného než celočíselnou finální odpověď po "Odpověď:".\n\n{question}""",
+    'vi': """Giải bài toán toán học này. Trước khi bạn cung cấp câu trả lời cuối cùng, hãy cung cấp các bước suy luận. Câu trả lời cuối cùng phải ở trên dòng cuối cùng trong định dạng "Câu trả lời:". Không thêm bất kỳ thứ gì khác ngoài câu trả lời số nguyên sau "Câu trả lời:".\n\n{question}""",
+    'ko': """이 수학 문제를 풀어주세요. 마지막 줄에 최종 답을 제시하기 전에 논리적 단계를 제공해주세요. 마지막 줄에는 "답:" 형식으로 최종 답을 제시해주세요. "답:" 뒤에 정수 답만 추가해주세요.\n\n{question}""",
+}
+
+mgsm_datasets = []
+for lang in ALL_LANGUAGES:
+    mgsm_reader_cfg = dict(input_columns=['question'], output_column='answer')
+
+    mgsm_infer_cfg = dict(
+        prompt_template=dict(
+            type=PromptTemplate,
+            template=dict(
+                round=[
+                    dict(role='HUMAN', prompt=LANG_TO_INSTRUCTIONS[lang]),
+                ]
+            ),
+        ),
+        retriever=dict(type=ZeroRetriever),
+        inferencer=dict(type=GenInferencer, max_out_len=512),
+    )
+
+    mgsm_eval_cfg = dict(
+        evaluator=dict(type=xMGSM_Evaluator),
+        pred_role='BOT',
+        pred_postprocessor=dict(type=mgsm_postprocess, lang=lang),
+    )
+
+    mgsm_datasets.append(
+        dict(
+            type=xMGSMSDataset,
+            abbr=f'mgsm_{lang}',
+            path=f'data/mgsm/mgsm_test_{lang}_google.jsonl',
+            reader_cfg=mgsm_reader_cfg,
+            infer_cfg=mgsm_infer_cfg,
+            eval_cfg=mgsm_eval_cfg,
+        )
+    )

opencompass/configs/datasets/xmgsm/mgsm_gen_d967bc_8shot.py

@@ -0,0 +1,90 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.openicl.icl_evaluator import JiebaRougeEvaluator
+from opencompass.datasets import xMGSMSDataset, xMGSM_Evaluator, mgsm_postprocess
+import os
+
+ALL_LANGUAGES = ['bn', 'de', 'en', 'es', 'fr', 'ja', 'ru', 'sw', 'te', 'th', 'zh', 'ar', 'sr', 'hu', 'cs', 'vi', 'ko']
+
+LANG_TO_INSTRUCTIONS = {
+    'en': """Solve this math problem. Give the reasoning steps before giving the final answer on the last line by itself in the format of "Answer:". Do not add anything other than the integer answer after "Answer:".\n\n{question}""",
+    'bn': """এই গণিতের সমস্যাটি সমাধান করুন। চূড়ান্ত উত্তর দেওয়ার আগে যুক্তিসম্পন্ন পদক্ষেপ প্রদান করুন। চূড়ান্ত উত্তরটি একক সংখ্যা হিসাবে "উত্তর:" এর পরে শেষ লাইনে দিন। "উত্তর:" এর পরে অন্য কিছু যুক্ত করবেন না।.\n\n{question}""",
+    'de': """Löse dieses Mathematikproblem. Gib die Schritte zur Begründung an, bevor du die endgültige Antwort in der letzten Zeile alleine im Format "Antwort:" gibst. Füge nichts anderes als die ganzzahlige Antwort nach "Antwort:" hinzu.\n\n{question}""",
+    'es': """Resuelve este problema matemático. Proporciona los pasos de razonamiento antes de dar la respuesta final en la última línea por sí misma en el formato de "Respuesta:". No añadas nada más que la respuesta entera después de "Respuesta:".\n\n{question}""",
+    'fr': """Résolvez ce problème de mathématiques. Donnez les étapes de raisonnement avant de fournir la réponse finale sur la dernière ligne elle-même dans le format de "Réponse:". N'ajoutez rien d'autre que la réponse entière après "Réponse:".\n\n{question}""",
+    'ja': """の数学の問題を解いてください。最終的な答えを出す前に、解答の推論過程を記述してください。そして最後の行には "答え:" の形式で答えを記述し、その後には整数の答え以外何も追加しないでください。\n\n{question}""",
+    'ru': """Решите эту математическую задачу. Объясните шаги рассуждения перед тем, как дать окончательный ответ в последней строке сам по себе в формате "Ответ:". Не добавляйте ничего, кроме целочисленного ответа после "Ответ:".\n\n{question}""",
+    'sw': """Suluhisha tatizo hili la hesabu. Toa hatua za mantiki kabla ya kutoa jibu la mwisho kwenye mstari wa mwisho peke yake katika muundo wa "Jibu:". Usiongeze chochote kingine isipokuwa jibu la integer baada ya "Jibu:".\n\n{question}""",
+    'te': """ఈ గణిత సమస్యను పరిష్కరించండి. చివరి సమాధానాన్ని ఇవ్వదానికి ముందు తర్కాత్మక అదుగులను ఇవ్వండి. చివరి పంక్తిలో మాత్రమే 'సమాధానం:' అనే ఆకారంలో చివరి సమాధానాద్ని ఇవ్వండి సమాధానం: తర్వాత పూర్ణాంక సమాధానానికి తప్పించి ఎదేనా చేర్చవద్దు.\n\n{question}""",
+    'th': """แก้ปัญหาคณิตศาสตร์นี้ ให้ให้ขั้นตอนการใช้เหตุผลก่อนที่จะให้คำตอบสุดท้ายในบรรทัดสุดท้ายโดยอยู่ในรูปแบบ "คำตอบ:" ไม่ควรเพิ่มอะไรนอกจากคำตอบที่เป็นจำนวนเต็มหลังจาก "คำตอบ:"\n\n{question}""",
+    'zh': """解决这个数学问题。在最后一行给出答案前，请提供推理步骤。最后一行应该以 "答案: " 的形式独立给出答案。在 "答案：" 后不要添加除整数答案之外的任何内容。\n\n{question}""",
+    'ar': """حل هذه المسألة الرياضية. قبل أن تعطي الإجابة النهائية، يرجى إعطاء الخطوات التفكيرية. يجب أن تكون الإجابة النهائية في السطر الأخير بالشكل "الإجابة:". لا تضيف أي شيء آخر غير الإجابة العددية النهائية بعد "الإجابة:".\n\n{question}""",
+    'sr': """Решење овог математичког проблема. Пре него што дате коначни одговор, пошаљите логички кораци. Коначни одговор мора бити на последњој линији у формату "Одговор:". Не додавајте ништа друго осим коначног целобројног одговора после "Одговор:".\n\n{question}""",
+    'hu': """Megoldás ezt a matematikai problémát. Mielőtt megadná a végső választ, adjon meg logikai lépéseket. A végső válasz a végső sorban kell legyen a "Válasz:" formátumban. Ne adjon hozzá semmit mást, csak a végső egész szám választ a "Válasz:" után.\n\n{question}""",
+    'cs': """Řešení tohoto matematického problému. Předtím, než dáte finální odpověď, přidejte logické kroky. Finální odpověď musí být na posledním řádku ve formátu "Odpověď:". Nezahrněte nic jiného než celočíselnou finální odpověď po "Odpověď:".\n\n{question}""",
+    'vi': """Giải bài toán toán học này. Trước khi bạn cung cấp câu trả lời cuối cùng, hãy cung cấp các bước suy luận. Câu trả lời cuối cùng phải ở trên dòng cuối cùng trong định dạng "Câu trả lời:". Không thêm bất kỳ thứ gì khác ngoài câu trả lời số nguyên sau "Câu trả lời:".\n\n{question}""",
+    'ko': """이 수학 문제를 풀어주세요. 마지막 줄에 최종 답을 제시하기 전에 논리적 단계를 제공해주세요. 마지막 줄에는 "답:" 형식으로 최종 답을 제시해주세요. "답:" 뒤에 정수 답만 추가해주세요.\n\n{question}""",
+}
+import json
+def load_jsonl(path):
+    with open(path, 'r', encoding='utf-8') as f:
+        return [json.loads(line) for line in f]
+USER_HOME = os.path.expanduser("~")
+DEFAULT_DATA_FOLDER = os.path.join(USER_HOME, '.cache/opencompass/')
+mgsm_datasets = []
+for lang in ALL_LANGUAGES:
+    mgsm_reader_cfg = dict(input_columns=['question'], output_column='answer')
+    cache_dir = DEFAULT_DATA_FOLDER
+
+    # For absolute path customized by the users
+    local_path = os.path.join(cache_dir, f'data/mgsm/mgsm_train_{lang}_google.jsonl')
+    data = load_jsonl(local_path)
+    questions = [item['question'] for item in data]
+    answers = [item['answer'] for item in data]
+
+    mgsm_infer_cfg = dict(
+        prompt_template=dict(
+            type=PromptTemplate,
+            template=dict(
+                round=[
+                    dict(role='HUMAN', prompt=questions[0]),
+                    dict(role='BOT', prompt=answers[0]),
+                    dict(role='HUMAN', prompt=questions[1]),
+                    dict(role='BOT', prompt=answers[1]),
+                    dict(role='HUMAN', prompt=questions[2]),
+                    dict(role='BOT', prompt=answers[2]),
+                    dict(role='HUMAN', prompt=questions[3]),
+                    dict(role='BOT', prompt=answers[3]),
+                    dict(role='HUMAN', prompt=questions[4]),
+                    dict(role='BOT', prompt=answers[4]),
+                    dict(role='HUMAN', prompt=questions[5]),
+                    dict(role='BOT', prompt=answers[5]),
+                    dict(role='HUMAN', prompt=questions[6]),
+                    dict(role='BOT', prompt=answers[6]),
+                    dict(role='HUMAN', prompt=questions[7]),
+                    dict(role='BOT', prompt=answers[7]),
+                    dict(role='HUMAN', prompt=LANG_TO_INSTRUCTIONS[lang]),
+                ]
+            ),
+        ),
+        retriever=dict(type=ZeroRetriever),
+        inferencer=dict(type=GenInferencer, max_out_len=512),
+    )
+
+    mgsm_eval_cfg = dict(
+        evaluator=dict(type=xMGSM_Evaluator),
+        pred_role='BOT',
+        pred_postprocessor=dict(type=mgsm_postprocess, lang=lang),
+    )
+
+    mgsm_datasets.append(
+        dict(
+            type=xMGSMSDataset,
+            abbr=f'mgsm_{lang}_8shot',
+            path=f'data/mgsm/mgsm_test_{lang}_google.jsonl',
+            reader_cfg=mgsm_reader_cfg,
+            infer_cfg=mgsm_infer_cfg,
+            eval_cfg=mgsm_eval_cfg,
+        )
+    )

opencompass/configs/datasets/xmgsm/xmgsm_ge_8shot.py

@@ -0,0 +1,4 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .mgsm_gen_d967bc_8shot import mgsm_datasets

opencompass/configs/datasets/xmgsm/xmgsm_gen.py

@@ -0,0 +1,4 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .mgsm_gen_d967bc import mgsm_datasets

opencompass/configs/datasets/xruler/README.md

@@ -0,0 +1,14 @@
+# Ruler
+OpenCompass now supports the brand new long-context language model evaluation benchmark — [RULER](https://arxiv.org/pdf/2404.06654). RULER provides an evaluation of long-context including retrieval, multi-hop tracing, aggregation, and question answering through flexible configurations.
+
+OpenCompass have providied two types of evaluation demo for using different tokenizers.
+
+For using the same tokenizer (typicall GPT-4), you can follow the demo (configs/eval_ruler_fix_tokenizer.py) where most of the settings are already defined.
+
+
+For evaluation using each model's own tokenizer, you have to build the settings when you run the demo (we do not know which model you are trying to evaluate!) you can create a new evaluation script following the example (configs/eval_ruler.py) and change the context window sizes or add models according to your settings.
+
+```bash
+python run.py configs/eval_ruler_fix_tokenizer.py # For evaluation with GPT-4 tokenizer
+python run.py configs/eval_ruler.py # For evaluation with model's tokenizer
+```

opencompass/configs/datasets/xruler/ruler_128k_gen.py

@@ -0,0 +1,28 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .ruler_cwe_gen import cwe_datasets as cwe  # CWE
+    from .ruler_fwe_gen import fwe_datasets as fwe  # FWE
+    from .ruler_niah_gen import niah_datasets as niah  # Niah
+    from .ruler_qa_gen import qa_datasets as qa  # QA
+    from .ruler_vt_gen import vt_datasets as vt  # VT
+
+
+import_ds = sum((cwe, fwe, niah, qa, vt), [])
+
+# Evaluation config
+NUM_SAMPLES = 100  # Change to the number of samples you need
+# Change the context lengths to be tested
+max_seq_lens = [1024 * 128]
+abbr_suffixs = ['128k']
+
+ruler_datasets = []
+
+# Different seq length
+for max_seq_len, abbr_suffix in zip(max_seq_lens, abbr_suffixs):
+    for dataset in import_ds:
+        tmp_dataset = dataset.deepcopy()
+        tmp_dataset['abbr'] = tmp_dataset['abbr'] + '_' + abbr_suffix
+        tmp_dataset['num_samples'] = NUM_SAMPLES
+        tmp_dataset['max_seq_length'] = max_seq_len
+        ruler_datasets.append(tmp_dataset)

opencompass/configs/datasets/xruler/ruler_16k_gen.py

@@ -0,0 +1,29 @@
+
+from mmengine.config import read_base
+
+with read_base():
+    from .ruler_cwe_gen import cwe_datasets as cwe  # CWE
+    from .ruler_fwe_gen import fwe_datasets as fwe  # FWE
+    from .ruler_niah_gen import niah_datasets as niah  # Niah
+    from .ruler_qa_gen import qa_datasets as qa  # QA
+    from .ruler_vt_gen import vt_datasets as vt  # VT
+
+
+import_ds = sum((cwe, fwe, niah, qa, vt), [])
+
+# Evaluation config
+NUM_SAMPLES = 100 # Change to the number of samples you need
+# Change the context lengths to be tested
+max_seq_lens = [1024 * 16]
+abbr_suffixs = ['16k']
+
+ruler_datasets = []
+
+# Different seq length
+for max_seq_len, abbr_suffix in zip(max_seq_lens, abbr_suffixs):
+    for dataset in import_ds:
+        tmp_dataset = dataset.deepcopy()
+        tmp_dataset['abbr'] = tmp_dataset['abbr'] + '_' + abbr_suffix
+        tmp_dataset['num_samples'] = NUM_SAMPLES
+        tmp_dataset['max_seq_length'] = max_seq_len
+        ruler_datasets.append(tmp_dataset)

opencompass/configs/datasets/xruler/ruler_1m_gen.py

@@ -0,0 +1,29 @@
+
+from mmengine.config import read_base
+
+with read_base():
+    from .ruler_cwe_gen import cwe_datasets as cwe  # CWE
+    from .ruler_fwe_gen import fwe_datasets as fwe  # FWE
+    from .ruler_niah_gen import niah_datasets as niah  # Niah
+    from .ruler_qa_gen import qa_datasets as qa  # QA
+    from .ruler_vt_gen import vt_datasets as vt  # VT
+
+
+import_ds = sum((cwe, fwe, niah, qa, vt), [])
+
+# Evaluation config
+NUM_SAMPLES = 100 # Change to the number of samples you need
+# Change the context lengths to be tested
+max_seq_lens = [1024 * 1024]
+abbr_suffixs = ['1m']
+
+ruler_datasets = []
+
+# Different seq length
+for max_seq_len, abbr_suffix in zip(max_seq_lens, abbr_suffixs):
+    for dataset in import_ds:
+        tmp_dataset = dataset.deepcopy()
+        tmp_dataset['abbr'] = tmp_dataset['abbr'] + '_' + abbr_suffix
+        tmp_dataset['num_samples'] = NUM_SAMPLES
+        tmp_dataset['max_seq_length'] = max_seq_len
+        ruler_datasets.append(tmp_dataset)

opencompass/configs/datasets/xruler/ruler_32k_gen.py

@@ -0,0 +1,29 @@
+
+from mmengine.config import read_base
+
+with read_base():
+    from .ruler_cwe_gen import cwe_datasets as cwe  # CWE
+    from .ruler_fwe_gen import fwe_datasets as fwe  # FWE
+    from .ruler_niah_gen import niah_datasets as niah  # Niah
+    from .ruler_qa_gen import qa_datasets as qa  # QA
+    from .ruler_vt_gen import vt_datasets as vt  # VT
+
+
+import_ds = sum((cwe, fwe, niah, qa, vt), [])
+
+# Evaluation config
+NUM_SAMPLES = 100 # Change to the number of samples you need
+# Change the context lengths to be tested
+max_seq_lens = [1024 * 32]
+abbr_suffixs = ['32k']
+
+ruler_datasets = []
+
+# Different seq length
+for max_seq_len, abbr_suffix in zip(max_seq_lens, abbr_suffixs):
+    for dataset in import_ds:
+        tmp_dataset = dataset.deepcopy()
+        tmp_dataset['abbr'] = tmp_dataset['abbr'] + '_' + abbr_suffix
+        tmp_dataset['num_samples'] = NUM_SAMPLES
+        tmp_dataset['max_seq_length'] = max_seq_len
+        ruler_datasets.append(tmp_dataset)

opencompass/configs/datasets/xruler/ruler_4k_gen.py

@@ -0,0 +1,28 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .ruler_cwe_gen import cwe_datasets as cwe  # CWE
+    from .ruler_fwe_gen import fwe_datasets as fwe  # FWE
+    from .ruler_niah_gen import niah_datasets as niah  # Niah
+    from .ruler_qa_gen import qa_datasets as qa  # QA
+    from .ruler_vt_gen import vt_datasets as vt  # VT
+
+
+import_ds = sum((cwe, fwe, niah, qa, vt), [])
+
+# Evaluation config
+NUM_SAMPLES = 100  # Change to the number of samples you need
+# Change the context lengths to be tested
+max_seq_lens = [1024 * 4]
+abbr_suffixs = ['4k']
+
+ruler_datasets = []
+
+# Different seq length
+for max_seq_len, abbr_suffix in zip(max_seq_lens, abbr_suffixs):
+    for dataset in import_ds:
+        tmp_dataset = dataset.deepcopy()
+        tmp_dataset['abbr'] = tmp_dataset['abbr'] + '_' + abbr_suffix
+        tmp_dataset['num_samples'] = NUM_SAMPLES
+        tmp_dataset['max_seq_length'] = max_seq_len
+        ruler_datasets.append(tmp_dataset)

opencompass/configs/datasets/xruler/ruler_64k_gen.py

@@ -0,0 +1,28 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .ruler_cwe_gen import cwe_datasets as cwe  # CWE
+    from .ruler_fwe_gen import fwe_datasets as fwe  # FWE
+    from .ruler_niah_gen import niah_datasets as niah  # Niah
+    from .ruler_qa_gen import qa_datasets as qa  # QA
+    from .ruler_vt_gen import vt_datasets as vt  # VT
+
+
+import_ds = sum((cwe, fwe, niah, qa, vt), [])
+
+# Evaluation config
+NUM_SAMPLES = 100  # Change to the number of samples you need
+# Change the context lengths to be tested
+max_seq_lens = [1024 * 64]
+abbr_suffixs: list[str] = ['64k']
+
+ruler_datasets = []
+
+# Different seq length
+for max_seq_len, abbr_suffix in zip(max_seq_lens, abbr_suffixs):
+    for dataset in import_ds:
+        tmp_dataset = dataset.deepcopy()
+        tmp_dataset['abbr'] = tmp_dataset['abbr'] + '_' + abbr_suffix
+        tmp_dataset['num_samples'] = NUM_SAMPLES
+        tmp_dataset['max_seq_length'] = max_seq_len
+        ruler_datasets.append(tmp_dataset)

opencompass/configs/datasets/xruler/ruler_8k_gen.py

@@ -0,0 +1,29 @@
+
+from mmengine.config import read_base
+
+with read_base():
+    from .ruler_cwe_gen import cwe_datasets as cwe  # CWE
+    from .ruler_fwe_gen import fwe_datasets as fwe  # FWE
+    from .ruler_niah_gen import niah_datasets as niah  # Niah
+    from .ruler_qa_gen import qa_datasets as qa  # QA
+    from .ruler_vt_gen import vt_datasets as vt  # VT
+
+
+import_ds = sum((cwe, fwe, niah, qa, vt), [])
+
+# Evaluation config
+NUM_SAMPLES = 100 # Change to the number of samples you need
+# Change the context lengths to be tested
+max_seq_lens = [1024 * 8]
+abbr_suffixs = ['8k']
+
+ruler_datasets = []
+
+# Different seq length
+for max_seq_len, abbr_suffix in zip(max_seq_lens, abbr_suffixs):
+    for dataset in import_ds:
+        tmp_dataset = dataset.deepcopy()
+        tmp_dataset['abbr'] = tmp_dataset['abbr'] + '_' + abbr_suffix
+        tmp_dataset['num_samples'] = NUM_SAMPLES
+        tmp_dataset['max_seq_length'] = max_seq_len
+        ruler_datasets.append(tmp_dataset)

opencompass/configs/datasets/xruler/ruler_combined_gen.py

@@ -0,0 +1,12 @@
+from mmengine.config import read_base
+
+with read_base():
+    from .ruler_1m_gen import ruler_datasets as ruler_1m_ds
+    from .ruler_4k_gen import ruler_datasets as ruler_4k_ds
+    from .ruler_8k_gen import ruler_datasets as ruler_8k_ds
+    from .ruler_16k_gen import ruler_datasets as ruler_16k_ds
+    from .ruler_32k_gen import ruler_datasets as ruler_32k_ds
+    from .ruler_64k_gen import ruler_datasets as ruler_64k_ds
+    from .ruler_128k_gen import ruler_datasets as ruler_128k_ds
+
+ruler_combined_datasets = sum((v for k, v in locals().items() if k.endswith('_ds')), [])

opencompass/configs/datasets/xruler/ruler_cwe_gen.py

@@ -0,0 +1,34 @@
+from opencompass.datasets.ruler.ruler_cwe import RulerCweDataset
+from opencompass.datasets.ruler.ruler_cwe import RulerCweEvaluator
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+
+# CWE Dataset
+cwe_datasets = [
+    {
+        'abbr': 'ruler_cwe',
+        'type': RulerCweDataset,
+        'freq_cw': 30,
+        'freq_ucw': 3,
+        'num_cw': 10,
+        'tokens_to_generate': 120,
+        'reader_cfg': dict(input_columns=['prompt'], output_column='answer'),
+        'infer_cfg': dict(
+            prompt_template=dict(
+                type=PromptTemplate,
+                template=dict(
+                    round=[
+                        dict(role='HUMAN', prompt='{prompt}'),
+                        dict(role='BOT', prompt='{answer}\n'),
+                    ]
+                ),
+            ),
+            retriever=dict(type=ZeroRetriever),
+            inferencer=dict(type=GenInferencer),
+        ),
+        'eval_cfg': dict(
+            evaluator=dict(type=RulerCweEvaluator),
+        ),
+    }
+]

opencompass/configs/datasets/xruler/ruler_fwe_gen.py

@@ -0,0 +1,33 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets.ruler.ruler_fwe import RulerFweDataset
+from opencompass.datasets.ruler.ruler_fwe import RulerFweEvaluator
+
+# FWE Dataset
+fwe_datasets = [
+    {
+        'abbr': 'ruler_fwe',
+        'type': RulerFweDataset,
+        'tokens_to_generate': 50,
+        'alpha': 2.0,
+        'coded_wordlen': 6,
+        'reader_cfg': dict(input_columns=['prompt'], output_column='answer'),
+        'infer_cfg': dict(
+            prompt_template=dict(
+                type=PromptTemplate,
+                template=dict(
+                    round=[
+                        dict(role='HUMAN', prompt='{prompt}'),
+                        dict(role='BOT', prompt='{answer}\n'),
+                    ]
+                ),
+            ),
+            retriever=dict(type=ZeroRetriever),
+            inferencer=dict(type=GenInferencer),
+        ),
+        'eval_cfg': dict(
+            evaluator=dict(type=RulerFweEvaluator),
+        ),
+    }
+]

opencompass/configs/datasets/xruler/ruler_niah_gen.py

@@ -0,0 +1,122 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets.xruler.xruler_niah import xRulerNiahDataset
+from opencompass.datasets.xruler.xruler_niah import xRulerNiahEvaluator
+
+
+# Ruler Dataset settings
+niah_configurations = [
+    {
+        'abbr': 'single_1',
+        'type_haystack': 'repeat',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'single_2',
+        'type_haystack': 'essay',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'single_3',
+        'type_haystack': 'essay',
+        'type_needle_k': 'words',
+        'type_needle_v': 'uuids',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'multikey_1',
+        'type_haystack': 'essay',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 4,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'multikey_2',
+        'type_haystack': 'needle',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'multikey_3',
+        'type_haystack': 'needle',
+        'type_needle_k': 'uuids',
+        'type_needle_v': 'uuids',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'multivalue',
+        'type_haystack': 'essay',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 1,
+        'num_needle_v': 4,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'multiquery',
+        'type_haystack': 'essay',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 4,
+    },
+]
+
+niah_datasets = []
+LANGS = ['en', 'zh']
+# NIAH Dataset
+base_path = './data/xruler/un_test'
+for lang in LANGS:
+    file_path = f'un_test.{lang}.jsonl'
+    for index, config in enumerate(niah_configurations):
+        dataset_dict = {
+            'abbr': f'ruler_niah_{config["abbr"]}',
+            'type': xRulerNiahDataset,
+            'base_path': base_path,
+            'file_path': file_path,
+            # 'tokenizer_model': model_path,
+            'tokens_to_generate': 128,
+            'type_haystack': config['type_haystack'],
+            'type_needle_k': config['type_needle_k'],
+            'type_needle_v': config['type_needle_v'],
+            'num_needle_k': config['num_needle_k'],
+            'num_needle_v': config['num_needle_v'],
+            'num_needle_q': config['num_needle_q'],
+            'reader_cfg': dict(input_columns=['prompt'], output_column='answer'),
+            'infer_cfg': dict(
+                prompt_template=dict(
+                    type=PromptTemplate,
+                    template=dict(
+                        round=[
+                            dict(role='HUMAN', prompt='{prompt}'),
+                            dict(role='BOT', prompt='{answer}\n'),
+                        ]
+                    ),
+                ),
+                retriever=dict(type=ZeroRetriever),
+                inferencer=dict(type=GenInferencer),
+            ),
+            'eval_cfg': dict(
+                evaluator=dict(type=xRulerNiahEvaluator),
+            ),
+        }
+        niah_datasets.append(dataset_dict)

opencompass/configs/datasets/xruler/ruler_vt_gen.py

@@ -0,0 +1,32 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets.ruler.ruler_vt import RulerVtDataset
+from opencompass.datasets.ruler.ruler_vt import RulerVtEvaluator
+
+# VT Dataset
+vt_datasets = [
+    {
+        'abbr': 'ruler_vt',
+        'type': RulerVtDataset,
+        'num_chains': 1,
+        'num_hops': 4,
+        'reader_cfg': dict(input_columns=['prompt'], output_column='answer'),
+        'infer_cfg': dict(
+            prompt_template=dict(
+                type=PromptTemplate,
+                template=dict(
+                    round=[
+                        dict(role='HUMAN', prompt='{prompt}'),
+                        dict(role='BOT', prompt='{answer}\n'),
+                    ]
+                ),
+            ),
+            retriever=dict(type=ZeroRetriever),
+            inferencer=dict(type=GenInferencer),
+        ),
+        'eval_cfg': dict(
+            evaluator=dict(type=RulerVtEvaluator),
+        ),
+    }
+]

opencompass/configs/datasets/xruler/xruler_niah_gen.py

@@ -0,0 +1,122 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets.xruler.xruler_niah import xRulerNiahDataset
+from opencompass.datasets.xruler.xruler_niah import xRulerNiahEvaluator
+
+
+# Ruler Dataset settings
+niah_configurations = [
+    {
+        'abbr': 'single_1',
+        'type_haystack': 'repeat',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'single_2',
+        'type_haystack': 'essay',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'single_3',
+        'type_haystack': 'essay',
+        'type_needle_k': 'words',
+        'type_needle_v': 'uuids',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'multikey_1',
+        'type_haystack': 'essay',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 4,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'multikey_2',
+        'type_haystack': 'needle',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'multikey_3',
+        'type_haystack': 'needle',
+        'type_needle_k': 'uuids',
+        'type_needle_v': 'uuids',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'multivalue',
+        'type_haystack': 'essay',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 1,
+        'num_needle_v': 4,
+        'num_needle_q': 1,
+    },
+    {
+        'abbr': 'multiquery',
+        'type_haystack': 'essay',
+        'type_needle_k': 'words',
+        'type_needle_v': 'numbers',
+        'num_needle_k': 1,
+        'num_needle_v': 1,
+        'num_needle_q': 4,
+    },
+]
+
+niah_datasets = []
+LANGS = ['en', 'zh', 'ar', 'bn', 'cs', 'de', 'es', 'fr', 'hu']
+# NIAH Dataset
+base_path = './data/xruler/un_test'
+for lang in LANGS:
+    file_path = f'un_test.{lang}.json'
+    for index, config in enumerate(niah_configurations):
+        dataset_dict = {
+            'abbr': f'ruler_niah_{config["abbr"]}_{lang}',
+            'type': xRulerNiahDataset,
+            'base_path': base_path,
+            'file_path': file_path,
+            # 'tokenizer_model': model_path,
+            'tokens_to_generate': 128,
+            'type_haystack': config['type_haystack'],
+            'type_needle_k': config['type_needle_k'],
+            'type_needle_v': config['type_needle_v'],
+            'num_needle_k': config['num_needle_k'],
+            'num_needle_v': config['num_needle_v'],
+            'num_needle_q': config['num_needle_q'],
+            'reader_cfg': dict(input_columns=['prompt'], output_column='answer'),
+            'infer_cfg': dict(
+                prompt_template=dict(
+                    type=PromptTemplate,
+                    template=dict(
+                        round=[
+                            dict(role='HUMAN', prompt='{prompt}'),
+                            dict(role='BOT', prompt='{answer}\n'),
+                        ]
+                    ),
+                ),
+                retriever=dict(type=ZeroRetriever),
+                inferencer=dict(type=GenInferencer),
+            ),
+            'eval_cfg': dict(
+                evaluator=dict(type=xRulerNiahEvaluator),
+            ),
+        }
+        niah_datasets.append(dataset_dict)

opencompass/configs/datasets/xruler/xruler_qa_gen.py

@@ -0,0 +1,40 @@
+from opencompass.openicl.icl_prompt_template import PromptTemplate
+from opencompass.openicl.icl_retriever import ZeroRetriever
+from opencompass.openicl.icl_inferencer import GenInferencer
+from opencompass.datasets.xruler.ruler_qa import xRulerQaDataset
+from opencompass.datasets.xruler.ruler_qa import xRulerQaEvaluator
+LANGS = ['ar', 'bn', 'cs']
+qa_configurations = [
+    {'lang': lang, 'dataset': 'squad', 'path': f'./data/xruler/xquad/xquad_{lang}.json'} for lang in LANGS
+    # {'dataset': 'hotpotqa', 'path': './data/ruler/hotpotqa.json'},
+]
+
+qa_datasets = []
+for index, config in enumerate(qa_configurations):
+    dataset_dict = {
+        'abbr': f'xruler_qa_{config["dataset"]}_{config["lang"]}',
+        'dataset': config['dataset'],
+        'path': config['path'],
+        'lang': config['lang'],
+        'type': xRulerQaDataset,
+        'tokens_to_generate': 50,
+        'max_seq_length': 1024 * 8,
+        'reader_cfg': dict(input_columns=['prompt'], output_column='answer'),
+        'infer_cfg': dict(
+            prompt_template=dict(
+                type=PromptTemplate,
+                template=dict(
+                    round=[
+                        dict(role='HUMAN', prompt='{prompt}'),
+                        dict(role='BOT', prompt='{answer}\n'),
+                    ]
+                ),
+            ),
+            retriever=dict(type=ZeroRetriever),
+            inferencer=dict(type=GenInferencer),
+        ),
+        'eval_cfg': dict(
+            evaluator=dict(type=xRulerQaEvaluator),
+        ),
+    }
+    qa_datasets.append(dataset_dict)

opencompass/datasets/__init__.py

@@ -150,3 +150,7 @@ from .xcopa import *  # noqa: F401, F403
 from .xiezhi import XiezhiDataset, XiezhiRetriever  # noqa: F401, F403
 from .xlsum import *  # noqa: F401, F403
 from .xsum import *  # noqa: F401, F403
+from .xhumaneval import xHumanevalDataset  # noqa: F401, F403
+from .xIFEval.ifeval import xIFEvalDataset, xIFEvaluator  # noqa: F401, F403
+from .xlivecodebench import xLCBCodeGenerationDataset, xLCBCodeGenerationEvaluator
+from .xmgsm import *  # noqa: F401, F403

opencompass/datasets/evalplus/__init__.py

@@ -0,0 +1,4 @@
+try:
+    from evalplus._version import __version__, __version_tuple__
+except ImportError:
+    __version__ = "local-dev"

opencompass/datasets/evalplus/codegen.py

@@ -0,0 +1,270 @@
+import json
+import os
+from typing import Dict, List, Optional
+
+from .data import get_evalperf_data, get_human_eval_plus, get_mbpp_plus
+from .provider import DecoderBase, make_model
+from .sanitize import sanitize
+from .utils import progress
+
+def codegen(
+    target_path: str,
+    model: DecoderBase,
+    dataset: Dict,
+    greedy=False,
+    n_samples=1,
+    id_range=None,
+    resume=True,
+):
+    task2nexist = {}
+    if resume and target_path.endswith(".jsonl") and os.path.isfile(target_path):
+        with open(target_path, "r") as f:
+            for line in f:
+                if not line.strip():
+                    continue
+                task_id = json.loads(line)["task_id"]
+                task2nexist[task_id] = task2nexist.get(task_id, 0) + 1
+
+    if target_path.endswith(".jsonl"):
+        raw_target_path = target_path.replace(".jsonl", ".raw.jsonl")
+    else:
+        raw_target_path = target_path + ".raw"
+        os.makedirs(target_path, exist_ok=True)
+
+    print(f"Sanitized code outputs will be saved to {target_path}")
+    print(f"Raw outputs will be saved to {raw_target_path}")
+
+    backend_type: str = type(model).__name__
+    with progress(backend_type) as p:
+        for task_id, task in p.track(dataset.items()):
+            if id_range is not None:
+                id_num = int(task_id.split("/")[1])
+                low, high = id_range
+                if id_num < low or id_num >= high:
+                    p.console.print(f"Skipping {task_id} as it is not in {id_range}")
+                    continue
+
+            if not target_path.endswith(".jsonl"):
+                p_name = task_id.replace("/", "_")
+                os.makedirs(os.path.join(target_path, p_name), exist_ok=True)
+                task2nexist[task_id] = len(
+                    [
+                        f
+                        for f in os.listdir(os.path.join(target_path, p_name))
+                        if f.endswith(".py")
+                    ]
+                )
+
+            n_more_samples = n_samples
+            log = f"Codegen: {task_id} @ {model}"
+            if resume and task2nexist.get(task_id, 0) > 0:
+                log += f" (resuming from {task2nexist[task_id]})"
+                n_more_samples -= task2nexist[task_id]
+
+            p.console.print(log)
+
+            sidx = n_samples - n_more_samples
+            while sidx < n_samples:
+                prompt = task["prompt"].strip() + "\n"
+                outputs = model.codegen(
+                    prompt,
+                    do_sample=not greedy,
+                    num_samples=n_samples - sidx,
+                )
+                assert outputs, "No outputs from model!"
+                for impl in outputs:
+                    solution = prompt + impl if model.is_direct_completion() else impl
+                    sanitized_solution = sanitize(
+                        solution, entrypoint=task["entry_point"]
+                    )
+                    if target_path.endswith(".jsonl"):
+                        # Writing the sanitized version
+                        with open(target_path, "a") as f:
+                            f.write(
+                                json.dumps(
+                                    {"task_id": task_id, "solution": sanitized_solution}
+                                )
+                                + "\n"
+                            )
+
+                        # Writing the raw version
+                        with open(raw_target_path, "a") as f:
+                            f.write(
+                                json.dumps({"task_id": task_id, "solution": solution})
+                                + "\n"
+                            )
+                    else:
+                        # Writing the sanitized version
+                        with open(
+                            os.path.join(target_path, p_name, f"{sidx}.py"),
+                            "w",
+                            encoding="utf-8",
+                        ) as f:
+                            f.write(sanitized_solution)
+
+                        # Writing the raw version
+                        with open(
+                            os.path.join(raw_target_path, p_name, f"{sidx}.py"),
+                            "w",
+                            encoding="utf-8",
+                        ) as f:
+                            f.write(solution)
+                    sidx += 1
+
+
+def run_codegen(
+    model: str,
+    dataset: str,
+    root: str = "evalplus_results",
+    bs: Optional[int] = None,
+    n_samples: int = 1,
+    temperature: float = 0.0,
+    resume: bool = True,
+    greedy: bool = False,
+    id_range: List = None,
+    version: str = "default",
+    backend: str = "vllm",
+    force_base_prompt: bool = False,
+    base_url: str = None,
+    tp: int = 1,
+    evalperf_type: str = None,  # For EvalPerf
+    jsonl_fmt: bool = True,
+    attn_implementation: str = "eager",
+    device_map: Optional[str] = None,
+    trust_remote_code: bool = False,
+    enable_prefix_caching: bool = False,
+    enable_chunked_prefill: bool = False,
+    dtype: str = "bfloat16",
+    gptqmodel_backend: str = "auto",
+    lang: str = "en" # For GPTQModel
+):
+    assert dataset in ["humaneval", "mbpp", "evalperf"], f"Invalid dataset {dataset}"
+    assert evalperf_type is None or evalperf_type in [
+        "instruct",
+        "perf-instruct",
+        "perf-CoT",
+    ]
+
+    # Make dir for codes generated by each model
+    identifier = model.strip("./").replace("/", "--") + f"_{backend}_temp_{temperature}"
+    if evalperf_type:
+        identifier += f"-{evalperf_type}"
+
+    target_path = os.path.join(root, dataset, identifier) if lang == "en" else os.path.join(root, dataset, f"{lang}_{identifier}")
+    if jsonl_fmt:
+        target_path += ".jsonl"
+    else:
+        os.makedirs(target_path, exist_ok=True)
+
+    if dataset == "humaneval":
+        dataset_dict = get_human_eval_plus(version=version, lang=lang)
+    elif dataset == "mbpp":
+        dataset_dict = get_mbpp_plus(version=version)
+    elif dataset == "evalperf":
+        original_dataset = {**get_human_eval_plus(), **get_mbpp_plus()}
+        dataset_dict = {k: original_dataset[k] for k in get_evalperf_data()}
+        assert id_range is None, "id_range not supported for evalperf"
+    else:
+        raise ValueError(f"Invalid dataset {dataset}")
+
+    all_tasks_complete = False
+    if jsonl_fmt and os.path.isfile(target_path):
+        task_counts = {}
+        with open(target_path, "r") as f:
+            for line in f:
+                if not line.strip():
+                    continue
+                data = json.loads(line)
+                task_id = data["task_id"]
+                task_counts[task_id] = task_counts.get(task_id, 0) + 1
+
+            all_tasks_complete = all(
+                task_counts.get(task_id, 0) >= n_samples
+                for task_id in dataset_dict.keys()
+            )
+
+    if all_tasks_complete:
+        print("All samples are already cached. Skipping codegen.")
+        return target_path
+
+    if greedy and (temperature != 0 or bs != 1 or n_samples != 1):
+        temperature = 0.0
+        bs = 1
+        n_samples = 1
+        print("Greedy decoding ON (--greedy): setting bs=1, n_samples=1, temperature=0")
+
+    if id_range is not None:
+        assert len(id_range) == 2, "id_range must be a list of length 2"
+        assert id_range[0] < id_range[1], "id_range must be increasing"
+        id_range = tuple(id_range)
+
+    if bs is None:
+        bs = min(n_samples, 32)
+        print(f"Setting batch size to {bs}")
+
+    # Make project dir
+    os.makedirs(root, exist_ok=True)
+    # Make dataset dir
+    os.makedirs(os.path.join(root, dataset), exist_ok=True)
+
+    # Model instructions
+    instruction_prefix = "Please provide a self-contained Python script that solves the following problem in a markdown code block:"
+    response_prefix = "Below is a Python script with a self-contained function that solves the problem and passes corresponding tests:"
+
+    if evalperf_type == "perf-instruct":
+        instruction_prefix = "Please provide an efficient and self-contained Python script that solves the following problem in a markdown code block:"
+        response_prefix = "Below is a Python script with a self-contained function that efficiently solves the problem and passes corresponding tests:"
+    elif evalperf_type == "perf-CoT":
+        instruction_prefix = "Think step by step: please provide an efficient and self-contained Python script that solves the following problem in a markdown code block:"
+        response_prefix = "Below is a Python script with a self-contained function that efficiently solves the problem and passes corresponding tests:"
+    elif evalperf_type is not None and evalperf_type != "instruct":
+        raise ValueError(f"Invalid evalperf_type: {evalperf_type}")
+
+    # Model creation
+    model_runner = make_model(
+        model=model,
+        backend=backend,
+        batch_size=bs,
+        temperature=temperature,
+        force_base_prompt=force_base_prompt,
+        dataset=dataset,
+        base_url=base_url,
+        tp=tp,
+        instruction_prefix=instruction_prefix,
+        response_prefix=response_prefix,
+        device_map=device_map,
+        attn_implementation=attn_implementation,
+        trust_remote_code=trust_remote_code,
+        enable_prefix_caching=enable_prefix_caching,
+        enable_chunked_prefill=enable_chunked_prefill,
+        dtype=dtype,
+        gptqmodel_backend=gptqmodel_backend,
+    )
+
+    codegen(
+        target_path=target_path,
+        dataset=dataset_dict,
+        greedy=greedy,
+        model=model_runner,
+        n_samples=n_samples,
+        resume=resume,
+        id_range=id_range,
+    )
+
+    # force shutdown the model runner
+    del model_runner
+    import gc
+
+    gc.collect()
+
+    return target_path
+
+
+def main():
+    from fire import Fire
+
+    Fire(run_codegen)
+
+
+if __name__ == "__main__":
+    main()

opencompass/datasets/evalplus/config.py

@@ -0,0 +1,16 @@
+## EvalPlus
+DEFAULT_GT_TIME_LIMIT_FACTOR = 4.0
+DEFAULT_MIN_TIME_LIMIT = 1.0
+
+## EvalPerf
+
+### General
+PERF_PROFILE_ROUNDS = 1
+PERF_RAM_GB_PER_PROC = 12
+
+### Evaluation Phase
+PERF_EVAL_TIMEOUT_SECOND = 45
+
+### Curation Phase
+PERF_CURATE_TIMEOUT_SECOND = 20
+PREF_CURATE_MIN_INSTRUCTION = 10000

opencompass/datasets/evalplus/eval/__init__.py

@@ -0,0 +1,316 @@
+# The MIT License
+#
+# Copyright (c) OpenAI (https://openai.com)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+import itertools
+import multiprocessing
+import os
+import time
+from multiprocessing import Array, Value
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+import numpy as np
+import psutil
+
+from opencompass.datasets.evalplus.config import *
+from opencompass.datasets.evalplus.eval._special_oracle import (
+    MBPP_OUTPUT_NOT_NONE_TASKS,
+    MBPP_OUTPUT_SET_EQ_TASKS,
+    _digit_distance_nums,
+    _poly,
+    _surface_Area,
+)
+from opencompass.datasets.evalplus.eval.utils import (
+    create_tempdir,
+    reliability_guard,
+    swallow_io,
+    time_limit,
+)
+
+
+def compatible_eval_result(results: Dict) -> Dict:
+    # compatibility
+    for task_results in results["eval"].values():
+        # update the "files" field to "nfiles"
+        if "files" in task_results and "nfiles" not in task_results:
+            task_results["nfiles"] = len(task_results.pop("files"))
+    return results
+
+
+# unbiased estimator from https://github.com/openai/human-eval
+def estimate_pass_at_k(
+    num_samples: Union[int, List[int], np.ndarray],
+    num_correct: Union[List[int], np.ndarray],
+    k: int,
+) -> np.ndarray:
+    """
+    Estimates pass@k of each problem and returns them in an array.
+    """
+
+    def estimator(n: int, c: int, k: int) -> float:
+        """
+        Calculates 1 - comb(n - c, k) / comb(n, k).
+        """
+        if n - c < k:
+            return 1.0
+        return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
+
+    if isinstance(num_samples, int):
+        num_samples_it = itertools.repeat(num_samples, len(num_correct))
+    else:
+        assert len(num_samples) == len(num_correct)
+        num_samples_it = iter(num_samples)
+
+    return np.array(
+        [estimator(int(n), int(c), k) for n, c in zip(num_samples_it, num_correct)]
+    )
+
+
+PASS = "pass"
+FAIL = "fail"
+TIMEOUT = "timeout"
+
+_SUCCESS = 0
+_FAILED = 1
+_TIMEOUT = 2
+_UNKNOWN = 3
+
+_mapping = {_SUCCESS: PASS, _FAILED: FAIL, _TIMEOUT: TIMEOUT, _UNKNOWN: None}
+
+
+def query_maximum_memory_bytes() -> Optional[int]:
+    # Disable functionalities that can make destructive changes to the test.
+    # allow only 4GB memory usage
+    maximum_memory_bytes = os.getenv(
+        "EVALPLUS_MAX_MEMORY_BYTES", 4 * 1024 * 1024 * 1024
+    )
+    maximum_memory_bytes = min(int(maximum_memory_bytes), psutil.virtual_memory().total)
+    if maximum_memory_bytes == -1:
+        return None
+    return maximum_memory_bytes
+
+
+def is_floats(x) -> bool:
+    # check if it is float; List[float]; Tuple[float]
+    if isinstance(x, float):
+        return True
+    if isinstance(x, (list, tuple)) and x:
+        return all(isinstance(i, float) for i in x)
+    if isinstance(x, np.ndarray):
+        return x.dtype == np.float64 or x.dtype == np.float32
+    return False
+
+
+def unsafe_execute(
+    dataset: str,
+    entry_point: str,
+    code: str,
+    inputs,
+    expected: List,
+    time_limits,
+    atol,
+    fast_check,
+    stat,  # Value
+    details,  # Array
+    progress,  # Value
+):
+    with create_tempdir():
+        # These system calls are needed when cleaning up tempdir.
+        import os
+        import shutil
+
+        rmtree = shutil.rmtree
+        rmdir = os.rmdir
+        chdir = os.chdir
+        reliability_guard(maximum_memory_bytes=query_maximum_memory_bytes())
+        exec_globals = {}
+        try:
+            with swallow_io():
+                exec(code, exec_globals)
+                fn = exec_globals[entry_point]
+
+            for i, inp in enumerate(inputs):
+                try:
+                    with time_limit(time_limits[i]):
+                        with swallow_io():
+                            out = fn(*inp)
+
+                    exp = expected[i]
+                    exact_match = out == exp
+
+                    # ================================================ #
+                    # ============== special oracles ================= #
+                    if dataset == "mbpp":
+                        if "are_equivalent" == entry_point:  # Mbpp/164 special oracle
+                            exact_match = exact_match or True
+                        elif "sum_div" == entry_point:  # Mbpp/295 special oracle
+                            exact_match = exact_match or out == 0
+                        elif "surface_Area" == entry_point:  # Mbpp/581 special oracle
+                            exact_match = (
+                                exact_match or abs(out - _surface_Area(*inp)) <= atol
+                            )
+                        elif (
+                            "digit_distance_nums" == entry_point
+                        ):  # Mbpp/558 special oracle
+                            exact_match = exact_match or out == _digit_distance_nums(
+                                *inp
+                            )
+                        elif entry_point in MBPP_OUTPUT_SET_EQ_TASKS:
+                            exact_match = set(out) == set(exp)
+                        elif entry_point in MBPP_OUTPUT_NOT_NONE_TASKS:
+                            # exp is True  if not None
+                            #        False if None
+                            if isinstance(out, bool):
+                                exact_match = out == exp
+                            else:
+                                exact_match = exp == (out is not None)
+
+                    if dataset == "humaneval":
+                        if "find_zero" == entry_point:
+                            assert abs(_poly(*inp, out)) <= atol
+                            details[i] = True
+                            progress.value += 1
+                            continue
+                    # ============== special oracles ================= #
+                    # ================================================ #
+
+                    if atol == 0 and is_floats(exp):
+                        atol = 1e-6  # enforce atol for float comparison
+                    if not exact_match and atol != 0:
+                        # explicitly set rtol=1e-07
+                        # to match `np.testing.assert_allclose`'s default values
+                        assert type(out) == type(exp)
+                        if isinstance(exp, (list, tuple)):
+                            assert len(out) == len(exp)
+                        assert np.allclose(out, exp, rtol=1e-07, atol=atol)
+                    else:
+                        assert exact_match
+                except BaseException:
+                    details[i] = False
+                    progress.value += 1
+                    if fast_check:
+                        raise
+                    continue
+
+                details[i] = True
+                progress.value += 1
+
+            stat.value = _SUCCESS
+        except BaseException:
+            stat.value = _FAILED
+        # Needed for cleaning up.
+        shutil.rmtree = rmtree
+        os.rmdir = rmdir
+        os.chdir = chdir
+
+
+def untrusted_check(
+    dataset: str,
+    code: str,
+    inputs: List[Any],
+    entry_point: str,
+    expected,
+    atol,
+    ref_time: List[float],
+    fast_check: bool = False,
+    min_time_limit: float = DEFAULT_MIN_TIME_LIMIT,
+    gt_time_limit_factor: float = DEFAULT_GT_TIME_LIMIT_FACTOR,
+) -> Tuple[str, np.ndarray]:
+    time_limits = [max(min_time_limit, gt_time_limit_factor * t) for t in ref_time]
+    timeout = min(os.getenv("EVALPLUS_TIMEOUT_PER_TASK", 60), sum(time_limits)) + 1
+    if not fast_check:
+        timeout += 1  # extra time for data collection
+
+    # shared memory objects
+    progress = Value("i", 0)
+    stat = Value("i", _UNKNOWN)
+    details = Array("b", [False for _ in range(len(inputs))])
+
+    p = multiprocessing.Process(
+        target=unsafe_execute,
+        args=(
+            dataset,
+            entry_point,
+            code,
+            inputs,
+            expected,
+            time_limits,
+            atol,
+            fast_check,
+            # return values
+            stat,
+            details,
+            progress,
+        ),
+    )
+    p.start()
+    p.join(timeout=timeout + 1)
+    if p.is_alive():
+        p.terminate()
+        time.sleep(0.1)
+    if p.is_alive():
+        p.kill()
+        time.sleep(0.1)
+
+    stat = _mapping[stat.value]
+    details = details[: progress.value]
+
+    if not stat:
+        stat = TIMEOUT
+
+    if stat == PASS:
+        if len(details) != len(inputs) or not all(details):
+            stat = FAIL
+
+    return stat, details
+
+
+def evaluate_files(
+    dataset: str,
+    files: List[str],
+    inputs: List,
+    expected: List,
+    entry_point: str,
+    atol: float,
+    ref_time: List[float],
+    fast_check: bool = False,
+    min_time_limit: float = DEFAULT_MIN_TIME_LIMIT,
+    gt_time_limit_factor: float = DEFAULT_GT_TIME_LIMIT_FACTOR,
+) -> List[Tuple[str, List[bool]]]:
+    ret = []
+    # sort files by the id in name (i.e., "../n.py")
+    files = sorted(files, key=lambda x: int(x.split("/")[-1].split(".")[0]))
+    for file in files:
+        code = open(file, "r").read()
+        stat, det = untrusted_check(
+            dataset,
+            code,
+            inputs,
+            entry_point,
+            expected=expected,
+            atol=atol,
+            ref_time=ref_time,
+            fast_check=fast_check,
+            min_time_limit=min_time_limit,
+            gt_time_limit_factor=gt_time_limit_factor,
+        )
+        ret.append((stat, det.tolist()))
+    return ret

opencompass/datasets/evalplus/eval/_special_oracle.py

@@ -0,0 +1,55 @@
+"""Special oracle handlings for problems where direct differential testing is not applicable."""
+
+import math
+
+# For tasks whose output are not serializable, we only check the output is not None, which
+# is also consistent with the original dataset.
+MBPP_OUTPUT_NOT_NONE_TASKS = ["check_str", "text_match_three", "text_starta_endb"]
+
+# Tasks that needs to perform set comparison over two lists
+MBPP_OUTPUT_SET_EQ_TASKS = [
+    "similar_elements",  # Mbpp/2
+    "find_char_long",  # Mbpp/7
+    "common_in_nested_lists",  # Mbpp/111
+    "extract_singly",  # Mbpp/140
+    "larg_nnum",  # Mbpp/232
+    "intersection_array",  # Mbpp/249
+    "find_dissimilar",  # Mbpp/579
+    "Diff",  # Mbpp/769
+]
+
+
+# oracle for Mbpp/581
+def _surface_Area(base_edge, height):
+    """
+    Recognizes the "height" as the perpendicular distance from the base to the apex of the pyramid
+    """
+    slant_height = math.sqrt((base_edge / 2) ** 2 + height**2)
+    base_area = base_edge**2
+    lateral_area = 4 * (base_edge * slant_height) / 2
+    total_surface_area = base_area + lateral_area
+    return round(total_surface_area)
+
+
+# oracle for Mbpp/558
+def _digit_distance_nums(num1, num2):
+    """
+    Preprocesses the two numbers to have the same length by padding with zeros
+    """
+    str_num1, str_num2 = str(num1), str(num2)
+    max_length = max(len(str_num1), len(str_num2))
+    str_num1, str_num2 = str_num1.zfill(max_length), str_num2.zfill(max_length)
+    total_difference = 0
+    for digit1, digit2 in zip(str_num1, str_num2):
+        difference = abs(int(digit1) - int(digit2))
+        total_difference += difference
+    return total_difference
+
+
+# oracle for HumaneEval/032
+def _poly(xs: list, x: float):
+    """
+    Evaluates polynomial with coefficients xs at point x.
+    return xs[0] + xs[1] * x + xs[1] * x^2 + .... xs[n] * x^n
+    """
+    return sum([coeff * math.pow(x, i) for i, coeff in enumerate(xs)])

opencompass/datasets/evalplus/eval/utils.py

@@ -0,0 +1,187 @@
+# The MIT License
+#
+# Copyright (c) OpenAI (https://openai.com)
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+import contextlib
+import faulthandler
+import io
+import os
+import platform
+import signal
+import tempfile
+from typing import Optional
+
+
+@contextlib.contextmanager
+def swallow_io():
+    stream = WriteOnlyStringIO()
+    with contextlib.redirect_stdout(stream):
+        with contextlib.redirect_stderr(stream):
+            with redirect_stdin(stream):
+                yield
+
+
+@contextlib.contextmanager
+def time_limit(seconds: float):
+    def signal_handler(signum, frame):
+        raise TimeoutException("Timed out!")
+
+    signal.setitimer(signal.ITIMER_REAL, seconds)
+    signal.signal(signal.SIGALRM, signal_handler)
+    try:
+        yield
+    finally:
+        signal.setitimer(signal.ITIMER_REAL, 0)
+
+
+@contextlib.contextmanager
+def create_tempdir():
+    with tempfile.TemporaryDirectory() as dirname:
+        with chdir(dirname):
+            yield dirname
+
+
+@contextlib.contextmanager
+def chdir(root):
+    if root == ".":
+        yield
+        return
+    cwd = os.getcwd()
+    os.chdir(root)
+    try:
+        yield
+    except BaseException as exc:
+        raise exc
+    finally:
+        os.chdir(cwd)
+
+
+class TimeoutException(Exception):
+    pass
+
+
+class WriteOnlyStringIO(io.StringIO):
+    """StringIO that throws an exception when it's read from"""
+
+    def read(self, *args, **kwargs):
+        raise IOError
+
+    def readline(self, *args, **kwargs):
+        raise IOError
+
+    def readlines(self, *args, **kwargs):
+        raise IOError
+
+    def readable(self, *args, **kwargs):
+        """Returns True if the IO object can be read."""
+        return False
+
+
+class redirect_stdin(contextlib._RedirectStream):  # type: ignore
+    _stream = "stdin"
+
+
+def reliability_guard(maximum_memory_bytes: Optional[int] = None):
+    """
+    This disables various destructive functions and prevents the generated code
+    from interfering with the test (e.g. fork bomb, killing other processes,
+    removing filesystem files, etc.)
+
+    WARNING
+    This function is NOT a security sandbox. Untrusted code, including, model-
+    generated code, should not be blindly executed outside of one. See the
+    Codex paper for more information about OpenAI's code sandbox, and proceed
+    with caution.
+    """
+
+    if maximum_memory_bytes is not None:
+        import resource
+
+        resource.setrlimit(
+            resource.RLIMIT_AS, (maximum_memory_bytes, maximum_memory_bytes)
+        )
+        resource.setrlimit(
+            resource.RLIMIT_DATA, (maximum_memory_bytes, maximum_memory_bytes)
+        )
+        if not platform.uname().system == "Darwin":
+            resource.setrlimit(
+                resource.RLIMIT_STACK, (maximum_memory_bytes, maximum_memory_bytes)
+            )
+
+    faulthandler.disable()
+
+    import builtins
+
+    builtins.exit = None
+    builtins.quit = None
+
+    import os
+
+    os.environ["OMP_NUM_THREADS"] = "1"
+
+    os.kill = None
+    os.system = None
+    os.putenv = None
+    os.remove = None
+    os.removedirs = None
+    os.rmdir = None
+    os.fchdir = None
+    os.setuid = None
+    os.fork = None
+    os.forkpty = None
+    os.killpg = None
+    os.rename = None
+    os.renames = None
+    os.truncate = None
+    os.replace = None
+    os.unlink = None
+    os.fchmod = None
+    os.fchown = None
+    os.chmod = None
+    os.chown = None
+    os.chroot = None
+    os.fchdir = None
+    os.lchflags = None
+    os.lchmod = None
+    os.lchown = None
+    os.getcwd = None
+    os.chdir = None
+    builtins.open = None
+
+    import shutil
+
+    shutil.rmtree = None
+    shutil.move = None
+    shutil.chown = None
+
+    import subprocess
+
+    subprocess.Popen = None  # type: ignore
+
+    __builtins__["help"] = None
+
+    import sys
+
+    sys.modules["ipdb"] = None
+    sys.modules["joblib"] = None
+    sys.modules["resource"] = None
+    sys.modules["psutil"] = None
+    sys.modules["tkinter"] = None

opencompass/datasets/evalplus/evalperf.py

@@ -0,0 +1,557 @@
+"""Compute the Differential Performance Scores (DPS) and DPS_{norm} of given samples from a model.
+
+Check our COLM paper for more details: https://www.arxiv.org/abs/2408.06450
+
+^Updates from the COLM paper:
+* Condition to activate efficiency evaluation for a task:
+  * Paper: as long as you have at least one correct solution, and we select up to 10 correct solutions for efficiency sampling
+  * Here: you need to have at least `min_correct` correct solutions, and we evaluate the efficiency of all correct solutions
+  * Updating rationale: to make the evaluation more statistically robust
+
+@inproceedings{liu2024evaluating,
+  title = {Evaluating Language Models for Efficient Code Generation},
+  author = {Liu, Jiawei and Xie, Songrun and Wang, Junhao and Wei, Yuxiang and Ding, Yifeng and Zhang, Lingming},
+  booktitle = {First Conference on Language Modeling},
+  year = {2024},
+  url = {https://openreview.net/forum?id=IBCBMeAhmC},
+}
+"""
+
+import json
+import multiprocessing
+import os
+import socket
+import time
+from collections import defaultdict
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from contextlib import closing
+from datetime import datetime
+from statistics import mean
+from typing import Dict, List, Optional, Tuple
+
+import rich
+from rich.rule import Rule
+from rich.syntax import Syntax
+from rich.table import Table
+
+from opencompass.datasets.evalplus.codegen import run_codegen
+from opencompass.datasets.evalplus.config import *
+from opencompass.datasets.evalplus.config import PERF_EVAL_TIMEOUT_SECOND
+from opencompass.datasets.evalplus.data import (
+    get_evalperf_data,
+    get_human_eval_plus,
+    get_human_eval_plus_hash,
+    get_mbpp_plus,
+    get_mbpp_plus_hash,
+)
+from opencompass.datasets.evalplus.data.mbpp import mbpp_deserialize_inputs
+from opencompass.datasets.evalplus.data.utils import stream_jsonl
+from opencompass.datasets.evalplus.eval import PASS, untrusted_check
+from opencompass.datasets.evalplus.eval._special_oracle import MBPP_OUTPUT_NOT_NONE_TASKS
+from opencompass.datasets.evalplus.evaluate import get_groundtruth
+from opencompass.datasets.evalplus.perf.profile import (
+    are_profiles_broken,
+    default_parallelism,
+    profile,
+    simple_test_profiler,
+)
+from opencompass.datasets.evalplus.utils import progress
+
+def rule(msg: str):
+    rich.print(Rule(msg))
+
+
+def not_none(l: list) -> list:
+    return [x for x in l if x is not None]
+
+
+def get_free_port():
+    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
+        s.bind(("", 0))
+        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+        return s.getsockname()[1]
+
+
+def correctness_check(
+    solution: str, dataset: str, task: Dict, expected_output: List
+) -> Tuple:
+    assert isinstance(solution, str)
+    result = untrusted_check(
+        dataset,
+        solution,
+        task["base_input"] + list(task["plus_input"]),
+        task["entry_point"],
+        expected_output["base"] + expected_output["plus"],
+        task["atol"],
+        expected_output["base_time"] + expected_output["plus_time"],
+        fast_check=True,
+        min_time_limit=DEFAULT_MIN_TIME_LIMIT,
+        gt_time_limit_factor=DEFAULT_GT_TIME_LIMIT_FACTOR,
+    )
+    return result, solution
+
+
+def get_evalplus_data():
+    problems_he = get_human_eval_plus(noextreme=True)
+    dataset_hash = get_human_eval_plus_hash(noextreme=True)
+    expected_output_human = get_groundtruth(problems_he, dataset_hash, [])
+    problems_mbpp = get_mbpp_plus(noextreme=True)
+    dataset_hash = get_mbpp_plus_hash(noextreme=True)
+    expected_output_mbpp = get_groundtruth(
+        problems_mbpp,
+        dataset_hash,
+        MBPP_OUTPUT_NOT_NONE_TASKS,
+    )
+    problems = {**problems_he, **problems_mbpp}
+    expected_output = {**expected_output_human, **expected_output_mbpp}
+    return problems, expected_output
+
+
+def table_print(table_name: str, kv: Dict):
+    table = Table(
+        title=table_name,
+        show_header=True,
+        header_style="bold",
+    )
+    for col_name in kv:
+        table.add_column(col_name)
+
+    table.add_row(*[str(v) for v in kv.values()])
+    rich.print(table)
+
+
+def correctness_worker(task_id: str, samples: list, ctask: Dict, expected_output: Dict):
+    assert isinstance(
+        samples, list
+    ), f"{task_id}: samples is not a list but {type(samples)}"
+
+    results = []
+
+    for solution in samples:
+        result, solution = correctness_check(
+            solution, task_id.split("/")[0].lower(), ctask, expected_output
+        )
+        results.append(
+            {
+                "solution": solution,
+                "pass": result[0] == PASS,
+                "profiled": False,
+                "matching_cluster_idx": None,
+                "dps": None,
+                "dps_norm": None,
+            }
+        )
+
+    return task_id, results
+
+
+def perf_worker(
+    task_id: str,
+    ptask: Dict,  # EvalPerf data
+    ret_dict: Dict,
+    lazy_evaluation: bool,
+    max_profile: int,
+):
+    rich.print(f"{task_id}: Started")
+    start_time = time.time()
+
+    ######################### Profiling Setup #########################
+    n_reference = len(ptask["reference"])
+    entry_point = ptask["entry_point"]
+    pe_input = (
+        mbpp_deserialize_inputs(task_id, ptask["pe_input"])[0]
+        if task_id.startswith("Mbpp/")
+        else ptask["pe_input"][0]
+    )
+    ####################################################################
+
+    ####################################################################
+    ############### Lazily profile reference solutions #################
+    ####################################################################
+    cache_ref_num_inst = [None] * n_reference
+
+    def get_avg_ref_profile(idx, check_order=True) -> Optional[Tuple]:
+        nonlocal cache_ref_num_inst
+
+        assert (
+            idx < n_reference - 1
+            and cache_ref_num_inst[idx + 1] is not None
+            or idx == n_reference - 1
+        ), f"Calling get_avg_ref_profile({idx}) before get_avg_ref_profile({idx+1}) is called, is not allowed! {n_reference = }"
+
+        if cache_ref_num_inst[idx] is not None:
+            return cache_ref_num_inst[idx], ptask["scores"][idx]
+
+        evaluation_time = PERF_EVAL_TIMEOUT_SECOND
+        ref_solution = ptask["reference"][idx]
+        for _ in range(2):  # at most retry twice
+            profiles = profile(
+                ref_solution,
+                entry_point,
+                [pe_input],
+                timeout_second_per_test=evaluation_time,
+            )
+
+            # Bad thing#1: timeout / failure happens
+            if are_profiles_broken(profiles):
+                print(f"{task_id}: [WARNING] Error in ref: {profiles}")
+                rich.print(Syntax(ref_solution, "python"))
+                print(f"{task_id}: Retrying w/ +10s timeout...")
+                evaluation_time += 10
+            else:
+                break
+
+        avg_profile = mean(profiles)
+        # Bad thing#2: if the current #instruction is faster than that of i+1
+        if idx < n_reference - 1 and avg_profile < cache_ref_num_inst[idx + 1]:
+            print(f"{task_id}: [WARNING] #{idx} ref faster than #{idx + 1}")
+            print(f"ref {idx}: #inst {avg_profile}\tscore {ptask['scores'][idx]:.1f}")
+            print(
+                f"ref {idx+1}: #inst {cache_ref_num_inst[idx+1]}\tscore {ptask['scores'][idx+1]:.1f}"
+            )
+            rich.print(Syntax(ref_solution, "python"))
+            if check_order:
+                return None
+
+        cache_ref_num_inst[idx] = avg_profile
+        ret_dict["ref"][idx]["_num_cpu_instructions"] = avg_profile
+        return cache_ref_num_inst[idx], ptask["scores"][idx]
+
+    ####################################################################
+    ############################## END #################################
+    ####################################################################
+
+    if not lazy_evaluation:  # compute everything ahead of time
+        for i in range(n_reference - 1, -1, -1):
+            if get_avg_ref_profile(i) is None:
+                break
+
+        assert (
+            None not in cache_ref_num_inst
+        ), f"{task_id}: Failed to profile certain reference: {cache_ref_num_inst = }"
+
+    profile_cache = {}
+
+    cur_profiled = 0
+    for result in ret_dict["results"]:
+        if cur_profiled >= max_profile:
+            rich.print(f"{task_id}: Reached max_profile limit {max_profile}, stopped")
+            break
+        if not result["pass"]:
+            continue
+
+        solution = result["solution"]
+
+        if solution in profile_cache:  # reuse cache
+            sample_profiles = profile_cache[solution]
+        else:
+            sample_profiles = profile(
+                solution,
+                entry_point,
+                [pe_input],
+                timeout_second_per_test=PERF_EVAL_TIMEOUT_SECOND,
+            )
+            profile_cache[solution] = sample_profiles  # store cache
+
+        score = 0
+        norm_score = 0
+        result["matching_cluster_idx"] = -1  # -1 means even slower than the slowest ref
+        # if the solution results in a timeout, score is 0
+        if are_profiles_broken(sample_profiles):
+            print(
+                f"{task_id}: Tested solution error'ed out: {sample_profiles} ... regarded as 0 score"
+            )
+            rich.print(Syntax(solution, "python"))
+        else:
+            avg_sample_profile = result["_num_cpu_instructions"] = mean(sample_profiles)
+            # Get profiles from fast to slow (back to front):
+            for j in range(n_reference - 1, -1, -1):
+                avg_ref_profile, ref_score = get_avg_ref_profile(j, check_order=False)
+                if avg_sample_profile <= avg_ref_profile:
+                    result["matching_cluster_idx"] = j
+                    score = ref_score
+                    norm_score = 100 * (j + 1) / n_reference
+                    break
+
+        result["dps"] = score
+        result["dps_norm"] = norm_score
+        result["profiled"] = True
+        cur_profiled += 1
+
+    ret_dict["dps"] = mean(not_none([r["dps"] for r in ret_dict["results"]]))
+    ret_dict["dps_norm"] = mean(not_none([r["dps_norm"] for r in ret_dict["results"]]))
+    ret_dict["n_profiled"] = cur_profiled
+
+    table_print(
+        f"[bold green]{task_id} Completed[/]",
+        {
+            "Duration": f"{time.time() - start_time:.1f}s",
+            "DPS": f"[green]{ret_dict['dps']:.1f}[/]",
+            "DPS_norm": f"[green]{ret_dict['dps_norm']:.1f}[/]",
+            "# Profiled": f"{cur_profiled} / {len(ret_dict['results'])}",
+            "Pass@1": f"{ret_dict['pass@1']:.1f}%",
+        },
+    )
+
+    return ret_dict
+
+
+# TODO(@ganler): OPTIMIZATION: reuse the samples from the generations of other datasets
+def script(
+    samples: Optional[str] = None,
+    min_correct: int = 10,
+    max_profile: Optional[int] = None,
+    n_samples: int = 100,
+    temperature: float = 1.0,
+    parallel: Optional[int] = None,
+    lazy_evaluation: bool = True,
+    i_just_wanna_run: bool = False,
+    **model_kwargs,
+):
+    max_profile = max_profile or min(min_correct * 2, n_samples)
+    assert min_correct <= max_profile <= n_samples
+    simple_test_profiler()  # test linux perf setup
+
+    if model_kwargs:
+        # To suppress the warning of tokenizers
+        os.environ["TOKENIZERS_PARALLELISM"] = os.environ.get(
+            "TOKENIZERS_PARALLELISM", "false"
+        )
+        # overwrite parameters
+        samples = run_codegen(
+            dataset="evalperf",
+            n_samples=n_samples,
+            temperature=temperature,
+            **model_kwargs,
+        )
+
+    assert samples is not None, "Please provide the path to the samples"
+
+    # Data loading
+    problems, expected_output = get_evalplus_data()
+    ptasks = get_evalperf_data()
+
+    # Parallelism
+    max_workers = parallel or max(1, default_parallelism(divisor=4))
+    assert 0 < max_workers < multiprocessing.cpu_count(), "Invalid max CPU workers"
+
+    if os.path.isdir(samples):
+        result_path = os.path.join(samples, "evalperf_results.json")
+    else:
+        assert samples.endswith(".jsonl")
+        result_path = samples.replace(".jsonl", "_evalperf_results.json")
+    brief_result_path = result_path.replace(
+        "evalperf_results.json", "evalperf_results.brief.json"
+    )
+
+    # resume results
+    eval_results = {}
+    if not i_just_wanna_run and os.path.exists(result_path):
+        resumed_result = json.load(open(result_path, "r"))
+        if (
+            resumed_result["n_samples"] == n_samples
+            and resumed_result["temperature"] == temperature
+            and resumed_result["min_correct"] == min_correct
+            and resumed_result["max_profile"] == max_profile
+        ):
+            eval_results = resumed_result["eval"]
+            for etask in eval_results:
+                ptasks.pop(etask, None)
+
+            rich.print(f"Resumed {len(eval_results)} results from {result_path}")
+
+    # Load model's samples: task_id -> a list of samples
+    sample_iter = stream_jsonl(samples)
+    samples = defaultdict(list)
+    for task in sample_iter:
+        samples[task["task_id"].replace("_", "/")].append(task["solution"])
+    samples = {k: v[:n_samples] for k, v in samples.items()}
+
+    # assert each task has n_samples
+    for task_id, s in samples.items():
+        assert len(s) == n_samples, f"{task_id} has {len(s)} samples != {n_samples}"
+
+    # Initialize eval_results
+    for task_id, ptask in ptasks.items():
+        eval_results[task_id] = {
+            "task_id": task_id,
+            "results": [],
+            "ref": [
+                {"solution": s, "score": r, "_num_cpu_instructions": None}
+                for s, r in zip(ptask["reference"], ptask["scores"])
+            ],
+            "dps": None,
+            "dps_norm": None,
+            "pass@1": None,
+            "n_profiled": None,
+        }
+
+    rule("Correctness Checking...")
+    with progress("Correctness") as p:
+        with ThreadPoolExecutor(max_workers=max_workers) as executor:
+            futures = [
+                executor.submit(
+                    correctness_worker,
+                    task_id,
+                    samples[task_id],
+                    problems[task_id],
+                    expected_output[task_id],
+                )
+                for task_id in ptasks
+            ]
+
+            for future in p.track(as_completed(futures), total=len(futures)):
+                task_id, results = future.result()
+                eval_results[task_id]["results"] = results
+                eval_results[task_id]["pass@1"] = (
+                    100 * len([r for r in results if r["pass"]]) / n_samples
+                )
+
+    rule("EvalPerf Configurations")
+    if lazy_evaluation:
+        rich.print(
+            "[bold yellow]Lazy evaluation is enabled[/]: "
+            "Fast evaluation without enumeratively checking reference order consistency."
+        )
+
+    table_print(
+        "Configurations",
+        {
+            "Max CPU": max_workers,
+            "#Tasks": len(ptasks),
+            "#Samples per task": n_samples,
+            "Min correct": min_correct,
+            "Max profile": max_profile,
+            "Result path": result_path,
+        },
+    )
+
+    rich.print(f"IDs of tasks to evaluate: {list(ptasks.keys())}")
+    rule("Evaluation Start")
+    undone = []
+    with progress("Profiling") as p:
+        with ThreadPoolExecutor(max_workers=max_workers) as executor:
+            futures = []
+            for task_id, ptask in ptasks.items():
+                n_pass = len([r for r in eval_results[task_id]["results"] if r["pass"]])
+                if n_pass < min_correct:
+                    rich.print(
+                        f"{task_id}: [bold yellow]{n_pass} < {min_correct} correct solutions, skipped[/]"
+                    )
+                    continue
+                futures.append(
+                    executor.submit(
+                        perf_worker,
+                        task_id,
+                        ptask,
+                        eval_results[task_id],
+                        lazy_evaluation,
+                        max_profile,
+                    )
+                )
+                undone.append(task_id)
+                rich.print(f"{task_id}: Queued")
+
+            for future in p.track(as_completed(futures), total=len(futures)):
+                result = future.result()
+                eval_results[result["task_id"]] = result
+                undone.remove(result["task_id"])
+                if undone and len(undone) < max_workers:
+                    print(f"Still running: {undone}")
+
+    rule("Evaluation Summary")
+    dps = mean(not_none([res["dps"] for res in eval_results.values()]))
+    dps_norm = mean(not_none([res["dps_norm"] for res in eval_results.values()]))
+    pass_1 = mean(not_none([res["pass@1"] for res in eval_results.values()]))
+    n_evalperfed = len(not_none([res["dps"] for res in eval_results.values()]))
+
+    table_print(
+        "EvalPerf Summary",
+        {
+            "DPS": f"{dps:.1f}",
+            "DPS_norm": f"{dps_norm:.1f}",
+            "Pass@1": f"{pass_1:.1f}%",
+            "#EvalPerf-ed tasks": f"{n_evalperfed} / {len(eval_results)}",
+            "min_correct": min_correct,
+            "n_samples": n_samples,
+            "temperature": temperature,
+        },
+    )
+
+    # Save full results
+    with open(result_path, "w") as f:
+        f.write(
+            json.dumps(
+                {
+                    "date": datetime.now().strftime("%Y-%m-%d %H:%M"),
+                    "n_samples": n_samples,
+                    "temperature": temperature,
+                    "min_correct": min_correct,
+                    "max_profile": max_profile,
+                    "eval": eval_results,
+                }
+            )
+        )
+    rich.print(f"Full results have been saved to {result_path}")
+
+    # Save brief results
+    with open(brief_result_path, "w") as f:
+        f.write(
+            json.dumps(
+                {
+                    "date": datetime.now().strftime("%Y-%m-%d %H:%M"),
+                    "config": {
+                        "n_samples": n_samples,
+                        "temperature": temperature,
+                        "min_correct": min_correct,
+                        "max_profile": max_profile,
+                    },
+                    "summary": {
+                        "dps": dps,
+                        "dps_norm": dps_norm,
+                        "pass@1": pass_1,
+                    },
+                    "eval": {
+                        task_id: {
+                            "dps": res["dps"],
+                            "dps_norm": res["dps_norm"],
+                            "pass@1": res["pass@1"],
+                            "profiled": [
+                                {
+                                    "solution": r["solution"],
+                                    "matching_cluster_idx": r["matching_cluster_idx"],
+                                }
+                                for r in res["results"]
+                                if r["profiled"]
+                            ],
+                        }
+                        for task_id, res in eval_results.items()
+                    },
+                }
+            )
+        )
+
+    rich.print(f"Brief results have been saved to {brief_result_path}")
+
+    rule("To visualize win-rates and pair-wise DPS, run:")
+    rich.print(
+        Syntax(
+            f"""\
+git clone git@github.com:evalplus/evalplus.github.io.git
+git --git-dir=evalplus.github.io/.git pull
+cp {brief_result_path} evalplus.github.io/results/evalperf
+python evalplus.github.io/results/evalperf/stats.py
+python -m http.server -d evalplus.github.io {get_free_port()}""",
+            "bash",
+        )
+    )
+
+
+def main():
+    from fire import Fire
+
+    Fire(script)
+
+
+if __name__ == "__main__":
+    main()

opencompass/datasets/evalplus/evaluate.py

@@ -0,0 +1,373 @@
+import json
+import multiprocessing
+import os
+import pickle
+import threading
+import time
+from collections import Counter, defaultdict
+from concurrent.futures import ProcessPoolExecutor, as_completed
+from datetime import datetime
+from typing import Any, Dict, List, Optional, Tuple
+from warnings import warn
+
+import numpy as np
+from termcolor import cprint
+from tqdm import tqdm
+
+from opencompass.datasets.evalplus.codegen import run_codegen
+from opencompass.datasets.evalplus.config import *
+from opencompass.datasets.evalplus.data import (
+    get_human_eval_plus,
+    get_human_eval_plus_hash,
+    get_mbpp_plus,
+    get_mbpp_plus_hash,
+    load_solutions,
+)
+from opencompass.datasets.evalplus.data.mbpp import mbpp_serialize_inputs
+from opencompass.datasets.evalplus.data.utils import CACHE_DIR
+from opencompass.datasets.evalplus.eval import (
+    PASS,
+    compatible_eval_result,
+    estimate_pass_at_k,
+    untrusted_check,
+)
+from opencompass.datasets.evalplus.eval._special_oracle import MBPP_OUTPUT_NOT_NONE_TASKS
+from opencompass.datasets.evalplus.gen.util import trusted_exec
+
+# 1st item: the status
+# 2nd item (optional): the detailed pass/fail boolean for each input
+Result = Tuple[str, List[bool]]
+
+def get_groundtruth(problems, hashcode, tasks_only_output_not_none):
+    cache_file = os.path.join(CACHE_DIR, f"{hashcode}.pkl")
+    if os.path.exists(cache_file):
+        print(f"Load from ground-truth from {cache_file}")
+        with open(cache_file, "rb") as f:
+            return pickle.load(f)
+
+    os.makedirs(CACHE_DIR, exist_ok=True)
+    print("Computing expected output...")
+    tbegin = time.time()
+    expected_output = {}
+    for task_id, problem in problems.items():
+        oracle = {}
+        oracle["base"], oracle["base_time"] = trusted_exec(
+            problem["prompt"] + problem["canonical_solution"],
+            problem["base_input"],
+            problem["entry_point"],
+            record_time=True,
+            output_not_none=problem["entry_point"] in tasks_only_output_not_none,
+        )
+
+        oracle["plus"], oracle["plus_time"] = trusted_exec(
+            problem["prompt"] + problem["canonical_solution"],
+            problem["plus_input"],
+            problem["entry_point"],
+            record_time=True,
+            output_not_none=problem["entry_point"] in tasks_only_output_not_none,
+        )
+        expected_output[task_id] = oracle
+        #print('oracle', oracle)
+    print(f"Expected outputs computed in {time.time() - tbegin:.2f}s")
+
+    with open(cache_file, "wb") as f:
+        pickle.dump(expected_output, f)
+    # print('get_groundtruth', expected_output)
+    return expected_output
+
+def check_correctness(
+    dataset: str,
+    completion_id: int,
+    problem: Dict[str, Any],
+    solution: str,
+    expected_output: Dict[str, List],
+    base_only=False,
+    fast_check=False,
+    identifier=None,
+    min_time_limit: float = DEFAULT_MIN_TIME_LIMIT,
+    gt_time_limit_factor: float = DEFAULT_GT_TIME_LIMIT_FACTOR,
+) -> Dict[str, Result]:  # {...}, "base" | "plus" -> (status, details)
+    ret = {
+        "completion_id": completion_id,
+        "task_id": problem["task_id"],
+        "_identifier": identifier,
+        "solution": solution,
+    }
+    ret["base"] = untrusted_check(
+        dataset,
+        solution,
+        problem["base_input"],
+        problem["entry_point"],
+        expected=expected_output["base"],
+        atol=problem["atol"],
+        ref_time=expected_output["base_time"],
+        fast_check=fast_check,
+        min_time_limit=min_time_limit,
+        gt_time_limit_factor=gt_time_limit_factor,
+    )
+
+    if not base_only:
+        ret["plus"] = untrusted_check(
+            dataset,
+            solution,
+            problem["plus_input"],
+            problem["entry_point"],
+            expected=expected_output["plus"],
+            atol=problem["atol"],
+            ref_time=expected_output["plus_time"],
+            fast_check=fast_check,
+            min_time_limit=min_time_limit,
+            gt_time_limit_factor=gt_time_limit_factor,
+        )
+
+    return ret
+
+
+def evaluate(
+    dataset: str,
+    samples: Optional[str] = None,
+    base_only: bool = False,
+    parallel: Optional[int] = None,
+    i_just_wanna_run: bool = False,
+    test_details: bool = False,
+    min_time_limit: float = DEFAULT_MIN_TIME_LIMIT,
+    gt_time_limit_factor: float = DEFAULT_GT_TIME_LIMIT_FACTOR,
+    mini: bool = False,
+    noextreme: bool = False,
+    version: str = "default",
+    output_file: Optional[str] = None,
+    lang: str = "ar",
+    **model_kwargs,
+):
+    if model_kwargs:
+        print('model_kwargs', model_kwargs)
+        # To suppress the warning of tokenizers
+        os.environ["TOKENIZERS_PARALLELISM"] = os.environ.get(
+            "TOKENIZERS_PARALLELISM", "false"
+        )
+        samples = run_codegen(
+            dataset=dataset,
+            **model_kwargs,
+        )
+    assert samples is not None, "No samples provided"
+
+    n_workers = parallel or max(1, multiprocessing.cpu_count() // 2)
+
+    if os.path.isdir(samples):
+        result_path = os.path.join(samples, "eval_results.json")
+    else:
+        assert samples.endswith(".jsonl")
+        result_path = samples.replace(".jsonl", "_eval_results.json")
+
+    if output_file is not None:
+        result_path = output_file
+
+    if os.path.isfile(result_path) and not i_just_wanna_run:
+        print(f"Load from previous results from {result_path}")
+        with open(result_path, "r") as f:
+            results = json.load(f)
+
+        results = compatible_eval_result(results)
+    else:
+        if dataset == "humaneval":
+            print('get_human_eval_plus')
+            problems = get_human_eval_plus(
+                mini=mini, noextreme=noextreme, version=version, lang=lang
+            )
+            dataset_hash = get_human_eval_plus_hash(
+                mini=mini, noextreme=noextreme, version=version
+            )
+            expected_output = get_groundtruth(problems, dataset_hash, [])
+        elif dataset == "mbpp":
+            problems = get_mbpp_plus(mini=mini, noextreme=noextreme, version=version)
+            dataset_hash = get_mbpp_plus_hash(
+                mini=mini, noextreme=noextreme, version=version
+            )
+            expected_output = get_groundtruth(
+                problems,
+                dataset_hash,
+                MBPP_OUTPUT_NOT_NONE_TASKS,
+            )
+
+        results = {
+            "date": datetime.now().strftime("%Y-%m-%d %H:%M"),
+            "hash": dataset_hash,
+            "eval": {},
+        }
+
+        with ProcessPoolExecutor(max_workers=n_workers) as executor:
+            futures = []
+            completion_id = Counter()
+            n_samples = 0
+            eval_results = defaultdict(list)  # task_id ->
+            remainings = set()
+
+            print("Reading samples...")
+            print('samples', samples)
+            for sample in tqdm(load_solutions(samples)):
+                task_id = sample["task_id"]
+                if task_id not in problems:
+                    warn(
+                        f"Task {task_id} is found in the samples but not found in the dataset"
+                    )
+                    continue
+                solution = (
+                    sample["solution"]
+                    if "solution" in sample
+                    else problems[task_id]["prompt"] + sample["completion"]
+                )
+                remainings.add(sample["_identifier"])
+                args = (
+                    dataset,
+                    completion_id[task_id],
+                    problems[task_id],
+                    solution,
+                    expected_output[task_id],
+                    base_only,
+                    not test_details,  # fast_check
+                    sample["_identifier"],
+                    min_time_limit,
+                    gt_time_limit_factor,
+                )
+                futures.append(executor.submit(check_correctness, *args))
+                completion_id[task_id] += 1
+                n_samples += 1
+
+            assert n_samples == len(remainings), "Missing problems in unfinished"
+            assert len(completion_id) == len(problems), "Missing problems in samples"
+
+            def stucking_checker():
+                while remainings:
+                    last_size = len(remainings)
+                    time.sleep(20)
+                    if last_size != len(remainings) or len(remainings) == 0:
+                        continue
+                    # Potential stucking
+                    warn("No samples had finished testing in the last 20s")
+                    warn(f"{len(remainings)} samples to be tested: {remainings}")
+
+            threading.Thread(target=stucking_checker).start()
+
+            for future in tqdm(as_completed(futures), total=n_samples):
+                result = future.result()
+                print('result 253', result)
+                remainings.remove(result["_identifier"])
+                eval_results[result["task_id"]].append(result)
+
+        # sort the results for each problem by completion_id
+        for task_id, task_results in eval_results.items():
+            task_results.sort(key=lambda x: x["completion_id"])
+            results["eval"][task_id] = []
+            for res in task_results:
+
+                def get_failed_tests(stat, details, inputs) -> List[Any]:
+                    if stat == PASS or not details:
+                        return []
+
+                    if test_details:
+                        return [
+                            inputs[i] for i in range(len(details)) if not details[i]
+                        ]
+
+                    # else => simply return the only and the last fail test
+                    return [inputs[len(details) - 1]]
+
+                base_stat, base_details = res["base"]
+                base_fail_tests = get_failed_tests(
+                    base_stat, base_details, problems[task_id]["base_input"]
+                )
+
+                # initialize plus tests
+                plus_stat = None
+                plus_fail_tests = []
+
+                # with plus tests
+                if not base_only:
+                    plus_stat, plus_details = res["plus"]
+                    plus_fail_tests = get_failed_tests(
+                        plus_stat, plus_details, problems[task_id]["plus_input"]
+                    )
+
+                if dataset == "mbpp":
+                    base_fail_tests = mbpp_serialize_inputs(task_id, base_fail_tests)
+                    plus_fail_tests = mbpp_serialize_inputs(task_id, plus_fail_tests)
+
+                results["eval"][task_id].append(
+                    {
+                        "task_id": task_id,
+                        "solution": res["solution"],
+                        "base_status": base_stat,
+                        "plus_status": plus_stat,
+                        "base_fail_tests": base_fail_tests,
+                        "plus_fail_tests": plus_fail_tests,
+                    }
+                )
+
+    # Calculate pass@k.
+    total = np.array([len(r) for r in results["eval"].values()])
+    base_correct = []
+    new_correct = []
+
+    for res in results["eval"].values():
+        bc = sum([r["base_status"] == PASS for r in res])
+        base_correct.append(bc)
+        if not base_only:
+            new_correct.append(
+                sum(
+                    [
+                        res[i]["base_status"] == res[i]["plus_status"] == PASS
+                        for i in range(len(res))
+                    ]
+                )
+            )
+    base_correct = np.array(base_correct)
+
+    pass_at_k = {
+        f"pass@{k}": estimate_pass_at_k(total, base_correct, k).mean()
+        for k in [1, 10, 100]
+        if total.min() >= k
+    }
+    cprint(f"{dataset} (base tests)", "red")
+    for k, v in pass_at_k.items():
+        cprint(f"{k}:\t{v:.3f}", "red")
+    results["pass_at_k"] = {"base": pass_at_k}
+
+    if new_correct:
+        cprint(f"{dataset}+ (base + extra tests)", "green")
+        pass_at_k = {
+            f"pass@{k}": estimate_pass_at_k(total, np.array(new_correct), k).mean()
+            for k in [1, 10, 100]
+            if (total >= k).all()
+        }
+        for k, v in pass_at_k.items():
+            cprint(f"{k}:\t{v:.3f}", "green")
+        results["pass_at_k"]["plus"] = pass_at_k
+
+    # save results
+    if os.path.isfile(result_path) and i_just_wanna_run:
+        decision = ""
+        while decision.lower() not in ["y", "n"]:
+            print(f"{result_path} already exists. Press [Y/N] to overwrite or exit...")
+            decision = input()
+
+        if decision.lower() == "y":
+            # mv the file to a backup
+            new_path = result_path + ".bak"
+            while os.path.isfile(new_path):
+                new_path += ".bak"
+            os.rename(result_path, new_path)
+            print(f"Backup {result_path} to {new_path}")
+
+    if not os.path.isfile(result_path):
+        with open(result_path, "w") as f:
+            json.dump(results, f)
+
+
+def main():
+    from fire import Fire
+
+    Fire(evaluate)
+
+
+if __name__ == "__main__":
+    main()

opencompass/datasets/evalplus/gen/__init__.py

@@ -0,0 +1,21 @@
+import copy
+from typing import Any, List
+
+
+class BaseGen(object):
+    def __init__(self, inputs: List[Any], entry_point: str, contract: str):
+        """Initializing a input mutator.
+
+        Args:
+            inputs (List[Any]): The set of initial inputs (i.e., seeds)
+            entry_point (str): The function name to invoke with the input
+            contract (str): The contract to verify input validity
+        """
+        self.contract = contract
+        self.entry_point = entry_point
+        self.seed_pool: List[Any] = copy.deepcopy(inputs)
+        self.new_inputs = []
+        self.seed_hash = set([hash(str(x)) for x in self.seed_pool])
+
+    def generate(self, num: int) -> List[Any]:
+        raise NotImplementedError

opencompass/datasets/evalplus/gen/chatgpt_gen.py

@@ -0,0 +1,78 @@
+import ast
+import random
+from typing import List
+
+import openai
+from openai.types.chat import ChatCompletion
+
+from evalplus.data.utils import to_raw
+from evalplus.gen import BaseGen
+from evalplus.gen.util import trusted_check_exec
+from evalplus.gen.util.openai_request import make_auto_request
+
+
+class ChatGPTGen(BaseGen):
+    def __init__(self, inputs: List, signature: str, contract_code: str, gd_code: str):
+        super().__init__(inputs, signature, contract_code)
+        self.gd_code = gd_code
+        self.prompt_messages = [
+            "Please generate complex inputs to test the function.",
+            "Please generate corner case inputs to test the function.",
+            "Please generate difficult inputs to test the function.",
+        ]
+        self.iteration = 20
+        self.client = openai.Client()
+
+    def seed_selection(self) -> List:
+        # get 5 for now.
+        return random.sample(self.seed_pool, k=min(len(self.seed_pool), 5))
+
+    @staticmethod
+    def _parse_ret(ret: ChatCompletion) -> List:
+        rets = []
+        output = ret.choices[0].message.content
+        if "```" in output:
+            for x in output.split("```")[1].splitlines():
+                if x.strip() == "":
+                    continue
+                try:
+                    # remove comments
+                    input = ast.literal_eval(f"[{x.split('#')[0].strip()}]")
+                except:  # something wrong.
+                    continue
+                rets.append(input)
+        return rets
+
+    def chatgpt_generate(self, selected_inputs: List) -> List:
+        # append the groundtruth function
+        # actually it can be any function (maybe we can generate inputs for each llm generated code individually)
+        message = f"Here is a function that we want to test:\n```\n{self.gd_code}\n```"
+        str_inputs = "\n".join(
+            [
+                ", ".join([f"'{to_raw(i)}'" if type(i) == str else str(i) for i in x])
+                for x in selected_inputs
+            ]
+        )
+        message += f"\nThese are some example inputs used to test the function:\n```\n{str_inputs}\n```"
+        message += f"\n{random.choice(self.prompt_messages)}"
+        ret = make_auto_request(
+            self.client,
+            message=message,
+            model="gpt-3.5-turbo",
+            max_tokens=256,
+            response_format={"type": "text"},
+        )
+        return self._parse_ret(ret)
+
+    def generate(self, num: int):
+        while len(self.new_inputs) < num and self.iteration >= 0:
+            seeds = self.seed_selection()
+            new_inputs = self.chatgpt_generate(seeds)
+            for new_input in new_inputs:
+                if hash(str(new_input)) not in self.seed_hash:
+                    if trusted_check_exec(self.contract, [new_input], self.entry_point):
+                        self.seed_pool.append(new_input)
+                        self.seed_hash.add(hash(str(new_input)))
+                        self.new_inputs.append(new_input)
+            self.iteration -= 1
+        return self.new_inputs[:num]

opencompass/datasets/evalplus/gen/mut_gen.py

@@ -0,0 +1,30 @@
+import random
+from abc import abstractmethod
+from typing import Any, List
+
+from evalplus.gen import BaseGen
+from evalplus.gen.util import trusted_check_exec
+
+
+class MutateGen(BaseGen):
+    def __init__(self, inputs: List, signature: str, contract_code: str):
+        super().__init__(inputs, signature, contract_code)
+
+    def seed_selection(self):
+        # random for now.
+        return random.choice(self.seed_pool)
+
+    @abstractmethod
+    def mutate(self, seed_input: Any) -> Any:
+        pass
+
+    def generate(self, num: int) -> List[Any]:
+        while len(self.new_inputs) < num:
+            seed = self.seed_selection()
+            new_input = self.mutate(seed)
+            if hash(str(new_input)) not in self.seed_hash:
+                if trusted_check_exec(self.contract, [new_input], self.entry_point):
+                    self.seed_pool.append(new_input)
+                    self.seed_hash.add(hash(str(new_input)))
+                    self.new_inputs.append(new_input)
+        return self.new_inputs[:num]

opencompass/datasets/evalplus/gen/type_mut.py

@@ -0,0 +1,340 @@
+import copy
+import random
+import string
+import time
+from typing import Any, Dict, List, Set, Tuple
+
+from multipledispatch import dispatch
+
+from evalplus.gen.mut_gen import MutateGen
+from evalplus.gen.util import trusted_check_exec
+
+MAX_MULTI_STEP_SIZE = 5
+MUTATE_BOUND_SIZE = 8
+
+NoneType = type(None)
+
+
+# decorator to use ingredients
+class use_ingredient:
+    def __init__(self, prob: float):
+        assert 0 <= prob <= 0.95
+        self.prob = prob
+
+    def __call__(obj, func):
+        def wrapper(self, seed_input):
+            if random.random() < obj.prob and self.ingredients[type(seed_input)]:
+                return random.choice(list(self.ingredients[type(seed_input)]))
+            else:
+                return func(self, seed_input)
+
+        return wrapper
+
+
+class TypedMutGen(MutateGen):
+    def __init__(self, inputs: List, signature: str, contract_code: str):
+        super().__init__(inputs, signature, contract_code)
+        self.timeout = 60 * 60  # 1 hour
+        self.ingredients = {
+            int: set(),
+            float: set(),
+            str: set(),
+            complex: set(),
+        }
+        for x in inputs:
+            self.fetch_ingredient(x)
+
+    def seed_selection(self):
+        # random for now.
+        return random.choice(self.seed_pool)
+
+    def mutate(self, seed_input: Any) -> List:
+        new_input = copy.deepcopy(seed_input)
+
+        patience = MUTATE_BOUND_SIZE
+        while new_input == seed_input or patience == 0:
+            new_input = self.typed_mutate(new_input)
+            patience -= 1
+
+        return new_input
+
+    #########################
+    # Type-aware generation #
+    #########################
+    @dispatch(NoneType)
+    def typed_gen(self, _):
+        return None
+
+    @dispatch(int)
+    def typed_gen(self, _):
+        @use_ingredient(0.5)
+        def _impl(*_):
+            return random.randint(-100, 100)
+
+        return _impl(self, _)
+
+    @dispatch(float)
+    def typed_gen(self, _):
+        @use_ingredient(0.5)
+        def _impl(*_):
+            return random.uniform(-100, 100)
+
+        return _impl(self, _)
+
+    @dispatch(bool)
+    def typed_gen(self, _):
+        return random.choice([True, False])
+
+    @dispatch(str)
+    def typed_gen(self, _):
+        @use_ingredient(0.5)
+        def _impl(*_):
+            return "".join(
+                random.choice(string.ascii_letters)
+                for _ in range(random.randint(0, 10))
+            )
+
+        return _impl(self, _)
+
+    def any_gen(self):
+        # weighted choose
+        choice = random.choices(
+            [
+                True,
+                1,
+                1.1,
+                "str",
+                [],  # list
+                tuple(),  # tuple
+                dict(),  # dict
+                None,  # None
+            ],
+            [0.2, 0.2, 0.2, 0.2, 0.05, 0.05, 0.05, 0.05],
+        )[0]
+        return self.typed_gen(choice)
+
+    @dispatch(list)
+    def typed_gen(self, _):
+        ret = []
+        size = random.randint(0, 10)
+        if random.randint(0, 4) == 0:  # heterogeneous
+            for _ in range(size):
+                ret.append(self.any_gen())
+        else:  # homogeneous
+            t = random.choice([bool(), int(), float(), str()])
+            for _ in range(size):
+                ret.append(self.typed_gen(t))
+        return ret
+
+    @dispatch(tuple)
+    def typed_gen(self, _):
+        return tuple(self.typed_gen([]))
+
+    # NOTE: disable set for now as Steven is too weak in Python (/s)
+    # @dispatch(set)
+    # def typed_gen(self, _):
+    #     return set(self.typed_gen([]))
+
+    @dispatch(dict)
+    def typed_gen(self, _):
+        ret = dict()
+        values = self.typed_gen([])
+        # NOTE: Assumption: nobody uses dict with heterogeneous keys
+        # NOTE: Assumption: nobody uses dict with boolean keys
+        key_type = random.choice([int(), float(), str()])
+        for v in values:
+            ret[self.typed_gen(key_type)] = self.typed_gen(v)
+        return ret
+
+    ########################
+    # Type-aware mutation  #
+    ########################
+    # Simple primitives
+    @dispatch(int)
+    def typed_mutate(self, seed_input: int):
+        @use_ingredient(0.5)
+        def _impl(_, seed_input: int):
+            return seed_input + random.randint(-1, 1)
+
+        return _impl(self, seed_input)
+
+    @dispatch(float)
+    def typed_mutate(self, seed_input: float):
+        @use_ingredient(0.5)
+        def _impl(_, seed_input: float):
+            if random.randint(0, 1):
+                return seed_input + random.uniform(-1, 1)
+            return seed_input * (1 + random.uniform(-0.5, 0.5))
+
+        return _impl(self, seed_input)
+
+    @dispatch(complex)
+    def typed_mutate(self, seed_input: complex):
+        @use_ingredient(0.5)
+        def _impl(_, seed_input: complex):
+            imag = seed_input.imag + random.uniform(-1, 1)
+            return complex(0, imag)
+
+        return _impl(self, seed_input)
+
+    @dispatch(bool)
+    def typed_mutate(self, seed_input: bool):
+        return random.choice([True, False])
+
+    @dispatch(NoneType)
+    def typed_mutate(self, seed_input: NoneType):
+        return None
+
+    # List-like
+    @dispatch(list)
+    def typed_mutate(self, seed_input: List):
+        if len(seed_input) == 0:
+            return self.typed_gen([])
+
+        choice = random.randint(0, 3)
+        idx = random.randint(0, len(seed_input) - 1)
+        if choice == 0:  # remove one element
+            seed_input.pop(random.randint(0, len(seed_input) - 1))
+        elif choice == 1 and len(seed_input) > 0:  # add one mutated element
+            seed_input.insert(
+                random.randint(0, len(seed_input) - 1),
+                self.typed_mutate(seed_input[idx]),
+            )
+        elif choice == 2 and len(seed_input) > 0:  # repeat one element
+            seed_input.append(seed_input[idx])
+        else:  # inplace element change
+            seed_input[idx] = self.typed_mutate(seed_input[idx])
+        return seed_input
+
+    @dispatch(tuple)
+    def typed_mutate(self, seed_input: Tuple):
+        return tuple(self.typed_mutate(list(seed_input)))
+
+    # String
+    @dispatch(str)
+    def typed_mutate(self, seed_input: str):
+        @use_ingredient(0.4)
+        def _impl(_, seed_input: str):
+            choice = random.randint(0, 2) if seed_input else 0
+            if choice == 0 and self.ingredients[str]:  # insert an ingredient
+                idx = random.randint(0, len(seed_input))
+                return (
+                    seed_input[:idx]
+                    + random.choice(list(self.ingredients[str]))
+                    + seed_input[idx:]
+                )
+            # other choices assume len(seed_input) > 0
+            elif choice == 1:  # replace a substring with empty or mutated string
+                start = random.randint(0, len(seed_input) - 1)
+                end = random.randint(start + 1, len(seed_input))
+                mid = (
+                    ""
+                    if random.randint(0, 1)
+                    else self.typed_mutate(seed_input[start:end])
+                )
+                return seed_input[:start] + mid + seed_input[end:]
+            elif choice == 2:  # repeat one element
+                idx = random.randint(0, len(seed_input) - 1)
+                return (
+                    seed_input[:idx]
+                    + seed_input[random.randint(0, len(seed_input) - 1)]
+                    + seed_input[idx:]
+                )
+
+            # random char
+            return self.typed_gen(str())
+
+        return _impl(self, seed_input)
+
+    # Set
+    @dispatch(set)
+    def typed_mutate(self, seed_input: Set):
+        return set(self.typed_mutate(list(seed_input)))
+
+    # Dict
+    @dispatch(dict)
+    def typed_mutate(self, seed_input: Dict):
+        if len(seed_input) == 0:
+            return self.typed_gen(dict())
+
+        choice = random.randint(0, 2)
+        if choice == 0:  # remove a kv
+            del seed_input[random.choice(list(seed_input.keys()))]
+        elif choice == 1:  # add a kv
+            k = self.typed_mutate(random.choice(list(seed_input.keys())))
+            v = self.typed_mutate(random.choice(list(seed_input.values())))
+            seed_input[k] = v
+        elif choice == 2:  # inplace value change
+            k0, v0 = random.choice(list(seed_input.items()))
+            seed_input[k0] = self.typed_mutate(v0)
+        return seed_input
+
+    ############################################
+    # Fetching ingredients to self.ingredients #
+    ############################################
+    def fetch_ingredient(self, seed_input):
+        self.typed_fetch(seed_input)
+
+    @dispatch(int)
+    def typed_fetch(self, seed_input: int):
+        self.ingredients[int].add(seed_input)
+
+    @dispatch(float)
+    def typed_fetch(self, seed_input: float):
+        self.ingredients[float].add(seed_input)
+
+    @dispatch(complex)
+    def typed_fetch(self, seed_input: complex):
+        self.ingredients[complex].add(seed_input)
+
+    @dispatch(str)
+    def typed_fetch(self, seed_input: str):
+        self.ingredients[str].add(seed_input)
+        for token in seed_input.strip().split():
+            self.ingredients[str].add(token)
+
+    # List-like
+    def _fetch_list_like(self, seed_input):
+        for x in seed_input:
+            if self.typed_fetch.dispatch(type(x)):
+                self.fetch_ingredient(x)
+
+    @dispatch(list)
+    def typed_fetch(self, seed_input: List):
+        self._fetch_list_like(seed_input)
+
+    @dispatch(tuple)
+    def typed_fetch(self, seed_input: Tuple):
+        self._fetch_list_like(seed_input)
+
+    # NOTE: disable set for now as Steven is too weak in Python (/s)
+    # @dispatch(set)
+    # def typed_fetch(self, seed_input: Set):
+    #     self._fetch_list_like(seed_input)
+
+    # Dict
+    @dispatch(dict)
+    def typed_fetch(self, seed_input: Dict):
+        self._fetch_list_like(seed_input.keys())
+        self._fetch_list_like(seed_input.values())
+
+    def generate(self, num: int):
+        start = time.time()
+        num_generated = 1
+        while len(self.new_inputs) < num and time.time() - start < self.timeout:
+            if num_generated % 1000 == 0:
+                print(
+                    f"generated {num_generated} already with {len(self.new_inputs)} new inputs ... "
+                )
+            new_input = self.seed_selection()
+            # Multi-step instead of single-step
+            for _ in range(random.randint(1, MAX_MULTI_STEP_SIZE)):
+                new_input = self.mutate(new_input)
+            num_generated += 1
+            if hash(str(new_input)) not in self.seed_hash:
+                if trusted_check_exec(self.contract, [new_input], self.entry_point):
+                    self.typed_fetch(new_input)
+                    self.seed_pool.append(new_input)
+                    self.new_inputs.append(new_input)
+                self.seed_hash.add(hash(str(new_input)))
+        return self.new_inputs[:num]

opencompass/datasets/evalplus/gen/util/__init__.py

@@ -0,0 +1,40 @@
+import time
+from copy import deepcopy
+
+from opencompass.datasets.evalplus.eval.utils import time_limit
+
+
+def trusted_exec(code, inputs, entry_point, record_time=False, output_not_none=False):
+    """Execute trusted code in place."""
+    exec_globals = {}
+    exec(code, exec_globals)
+    fn = exec_globals[entry_point]
+
+    rtime = []
+    ret = []
+    for inp in inputs:
+        inp = deepcopy(inp)
+        if record_time:
+            start = time.time()
+            ret.append(fn(*inp))
+            rtime.append(time.time() - start)
+        else:
+            ret.append(fn(*inp))
+
+    if output_not_none:
+        ret = [i is not None for i in ret]
+
+    if record_time:
+        return ret, rtime
+    else:
+        return ret
+
+
+def trusted_check_exec(code, inputs, entry_point):
+    """Check trusted_exec success."""
+    try:
+        with time_limit(seconds=1.0):
+            trusted_exec(code, inputs, entry_point)
+    except Exception:
+        return False
+    return True

opencompass/datasets/evalplus/inputgen.py

@@ -0,0 +1,107 @@
+"""Generate a .jsonl file where each line is a json object
+representing a programming problem with a task ID ("task_id")
+and a list of enhanced inputs ("inputs") for that task.
+"""
+
+import argparse
+import json
+import os
+
+from opencompass.datasets.evalplus.data.mbpp import mbpp_serialize_inputs
+from opencompass.datasets.evalplus.gen.chatgpt_gen import ChatGPTGen
+from opencompass.datasets.evalplus.gen.type_mut import TypedMutGen
+
+class SetEncoder(json.JSONEncoder):
+    def default(self, obj):
+        if isinstance(obj, set):
+            return list(obj)
+        return json.JSONEncoder.default(self, obj)
+
+
+# Used for MBPP as MBPP's prompt is not a formal function signature
+def insert_contract_into_code(entry_point, code, contract):
+    lines = code.split("\n")
+    index = lines.index(
+        next(line for line in lines if line.startswith(f"def {entry_point}"))
+    )
+    lines.insert(index + 1, contract)
+    return "\n".join(lines)
+
+
+def input_generation(args, problems):
+    with open(args.output, "w") as file:
+        for problem in problems.values():
+            new_input = {}
+            task_id = problem["task_id"]
+            print(f"generating inputs for {task_id} ...")
+            # by default we do not include constraints in the prompt (code)
+            code = problem["prompt"] + problem["canonical_solution"]
+            # but we use c_code to include contract which checks input validity at execution time
+            if args.dataset == "humaneval":
+                c_code = (
+                    problem["prompt"]
+                    + problem["contract"]
+                    + problem["canonical_solution"]
+                )
+            elif args.dataset == "mbpp":
+                c_code = problem["prompt"] + insert_contract_into_code(
+                    entry_point=problem["entry_point"],
+                    code=problem["canonical_solution"],
+                    contract=problem["contract"],
+                )
+
+            # first generate chatgpt
+            input_gen = ChatGPTGen(
+                problem["base_input"], problem["entry_point"], c_code, code
+            ).generate(args.chatgpt_len)
+            # generate mutation next
+
+            if input_gen is None or len(input_gen) == 0:
+                new_input["task_id"] = task_id
+                new_input["inputs"] = {}
+                file.write(json.dumps(new_input, cls=SetEncoder) + "\n")
+                continue
+
+            input_gen.extend(
+                TypedMutGen(input_gen, problem["entry_point"], c_code).generate(
+                    args.mut_len
+                )
+            )
+            print(f"generated {len(input_gen)} inputs")
+            new_input["task_id"] = task_id
+            if args.dataset == "mbpp":
+                new_input["inputs"] = mbpp_serialize_inputs(task_id, input_gen)
+            new_input["inputs"] = input_gen
+            file.write(json.dumps(new_input, cls=SetEncoder) + "\n")
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--dataset", required=True, type=str, choices=["humaneval", "mbpp"]
+    )
+    parser.add_argument("--chatgpt_len", required=True, type=int)
+    parser.add_argument("--mut_len", required=True, type=int)
+    parser.add_argument("--output", type=str, help="Output .jsonl path")
+    args = parser.parse_args()
+
+    problems = None
+    if args.dataset == "humaneval":
+        from evalplus.data import get_human_eval_plus
+
+        # Allow it to be incomplete
+        problems = get_human_eval_plus(err_incomplete=False)
+        args.output = args.output or "HumanEvalPlusInputs.jsonl"
+
+    if args.dataset == "mbpp":
+        from evalplus.data import get_mbpp_plus
+
+        problems = get_mbpp_plus(err_incomplete=False)
+        args.output = args.output or "MbppPlusInput.jsonl"
+
+    assert not os.path.isfile(args.output), f"{args.output} already exists!"
+    input_generation(args, problems)
+
+
+if __name__ == "__main__":
+    main()

opencompass/datasets/evalplus/lecacy_sanitize.py

@@ -0,0 +1,200 @@
+"""Legacy version of post-processing LLM-generated Python code.
+This sanitizer is implemented using regex and string manipulation.
+You might want to use the latest tree-sitter-based sanitizer (evalplus.sanitize) instead.
+"""
+
+import os
+import pathlib
+import re
+from typing import List, Optional
+
+from tqdm import tqdm
+
+from opencompass.datasets.evalplus.data import (
+    get_human_eval_plus,
+    get_mbpp_plus,
+    load_solutions,
+    write_directory,
+    write_jsonl,
+)
+from opencompass.datasets.evalplus.syncheck import syntax_check
+
+def remove_unindented_lines(
+    code: str, protect_before: str, execeptions: List[str], trim_tails: List[str]
+) -> str:
+    lines = code.splitlines()
+    cut_idx = []
+    cut_enabled = False
+    for i, line in enumerate(lines):
+        if not cut_enabled and line.startswith(protect_before):
+            cut_enabled = True
+            continue
+        if line.strip() == "":
+            continue
+        if any(line.startswith(e) for e in execeptions):
+            continue
+
+        lspace = len(line) - len(line.lstrip())
+        if lspace == 0:
+            cut_idx.append(i)
+
+        if any(line.rstrip().startswith(t) for t in trim_tails):
+            # cut off everything behind
+            cut_idx.extend(list(range(i, len(lines))))
+            break
+
+    return "\n".join([line for i, line in enumerate(lines) if i not in cut_idx])
+
+
+def to_four_space_indents(old_code):
+    new_code = ""
+    for line in old_code.splitlines():
+        lspace = len(line) - len(line.lstrip())
+        if lspace == 3:
+            new_code += " "
+        new_code += line + "\n"
+    return new_code
+
+
+def sanitize(
+    old_code: str,
+    entry_point: str,
+    rm_prefix_lines: Optional[str] = None,
+    eofs: List = None,
+):
+    new_code = old_code
+    if rm_prefix_lines is not None:
+        new_code = "\n".join(
+            [
+                line
+                for line in old_code.splitlines()
+                if not line.startswith(rm_prefix_lines)
+            ]
+        )
+
+    new_code = "\n" + new_code
+    def_left = "def " + entry_point
+
+    # basic handling of chat output
+    new_code = new_code.replace("\n```python\n", "\n```\n")
+    for chunk in new_code.split("\n```\n"):
+        if def_left in chunk:
+            new_code = chunk
+            break
+
+    chunks = [chunk for chunk in re.split(f"{def_left}\\s*\\(", new_code)]
+    # TODO: having return does not mean this is complete
+    bodies = [chunk for chunk in chunks[1:] if "    return " in chunk.split("\ndef")[0]]
+    def_left = def_left + "("
+    new_code = def_left + def_left.join(bodies) if len(bodies) > 0 else ""  # fn + impl
+    new_code = to_four_space_indents(new_code)
+
+    for eof in eofs or []:
+        new_code = new_code.split(eof)[0]
+
+    # remove lines starting from the first unindented line after def_left
+    new_code = remove_unindented_lines(
+        new_code,
+        protect_before=def_left,
+        execeptions=["def ", "import ", "from "],
+        trim_tails=['"""', "if", "print"],
+    )
+    new_code = chunks[0] + new_code
+
+    # cut all functions that are not syntactically correct && not the entry point
+    parts = new_code.split("\ndef ")
+    includes = [parts[0]]
+    for fn in new_code.split("\ndef ")[1:]:
+        if (
+            fn.strip().startswith(entry_point + " ")
+            or fn.strip().startswith(entry_point + "(")
+            or syntax_check("\ndef " + fn)
+        ):
+            includes.append(fn)
+    new_code = "\ndef ".join(includes)
+    return new_code.strip()
+
+
+def script(
+    samples: str,
+    eofs: List[str] = [],
+    inplace: bool = False,
+    rm_prefix_lines: str = None,
+    debug_task: str = None,
+    mbpp_version: str = "default",
+):
+    # task_id -> entry_point
+    entry_point = {}
+    dataset = {**get_human_eval_plus(), **get_mbpp_plus(version=mbpp_version)}
+
+    for task_id, problem in dataset.items():
+        entry_point[task_id] = problem["entry_point"]
+
+    # make a new folder with "-sanitized" suffix
+    is_folder = os.path.isdir(samples)
+    target_path = pathlib.Path(samples)
+    if not inplace:
+        if is_folder:
+            new_name = target_path.name + "-sanitized"
+        else:
+            new_name = target_path.name.replace(".jsonl", "-sanitized.jsonl")
+        target_path = target_path.parent / new_name
+    target_path = str(target_path)
+
+    nsan = 0
+    ntotal = 0
+
+    new_solutions = []
+
+    for solution in tqdm(load_solutions(samples)):
+        task_id = solution["task_id"]
+        dbg_identifier = solution["_identifier"]
+        if debug_task is not None and task_id != debug_task:
+            continue
+
+        ntotal += 1
+        if "solution" in solution:
+            old_code = solution["solution"]
+        else:
+            assert "completion" in solution
+            old_code = dataset[task_id]["prompt"] + "\n" + solution["completion"]
+
+        old_code = old_code.strip()
+
+        new_code = sanitize(
+            old_code=old_code,
+            entry_point=entry_point[task_id],
+            rm_prefix_lines=rm_prefix_lines,
+            eofs=eofs,
+        ).strip()
+
+        # if changed, print the message
+        if new_code != old_code:
+            msg = "Sanitized: " + dbg_identifier
+            if is_folder:
+                msg += " -> " + dbg_identifier.replace(samples, target_path)
+            print(msg)
+            nsan += 1
+
+        new_solutions.append({"task_id": task_id, "solution": new_code})
+
+    if is_folder:
+        write_directory(target_path, new_solutions)
+    else:
+        write_jsonl(target_path, new_solutions)
+
+    if nsan > 0:
+        print(f"Sanitized {nsan} out of {ntotal} files.")
+    else:
+        print(f"All files seems valid -- no files are sanitized.")
+    print(f"Check the sanitized files at {target_path}")
+
+
+def main():
+    from fire import Fire
+
+    Fire(script)
+
+
+if __name__ == "__main__":
+    main()

opencompass/datasets/evalplus/perf/profile.py

@@ -0,0 +1,183 @@
+import time
+from concurrent.futures import ProcessPoolExecutor
+from multiprocessing import Process, Value, cpu_count
+from platform import system
+from time import perf_counter
+from traceback import format_exc
+from typing import Any, Callable, List, Optional
+
+import psutil
+from cirron import Collector
+
+from evalplus.config import PERF_PROFILE_ROUNDS, PERF_RAM_GB_PER_PROC
+from evalplus.eval.utils import (
+    TimeoutException,
+    create_tempdir,
+    reliability_guard,
+    swallow_io,
+    time_limit,
+)
+
+
+def get_max_ram_gb():
+    total_ram = psutil.virtual_memory().total
+    return total_ram / (1024**3)
+
+
+def default_parallelism(divisor=4):
+    return max(1, max(cpu_count(), get_max_ram_gb() // PERF_RAM_GB_PER_PROC) // divisor)
+
+
+def simple_test_profiler():
+    # assert linux
+    assert system() == "Linux", "EvalPerf requires Linux's perf_event_open"
+    try:
+        with Collector():
+            pass
+    except Exception as e:
+        print("It seems your system does not support instruction counting.")
+        print("Try this on Linux:")
+        print("   sudo sh -c 'echo 0 > /proc/sys/kernel/perf_event_paranoid'   ")
+        print("Also check more info at: https://github.com/s7nfo/Cirron")
+        print("Re-raising the original exception...")
+        raise e
+
+
+def are_profiles_broken(profiles) -> bool:
+    return not all(isinstance(profile, (float, int)) for profile in profiles)
+
+
+def physical_runtime_profiler(function, test_inputs) -> float:
+    start = perf_counter()
+    for test_input in test_inputs:
+        function(*test_input)
+    return perf_counter() - start
+
+
+def num_instruction_profiler(function, test_inputs) -> int:
+    with Collector() as c:
+        for test_input in test_inputs:
+            function(*test_input)
+    return int(c.counters.instruction_count)
+
+
+_STAT_NONE = 0
+_STAT_START = 1
+_STAT_SUCC = 2
+_STAT_ERROR = 3
+
+
+def get_instruction_count_shared_mem(
+    profiler: Callable,
+    func_code: str,
+    entry_point: str,
+    test_inputs: List[Any],
+    timeout_second_per_test: float,
+    memory_bound_gb: int,
+    warmup_inputs: Optional[List[Any]],
+    # shared memory
+    compute_cost,  # Value("d", 0.0),
+    progress,  # Value("i", 0),
+) -> Optional[float]:
+
+    error = None
+
+    with create_tempdir():
+        # These system calls are needed when cleaning up tempdir.
+        import os
+        import shutil
+
+        rmtree = shutil.rmtree
+        rmdir = os.rmdir
+        chdir = os.chdir
+
+        # Disable functionalities that can make destructive changes to the test.
+        maximum_memory_bytes = memory_bound_gb * 1024 * 1024 * 1024
+        reliability_guard(maximum_memory_bytes=maximum_memory_bytes)
+        exec_globals = {}
+
+        # run (eval) the func def
+        exec(func_code, exec_globals)
+        fn = exec_globals[entry_point]
+
+        # warmup the function
+        if warmup_inputs:
+            for _ in range(3):
+                fn(*warmup_inputs)
+
+        progress.value = _STAT_START
+        try:  # run the function
+            with time_limit(timeout_second_per_test):
+                with swallow_io():
+                    compute_cost.value = profiler(fn, test_inputs)
+                    progress.value = _STAT_SUCC
+        except TimeoutException:
+            print("[Warning] Profiling hits TimeoutException")
+        except MemoryError:
+            print("[Warning] Profiling hits MemoryError")
+        except:
+            print("[CRITICAL] ! Unknown exception during profiling !")
+            error = format_exc()
+            print(error)
+
+        if progress.value != _STAT_SUCC:
+            progress.value = _STAT_ERROR
+
+        # Needed for cleaning up.
+        shutil.rmtree = rmtree
+        os.rmdir = rmdir
+        os.chdir = chdir
+
+
+def profile(
+    func_code: str,
+    entry_point: str,
+    test_inputs: List[Any],
+    timeout_second_per_test: float,
+    memory_bound_gb: int = PERF_RAM_GB_PER_PROC,
+    profile_rounds: int = PERF_PROFILE_ROUNDS,
+    profiler: Callable = num_instruction_profiler,
+    warmup_inputs: Optional[List[Any]] = None,  # multiple inputs
+) -> List[int | float | str]:
+    """Profile the func_code against certain input tests.
+    The function code is assumed to be correct and if a string is returned, it is an error message.
+    """
+    timeout = timeout_second_per_test * len(test_inputs) * profile_rounds
+
+    def _run():
+        compute_cost = Value("d", 0.0)
+        progress = Value("i", _STAT_NONE)
+
+        p = Process(
+            target=get_instruction_count_shared_mem,
+            args=(
+                profiler,
+                func_code,
+                entry_point,
+                test_inputs,
+                timeout_second_per_test,
+                memory_bound_gb,
+                warmup_inputs,
+                # shared memory
+                compute_cost,
+                progress,
+            ),
+        )
+        p.start()
+        p.join(timeout=timeout + 1)
+        if p.is_alive():
+            p.terminate()
+            time.sleep(0.1)
+
+        if p.is_alive():
+            p.kill()
+            time.sleep(0.1)
+
+        if progress.value == _STAT_SUCC:
+            return compute_cost.value
+        elif progress.value == _STAT_NONE:
+            return "PROFILING DID NOT START"
+        elif progress.value == _STAT_ERROR:
+            return "SOLUTION ERROR ENCOUNTERED WHILE PROFILING"
+
+    return [_run() for _ in range(profile_rounds)]

opencompass/datasets/evalplus/perf/sampling.py

@@ -0,0 +1,295 @@
+import json
+import os
+import re
+from concurrent.futures import ProcessPoolExecutor
+from copy import deepcopy
+from traceback import format_exc
+from typing import Any, List, Optional, Tuple
+
+from pympler.asizeof import asizeof
+from rich.syntax import Syntax
+from termcolor import colored
+
+from evalplus.config import PERF_CURATE_TIMEOUT_SECOND, PERF_RAM_GB_PER_PROC
+from evalplus.data import get_human_eval_plus, get_mbpp_plus
+from evalplus.data.mbpp import mbpp_serialize_inputs
+from evalplus.eval.utils import TimeoutException, reliability_guard, time_limit
+from evalplus.sanitize import syntax_check
+from evalplus.utils import progress
+
+
+# this is more of a hack... rather than a "verified" implementation
+def insert_contract(entry_point: str, code: str, contract: str):
+    # why is this so complicated? because the contract might be mis-indented...
+    def get_first_indent_size(source, body_char_start_idx):
+        assert source.strip()
+        indent_size = 0
+        while source[body_char_start_idx - indent_size - 1] == " ":
+            indent_size += 1
+        return indent_size
+
+    code = code.replace("\t", " " * 4)
+    contract = contract.replace("\t", " " * 4)
+
+    lines = [line for line in code.split("\n") if line.strip()]
+    fn_def_line = [line for line in lines if line.startswith(f"def {entry_point}")][0]
+    def_line_idx = lines.index(fn_def_line)
+    body_start_idx = code.index(code.split(fn_def_line)[1].lstrip())
+
+    source_indent: int = get_first_indent_size(code, body_start_idx)
+    contract_indent: int = get_first_indent_size(
+        contract, len(contract) - len(contract.lstrip())
+    )
+    return "\n".join(
+        lines[: def_line_idx + 1]
+        + [
+            " " * max(0, source_indent - contract_indent) + cline
+            for cline in contract.split("\n")
+            if cline
+        ]
+        + [
+            " " * max(0, contract_indent - source_indent) + sline
+            for sline in lines[def_line_idx + 1 :]
+            if sline
+        ]
+    )
+
+
+def post_process(text: str) -> Optional[str]:
+    """Post-process the LLM generated text to make it valid."""
+    if "\n```" not in text:
+        return None
+
+    # split ```python3 or ```python
+    text = re.split(r"\n```python3?\n", text)[1]
+    text = text.split("\n```")[0].strip()
+
+    # perform syntax check
+    if not syntax_check(text):
+        print(colored("⚠️ Syntax check failed for the code below:", "red"))
+        print(text[:256], "..." if len(text) > 256 else "")
+        return None
+
+    return text
+
+
+# returns:
+# 1. generated and validated (by the contract) inputs
+# 2. whether the generator stops in a well-defined manner
+#    -- if False, we might want to try another generator
+def sample_one_input(
+    ref_code_with_contract: str,
+    entry_point: str,
+    generator_code: str,
+    timeout_second: float = PERF_CURATE_TIMEOUT_SECOND + 1,
+) -> Tuple[List[Any], bool]:
+    # These system calls are needed when cleaning up tempdir.
+    import os
+    import shutil
+
+    rmtree = shutil.rmtree
+    rmdir = os.rmdir
+    chdir = os.chdir
+    # Disable functionalities that can make destructive changes to the test.
+    # :imit memory usages.
+    maximum_memory_bytes = PERF_RAM_GB_PER_PROC * 1024 * 1024 * 1024
+    reliability_guard(maximum_memory_bytes=maximum_memory_bytes)
+    exec_globals = {}
+
+    # eval the func def with contract
+    exec(ref_code_with_contract, exec_globals)
+    fn = exec_globals[entry_point]
+
+    # eval the generator
+    generator_code = "from typing import *\n" + generator_code
+    try:
+        exec(generator_code, exec_globals)
+        generator = exec_globals["perf_input_gen"]
+    except Exception:
+        print(colored(f"⚠️ [GEN EVAL] Exception ~ {entry_point}:", "red"))
+        print(colored(format_exc(), "red"))
+        return [], False
+
+    well_defined_exit = True
+    return_inputs = []
+
+    for fac in range(1, 27):
+        scale = 2**fac
+        print(f"[INPUT GEN] scale=2**{fac}")
+        try:
+            with time_limit(timeout_second):
+                test_input = generator(scale)
+            if not isinstance(test_input, tuple):
+                test_input = (test_input,)
+            # integers should stay in the range of 64-bit
+            if any(
+                isinstance(arg, int) and not (-(2**63) <= arg < 2**63)
+                for arg in test_input
+            ):
+                print(colored(f"[INPUT GEN] Int overflow against 64bit", "yellow"))
+                break
+            # hack list integer
+            if isinstance(test_input[0], list) and any(
+                not (-(2**63) <= v < 2**63)
+                for v in test_input[0]
+                if isinstance(v, int)
+            ):
+                print(colored(f"[INPUT GEN] Int overflow against 64bit", "yellow"))
+                break
+            # stop here if the input is of 64M.
+            INPUT_LIMIT_MB = 64
+            if asizeof(test_input) > 1024 * 1024 * INPUT_LIMIT_MB:
+                print(colored(f"[INPUT GEN] Size > {INPUT_LIMIT_MB}MB", "yellow"))
+                break
+        except TimeoutException:
+            print(colored(f"[INPUT GEN] TimeoutException at scale=2**{fac}", "yellow"))
+            break
+        except MemoryError:
+            print(colored(f"[INPUT GEN] MemoryError at scale=2**{fac}", "yellow"))
+            break
+        except Exception:
+            print(colored(f"⚠️ [INPUT GEN] Exception at scale=2**{fac}", "red"))
+            print(colored(format_exc(), "red"))
+            well_defined_exit = False
+            break
+
+        try:
+            with time_limit(timeout_second):
+                # deepcopy in case fn modifies the input
+                fn(*deepcopy(test_input))
+            return_inputs = [test_input]  # only keep on input
+        except TimeoutException:
+            print(colored(f"[Testing] Timeout at scale=2**{fac}", "yellow"))
+            break
+        except MemoryError:
+            print(colored(f"[Testing] MemoryError at scale=2**{fac}", "yellow"))
+            break
+        except Exception:
+            print(colored(f"⚠️ [Testing] Exception ~ {entry_point}", "red"))
+            print(colored(format_exc(), "red"))
+            well_defined_exit = False
+            break
+
+    # Needed for cleaning up.
+    shutil.rmtree = rmtree
+    os.rmdir = rmdir
+    os.chdir = chdir
+
+    return return_inputs, well_defined_exit
+
+
+def main(input: str, output: str):
+    """In the synthesizer file, each line includes a set of input generators for a task.
+    The goal of this script is to use these generators to sample inputs for each task.
+    The generated inputs are expected to be valid.
+    """
+    assert output.endswith(".jsonl"), "output must be a .jsonl file"
+
+    id2task = {}
+    for task_id, item in get_human_eval_plus().items():
+        id2task[task_id] = item
+
+    for task_id, item in get_mbpp_plus().items():
+        id2task[task_id] = item
+
+    # loading the synthesizers
+    with open(input, "r") as f:
+        synthesizers = [json.loads(l) for l in f]
+
+    n_total = 0
+    n_parsed = 0
+    n_dedup = 0
+
+    for item in synthesizers:
+        item["synthesizers"] = [post_process(s) for s in item["synthesizers"]]
+        n_total += len(item["synthesizers"])
+        item["synthesizers"] = [s for s in item["synthesizers"] if s is not None]
+        n_parsed += len(item["synthesizers"])
+
+        dedup_set = set()
+        for s in item["synthesizers"]:
+            dedup_set.add(
+                "\n".join(
+                    [l for l in s.splitlines() if l.strip() and not l.startswith("#")]
+                )
+            )
+        item["synthesizers"] = list(dedup_set)
+        n_dedup += len(item["synthesizers"])
+
+    print(
+        colored(
+            f"#Total {n_total} with {n_parsed} parsed => {100 * (1 - n_parsed / n_total) :.1f}% syntax err",
+            "green",
+        )
+    )
+
+    print(
+        colored(
+            f"#Parsed {n_parsed} with {n_dedup} dedup => {100 * (1 - n_dedup / n_parsed) :.1f}% duplicate",
+            "green",
+        )
+    )
+
+    # resume mode check finished tasks
+    finished_tasks = set()
+    if os.path.isfile(output):
+        with open(output, "r") as f:
+            for l in f:
+                item = json.loads(l)
+                finished_tasks.add(item["task_id"])
+
+    print("Resumed finished tasks:", finished_tasks)
+    with open(output, "ab+") as f:
+        with progress() as p:
+            for item in p.track(synthesizers):
+                task_id = item["task_id"]
+                entry_point = id2task[task_id]["entry_point"]
+                if task_id in finished_tasks:
+                    p.console.print(f"{task_id}: {entry_point} ~ Resumed")
+                    continue
+
+                ref_code_with_contract = insert_contract(
+                    entry_point, item["ref_code"], id2task[task_id]["contract"]
+                )
+                p.console.print(f"{task_id}: PE input generation...")
+                p.console.print(Syntax(ref_code_with_contract.strip(), "python"))
+
+                results = []
+                for i, generator_code in enumerate(item["synthesizers"]):
+                    p.console.print(
+                        f"Using generator {i+1}/{len(item['synthesizers'])}:"
+                    )
+                    p.console.print(Syntax(generator_code, "python"))
+                    args = (
+                        ref_code_with_contract,
+                        entry_point,
+                        generator_code,
+                    )
+                    with ProcessPoolExecutor(max_workers=1) as executor:
+                        tmp_results, status = executor.submit(
+                            sample_one_input, *args
+                        ).result()
+
+                    results.extend(tmp_results)
+
+                    # if the func returns in a well-defined manner, we can stop here.
+                    if status:
+                        break
+
+                p.console.print("Serializing and storing results...")
+
+                if "Mbpp/" in task_id:
+                    results = mbpp_serialize_inputs(task_id, results)
+
+                to_write = {"task_id": item["task_id"], "inputs": results}
+                to_write = (json.dumps(to_write) + "\n").encode("utf-8")
+
+                # task_id => list of inputs
+                f.write(to_write)
+                f.flush()
+
+
+if __name__ == "__main__":
+    import fire
+
+    fire.Fire(main)

opencompass/datasets/evalplus/perf/sas.py

@@ -0,0 +1,168 @@
+"""This file implements the "Synthesizing an Synthesizer" idea using OpenAI API.
+Specifically, for each HumanEval+ and MBPP+ task, we generate n test input synthesizers
+by querying a vLLM server (https://docs.vllm.ai/en/latest/).
+"""
+
+import json
+from typing import Optional
+
+import openai
+from tqdm import tqdm
+
+from evalplus.data import get_human_eval_plus, get_mbpp_plus
+
+
+def fewshot_cot(
+    task_id,
+    client: openai.OpenAI,
+    entry_point: str,
+    code: str,
+    model: str,
+    n: int = 1,
+    max_tokens: int = 2048,
+):
+    responses = client.completions.create(
+        model=model,
+        prompt=f'''\
+You are an AI programming assistant, proficient in analyzing and generating Python code. \
+You are going to produce a self-contained Python function to generate a large input for a given function, \
+to test its performance at scale.
+### Instruction:
+Generate a `perf_input_gen(scale: int)` function to produce a "large" input to exercise the performance of the `add` function:
+```python3
+def add(x: int, y: int):
+    """Add two numbers x and y
+    >>> add(2, 3)
+    5
+    >>> add(5, 7)
+    12
+    """
+    return x + y
+```
+### Response:
+Analysis:
+1. Input format: two integers `x` and `y`
+2. Is this task O(1) solvable? Yes
+### Instruction:
+Generate a `perf_input_gen(scale: int)` function to produce a "large" input to exercise the performance of the `prime_num` function:
+```python3
+"""
+Write a function to check if a number is prime or not.
+assert prime_num(2) == True
+"""
+import math
+def prime_num(num):
+    if num < 2: return False
+    for i in range(2, math.isqrt(num)):
+        if num % i == 0:
+            return False
+    return True
+```
+### Response:
+Analysis:
+1. Input format: An integer `n`
+2. Is this task O(1) solvable? No
+3. Time complexity: O(n)
+4. Space complexity: O(1)
+5. What kind of input can exercise its performance? Large prime numbers
+```python3
+# Can reuse the `prime_num` function
+# `scale` is a rough estimate of the input size -- larger `scale` means larger input
+# use case: prime_num(*perf_input_gen(scale))
+import random
+def perf_input_gen(scale: int):
+    for i in range(scale, 2, -1):
+        if prime_num(i):
+            return (i,)
+    return (2,)
+```
+### Instruction:
+Generate a `perf_input_gen(scale: int)` function to produce a "large" input to exercise the performance of the `{entry_point}` function:
+```python3
+{code}
+```
+### Response:
+Analysis:
+1. Input format: ''',
+        n=n,
+        stop=["\n```\n", "\n2. Is this task O(1) solvable? Yes"],
+        max_tokens=max_tokens,
+        temperature=0.2,
+    )
+
+    # warn if any response is out of context
+    for r in responses.choices:
+        if r.finish_reason == "length":
+            print(f"Warning: response is too long for {task_id}")
+
+    return [r.text for r in responses.choices]
+
+
+def main(
+    output: str,  # output file
+    n: int = 16,  # sample size and batch size
+    model: Optional[str] = "TheBloke/deepseek-coder-33B-instruct-AWQ",
+    port: str = 8088,
+):
+    assert output.endswith(".jsonl"), "output must be a .jsonl file"
+
+    base_url = f"http://localhost:{port}/v1"
+    print(f"Trying to query vLLM model: {model} at {base_url}")
+    print(f"Note: To use SaS, you need to first set up a vLLM server for {model}")
+    print(f"For example:")
+    print(
+        f"""python -m vllm.entrypoints.openai.api_server \\
+--model "{model}" \\
+--port {port} \\
+--tensor-parallel-size 2 \\
+--max-num-seqs 16 \\
+--gpu-memory-utilization 1.0"""
+    )
+
+    # "task_id" -> { "task_id", "entry_point", "ref_code", }
+    tasks = {}
+    for task_id, item in get_human_eval_plus().items():
+        tasks[task_id] = {
+            "task_id": task_id,
+            "entry_point": item["entry_point"],
+            "ref_code": item["prompt"] + item["canonical_solution"],
+        }
+
+    for task_id, item in get_mbpp_plus().items():
+        tasks[task_id] = {
+            "task_id": task_id,
+            "entry_point": item["entry_point"],
+            "ref_code": item["prompt"] + item["canonical_solution"],
+        }
+
+    # Using vLLM as a backend, please make sure that a vLLM server is available first.
+    # vLLM document: https://docs.vllm.ai/en/latest/
+    client = openai.OpenAI(api_key="none", base_url=base_url)
+
+    with open(output, "w") as f:
+        for task_id, item in tqdm(tasks.items(), total=len(tasks)):
+            responses = fewshot_cot(
+                task_id=task_id,
+                client=client,
+                entry_point=item["entry_point"],
+                code=item["ref_code"],
+                model=model,
+                n=n,
+            )
+            f.write(
+                json.dumps(
+                    {
+                        "task_id": task_id,
+                        "ref_code": item["ref_code"],
+                        "synthesizers": responses,
+                    }
+                )
+                + "\n"
+            )
+            f.flush()
+
+
+if __name__ == "__main__":
+    import fire
+
+    fire.Fire(main)

opencompass/datasets/evalplus/perf/select_pe_inputs.py

@@ -0,0 +1,126 @@
+"""Select the most performance-exercising inputs from pe_inputs obtained from `sampling.py`.
+"""
+
+import json
+from statistics import median
+
+from tqdm import tqdm
+
+from evalplus.config import PERF_CURATE_TIMEOUT_SECOND
+from evalplus.data import get_human_eval_plus, get_mbpp_plus
+from evalplus.data.mbpp import mbpp_deserialize_inputs, mbpp_serialize_inputs
+from evalplus.perf.profile import are_profiles_broken, profile
+
+
+def script(solutions: str, output_profiled_solutions: str, pe_inputs: str = None):
+    assert solutions.endswith(".jsonl")
+    assert pe_inputs is None or pe_inputs.endswith(".jsonl")
+    assert output_profiled_solutions.endswith(".jsonl")
+
+    evalplus = get_human_eval_plus(noextreme=True)
+    mbppplus = get_mbpp_plus(noextreme=True)
+    tasks = {**evalplus, **mbppplus}
+
+    # assume each line's format is: {
+    # "task_id": task's id,
+    # "inputs": a list of inputs,
+    inputs_dict = None
+
+    if pe_inputs is not None:
+        print("Loading performance-exercising inputs...")
+        with open(pe_inputs, "r") as f:
+            inputs_dict = {
+                task["task_id"]: task["inputs"] for l in f for task in [json.loads(l)]
+            }
+
+    # Notably, the solutions are already validated and cleaned.
+    with open(solutions, "r") as f:
+        solutions = {}
+        for l in f:
+            solution = json.loads(l)
+            solutions[solution["task_id"]] = solution["solution"]
+
+    for task_id, task in tqdm(tasks.items()):
+        if inputs_dict:
+            inputs = (
+                mbpp_deserialize_inputs(task_id, inputs_dict[task_id])
+                if "Mbpp/" in task_id
+                else inputs_dict[task_id]
+            )
+        else:
+            inputs = task["base_input"] + list(task["plus_input"])
+
+        input_costs = []
+
+        if task_id.startswith("HumanEval"):
+            canonical_solution = task["prompt"] + task["canonical_solution"]
+        else:
+            canonical_solution = task["canonical_solution"]
+
+        for inp in inputs:
+            costs = profile(
+                canonical_solution,
+                task["entry_point"],
+                [inp],
+                timeout_second_per_test=PERF_CURATE_TIMEOUT_SECOND,
+            )
+            if are_profiles_broken(costs):
+                continue
+            input_costs.append((median(costs), inp))
+        input_costs.sort(reverse=True, key=lambda x: x[0])
+
+        for _, pe_input in input_costs:
+            solution_costs = []
+
+            for solution in solutions[task_id]:
+                costs = profile(
+                    solution,
+                    task["entry_point"],
+                    [pe_input],
+                    timeout_second_per_test=PERF_CURATE_TIMEOUT_SECOND,
+                )
+                if not are_profiles_broken(costs):
+                    solution_costs.append(costs)
+                    continue
+
+                # stop once we find the first also the most performance-exercising input
+                break
+
+            # This means no timeouts happen for the input, so we use it.
+            if len(solution_costs) == len(solutions[task_id]):
+                break
+
+        # If no satisfied input found, we don't save any profiled data.
+        if len(input_costs) == 0 or len(solution_costs) != len(solutions[task_id]):
+            print(f"Skipping {task_id}...")
+            pe_input = None
+            solution_costs = None
+        else:
+            pe_input = (
+                mbpp_serialize_inputs(task_id, [pe_input])
+                if task_id.startswith("Mbpp/")
+                else [pe_input]
+            )
+
+        with open(output_profiled_solutions, "a") as f:
+            f.write(
+                json.dumps(
+                    {
+                        "task_id": task_id,
+                        "pe_input": pe_input,
+                        "solutions": solutions[task_id],
+                        "counter_profile": solution_costs,
+                    }
+                )
+                + "\n"
+            )
+
+
+def main():
+    from fire import Fire
+
+    Fire(script)
+
+
+if __name__ == "__main__":
+    main()