mirror of
https://github.com/open-compass/opencompass.git
synced 2025-05-30 16:03:24 +08:00
feat(text_postprocessors): add parse_bracketed_answer function to extract answers enclosed in brackets
This function is designed to extract option answers enclosed in brackets from text, and replaces the use of first_option_postprocess in the MaritimeBench dataset. This change aims to support the answer format requirements of the MaritimeBench dataset and ensure that model outputs conform to the <option> format.
This commit is contained in:
parent
b0bbd4a96a
commit
a0bac8e604
61
opencompass/configs/datasets/maritimebench/README.md
Normal file
61
opencompass/configs/datasets/maritimebench/README.md
Normal file
@ -0,0 +1,61 @@
|
|||||||
|
## 📘 About MaritimeBench
|
||||||
|
|
||||||
|
**MaritimeBench** 是航运行业首个基于“学科(一级)- 子学科(二级)- 具体考点(三级)”分类体系构建的专业知识评测集。该数据集包含 **1888 道客观选择题**,覆盖以下核心领域:
|
||||||
|
|
||||||
|
- 航海
|
||||||
|
- 轮机
|
||||||
|
- 电子电气员
|
||||||
|
- GMDSS(全球海上遇险与安全系统)
|
||||||
|
- 船员培训
|
||||||
|
|
||||||
|
评测内容涵盖理论知识、操作技能及行业规范,旨在:
|
||||||
|
|
||||||
|
- 提升 AI 模型在航运领域的 **理解与推理能力**
|
||||||
|
- 确保其在关键知识点上的 **准确性与适应性**
|
||||||
|
- 支持航运专业考试、船员培训及资质认证的 **自动化测评**
|
||||||
|
- 优化船舶管理、导航操作、海上通信等场景下的 **智能问答与决策系统**
|
||||||
|
|
||||||
|
MaritimeBench 基于行业权威标准,构建了 **系统、科学的知识评测体系**,全面衡量模型在航运各专业领域的表现,助力其专业化发展。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧪 示例
|
||||||
|
|
||||||
|
请回答单选题。要求只输出选项,不输出解释,将选项放在 `< >` 内,直接输出答案。
|
||||||
|
|
||||||
|
**题目 1:**
|
||||||
|
在船舶主推进动力装置中,传动轴系在运转中承受以下复杂的应力和负荷,但不包括______。
|
||||||
|
选项:
|
||||||
|
A. 电磁力
|
||||||
|
B. 压拉应力
|
||||||
|
C. 弯曲应力
|
||||||
|
D. 扭应力
|
||||||
|
**答:** `<A>`
|
||||||
|
|
||||||
|
**题目 2:**
|
||||||
|
当船舶实行 PMS 检验时,应将 CCS 现行规范中规定的特别检验纳入在 PMS 计划表中,下列应包括______。
|
||||||
|
① 每年应进行的确认性检查项目
|
||||||
|
② 每年应进行的拆检项目
|
||||||
|
③ 5 年内应拆检的项目
|
||||||
|
④ 5 年内应进行的确认性检查项目
|
||||||
|
选项:
|
||||||
|
A. ①④
|
||||||
|
B. ②④
|
||||||
|
C. ①③
|
||||||
|
D. ①②③④
|
||||||
|
**答:** `<C>`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📂 Dataset Links
|
||||||
|
|
||||||
|
- [MaritimeBench on Hugging Face](https://huggingface.co/datasets/Hi-Dolphin/MaritimeBench)
|
||||||
|
- [MaritimeBench on ModelScope](https://modelscope.cn/datasets/HiDolphin/MaritimeBench/summary)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 模型测试结果
|
||||||
|
|
||||||
|
| dataset | version | metric | mode | Qwen2.5-32B |
|
||||||
|
|----- | ----- | ----- | ----- | -----|
|
||||||
|
| maritimebench | 6d56ec | accuracy | gen | 72.99 |
|
@ -1,6 +1,6 @@
|
|||||||
from opencompass.datasets import MaritimeBenchDataset
|
from opencompass.datasets import MaritimeBenchDataset
|
||||||
from opencompass.openicl.icl_prompt_template import PromptTemplate
|
from opencompass.openicl.icl_prompt_template import PromptTemplate
|
||||||
from opencompass.utils.text_postprocessors import first_option_postprocess
|
from opencompass.utils.text_postprocessors import parse_bracketed_answer
|
||||||
from opencompass.openicl.icl_evaluator import AccEvaluator
|
from opencompass.openicl.icl_evaluator import AccEvaluator
|
||||||
from opencompass.openicl.icl_retriever import ZeroRetriever
|
from opencompass.openicl.icl_retriever import ZeroRetriever
|
||||||
from opencompass.openicl.icl_inferencer import GenInferencer
|
from opencompass.openicl.icl_inferencer import GenInferencer
|
||||||
@ -26,7 +26,7 @@ maritimebench_infer_cfg = dict(
|
|||||||
|
|
||||||
maritimebench_eval_cfg = dict(
|
maritimebench_eval_cfg = dict(
|
||||||
evaluator=dict(type=AccEvaluator),
|
evaluator=dict(type=AccEvaluator),
|
||||||
pred_postprocessor=dict(type=first_option_postprocess, options='ABCD')
|
pred_postprocessor=dict(type=parse_bracketed_answer, options='A|B|C|D')
|
||||||
)
|
)
|
||||||
|
|
||||||
maritimebench_datasets = [
|
maritimebench_datasets = [
|
||||||
|
@ -283,3 +283,10 @@ def extract_non_reasoning_content(
|
|||||||
re.DOTALL)
|
re.DOTALL)
|
||||||
non_reasoning_content = reasoning_regex.sub('', text).strip()
|
non_reasoning_content = reasoning_regex.sub('', text).strip()
|
||||||
return non_reasoning_content
|
return non_reasoning_content
|
||||||
|
|
||||||
|
|
||||||
|
def parse_bracketed_answer(text: str, options: str) -> str:
|
||||||
|
match = re.search(rf'<({options})>', text)
|
||||||
|
if match:
|
||||||
|
return match.group(1)
|
||||||
|
return ''
|
||||||
|
Loading…
Reference in New Issue
Block a user