name: 🐞 Bug report
description: Create a report to help us improve
labels: ["bug"]
title: "[Bug] "
body:
  - type: markdown
    attributes:
      value: |
        For general questions or idea discussions, please post it to our [**Forum**](https://github.com/open-compass/opencompass/discussions).
        If you have already identified the reason, we strongly appreciate you creating a new PR according to [the tutorial](https://opencompass.readthedocs.io/en/master/community/CONTRIBUTING.html)!
        If you need our help, please fill in the following form to help us to identify the bug.

  - type: checkboxes
    attributes:
      label: Prerequisite
      description: Please check the following items before creating a new issue.
      options:
      - label: I have searched [Issues](https://github.com/open-compass/opencompass/issues/) and [Discussions](https://github.com/open-compass/opencompass/discussions) but cannot get the expected help.
        required: true
      - label: The bug has not been fixed in the [latest version](https://github.com/open-compass/opencompass).
        required: true

  - type: dropdown
    id: task
    attributes:
      label: Type
      description: The problem arises when
      options:
        - I'm evaluating with the officially supported tasks/models/datasets.
        - I have modified the code (config is not considered code), or I'm working on my own tasks/models/datasets.
    validations:
      required: true

  - type: textarea
    id: environment
    validations:
      required: true
    attributes:
      label: Environment
      description: |
        Please run `python -c "import opencompass.utils;import pprint;pprint.pprint(dict(opencompass.utils.collect_env()))"` to collect necessary environment information and paste it here.
      placeholder: |
        ```python
        # The output the above command
        ```

  - type: textarea
    attributes:
      label: Reproduces the problem - code/configuration sample
      description: |
        Please provide a code or configuration sample that reproduces the problem you ran into. It can be a Colab link or just a code snippet.
      placeholder: |
        ```python
        # Sample code to reproduce the problem
        ```
    validations:
      required: true

  - type: textarea
    attributes:
      label: Reproduces the problem - command or script
      description: |
        What command or script did you run?
      placeholder: |
        ```shell
        The command or script you run.
        ```
    validations:
      required: true

  - type: textarea
    attributes:
      label: Reproduces the problem - error message
      description: |
        Please provide the error message or logs you got, with the full traceback.

        Tip: You can attach images or log files by dragging them into the text area..
      placeholder: |
        ```
        The error message or logs you got, with the full traceback.
        ```
    validations:
      required: true

  - type: textarea
    id: other
    attributes:
      label: Other information
      description: |
        Tell us anything else you think we should know.

        1. What's your expected result?
        2. What dataset did you use?
        3. What do you think might be the reason?