AISE-Bench Leaderboard

How to Submit

1. Fork this repository.
2. Create a new branch for your submission.
3. Add your submission folder under submissions/<organization>__<model>__<input_config>/.
4. Open a Pull Request with the new submission folder.

Submission Directory Requirements

Each submission directory must contain the metadata and predictions for one model/input configuration pair:

<organization>__<model>__<input_config>/
  metadata.yaml
  predictions.jsonl
  generation_config.json      # optional, recommended
  artifacts/                  # optional logs or prompt notes

Use URL-safe directory names. Replace spaces, slashes, and special characters with hyphens; keep input_config as TEXT.

metadata.yaml

model_name: "My Model"
model_url: https://...

predictions.jsonl

Each line must be one JSON object:

{
  "index": 1,
  "qid": 3,
  "question": "question text",
  "planning_text": [],
  "api_input": {},
  "api_output": {},
  "result_edit": "{\"answer\":\"...\",\"reference\":{\"[1]\":\"...\"}}"
}

The file must contain exactly one line for every item in test.json. qid must match the corresponding item in test.json.