# Config file reference

Every field of `run.yaml`, with its type and default.

The same fields make up a `RunConfig` in Python, and the reproducibility config that every run writes has this config under its `config` key. Unknown fields are errors, so a typo never goes unnoticed.

## Example

A short config, and the same config with every default written out, the way it appears in the reproducibility config:

```yaml
dataset:
  path: data.xlsx
  sheet: test
mapping:
  input: text
  context:
  - language
  expected: label
  id: id
prompt:
  system: You are a careful annotator.
  user: 'Classify the sentiment of this {language} text: {text}'
models:
- openrouter/nvidia/nemotron-3-super-120b-a12b:free
metrics:
- name: accuracy
  labels:
  - positive
  - negative
seed: 42
```

```yaml
dataset:
  path: data.xlsx
  sheet: test
mapping:
  input:
  - text
  context:
  - language
  expected: label
  id: id
  answer_delimiter: null
prompt:
  user: 'Classify the sentiment of this {language} text: {text}'
  system: You are a careful annotator.
models:
- id: openrouter/nvidia/nemotron-3-super-120b-a12b:free
  label: null
  temperature: 0.0
  max_tokens: null
  top_p: null
  seed: null
  requests_per_minute: null
  concurrency: null
  params: {}
metrics:
- name: accuracy
  label: null
  params:
    labels:
    - positive
    - negative
seed: 42
concurrency: 4
retry:
  max_attempts: 5
  base_delay: 2.0
  max_delay: 60.0
  timeout: 120.0
```

## Top level

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `dataset` | object, see [`dataset`](#dataset) | none | Where to load the data from. Optional when the data is passed to `run` directly. |
| `mapping` | object, see [`mapping`](#mapping) | required | Which columns are inputs, context, the answer key, and the row ID. |
| `prompt` | object, see [`prompt`](#prompt) | required | The prompt template: a string, or an object with `user` and `system`. |
| `models` | list, see [`models`](#models) | required | Models to evaluate, as IDs or objects with options. |
| `metrics` | list, see [`metrics`](#metrics) | `[]` | Metrics to compute, as names or objects with options. Empty collects outputs only. |
| `seed` | integer | none | Default seed for models that do not set their own. |
| `concurrency` | integer, > 0 | `4` | Parallel requests per model (rate limits still apply). |
| `retry` | object, see [`retry`](#retry) | see below | Retry and timeout settings. |

## dataset

Where the data comes from. Optional when the dataset is passed in directly.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `path` | string | required | Data file. A relative path is resolved against the config file's folder. |
| `sheet` | string or integer | none | Excel sheet name or zero-based index. Defaults to the first sheet. |

## mapping

Which columns are inputs, extra context, the expected answer, and the row ID.

`input` and `context` take one column name or a list.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `input` | list of strings | required | Columns available to the prompt template. One name or a list. |
| `context` | list of strings | `[]` | More columns available to the prompt, such as metadata. |
| `expected` | string | none | The answer key that outputs are scored against. Never available to the prompt. |
| `id` | string | none | A stable row ID. Defaults to the 1-based row number. |
| `answer_delimiter` | string | none | Split the expected cell on this string to allow several valid answers, e.g. `"\|"`. |

## prompt

A user prompt (and optional system prompt) with column placeholders.

A plain string is the user message: `prompt: "Translate: {text}"`.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `user` | string | required | The user message, with `{column}` placeholders. |
| `system` | string | none | An optional system message, with the same placeholders. |

## models

A model to evaluate, addressed by its LiteLLM ID (`provider/model`).

Each entry is a model ID string, or an object with options.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `id` | string | required | LiteLLM model ID with its provider prefix, e.g. `gemini/gemini-3.5-flash`. |
| `label` | string | none | Display name in tables. Defaults to the model ID without the provider prefix. |
| `temperature` | number, ≥ 0 | `0.0` | Sampling temperature. 0 gives the most repeatable outputs. |
| `max_tokens` | integer, > 0 | none | Cap on output tokens per response. |
| `top_p` | number, > 0, ≤ 1 | none | Nucleus sampling cutoff. |
| `seed` | integer | none | Overrides the run-level `seed` for this model. |
| `requests_per_minute` | number, > 0 | none | Override the default rate limit for this model. |
| `concurrency` | integer, > 0 | none | Override the run-level number of parallel requests for this model. |
| `params` | mapping | `{}` | Extra provider parameters passed through to LiteLLM. |

## metrics

A metric name plus its options, e.g. `{"name": "exact_match", "ignore_case": false}`.

Each entry is a metric name, or an object with options. Options can sit next to `name` (`{name: f1, average: macro}`) or under `params`. See the [metrics reference](https://paperlab.space/docs/reference/metrics.md) for each metric's options.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `name` | string | required | A registered metric name; `paperlab-space metrics` lists them. |
| `label` | string | none | Column name in outputs; needed to use the same metric twice with different options. |
| `params` | mapping | `{}` | Metric options such as `labels` or `ignore_case`. In YAML they can also sit next to `name`. |

## retry

How failed requests are retried, and how long to wait for a response.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `max_attempts` | integer, ≥ 1 | `5` | Tries per row, including the first. |
| `base_delay` | number, ≥ 0 | `2.0` | Seconds before the first retry; doubles on each attempt. |
| `max_delay` | number, ≥ 0 | `60.0` | Upper bound on the wait between retries, in seconds. |
| `timeout` | number, > 0 | `120.0` | Seconds to wait for one model response. |