Reference

Config file reference

Every field of run.yaml, with its type and default.

The same fields make up a RunConfig in Python, and the reproducibility config that every run writes has this config under its config key. Unknown fields are errors, so a typo never goes unnoticed.

Example

A short config, and the same config with every default written out, the way it appears in the reproducibility config:

YAML

dataset:
  path: data.xlsx
  sheet: test
mapping:
  input: text
  context:
  - language
  expected: label
  id: id
prompt:
  system: You are a careful annotator.
  user: 'Classify the sentiment of this {language} text: {text}'
models:
- openrouter/nvidia/nemotron-3-super-120b-a12b:free
metrics:
- name: accuracy
  labels:
  - positive
  - negative
seed: 42

YAML

dataset:
  path: data.xlsx
  sheet: test
mapping:
  input:
  - text
  context:
  - language
  expected: label
  id: id
  answer_delimiter: null
prompt:
  user: 'Classify the sentiment of this {language} text: {text}'
  system: You are a careful annotator.
models:
- id: openrouter/nvidia/nemotron-3-super-120b-a12b:free
  label: null
  temperature: 0.0
  max_tokens: null
  top_p: null
  seed: null
  requests_per_minute: null
  concurrency: null
  params: {}
metrics:
- name: accuracy
  label: null
  params:
    labels:
    - positive
    - negative
seed: 42
concurrency: 4
retry:
  max_attempts: 5
  base_delay: 2.0
  max_delay: 60.0
  timeout: 120.0

Top level

Field	Type	Default	Description
`dataset`	object, see `dataset`	none	Where to load the data from. Optional when the data is passed to `run` directly.
`mapping`	object, see `mapping`	required	Which columns are inputs, context, the answer key, and the row ID.
`prompt`	object, see `prompt`	required	The prompt template: a string, or an object with `user` and `system`.
`models`	list, see `models`	required	Models to evaluate, as IDs or objects with options.
`metrics`	list, see `metrics`	`[]`	Metrics to compute, as names or objects with options. Empty collects outputs only.
`seed`	integer	none	Default seed for models that do not set their own.
`concurrency`	integer, > 0	`4`	Parallel requests per model (rate limits still apply).
`retry`	object, see `retry`	see below	Retry and timeout settings.

dataset

Where the data comes from. Optional when the dataset is passed in directly.

Field	Type	Default	Description
`path`	string	required	Data file. A relative path is resolved against the config file's folder.
`sheet`	string or integer	none	Excel sheet name or zero-based index. Defaults to the first sheet.

mapping

Which columns are inputs, extra context, the expected answer, and the row ID.

input and context take one column name or a list.

Field	Type	Default	Description
`input`	list of strings	required	Columns available to the prompt template. One name or a list.
`context`	list of strings	`[]`	More columns available to the prompt, such as metadata.
`expected`	string	none	The answer key that outputs are scored against. Never available to the prompt.
`id`	string	none	A stable row ID. Defaults to the 1-based row number.
`answer_delimiter`	string	none	Split the expected cell on this string to allow several valid answers, e.g. `"\|"`.

prompt

A user prompt (and optional system prompt) with column placeholders.

A plain string is the user message: prompt: "Translate: {text}".

Field	Type	Default	Description
`user`	string	required	The user message, with `{column}` placeholders.
`system`	string	none	An optional system message, with the same placeholders.

models

A model to evaluate, addressed by its LiteLLM ID (provider/model).

Each entry is a model ID string, or an object with options.

Field	Type	Default	Description
`id`	string	required	LiteLLM model ID with its provider prefix, e.g. `gemini/gemini-3.5-flash`.
`label`	string	none	Display name in tables. Defaults to the model ID without the provider prefix.
`temperature`	number, ≥ 0	`0.0`	Sampling temperature. 0 gives the most repeatable outputs.
`max_tokens`	integer, > 0	none	Cap on output tokens per response.
`top_p`	number, > 0, ≤ 1	none	Nucleus sampling cutoff.
`seed`	integer	none	Overrides the run-level `seed` for this model.
`requests_per_minute`	number, > 0	none	Override the default rate limit for this model.
`concurrency`	integer, > 0	none	Override the run-level number of parallel requests for this model.
`params`	mapping	`{}`	Extra provider parameters passed through to LiteLLM.

metrics

A metric name plus its options, e.g. {"name": "exact_match", "ignore_case": false}.

Each entry is a metric name, or an object with options. Options can sit next to name ({name: f1, average: macro}) or under params. See the metrics reference for each metric's options.

Field	Type	Default	Description
`name`	string	required	A registered metric name; `paperlab-space metrics` lists them.
`label`	string	none	Column name in outputs; needed to use the same metric twice with different options.
`params`	mapping	`{}`	Metric options such as `labels` or `ignore_case`. In YAML they can also sit next to `name`.

retry

How failed requests are retried, and how long to wait for a response.

Field	Type	Default	Description
`max_attempts`	integer, ≥ 1	`5`	Tries per row, including the first.
`base_delay`	number, ≥ 0	`2.0`	Seconds before the first retry; doubles on each attempt.
`max_delay`	number, ≥ 0	`60.0`	Upper bound on the wait between retries, in seconds.
`timeout`	number, > 0	`120.0`	Seconds to wait for one model response.

Edit this page on GitHub