# Metrics reference

Every built-in metric and its options.

Run `paperlab-space metrics` to list the metrics in your installed version. For how normalisation and label extraction work, see the [metrics guide](https://paperlab.space/docs/guides/metrics.md).

## accuracy

Classification accuracy. With `labels`, a free-form answer that mentions exactly one label counts as that label.

Needs an answer key: yes. Higher is better: yes.

| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `labels` | list of strings | none | Class labels. A free-form answer that mentions exactly one of them counts as it. |
| `ignore_case` | boolean | `true` | Compare case-insensitively. |
| `trim_punctuation` | boolean | `true` | Strip punctuation and quotes from the start and end. |
| `ignore_punctuation` | boolean | `false` | Remove all punctuation, including inside the text. |
| `nfkc` | boolean | `true` | Apply Unicode NFKC, so full-width characters equal their ASCII forms. |
| `count_failed` | boolean | `true` | Score failed requests as wrong. `false` leaves them out. |

## exact_match

Share of rows where the output equals a reference answer after normalisation.

Needs an answer key: yes. Higher is better: yes.

| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `ignore_case` | boolean | `true` | Compare case-insensitively. |
| `trim_punctuation` | boolean | `true` | Strip punctuation and quotes from the start and end. |
| `ignore_punctuation` | boolean | `false` | Remove all punctuation, including inside the text. |
| `nfkc` | boolean | `true` | Apply Unicode NFKC, so full-width characters equal their ASCII forms. |
| `count_failed` | boolean | `true` | Score failed requests as wrong. `false` leaves them out. |

## f1

Classification precision, recall, and F1 over the gold labels (macro-averaged by default).

Needs an answer key: yes. Higher is better: yes.

| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `average` | `macro`, `micro`, `weighted` | `"macro"` | How per-class F1 scores are combined. |
| `labels` | list of strings | none | Class labels. A free-form answer that mentions exactly one of them counts as it. |
| `ignore_case` | boolean | `true` | Compare case-insensitively. |
| `trim_punctuation` | boolean | `true` | Strip punctuation and quotes from the start and end. |
| `ignore_punctuation` | boolean | `false` | Remove all punctuation, including inside the text. |
| `nfkc` | boolean | `true` | Apply Unicode NFKC, so full-width characters equal their ASCII forms. |
| `count_failed` | boolean | `true` | Score failed requests as wrong. `false` leaves them out. |

## token_f1

Token-overlap F1 between output and reference (SQuAD-style), averaged over rows. Characters count as tokens in Chinese, Japanese, Thai, and Burmese.

Needs an answer key: yes. Higher is better: yes.

| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `ignore_case` | boolean | `true` | Compare case-insensitively. |
| `ignore_punctuation` | boolean | `true` | Remove all punctuation, including inside the text. |
| `remove_articles` | boolean | `false` | Remove English articles (a, an, the), as in SQuAD. |
| `nfkc` | boolean | `true` | Apply Unicode NFKC, so full-width characters equal their ASCII forms. |
| `count_failed` | boolean | `true` | Score failed requests as wrong. `false` leaves them out. |