Reference

Metrics reference

Every built-in metric and its options.

Run paperlab-space metrics to list the metrics in your installed version. For how normalisation and label extraction work, see the metrics guide.

accuracy

Classification accuracy. With labels, a free-form answer that mentions exactly one label counts as that label.

Needs an answer key: yes. Higher is better: yes.

Option	Type	Default	Description
`labels`	list of strings	none	Class labels. A free-form answer that mentions exactly one of them counts as it.
`ignore_case`	boolean	`true`	Compare case-insensitively.
`trim_punctuation`	boolean	`true`	Strip punctuation and quotes from the start and end.
`ignore_punctuation`	boolean	`false`	Remove all punctuation, including inside the text.
`nfkc`	boolean	`true`	Apply Unicode NFKC, so full-width characters equal their ASCII forms.
`count_failed`	boolean	`true`	Score failed requests as wrong. `false` leaves them out.

Share of rows where the output equals a reference answer after normalisation.

Needs an answer key: yes. Higher is better: yes.

Option	Type	Default	Description
`ignore_case`	boolean	`true`	Compare case-insensitively.
`trim_punctuation`	boolean	`true`	Strip punctuation and quotes from the start and end.
`ignore_punctuation`	boolean	`false`	Remove all punctuation, including inside the text.
`nfkc`	boolean	`true`	Apply Unicode NFKC, so full-width characters equal their ASCII forms.
`count_failed`	boolean	`true`	Score failed requests as wrong. `false` leaves them out.

Classification precision, recall, and F1 over the gold labels (macro-averaged by default).

Needs an answer key: yes. Higher is better: yes.

Option	Type	Default	Description
`average`	`macro`, `micro`, `weighted`	`"macro"`	How per-class F1 scores are combined.
`labels`	list of strings	none	Class labels. A free-form answer that mentions exactly one of them counts as it.
`ignore_case`	boolean	`true`	Compare case-insensitively.
`trim_punctuation`	boolean	`true`	Strip punctuation and quotes from the start and end.
`ignore_punctuation`	boolean	`false`	Remove all punctuation, including inside the text.
`nfkc`	boolean	`true`	Apply Unicode NFKC, so full-width characters equal their ASCII forms.
`count_failed`	boolean	`true`	Score failed requests as wrong. `false` leaves them out.

Token-overlap F1 between output and reference (SQuAD-style), averaged over rows. Characters count as tokens in Chinese, Japanese, Thai, and Burmese.

Needs an answer key: yes. Higher is better: yes.

Option	Type	Default	Description
`ignore_case`	boolean	`true`	Compare case-insensitively.
`ignore_punctuation`	boolean	`true`	Remove all punctuation, including inside the text.
`remove_articles`	boolean	`false`	Remove English articles (a, an, the), as in SQuAD.
`nfkc`	boolean	`true`	Apply Unicode NFKC, so full-width characters equal their ASCII forms.
`count_failed`	boolean	`true`	Score failed requests as wrong. `false` leaves them out.