Extend

Custom metrics

Add a metric of your own and use it by name, like the built-in ones.

Write the metric

Subclass Metric, declare its options as __init__ arguments, and register the class:

Python

from paperlab_space import register
from paperlab_space.metrics import Metric


@register
class StartsWith(Metric):
    name = "starts_with"
    description = "Share of outputs that start with the reference answer."

    def __init__(self, ignore_case: bool = True) -> None:
        self.ignore_case = ignore_case

    def compute(self, predictions, references):
        scores = []
        for output, refs in zip(predictions, references):
            if not refs:  # no answer key for this row
                scores.append(None)
            elif output is None:  # the request failed
                scores.append(0.0)
            else:
                fold = str.casefold if self.ignore_case else str
                scores.append(float(any(fold(output).startswith(fold(r)) for r in refs)))
        values = [s for s in scores if s is not None]
        mean = sum(values) / len(values) if values else None
        return self.result(mean, per_row=scores, n_scored=len(values))

compute gets every row of one model at once:

predictions: one output per row, or None where the request failed.
references: the valid answers per row, or None where the row has no answer key entry.

It returns a MetricResult, built with self.result(...):

Argument	What it is
`value`	The headline number, or `None` if no row could be scored.
`per_row`	Row-level scores aligned with the rows (`None` for rows not scored). These appear in the Outputs sheet.
`n_scored`	How many rows counted.
`extra`	Secondary numbers by name, such as precision next to F1. They are saved in `results.json`.

Class attributes

Attribute	Default	Meaning
`name`	required	The name used in run configs. It must be unique.
`description`	`""`	One sentence, shown by `paperlab-space metrics`.
`requires_reference`	`True`	Set `False` for a metric that does not need an answer key, such as output length. Without an answer key, `references` is all `None`.
`higher_is_better`	`True`	Set `False` where lower is better, so the LaTeX table bolds the right score.

Use it

Once the module that defines it has been imported, the metric works by name in a RunConfig, including one loaded from YAML:

Python

import my_metrics  # registers starts_with
from paperlab_space import RunConfig, run

config = RunConfig.from_yaml("run.yaml")  # metrics: [{name: starts_with, ignore_case: false}]
results = run(config)

Options are checked against the __init__ arguments, so a misspelt option is an error before any request is sent. Like the built-in metrics, a custom metric can appear twice with different options if each entry has its own label.

The paperlab-space command does not import your code, so custom metrics are for Python use.

Reuse the built-in normalisation

To compare answers the way the built-in metrics do, use Normalizer, tokenize, and extract_label from paperlab_space.metrics:

Python

from paperlab_space.metrics import Normalizer, tokenize

normalize = Normalizer(ignore_punctuation=True)
tokens = tokenize(normalize("今日は 晴れ!"))  # ['今', '日', 'は', '晴', 'れ']

See the Python API reference for their options.

Score existing results

score recomputes every configured metric for results you already have, without new requests. Use it to add a metric to an old run:

Python

from paperlab_space import MetricSpec, RunResults, export, score

results = RunResults.load_json("results-run/results.json")
results.config = results.config.model_copy(
    update={"metrics": [*results.config.metrics, MetricSpec(name="starts_with")]}
)
results.scores = score(results)
export.write_all(results, "results-run/")

Edit this page on GitHub