Metrics and objectives

Closed beta (nothing installed): the examples on this page use the Python client and the engine’s own types, which the beta does not distribute; the same frontier over plain HTTP is multimetric_rest.py in the hosted examples.

One metric or a metrics list

An experiment declares either a single metric or a metrics list, never both. The single form is a name plus an objective of maximize (the default) or minimize; a bare string (metric="loss") is refused with a 400 validation_error that quotes the dict form, because the shorthand named no objective and used to maximize a loss silently. Spell out objective for anything you minimize. In the single form observations carry one value, and that is the whole contract. The list form switches the experiment into metrics mode: observations carry per-metric values, every optimized and constraint metric is required on each report, stored metrics are optional, and the engine mirrors the first optimized metric as the legacy objective so older readers still find one.

A single metric can carry neither a role other than optimize nor a threshold; the driver refuses both and points you at the list form, since optimizing a metric you meant as a constraint would lose what you said.

Metric roles

Each entry in the list carries a strategy field: its role, optimize, store, or constraint, unrelated to the suggestion strategies of the ladder page. optimize metrics drive the search – one to four of them, and two or more means a Pareto frontier rather than a scalar best. store metrics are recorded with every observation and never modeled; they cannot carry a threshold. constraint metrics are black-box feasibility requirements and must carry a threshold: with maximize, a value at or above the threshold is feasible; with minimize, at or below. In gp_ei and gp_nehvi each constraint metric gets its own Gaussian-process (GP) component, and the acquisition (the score the strategy maximizes to choose the next point) is weighted by the probability that a candidate turns out feasible. ucb_cl models the objective only, and sobol and random ignore outcome constraints, so a suggestion answered from one of those rungs can land in the infeasible region.

The ceiling of four optimized metrics is a compute envelope for the hypervolume acquisition; keep secondary quantities as stored metrics and promote them when they become decisions.

Thresholds and the frontier reference point

A threshold on an optimized metric has meaning only when two or more metrics are optimized; on a single optimized metric it is refused, since a constraint metric is the right spelling of that intent. In the frontier setting thresholds filter – a point failing any threshold is excluded from the reported frontier – and anchor: the multi-objective rung uses them as the hypervolume reference point, so improvement is measured against the region you declared acceptable. A metric without a threshold anchors ten percent of the observed span beyond the worst observed value, away from the optimum (the engine computes it in its maximization frame as the minimum minus a tenth of the span; a zero span counts as one). Thresholds may move mid-experiment through update_experiment; the structure around them may not.

Two optimized metrics on one run: accuracy maximized with a threshold of 0.80, latency_ms minimized with a threshold of 60. The dashed lines are the thresholds; they meet at the reference point the gp_nehvi rung anchors its hypervolume on. best returns the solid blue points, the non-dominated observations among those within both thresholds, and the frontier runs through them. Hollow points are dominated: another observation beats them on both metrics. Grey points fail a threshold and are left out before dominance is tested, the top-right one even though no observation beats it. Measured with the local driver on the multimetric_local.py example family: seed 7, 40 observations, engine 0.4.0.

from tauopt import ExperimentState, MetricSpec, Parameter, Space
from tauopt.engine import ObservationRecord, derive_objective

metrics = (
    MetricSpec(name="accuracy", goal="max"),
    MetricSpec(name="latency_ms", goal="min", threshold=50.0),
    MetricSpec(name="memory_mb", goal="min", strategy="constraint", threshold=512.0),
    MetricSpec(name="epochs", strategy="store"),
)
space = Space(parameters=(Parameter(name="x", type="double", bounds=(0.0, 1.0)),))
state = ExperimentState(space=space, objective=derive_objective(metrics), metrics=metrics)
rows = [
    ({"x": 0.1}, {"accuracy": 0.90, "latency_ms": 40.0, "memory_mb": 400.0, "epochs": 3}),
    ({"x": 0.2}, {"accuracy": 0.95, "latency_ms": 45.0, "memory_mb": 600.0}),
    ({"x": 0.3}, {"accuracy": 0.97, "latency_ms": 70.0, "memory_mb": 300.0}),
    ({"x": 0.4}, {"accuracy": 0.80, "latency_ms": 20.0, "memory_mb": 300.0}),
]
for a, v in rows:
    state = state.with_observation(ObservationRecord(assignments=a, values=v))
state = state.with_observation(ObservationRecord(assignments={"x": 0.5}, failed=True))
print(state.is_multimetric, [m.name for m in state.optimized_metrics], state.objective)
print([o.assignments["x"] for o in state.pareto_frontier()])
print(len(state.observations), len(state.scored_observations()))
try:
    state.best_observation()
except ValueError as e:
    print("refused:", e)

True ['accuracy', 'latency_ms'] metric='accuracy' goal='max'
[0.1, 0.4]
5 4
refused: multimetric experiment: use pareto_frontier(), not best

The point at 0.2 is infeasible (memory over the constraint), the point at 0.3 fails the latency threshold, and the two survivors do not dominate each other.

What best means in each mode

best answers three questions, depending on the experiment. With one optimized metric it returns the single best feasible scored (non-failed) observation, where feasible means every constraint metric is satisfied. With two or more it returns the non-dominated, feasible, threshold-satisfying frontier, and the scalar progress.best_observation is undefined. With num_solutions above one (single-objective only) it returns that many diverse good solutions: a greedy max-min-distance pick in the unit cube, seeded at the best point, drawn from the top candidates by value. Before any successful observation the answer is an empty set rather than an error.

Failed observations and noise

failed=True records an attempt that produced no value; a value alongside it is refused. Failed rows are excluded from modeling and from best, and the policy’s space-filling bootstrap counts only successful rows. Failed rows still count as tried points for deduplication, and the gp_ei rung can be configured to model the probability of success from them.

from pydantic import ValidationError

from tauopt import ExperimentState, Parameter, Space
from tauopt.engine import Objective, ObservationRecord

space = Space(parameters=(Parameter(name="x", type="double", bounds=(0.0, 1.0)),))
state = ExperimentState(space=space, objective=Objective(metric="loss", goal="min"))
state = state.with_observation(ObservationRecord(assignments={"x": 0.2}, value=0.9, value_stddev=0.1))
state = state.with_observation(ObservationRecord(assignments={"x": 0.6}, value=0.4))
state = state.with_observation(ObservationRecord(assignments={"x": 0.9}, failed=True))
best = state.best_observation()
print(best.assignments, best.value, state.metric_stddev(best, "loss"))
print(len(state.observations), len(state.scored_observations()))
for bad in ({"value": float("nan")}, {"value": 1.0, "failed": True}):
    try:
        ObservationRecord(assignments={"x": 0.1}, **bad)
    except ValidationError as e:
        print("refused:", e.errors()[0]["msg"])

{'x': 0.6} 0.4 None
3 2
refused: Value error, observation value must be finite, got nan
refused: Value error, a failed observation carries no value

Reported noise – value_stddev, or values_stddev per metric, each finite and positive – enters the GP squared, as a fixed per-point variance. When some points report a stddev and others do not, the missing stddevs are imputed at the median of the reported stddevs (or, under gp_ei’s mll_impute noise mode, at the stage-1 inferred level); when none report, the model infers one noise level for all. A floor keeps every fit conditioned: the inferred-noise path always keeps at least the 1e-4 floor; the fixed path keeps your variances verbatim once base_noise is 0. On the inferred-noise path the fit’s noise level is constrained to at least the value resolve_floor_std returns, max(1e-4, base_noise, s2_pooled / stdvs**2) in standardized-variance units, where base_noise defaults to 1e-3 and s2_pooled is the disagreement among repeated assignments; an explicit base_noise=0 lowers that floor to the 1e-4 numerical minimum rather than removing it. On the fixed-noise path each reported variance is raised to at least base_noise times the standardized variance, and there an explicit zero leaves the reported values verbatim. Repeated assignments are collapsed to per-point means at fit time with their disagreement folded into the noise, while the raw rows stay untouched in the record. Values must be finite: a NaN is refused at the door, never discovered inside a fit. Field names and defaults are in the strategies and config reference.

Do it: Optimize multiple metrics, Report failures and noise, and Request multiple solutions.