Optimize multiple metrics#

Use metrics=[...] when more than one measured quantity drives the search. The engine then keeps a Pareto frontier (the observed points no other observed point beats on every optimized metric) instead of a single best point. For what a frontier, a threshold, and a stored metric mean, read Metrics and objectives; field-level detail lives in the Python API reference. The printed values are specific to the machine and dependency set that ran them.

Closed beta (nothing installed): the same experiment over plain HTTP is multimetric_rest.py. The metrics list is identical in the JSON body of POST /v1/experiments, an observation sends the same values object in its JSON body, and best_assignments returns the frontier.

Declare the metrics#

  1. Pass metrics=[...] and omit metric=. Passing both, or neither, is refused with provide exactly one of metric | metrics.

  2. Give every entry a name and an objective ("maximize" or "minimize"). Entries default to "strategy": "optimize" (the metric’s role, unrelated to the suggestion strategies gp_ei, sobol, and the rest); between one and four metrics may be optimized.

  3. To bound the region of the frontier you care about, add a threshold to an optimized metric: an upper bound for a minimized metric, a lower bound for a maximized one. A threshold on the only optimized metric is refused; use a constraint metric for that (see Add constraints).

  4. Add "strategy": "store" entries for values you want recorded but never modeled. They take no threshold.

  5. On every successful observe, pass values={...} naming each optimized and constraint metric; stored metrics are optional (a failed=True observation carries no values).

Run the loop and read the frontier#

from tauopt import Client

client = Client(driver="local")
exp = client.create_experiment(
    name="two-metric-tradeoff",
    parameters=[
        {"name": "x1", "type": "double", "bounds": (0.0, 1.0)},
        {"name": "x2", "type": "double", "bounds": (0.0, 1.0)},
    ],
    metrics=[
        # the threshold bounds the reported frontier and anchors the hypervolume the search grows (the reference point)
        {"name": "err", "objective": "minimize", "threshold": 0.6},
        {"name": "cost", "objective": "minimize"},
        {"name": "note", "objective": "maximize", "strategy": "store"},  # recorded only
    ],
    observation_budget=16,
)

for i in range(16):
    sug = client.suggest(
        exp.id, config={"strategy": "auto", "seed": 7, "timebox_mode": "caps"}
    )
    x1, x2 = sug.assignments["x1"], sug.assignments["x2"]
    err = x1**2 + 0.1 * x2**2                   # best at x1 = 0
    cost = (1 - x1) ** 2 + 0.1 * (1 - x2) ** 2  # best at x1 = 1
    client.observe(
        exp.id,
        suggestion_id=sug.id,
        values={"err": err, "cost": cost, "note": float(i)},
    )
    print(i, sug.provenance.strategy, round(err, 4), round(cost, 4))

frontier = client.best(exp.id)
print("frontier size:", frontier.count)
for entry in frontier.data:
    print({k: round(v, 3) for k, v in entry.values.items()})
print("scalar best_observation:", client.experiment(exp.id).progress.best_observation)

0 sobol 0.3903 0.1838
1 sobol 0.0017 1.0184
2 sobol 0.2894 0.2767
3 sobol 0.82 0.0382
4 gp_nehvi 0.0925 0.5939
5 gp_nehvi 0.1745 0.5285
6 gp_nehvi 0.0443 0.7053
7 gp_nehvi 0.1623 0.4277
8 gp_nehvi 0.4564 0.1406
9 gp_nehvi 0.2167 0.3442
10 gp_nehvi 0.0202 0.8359
11 gp_nehvi 0.1158 0.5097
12 gp_nehvi 0.3272 0.2274
13 gp_nehvi 0.5264 0.1053
14 gp_nehvi 0.0676 0.625
15 gp_nehvi 0.2497 0.3017
frontier size: 14
{'err': 0.39, 'cost': 0.184, 'note': 0.0}
{'err': 0.002, 'cost': 1.018, 'note': 1.0}
{'err': 0.289, 'cost': 0.277, 'note': 2.0}
{'err': 0.092, 'cost': 0.594, 'note': 4.0}
{'err': 0.044, 'cost': 0.705, 'note': 6.0}
{'err': 0.162, 'cost': 0.428, 'note': 7.0}
{'err': 0.456, 'cost': 0.141, 'note': 8.0}
{'err': 0.217, 'cost': 0.344, 'note': 9.0}
{'err': 0.02, 'cost': 0.836, 'note': 10.0}
{'err': 0.116, 'cost': 0.51, 'note': 11.0}
{'err': 0.327, 'cost': 0.227, 'note': 12.0}
{'err': 0.526, 'cost': 0.105, 'note': 13.0}
{'err': 0.068, 'cost': 0.625, 'note': 14.0}
{'err': 0.25, 'cost': 0.302, 'note': 15.0}
scalar best_observation: None

best(exp.id) returns the frontier: count entries, each carrying values (and value=None). An entry is on the frontier when no other observation beats it on every optimized metric, it is feasible, and it satisfies every threshold. Two observations are missing from the list: number 3, whose err of 0.82 fails the threshold, and number 5, which number 7 beats on both metrics. The default policy routes to gp_nehvi once the space-filling bootstrap is done; the multimetric fallback ladder is gp_nehvi -> sobol -> random (see Strategies and the never-fail ladder). experiment(exp.id).progress.best_observation is None when two or more metrics are optimized, because a scalar best does not exist there (a metrics-mode experiment with one optimized metric keeps its scalar best).

Move a threshold mid-experiment#

Thresholds are the one part of the metric cluster you can change after creation. Names and objectives stay fixed.

  1. Call update_experiment(exp.id, metric_thresholds={"<name>": <float>}); pass None for a name to clear its threshold. Over REST the same change is PATCH /v1/experiments/{id} with {"metric_thresholds": {"<name>": <float>}}.

  2. Read best(exp.id) again; the frontier is recomputed from the stored observations.

from tauopt import ApiError, Client

client = Client(driver="local")
exp = client.create_experiment(
    name="moving-threshold",
    parameters=[{"name": "x1", "type": "double", "bounds": (0.0, 1.0)}],
    metrics=[
        {"name": "err", "objective": "minimize", "threshold": 0.6},
        {"name": "cost", "objective": "minimize"},
    ],
)
for x1 in (0.1, 0.4, 0.7, 0.9):   # four reports along the tradeoff, no asks needed
    client.observe(exp.id, assignments={"x1": x1}, values={"err": x1**2, "cost": (1 - x1) ** 2})
print("frontier at err <= 0.6:", client.best(exp.id).count)

exp = client.update_experiment(exp.id, metric_thresholds={"err": 0.2})
print("thresholds now:", [(m.name, m.threshold) for m in exp.metrics])
print("frontier at err <= 0.2:", client.best(exp.id).count)

exp = client.update_experiment(exp.id, metric_thresholds={"err": None})   # None clears it
print("frontier unbounded:", client.best(exp.id).count)

try:
    client.update_experiment(exp.id, metric_thresholds={"latency": 1.0})
except ApiError as e:
    print(e)

frontier at err <= 0.6: 3
thresholds now: [('err', 0.2), ('cost', None)]
frontier at err <= 0.2: 2
frontier unbounded: 4
[400] validation_error: unknown metrics ['latency']

A single-metric experiment refuses the call with metric_thresholds requires metrics mode.

The runnable example is python/examples/multimetric_local.py in-process and its package-free twin multimetric_rest.py against the hosted service with nothing installed.