Report failures and noise#

Every observation is either a result or a failure, and a result can carry its own measurement noise. Each shape changes what the engine learns and what best reports. The modeling side is in Metrics and objectives. The values printed below depend on the machine and dependency set.

Closed beta (nothing installed): the noise half over plain HTTP is experimental_design_rest.py: value_stddev rides beside value in the JSON body of POST /v1/experiments/{id}/observations; a failure is the same body with "failed": true and no value.

Report a failed evaluation#

  1. When the evaluation itself breaks (a diverged fit, a crashed simulator), call observe(exp.id, suggestion_id=sug.id, failed=True) with no value. A value and failed=True together are refused: provide value(s) XOR failed=true. The failed point is excluded from modeling and from best but stays in the dedup universe, so the engine will not suggest it again.

  2. Budget for a longer bootstrap: the default policy counts non-failed observations only, so a failure does not advance the switch from sobol to the Gaussian-process (GP) rung.

Report measurement noise#

  1. Pass value_stddev=<float> with value (single-metric mode), or values_stddev={"<metric>": <float>, ...} with values (metrics mode; every name must also appear in values). The GP strategies square it into that point’s fixed noise variance.

  2. Keep each stddev finite and greater than zero; 0.0 is refused.

Report a point you evaluated without a suggestion#

Call observe(exp.id, assignments={...}, value=...), or with failed=True. The assignments are validated against the space, the observation counts as evidence, and its suggestion_id is None. Pass exactly one of suggestion_id or assignments.

Read best with failures present#

best(exp.id) returns the best non-failed, feasible observation and is empty before one exists. experiment(exp.id).progress.observation_count counts every observation, failures included. The run below covers all four sections: two suggested points fail, one reported point fails, and the bootstrap stays on sobol until four non-failed observations exist.

from tauopt import ApiError, Client

client = Client(driver="local")
exp = client.create_experiment(
    name="noisy-with-failures",
    parameters=[{"name": "x", "type": "double", "bounds": (0.0, 1.0)}],
    metric={"name": "score", "objective": "maximize"},
)


def measure(x):
    if x < 0.25:
        raise RuntimeError("diverged")   # the evaluation itself fails
    return 1.0 - (x - 0.5) ** 2, 0.05    # mean, measurement stddev


for i in range(8):
    sug = client.suggest(
        exp.id, config={"strategy": "auto", "seed": 2, "timebox_mode": "caps"}
    )
    x = round(sug.assignments["x"], 3)
    try:
        value, stddev = measure(sug.assignments["x"])
    except RuntimeError:
        obs = client.observe(exp.id, suggestion_id=sug.id, failed=True)
        print(i, sug.provenance.strategy, x, "failed:", obs.failed)
        continue
    obs = client.observe(exp.id, suggestion_id=sug.id, value=value, value_stddev=stddev)
    print(i, sug.provenance.strategy, x, round(value, 4), "stddev:", obs.value_stddev)

# points you evaluated without asking for them
client.observe(exp.id, assignments={"x": 0.1}, failed=True)
client.observe(exp.id, assignments={"x": 0.52}, value=0.9996, value_stddev=0.05)

print("observations:", client.experiment(exp.id).progress.observation_count)
print("best:", client.best(exp.id).data[0])

# a result and a failure flag together are refused; so is a zero stddev
try:
    client.observe(exp.id, assignments={"x": 0.3}, value=0.5, failed=True)
except ApiError as e:
    print(e)
try:
    client.observe(exp.id, assignments={"x": 0.3}, value=0.5, value_stddev=0.0)
except ApiError as e:
    print(e.message.splitlines()[2].strip())

0 sobol 0.456 0.998 stddev: 0.05
1 sobol 0.504 1.0 stddev: 0.05
2 sobol 0.812 0.9027 stddev: 0.05
3 sobol 0.228 failed: True
4 sobol 0.071 failed: True
5 sobol 0.905 0.8359 stddev: 0.05
6 gp_ei 0.284 0.9534 stddev: 0.05
7 gp_ei 0.5 1.0 stddev: 0.05
observations: 10
best: assignments={'x': 0.5004241997025637} value=0.9999998200546123 values=None
[400] validation_error: provide value(s) XOR failed=true
Value error, value_stddev must be finite and > 0, got 0.0 [type=value_error, input_value=0.0, input_type=float]

The second refusal is a multi-line pydantic message on e.message; the run prints its third line. Observing the same suggestion twice is refused with a 409 conflict, so report a repeat measurement of a point by assignments instead, or request an authorized replica (see Use duplicate intents).