Experiments and the ask loop

Closed beta (nothing installed): the examples on this page use the Python client, which the beta does not distribute; the same loop over plain HTTP is quickstart_rest.py in the hosted examples, and Hosted beta: getting started walks it with curl.

The experiment is the unit of work

An experiment binds four things:

• a search space: parameters plus optional linear constraints;

• one metric, or a metrics list when several quantities are measured (the roles are on metrics and objectives);

• a budget hint, observation_budget;

• a bandwidth, parallel_bandwidth: the number of suggestions that may be open at once, default 1.

Everything the engine knows lives in one record: the space, the objective, the observation history, and the open suggestions.

That record is append-only. Each observation produces a successor state; history is never rewritten, which is what lets any served suggestion be re-derived later. Space and metric structure are immutable after creation; update_experiment moves only the name, the budget, the bandwidth, metadata, and metric thresholds. Archiving is a soft delete: the experiment stays readable (experiment, best, listings, dismissals keep working) while suggest, observe, and updates refuse with a 404. There is no unarchive.

The loop

You ask for a point, evaluate it, and report what you saw. suggest returns a suggestion carrying assignments and a provenance receipt; observe reports a value (or per-metric values, a failed=True attempt, or a value with its measurement value_stddev); best reads the incumbent or the frontier. Each ask (one suggest call) is a fresh routing decision over the current state, so the strategy changes mid-experiment without any switch on your side. The run below opens with four space-filling points and moves to the Gaussian-process (GP) rung of the never-fail ladder on the fifth ask (needs tauopt[all]):

The same loop, with what the engine keeps and checks: the record’s parts, and the two checks every ask passes before the model runs. A full bandwidth answers 429 bandwidth_full; a space with no point left raises SpaceExhausted, which is completion rather than failure.

from tauopt import Client

client = Client(driver="local")
exp = client.create_experiment(
    name="ask-loop",
    parameters=[{"name": "x", "type": "double", "bounds": [-2.0, 2.0]}],
    metric={"name": "loss", "objective": "minimize"},
    observation_budget=3,
)
for i in range(5):
    sug = client.suggest(exp.id)
    x = sug.assignments["x"]
    client.observe(exp.id, suggestion_id=sug.id, value=(x - 0.5) ** 2)
    print(i + 1, sug.provenance.strategy, round(x, 3))
best = client.best(exp.id).data[0]
print(round(best.assignments["x"], 3), round(best.value, 4))

1 sobol 1.688
2 sobol -1.091
3 sobol -0.803
4 sobol 0.417
5 gp_ei 0.739
0.417 0.0068

You may also report results the engine never suggested, by passing assignments instead of a suggestion_id; see suggestions and observations.

The observation budget is a hint rather than a stop

The run above declares a budget of 3 and asks five times; every ask answers. observation_budget is a planning input: the default policy ignores it (one optional config block, queue_selection, reads it as its planning horizon when the block names none; the block’s fields are in strategies and config). Your loop decides when to quit – a budget you set, a convergence rule you own, or a completion signal from the engine.

The alternative, an enforced stop at the budget, would turn a planning estimate into a hard refusal and force a schema change every time you wanted ten more points. Treating the budget as advice keeps the stop where the knowledge is.

Suggest is a pure function of state and config

The engine’s ask maps a state, a config, and the ask’s explicit intents (none unless you pass them) to drafts (candidate points; the driver opens each one as a suggestion) plus a seed, with no hidden input: the seed derives from the config and a scope hash (the space for sobol, the full state otherwise) unless you pin one, and the same inputs give the same answer in a different process. Two consequences follow.

A same-state re-ask is a deterministic repeat rather than a retry strategy. A 409 unique_candidate_unavailable answer stays the same until the state changes – observe or dismiss something first. Through the client, every accepted ask opens a suggestion, which changes the state, so consecutive asks differ without any effort on your part.

Which rung answers is a separate question. Under the default wall-clock timebox, a slow machine can push an ask down the fallback ladder, so the executed strategy may differ between runs; the receipt records the executed strategy and seed, and the served point re-derives exactly from those. The details live in provenance and determinism.

What completion means

The engine declares completion only when it can prove that every input-feasible point has been suggested or observed, which is possible only on a finite space (every parameter integer, categorical, or a numeric grid); a space with an ungridded double parameter never completes, so there your budget or your own convergence rule is the stop. That proof arrives as SpaceExhausted, which is both an OptimizationComplete and an ApiError, so one except OptimizationComplete: closes the loop and old except ApiError: handlers keep working. A shortfall without proof is a plain 409 and is not completion. Reaching your budget is your decision; the engine never enforces it. The proof, the universe it ranges over, and the continuation paths are in dedup and completion.

Do it: Quickstart and Run parallel workers.