Strategies and the never-fail ladder

The five strategies

Each strategy owns one whole suggest step as a pure function of the experiment state and its config; the GP rungs fit a model and maximize an acquisition, the space-filling rungs draw and decode. The acquisition is the score a strategy maximizes to choose the next point.

• random draws one uniform random coordinate per parameter and decodes it, using the standard library. It needs no extras, honors linear constraints by rejection, and is the floor of both ladders.

• sobol draws evenly spread quasi-random points: a scrambled Sobol sequence whose scramble seed is fixed per space, skipping past every point already issued so successive asks continue one sequence. It needs tauopt[local].

• gp_ei fits a Gaussian process (GP) to the scored results and picks the point with the best expected improvement over the best feasible observation (the worst observed value until one is feasible), scored as log expected improvement (analytic for one point, Monte Carlo for a batch). Each open suggestion enters as a constant-liar fantasy (the objective at the worst value seen; each outcome-constraint component at its threshold); constraint metrics get their own GP components and weight the acquisition by feasibility.

• ucb_cl fits the objective GP once and scores candidates by the GP’s mean plus a multiple of its uncertainty (an upper confidence bound), with open points treated as if they had returned the worst value seen; it fills slots one point at a time, conditioning each filled slot as a lie before the next. It models the objective only.

• gp_nehvi is the multi-objective version: one GP per optimized and constraint metric, and the point expected to grow the volume dominated by the frontier the most (log noisy expected hypervolume improvement), with metric thresholds as the reference point and open suggestions integrated as pending points. It requires at least two optimized metrics.

The three GP rungs need tauopt[botorch]. Parameter keys and defaults for each are in the strategies and config reference; the model each GP rung builds, and where it gets expensive, is in limits and when not to use TauOpt.

The two ladders

A single-objective ask descends gp_ei, then ucb_cl, then sobol, then random. A multi-objective ask descends gp_nehvi, then sobol, then random, skipping ucb_cl because it is single-objective. The chain starts at the intended strategy and only ever moves down: fallbacks get cheaper and harder to break, never the reverse. A strategy name that is unknown, or pinned against the wrong metric shape, falls through the whole ladder.

The ask enters at the intended strategy, the policy’s route or your pinned config, and descends one rung at a time until a rung answers: gp_ei, ucb_cl, sobol, random for one optimized metric; gp_nehvi, sobol, random for two or more, with no ucb_cl rung. Every descent lands on the receipt.

Each fallback rung runs with its own default parameters – a timebox_s written for gp_ei must not poison the rung saving the ask – while the top-level config fields (seed, center_in_bootstrap, dedup and intents blocks, queue selection, categorical and timebox modes, noise floor) carry down unchanged. A rung whose extras are not installed is skipped without an attempt record.

from tauopt import ExperimentState, OptimizerConfig, Parameter, Space, ask
from tauopt.engine import Objective
from tauopt.engine.policy.default import FALLBACK_LADDER, MOO_FALLBACK_LADDER

print(FALLBACK_LADDER)
print(MOO_FALLBACK_LADDER)
space = Space(parameters=(Parameter(name="x", type="double", bounds=(0.0, 1.0)),))
state = ExperimentState(space=space, objective=Objective(metric="score", goal="max"))
result = ask(state, config=OptimizerConfig(strategy="gp_ei"))
p = result.provenance
print(p.strategy, p.fallback_from, p.dedup.ladder_hops)
for attempt in p.fallback_attempts:
    print(" ", attempt.rung, attempt.exception_type, attempt.exception_text)

('gp_ei', 'ucb_cl', 'sobol', 'random')
('gp_nehvi', 'sobol', 'random')
sobol gp_ei ('gp_ei', 'ucb_cl', 'sobol')
  gp_ei ValueError GP strategies require at least one successful observation
  ucb_cl ValueError GP strategies require at least one successful observation

What triggers a descent

A rung raises; the next rung answers. The causes:

• A wall-clock overrun. In the default budget timebox mode each GP rung has a cooperative budget (10 seconds by default, 300 at most) checked at phase boundaries. With on_timeout="fail" the ask raises the timeout instead of descending.

• States the rung cannot model: no scored observation yet, a multi-objective state for gp_ei or ucb_cl, fewer than two optimized metrics for gp_nehvi. Bad config is not a descent: a params value the intended strategy’s reader refuses (the strategy the config names, or the one the policy routes under auto with the config’s top-level flags carried into it; the wrong type, out of range, an unknown enumerated value, a two-writer conflict, or a composition of valid values the strategy cannot run on this space – the mixed categorical mode on a conditional space, the fixed feasibility GP with the mixed mode on a space with categoricals) is the caller’s 400 validation_error at the ask boundary, before any rung runs. Under auto the refusal lands on the first ask the policy routes to a GP rung; the bootstrap’s sobol asks carry the flag inertly, so a run that starts clean can meet the 400 at its first GP-routed ask.

• Numerical failure that survives the conditioning controller. A typed linear-algebra error (a non-positive-definite matrix, NaNs) retries the rung twice at a noise floor raised tenfold each time; gp_nehvi also arms a finite-filtering initializer against non-finite (NaN or inf) acquisition values. Only exhaustion of those retries descends.

• Repair exhaustion under dedup: the rung keeps the unique points it found and the ladder fills the rest.

• Integrity failures: a non-finite coordinate, the wrong number of drafts (candidate points), or, with dedup off, a draft carrying a forged duplicate authorization (the dedup boundary drops such a draft as boundary_invalid instead).

from tauopt import ExperimentState, OptimizerConfig, Parameter, Space, ask
from tauopt.engine import Objective, ObservationRecord

space = Space(parameters=(Parameter(name="x", type="double", bounds=(0.0, 1.0)),))
state = ExperimentState(space=space, objective=Objective(metric="score", goal="max"))
for i in range(4):
    state = state.with_observation(ObservationRecord(assignments={"x": i / 4}, value=float(i)))
result = ask(state, config=OptimizerConfig(strategy="gp_ei", params={"timebox_s": 1e-6}))
p = result.provenance
print(p.strategy, p.fallback_from, p.dedup.ladder_hops)
for attempt in p.fallback_attempts:
    print(" ", attempt.rung, attempt.stage, attempt.exception_type, attempt.exception_text)
print(p.timings)

ucb_cl gp_ei ('gp_ei', 'ucb_cl')
  gp_ei None TimeboxExceeded gp_ei exceeded timebox_s=1e-06
fit_s=0.11919106682762504 acq_s=0.005085200071334839

The microsecond timebox applied to gp_ei alone; ucb_cl ran with its own 10-second default and answered. The timings line varies from run to run; the strategy and ladder lines are the stable part on an unloaded machine; under load ucb_cl’s own budget can bind and the descent continues.

How a fallback is recorded

strategy names the rung that answered and fallback_from the one you asked for; fallback_attempts keeps one record per failed rung, in ladder order, with the phase it died in (fit, fantasy, acq, or none), the exception type and its verbatim text, and any conditioning retries it burned. The dedup record’s ladder_hops lists every rung attempted, and timings accumulate across all of them. Through the Client the attempts arrive as a list of dicts (a["rung"], a["exception_type"], a["exception_text"]); the engine-level AskResult above carries records with attribute access. The hosted plane serves the provenance without its exception_text: the fallback_attempts and conditioning_events records keep rung, stage, exception_type and their other typed fields, and the verbatim text stays in the service’s stored provenance – the operator’s database, never the wire or the request log; the local driver, your own process, keeps it.

Why the ladder exists

A suggestion service has one job: answer. Raising on a numerics failure turns a transient conditioning problem into an outage for every worker polling the experiment. Repairing the config silently – clamping a bad parameter, swapping a model – hides the degradation from the one person who could fix it. The ladder always answers, from a documented rung, with the descent on the receipt. The bottom rung is standard-library random, which cannot fail to import. It can fail to draw in one case: under linear constraints it samples by rejection, and after 100,000 consecutive rejected draws it raises rather than loop, so a pathologically thin feasible region (the creation-time probe of 4,096 points screens out most of them) is the one refusal the ladder lets through. Because random is the last rung, the ask then raises AskFailed (error_type all_rungs_failed) with no strategy could produce a suggestion ... and the rejection text in its attempts summary, a 500 of the same type through either driver; the entry is in Troubleshooting.

Do it: Pin a strategy and seed.