Replay a suggestion#

A suggestion is a pure function of the experiment state, the config, and the ask’s explicit intents (none in this example), and its provenance receipt identifies the run that produced it. Replay needs three things: the recorded observations in order, the pinned config (with the ask’s intents, when it carried any), and the receipt to compare against. Why this holds, and what each receipt field means, is in Provenance and determinism. The values below come from one machine and dependency set.

Closed beta: this flow has no *_rest.py twin and stays local-only during the beta: replay_suggestion.py rebuilds the engine state from its JSON and calls ask in-process, which the hosted service does not expose.

Record what replay needs#

  1. Pin the config on every ask, including seed and a load-independent timebox_mode ("caps" or "off"); see Pin a strategy and seed.

  2. Log the full payload of every observe in order: assignments plus value or values, any stddevs, and failed=True rows. This log is also the local driver’s crash-recovery record (see Troubleshooting).

  3. For the suggestion you want to replay, keep assignments, the ask’s intents if it carried any, and the provenance fields strategy, config_hash, seed, engine_version, engine_sha, and deps_fingerprint.

  4. Make sure nothing else was in the state at that ask: open suggestions and dismissals are part of the state too, so observe pending work (rather than dismissing it) before the ask you intend to replay.

import json

from tauopt import Client

PARAMETERS = [
    {"name": "x1", "type": "double", "bounds": (0.0, 1.0)},
    {"name": "x2", "type": "double", "bounds": (0.0, 1.0)},
]
METRIC = {"name": "y", "objective": "minimize"}
CONFIG = {"strategy": "auto", "seed": 11, "timebox_mode": "caps"}

client = Client(driver="local")
exp = client.create_experiment(name="recorded-run", parameters=PARAMETERS, metric=METRIC)
log = []
for _ in range(7):
    sug = client.suggest(exp.id, config=CONFIG)
    a = sug.assignments
    value = (a["x1"] - 0.3) ** 2 + (a["x2"] - 0.6) ** 2
    client.observe(exp.id, suggestion_id=sug.id, value=value)
    log.append({"assignments": a, "value": value})   # the full observe payload

target = client.suggest(exp.id, config=CONFIG)       # the ask to replay (left open)
p = target.provenance
receipt = {
    "assignments": target.assignments,
    "strategy": p.strategy,
    "config_hash": p.config_hash,
    "seed": p.seed,
    "engine_version": p.engine_version,
    "engine_sha": p.engine_sha,
    "deps_fingerprint": p.deps_fingerprint,
}
with open("replay-record.json", "w") as f:
    json.dump({"parameters": PARAMETERS, "metric": METRIC, "config": CONFIG,
               "observations": log, "receipt": receipt}, f, indent=1)
print(json.dumps(receipt, sort_keys=True))

{"assignments": {"x1": 0.0, "x2": 0.0}, "config_hash": "blake2b_7d6c359d8a47e08c", "deps_fingerprint": "env_blake2b_60eded5dc28f0f6c", "engine_sha": "0bd90a9c357fda40145ff4ae829789ef6b38c49b", "engine_version": "0.3.2", "seed": 11, "strategy": "gp_ei"}

The sample was captured at engine commit 0bd90a9; your receipt shows your checkout’s HEAD, which docs-only commits move too.

Rebuild and re-ask#

  1. In a fresh process, create an experiment with the same parameters and metric.

  2. Replay the log through observe(assignments=..., ...) in the recorded order; no suggestion ids are needed.

  3. Call suggest with the recorded config, and with the recorded intents if the ask carried any.

  4. Compare engine_version, engine_sha, and deps_fingerprint first: a mismatch explains a different Gaussian-process (GP) answer before anything else does. engine_sha is stamped into every built wheel and sdist at build time, so a plain pip install of a built wheel carries the real commit; it reads unknown only for an install built without the stamp (a source tree copied by hand, or an older build) that sits outside any checkout, and two unknown values match without saying anything about the build. A checkout, an editable install (pip install -e "./python[all]"), or TAUOPT_ENGINE_SHA=$(git rev-parse HEAD) exported before each run gives a real value to compare (see Provenance and determinism).

  5. Assert config_hash, seed, strategy, and assignments against the receipt.

import json

from tauopt import Client

with open("replay-record.json") as f:
    rec = json.load(f)

client = Client(driver="local")   # a fresh process: nothing survives from the recording run
exp = client.create_experiment(name="replayed-run", parameters=rec["parameters"], metric=rec["metric"])
for row in rec["observations"]:
    client.observe(exp.id, assignments=row["assignments"], value=row["value"])

sug = client.suggest(exp.id, config=rec["config"])
p, want = sug.provenance, rec["receipt"]
print("same environment:", (p.engine_version, p.engine_sha, p.deps_fingerprint)
      == (want["engine_version"], want["engine_sha"], want["deps_fingerprint"]))
print("config_hash match:", p.config_hash == want["config_hash"])
print("seed match:", p.seed == want["seed"])
print("strategy match:", p.strategy == want["strategy"])
print("assignments match:", sug.assignments == want["assignments"])

same environment: True
config_hash match: True
seed match: True
strategy match: True
assignments match: True

The same recipe serves a hosted experiment: client.observations(exp.id) lists its observations in report order, and every hosted suggestion carries the receipt. Replay it by re-asking the service with the same recorded state and config. A hosted receipt’s deps_fingerprint is a build stamp of the service’s lock file rather than a hash of installed versions, so it never equals a fingerprint computed on your machine, and a local replay of a hosted suggestion needs that build’s dependency set, which the receipt does not carry.

Replay at the engine level#

The same replay works without a driver, against a serialized state. ExperimentState.to_json() is the canonical snapshot and ExperimentState.from_json() restores it exactly; state_hash() is the identity the derived seeds come from. Objective takes "min" or "max" where the client takes "minimize" or "maximize".

import json

from tauopt import ExperimentState, OptimizerConfig, Parameter, Space, ask
from tauopt.engine import Objective, ObservationRecord

with open("replay-record.json") as f:
    rec = json.load(f)

state = ExperimentState(
    space=Space(parameters=tuple(Parameter(**p) for p in rec["parameters"])),
    objective=Objective(metric="y", goal="min"),   # "minimize" -> "min", "maximize" -> "max"
)
for row in rec["observations"]:
    state = state.with_observation(ObservationRecord(assignments=row["assignments"], value=row["value"]))

snapshot = state.to_json()                       # canonical JSON: store this with the receipt
restored = ExperimentState.from_json(snapshot)   # the exact state, every validator re-run
print("round-trip exact:", restored.to_json() == snapshot, "state_hash", restored.state_hash())

result = ask(restored, config=OptimizerConfig(**rec["config"]))
draft, prov, want = result.drafts[0], result.provenance, rec["receipt"]
print("config_hash match:", prov.config_hash == want["config_hash"])
print("seed match:", prov.seed == want["seed"])
print("assignments match:", draft.assignments == want["assignments"])

round-trip exact: True state_hash blake2b_72a5ed3fa5e0ab41
config_hash match: True
seed match: True
assignments match: True

ask returns an AskResult: drafts[0].assignments is the point, and provenance the same receipt the client exposes. The engine path opens no suggestion and holds no bandwidth slot.