Run the benchmark suite#

tauopt-bench is the benchmark suite that ships in the repository beside the engine, under bench/python/. It runs a declared set of optimization problems against a declared set of methods (TauOpt’s open default policy, two reference floors, and the external libraries it has adapters for), writes one receipt per cell (one problem, one method, one replicate seed), and renders a report from the receipts. This page is about the machinery: installing the suite, listing what it can run, running one cell, reading what a run writes, bounding the CPU a run takes, and what the fleet directories are for. It states no result. What any method scores on any problem is the business of a reviewed campaign report, never of this site; The benchmark suite explains why.

Every command on this page was run on 2026-10-23 from a clone at engine version 0.4.0, from the bench/python directory the install step enters, and its output is pasted as printed; the install section’s commands (the sync, the checkpoint, the banner) were run again on 2026-10-26, when the suite became its own uv project. Two exceptions: the receipt section prints the file’s field names rather than its values, and the install checkpoint shows its path with the clone’s location abbreviated. The fleet section is read from the code and was not run.

Before you start#

Access to the repository: it is private during the beta; request access first, or the clone below fails.
git, and uv: the suite is its own uv project in the repository, beside the engine, and installs with uv only; it declares the same CPU-torch pin as the engine (Install with uv).
Python 3.12 or newer for the suite (the engine alone runs on 3.10); uv sync downloads one when the box lacks it.
Disk and network: the gp extra pulls CPU torch, botorch, and gpytorch; the lasso extra pulls two commit-pinned git sources from GitHub. A real lasso cell downloads its dataset on first use into the libsvmdata cache (LIBSVMDATA_HOME picks the place; the three real datasets total about 18 MB), because constructing a real lasso problem loads its data; a synthetic lasso cell downloads nothing, and list downloads nothing (it builds no lasso problem; the botorch and multi-objective families it does build need no download).

Install from the clone#

The suite is its own uv project with its own lock (bench/python/uv.lock), so uv sync from its directory installs it, with the engine installed from the sibling python/ as an editable path dependency, into the suite’s own environment at bench/python/.venv:

git clone https://github.com/talariasci/tauopt && cd tauopt/bench/python
uv sync --extra gp --extra bbob --extra real --extra lasso

The gp sync is the long step: CPU torch, botorch, and gpytorch, several hundred megabytes.

The gp extra is tauopt[botorch], the Gaussian-process (GP) strategies TauOpt’s methods route to, so a run with any TauOpt method needs it. bbob, real, and lasso are the three battery extras: a run needs the extras its own cells use, and list needs none of them (the bbob, bayesmark, and lasso families answer list from the registry, without their library; the botorch and multi-objective families build their problems from the library for their metadata, so list needs gp). Add one adapter extra per external library you will run. Not --all-extras: skypilot is pinned exact and belongs in a venv of its own. Everything else sits behind an extra declared in bench/python/pyproject.toml, and the suite’s core never imports any of them: a missing one fails at preflight, naming its install target, before a receipt is written.

Extra	What it enables
`gp`	the GP strategies through the native adapter (`tauopt[botorch]`)
`optuna`, `ax`, `skopt`, `fmfn`, `hyperopt`, `vizier`	one external library each, behind its adapter
`bbob`	the BBOB battery (`coco-experiment`; wrapped, never vendored)
`real`	the bayesmark task battery (`bayesmark`, `scikit-learn`)
`lasso`	the LassoBench task family (two commit-pinned git sources)
`gcs`, `skypilot`	the fleet’s receipts handoff and launch backend, maintainer-side; `skypilot` is pinned exact and belongs in a venv of its own

The sync is exact within this environment: a later uv sync here with fewer extras removes the ones left out. The engine’s own install line (uv sync from python/) builds a separate environment and never touches this one.

HEBO has no extra by design: its release declares a numpy below 1.25, which cannot co-resolve with the suite’s lock. bench/hebo-venv.sh, run with bash from any directory (the script changes to the repository root itself), builds an isolation venv at bench/.venv-hebo holding HEBO and its own math. The adapter finds that venv from any working directory (the path is resolved beside the checkout’s bench/, not against the working directory); a venv built elsewhere (bash bench/hebo-venv.sh <path>, where a relative path is taken against the repository root) needs TAUOPT_BENCH_HEBO_VENV=<absolute path to the venv>/bin/python exported before any hebo/default cell (the adapter uses the variable as given, from whatever directory the run starts in), and the variable wins when both exist; with neither in place, preflight refuses with ImportError: the hebo boundary worker needs its ISOLATION VENV (bench/hebo-venv.sh; or set TAUOPT_BENCH_HEBO_VENV) .... The venv runs a self-contained worker script over a JSON-lines pipe; the objective is evaluated on the core side for every method.

[project.scripts] installs the console script tauopt-bench; python -m tauopt_bench.cli is the same entry point (the suite’s own pins invoke it that way). Every subcommand’s flags and help text are on the tauopt-bench CLI reference, rendered from the command’s own argparse tree; this page names the flags it uses in prose. With no arguments the command prints its banner and usage:

uv run tauopt-bench

tauopt-bench -- benchmark suite for Bayesian-optimization methods.

This release claims the package name; the real package ships from
https://github.com/talariasci/tauopt -- see https://talariasci.com

usage: tauopt-bench [-h]
                    {list,plan,run,fanout,fanout-bake,fanout-bake-candidate,skyseam-emit,merge,canonical,report}
                    ...

The clone prints the same banner as the PyPI placeholder; seeing it does not mean you installed the placeholder. The checkpoint is the module path:

uv run python -c "import tauopt_bench; print(tauopt_bench.__file__)"

<clone>/bench/python/tauopt_bench/__init__.py

The line is printed with your clone’s absolute path in front; a path that ends anywhere else is another install answering.

Check the install#

One small file, as a check that the package is wired:

TAUOPT_CPU_CAP=4 uv run pytest -q tests/test_skeleton.py

....                                                                     [100%]
4 passed in 0.13s

List the problems, methods, and presets#

tauopt-bench list prints three sections: problems: (one line per registered problem id with its dimension and its known optimum, None where the problem declares only an empirical best), methods: (method id, the adapter that drives it, and whether it takes the replicate seed), and presets: (every frozen protocol with its hash). Before you run it, have the gp extra installed, as the install line above does: the botorch and multi-objective families build their problems from the library for their metadata, and without it the listing stops at the first botorch problem, after the bayesmark and bbob lines, with ImportError: botorch/dixon-price-4d needs the GP extra: pip install 'tauopt-bench[gp]' (do not run the pip install the message prints, that name is the PyPI placeholder: re-run the uv sync line with --extra gp). The three battery extras are not needed for list: the bbob, bayesmark, and lasso families register their metadata beside their factories and answer from the registry. To print one section, pipe through sed as the commands below do; head works too: when the reader closes the pipe early the listing stops with no traceback, and the command itself exits 1 (the pipeline’s status is head’s). The methods section:

uv run tauopt-bench list | sed -n '/^methods:/,/^presets:/p'

methods:
  ax/default  adapter=ax  seeded=True
  floor/random  adapter=native  seeded=True
  floor/sobol  adapter=native  seeded=True
  fmfn/default  adapter=fmfn  seeded=True
  hebo/default  adapter=hebo  seeded=True
  hyperopt/default  adapter=hyperopt  seeded=True
  optuna/default  adapter=optuna  seeded=True
  skopt/default  adapter=skopt  seeded=True
  tauopt/default  adapter=native  seeded=True
  tauopt/gp_ei  adapter=native  seeded=True
  tauopt/gp_ei-catmixed  adapter=native  seeded=True
  tauopt/gp_ei-catmixed-capped  adapter=native  seeded=True
  tauopt/gp_ei-feasgp  adapter=native  seeded=True
  tauopt/gp_ei-feasgp-fixed  adapter=native  seeded=True
  tauopt/gp_ei-mllimpute  adapter=native  seeded=True
  tauopt/wave-e2-center  adapter=native  seeded=True
  tauopt/wave-e3-dedup  adapter=native  seeded=True
  tauopt/wave-e35-select-t  adapter=native  seeded=True
  tauopt/wave-e35-select-td  adapter=native  seeded=True
  tauopt/wave-e55-cat  adapter=native  seeded=True
  tauopt/wave-e55-composed  adapter=native  seeded=True
  tauopt/wave-e55r-cat  adapter=native  seeded=True
  tauopt/wave-e55r-center  adapter=native  seeded=True
  tauopt/wave-e55r-composed  adapter=native  seeded=True
  tauopt/wave-e55r-default  adapter=native  seeded=True
  vizier/default  adapter=vizier  seeded=True
presets:

Method ids read <library>/<name>. tauopt/default is the engine’s open default policy (default@v1, described in Policies), the one surface public claims use, registered as {"strategy": "auto"} plus the replicate seed and nothing else, so it runs under the engine’s default timebox (budget mode, 10 s of wall per GP rung) exactly as a user gets it; the tauopt/wave-e55r-* arms are the ones that pin timebox_mode: "off"; floor/random and floor/sobol are the two reference floors; <library>/default is that library driven at its documented defaults through the suite’s shared ask/tell boundary; the tauopt/gp_ei* ids pin one explicit gp_ei configuration each: catmixed and catmixed-capped set categorical_mode to mixed and mixed_capped, mllimpute sets noise_mode to mll_impute, feasgp and feasgp-fixed set failure_mode to feasibility_gp and feasibility_gp_fixed (the experimental values in Strategies and config), and the tauopt/wave-* ids are candidate engine configurations under measurement, development arms rather than public method surfaces. seeded=True on every row means the replicate seed reaches the method: into the engine config for the native adapter, into the library’s own seeding for an external one. It is not the determinism class, which is declared per adapter and appears in the receipt as determinism_class: bit_exact for native, seeded for every external adapter at this version (ax, fmfn, hebo, hyperopt, optuna, skopt, vizier); no adapter declares stochastic.

The presets, with the hash that is each one’s identity:

uv run tauopt-bench list | sed -n '/^presets:/,$p'

presets:
  v1-dev  blake2b_8044d3357d4a145d
  v1-dev-con-xroster  blake2b_b91c9722e435623f
  v1-dev-cond-xroster  blake2b_6e85335724a614a1
  v1-dev-hdbo-probe  blake2b_3044a93d703036be
  v1-dev-mcm  blake2b_e3f7e0edc7664a53
  v1-dev-mcm-xlib  blake2b_0376a15c662368a8
  v1-dev-mcm-xroster  blake2b_74e8ed91dd20aa70
  v1-dev-par-hebo-probe  blake2b_4af701ae8112f4a2
  v1-dev-par-xroster  blake2b_b6c4cb1689168ca2
  v1-dev-real-xroster  blake2b_8f27f8f13e66b3b2
  v1-dev-xlib  blake2b_a33b3c4ac0eb67c0
  v1-dev-xroster  blake2b_8877037d21bf0973
  v1-full  blake2b_ae93e6c385a896fa
  v1-full-xlib  blake2b_01db4dc51342b918
  v1-full-xlib2  blake2b_f34c018c271af537
  v1-full-xlib3  blake2b_7cf17cdff5b27f32
  v1-real  blake2b_db88d4e6bd3ba50c
  v1-real-par-xroster  blake2b_a868864d7e48c05f
  v1-real-xroster  blake2b_92c255ae548f7824
  v1-real-xroster2  blake2b_0e8f7cd1460684bc
  v1-s6-cat  blake2b_8db0a8ec027c5653
  v1-s6-cat-e5b  blake2b_2a7c0c8f406cf219
  v1-s6-fail  blake2b_1a8fcec7152ae803
  v1-s6-fail-e5a  blake2b_d9cdd14214b848c6
  v1-s6-noise  blake2b_ba25c60005500c79
  v1-smoke  blake2b_c32f561bed206128
  v1-wave1-confirmation  blake2b_3f587694f43b1cf7
  v1-wave1-confirmation-b  blake2b_15b95b1ef6125df3
  v1-wave1-confirmation-b-par  blake2b_13fbb04774199249
  v1-wave1-confirmation-par  blake2b_dc27734e2fa7a48b
  v1-wave1-discovery  blake2b_6aa9b98032fb718b
  v1-wave1-discovery-par  blake2b_f3094c207a27e736
  v1-wave1-e2-discovery  blake2b_3161ec73c631d8af
  v1-wave1-e2-discovery-par  blake2b_37fee15cb8d949d7
  v1-wave1-e3-discovery  blake2b_4eca6bebf2e2db56
  v1-wave1-e3-discovery-par  blake2b_1732c3f0bf1f1529
  v1-wave1-e35-discovery  blake2b_ced21e7be103b2af
  v1-wave1-e35-discovery-par  blake2b_27130e6d32229492
  v1-wave1-e55-reveal  blake2b_6e99f768508f38c1
  v1-wave1-e55-reveal-par  blake2b_348ff2612facc229
  v1-wave1-e55r-reveal  blake2b_65c05c13bbbf43ca
  v1-wave1-e55r-reveal-par  blake2b_3f212153fc460cfb

No preset puts a development arm beside an external library: the v1-s6-* presets measure one tauopt/gp_ei* arm against tauopt/gp_ei and the random floor, the v1-wave1-* presets put tauopt/wave-* arms beside tauopt/default (the e55r pair, beside tauopt/wave-e55r-default) and both floors, and the xlib and xroster presets carry tauopt/default, both floors, and the external libraries.

Problem ids read <family>/<name>-<dim>d. The families registered at this version (a battery is the module that registers one family; the two words name the same thing on this page), from the same listing:

uv run tauopt-bench list | sed -n '/^problems:/,/^methods:/p' | awk -F/ 'NF>1{print $1}' | sort -u

  bayesmark
  bbob
  botorch
  cond
  evalset
  lasso
  mixed
  moo

evalset is a port of the ICML 2016 evalset functions, with golden fixtures pinning it to the reference implementation; botorch wraps that library’s test functions; bbob wraps COCO; bayesmark and lasso are the two real-task batteries; mixed holds the categorical and mixed problems of the COMBO lineage; cond holds constructed conditional spaces whose optima are known by construction; moo holds multi-objective problems with fixed reference points.

A variant of a base problem appends a tag to its id (evalset/ackley-5d+fail:p0.15, evalset/ackley-3d+disc:r1, evalset/ackley-5d+con:half, evalset/ackley-11d+int:0,1,2; the +noise: tags add gaussian noise); tags compose left to right, every draw they make derives from the receipt’s seed chain, and the grammar lives in bench/python/tauopt_bench/problems/variants.py. Whether a method can run a problem is decided per cell at preflight: an adapter that cannot express one (a conditional space on a library without one, a linear constraint on a library without a feasible ask path) refuses the run before any receipt exists, never as error rows in the middle of it.

Select problems and methods with a protocol#

A run is driven by a protocol: the problem ids, the method ids, how many replicates and how they are seeded, and the evaluation budget per cell. The presets list prints are protocols frozen as JSON files under bench/python/tauopt_bench/protocols/presets/; a test pins every preset’s hash, so editing one is a failing test and a new protocol is a new file. --protocol takes a preset name or a path to a JSON file of your own with the same shape:

{
  "protocol_id": "one-cell",
  "version": 1,
  "problem_ids": ["evalset/branin01-2d"],
  "method_ids": ["tauopt/default"],
  "seed_policy": {"master_seed": 20261020, "n_replicates": 1},
  "budget_rule": {"rule": "fixed", "fixed": 12, "cap": null},
  "device": "cpu",
  "notes": "docs example: one problem, one method, one replicate"
}

seed_policy carries master_seed and exactly one of n_replicates (replicate seeds derive per cell) or seeds (an explicit list used verbatim, the way to reproduce one specific run).
budget_rule is fixed with fixed: N, or the registered formula "20+40*ceil(sqrt(d))" of the problem’s dimension, with an optional cap. Rules are looked up by their literal string, never evaluated from the file.
An optional parallel: {"rounds": R, "batch_size": B} block runs in batch mode: each round asks for B points conditioned on every observation up to and including the previous round. The budget must then be fixed and equal R * B, and every method must have a native batch-ask API or preflight refuses.
version is the protocol schema version; 1 is the only value today.
device accepts cpu only: the engine never runs on a GPU, and the field exists so a protocol says so explicitly.
protocol_hash is a BLAKE2b hash of the canonical JSON (sorted keys, compact, defaults dropped); it keys every receipt and every seed. The id is a label; the hash is the identity.

The cell grid is problem_ids x method_ids x replicates, enumerated in list order.

Run one problem with one method for one seed#

Save the JSON as one-cell.json under bench/runs/, the directory runs belong in; create it first (mkdir -p ../runs): the repository’s .gitignore excludes it, so it is absent from a fresh clone and results never land in a commit by accident. --out creates its own directory. Then run it with one worker:

uv run tauopt-bench run --protocol ../runs/one-cell.json --out ../runs/one-cell --workers 1

receipts: ../runs/one-cell/receipts.jsonl

run prints nothing until it finishes; rows are appended to receipts.jsonl as cells complete, so wc -l ../runs/one-cell/receipts.jsonl is the progress gauge. An adapter failure inside a cell becomes a status: "error" row carrying the error text, the run continues, and the exit code says so: an exit of 2 names the count on stderr (ERROR: N cell(s) failed -- status=error rows recorded in receipts, with the count in place of N); the count is the file’s, every error row in receipts.jsonl, a resume’s included; list the rows with jq -c 'select(.status=="error") | [.run_id, .failure_mode]' ../runs/one-cell/receipts.jsonl. failure_mode is the exception’s type and text with memory addresses scrubbed, cut at 400 characters; the full traceback is not kept and no flag prints it.

Preflight runs before any cell: an unknown id, a missing extra, a cell the adapter cannot express, or two replicates whose reduced seeds would collide all refuse the whole run up front (the messages are listed under When a run or report refuses). --out is created if absent; a directory that already holds this protocol’s receipts.jsonl is resumed (its cells are skipped and the rest appended, under Resume an interrupted run), and one holding another protocol’s receipts refuses (so do one another run holds, one holding another shard’s cells, one written under another environment, and one carrying a cell twice; Resume an interrupted run names each), here the v1-smoke preset into the one-cell directory:

uv run tauopt-bench run --protocol v1-smoke --out ../runs/one-cell --workers 1

ValueError: refusing to resume ../runs/one-cell/receipts.jsonl: its receipts carry protocol_hash blake2b_7e551d03cf722750 but this run is blake2b_c32f561bed206128 -- use a fresh --out directory

run exits 0 when every cell produced a receipt and none is an error row, 2 when the file carries an error row (it counts every error row in the file, a resume’s included, so a directory that still carries one exits 2 on every run into it; and when neither or both of --protocol and --shard are given). Every other refusal is an uncaught exception: the message is the last line of a Python traceback and the exit code is 1. Preflight refuses with KeyError: "unknown problem '<id>' (known: [...])" or KeyError: "unknown method '<id>' (known: [...])" for an unknown id, ImportError: boundary worker for '<library>' unavailable: ... for a missing adapter extra, tauopt_bench.adapters.base.UnsupportedProblemError: cell (<problem>, <method>): ... for a cell the adapter cannot express, and ValueError: reduced internal seed collision in (<problem>, <method>): replicates i and j share a 32-bit seed -- re-key the protocol (master_seed/notes) or use explicit seeds for a seed collision. merge refuses with ValueError: refusing to merge mixed protocol_hash receipts: [...] or ValueError: duplicate cell in merge inputs: (...). Every claims-mode refusal begins claims mode refused:. list without the gp extra ends in the botorch family’s ImportError (under List the problems, methods, and presets).

Resume an interrupted run#

After an interrupted run (a kill, an out-of-memory, a closed laptop) the directory holds the receipts of the cells that completed. Run the same protocol, or the same shard spec, into it again: run reads the file, skips every cell it already carries, runs the rest, and appends them, printing one line before the receipts: line that says how many it skipped. The resumed file equals a single uninterrupted run canonically (a pin; a row is a function of the protocol, the cell, and the environment, never of which process wrote it). The one-cell run above is complete, so running it again skips its one cell and appends nothing:

uv run tauopt-bench run --protocol ../runs/one-cell.json --out ../runs/one-cell --workers 1

resumed: 1 of 1 cell(s) already in ../runs/one-cell/receipts.jsonl
receipts: ../runs/one-cell/receipts.jsonl

A resume is the same committed code (the engine and bench HEAD the receipts name; an uncommitted edit between the kill and the resume is not seen) with the same installed versions, on the same machine with the same knobs (the claims gate’s one-substrate rule): run compares each present row’s env fingerprint with its own and refuses a file written under another engine or bench commit, other installed versions, another CPU, or another --torch-threads value (ValueError: refusing to resume ...: its receipts carry another environment ...); run into a fresh --out instead. The resume checks the core environment fingerprint; a subprocess method’s worker environment (worker_env, the HEBO venv) is the claims gate’s check at report time, so keep the same worker venv across a resume. A directory holds one shard: a same-hash file carrying cells outside this run’s (another shard’s) refuses too (ValueError: refusing to resume ...: its receipts carry cells outside this run's ...). One run at a time per directory: run holds an exclusive lock on <out>/.run.lock from the moment it reads the file until the manifest is written (a --protocol run) or the last receipt is appended (a --shard run), and a second run into the directory refuses while it is held (ValueError: another run holds <out> (its .run.lock is taken) ...); wait for it, or use a fresh --out. The lock file stays behind after a run or a refusal, unlocked and empty; it is never a receipt.

A cell counts as present under any status, an error row included: receipts are append-only per run and merge refuses a duplicate cell, so the resume never re-runs a failed cell into the same file, and the exit code counts every error row in the file, a resume’s included, so a directory that still carries one exits 2 on every run into it; to re-run the cell, fix the cause and either run into a fresh --out or delete the error row from the file and run again (a resume runs every cell the file no longer carries). A file carrying one cell twice (a hand-duplicated row) refuses before any bookkeeping (ValueError: refusing to resume ...: it carries cell (...) twice ...): a receipts file holds one row per cell, so prune the extra row and run again. A receipts file whose last line has no trailing newline (a kill during a write cut the line) refuses before any append (ValueError: refusing to resume ...: its last line has no trailing newline ...); delete that partial last line and run again. An empty file (a kill before the first row was written) refuses too (ValueError: <path>: empty receipts file ...); delete it. A complete line that is not JSON refuses as corrupt (ValueError: <path>:<n>: malformed receipt line: ..., naming the line); a line that is valid JSON but not a receipt row ({}) ends in the row model’s own ValidationError: N validation errors for ReceiptRow, with no file or line prefix, the file having been hand-edited; restore the line. After a resume of a --protocol run, manifest.json carries resumed_cells and its n_errors counts the error rows of the whole file; a resume that runs nothing keeps the manifest’s wall time and knobs. A --shard run writes receipts only, no manifest. A hand-written protocol of the missing cells is not a resume: it carries a different hash, and merge refuses to mix it with the original. At campaign scale, fanout --reuse-attempts resumes a bucket prefix for the same code and protocol.

The run directory holds three files (ls hides a fourth, .run.lock, the empty file the run holds its lock on; it is not a receipt):

ls ../runs/one-cell

manifest.json
protocol.json
receipts.jsonl

protocol.json is the resolved protocol in canonical form, the bytes protocol_hash is computed over.
manifest.json is run metadata that enters no identity check: the protocol id and hash, the cell and error counts (the error count is the file’s, across a resume), the wall time of the run that wrote it (a resume that runs cells rewrites it with its own), the three worker knobs (workers, torch_threads, max_tasks_per_worker), and after a resume resumed_cells. --protocol runs write it; --shard runs write receipts only.
receipts.jsonl holds one JSON row per cell, appended as cells complete (a worker pool appends in completion order, not grid order; every consumer sorts by cell).

Read a receipt#

The snippet prints a row’s field names. Save it as read-receipt.py in bench/python and run uv run python read-receipt.py; it reads JSON only, so any Python 3 works.

# docs: no-run
# (needs the run directory the previous step wrote)
import json

with open("../runs/one-cell/receipts.jsonl") as f:
    row = json.loads(f.readline())
print(sorted(row))
print(row["status"], row["budget"], len(row["trace"]), len(row["best_trace"]))
for key in ("seeds", "env", "timings", "memory"):
    print(key, sorted(row[key]))
print(sorted(row["trace"][0]))
print(row["boundary"])

['adapter', 'adapter_version', 'best_trace', 'boundary', 'budget', 'determinism_class', 'env', 'failure_mode', 'memory', 'method_id', 'moo', 'objective', 'problem_id', 'protocol_hash', 'replicate_k', 'run_id', 'seeds', 'status', 'timings', 'trace']
ok 12 12 12
seeds ['adapter_internal_seed', 'engine_config_seed', 'instance_seed_root', 'master_seed', 'replicate_seed']
env ['bench_sha', 'bench_version', 'cpu_model', 'deps_fingerprint', 'engine_sha', 'engine_version', 'extras', 'platform', 'python', 'torch_num_threads']
timings ['acq_total_s', 'evaluate_total_s', 'fit_total_s', 'suggest_rounds_s', 'suggest_total_s']
memory ['children_peak_rss_bytes_end', 'children_peak_rss_bytes_start', 'worker_peak_rss_bytes_end', 'worker_peak_rss_bytes_start']
['assignments', 'config_hash', 'dedup', 'failed', 'fallback_from', 'iter', 'latent_value', 'policy', 'seed', 'strategy', 'value', 'value_stddev', 'values', 'values_stddev']
{'v': 1, 'transport': 'in-process'}

Every top-level field prints on every row, null where it does not apply (failure_mode and moo on this ok, single-objective row), except those that appear only when they carry a value: terminal_dedup on a terminal row, parallel on a batch-mode run, worker_env on a subprocess method’s row (boundary and memory drop the same way when empty, which no row this version writes is). The fields, by group:

Identity#

run_id is <protocol_hash>:<problem_id>:<method_id>:k<replicate>; protocol_hash, problem_id, method_id, replicate_k, budget, and objective (minimize or maximize, the problem’s own direction) name the cell; adapter, adapter_version (the driven library’s version; TauOpt’s for the native adapter), and determinism_class (bit_exact for the native engine, seeded for the external libraries, stochastic for a library that cannot thread a seed; no adapter declares it at this version) name what ran. status is ok, error, space_exhausted, or unique_candidate_unavailable. On error, failure_mode holds the error text with memory addresses scrubbed to 0x... and cut at 400 characters. The last two are structured terminals (not errors): the cell ended early at an ask the engine refused. space_exhausted carries the engine’s proof that no candidate remains; unique_candidate_unavailable means the dedup repair ran dry without a proof. terminal_dedup is that ask’s dedup record: on space_exhausted its exhaust_proof, proof_method, and universe_size are the proof; on unique_candidate_unavailable it is evidence only. A terminal row passes the claims gate’s grid check, which refuses error rows and an incomplete grid; it is listed in the report’s coverage section and is otherwise unscored, because the statistics read ok rows only (a method whose every replicate on a problem ended that way reads as having no receipt there). A cell whose every rung failed is an error row (the engine raises AskFailed), never a terminal.

Seeds#

replicate_seed is the explicit list entry or stable_seed(protocol_hash, problem_id, method_id, k); engine_config_seed is that seed as merged into the engine config for a native method; instance_seed_root seeds the problem’s own randomness (noise, failures), one derived seed per evaluation; adapter_internal_seed is the 32-bit reduction an external library received. Every seed in the chain is in the row, so a cell re-derives from its receipt alone.

Environment#

env is the fingerprint the claims gate (the checks a publishable report must pass; see Render a report) compares: the engine’s identity triple engine_version, engine_sha, and deps_fingerprint, resolved exactly as Provenance and determinism describes for a suggestion’s receipt; bench_version and bench_sha (TAUOPT_BENCH_SHA when set, else the checkout’s HEAD when the module is a tracked file, else unknown); extras, which math and adapter extras are importable; torch_num_threads; and cpu_model. boundary records the ask/tell protocol version and the transport (in-process for the native engine, subprocess for an external library), and a subprocess method adds worker_env: that interpreter’s Python, the library version, a hash over its installed distributions, and its torch thread pin.

The trace#

trace has one row per evaluation, iter from 0: assignments (the suggested point), the observed value (or values in metrics mode), value_stddev when the problem reports one, failed, latent_value (the noiseless value on noise variants: scoring adopts a new incumbent on observed improvement and reads it at its latent value, the quality of the point you would deploy), aux (channels the optimizer never saw, such as the bayesmark generalization loss), and on native rows the suggestion’s engine provenance (strategy, fallback_from, seed, config_hash, policy) and, on an authorized duplicate, the suggestion’s own intent tag, with the engine’s audit records the row carries: dedup; fits, work, and acq_events once a GP rung has run; aggregation, conditioning_events, and fallback_attempts when a rung aggregated duplicate points, escalated conditioning, or failed; center and selection on arms whose flags produce them. A record that is absent prints no key. The engine’s per-ask timings fold into the cell’s timings; its base_noise and acq_init_filters records are not carried. best_trace is the running best observed value in the problem’s direction, null until the first successful evaluation. A multi-objective problem adds moo (the reference point, the known front’s hypervolume, and the running hypervolume per iteration); a batch-mode run adds parallel (its rounds and batch size).

Measurements#

timings (suggest, evaluate, fit, and acquisition wall seconds, plus the per-ask walls) and memory (peak resident-set marks around the cell) are measurements, never identity.

tauopt-bench canonical prints the identity view: every row minus timings and memory (and minus the verbatim exception text inside the audit records), keys sorted, compact, rows sorted by (problem_id, method_id, replicate_k):

uv run tauopt-bench canonical ../runs/one-cell | uv run python -c "import json, sys; print(sorted(json.loads(sys.stdin.readline())))"

['adapter', 'adapter_version', 'best_trace', 'boundary', 'budget', 'determinism_class', 'env', 'failure_mode', 'method_id', 'moo', 'objective', 'problem_id', 'protocol_hash', 'replicate_k', 'run_id', 'seeds', 'status', 'trace']

Two runs of one protocol in fresh processes print identical canonical lines. That is the suite’s first pin (a pin is a test that fails the moment the invariant moves), and it rests on the engine’s cross-process determinism, which in the default budget timebox mode holds while no GP rung overruns its box: a rung that did descends the ladder, and the row’s fallback_from shows it.

Render a report#

uv run tauopt-bench report --receipts ../runs/one-cell --out ../runs/one-cell-report

report: ../runs/one-cell-report/report.md
  protocol blake2b_7e551d03cf722750  preset ad-hoc (not claims-grade)  mode dev
  1 rows (0 error), 1 problems (0 excluded, 1 not normalizable)

The report is a pure function of the receipt bytes: no timestamps, fixed float formats, sorted iteration, so rendering the same receipts twice gives the same bytes (a golden fixture pins it). It writes report.md and one CSV per statistic family under tables/:

find ../runs/one-cell-report -type f | sort

../runs/one-cell-report/report.md
../runs/one-cell-report/tables/borda.csv
../runs/one-cell-report/tables/borda_sidak2016.csv
../runs/one-cell-report/tables/failure_violation.csv
../runs/one-cell-report/tables/friedman_ranks.csv
../runs/one-cell-report/tables/frontier.csv
../runs/one-cell-report/tables/mean_ranks.csv
../runs/one-cell-report/tables/mean_se.csv
../runs/one-cell-report/tables/mwu_pairwise.csv
../runs/one-cell-report/tables/mwu_pairwise_sidak2016.csv
../runs/one-cell-report/tables/normalized_anytime.csv
../runs/one-cell-report/tables/normalized_final.csv
../runs/one-cell-report/tables/problem_tiers.csv
../runs/one-cell-report/tables/problem_tiers_sidak2016.csv
../runs/one-cell-report/tables/slice_deltas.csv
../runs/one-cell-report/tables/timings.csv

The files, by section: borda.csv, mwu_pairwise.csv, and problem_tiers.csv are the pairwise lane (a lane, on this page and the explanation, is one family of rank tests) under Holm correction, the primary ranking (problem_tiers is each problem’s partial order, the ballot its Borda points come from); the three *_sidak2016.csv files are the same lane under the 2016 lineage’s fixed per-test Sidak alpha, reported beside the Holm lane for comparability with the original battery papers; friedman_ranks.csv (per-problem ranks) and mean_ranks.csv (mean rank per method) feed the Friedman and Nemenyi section; normalized_final.csv and normalized_anytime.csv are the normalized score at the final iteration and at every iteration; frontier.csv is the multi-objective slice, failure_violation.csv the failure variants, slice_deltas.csv the primary lane’s significant final-best wins minus losses re-counted within each family and tag, and timings.csv and mean_se.csv the itemized timing and the per-cell mean with standard error.

The sections, in the report’s order:

coverage and integrity: which problems were scored, on what basis, and which were excluded, with the reason
the methodology self-audit table
pairwise Mann-Whitney U tests aggregated by Borda count
Friedman mean ranks with Nemenyi critical differences
the normalized score, which puts the same campaign’s floor/random median at 0 and the problem’s anchor at 100 (the known optimum, else the best-known value, else the campaign pool’s best; a problem with no random-floor receipts, no anchor, or an anchor its floor meets is listed as not normalizable and still ranked)
frontier diversity on multi-objective problems
failure-region respect on failure variants
per-family win and loss deltas
itemized timing
mean with standard error per cell

A one-cell run renders every section with nothing to compare, which is what makes it a safe way to learn the shape.

--claims turns on the gate a publishable report must pass:

the protocol hash must match a frozen preset;
the grid must be complete: every problem, method, and replicate of the preset present, nothing extra, nothing twice, and no error row (a structured terminal row passes);
every row’s budget must be the one the preset plans;
the substrate (engine and bench version and build, Python, platform, CPU model, torch thread pin) must be one across the whole campaign;
the full fingerprint must be one within each method, which admits exactly the isolated-venv split across methods;
one adapter version per method;
on a preset whose notes carry s8-floor, every row of the methodology self-audit table must pass.

An ad-hoc protocol refuses at the first check, with a message that begins claims mode refused: protocol_hash ... matches no frozen preset and ends Ad-hoc protocols report in dev mode only.

Bound the CPU a run uses#

--workers N (default auto, every core) sizes the process pool across cells. Receipts are invariant to it: every cell runs with torch pinned to the same intra-op thread count, pooled or sequential, so --workers 1 and --workers auto produce identical canonical rows (a pin). Use it to leave cores free.
--torch-threads N (default 1) is that pin. It is identity, recorded as env.torch_num_threads: a different value is a different environment, because the thread count moves torch’s numerics. The pin governs the evaluate path (the botorch family runs in torch) and the external workers; the engine’s own GP rungs set one thread for themselves and restore it afterwards whatever the pin, which is why native rows are invariant to --workers by construction.
--max-tasks-per-worker N (default 1, a fresh process per cell; 0 keeps workers alive) recycles pooled workers; it is recorded in the manifest and never in a receipt.
tauopt-bench run does not read TAUOPT_CPU_CAP; --workers is a run’s only core bound. The test suite honors TAUOPT_CPU_CAP=<n> (CI_CPU_CAP is a box-wide alias; the TauOpt name wins when set; 0 disables; an empty or malformed value refuses with pytest.UsageError: TAUOPT_CPU_CAP must be a positive integer (or 0 to disable), got ''): it shrinks the process’s CPU affinity to n cores before numpy or torch load and defaults the BLAS and OpenMP pool variables to one thread (a value you already export wins). The full suite is a long run, and its external-library tests skip without their extras; CI runs it with every adapter and battery extra and the HEBO venv present and turns those skips into failures (TAUOPT_BENCH_REQUIRE_EXTERNALS=1). The one-file smoke under Check the install is the short form.

Shard a protocol and merge the receipts#

A protocol splits into shard specs that run independently and merge back. plan writes one shard-NNNN.json per shard, each embedding the full resolved protocol so a spec file is self-contained; run --shard runs one; merge unions receipt files, refusing mixed protocol hashes and duplicate cells, sorts by cell, and checks completeness against the protocol when --protocol names it. Shards plus merge equal a single-process run, canonically. Cells go to shards round-robin, and under fanout by longest-processing-time packing when committed campaign timing tables are present. --isolate-adapter NAME pulls one adapter’s cells into a trailing shard of their own, the split the HEBO venv needs.

uv run tauopt-bench plan --protocol v1-smoke --shards 2 --out ../runs/plan-smoke

planned 2 shard(s) for v1-smoke (blake2b_c32f561bed206128) -> ../runs/plan-smoke

ls ../runs/plan-smoke

protocol.json
shard-0000.json
shard-0001.json

v1-smoke is the preset the suite’s own pins run in fresh processes: two small problems, three native methods, two replicates, a fixed budget, to prove the whole path from plan to report rather than to rank anything. Run each shard into a directory of its own, then merge them:

uv run tauopt-bench run --shard ../runs/plan-smoke/shard-0000.json --out ../runs/smoke-0 --workers 1
uv run tauopt-bench run --shard ../runs/plan-smoke/shard-0001.json --out ../runs/smoke-1 --workers 1
uv run tauopt-bench merge --inputs ../runs/smoke-0 ../runs/smoke-1 --out ../runs/smoke-merged --protocol v1-smoke

receipts: ../runs/smoke-0/receipts.jsonl
receipts: ../runs/smoke-1/receipts.jsonl
merged -> ../runs/smoke-merged/receipts.jsonl

Shards that ran on different machines make merge print WARNING: mixed <field> across inputs: [...] lines on stderr before the merged -> line (none on one machine); --protocol makes it check that the union is exactly the preset’s cell set. merge refuses a duplicate cell (ValueError: duplicate cell in merge inputs: ('evalset/branin01-2d', 'tauopt/default', 0), as printed when the one-cell run directory is listed twice) and mixed protocol hashes (ValueError: refusing to merge mixed protocol_hash receipts: [...]); drop the offending file from --inputs.

When a run or report refuses#

One line per refusal, the message’s first words first. Each is the last line of a Python traceback (exit 1) unless the line says otherwise.

ImportError: bayesmark tasks need the [real] extra: pip install 'tauopt-bench[real]', or the bbob or lasso twin, from preflight: a battery extra one of the run’s cells needs is missing; do not run the pip install the message prints (that name is the PyPI placeholder): re-run the uv sync line above, which carries all three.
ImportError: <problem> needs the GP extra: pip install 'tauopt-bench[gp]', from list (at the first botorch problem) or from preflight: the gp extra is missing; re-run the uv sync line above, which carries it.
ImportError: boundary worker for '<library>' unavailable: ...: the adapter extra for that library is missing; add --extra <library> to the sync.
ImportError: the hebo boundary worker needs its ISOLATION VENV (bench/hebo-venv.sh; or set TAUOPT_BENCH_HEBO_VENV) ...: build the venv and export the variable, as Install from the clone shows.
KeyError: "unknown problem '<id>' (known: [...])" or KeyError: "unknown method '<id>' (known: [...])": the id is not registered; list prints the registered ids.
tauopt_bench.adapters.base.UnsupportedProblemError: cell (<problem>, <method>): ...: the adapter cannot express that cell, and the text names the feature it lacks; take the cell out of the protocol.
tauopt_bench.adapters.base.UnsupportedProblemError: method '<id>' (adapter '<name>') has no native batch-suggest API -- ...: a parallel protocol with a batchless method; drop the method or the parallel block.
ValueError: reduced internal seed collision in (<problem>, <method>): replicates i and j share a 32-bit seed -- re-key the protocol (master_seed/notes) or use explicit seeds: do what it says.
ValueError: refusing to resume <path>/receipts.jsonl: its receipts carry protocol_hash <a> but this run is <b> -- use a fresh --out directory: the --out directory holds another protocol’s receipts; pick a fresh one (the same protocol’s directory is resumed instead, under Resume an interrupted run).
ValueError: refusing to resume <path>/receipts.jsonl: its receipts carry cells outside this run's (another shard's?) -- one directory per shard, or a fresh --out: the --out directory holds another shard of the same protocol; run each shard into a directory of its own.
ValueError: refusing to resume <path>/receipts.jsonl: its receipts carry another environment (engine or bench build, dependencies, cpu, or torch threads differ) -- use a fresh --out directory: the receipts were written under another environment fingerprint (another build, machine, or --torch-threads value); a resume is the same substrate, so pick a fresh directory.
ValueError: refusing to resume <path>/receipts.jsonl: its last line has no trailing newline (a write cut short) -- delete that line or restore it, then run again: the last row was cut by a kill during its write; delete the partial line (or restore it whole) and run again.
ValueError: refusing to resume <path>/receipts.jsonl: it carries cell (<problem>, <method>, <k>) twice -- a receipts file holds one row per cell; prune the file: a hand-duplicated row; delete the extra row and run again.
ValueError: another run holds <path> (its .run.lock is taken) -- wait for it or use a fresh --out directory: a run is in progress in that directory; wait for it to end, or pick a fresh one.
ValueError: <path>/receipts.jsonl:<n>: malformed receipt line: ...: line n is not JSON (a hand edit; canonical, merge, and report say this of a file cut in the middle of a line, which run refuses by the trailing-newline line above); fix or delete the line and run again.
pydantic_core._pydantic_core.ValidationError: N validation errors for ReceiptRow (the first line of a multi-line message; the missing fields follow it, and no file path or line number prefixes it): a line that is valid JSON but not a receipt row ({}); the file was hand-edited; restore the line and run again.
ValueError: <path>/receipts.jsonl: empty receipts file ...: the receipts file exists and is empty (a kill before the first row); delete it and run again.
run: provide exactly one of --protocol | --shard (one line on stderr, exit 2): the run was given neither or both.
ERROR: N cell(s) failed -- status=error rows recorded in receipts (on stderr after the receipts: line, exit 2): adapter failures inside cells; list the rows as Run one problem with one method for one seed shows.
ValueError: duplicate cell in merge inputs: (...): one cell arrived twice; drop the file carrying it from --inputs.
ValueError: refusing to merge mixed protocol_hash receipts: [...]: the inputs come from two protocols; merge takes the shards of one plan.
ValueError: campaign incomplete: N missing cells (first: [...]), M extra: merge --protocol found that the union is not the preset’s cell set; a shard is missing or does not belong.
ValueError: claims mode refused: ...: one of the gate’s clauses under Render a report, named in the text; fix it or render in dev mode.
pytest.UsageError: TAUOPT_CPU_CAP must be a positive integer (or 0 to disable), got '' (exit 4, from pytest): the cap is set to an empty or malformed value; set a positive integer or 0.

Banked campaigns and what you can verify#

bench/campaigns/ holds the banked campaigns, one directory per frozen preset run to completion: the report of record (report.md) with its tables/, fanout-run.json where the orchestrator completed the campaign end to end, and a BANKING.md exception record where a campaign was merged by hand (campaign id, source tarball hash, the incident, the exact merge command). The raw per-cell receipts are not in the checkout: bench/runs/ is ignored, and durable copies live in a private blob store whose contents are pinned by the committed bench/receipts.blobmanifest.json. The committed timing tables double as the fan-out planner’s cost model. What a reader can verify without rerunning anything, and what needs receipt access, is laid out in the repository’s bench/REPRO.md.

The fleet, maintainer-side#

Everything above runs on one machine. Campaign-scale runs fan out: tauopt-bench fanout plans the shards, writes one SkyPilot task YAML per shard under tasks/ (the file the backend launches, so --emit-yaml-only hands a scheduler of your own the same artifact), launches one managed cloud job per shard, polls, pulls each shard’s receipts from a cloud bucket with hash verification, merges them, and writes fanout-run.json: the source tarball’s hash, per-shard attempt digests, the boot image, every job’s status, and the environment warnings. One image plus one machine type per fleet (a machine family with one CPU platform; the fleet’s default is the c3 family for exactly this reason) gives one environment fingerprint, which is what the claims gate requires; bench/docker/ builds that image at a committed commit only (the build refuses a dirty tree, because the baked SHA stamps every receipt), and fanout-bake bakes a VM image for the source-tarball mode; fanout-bake-candidate bakes and stamps a sealed, commit-bound image for one engine-wave envelope, and skyseam-emit writes a self-contained SkyPilot task YAML for a scheduler of your own, launchable with no tauopt import. The cloud dependencies live in a launcher venv ([skypilot]) and in the fleet image ([gcs]); the suite’s core never imports them. None of this is a beta-user path, and this page shows no cloud command.

bench/infra/ is the maintainer-side VM dispatch kit for the calibration probes that set the caps work ceilings; it is not a user path, and its own README names the scripts.