The benchmark suite#

What it measures, and why it exists#

tauopt-bench measures how optimization methods perform on a declared set of problems under a declared protocol: every method gets the same fixed number of evaluations on each problem, repeated over several seeds, through one interface that asks the method for a point and tells it the result (ask/tell: the suggest/observe pair of the ask loop, in the vocabulary the external libraries share). One method on one problem with one seed is a cell. A protocol is data: problem ids, method ids, a seed policy, a budget rule, and optionally a batch geometry; its hash, not its name, is its identity.

It exists because the engine’s design choices (the default policy’s routing rule, the never-fail ladder, the dedup boundary, the batch shape) are empirical claims, and the repository’s rule is that a public claim rests on evidence a reader could regenerate: a frozen protocol, receipts, and a report that is a pure function of the receipts.

The rule for what may be claimed is one line in the repository’s AGENTS.md: public benchmark claims use the open default policy only. That policy is default@v1 (see Policies); its method id is tauopt/default, the engine as a user gets it: {"strategy": "auto"} plus the replicate seed, under the engine’s default budget timebox. The other tauopt/* method ids, the explicit gp_ei configurations and the wave-* candidate arms, exist to measure one change at a time inside the engine and are registered as development arms, not public method surfaces.

The suite is also where the engine’s guarantees get exercised at scale: a bit-exact receipt across fresh processes is the engine’s cross-process determinism, observed over every cell of a campaign rather than over one ask.

Problems and rosters#

A problem owns three things: a search space written in the engine’s own parameter language (the same dictionaries create_experiment takes), an evaluate function, and metadata (id, family, dimension, the dimension the surrogate model sees, tags, the known optimum or best known value, and multi-objective anchors). Ids are stable strings, <family>/<name>-<dim>d, and the registry is a dictionary that every battery module fills on import. Variants wrap a base problem and append a tag to its id: additive or multiplicative gaussian noise (with the noiseless value kept on the receipt as latent_value: scoring walks the incumbent chosen on observed value and reads its latent value, the quality of the point you would deploy), a deterministic or random failure region, a discretized objective, one linear constraint through the box center, or integer-valued dimensions. Every draw a variant makes comes from the receipt’s seed chain, never from global random state.

The batteries are ports or wrappers, with the port risk pinned: the ICML 2016 evalset functions are transcribed and checked against the reference implementation by golden fixtures; the botorch, COCO (BBOB), bayesmark, and LassoBench families are wrapped rather than copied (the coco-experiment package is BSD-3-Clause, but the COCO repository also carries GPL and LGPL components, so the suite imports the package and copies nothing); the categorical and mixed problems follow the COMBO lineage; the conditional mini-battery is constructed so its optima are known by construction. Sources, named once: the evalset functions are ported from sigopt/evalset, whose source of record is Dewancker et al., “A Strategy for Ranking Optimization Methods using Multiple Criteria” (ICML AutoML workshop, 2016); the categorical and mixed problems follow QUVA-Lab/COMBO and the MCBO formulations in the HEBO repository; COCO is numbbo/coco through the coco-experiment package; bayesmark is uber/bayesmark; LassoBench is ksehic/LassoBench. The rank tests follow that paper (pairwise Mann-Whitney aggregated by Borda count) and Demsar 2006 (Friedman with Nemenyi critical differences). Tags (noisy, failure, mixed, conditional, moo, constrained, boundary_optimum) let the report slice results by problem class. Some problems are saturated: on the banked campaign that measured it, the random floor met their best-known value at the standing budget. They stay registered and runnable, but derived presets leave them out by default, and the membership is pinned to the receipts that measured it.

A roster is the method side of a protocol. Every method drives its library through one ask/tell boundary: the external libraries run as subprocess workers of the workspace interpreter (HEBO alone runs its worker from a venv of its own, the one the workspace cannot co-resolve), speaking a JSON-lines protocol in which objective values cross as hex floats, bit-exact; the native engine runs the same interface in-process, because it is the process the problems evaluate in. Problems always evaluate on the core side, so every method solved the same implementation of the same function. Each adapter declares what it can express, and a cell it cannot express refuses at preflight. Two reference floors, seeded random and scrambled Sobol through the engine’s own strategies, ride every preset that compares methods across libraries; the single-arm A/B presets carry the random floor alone, and an ad-hoc protocol may omit both, at the price of a normalized score with no floor. The suite does not patch a library’s method: a batchless library sits out batch mode rather than being wrapped in a constant-liar loop of the suite’s making, and a library whose tell has no failure channel (skopt and fmfn at this version) receives a pre-declared impute the suite documents as its own: the worst observed success pushed beyond the worst by half the observed range in the problem’s direction, and a constant 0.0 before any success; the receipt records the true failed row, and the impute shapes only the library’s search. HEBO receives NaN, Optuna, Ax, and hyperopt a failed trial state, Vizier an infeasible trial.

Seeds and replicates#

A protocol’s hash is a BLAKE2b hash of its canonical JSON, and the hash, not the name, is the identity every receipt carries and every seed derives from. Presets are protocols frozen as files; they are additive, never revised, because every banked campaign’s receipts carry their hashes.

Replicates are seeds. The replicate seed for a cell is stable_seed(protocol_hash, problem_id, method_id, k) unless the policy lists explicit seeds; it reaches the engine as the config seed for a native method, and an external library as a 32-bit reduction the receipt records (preflight refuses a protocol in which two replicates would reduce to the same value). A second chain seeds the problem: instance_seed_root derives from the replicate seed, and each evaluation’s instance seed derives from it and the iteration index, so noise and failures are reproducible per evaluation. Because a derived replicate seed is keyed by method, the instance chain is too: on a noise or failure variant, two methods on the same problem and replicate face different draws (an explicit seeds list is the exception: the listed seed, and so the instance chain, is the same for every method). The design is unpaired across methods by construction and the rank tests read it that way; problems, not instances, are the paired unit of the Friedman lane (a lane, on the two benchmark pages, is one family of rank tests). Three determinism classes are declared per adapter, never inferred at run time: bit_exact (the native engine: the same inputs give byte-identical canonical rows across fresh processes; in the default budget timebox mode this holds while no rung overruns its wall box, the engine’s own stated condition, and a rung that did is visible as fallback_from), seeded (an external library reproduces its own trajectory given its seed, within one library version and, for some libraries, one OS and architecture), and stochastic (a library that cannot thread a seed; documented and surfaced; no registered adapter declares it at this version). The public arm tauopt/default runs under the engine’s default wall-clock timebox, so its receipts are bit-exact only while no rung’s box binds; the suite’s planned posture for campaign arms is timebox-free (algorithm-natural termination, the timebox_mode: "off" the wave-e55r-* arms already pin), a change to the public arm’s configuration that lands with an era roll of the frozen config surface, never silently.

Receipts and reproducibility#

A receipt is one JSON row per cell: identity (protocol hash, problem, method, replicate, budget), the full seed chain, the environment fingerprint, the trace of every evaluation with the engine’s per-suggestion provenance on native rows, the running best, the status, and the measurements (wall times, peak memory). Its canonical form drops the measurements and the verbatim third-party exception text, sorts keys, and sorts rows by cell; that is the object the suite’s pins compare. The pins: one protocol in two fresh processes gives identical canonical rows; --workers 1 and a full pool give identical rows; shards run separately and merged give the single-process run’s rows; and every method has committed golden trajectories (exact traces at budgets that cross the model phase, on each of the sequential, batch, and failure paths the method supports; the two batchless libraries have no batch golden) that a change to the suite or a library must replay exactly or regenerate deliberately.

The environment fingerprint is the engine’s identity triple (version, build SHA, dependency fingerprint; see Provenance and determinism), the suite’s own version and SHA, the Python and platform, which extras are importable, the torch thread pin, and the CPU model, because thread count and microarchitecture move numerics. A method that ran in its own interpreter adds that interpreter’s identity. Merging receipts from several machines is legal and never silent: differences in SHA, fingerprint, CPU, or adapter version are listed as warnings, and the claims gate refuses them where it matters.

An adapter failure inside a cell is evidence (not an excuse): it becomes an error row with the error text, and the run continues. A cell the engine ends early with a proof that no candidate remains is a structured terminal status with the proof attached (not an error).

The campaign model#

A campaign is one frozen protocol run to completion. At scale the protocol is planned into shards, each shard’s cells run as one managed cloud job on one image and one machine type, the receipts come back through a bucket with hash verification, and merge produces one receipts file; the orchestrator writes a run record (the source tarball’s hash, per-shard attempt digests, the boot image, every job’s status) beside it. The report renders from the merged receipts, in the sections Run the benchmark suite lists; rendering is byte-stable: the same receipts give the same report, and a golden fixture pins that.

A claims-mode report is one that passed the gate the same page enumerates: a frozen preset, a complete and clean grid, planned budgets, one substrate across the campaign (one engine build and bench build, Python, platform, CPU model, and thread pin), one full fingerprint within each method, one adapter version per method, and, on an s8-floor preset, a passing methodology self-audit table. Banked campaigns live in the repository with their report of record, their tables, their run record, and, where a campaign was merged by hand, an exception record naming the incident and the exact merge; the raw receipts live in a private store pinned by a committed manifest.

The statistics, one line each. The pairwise lane runs one-sided Mann-Whitney U tests on each replicate’s final best, in both directions of every method pair, per problem; the family-wise error is controlled per problem by Holm step-down (the primary lane) and, beside it, by the 2016 lineage’s fixed per-test Sidak alpha; each problem casts its resulting partial order as a Borda ballot. The Friedman lane takes problems as blocks and methods as treatments, the per-block value being the median of a method’s final scores, and reports Nemenyi critical differences. The normalized score puts 0 at the same campaign’s random-floor median at the same iteration and 100 at the problem’s known optimum, its best-known value, or, where neither is declared, the campaign pool’s best (a pool anchor moves with the roster, so pool-anchored scores compare within one campaign only); a problem with no random-floor receipts, no anchor, or an anchor its floor meets is listed as not normalizable and still ranked. Frontier diversity (IGD+ and coverage beside hypervolume) runs on multi-objective problems; the failure-violation rate (failed asks over the budget, whole run and second half) on failure variants; slice deltas re-count the primary lane’s significant final-best wins and losses within each family and tag. The report’s methodology self-audit table checks the survey floor the suite holds itself to (replication count, a recognized battery, anytime curves, scale-appropriate aggregation, uncertainty bands, strong baselines, equal budgets, a reproduction one-liner, itemized compute).

What is published, and where#

Results are published only through a reviewed report whose every number traces to receipts under a claims-grade protocol; nothing on this site is such a report. The reason is what a result is bound to. A number is a property of one campaign: its frozen protocol, its substrate (one engine build, one CPU model, one thread pin), and its reviewed report; elsewhere it is comparable through rank tables, never as a value. These pages are versioned with the engine and outlive every campaign, so a number here would outlive the receipts that produced it and read as a standing claim about the method, re-reviewed at no release. The report carries the number with its receipts; this site documents the engine and the suite’s machinery.

What the suite is not#

  • Not a leaderboard on this site: a result lives in a reviewed report with its receipts (see What is published, and where).

  • Not a published package. The PyPI and npm names hold placeholders; the suite ships in the repository under bench/python/ and is installed from a clone (see Run the benchmark suite).

  • Not part of the engine. Nothing in tauopt imports it; it depends on the engine, not the reverse, and installing the engine installs none of it.

  • Not a tuning harness for other libraries. Each is driven at its documented defaults through its own API, as a competent user would drive it; the suite does not fix, wrap, or extend a library’s method.

  • Not a claim about other hardware. A campaign’s receipts carry the CPU model and the thread pin, and a reproduction on different hardware is comparable through its rank tables (not byte-identical).

  • Not a pre-registration. The gate proves that a report matches its frozen protocol, not that the protocol was fixed before any result was seen; presets are additive, and a campaign is banked when it completes.

  • Not a plug-in harness. A method or a problem is a code change in bench/python/tauopt_bench/adapters/ or problems/ (each module registers into the registry on import, and a method ships with golden trajectories); there is no registration API.