Dedup and completion

Closed beta (nothing installed): the same completion over plain HTTP is exhaustion_rest.py: the ask that drains the space answers 409 space_exhausted with the proof in details.dedup, and the script reads that answer as completion.

Dedup is the engine default

Since 0.3.0 every ask refuses to suggest a point that is already observed, already open, or already in the current batch. Equality is exact and canonical: the active assignments, with integers normalized to integers, doubles to floats (negative zero folded), and categories to strings, serialized as compact JSON with sorted keys. Near-duplicates are not collapsed.

The default is on because re-evaluating a known point of a deterministic objective spends budget to learn nothing, and a strategy quietly returning the incumbent on a flat acquisition surface is the failure mode most likely to look like progress.

The universe

The exclusion universe is observed points (failed ones included), open suggestions, and whatever the current batch has already selected. Dismissed suggestions leave it, so a declined point may come back later. Every receipt’s dedup record carries collision_count, the repair counters, and per_draft_origin.

Repair belongs to the strategy that collided. The sequence rungs advance their stream, examining at most four candidates per requested point; the Gaussian-process (GP) rungs walk their already-evaluated restart pool, then try up to four bounded re-optimizations per slot with the collision excluded (gp_nehvi walks its pool only). A rung whose repair runs dry keeps what it found, and the fallback ladder fills the rest with the exclusions carried down.

The completion ladder

On a finite space the ask boundary (the layer between the strategy and the caller that enforces dedup and intents) can prove there is nothing left. When every axis is finite and the raw product of axis sizes is at most 1,024, it enumerates the feasible universe in a fixed order, fills remaining slots from it, and proves exhaustion if the walk ends with slots still empty. Beyond that bound, an unconditional, unconstrained all-finite space gets a cardinality certificate: distinct on-axis keys equal to the raw product prove exhaustion without enumeration.

Proven exhaustion is a completion state rather than a failure. Both drivers answer with a 409 of type space_exhausted, raised as SpaceExhausted – an OptimizationComplete and an ApiError at once, so one clause catches the terminal state while older handlers keep working. A shortfall without proof (repair ran dry on a continuous or large space) is a plain 409 unique_candidate_unavailable and never completion: a same-state re-ask returns the same answer, so observe or dismiss pending work first.

The completion ladder, for a one-point ask with no duplicate intents. (1) The unique search ran down every rung, repair included, and the slot is still empty. (2) When every axis is finite and the raw product of the axis sizes is at most 1,024, the boundary walks the feasible universe in a fixed order and an untaken point fills the slot; a walk that ends with the slot still empty is the enumeration proof. (3) Beyond that bound, an all-finite space with no conditions and no constraints gets the cardinality certificate: the distinct taken points on the axes are counted, and a count equal to the raw product is the proof. (4) Any other space, or a count that falls short, has no proof. (5) A proof is a 409 space_exhausted on both drivers, raised as SpaceExhausted, an OptimizationComplete and an ApiError at once: the optimization is complete, so read the best point or continue through an authorized duplicate intent. (6) No proof is a plain 409 unique_candidate_unavailable, never completion; a same-state re-ask returns the same answer, so observe or dismiss pending work before you ask again.

from tauopt import Client, OptimizationComplete

client = Client(driver="local")
exp = client.create_experiment(
    name="tiny",
    parameters=[
        {"name": "kernel", "type": "categorical", "categorical_values": ["rbf", "linear"]},
        {"name": "degree", "type": "integer", "bounds": [1, 2]},
    ],
    metric={"name": "score", "objective": "maximize"},
)
for i in range(4):
    sug = client.suggest(exp.id)
    client.observe(exp.id, suggestion_id=sug.id, value=float(i))
    d = sug.provenance.dedup
    print(sug.assignments, sug.provenance.strategy, d["per_draft_origin"], d["collision_count"])
try:
    client.suggest(exp.id)
except OptimizationComplete as e:
    d = e.details["dedup"]
    print(type(e).__name__, e.status_code, e.error_type, d["exhaust_proof"], d["proof_method"], d["universe_size"])

{'kernel': 'linear', 'degree': 2} sobol ['sobol'] 0
{'kernel': 'rbf', 'degree': 1} sobol ['sobol'] 0
{'kernel': 'rbf', 'degree': 2} sobol ['sobol'] 0
{'kernel': 'linear', 'degree': 1} sobol ['sobol'] 0
SpaceExhausted 409 space_exhausted True enumeration 4

The proof receipt

The proof rides details.dedup on the error, or the receipt when an ask proved exhaustion while granting duplicates. exhaust_proof is the verdict, proof_method names enumeration or cardinality, universe_size counts the feasible points, and the cardinality method adds matched, which must equal that size.

Authorized duplicates

A noisy objective still has work left at exhaustion, so after the unique search the boundary accepts typed duplicate requests – {kind, source_id, ordinal} – for the slots that remain. source_id is the canonical assignment key of the point to repeat. Authorization is recomputed from state and config on every request, never trusted: the config must enable the kind, the source must be the one the policy selects, and the ordinal must be the next expected count. Refusals carry a typed reason in the receipt.

• variance_replication re-evaluates an observed point to measure its spread. The policy picks the least-replicated source first, then the best observed value; max_replicas caps replicas per point.

• liar_verification re-evaluates a stale open suggestion: at least three observations have landed since it opened and its placeholder value sits within ten percent of the incumbent (the band is 0.1 * max(abs(incumbent), 1), so an absolute 0.1 when the incumbent’s magnitude is below 1). One grant per ask; max_verifications caps grants per source. Only suggestions opened under a config carrying a duplicate_intents or queue_selection block record the placeholder, so rows opened without one never qualify.

Both kinds are single-objective; on multi-metric experiments they are inert.

import json

from tauopt import Client, SpaceExhausted

client = Client(driver="local")
exp = client.create_experiment(
    name="tiny",
    parameters=[
        {"name": "kernel", "type": "categorical", "categorical_values": ["rbf", "linear"]},
        {"name": "degree", "type": "integer", "bounds": [1, 2]},
    ],
    metric={"name": "score", "objective": "maximize"},
)
for i in range(4):
    sug = client.suggest(exp.id)
    client.observe(exp.id, suggestion_id=sug.id, value=float(i))

cfg = {"strategy": "auto", "duplicate_intents": {"variance_replication": {"max_replicas": 1}}}
best = max(client.observations(exp.id), key=lambda o: o.value)
source = json.dumps(best.assignments, sort_keys=True, separators=(",", ":"))
intent = {"kind": "variance_replication", "source_id": source, "ordinal": 0}
rep = client.suggest(exp.id, config=cfg, intents=[intent])
print(rep.assignments, rep.intent, rep.provenance.dedup["per_draft_origin"])
client.observe(exp.id, suggestion_id=rep.id, value=best.value + 0.1)
try:
    client.suggest(exp.id, config=cfg, intents=[intent])
except SpaceExhausted as e:
    print("refused:", e.details["dedup"]["intent_refusals"])
dup = client.suggest(exp.id, config={"strategy": "auto", "dedup": False})
print(dup.assignments, dup.provenance.strategy, dup.provenance.dedup)

{'kernel': 'linear', 'degree': 1} {'kind': 'variance_replication', 'source_id': '{"degree":1,"kernel":"linear"}', 'ordinal': 0} ['intent_grant']
refused: [{'kind': 'variance_replication', 'source_id': '{"degree":1,"kernel":"linear"}', 'ordinal': 0, 'reason': 'source_not_in_q'}]
{'kernel': 'linear', 'degree': 1} gp_ei None

The second request is refused: the cap of one replica is spent, so the source dropped out of the qualifying set.

Turning dedup off

dedup: false is the per-config escape hatch: the ask returns exactly the requested number of drafts (candidate points) with no boundary, no proof, and no dedup record, as the last line above shows. It also refuses intents, since a grant authorizes passage through a refusal that no longer exists. Reach for it when you would rather manage repeats yourself; otherwise the intents path keeps the accounting on the receipt. The field reference is in strategies and config.

Do it: Use duplicate intents.