Use conditional parameters#

A conditional parameter exists only on some branches of a categorical choice: momentum matters for SGD and not for L-BFGS. Declare the dependency with conditions, and the engine searches each branch with the right variables present. How absent parameters enter the model is covered in Parameters and search spaces. The assignments printed below are what one machine and dependency set produced; yours will differ.

Closed beta (nothing installed): the same space over plain HTTP is mixed_space_rest.py. The parameter objects are identical in the JSON body of POST /v1/experiments: the child carries "conditions": {"opt": ["sgd"]} and bounds are JSON arrays; the script adds a linear constraint on top of what this page declares.

Declare a parent and a child#

Declare the parent as a categorical parameter with categorical_values (at least two distinct, non-empty strings), and place it earlier in the parameters list than any child. A later or non-categorical parent is refused: condition parent 'opt' must be an EARLIER parameter in the list.
Give the child "conditions": {"<parent>": ["<value>", ...]}. The child is active only when every named parent takes one of its listed values; each listed value must be one of the parent’s categorical_values, and the lists must be non-empty.
Chain as needed: a child may itself be the parent of a further conditional parameter, as long as the list order runs parent before child.

Read assignments with absent children#

An inactive parameter is absent from assignments, so read a child with "momentum" in a or a.get("momentum"), never a["momentum"]. When you report a point you evaluated yourself, pass exactly the active set: an inactive child in assignments is refused.

import json
import math

from tauopt import ApiError, Client

client = Client(driver="local")
exp = client.create_experiment(
    name="optimizer-choice",
    parameters=[
        {"name": "opt", "type": "categorical", "categorical_values": ["adam", "sgd", "lbfgs"]},
        {"name": "lr", "type": "double", "bounds": (1e-4, 1e-1), "log_scale": True},
        {"name": "width", "type": "integer", "grid": [16, 64, 256]},
        # active only when opt == "sgd"; the parent must be declared EARLIER in the list
        {"name": "momentum", "type": "double", "bounds": (0.0, 0.99), "conditions": {"opt": ["sgd"]}},
    ],
    metric={"name": "loss", "objective": "minimize"},
)


def loss(a):
    branch = {"adam": 0.8, "sgd": 0.0, "lbfgs": 1.6}[a["opt"]]
    lr_term = (math.log10(a["lr"]) - math.log10(3e-3)) ** 2
    width_term = {16: 0.5, 64: 0.0, 256: 0.3}[a["width"]]
    momentum_term = (a["momentum"] - 0.9) ** 2 if "momentum" in a else 0.25
    return branch + lr_term + width_term + momentum_term


for i in range(12):
    sug = client.suggest(
        exp.id, config={"strategy": "auto", "seed": 7, "timebox_mode": "caps"}
    )
    a = sug.assignments
    client.observe(exp.id, suggestion_id=sug.id, value=loss(a))
    print(i, sug.provenance.strategy, json.dumps(a, sort_keys=True))

print("best:", client.best(exp.id).data[0].assignments)

# an off-suggestion report must omit inactive children
try:
    client.observe(exp.id, assignments={"opt": "adam", "lr": 0.01, "width": 64, "momentum": 0.5}, value=1.0)
except ApiError as e:
    print(e)

sobol {"lr": 0.016628537001972393, "momentum": 0.6886746892891824, "opt": "sgd", "width": 256}
sobol {"lr": 0.0007613923100241095, "momentum": 0.39355862748809156, "opt": "sgd", "width": 16}
sobol {"lr": 0.02915727609453412, "opt": "adam", "width": 64}
sobol {"lr": 0.0002708028238357388, "opt": "lbfgs", "width": 16}
sobol {"lr": 0.06060475236965854, "opt": "lbfgs", "width": 16}
sobol {"lr": 0.0002116344381045821, "opt": "adam", "width": 256}
sobol {"lr": 0.003394643095813996, "opt": "adam", "width": 64}
sobol {"lr": 0.0022936004267233375, "opt": "lbfgs", "width": 256}
gp_ei {"lr": 0.00038449088917405613, "opt": "adam", "width": 64}
gp_ei {"lr": 0.004010705638607777, "momentum": 0.5996184386177472, "opt": "sgd", "width": 64}
gp_ei {"lr": 0.0032519536625094576, "momentum": 0.9273096638852648, "opt": "sgd", "width": 64}
gp_ei {"lr": 0.1, "momentum": 0.99, "opt": "sgd", "width": 16}
best: {'opt': 'sgd', 'lr': 0.0032519536625094576, 'width': 64, 'momentum': 0.9273096638852648}
[400] validation_error: assignments keys ['lr', 'momentum', 'opt', 'width'] != active parameters ['lr', 'opt', 'width']

The space-filling bootstrap runs for max(4, 2 * dim) observations, eight here with four parameters, before the GP rung takes over (see Strategies and the never-fail ladder).

Combine with log-scale doubles and integer grids#

The space in the run mixes the other parameter kinds:

{"name": "lr", "type": "double", "bounds": (1e-4, 1e-1), "log_scale": True} searches in log10 space; both bounds must be positive, and log_scale is refused on integers.
{"name": "width", "type": "integer", "grid": [16, 64, 256]} restricts the parameter to an explicit value list; grids are stored ascending, integer grids need whole numbers, and a numeric parameter takes exactly one of bounds or grid.

Conditions, grids, and log scaling compose with each other. Linear constraints do not: a constrained parameter must be a plain, unconditional double (see Add constraints).

The runnable example is python/examples/mixed_space_local.py in-process and its package-free twin mixed_space_rest.py against the hosted service with nothing installed.