Skip to content

coleman.api

Library-first public API for driving experiments programmatically.

coleman.api

coleman.api - Library-first public API.

External projects can pip install coleman and drive experiments programmatically via :func:run, :func:run_many, or :func:sweep.

Functions:

Name Description
run

Execute a single experiment from a resolved RunSpec.
run_many

Execute multiple specs, optionally in parallel.
sweep

Expand a base spec × sweep definition and return concrete specs.
load_spec

Load and validate a RunSpec from YAML (with pack resolution).
save_resolved

Persist a resolved spec as canonical JSON.

RunResult

Lightweight container for the outcome of a single run.

Parameters:

Name Type Description Default
run_id str

Deterministic run identifier.

 required
spec RunSpec

The resolved spec that was executed.

 required
metrics dict[str, Any]

Summary metrics collected during the run.

 None
artifacts_dir str | None

Path to the run's artefact directory, if any.

 None

__init__ 

__init__(*, run_id, spec, metrics=None, artifacts_dir=None)

Initialise a RunResult.

Parameters:

Name Type Description Default
run_id str

Deterministic run identifier.

 required
spec RunSpec

The resolved spec that was executed.

 required
metrics dict[str, Any] or None

Summary metrics. Defaults to empty dict.

 None
artifacts_dir str or None

Path to the run artefact directory, if any.

 None

__repr__ 

__repr__()

Return a concise string representation of the result.

load_spec 

load_spec(path, *, packs_dir=None)

Load a :class:RunSpec from a YAML config file.

Pack references (packs: key) are resolved and deep-merged before validation.

Parameters:

Name Type Description Default
path str | Path

Path to the YAML config file.

 required
packs_dir str | Path | None

Root directory for config packs. When None (the default), the directory is derived as <config_dir>/packs so that configs remain relocatable regardless of the working directory.

 None

Returns:

Type Description
RunSpec

Validated run specification.

Raises:

Type Description
FileNotFoundError

If path or a referenced pack does not exist.
ValidationError

If the resolved dict fails schema validation.

run 

run(spec)

Execute a single experiment from a resolved spec.

This is the canonical evaluation entry-point. Both the CLI and library callers route through here.

Parameters:

Name Type Description Default
spec RunSpec

Fully resolved run specification.

 required

Returns:

Type Description
RunResult

Outcome container with deterministic run_id and summary metrics.

run_many 

run_many(specs, *, max_workers=1)

Execute multiple specs, optionally in parallel.

Parameters:

Name Type Description Default
specs list[RunSpec]

List of resolved run specifications.

 required
max_workers int

Maximum number of parallel workers (1 = sequential).

 1

Returns:

Type Description
list[RunResult]

Results in the same order as specs.

Raises:

Type Description
ValueError

If duplicate run_id values are detected when max_workers > 1 (parallel writes to the same directory would cause data corruption).

run_with_extension 

run_with_extension(spec, extension)

Execute one run using runner extension callbacks without replacing orchestration.

save_resolved 

save_resolved(spec, path, *, redact_sensitive=True)

Persist spec as canonical JSON.

Parameters:

Name Type Description Default
spec RunSpec

Resolved run specification.

 required
path str | Path

Destination file path.

 required
redact_sensitive bool

If True, redact likely sensitive values before persisting.

 True

Returns:

Type Description
Path

The written file path.

sweep 

sweep(base, sweep_spec)

Expand base × sweep_spec into concrete specs.

Parameters:

Name Type Description Default
base RunSpec

Template specification.

 required
sweep_spec SweepSpec

Sweep definition (axes + optional seeds).

 required

Returns:

Type Description
list[RunSpec]

Expanded specs in deterministic order.