Skip to content

Coleman

Multi-Armed Bandit based Test Case Prioritization for Continuous Integration

Coleman is a framework that applies Multi-Armed Bandit (MAB) algorithms to solve the Test Case Prioritization problem in Continuous Integration (CI) environments.

Features

  • Adaptive learning from test execution feedback
  • Multiple MAB policies: baseline, Bayesian/stochastic, adversarial, and non-stationary variants
  • Contextual bandits: LinUCB/SWLinUCB plus LinTS and contextual epsilon-greedy variants
  • HCS support with WTS and VTS strategies
  • Cost-effective prioritization under time budgets
  • Typed configuration — Pydantic v2 models with YAML configs and composable config packs
  • Library-first API — run(), run_many(), sweep(), load_spec()
  • coleman CLI — thin wrapper: coleman run, coleman sweep, coleman validate
  • Sweep engine — grid (Cartesian) and zip (paired) parameter expansion with seed replication
  • Deterministic run_id — sha256(canonical_json(spec))[:12] for exact replication
  • Provenance tracking — spec.resolved.json + provenance.json per run

Quick Start

pip install coleman
coleman run --config my-experiment.yaml

Or use the library API:

from coleman.api import run, load_spec

spec = load_spec("my-experiment.yaml")
result = run(spec)

See the Getting Started guide for full instructions and the Configuration guide for the YAML schema, config packs, sweep engine, and determinism contract. For deep post-run analytics, see the Analysis Playbook. For advanced customization and parallel extension contracts, see Extensibility & Parallelism.