Skip to content

COLEMAN 4.10 — Inspect controlled faults across Java codebases

Reproduce a small, explicit matrix of real upstream fixes before making a benchmark claim. COLEMAN now collects CLI-13, CLI-51 and CODEC-65 with frozen sources, common test fixtures and the existing bounded JUnit producer. Every selected result remains inspectable, including errors and failures that persist in both revisions.

What you can do

  • Collect a declared matrix. One typed plan describes three focal fixes across Apache Commons CLI and Codec. Pinned source views, original notices, JARs, runtime image, architecture, timezone and harness make the protocol explicit.
  • Keep negative evidence. The initial qualification retained 714 role observations / 357 same-test pairs: 348 both-pass, four both-fail, four assertion-failed-to-passed and one inconclusive comparison. CLI-51's test error remains inconclusive under the existing classifier. Three Codec triggering tests do not become three faults.
  • Resume without inventing observations. Existing role acknowledgments and raw packets retain interruption semantics. Typed indices distinguish observed pairs from entirely unobserved tests; incomplete collection has no completion marker. A verified completed matrix reopens offline without querying Docker.
  • Verify the evidence behind the index. Offline validation checks actual source/context bytes, raw protocols, normalized observations and diagnostic references before accepting counts, identities or collection status.

The complete matrix guide provides installation, explicit runtime preparation, source acquisition, typed Python use, CLI collection/validation, recovery and the installed-package workflow. Foreign sources, dependencies and captures stay in operator-owned external storage.

Compatibility and limits

The new optional transport adds versioned matrix plans/indices while reusing the existing generic controlled domain, source cache, JUnit producer and immutable artifact writer. Old single-pair recipes, controlled artifacts, native results, RunSpec identities and learning checkpoints retain their semantics and require no migration. Unknown matrix schemas or producers reject; there is no implicit reinterpretation of prior evidence. The producer identity conservatively binds all COLEMAN packaged code/resources, so even an unrelated packaged change selects a new generation. Incidental bytecode, hidden files and interpreter caches do not affect that identity.

Codec's common fixture is explicitly partial. Before outcomes were observed, three fixture files were excluded because of separately noticed third-party data, non-UTF8 source bytes or dependencies on excluded classes. Remaining source bytes and original LICENSE/NOTICE are preserved. This is direct-upstream reproduction, not the complete original build or a full Defects4J reconstruction. Upstream Base64/Hex/Digest fixtures also generate unseeded random inputs whose bytes/seeds are not captured. Those tests remain included with an explicit limitation; deterministic saved-artifact replay does not reconstruct fresh inputs.

The CLI fixes share project history. Multiple tests, seeds or trials do not establish independent projects or unique faults. Collection clocks and harness nanoseconds are not historical CI events, runner work or parallel wall-clock savings. Shared-JVM test-order independence remains unverified. All outputs retain not_evaluated and insufficient_evidence; diversified source admission, baseline comparisons and confirmation in #261 remain open.

Validation covers versioned goldens, old recipes, strict units/missingness, non-finite rejection, deterministic identities, duplicate accounting, negative results, source/raw/index correspondence, interrupted publication, concurrency, offline continuation and domain import boundaries. Installed-package checks use actual Java source collection and a separate native CI exact-answer/checkpoint workflow. This validates the evidence pipeline, not algorithm effectiveness.