COLEMAN 4.16 — Compare projects without counting repetitions as evidence¶

Repeated seeds and thousands of builds can make a CI experiment look larger than its independent evidence. COLEMAN now connects a frozen corpus of those repetitions to its existing protected evaluator, preserving the raw observations and giving each reviewed repository family one paired mean.

What you can use¶

• Freeze one fold, native-work budget, complete build/seed membership, source clocks, project families and metric semantics with RepetitionDesign.
• Keep completed, failed, aborted and skipped attempts in a versioned RepeatedCorpusReport, bound to the original data and frozen candidates.
• Bind each immutable raw key to its declared candidate, build and seed so rearranging unchanged results cannot change their source attribution.
• Prepare equal-seed, then equal-build means. Zero and negative measurements remain valid; any missing input makes its whole unit mean unavailable.
• Publish immutable raw evidence, then use the existing protected access, paired effects, multiplicity corrections and conservative conclusions.

The complete installed TCPCI example runs actual Random/UCB policies, independently checks native NAPFD/APFDc, retains sixteen raw cells and groups its related histories into one constructed family with insufficient evidence. Its retry and native checkpoint continuation preserve the distinction between reproducibility and fresh protected evidence.

Metric clarity and compatibility¶

The guide documents the full-universe denominator of historical native verdict NAPFD, its distinction from a selected execution horizon, APFDc's cost denominator and binary32 duration boundaries in the native bandit. These are explicit definitions and independent checks, not silent changes to old results.

The native runner, existing SDK modules, result formats, source receipts, checkpoints and protected evaluator retain their behavior. New corpus contracts require explicit schema version 1 and reject unknown versions; old native artifacts remain native artifacts and are not automatically migrated into a scientific comparison.

Evidence and limits¶

Qualification includes versioned goldens, signed/zero/null/nonfinite boundaries, full-grid and pseudoreplication guards, deterministic arithmetic metamorphisms, normal null/known-effect simulations, real metric mutations, offline installed execution, process interruption, concurrent publication and checkpoint recovery.

Declared family identities, normal differences and source admission still require scientific justification. This release does not add clustered inference, a formal equivalence test or a cross-fold significance engine. It does not claim new historical source admission, prospective savings or algorithm superiority. The remaining benchmark corpus work continues under #261.

The actual signed wheel also passes installed qualification outside checkout: all 308 SDK files match the release tag; the tagged Random/UCB example preserves 24 native files on retry and rejects swapped immutable build/seed records. Original Karaf data yields sixteen independently checked native rows and resumes checkpoint step 8. Actual published 4.15 results continue with 28 protected files unchanged. Revalidating 61 original source bindings preserves the existing admission report and its four uncovered historical/HCS regimes, using the original 4.13 producer for controlled captures. Both distribution signatures, all six asset digests, SBOM and version-only automatic bump were checked.