Source materialization API¶
Use SourceCache and a frozen MaterializationSpec to verify permitted source
bytes and selected archive members. Network readers are explicit adapters;
validation works offline. Materialization does not grant scientific admission.
Start with the source-cache workflow or choose a dataset adapter. The docstrings below describe byte bounds, side effects, errors and concurrency limits.
coleman.corpus.materialization ¶
Content-addressed local source cache and bounded, restartable package integrity operations.
SourceCache ¶
Store content-addressed binary source blobs outside Git repositories.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
str or Path
|
Caller-owned trusted local cache root. Construction resolves it, validates storage placement and creates/verifies a small v1 format marker. |
required |
Notes
Every accessed blob is rechecked by size and SHA-256. Unreferenced files are neither adopted as receipts nor deleted; unknown format markers reject. Identical concurrent writers converge through immutable publication, while conflicting bytes reject. Requires the artifact writer's local filesystem operations; process-interruption limits do not guarantee power-loss recovery. This is not a hostile-filesystem sandbox or global disk quota.
root
property
¶
Return the immutable physical cache root selected and validated at construction.
__init__ ¶
Validate a root and create or verify the deterministic cache marker.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
str or Path
|
Trusted local root outside Git; existing symlink parents are resolved. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If placement or the existing marker is invalid, oversized or unsupported. |
OSError
|
If marker publication or local filesystem access fails. |
member_path ¶
Resolve and verify a declared member blob from a receipt.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
receipt
|
MaterializationReceipt
|
Validated v1 source/member declaration; metadata alone is not proof. |
required |
name
|
str
|
Exact logical member key, never interpreted as a filesystem path. |
required |
Returns:
| Type | Description |
|---|---|
Path
|
Cache path whose complete selected bytes match the declared size/hash. |
Raises:
| Type | Description |
|---|---|
KeyError
|
If the declaration does not contain the name. |
ValueError
|
If metadata, size or checksum validation fails. |
OSError
|
If cached bytes are missing or unreadable. |
Notes
Validates only this blob, without writes or network access. Use
validate_materialization to verify the whole archive/member relationship.
store_blob ¶
Verify a complete stream before immutable content-addressed publication.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
digest
|
str
|
Expected identified SHA-256 of the complete stream. |
required |
size
|
int
|
Exact byte count, from zero through 8 GiB; booleans reject. |
required |
reader
|
ByteReader
|
Caller-owned binary reader honoring requested chunk sizes. The caller closes it; even an identical replay consumes and verifies the stream. |
required |
Returns:
| Type | Description |
|---|---|
Path
|
Verified immutable blob path. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If identity, bounds, size, checksum or existing contents conflict. |
TypeError
|
If the reader returns a non-byte value. |
OSError
|
If reading or local immutable publication fails. |
Notes
Interrupted writes do not publish partial blobs. This low-level operation
grants no permission/admission; acquisition uses materialize_source and
its frozen component-terms gate. Local filesystem durability limits apply.
verify_blob ¶
Verify complete cached bytes offline without repair or publication.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
digest
|
str
|
Identified lowercase SHA-256: |
required |
size
|
int
|
Exact expected byte count, from zero through 8 GiB; booleans reject. |
required |
Returns:
| Type | Description |
|---|---|
Path
|
Verified blob path beneath the cache. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If identity, bound, actual size or checksum is invalid. |
OSError
|
If the blob is missing or cannot be read. |
Notes
Reads in bounded chunks with no network access. A filename alone does not certify contents, archive membership, permission or source admission.
materialize_source ¶
Acquire and verify a frozen raw/ZIP package and its declared members.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
spec
|
MaterializationSpec
|
Versioned source/member identities, component terms and resource bounds. |
required |
cache
|
SourceCache
|
Validated external binary cache for immutable source/member bytes. |
required |
source
|
ReaderFactory or None
|
Explicit context-managed binary reader factory. Lazy on a verified cache hit; required when the source is absent. No implicit HTTP occurs. |
None
|
refresh_source
|
bool
|
Force verification of the supplied source even on a cache hit. |
False
|
Returns:
| Type | Description |
|---|---|
MaterializationReceipt
|
Metadata returned only after the whole source and every declared selected member have been verified against actual bytes. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If terms/pins/bounds, source presence, archive structure, checksums or the boolean refresh flag fail validation. |
OSError
|
If caller-supplied reading or cache publication fails. |
Notes
Validates the frozen declaration before reader/cache access. ZIP source identity is checked before metadata/member decoding. Actual member streams are verified even on cached replay; an unrelated blob cannot prove archive membership. Interrupted work leaves only individually completed blobs. Retry reuses the frozen source and repeats bounded member validation. This does not grant scientific admission or create a second runner/checkpoint.
publish_materialization ¶
Verify cached bytes and publish an immutable metadata-only receipt.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
receipt
|
MaterializationReceipt
|
Frozen source/member declaration, revalidated against actual cached bytes. |
required |
cache
|
SourceCache
|
External binary cache; raw source/member bytes remain here. |
required |
writer
|
ArtifactWriter
|
Trusted local metadata writer with a physically separate publication root. |
required |
Returns:
| Type | Description |
|---|---|
Path
|
Deterministic receipt path under |
Raises:
| Type | Description |
|---|---|
ValueError
|
If roots overlap, metadata/bytes fail validation or published contents conflict. |
OSError
|
If offline reading or local immutable publication fails. |
Notes
Identical replay is idempotent. No falsely complete receipt is published before verification, and raw source data is not exported. The writer's trusted-parent and process-interruption/power-loss limits apply.
validate_cache_separation ¶
Reject overlapping physical cache and publication roots before writes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
cache_root
|
str or Path
|
Proposed external source cache, resolved through existing symlink parents. |
required |
output_root
|
str or Path
|
Proposed metadata publication root, with its own contained artifact family. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If roots coincide, either contains the other, or the materialization artifact family escapes the publication root. |
OSError
|
If filesystem resolution fails. |
Notes
Creates no files. Run this preflight before cache construction when planning acquisition/publication; publication repeats it. Trusted parents are required; hostile concurrent filesystem mutation is outside the guarantee.
validate_materialization ¶
Reverify complete cached source and member correspondence offline.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
receipt
|
MaterializationReceipt
|
Versioned package declaration to check against current bytes. |
required |
cache
|
SourceCache
|
External source/member cache used by the declaration. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If metadata, source identity, archive membership or member bytes disagree. |
OSError
|
If required cached files are missing or unreadable. |
Notes
Rehashes the complete source and decoded selected members and checks their independently cached copies. Publishes/repairs nothing, constructs no network reader and changes no source-availability or component-terms declaration.
validate_source_storage ¶
Reject source storage inside a Git worktree without creating files.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root
|
str or Path
|
Proposed source/cache/diagnostic location, resolved through existing symlink parents before checking ancestor Git markers. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the path is Git-backed or includes a reserved Git directory component. |
OSError
|
If path resolution or filesystem inspection fails. |
Notes
Reads only ancestor marker existence, not marker contents or sibling trees. No Git process is launched. Metadata-only exports retain their own rules. Trusted parents are required; future Git initialization and hostile concurrent filesystem mutation are outside this preflight guarantee.