arborist/docs/tickets/ticket-000061-cold-object-store-tier.md

Ticket #000061 — Cold-pack distribution tier

Status: in progress — opened 2026-05-25 Opened: 2026-05-25 Scope: ship arborist corpus state to new peers (and to DVD-R archival) via point-in-time tar.zst packs hosted on an S3-compatible bucket and/or burned to physical media. One artifact serves both channels. Audience: dav1d (architectural inflection — new optional dep, new optional network egress, new public-readable surface when CDN is enabled, new "delayed snapshot" semantics around falsifications). Hard constraint: packs are content-addressed (pack_hash = hash_leaf(manifest_bytes) over a leaf-hash-sorted, deduped manifest). Same chunk set → same pack_hash. Each pack pins the corpus snapshot_root it covers in the audit chain — consumers can detect drift between the pack and current corpus state.

Problem

A new peer comes up with an empty SQLite shard. How does it become a working arborist node?

Options today:

1. Re-ingest every source from upstream (Wikipedia dumps, textbooks, crawls). Hours-to-days; depends on every upstream being reachable.
2. rsync someone else's shard. Works but bypasses the audit chain — the receiving node has no proof the bytes came from a trusted producer with verifiable provenance.
3. Download a tarball. Fast, content-addressed, audit-row-pinned, verifiable on unpack.

snapshot.py already covers corpus identity (the snapshot_root Merkle hash over sorted document_roots). What's missing is the delivery — getting the bytes to a fresh node.

Design

What goes in the bucket

Only packs. No individual chunk objects. Layout:

<bucket>/packs/<pack_hash>.tar.zst              # pack body
<bucket>/packs/<pack_hash>.manifest.ndjson      # contents sidecar

Per-pack contents (inside the tar):

manifest.ndjson                  # one line per chunk: {leaf_hash, size}
blobs/<hash[:2]>/<hash[2:]>      # one tar member per chunk, raw UTF-8 body

The blobs/ prefix inside the tar is an in-pack convention, not a bucket layout — there is no blobs/ prefix in the bucket itself.

Pack identity and idempotence

• pack_hash = hash_leaf(manifest_bytes) where the manifest is sorted by leaf_hash and deduped before hashing.
• Same chunk set → same pack_hash. Two writers building the same pack collide on bucket upload — no GC after duplicate runs.
• Pack uploads are idempotent. Pack contents are append-only by construction.

Pack-size cap (DVD-R safe-fit)

Default max_pack_bytes = 4_400_000_000 (4.4 GB) — sits ~6.5 % below the 4.7 GB DVD-R marketing capacity to absorb:

• ISO9660 / UDF filesystem overhead
• growisofs lead-in / lead-out
• Media manufacturing variance (~1–2 %)
• Older drives refusing the outer edge (~1–3 %)

Sits between industry-standard tool defaults (HandBrake DVD-5 = 4.59 GB; DVDFab fit-to-DVD-5 = 4.3 GB; mkisofs default DVD = 4.59 GB). Cap applies to uncompressed bytes so the compressed .tar.zst is ≤ cap by construction. v1 produces ~30–50 % media fill on prose; future streaming-compressed cap fills discs better.

Multi-pack splitting is greedy first-fit by accumulated raw bytes.

Two channels — one artifact

                 ┌── S3CompatibleBackend.put_pack ── DO Spaces / R2 / S3
build_pack ─┬────┤
            └────└── --local-dir DIR ─── growisofs ─── /dev/sr0

Byte-identical packs in both channels. A pack burned at one site and fetched from CDN at another collide on sha256sum.

Delayed-snapshot discipline

Packs are not live mirrors. The bucket is eventually consistent with intent, not with the live corpus. Three operational consequences:

1. Each pack pins a snapshot_root (compute_snapshot_root(conn) at pack time). Recorded in the cold_pack_pushed audit row and in the push_pack return body.
2. Falsifications produce new packs. Between repacks, drift detection / arborist falsify / rehydrate_drift can flip providence_cache rows to stale. Chunk content is immutable; what changes is the corpus-state envelope. A re-pack with the same chunk set produces a different snapshot_doc_count if documents were added; same hash if not — but the audit row's snapshot_root distinguishes the moments.
3. Stale packs accumulate. No automatic GC in v1. Old pack_hashes stay in the bucket as valid snapshots of past corpus states. GC is a separate ticket once we have a re-pack cadence to GC against.

Cores never evict; packs include them

Packs cover every hot chunk with local content — surfaces AND cores. Cores carry the distillation derivations a new peer needs to bootstrap the v9.8 chain. (Earlier draft restricted packs to surfaces only — a bug that would have shipped a fresh peer with no derivation roots.)

Verification on unpack

Every chunk's leaf_hash is the tar member name. open_pack recomputes hash_leaf(body).hex() per member and refuses to restore on mismatch. Hostile-bucket bytes never reach the local DB.

After unpack, consumers can arborist snapshot verify <root> against the pack's recorded snapshot_root to confirm clean restore (or detect drift if the local corpus has moved on since).

What was deleted from this ticket's first attempt

The original implementation included per-chunk individual blob storage (evict_to_object, rehydrate_from_object, cold push, cold pull, blob_key(), BLOB_PREFIX, put_chunk/get_chunk/has_chunk/ list_chunk_hashes on the backend ABC). Fox's correction same-day (2026-05-25): "what ever was blobs? I wanted a way to hydrate using tarballs (the core and important data) for bringing new machines up".

Five-step deletion record:

• Step 1 — Make the requirement less dumb: the requirement was always "ship a corpus to a new peer," not "expose every chunk as an S3 object." Individual blobs solved a problem nobody asked for.
• Step 2 — Delete the part: 14M-object storage, ~$70/hydrate request cost, 14M-entry LIST walks — all deleted. ~250 lines of source + 7 tests gone.
• Result: packs-only design lands at ~17 GB bucket storage (vs 38 GB for raw blobs), ~4 GETs/hydrate (vs 14M; packs filled to 4.4 GB compressed each via streaming zstd), DO Spaces request cost falls from ~$70 to ~$0.00002 per fresh peer.

Implementation

• arborist/cold_object.py — ObjectStoreBackend ABC (raw bytes, pack-shaped wrappers) + S3CompatibleBackend (boto3) + MemoryBackend (tests) + build_pack / open_pack / parse_manifest.
• arborist/evict.py — push_pack / pull_pack. Per-call binds the corpus snapshot_root into pack audit + return body.
• arborist/cli.py — arborist cold pack | unpack | stats.
• tests/test_cold_object.py — pack invariants, splitting, snapshot binding, local-dir, no-push mode. Default MemoryBackend so the default suite runs without boto3.
• tests/test_cold_object_boto3.py — moto-mocked S3 wire test (boto3 + moto gated).
• pyproject.toml — [object-store] = boto3>=1.34; dev extras pull moto>=5.0.
• Makefile — bootstrap-object-store, cold-pack, cold-pack-dvd, cold-unpack, cold-stats.
• docs/cold-object-store.md — full design, DO Spaces quickstart, CDN hydrate recipe, DVD-burn recipe, delayed-snapshot discipline, failure-mode table.

Scope boundaries

In scope: pack build / push / pull / unpack, snapshot binding, hash verification, audit chain, DVD-R local-dir output, multi-pack splitting.

Out of scope (future tickets if needed):

• Multipart upload (single-pack > 5 GB on DO Spaces / AWS).
• Streaming pack builder (lift the in-memory tar ceiling, target compressed-bytes cap).
• Stale-pack GC.
• Range-fetch partial pack pulls.
• Server-side encryption (KMS / SSE-S3); mesh ciphertext on public bucket is the v1 confidentiality path.
• Cross-provider replication.

Status

In progress. Code lands incrementally on main.

Pack format v2 (2026-05-26): v1 (chunks-only) was correctly flagged by fox as under-engineered — packs contained only chunk bodies, not the shard tables a new peer needs to actually bootstrap. v2 ships every load-bearing table inside the same tar.zst alongside the chunk blobs. Tables go in as tables/<name>.jsonl (array-per-line columnar JSONL). The edges table goes through a fan-in restructure at pack-build time (group by dst_uri + edge_type + anchor, ship src_roots as an array) which yields ~5-10× compression vs the live-schema row layout. Live SQLite schema is unchanged; the restore step expands fan-in rows back to the per-edge form. Pack uploads NO LONGER append a cold_pack_pushed audit event (would leak into the next push's dumped metadata and break the "two writers at the same corpus state produce the same pack_hash" determinism property — the bucket / disc file IS the receipt). 24 tests pass including a new test_push_pack_v2_hydrates_fresh_empty_db that round-trips push from a populated DB into a completely empty DB.

Sizing (v2, current shards): ~2.1 GB per shard pack compressed (chunk content 1.78 GB + tables ~0.3 GB), ~8.5 GB total bucket footprint for new-peer-ready packs across 4 shards.

Per-peer state NOT shipped: mesh_, selfmodel_, capital_ledger, memory_, controller_events, fork_score_branches, adapter_loss_reports, falsifications, schema_meta, meta. FTS5 (chunks_fts, documents_fts*) not shipped either — rebuilt from chunks.content + documents.title on unpack.