# Ticket #000061 — Cold-pack distribution tier

**Status:** in progress — opened 2026-05-25
**Opened:** 2026-05-25
**Scope:** ship arborist corpus state to new peers (and to DVD-R archival)
  via point-in-time `tar.zst` packs hosted on an S3-compatible bucket
  and/or burned to physical media. One artifact serves both channels.
**Audience:** dav1d (architectural inflection — new optional dep, new
  optional network egress, new public-readable surface when CDN is
  enabled, new "delayed snapshot" semantics around falsifications).
**Hard constraint:** packs are content-addressed
  (`pack_hash = hash_leaf(manifest_bytes)` over a leaf-hash-sorted,
  deduped manifest). Same chunk set → same pack_hash. Each pack pins the
  corpus `snapshot_root` it covers in the audit chain — consumers can
  detect drift between the pack and current corpus state.

## Problem

A new peer comes up with an empty SQLite shard. How does it become a
working arborist node?

Options today:
1. Re-ingest every source from upstream (Wikipedia dumps, textbooks,
   crawls). Hours-to-days; depends on every upstream being reachable.
2. `rsync` someone else's shard. Works but bypasses the audit chain —
   the receiving node has no proof the bytes came from a trusted
   producer with verifiable provenance.
3. **Download a tarball.** Fast, content-addressed, audit-row-pinned,
   verifiable on unpack.

`snapshot.py` already covers corpus *identity* (the `snapshot_root`
Merkle hash over sorted document_roots). What's missing is the
*delivery* — getting the bytes to a fresh node.

## Design

### What goes in the bucket

Only packs. No individual chunk objects. Layout:

```
<bucket>/packs/<pack_hash>.tar.zst              # pack body
<bucket>/packs/<pack_hash>.manifest.ndjson      # contents sidecar
```

Per-pack contents (inside the tar):

```
manifest.ndjson                  # one line per chunk: {leaf_hash, size}
blobs/<hash[:2]>/<hash[2:]>      # one tar member per chunk, raw UTF-8 body
```

The `blobs/` prefix inside the tar is an in-pack convention, not a
bucket layout — there is no `blobs/` prefix in the bucket itself.

### Pack identity and idempotence

- `pack_hash = hash_leaf(manifest_bytes)` where the manifest is sorted
  by `leaf_hash` and deduped before hashing.
- Same chunk set → same pack_hash. Two writers building the same pack
  collide on bucket upload — no GC after duplicate runs.
- Pack uploads are idempotent. Pack contents are append-only by
  construction.

### Pack-size cap (DVD-R safe-fit)

Default `max_pack_bytes = 4_400_000_000` (4.4 GB) — sits ~6.5 % below
the 4.7 GB DVD-R marketing capacity to absorb:

- ISO9660 / UDF filesystem overhead
- growisofs lead-in / lead-out
- Media manufacturing variance (~1–2 %)
- Older drives refusing the outer edge (~1–3 %)

Sits between industry-standard tool defaults (HandBrake DVD-5 = 4.59 GB;
DVDFab fit-to-DVD-5 = 4.3 GB; mkisofs default DVD = 4.59 GB). Cap
applies to uncompressed bytes so the compressed `.tar.zst` is ≤ cap by
construction. v1 produces ~30–50 % media fill on prose; future
streaming-compressed cap fills discs better.

Multi-pack splitting is greedy first-fit by accumulated raw bytes.

### Two channels — one artifact

```
                 ┌── S3CompatibleBackend.put_pack ── DO Spaces / R2 / S3
build_pack ─┬────┤
            └────└── --local-dir DIR ─── growisofs ─── /dev/sr0
```

Byte-identical packs in both channels. A pack burned at one site and
fetched from CDN at another collide on `sha256sum`.

### Delayed-snapshot discipline

Packs are *not* live mirrors. The bucket is eventually consistent with
intent, not with the live corpus. Three operational consequences:

1. **Each pack pins a `snapshot_root`** (`compute_snapshot_root(conn)`
   at pack time). Recorded in the `cold_pack_pushed` audit row and in
   the `push_pack` return body.
2. **Falsifications produce new packs.** Between repacks, drift
   detection / `arborist falsify` / `rehydrate_drift` can flip
   `providence_cache` rows to `stale`. Chunk content is immutable; what
   changes is the corpus-state envelope. A re-pack with the same chunk
   set produces a different `snapshot_doc_count` if documents were
   added; same hash if not — but the audit row's `snapshot_root`
   distinguishes the moments.
3. **Stale packs accumulate.** No automatic GC in v1. Old `pack_hash`es
   stay in the bucket as valid snapshots of past corpus states. GC is
   a separate ticket once we have a re-pack cadence to GC against.

### Cores never evict; packs include them

Packs cover every hot chunk with local content — surfaces AND cores.
Cores carry the distillation derivations a new peer needs to bootstrap
the v9.8 chain. (Earlier draft restricted packs to surfaces only — a
bug that would have shipped a fresh peer with no derivation roots.)

### Verification on unpack

Every chunk's `leaf_hash` is the tar member name. `open_pack` recomputes
`hash_leaf(body).hex()` per member and refuses to restore on mismatch.
Hostile-bucket bytes never reach the local DB.

After unpack, consumers can `arborist snapshot verify <root>` against
the pack's recorded `snapshot_root` to confirm clean restore (or detect
drift if the local corpus has moved on since).

## What was deleted from this ticket's first attempt

The original implementation included per-chunk individual blob storage
(`evict_to_object`, `rehydrate_from_object`, `cold push`, `cold pull`,
`blob_key()`, `BLOB_PREFIX`, `put_chunk`/`get_chunk`/`has_chunk`/
`list_chunk_hashes` on the backend ABC). Fox's correction same-day
(2026-05-25): "what ever was blobs? I wanted a way to hydrate using
tarballs (the core and important data) for bringing new machines up".

Five-step deletion record:

- **Step 1 — Make the requirement less dumb:** the requirement was
  always "ship a corpus to a new peer," not "expose every chunk as an
  S3 object." Individual blobs solved a problem nobody asked for.
- **Step 2 — Delete the part:** 14M-object storage, ~$70/hydrate
  request cost, 14M-entry LIST walks — all deleted. ~250 lines of
  source + 7 tests gone.
- **Result:** packs-only design lands at ~17 GB bucket storage (vs
  38 GB for raw blobs), ~4 GETs/hydrate (vs 14M; packs filled to 4.4 GB
  compressed each via streaming zstd), DO Spaces request cost falls
  from ~$70 to ~$0.00002 per fresh peer.

## Implementation

- `arborist/cold_object.py` — `ObjectStoreBackend` ABC (raw bytes,
  pack-shaped wrappers) + `S3CompatibleBackend` (boto3) + `MemoryBackend`
  (tests) + `build_pack` / `open_pack` / `parse_manifest`.
- `arborist/evict.py` — `push_pack` / `pull_pack`. Per-call binds the
  corpus `snapshot_root` into pack audit + return body.
- `arborist/cli.py` — `arborist cold pack | unpack | stats`.
- `tests/test_cold_object.py` — pack invariants, splitting, snapshot
  binding, local-dir, no-push mode. Default MemoryBackend so the
  default suite runs without boto3.
- `tests/test_cold_object_boto3.py` — moto-mocked S3 wire test
  (boto3 + moto gated).
- `pyproject.toml` — `[object-store] = boto3>=1.34`; dev extras pull
  `moto>=5.0`.
- `Makefile` — `bootstrap-object-store`, `cold-pack`, `cold-pack-dvd`,
  `cold-unpack`, `cold-stats`.
- `docs/cold-object-store.md` — full design, DO Spaces quickstart,
  CDN hydrate recipe, DVD-burn recipe, delayed-snapshot discipline,
  failure-mode table.

## Scope boundaries

In scope: pack build / push / pull / unpack, snapshot binding, hash
verification, audit chain, DVD-R local-dir output, multi-pack
splitting.

Out of scope (future tickets if needed):
- Multipart upload (single-pack > 5 GB on DO Spaces / AWS).
- Streaming pack builder (lift the in-memory tar ceiling, target
  compressed-bytes cap).
- Stale-pack GC.
- Range-fetch partial pack pulls.
- Server-side encryption (KMS / SSE-S3); mesh ciphertext on public
  bucket is the v1 confidentiality path.
- Cross-provider replication.

## Status

In progress. Code lands incrementally on `main`.

**Pack format v2 (2026-05-26):** v1 (chunks-only) was correctly flagged
by fox as under-engineered — packs contained only chunk bodies, not the
shard tables a new peer needs to actually bootstrap. v2 ships every
load-bearing table inside the same tar.zst alongside the chunk blobs.
Tables go in as `tables/<name>.jsonl` (array-per-line columnar JSONL).
The `edges` table goes through a fan-in restructure at pack-build time
(group by `dst_uri + edge_type + anchor`, ship `src_roots` as an array)
which yields ~5-10× compression vs the live-schema row layout. Live
SQLite schema is unchanged; the restore step expands fan-in rows back
to the per-edge form. Pack uploads NO LONGER append a
`cold_pack_pushed` audit event (would leak into the next push's dumped
metadata and break the "two writers at the same corpus state produce
the same pack_hash" determinism property — the bucket / disc file IS
the receipt). 24 tests pass including a new `test_push_pack_v2_hydrates_fresh_empty_db`
that round-trips push from a populated DB into a completely empty DB.

**Sizing (v2, current shards):** ~2.1 GB per shard pack compressed
(chunk content 1.78 GB + tables ~0.3 GB), ~8.5 GB total bucket
footprint for new-peer-ready packs across 4 shards.

Per-peer state NOT shipped: mesh_*, selfmodel_*, capital_ledger,
memory_*, controller_events, fork_score_branches, adapter_loss_reports,
falsifications, schema_meta, meta. FTS5 (chunks_fts*, documents_fts*)
not shipped either — rebuilt from chunks.content + documents.title on
unpack.
