arborist/docs/cold-object-store.md

Cold-pack distribution tier (ticket #000061)

A point-in-time corpus distribution mechanism. arborist serializes its local chunks into tar.zst packs, ships them to an S3-compatible bucket (and/or to local disk for DVD-burning), and any new peer hydrates by downloading those packs from the bucket's CDN edge and unpacking them into a fresh shard.

What this is, and what it is not

Is: a backup-and-distribution unit. Pack bytes are content-addressed. Same chunk set on two writers → same pack_hash. The bucket is a delivery medium for a delayed snapshot of the corpus — repackaging after falsifications produces a new pack with a new hash.

Is not: a live mirror. Packs do not see falsifications that happen after the pack was built. They do not see ingests after the pack was built. They are frozen artifacts, identified by snapshot_root of the corpus state at pack time.

Is not: an individual-chunk fetch tier. There is no per-chunk URL in the bucket — corpus chunks live exclusively inside packs. New peers and backup consumers download packs whole.

Hard invariants

1. Bucket holds packs only. Layout:

   <bucket>/packs/<pack_hash>.tar.zst              # pack body
   <bucket>/packs/<pack_hash>.manifest.ndjson      # pack contents sidecar
   

   No blobs/ prefix, no per-chunk objects. (One pack ↔ one disc ↔ one bucket object.)

   Pack contents (v2 format, self-sufficient for new-peer hydration):

   manifest.ndjson                              # chunk leaf_hash + size catalog
   tables/documents.jsonl                       # array-per-line, sorted
   tables/chunks.jsonl                          # without content column
   tables/merkle_nodes.jsonl
   tables/edges.jsonl                           # FAN-IN restructured
   tables/audit_events.jsonl
   tables/derivations.jsonl
   tables/concept_relations.jsonl
   tables/concept_token_idf.jsonl
   tables/providence_cache.jsonl
   tables/citation_aliases.jsonl
   tables/term_aliases.jsonl
   tables/snapshots.jsonl
   tables/document_http_meta.jsonl
   blobs/<hash[:2]>/<hash[2:]>                  # raw UTF-8 chunk bodies
   

   Not shipped (per-peer or rebuildable): mesh_*, selfmodel_*, capital_ledger, controller_events, fork_score_branches, memory_*, adapter_loss_reports, falsifications, schema_meta, meta, chunks_fts*, documents_fts*.

2. pack_hash = hash_leaf(manifest_bytes). The manifest is sorted by leaf_hash and deduped before hashing, so input order and accidental duplicates don't move the hash. Two writers producing the same chunk set produce the same pack_hash — bucket upload is idempotent, DVD burns at two sites are byte-identical.

3. hash_leaf(chunk_body).hex() == leaf_hash is verified on every open_pack member. The pack's tar member name is blobs/<hash[:2]>/<hash[2:]> — that's a within-tar convention, not a bucket layout. Tampering with pack bytes is caught at unpack time, never reaches the local DB.

4. Every pack pins a snapshot_root. Pack creation reads the corpus's current snapshot root (arborist/snapshot.py:compute_snapshot_root) and records it in:

   • the audit row (cold_pack_pushed.body.snapshot_root)
   • the push_pack return body
   • the local-dir filenames implicitly (pack_hash itself encodes the manifest, which encodes the chunk set, which encodes that snapshot's content)

   Consumers can run arborist snapshot verify <root> after unpack to detect drift between the pack and the corpus state on the consuming node.

5. Cores never evict (CLAUDE.md rule). Packs include cores AND surfaces — cores carry the distillation derivations a new peer needs to bootstrap the v9.8 chain.

6. No credentials in audit body. Backend identity is endpoint URL + bucket name only. Credentials live in env vars / ~/.aws/credentials via standard boto3 discovery — Operation Voyeur.

Delayed snapshots and falsifications

Packs are not live. Between two pack runs, three things can happen:

1. New ingest. ingest_source adds new documents. They aren't in the old pack; they show up in the next pack. The old pack stays a valid snapshot of its state.

2. Falsification. Drift detection, arborist falsify, or rehydrate_drift flips a providence_cache row to falsification_state='stale' and/or marks a document for re-derivation. Chunk content does NOT change (chunks are immutable; content-addressed). A new pack covers the same chunk bytes but with a different providence_cache view.

3. Re-pack. A new pack run reads the current corpus and produces a pack with a new pack_hash (because the manifest covers a different chunk set — newly ingested, possibly with the same hashes minus any superseded ones).

Three operational consequences:

• Stale packs accumulate. Old pack_hashes stay in the bucket until explicitly garbage-collected. They're still valid snapshots of past corpus states. There's no automatic cleanup; that's a future ticket.
• A peer hydrated from an old pack is honestly old. It has the corpus state from the pack's snapshot_root. To catch up, it follows the same path any live peer does — ingest new sources, receive falsification events on the mesh, re-derive cores.
• The bucket is eventually consistent with intent, not with the live corpus. Re-pack cadence (daily? weekly? per-event?) is an operational policy, not a code property.

Two distribution channels — same artifact

The same .tar.zst file serves two channels:

Channel	Transport	Default cap
Bucket + CDN	S3CompatibleBackend.put_pack → public-read DO Spaces / R2 / S3, CDN edge serves consumers	4.4 GB / pack
DVD-R archival	--local-dir DIR → growisofs -dvd-compat -Z /dev/sr0=<pack>	4.4 GB / pack

Pack files are byte-identical between channels. A DVD burned from one local-dir pack and a CDN-fetched pack of the same content collide on sha256sum.

DO Spaces quickstart

# 1. Install the optional backend.
make bootstrap-object-store

# 2. Set boto3 standard env vars (never hard-code in scripts).
export AWS_ACCESS_KEY_ID=<your-spaces-key>
export AWS_SECRET_ACCESS_KEY=<your-spaces-secret>

# 3. Set bucket config.
export ARBORIST_COLD_ENDPOINT_URL=https://nyc3.digitaloceanspaces.com
export ARBORIST_COLD_BUCKET=arborist-corpus

# 4. Build packs and push them. Default cap = 4.4 GB / pack (DVD-R safe-
# fit). One shard typically yields 1-3 packs.
make cold-pack

# 5. Confirm what's in the bucket.
make cold-stats

Same flow works on AWS S3 (endpoint_url=https://s3.<region>.amazonaws.com), Cloudflare R2, Backblaze B2, GCS S3-interop, MinIO.

Hydrating a new peer from CDN

# 1. On the fresh node, install arborist + the [object-store] extra.
make bootstrap-object-store

# 2. List packs the publisher made available.
ARBORIST_COLD_ENDPOINT_URL=... ARBORIST_COLD_BUCKET=... \
    arborist cold stats

# 3. For each pack, unpack into a local shard. Verifies every chunk on
# the way in; bad bytes from a hostile CDN never reach the DB.
for hash in <pack-hashes>; do
    arborist --db ~/.arborist/shards/000.db cold unpack $hash
done

# 4. (Optional) Pin which corpus state we're at.
arborist --db ~/.arborist/shards/000.db snapshot list | head -1

The snapshot_root the publisher pinned at pack time is in the audit row; the verifier on the consumer side recomputes snapshot_root after unpack and they should match if the corpus is a clean restore.

DVD-R archival workflow

# 1. Write packs to a staging dir; skip the bucket entirely.
make cold-pack-dvd LOCAL_DIR=/mnt/dvd-staging

# 2. Each pack is one disc. Burn with growisofs.
for pack in /mnt/dvd-staging/arborist-pack-*.tar.zst; do
    growisofs -dvd-compat -Z /dev/sr0="$pack"
    # ... eject, insert next blank, repeat ...
done

# 3. On a fresh node, copy a pack from disc and unpack:
mount /dev/sr0 /mnt/dvd
arborist --db fresh.db cold unpack \
    "$(basename /mnt/dvd/arborist-pack-*.tar.zst .tar.zst | cut -d- -f3)"

The pack_hash is in the filename (arborist-pack-<hash[:16]>.tar.zst) so the disc itself is self-describing — no separate index needed.

Pack-size cap — fit on a 4.7 GB DVD-R, safely

Default cap is 4,400,000,000 bytes (4.4 GB, ~6.5 % buffer below the 4.7 GB marketing capacity). Targeting 4.7 GB directly is unsafe: filesystem overhead, media manufacturing variance, growisofs lead-in/lead-out, and older drives refusing the outer edge all eat into nominal capacity. 4.4 GB sits between the industry-standard tool defaults (HandBrake DVD-5 = 4,377 MiB ≈ 4.59 GB; DVDFab fit-to-DVD-5 = 4.3 GB; mkisofs default DVD = 4,377 MiB).

The cap applies to compressed bytes per pack. stream_packs uses streaming zstd compression and peeks the compressed-buffer size after every chunk (via FLUSH_BLOCK, which preserves the compressor's dictionary so block boundaries cost almost nothing in ratio). When the buffer reaches the cap, the pack is finalized and a new one starts. So each disc fills to ~4.4 GB of recorded data, not 30–50 % of capacity.

Overshoot bound: tar trailer (~1 KB padding) + zstd frame footer (~10 B) get emitted after the last in-loop size check, so actual compressed size can land at cap + ~2 KB. Trivial for a 4.4 GB cap.

For larger media:

Media	--max-pack-bytes	Marketing
DVD-R (default)	4_400_000_000 (4.4 GB)	4.7 GB
DVD+R DL	8_000_000_000 (8.0 GB)	8.5 GB
BD-R	24_000_000_000 (24 GB)	25 GB
BD-R DL	48_000_000_000 (48 GB)	50 GB

Cost model (DO Spaces, current corpus)

Pack format v2 (self-sufficient for new-peer hydration). Numbers measured 2026-05-26 against the live 4-shard corpus (14.1M total chunks; the 1.56M hot-content chunks per shard go into packs; metadata is added on top via the v2 dump path):

Path	Count	Storage	Cost
v2 pack storage (per shard)	1 pack	~2.1 GB	—
v2 pack storage (all 4)	4 packs	~8.5 GB	$0.17/mo (@ $0.02/GB)
Full-corpus hydrate (CDN)	~4 GETs	—	~$0.00002 in requests
Egress (in-region)	0	—	$0
Egress (CDN to public)	8.5 GB/peer	—	$0.09 per fresh peer (@ $0.01/GB)

The v1 chunks-only format produced ~1.78 GB per shard (7.1 GB total). v2 adds ~0.3–0.4 GB per shard for the metadata tables (chunks-meta, documents, audit_events, merkle_nodes, edges fan-in restructured, plus small tables). Trade: ~20 % more storage for a self-sufficient pack that a fresh peer can unpack into a working shard with no other inputs.

Repacking after a falsification event costs the same as the initial pack — one full corpus serialization per event-batched run, gated by re-pack cadence (operational policy).

Failure modes

Symptom	Cause	Recovery
pack chunk hash mismatch on unpack	Pack bytes corrupted in transit or on disc	Re-download / re-burn; pack is content-addressed so a fresh fetch is verifiable.
cold pack produces no packs	No hot chunks with non-null content	cold pack operates on local content. Confirm shard isn't empty / fully evicted.
Peer's snapshot_root differs from pack's	Local corpus drifted after unpack (ingest, falsification, etc.)	Expected. Pack is a delayed snapshot; the peer has moved on. Re-pack to re-baseline.
Bucket missing a pack	GC'd, never uploaded, wrong bucket	Re-build pack from any shard that still has the source content.

Future work

• Multipart upload for packs. Provider single-object limits (DO Spaces = 5 GB non-multipart, AWS S3 = 5 GB; both support multipart up to 5 TB). Today's code uses put_object which is single-shot. boto3 upload_file is the one-line drop-in.
• Streaming pack builder. ✅ Landed as stream_packs. Caps target compressed bytes; each disc fills. build_pack stays for tests + small/known-set callers.
• Pack GC. Stale packs (those whose snapshot_root is older than N re-pack cycles) get bucket-deleted automatically.
• Range-fetch partial pack pulls. Manifest carries offsets; GET .tar.zst Range: bytes=X-Y would let a consumer pull one chunk from a huge pack without downloading the whole thing.
• KMS / SSE-S3. Server-side encryption (mesh ciphertext on a public bucket is the v1 confidentiality path).
• Multi-region replication. Handled by the provider within a region; cross-provider replication is a separate distribution-policy question.