arborist/docs/cold-object-store.md

Cold-pack distribution tier (ticket #000061)

A point-in-time corpus distribution mechanism. arborist serializes its local chunks into tar.zst packs, ships them to an S3-compatible bucket (and/or to local disk for DVD-burning), and any new peer hydrates by downloading those packs from the bucket's CDN edge and unpacking them into a fresh shard.

What this is, and what it is not

Is: a backup-and-distribution unit. Pack bytes are content-addressed. Same chunk set on two writers → same pack_hash. The bucket is a delivery medium for a delayed snapshot of the corpus — repackaging after falsifications produces a new pack with a new hash.

Is not: a live mirror. Packs do not see falsifications that happen after the pack was built. They do not see ingests after the pack was built. They are frozen artifacts, identified by snapshot_root of the corpus state at pack time.

Is not: an individual-chunk fetch tier. There is no per-chunk URL in the bucket — corpus chunks live exclusively inside packs. New peers and backup consumers download packs whole.

Hard invariants

1. Bucket holds packs by default; blobs/ is opt-in for JIT. Layout:

   <bucket>/packs/<pack_hash>.tar.zst              # pack body
   <bucket>/packs/<pack_hash>.manifest.ndjson      # pack contents sidecar
   <bucket>/blobs/<hash[:2]>/<hash[2:]>            # per-chunk body — opt-in,
                                                   # populated only by
                                                   # `cold pack --jit-blobs`
                                                   # (online-JIT consumer flow)
   <bucket>/clones/<snapshot_id>/<NNN>.db          # raw shard clones — opt-in,
                                                   # populated only by
                                                   # `cold stream-snapshot`
                                                   # (Tier A raw-clone flow)
   

   The classic pack flow (DVD-burn channel) keeps one pack ↔ one disc ↔ one bucket object. The two opt-in prefixes (blobs/, clones/) light up additional consumer flows; see "Three consumer modes" below.

   Pack contents (v2 format, self-sufficient for new-peer hydration):

   manifest.ndjson                              # chunk leaf_hash + size catalog
   tables/documents.jsonl                       # array-per-line, sorted
   tables/chunks.jsonl                          # without content column
   tables/merkle_nodes.jsonl
   tables/edges.jsonl                           # FAN-IN restructured
   tables/audit_events.jsonl
   tables/derivations.jsonl
   tables/concept_relations.jsonl
   tables/concept_token_idf.jsonl
   tables/providence_cache.jsonl
   tables/citation_aliases.jsonl
   tables/term_aliases.jsonl
   tables/snapshots.jsonl
   tables/document_http_meta.jsonl
   blobs/<hash[:2]>/<hash[2:]>                  # raw UTF-8 chunk bodies
   

   Not shipped (per-peer or rebuildable): mesh_*, selfmodel_*, capital_ledger, controller_events, fork_score_branches, memory_*, adapter_loss_reports, falsifications, schema_meta, meta, chunks_fts*, documents_fts*.

   The FTS5 shadow tables (chunks_fts* / documents_fts*) are derived from chunk content, so cold pack defaults to --no-fts and the consumer regenerates them on hydrate. They CAN be shipped (--with-fts) and restored faithfully (~24 s/shard, no re-tokenize) for constrained consumers — see "FTS: restore or rebuild".

2. pack_hash = hash_leaf(manifest_bytes). The manifest is sorted by leaf_hash and deduped before hashing, so input order and accidental duplicates don't move the hash. Two writers producing the same chunk set produce the same pack_hash — bucket upload is idempotent, DVD burns at two sites are byte-identical.

3. hash_leaf(chunk_body).hex() == leaf_hash is verified on every open_pack member. The pack's tar member name is blobs/<hash[:2]>/<hash[2:]> — that's a within-tar convention, not a bucket layout. Tampering with pack bytes is caught at unpack time, never reaches the local DB.

4. Every pack pins a snapshot_root. Pack creation reads the corpus's current snapshot root (arborist/snapshot.py:compute_snapshot_root) and records it in:

   • the audit row (cold_pack_pushed.body.snapshot_root)
   • the push_pack return body
   • the local-dir filenames implicitly (pack_hash itself encodes the manifest, which encodes the chunk set, which encodes that snapshot's content)

   Consumers can run arborist snapshot verify <root> after unpack to detect drift between the pack and the corpus state on the consuming node.

5. Cores never evict (CLAUDE.md rule). Packs include cores AND surfaces — cores carry the distillation derivations a new peer needs to bootstrap the v9.8 chain.

6. No credentials in audit body. Backend identity is endpoint URL + bucket name only. Credentials live in env vars / ~/.aws/credentials via standard boto3 discovery — Operation Voyeur.

Delayed snapshots and falsifications

Packs are not live. Between two pack runs, three things can happen:

1. New ingest. ingest_source adds new documents. They aren't in the old pack; they show up in the next pack. The old pack stays a valid snapshot of its state.

2. Falsification. Drift detection, arborist falsify, or rehydrate_drift flips a providence_cache row to falsification_state='stale' and/or marks a document for re-derivation. Chunk content does NOT change (chunks are immutable; content-addressed). A new pack covers the same chunk bytes but with a different providence_cache view.

3. Re-pack. A new pack run reads the current corpus and produces a pack with a new pack_hash (because the manifest covers a different chunk set — newly ingested, possibly with the same hashes minus any superseded ones).

Three operational consequences:

• Stale packs accumulate. Old pack_hashes stay in the bucket until explicitly garbage-collected. They're still valid snapshots of past corpus states. There's no automatic cleanup; that's a future ticket.
• A peer hydrated from an old pack is honestly old. It has the corpus state from the pack's snapshot_root. To catch up, it follows the same path any live peer does — ingest new sources, receive falsification events on the mesh, re-derive cores.
• The bucket is eventually consistent with intent, not with the live corpus. Re-pack cadence (daily? weekly? per-event?) is an operational policy, not a code property.

Two distribution channels — same artifact

The same .tar.zst file serves two channels:

Channel	Transport	Default cap
Bucket + CDN	S3CompatibleBackend.put_pack → public-read DO Spaces / R2 / S3, CDN edge serves consumers	4.4 GB / pack
DVD-R archival	--local-dir DIR → growisofs -dvd-compat -Z /dev/sr0=<pack>	4.4 GB / pack

Pack files are byte-identical between channels. A DVD burned from one local-dir pack and a CDN-fetched pack of the same content collide on sha256sum.

Three consumer modes

A fresh peer has three ways to hydrate from the bucket; producer mode decides which the bucket supports.

mode	producer	consumer	local disk	bucket bytes pulled	first-query latency
full pack	cold pack (default)	cold unpack --mode full	full corpus (~35 GB)	metadata + every chunk pack	seconds (all data local)
JIT-blobs	cold pack --jit-blobs	cold unpack --mode just-enough + queries with ARBORIST_JIT_CHUNKS=1	metadata-only (~27 GB at FTS+schema floor)	metadata + N×blob per query	one round-trip per chunk on first hit, cached after
raw clone (Tier A)	cold stream-snapshot	cold clone	full corpus (~35 GB)	raw .db files via SQLite Backup API	seconds (all data local)

full-pack mode is the canonical path: hardened, fully tested, FTS restorable or rebuildable. Use for genesis recovery of a self-hosting peer, DVD-archival workflows, and any consumer that wants the full corpus offline-queryable.

JIT-blobs mode trades steady-state download for fastest time-to- queryable. The metadata pack is small; chunk bodies stream in as queries access them. Good for ephemeral nodes, demo VMs, edge servers that only serve a subset of the corpus. The blobs/<hash[:2]>/<hash[2:]> prefix is content-addressed: a chunk that exists in two snapshots costs one bucket object, and producer reuploads are free idempotent no-ops. Failure mode is graceful — hydrate_doc_jit skips missing blobs and hash-mismatches rather than crashing the query.

raw-clone mode is the simplest. cold stream-snapshot runs the SQLite Backup API over each shard to a tempfile and PUTs the bytes; the consumer DOWNLOAD the .db files and is queryable immediately. No compression, no restore phase. Larger bucket footprint than the pack flow but skips the slow metadata-restore step entirely.

# Producer (JIT-blobs mode):
arborist cold pack --jit-blobs --shards-dir ~/.arborist/shards

# Consumer:
arborist cold unpack --mode just-enough <metadata_pack_hash> \
    --hydrate-shards-dir ~/.arborist/shards --hydrate-M 4
export ARBORIST_JIT_CHUNKS=1
arborist query --shards-dir ~/.arborist/shards "what is X?"
# → first hit on each chunk fetches blobs/<hash> from the bucket and
#   caches it back into the local row; subsequent reads are free.

DO Spaces quickstart

# 1. Install the optional backend.
make bootstrap-object-store

# 2. Set boto3 standard env vars (never hard-code in scripts).
export AWS_ACCESS_KEY_ID=<your-spaces-key>
export AWS_SECRET_ACCESS_KEY=<your-spaces-secret>

# 3. Set bucket config.
export ARBORIST_COLD_ENDPOINT_URL=https://nyc3.digitaloceanspaces.com
export ARBORIST_COLD_BUCKET=arborist-corpus

# 4. Build packs and push them. Default cap = 4.4 GB / pack (DVD-R safe-
# fit). One shard typically yields 1-3 packs.
make cold-pack

# 5. Confirm what's in the bucket.
make cold-stats

Same flow works on AWS S3 (endpoint_url=https://s3.<region>.amazonaws.com), Cloudflare R2, Backblaze B2, GCS S3-interop, MinIO.

Hydrating a new peer from CDN

Genesis recovery is M-aware — every document routes to shard_for_document(root, M), the same deterministic function the producer used — and is driven by one make target:

# On the fresh node: install arborist + the [object-store] extra, set
# the bucket env (ARBORIST_COLD_BUCKET / _ENDPOINT_URL) + boto3 creds,
# then hydrate the full corpus into M target shards:
make cold-hydrate HYDRATE_DIR=~/.arborist/shards HYDRATE_M=4

What it does, in order:

1. Pull + restore metadata and chunk content — serially. Each metadata pack routes its rows into all M shared target shards, so parallel workers contend on the same files (and an unbounded restore transaction blows up RAM). The hydrate runs serial by default (COLD_PACK_JOBS=1; override at your own risk). The restore is bulk-tuned — 512 MB page cache applied before the heavy edge fan-out, plus bounded incremental commits — so it runs in ~8–20 min/pack, not the hours an untuned restore took (the edge fan-out into an indexed table thrashes a default ~2 MB cache).

2. Get a searchable FTS index. If the packs shipped FTS, it was restored during step 1 (fast); otherwise rebuild from content in parallel (cold rebuild-fts, one process per shard). This is cold verify-gated, so the rebuild only runs when the index isn't already searchable. See "FTS: restore or rebuild" below.

3. Self-verify (cold verify). Every non-empty shard must have materialized chunk content (not zero-filled placeholders) and a searchable FTS index. A bad recovery exits non-zero here — it fails loudly instead of silently serving empty results. Run it any time: arborist cold verify --shards-dir ~/.arborist/shards.

The publisher's pinned snapshot_root is in each pack's audit row; the consumer recomputes it after unpack and they match on a clean restore.

FTS: restore or rebuild (both work)

FTS is a deterministic function of chunk content, so a fresh peer has two ways to get a searchable index after hydrate. Measured 2026-05-29 on the live 4-shard / 6.2M-chunk corpus:

path	extra download	consumer work	result
restore prebuilt FTS pack	+4.76 GB	~24 s/shard	working index
rebuild from content	none	~5 min (4-way parallel)	working index

Both produce a byte-identical, queryable index (identical MATCH counts). Restore skips re-tokenization (~12× less consumer CPU) but adds ~20 % to the download (FTS packs were 4.76 GB of a 23.8 GB bucket). So:

• Bandwidth-sensitive / capable consumer (the producer's own hosts): drop FTS packs — cold pack --no-fts (the default) — and rebuild on hydrate.
• Constrained consumer (slow CPU, mobile / SPV peer): ship FTS packs — cold pack --with-fts — and restore them, skipping the rebuild.

make cold-hydrate picks automatically: it runs cold verify and only rebuilds when the restored index isn't already searchable. So shipping FTS packs makes recovery fast; dropping them makes the bucket small; either way the result is verified before the run reports success.

Faithful FTS5 transport (the headless-index bug, fixed 2026-05-29). An FTS5 index is not a plain row set — its shadow tables (_data, _idx, _docsize, _config) are mutually-dependent internal state, and _data rowid 1 is the segment "structure" record FTS5 reads first to learn which segments exist. The restore copies those shadow tables verbatim (correct in principle), but CREATE VIRTUAL TABLE first seeds an empty structure record at id 1, and the old restore used INSERT OR IGNORE — so the pack's real structure record lost the PK conflict and was silently dropped, leaving an empty "0 segments" header over a full body of orphaned segments (count(*) via _docsize looked right; every MATCH returned 0). Fix: clear the seeded rows then copy verbatim (DELETE FROM main.<shadow>; INSERT … SELECT * FROM fts_src.<shadow>), so each shadow table becomes a byte-for-byte copy of the producer's index and the real structure record survives.

DVD-R archival workflow

# 1. Write packs to a staging dir; skip the bucket entirely.
make cold-pack-dvd LOCAL_DIR=/mnt/dvd-staging

# 2. Each pack is one disc. Burn with growisofs.
for pack in /mnt/dvd-staging/arborist-pack-*.tar.zst; do
    growisofs -dvd-compat -Z /dev/sr0="$pack"
    # ... eject, insert next blank, repeat ...
done

# 3. On a fresh node, copy a pack from disc and unpack:
mount /dev/sr0 /mnt/dvd
arborist --db fresh.db cold unpack \
    "$(basename /mnt/dvd/arborist-pack-*.tar.zst .tar.zst | cut -d- -f3)"

The pack_hash is in the filename (arborist-pack-<hash[:16]>.tar.zst) so the disc itself is self-describing — no separate index needed.

Pack-size cap — fit on a 4.7 GB DVD-R, safely

Default cap is 4,400,000,000 bytes (4.4 GB, ~6.5 % buffer below the 4.7 GB marketing capacity). Targeting 4.7 GB directly is unsafe: filesystem overhead, media manufacturing variance, growisofs lead-in/lead-out, and older drives refusing the outer edge all eat into nominal capacity. 4.4 GB sits between the industry-standard tool defaults (HandBrake DVD-5 = 4,377 MiB ≈ 4.59 GB; DVDFab fit-to-DVD-5 = 4.3 GB; mkisofs default DVD = 4,377 MiB).

The cap applies to compressed bytes per pack. stream_packs uses streaming zstd compression and peeks the compressed-buffer size after every chunk (via FLUSH_BLOCK, which preserves the compressor's dictionary so block boundaries cost almost nothing in ratio). When the buffer reaches the cap, the pack is finalized and a new one starts. So each disc fills to ~4.4 GB of recorded data, not 30–50 % of capacity.

Overshoot bound: tar trailer (~1 KB padding) + zstd frame footer (~10 B) get emitted after the last in-loop size check, so actual compressed size can land at cap + ~2 KB. Trivial for a 4.4 GB cap.

For larger media:

Media	--max-pack-bytes	Marketing
DVD-R (default)	4_400_000_000 (4.4 GB)	4.7 GB
DVD+R DL	8_000_000_000 (8.0 GB)	8.5 GB
BD-R	24_000_000_000 (24 GB)	25 GB
BD-R DL	48_000_000_000 (48 GB)	50 GB

Cost model (DO Spaces, current corpus)

Pack format v2 (self-sufficient for new-peer hydration). Numbers measured 2026-05-26 against the live 4-shard corpus (14.1M total chunks; the 1.56M hot-content chunks per shard go into packs; metadata is added on top via the v2 dump path):

Path	Count	Storage	Cost
v2 pack storage (per shard)	1 pack	~2.1 GB	—
v2 pack storage (all 4)	4 packs	~8.5 GB	$0.17/mo (@ $0.02/GB)
Full-corpus hydrate (CDN)	~4 GETs	—	~$0.00002 in requests
Egress (in-region)	0	—	$0
Egress (CDN to public)	8.5 GB/peer	—	$0.09 per fresh peer (@ $0.01/GB)

The v1 chunks-only format produced ~1.78 GB per shard (7.1 GB total). v2 adds ~0.3–0.4 GB per shard for the metadata tables (chunks-meta, documents, audit_events, merkle_nodes, edges fan-in restructured, plus small tables). Trade: ~20 % more storage for a self-sufficient pack that a fresh peer can unpack into a working shard with no other inputs.

Repacking after a falsification event costs the same as the initial pack — one full corpus serialization per event-batched run, gated by re-pack cadence (operational policy).

Failure modes

Symptom	Cause	Recovery
pack chunk hash mismatch on unpack	Pack bytes corrupted in transit or on disc	Re-download / re-burn; pack is content-addressed so a fresh fetch is verifiable.
cold pack produces no packs	No hot chunks with non-null content	cold pack operates on local content. Confirm shard isn't empty / fully evicted.
Peer's snapshot_root differs from pack's	Local corpus drifted after unpack (ingest, falsification, etc.)	Expected. Pack is a delayed snapshot; the peer has moved on. Re-pack to re-baseline.
Bucket missing a pack	GC'd, never uploaded, wrong bucket	Re-build pack from any shard that still has the source content.
cold verify fails: content zero-filled	Phase-2 chunk-body fill never completed (crashed/interrupted restore, or --just-enough mode)	Re-run make cold-hydrate (serial, full mode). Pre-sized zeroblob placeholders are present but never overwritten.
cold verify fails: FTS dead (MATCH=0)	No fts pack shipped and the rebuild didn't run (or a pre-2026-05-29 headless-restore)	arborist cold rebuild-fts --shards-dir DIR (rebuilds from content).
Search empty despite docs present	Dead / missing FTS index	cold verify to confirm, then cold rebuild-fts.

Future work

• Multipart upload for packs. Provider single-object limits (DO Spaces = 5 GB non-multipart, AWS S3 = 5 GB; both support multipart up to 5 TB). Today's code uses put_object which is single-shot. boto3 upload_file is the one-line drop-in.
• Streaming pack builder. ✅ Landed as stream_packs. Caps target compressed bytes; each disc fills. build_pack stays for tests + small/known-set callers.
• Pack GC. Stale packs (those whose snapshot_root is older than N re-pack cycles) get bucket-deleted automatically.
• Range-fetch partial pack pulls. Manifest carries offsets; GET .tar.zst Range: bytes=X-Y would let a consumer pull one chunk from a huge pack without downloading the whole thing.
• KMS / SSE-S3. Server-side encryption (mesh ciphertext on a public bucket is the v1 confidentiality path).
• Multi-region replication. Handled by the provider within a region; cross-provider replication is a separate distribution-policy question.