arborist/docs/cold-object-store.md

# Cold-pack distribution tier (ticket #000061)

A point-in-time corpus distribution mechanism. arborist serializes its
local chunks into `tar.zst` packs, ships them to an S3-compatible bucket
(and/or to local disk for DVD-burning), and any new peer hydrates by
downloading those packs from the bucket's CDN edge and unpacking them
into a fresh shard.

## What this is, and what it is not

**Is:** a backup-and-distribution unit. Pack bytes are content-addressed.
Same chunk set on two writers → same `pack_hash`. The bucket is a
delivery medium for a *delayed* snapshot of the corpus — repackaging
after falsifications produces a new pack with a new hash.

**Is not:** a live mirror. Packs do not see falsifications that happen
*after* the pack was built. They do not see ingests after the pack was
built. They are frozen artifacts, identified by `snapshot_root` of the
corpus state at pack time.

**Is not:** an individual-chunk fetch tier. There is no per-chunk URL in
the bucket — corpus chunks live exclusively inside packs. New peers and
backup consumers download packs whole.

## Hard invariants

1. **Bucket holds packs only.** Layout:

   ```
   <bucket>/packs/<pack_hash>.tar.zst              # pack body
   <bucket>/packs/<pack_hash>.manifest.ndjson      # pack contents sidecar
   ```

   No `blobs/` prefix, no per-chunk objects. (One pack ↔ one disc ↔ one
   bucket object.)

   Pack contents (v2 format, self-sufficient for new-peer hydration):
   ```
   manifest.ndjson                              # chunk leaf_hash + size catalog
   tables/documents.jsonl                       # array-per-line, sorted
   tables/chunks.jsonl                          # without content column
   tables/merkle_nodes.jsonl
   tables/edges.jsonl                           # FAN-IN restructured
   tables/audit_events.jsonl
   tables/derivations.jsonl
   tables/concept_relations.jsonl
   tables/concept_token_idf.jsonl
   tables/providence_cache.jsonl
   tables/citation_aliases.jsonl
   tables/term_aliases.jsonl
   tables/snapshots.jsonl
   tables/document_http_meta.jsonl
   blobs/<hash[:2]>/<hash[2:]>                  # raw UTF-8 chunk bodies
   ```

   Not shipped (per-peer or rebuildable):
   `mesh_*`, `selfmodel_*`, `capital_ledger`, `controller_events`,
   `fork_score_branches`, `memory_*`, `adapter_loss_reports`,
   `falsifications`, `schema_meta`, `meta`, `chunks_fts*`,
   `documents_fts*`.

2. **`pack_hash = hash_leaf(manifest_bytes)`.** The manifest is sorted
   by `leaf_hash` and deduped before hashing, so input order and
   accidental duplicates don't move the hash. Two writers producing the
   same chunk set produce the same `pack_hash` — bucket upload is
   idempotent, DVD burns at two sites are byte-identical.

3. **`hash_leaf(chunk_body).hex() == leaf_hash`** is verified on every
   `open_pack` member. The pack's tar member name is `blobs/<hash[:2]>/<hash[2:]>` —
   that's a within-tar convention, not a bucket layout. Tampering with
   pack bytes is caught at unpack time, never reaches the local DB.

4. **Every pack pins a `snapshot_root`.** Pack creation reads the
   corpus's current snapshot root (`arborist/snapshot.py:compute_snapshot_root`)
   and records it in:
   - the audit row (`cold_pack_pushed.body.snapshot_root`)
   - the `push_pack` return body
   - the local-dir filenames implicitly (pack_hash itself encodes the
     manifest, which encodes the chunk set, which encodes that snapshot's
     content)

   Consumers can run `arborist snapshot verify <root>` after unpack to
   detect drift between the pack and the corpus state on the consuming
   node.

5. **Cores never evict** (CLAUDE.md rule). Packs include cores AND
   surfaces — cores carry the distillation derivations a new peer needs
   to bootstrap the v9.8 chain.

6. **No credentials in audit body.** Backend identity is endpoint URL +
   bucket name only. Credentials live in env vars / `~/.aws/credentials`
   via standard boto3 discovery — Operation Voyeur.

## Delayed snapshots and falsifications

Packs are not live. Between two pack runs, three things can happen:

1. **New ingest.** `ingest_source` adds new documents. They aren't in
   the old pack; they show up in the next pack. The old pack stays a
   valid snapshot of *its* state.

2. **Falsification.** Drift detection, `arborist falsify`, or
   `rehydrate_drift` flips a `providence_cache` row to
   `falsification_state='stale'` and/or marks a document for
   re-derivation. Chunk content does NOT change (chunks are immutable;
   content-addressed). A new pack covers the same chunk *bytes* but with
   a different `providence_cache` view.

3. **Re-pack.** A new pack run reads the current corpus and produces a
   pack with a new `pack_hash` (because the manifest covers a different
   chunk set — newly ingested, possibly with the same hashes minus any
   superseded ones).

Three operational consequences:

- **Stale packs accumulate.** Old `pack_hash`es stay in the bucket
  until explicitly garbage-collected. They're still valid snapshots
  of past corpus states. There's no automatic cleanup; that's a future
  ticket.
- **A peer hydrated from an old pack is honestly old.** It has the
  corpus state from the pack's `snapshot_root`. To catch up, it
  follows the same path any live peer does — ingest new sources,
  receive falsification events on the mesh, re-derive cores.
- **The bucket is eventually consistent with intent**, not with the
  live corpus. Re-pack cadence (daily? weekly? per-event?) is an
  operational policy, not a code property.

## Two distribution channels — same artifact

The same `.tar.zst` file serves two channels:

| Channel             | Transport                  | Default cap   |
|---------------------|----------------------------|---------------|
| **Bucket + CDN**    | `S3CompatibleBackend.put_pack` → public-read DO Spaces / R2 / S3, CDN edge serves consumers | 4.4 GB / pack |
| **DVD-R archival**  | `--local-dir DIR` → `growisofs -dvd-compat -Z /dev/sr0=<pack>` | 4.4 GB / pack |

Pack files are byte-identical between channels. A DVD burned from one
local-dir pack and a CDN-fetched pack of the same content collide on
`sha256sum`.

## DO Spaces quickstart

```bash
# 1. Install the optional backend.
make bootstrap-object-store

# 2. Set boto3 standard env vars (never hard-code in scripts).
export AWS_ACCESS_KEY_ID=<your-spaces-key>
export AWS_SECRET_ACCESS_KEY=<your-spaces-secret>

# 3. Set bucket config.
export ARBORIST_COLD_ENDPOINT_URL=https://nyc3.digitaloceanspaces.com
export ARBORIST_COLD_BUCKET=arborist-corpus

# 4. Build packs and push them. Default cap = 4.4 GB / pack (DVD-R safe-
# fit). One shard typically yields 1-3 packs.
make cold-pack

# 5. Confirm what's in the bucket.
make cold-stats
```

Same flow works on AWS S3 (`endpoint_url=https://s3.<region>.amazonaws.com`),
Cloudflare R2, Backblaze B2, GCS S3-interop, MinIO.

## Hydrating a new peer from CDN

```bash
# 1. On the fresh node, install arborist + the [object-store] extra.
make bootstrap-object-store

# 2. List packs the publisher made available.
ARBORIST_COLD_ENDPOINT_URL=... ARBORIST_COLD_BUCKET=... \
    arborist cold stats

# 3. For each pack, unpack into a local shard. Verifies every chunk on
# the way in; bad bytes from a hostile CDN never reach the DB.
for hash in <pack-hashes>; do
    arborist --db ~/.arborist/shards/000.db cold unpack $hash
done

# 4. (Optional) Pin which corpus state we're at.
arborist --db ~/.arborist/shards/000.db snapshot list | head -1
```

The snapshot_root the publisher pinned at pack time is in the audit row;
the verifier on the consumer side recomputes `snapshot_root` after
unpack and they should match if the corpus is a clean restore.

## DVD-R archival workflow

```bash
# 1. Write packs to a staging dir; skip the bucket entirely.
make cold-pack-dvd LOCAL_DIR=/mnt/dvd-staging

# 2. Each pack is one disc. Burn with growisofs.
for pack in /mnt/dvd-staging/arborist-pack-*.tar.zst; do
    growisofs -dvd-compat -Z /dev/sr0="$pack"
    # ... eject, insert next blank, repeat ...
done

# 3. On a fresh node, copy a pack from disc and unpack:
mount /dev/sr0 /mnt/dvd
arborist --db fresh.db cold unpack \
    "$(basename /mnt/dvd/arborist-pack-*.tar.zst .tar.zst | cut -d- -f3)"
```

The `pack_hash` is in the filename (`arborist-pack-<hash[:16]>.tar.zst`)
so the disc itself is self-describing — no separate index needed.

## Pack-size cap — fit on a 4.7 GB DVD-R, safely

Default cap is **4,400,000,000 bytes (4.4 GB, ~6.5 % buffer below the
4.7 GB marketing capacity)**. Targeting 4.7 GB directly is unsafe:
filesystem overhead, media manufacturing variance, growisofs
lead-in/lead-out, and older drives refusing the outer edge all eat
into nominal capacity. 4.4 GB sits between the industry-standard tool
defaults (HandBrake DVD-5 = 4,377 MiB ≈ 4.59 GB; DVDFab fit-to-DVD-5 =
4.3 GB; mkisofs default DVD = 4,377 MiB).

The cap applies to *compressed* bytes per pack. `stream_packs` uses
streaming zstd compression and peeks the compressed-buffer size after
every chunk (via `FLUSH_BLOCK`, which preserves the compressor's
dictionary so block boundaries cost almost nothing in ratio). When the
buffer reaches the cap, the pack is finalized and a new one starts. So
each disc fills to ~4.4 GB of recorded data, not 30–50 % of capacity.

Overshoot bound: tar trailer (~1 KB padding) + zstd frame footer (~10 B)
get emitted after the last in-loop size check, so actual compressed
size can land at cap + ~2 KB. Trivial for a 4.4 GB cap.

For larger media:

| Media                | `--max-pack-bytes`        | Marketing |
|----------------------|---------------------------|-----------|
| **DVD-R (default)**  | `4_400_000_000` (4.4 GB)  | 4.7 GB    |
| DVD+R DL             | `8_000_000_000` (8.0 GB)  | 8.5 GB    |
| BD-R                 | `24_000_000_000` (24 GB)  | 25 GB     |
| BD-R DL              | `48_000_000_000` (48 GB)  | 50 GB     |

## Cost model (DO Spaces, current corpus)

Pack format v2 (self-sufficient for new-peer hydration). Numbers measured
2026-05-26 against the live 4-shard corpus (14.1M total chunks; the
1.56M hot-content chunks per shard go into packs; metadata is added on
top via the v2 dump path):

| Path                          | Count       | Storage  | Cost                          |
|-------------------------------|-------------|----------|-------------------------------|
| v2 pack storage (per shard)   | 1 pack      | ~2.1 GB  | —                             |
| v2 pack storage (all 4)       | 4 packs     | ~8.5 GB  | $0.17/mo (@ $0.02/GB)         |
| Full-corpus hydrate (CDN)     | ~4 GETs     | —        | ~$0.00002 in requests         |
| Egress (in-region)            | 0           | —        | $0                            |
| Egress (CDN to public)        | 8.5 GB/peer | —        | $0.09 per fresh peer (@ $0.01/GB) |

The v1 chunks-only format produced ~1.78 GB per shard (7.1 GB total).
v2 adds ~0.3–0.4 GB per shard for the metadata tables (chunks-meta,
documents, audit_events, merkle_nodes, edges fan-in restructured, plus
small tables). Trade: ~20 % more storage for a self-sufficient pack
that a fresh peer can unpack into a working shard with no other inputs.

Repacking after a falsification event costs the same as the initial
pack — one full corpus serialization per event-batched run, gated by
re-pack cadence (operational policy).

## Failure modes

| Symptom                                  | Cause                              | Recovery                             |
|------------------------------------------|------------------------------------|--------------------------------------|
| `pack chunk hash mismatch` on unpack     | Pack bytes corrupted in transit or on disc | Re-download / re-burn; pack is content-addressed so a fresh fetch is verifiable. |
| `cold pack` produces no packs            | No hot chunks with non-null content | `cold pack` operates on local content. Confirm shard isn't empty / fully evicted. |
| Peer's snapshot_root differs from pack's | Local corpus drifted after unpack (ingest, falsification, etc.) | Expected. Pack is a delayed snapshot; the peer has moved on. Re-pack to re-baseline. |
| Bucket missing a pack                    | GC'd, never uploaded, wrong bucket | Re-build pack from any shard that still has the source content. |

## Future work

- **Multipart upload for packs.** Provider single-object limits (DO
  Spaces = 5 GB non-multipart, AWS S3 = 5 GB; both support multipart up
  to 5 TB). Today's code uses `put_object` which is single-shot. boto3
  `upload_file` is the one-line drop-in.
- **Streaming pack builder.** ✅ Landed as `stream_packs`. Caps target
  compressed bytes; each disc fills. `build_pack` stays for tests +
  small/known-set callers.
- **Pack GC.** Stale packs (those whose `snapshot_root` is older than N
  re-pack cycles) get bucket-deleted automatically.
- **Range-fetch partial pack pulls.** Manifest carries offsets;
  `GET .tar.zst Range: bytes=X-Y` would let a consumer pull one chunk
  from a huge pack without downloading the whole thing.
- **KMS / SSE-S3.** Server-side encryption (mesh ciphertext on a
  public bucket is the v1 confidentiality path).
- **Multi-region replication.** Handled by the provider within a region;
  cross-provider replication is a separate distribution-policy question.