diff --git a/docs/cold-object-store.md b/docs/cold-object-store.md index 4c60518..623300e 100644 --- a/docs/cold-object-store.md +++ b/docs/cold-object-store.md @@ -59,6 +59,12 @@ backup consumers download packs whole. `falsifications`, `schema_meta`, `meta`, `chunks_fts*`, `documents_fts*`. + The FTS5 shadow tables (`chunks_fts*` / `documents_fts*`) are + **rebuilt on the consumer** from chunk content after hydrate, not + shipped — the index is derived, and a row-level FTS-pack restore + produces a non-functional (headless) index anyway. `cold pack` + defaults to `--no-fts`; see "FTS: rebuild, don't restore". + 2. **`pack_hash = hash_leaf(manifest_bytes)`.** The manifest is sorted by `leaf_hash` and deduped before hashing, so input order and accidental duplicates don't move the hash. Two writers producing the @@ -165,27 +171,76 @@ Cloudflare R2, Backblaze B2, GCS S3-interop, MinIO. ## Hydrating a new peer from CDN +Genesis recovery is **M-aware** — every document routes to +`shard_for_document(root, M)`, the same deterministic function the +producer used — and is driven by one make target: + ```bash -# 1. On the fresh node, install arborist + the [object-store] extra. -make bootstrap-object-store - -# 2. List packs the publisher made available. -ARBORIST_COLD_ENDPOINT_URL=... ARBORIST_COLD_BUCKET=... \ - arborist cold stats - -# 3. For each pack, unpack into a local shard. Verifies every chunk on -# the way in; bad bytes from a hostile CDN never reach the DB. -for hash in ; do - arborist --db ~/.arborist/shards/000.db cold unpack $hash -done - -# 4. (Optional) Pin which corpus state we're at. -arborist --db ~/.arborist/shards/000.db snapshot list | head -1 +# On the fresh node: install arborist + the [object-store] extra, set +# the bucket env (ARBORIST_COLD_BUCKET / _ENDPOINT_URL) + boto3 creds, +# then hydrate the full corpus into M target shards: +make cold-hydrate HYDRATE_DIR=~/.arborist/shards HYDRATE_M=4 ``` -The snapshot_root the publisher pinned at pack time is in the audit row; -the verifier on the consumer side recomputes `snapshot_root` after -unpack and they should match if the corpus is a clean restore. +What it does, in order: + +1. **Pull + restore metadata and chunk content — serially.** Each + metadata pack routes its rows into *all M shared target shards*, so + parallel workers contend on the same files (and an unbounded restore + transaction blows up RAM). The hydrate runs **serial by default** + (`COLD_PACK_JOBS=1`; override at your own risk). The restore is + bulk-tuned — 512 MB page cache applied *before* the heavy edge + fan-out, plus bounded incremental commits — so it runs in + ~8–20 min/pack, not the hours an untuned restore took (the edge + fan-out into an indexed table thrashes a default ~2 MB cache). + +2. **Rebuild FTS from content — in parallel** (`cold rebuild-fts`, one + process per shard). See "FTS: rebuild, don't restore" below. + +3. **Self-verify** (`cold verify`). Every non-empty shard must have + materialized chunk content (not zero-filled placeholders) **and** a + searchable FTS index. A bad recovery exits non-zero here — it fails + **loudly** instead of silently serving empty results. Run it any + time: `arborist cold verify --shards-dir ~/.arborist/shards`. + +The publisher's pinned `snapshot_root` is in each pack's audit row; the +consumer recomputes it after unpack and they match on a clean restore. + +### FTS: rebuild, don't restore + +FTS is a deterministic function of chunk content, and the chunk packs +already ship that content — so the consumer **rebuilds** the index +rather than downloading a prebuilt one. `cold pack` therefore defaults +to `--no-fts`. Measured 2026-05-29 on the live 4-shard / 6.2M-chunk +corpus: + +| path | extra download | consumer work | result | +|-----------------------------------|----------------|------------------------|-----------------------| +| rebuild from content *(default)* | none | ~5 min (4-way parallel)| working index | +| restore prebuilt FTS pack | +4.76 GB | ~24 s/shard | dead index *(today)* | + +Restore skips re-tokenization so it's cheaper on CPU, but it adds ~20 % +to the download (FTS packs were 4.76 GB of a 23.8 GB bucket) **and the +shipped FTS-pack restore is non-functional** — rows present, `MATCH` +returns 0. So rebuild is the correct, smaller default; a *fixed* FTS +pack would only pay off for a slow-CPU / fast-link consumer (e.g. a +mobile SPV peer). `--with-fts` opts back in (not recommended until the +shadow-table restore is fixed). + +**Why the pack restore is dead (headless index).** An FTS5 index is not +a row set you can dump-and-reinsert. Its shadow tables (`_data`, `_idx`, +`_docsize`, `_config`) are mutually-dependent internal state. In +particular, `_data` **rowid 1 is the segment "structure" record** — +FTS5 reads it first to learn which segments exist. The pack restore does +`CREATE VIRTUAL TABLE … USING fts5(…)` (which auto-writes an *empty* +structure record at id 1) and then bulk-loads the dumped shadow rows +with `INSERT OR IGNORE`. The pack's real structure record collides with +the fresh empty one on the primary key and is **silently dropped**, so +the index keeps an empty "0 segments" header over a full body of +orphaned segments — `count(*)` (via `_docsize`) looks right, every +`MATCH` returns 0. Faithful FTS5 transport requires copying the whole +DB file or re-inserting content; row-level shadow-table dumps don't +round-trip. We re-insert content (`cold rebuild-fts`). ## DVD-R archival workflow @@ -271,6 +326,9 @@ re-pack cadence (operational policy). | `cold pack` produces no packs | No hot chunks with non-null content | `cold pack` operates on local content. Confirm shard isn't empty / fully evicted. | | Peer's snapshot_root differs from pack's | Local corpus drifted after unpack (ingest, falsification, etc.) | Expected. Pack is a delayed snapshot; the peer has moved on. Re-pack to re-baseline. | | Bucket missing a pack | GC'd, never uploaded, wrong bucket | Re-build pack from any shard that still has the source content. | +| `cold verify` fails: content zero-filled | Phase-2 chunk-body fill never completed (crashed/interrupted restore, or `--just-enough` mode) | Re-run `make cold-hydrate` (serial, full mode). Pre-sized zeroblob placeholders are present but never overwritten. | +| `cold verify` fails: FTS dead (`MATCH`=0) | FTS pack restored a headless index, or the rebuild step didn't run | `arborist cold rebuild-fts --shards-dir DIR` (rebuilds from content). | +| Search empty despite docs present | Dead / missing FTS index | `cold verify` to confirm, then `cold rebuild-fts`. | ## Future work