docs/cold-object-store: rewrite recovery section for the hardened path
- Hydrating a new peer: M-aware `make cold-hydrate` (serial default, bulk-tuned restore, parallel FTS rebuild from content, cold verify self-check) — replaces the stale single-shard `cold unpack` loop. - New "FTS: rebuild, don't restore" with the measured rebuild-vs-restore comparison (~5 min rebuild + no extra download vs +4.76 GB for a dead pack-restored index) and WHY the pack restore is dead: FTS5's _data rowid-1 "structure" record is left empty because INSERT OR IGNORE collides with the freshly-created vtable's empty header — a headless index over orphaned segments (count looks right, MATCH returns 0). - Invariants: FTS shadow tables are rebuilt on the consumer, not shipped (cold pack defaults to --no-fts). - Failure modes: cold verify's zero-filled-content + dead-FTS classes.
This commit is contained in:
parent
6d2a75d80b
commit
dfcd132017
1 changed files with 76 additions and 18 deletions
|
|
@ -59,6 +59,12 @@ backup consumers download packs whole.
|
|||
`falsifications`, `schema_meta`, `meta`, `chunks_fts*`,
|
||||
`documents_fts*`.
|
||||
|
||||
The FTS5 shadow tables (`chunks_fts*` / `documents_fts*`) are
|
||||
**rebuilt on the consumer** from chunk content after hydrate, not
|
||||
shipped — the index is derived, and a row-level FTS-pack restore
|
||||
produces a non-functional (headless) index anyway. `cold pack`
|
||||
defaults to `--no-fts`; see "FTS: rebuild, don't restore".
|
||||
|
||||
2. **`pack_hash = hash_leaf(manifest_bytes)`.** The manifest is sorted
|
||||
by `leaf_hash` and deduped before hashing, so input order and
|
||||
accidental duplicates don't move the hash. Two writers producing the
|
||||
|
|
@ -165,27 +171,76 @@ Cloudflare R2, Backblaze B2, GCS S3-interop, MinIO.
|
|||
|
||||
## Hydrating a new peer from CDN
|
||||
|
||||
Genesis recovery is **M-aware** — every document routes to
|
||||
`shard_for_document(root, M)`, the same deterministic function the
|
||||
producer used — and is driven by one make target:
|
||||
|
||||
```bash
|
||||
# 1. On the fresh node, install arborist + the [object-store] extra.
|
||||
make bootstrap-object-store
|
||||
|
||||
# 2. List packs the publisher made available.
|
||||
ARBORIST_COLD_ENDPOINT_URL=... ARBORIST_COLD_BUCKET=... \
|
||||
arborist cold stats
|
||||
|
||||
# 3. For each pack, unpack into a local shard. Verifies every chunk on
|
||||
# the way in; bad bytes from a hostile CDN never reach the DB.
|
||||
for hash in <pack-hashes>; do
|
||||
arborist --db ~/.arborist/shards/000.db cold unpack $hash
|
||||
done
|
||||
|
||||
# 4. (Optional) Pin which corpus state we're at.
|
||||
arborist --db ~/.arborist/shards/000.db snapshot list | head -1
|
||||
# On the fresh node: install arborist + the [object-store] extra, set
|
||||
# the bucket env (ARBORIST_COLD_BUCKET / _ENDPOINT_URL) + boto3 creds,
|
||||
# then hydrate the full corpus into M target shards:
|
||||
make cold-hydrate HYDRATE_DIR=~/.arborist/shards HYDRATE_M=4
|
||||
```
|
||||
|
||||
The snapshot_root the publisher pinned at pack time is in the audit row;
|
||||
the verifier on the consumer side recomputes `snapshot_root` after
|
||||
unpack and they should match if the corpus is a clean restore.
|
||||
What it does, in order:
|
||||
|
||||
1. **Pull + restore metadata and chunk content — serially.** Each
|
||||
metadata pack routes its rows into *all M shared target shards*, so
|
||||
parallel workers contend on the same files (and an unbounded restore
|
||||
transaction blows up RAM). The hydrate runs **serial by default**
|
||||
(`COLD_PACK_JOBS=1`; override at your own risk). The restore is
|
||||
bulk-tuned — 512 MB page cache applied *before* the heavy edge
|
||||
fan-out, plus bounded incremental commits — so it runs in
|
||||
~8–20 min/pack, not the hours an untuned restore took (the edge
|
||||
fan-out into an indexed table thrashes a default ~2 MB cache).
|
||||
|
||||
2. **Rebuild FTS from content — in parallel** (`cold rebuild-fts`, one
|
||||
process per shard). See "FTS: rebuild, don't restore" below.
|
||||
|
||||
3. **Self-verify** (`cold verify`). Every non-empty shard must have
|
||||
materialized chunk content (not zero-filled placeholders) **and** a
|
||||
searchable FTS index. A bad recovery exits non-zero here — it fails
|
||||
**loudly** instead of silently serving empty results. Run it any
|
||||
time: `arborist cold verify --shards-dir ~/.arborist/shards`.
|
||||
|
||||
The publisher's pinned `snapshot_root` is in each pack's audit row; the
|
||||
consumer recomputes it after unpack and they match on a clean restore.
|
||||
|
||||
### FTS: rebuild, don't restore
|
||||
|
||||
FTS is a deterministic function of chunk content, and the chunk packs
|
||||
already ship that content — so the consumer **rebuilds** the index
|
||||
rather than downloading a prebuilt one. `cold pack` therefore defaults
|
||||
to `--no-fts`. Measured 2026-05-29 on the live 4-shard / 6.2M-chunk
|
||||
corpus:
|
||||
|
||||
| path | extra download | consumer work | result |
|
||||
|-----------------------------------|----------------|------------------------|-----------------------|
|
||||
| rebuild from content *(default)* | none | ~5 min (4-way parallel)| working index |
|
||||
| restore prebuilt FTS pack | +4.76 GB | ~24 s/shard | dead index *(today)* |
|
||||
|
||||
Restore skips re-tokenization so it's cheaper on CPU, but it adds ~20 %
|
||||
to the download (FTS packs were 4.76 GB of a 23.8 GB bucket) **and the
|
||||
shipped FTS-pack restore is non-functional** — rows present, `MATCH`
|
||||
returns 0. So rebuild is the correct, smaller default; a *fixed* FTS
|
||||
pack would only pay off for a slow-CPU / fast-link consumer (e.g. a
|
||||
mobile SPV peer). `--with-fts` opts back in (not recommended until the
|
||||
shadow-table restore is fixed).
|
||||
|
||||
**Why the pack restore is dead (headless index).** An FTS5 index is not
|
||||
a row set you can dump-and-reinsert. Its shadow tables (`_data`, `_idx`,
|
||||
`_docsize`, `_config`) are mutually-dependent internal state. In
|
||||
particular, `_data` **rowid 1 is the segment "structure" record** —
|
||||
FTS5 reads it first to learn which segments exist. The pack restore does
|
||||
`CREATE VIRTUAL TABLE … USING fts5(…)` (which auto-writes an *empty*
|
||||
structure record at id 1) and then bulk-loads the dumped shadow rows
|
||||
with `INSERT OR IGNORE`. The pack's real structure record collides with
|
||||
the fresh empty one on the primary key and is **silently dropped**, so
|
||||
the index keeps an empty "0 segments" header over a full body of
|
||||
orphaned segments — `count(*)` (via `_docsize`) looks right, every
|
||||
`MATCH` returns 0. Faithful FTS5 transport requires copying the whole
|
||||
DB file or re-inserting content; row-level shadow-table dumps don't
|
||||
round-trip. We re-insert content (`cold rebuild-fts`).
|
||||
|
||||
## DVD-R archival workflow
|
||||
|
||||
|
|
@ -271,6 +326,9 @@ re-pack cadence (operational policy).
|
|||
| `cold pack` produces no packs | No hot chunks with non-null content | `cold pack` operates on local content. Confirm shard isn't empty / fully evicted. |
|
||||
| Peer's snapshot_root differs from pack's | Local corpus drifted after unpack (ingest, falsification, etc.) | Expected. Pack is a delayed snapshot; the peer has moved on. Re-pack to re-baseline. |
|
||||
| Bucket missing a pack | GC'd, never uploaded, wrong bucket | Re-build pack from any shard that still has the source content. |
|
||||
| `cold verify` fails: content zero-filled | Phase-2 chunk-body fill never completed (crashed/interrupted restore, or `--just-enough` mode) | Re-run `make cold-hydrate` (serial, full mode). Pre-sized zeroblob placeholders are present but never overwritten. |
|
||||
| `cold verify` fails: FTS dead (`MATCH`=0) | FTS pack restored a headless index, or the rebuild step didn't run | `arborist cold rebuild-fts --shards-dir DIR` (rebuilds from content). |
|
||||
| Search empty despite docs present | Dead / missing FTS index | `cold verify` to confirm, then `cold rebuild-fts`. |
|
||||
|
||||
## Future work
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue