cold-recovery: fix FTS-pack restore (headless index), verify-gate rebuild
The FTS pack restore produced a DEAD index — segments present, MATCH=0 — because the verbatim shadow-table copy used INSERT OR IGNORE, so the pack's real `_data` rowid-1 "structure" record lost the primary-key conflict to the empty one `CREATE VIRTUAL TABLE` seeds, leaving a "0 segments" header over orphaned segments. Fix (evict.py): clear the seeded rows, then copy verbatim (DELETE + INSERT ... SELECT), so each fts5 shadow table becomes a byte-for-byte copy of the producer's index and the real structure record survives. Validated on the 3090: restore one fts pack, NO rebuild -> MATCH 'anarchism'=1189 / 'the'=1.42M (identical to rebuild-from-content); `_data` id=1 structure record non-empty. make cold-hydrate: rebuild FTS only when the restored index isn't already searchable (cold verify-gated). Shipping FTS packs now makes recovery fast (~24s/shard restore, skip the ~5min rebuild); dropping them (--no-fts default) keeps the bucket small. Either way cold verify gates success. docs/cold-object-store: FTS packs restore correctly now; documented the restore-vs-rebuild tradeoff, per-consumer guidance, and the fixed bug.
This commit is contained in:
parent
dfcd132017
commit
1547259163
3 changed files with 64 additions and 44 deletions
12
Makefile
12
Makefile
|
|
@ -979,11 +979,15 @@ cold-hydrate: bootstrap ## genesis a fresh peer from cloud, M-aware: pull every
|
|||
--hydrate-shards-dir $(HYDRATE_DIR) \
|
||||
--hydrate-M $(HYDRATE_M) \
|
||||
$$MODE_FLAG
|
||||
@if [ "$${HYDRATE_MODE:-full}" = "full" ] && [ "$${HYDRATE_REBUILD_FTS:-1}" != "0" ]; then \
|
||||
echo ">> rebuilding FTS from content (parallel; cold-pack FTS restores a dead index, so always rebuild)"; \
|
||||
$(ARBORIST) cold rebuild-fts --shards-dir $(HYDRATE_DIR); \
|
||||
@if [ "$${HYDRATE_MODE:-full}" = "full" ] && [ "$${HYDRATE_REBUILD_FTS:-auto}" != "0" ]; then \
|
||||
if [ "$${HYDRATE_REBUILD_FTS:-auto}" = "1" ] || ! $(ARBORIST) cold verify --shards-dir $(HYDRATE_DIR) >/dev/null 2>&1; then \
|
||||
echo ">> FTS not searchable (no fts pack, or rebuild forced) → rebuilding from content (parallel)"; \
|
||||
$(ARBORIST) cold rebuild-fts --shards-dir $(HYDRATE_DIR); \
|
||||
else \
|
||||
echo ">> FTS restored from packs and verified searchable — skipping rebuild"; \
|
||||
fi; \
|
||||
fi
|
||||
@echo ">> verifying recovery (content materialized + searchable; fails loudly if not)"
|
||||
@echo ">> final verify (content materialized + searchable; fails loudly if not)"
|
||||
@$(ARBORIST) cold verify --shards-dir $(HYDRATE_DIR)
|
||||
@echo ">> hydration complete:"
|
||||
@ls -lh $(HYDRATE_DIR)/*.db 2>/dev/null
|
||||
|
|
|
|||
|
|
@ -1013,9 +1013,17 @@ def _pull_fts_pack_into_targets(
|
|||
).fetchone()
|
||||
if exists is None:
|
||||
continue
|
||||
# Copy the shadow table VERBATIM. CREATE VIRTUAL TABLE
|
||||
# already seeded these (e.g. chunks_fts_data id=1 = the
|
||||
# empty segment "structure" record, id=10 = averages),
|
||||
# so a plain INSERT OR IGNORE would drop the pack's REAL
|
||||
# structure record on the PK conflict and leave a
|
||||
# headless index (segments present, MATCH returns 0).
|
||||
# Clear the seed rows first, then insert, so main.<tbl>
|
||||
# becomes a byte-for-byte copy of the producer's index.
|
||||
target.execute(f"DELETE FROM main.{tbl}")
|
||||
target.execute(
|
||||
f"INSERT OR IGNORE INTO main.{tbl} "
|
||||
f"SELECT * FROM fts_src.{tbl}"
|
||||
f"INSERT INTO main.{tbl} SELECT * FROM fts_src.{tbl}"
|
||||
)
|
||||
total_rows += target.execute(
|
||||
"SELECT changes()"
|
||||
|
|
|
|||
|
|
@ -60,10 +60,10 @@ backup consumers download packs whole.
|
|||
`documents_fts*`.
|
||||
|
||||
The FTS5 shadow tables (`chunks_fts*` / `documents_fts*`) are
|
||||
**rebuilt on the consumer** from chunk content after hydrate, not
|
||||
shipped — the index is derived, and a row-level FTS-pack restore
|
||||
produces a non-functional (headless) index anyway. `cold pack`
|
||||
defaults to `--no-fts`; see "FTS: rebuild, don't restore".
|
||||
**derived** from chunk content, so `cold pack` defaults to `--no-fts`
|
||||
and the consumer regenerates them on hydrate. They CAN be shipped
|
||||
(`--with-fts`) and restored faithfully (~24 s/shard, no re-tokenize)
|
||||
for constrained consumers — see "FTS: restore or rebuild".
|
||||
|
||||
2. **`pack_hash = hash_leaf(manifest_bytes)`.** The manifest is sorted
|
||||
by `leaf_hash` and deduped before hashing, so input order and
|
||||
|
|
@ -194,8 +194,11 @@ What it does, in order:
|
|||
~8–20 min/pack, not the hours an untuned restore took (the edge
|
||||
fan-out into an indexed table thrashes a default ~2 MB cache).
|
||||
|
||||
2. **Rebuild FTS from content — in parallel** (`cold rebuild-fts`, one
|
||||
process per shard). See "FTS: rebuild, don't restore" below.
|
||||
2. **Get a searchable FTS index.** If the packs shipped FTS, it was
|
||||
restored during step 1 (fast); otherwise rebuild from content in
|
||||
parallel (`cold rebuild-fts`, one process per shard). This is
|
||||
`cold verify`-gated, so the rebuild only runs when the index isn't
|
||||
already searchable. See "FTS: restore or rebuild" below.
|
||||
|
||||
3. **Self-verify** (`cold verify`). Every non-empty shard must have
|
||||
materialized chunk content (not zero-filled placeholders) **and** a
|
||||
|
|
@ -206,41 +209,46 @@ What it does, in order:
|
|||
The publisher's pinned `snapshot_root` is in each pack's audit row; the
|
||||
consumer recomputes it after unpack and they match on a clean restore.
|
||||
|
||||
### FTS: rebuild, don't restore
|
||||
### FTS: restore or rebuild (both work)
|
||||
|
||||
FTS is a deterministic function of chunk content, and the chunk packs
|
||||
already ship that content — so the consumer **rebuilds** the index
|
||||
rather than downloading a prebuilt one. `cold pack` therefore defaults
|
||||
to `--no-fts`. Measured 2026-05-29 on the live 4-shard / 6.2M-chunk
|
||||
corpus:
|
||||
FTS is a deterministic function of chunk content, so a fresh peer has
|
||||
two ways to get a searchable index after hydrate. Measured 2026-05-29 on
|
||||
the live 4-shard / 6.2M-chunk corpus:
|
||||
|
||||
| path | extra download | consumer work | result |
|
||||
|-----------------------------------|----------------|------------------------|-----------------------|
|
||||
| rebuild from content *(default)* | none | ~5 min (4-way parallel)| working index |
|
||||
| restore prebuilt FTS pack | +4.76 GB | ~24 s/shard | dead index *(today)* |
|
||||
| path | extra download | consumer work | result |
|
||||
|----------------------------|----------------|-------------------------|---------------|
|
||||
| restore prebuilt FTS pack | +4.76 GB | ~24 s/shard | working index |
|
||||
| rebuild from content | none | ~5 min (4-way parallel) | working index |
|
||||
|
||||
Restore skips re-tokenization so it's cheaper on CPU, but it adds ~20 %
|
||||
to the download (FTS packs were 4.76 GB of a 23.8 GB bucket) **and the
|
||||
shipped FTS-pack restore is non-functional** — rows present, `MATCH`
|
||||
returns 0. So rebuild is the correct, smaller default; a *fixed* FTS
|
||||
pack would only pay off for a slow-CPU / fast-link consumer (e.g. a
|
||||
mobile SPV peer). `--with-fts` opts back in (not recommended until the
|
||||
shadow-table restore is fixed).
|
||||
Both produce a byte-identical, queryable index (identical `MATCH`
|
||||
counts). Restore skips re-tokenization (~12× less consumer CPU) but adds
|
||||
~20 % to the download (FTS packs were 4.76 GB of a 23.8 GB bucket). So:
|
||||
|
||||
**Why the pack restore is dead (headless index).** An FTS5 index is not
|
||||
a row set you can dump-and-reinsert. Its shadow tables (`_data`, `_idx`,
|
||||
`_docsize`, `_config`) are mutually-dependent internal state. In
|
||||
particular, `_data` **rowid 1 is the segment "structure" record** —
|
||||
FTS5 reads it first to learn which segments exist. The pack restore does
|
||||
`CREATE VIRTUAL TABLE … USING fts5(…)` (which auto-writes an *empty*
|
||||
structure record at id 1) and then bulk-loads the dumped shadow rows
|
||||
with `INSERT OR IGNORE`. The pack's real structure record collides with
|
||||
the fresh empty one on the primary key and is **silently dropped**, so
|
||||
the index keeps an empty "0 segments" header over a full body of
|
||||
orphaned segments — `count(*)` (via `_docsize`) looks right, every
|
||||
`MATCH` returns 0. Faithful FTS5 transport requires copying the whole
|
||||
DB file or re-inserting content; row-level shadow-table dumps don't
|
||||
round-trip. We re-insert content (`cold rebuild-fts`).
|
||||
- **Bandwidth-sensitive / capable consumer** (the producer's own hosts):
|
||||
drop FTS packs — `cold pack --no-fts` (the default) — and rebuild on
|
||||
hydrate.
|
||||
- **Constrained consumer** (slow CPU, mobile / SPV peer): ship FTS packs
|
||||
— `cold pack --with-fts` — and restore them, skipping the rebuild.
|
||||
|
||||
`make cold-hydrate` picks automatically: it runs `cold verify` and only
|
||||
rebuilds when the restored index isn't already searchable. So shipping
|
||||
FTS packs makes recovery fast; dropping them makes the bucket small;
|
||||
either way the result is verified before the run reports success.
|
||||
|
||||
**Faithful FTS5 transport (the headless-index bug, fixed 2026-05-29).**
|
||||
An FTS5 index is not a plain row set — its shadow tables (`_data`,
|
||||
`_idx`, `_docsize`, `_config`) are mutually-dependent internal state, and
|
||||
`_data` **rowid 1 is the segment "structure" record** FTS5 reads first to
|
||||
learn which segments exist. The restore copies those shadow tables
|
||||
verbatim (correct in principle), but `CREATE VIRTUAL TABLE` first seeds
|
||||
an *empty* structure record at id 1, and the old restore used
|
||||
`INSERT OR IGNORE` — so the pack's real structure record lost the PK
|
||||
conflict and was silently dropped, leaving an empty "0 segments" header
|
||||
over a full body of orphaned segments (`count(*)` via `_docsize` looked
|
||||
right; every `MATCH` returned 0). **Fix:** clear the seeded rows then
|
||||
copy verbatim (`DELETE FROM main.<shadow>; INSERT … SELECT * FROM
|
||||
fts_src.<shadow>`), so each shadow table becomes a byte-for-byte copy of
|
||||
the producer's index and the real structure record survives.
|
||||
|
||||
## DVD-R archival workflow
|
||||
|
||||
|
|
@ -327,7 +335,7 @@ re-pack cadence (operational policy).
|
|||
| Peer's snapshot_root differs from pack's | Local corpus drifted after unpack (ingest, falsification, etc.) | Expected. Pack is a delayed snapshot; the peer has moved on. Re-pack to re-baseline. |
|
||||
| Bucket missing a pack | GC'd, never uploaded, wrong bucket | Re-build pack from any shard that still has the source content. |
|
||||
| `cold verify` fails: content zero-filled | Phase-2 chunk-body fill never completed (crashed/interrupted restore, or `--just-enough` mode) | Re-run `make cold-hydrate` (serial, full mode). Pre-sized zeroblob placeholders are present but never overwritten. |
|
||||
| `cold verify` fails: FTS dead (`MATCH`=0) | FTS pack restored a headless index, or the rebuild step didn't run | `arborist cold rebuild-fts --shards-dir DIR` (rebuilds from content). |
|
||||
| `cold verify` fails: FTS dead (`MATCH`=0) | No fts pack shipped and the rebuild didn't run (or a pre-2026-05-29 headless-restore) | `arborist cold rebuild-fts --shards-dir DIR` (rebuilds from content). |
|
||||
| Search empty despite docs present | Dead / missing FTS index | `cold verify` to confirm, then `cold rebuild-fts`. |
|
||||
|
||||
## Future work
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue