engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

cold-recovery: fix FTS-pack restore (headless index), verify-gate rebuild

Browse source 
The FTS pack restore produced a DEAD index — segments present, MATCH=0 —
because the verbatim shadow-table copy used INSERT OR IGNORE, so the
pack's real `_data` rowid-1 "structure" record lost the primary-key
conflict to the empty one `CREATE VIRTUAL TABLE` seeds, leaving a "0
segments" header over orphaned segments. Fix (evict.py): clear the
seeded rows, then copy verbatim (DELETE + INSERT ... SELECT), so each
fts5 shadow table becomes a byte-for-byte copy of the producer's index
and the real structure record survives.

Validated on the 3090: restore one fts pack, NO rebuild -> MATCH
'anarchism'=1189 / 'the'=1.42M (identical to rebuild-from-content);
`_data` id=1 structure record non-empty.

make cold-hydrate: rebuild FTS only when the restored index isn't
already searchable (cold verify-gated). Shipping FTS packs now makes
recovery fast (~24s/shard restore, skip the ~5min rebuild); dropping
them (--no-fts default) keeps the bucket small. Either way cold verify
gates success.

docs/cold-object-store: FTS packs restore correctly now; documented the
restore-vs-rebuild tradeoff, per-consumer guidance, and the fixed bug.
This commit is contained in:
russell@unturf.com 2026-05-29 14:36:29 -04:00
parent dfcd132017
commit 1547259163
No known key found for this signature in database
3 changed files with 64 additions and 44 deletions
Download patch file Download diff file Expand all files Collapse all files

12
Makefile
View file

@ -979,11 +979,15 @@ cold-hydrate: bootstrap ## genesis a fresh peer from cloud, M-aware: pull every
--hydrate-shards-dir $(HYDRATE_DIR) \
--hydrate-M $(HYDRATE_M) \
$$MODE_FLAG
@if [ "$${HYDRATE_MODE:-full}" = "full" ] && [ "$${HYDRATE_REBUILD_FTS:-1}" != "0" ]; then \
echo ">> rebuilding FTS from content (parallel; cold-pack FTS restores a dead index, so always rebuild)"; \
$(ARBORIST) cold rebuild-fts --shards-dir $(HYDRATE_DIR); \
@if [ "$${HYDRATE_MODE:-full}" = "full" ] && [ "$${HYDRATE_REBUILD_FTS:-auto}" != "0" ]; then \
if [ "$${HYDRATE_REBUILD_FTS:-auto}" = "1" ] || ! $(ARBORIST) cold verify --shards-dir $(HYDRATE_DIR) >/dev/null 2>&1; then \
echo ">> FTS not searchable (no fts pack, or rebuild forced) → rebuilding from content (parallel)"; \
$(ARBORIST) cold rebuild-fts --shards-dir $(HYDRATE_DIR); \
else \
echo ">> FTS restored from packs and verified searchable — skipping rebuild"; \
fi; \
fi
@echo ">> verifying recovery (content materialized + searchable; fails loudly if not)"
@echo ">> final verify (content materialized + searchable; fails loudly if not)"
@$(ARBORIST) cold verify --shards-dir $(HYDRATE_DIR)
@echo ">> hydration complete:"
@ls -lh $(HYDRATE_DIR)/*.db 2>/dev/null