engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

cold-recovery hardening: drop FTS packs, parallel rebuild, loud self-check

Browse source 
Make cold recovery correct-by-default and fail-loud, closing the class
that silently produced a corrupt, unqueryable corpus (2026-05-29):

- cold rebuild-fts now runs in PARALLEL (one process per shard; separate
  files, no contention) and clears-first, so it also repairs the dead
  index a cold-pack FTS restore leaves. Replaces the serial post-pass.
- new `cold verify`: self-check a hydrated shard set — chunk content
  materialized (not zero-filled) AND FTS searchable (MATCH a word taken
  from sampled content). Non-zero exit if any shard fails. Validated: it
  passes the good recovery and fails the corrupt genesis-test, catching
  both the zero-filled-bodies and dead-FTS classes.
- cold pack defaults to --no-fts (FTS is derived from content and the
  FTS pack restore is non-functional anyway); --with-fts to opt back in.
- make cold-hydrate: serial by default (M-aware hydrate routes every
  pack into all M shared target shards, so parallel workers contend —
  #54); always rebuild FTS from content (drop the broken shard-000-only
  gate); run `cold verify` at the end so a bad recovery fails loudly.
This commit is contained in:
russell@unturf.com 2026-05-29 14:10:09 -04:00
parent 7f7eeefeb9
commit 6d2a75d80b
No known key found for this signature in database
2 changed files with 153 additions and 45 deletions
Download patch file Download diff file Expand all files Collapse all files

16
Makefile
View file

@ -965,7 +965,8 @@ cold-hydrate: bootstrap ## genesis a fresh peer from cloud, M-aware: pull every
just-enough) MODE_FLAG="--just-enough" ;; \
*) echo "HYDRATE_MODE=$$MODE invalid (full | just-enough)"; exit 2 ;; \
esac; \
$(COLD_PACK_PICK_JOBS); \
JOBS="$(COLD_PACK_JOBS)"; [ -z "$$JOBS" ] && JOBS=1; \
echo ">> COLD_PACK_JOBS=$$JOBS (serial default: M-aware hydrate routes every pack into all M shared target shards, so parallel workers contend on the same files — #54. Override COLD_PACK_JOBS=N at your own risk.)"; \
echo ">> discovering metadata packs in bucket"; \
hashes=$$($(ARBORIST) cold list --no-manifest \
| $(VENV)/bin/python -c \
@ -978,15 +979,12 @@ cold-hydrate: bootstrap ## genesis a fresh peer from cloud, M-aware: pull every
--hydrate-shards-dir $(HYDRATE_DIR) \
--hydrate-M $(HYDRATE_M) \
$$MODE_FLAG
@if [ "$${HYDRATE_MODE:-full}" = "full" ] && [ "$${HYDRATE_REBUILD_FTS:-auto}" != "0" ]; then \
has_fts=$$(python3 -c "import sqlite3; c=sqlite3.connect('$(HYDRATE_DIR)/000.db'); n=c.execute('SELECT COUNT(*) FROM chunks_fts_data').fetchone()[0]; print(n)" 2>/dev/null); \
if [ "$${HYDRATE_REBUILD_FTS:-auto}" = "1" ] || [ -z "$$has_fts" ] || [ "$$has_fts" = "0" ]; then \
echo ">> chunks_fts_data empty on shard 000 → serial FTS5 rebuild"; \
$(ARBORIST) cold rebuild-fts --shards-dir $(HYDRATE_DIR); \
else \
echo ">> chunks_fts_data already populated (fts pack restored) — skipping rebuild"; \
fi; \
@if [ "$${HYDRATE_MODE:-full}" = "full" ] && [ "$${HYDRATE_REBUILD_FTS:-1}" != "0" ]; then \
echo ">> rebuilding FTS from content (parallel; cold-pack FTS restores a dead index, so always rebuild)"; \
$(ARBORIST) cold rebuild-fts --shards-dir $(HYDRATE_DIR); \
fi
@echo ">> verifying recovery (content materialized + searchable; fails loudly if not)"
@$(ARBORIST) cold verify --shards-dir $(HYDRATE_DIR)
@echo ">> hydration complete:"
@ls -lh $(HYDRATE_DIR)/*.db 2>/dev/null