engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000067 phase 2: 3rd "fts" pack kind for skip-rebuild hydrate

Browse source 
Each producer shard now optionally emits a THIRD pack alongside its
metadata and chunks packs: an "fts" pack containing the FTS5 shadow
tables (chunks_fts_data, chunks_fts_idx, chunks_fts_docsize,
chunks_fts_config + the documents_fts_* counterparts) packed as a
fresh SQLite file inside the tar so BLOB columns round-trip natively.

Consumer detects fts_pack_hashes in the metadata pack's manifest,
pulls each fts pack, ATTACHes the embedded sqlite, INSERTs every
shadow-table row into its target's empty shadow tables, and SKIPS
the local FTS rebuild entirely.

Producer side:
  arborist/cold_object.py
    + PACK_KIND_FTS = "fts"
    + FTS_SHADOW_TABLES tuple (8 shadow tables)
    + build_fts_pack(src_db_path, ...)
        creates a temp sqlite, applies SCHEMA_SQL (so destination
        has FTS virtual tables → shadow tables auto-created), copies
        every shadow-table row from src via cursor iteration, packs
        the sqlite file into tar.zst
    + ParsedManifest.fts_pack_hashes
    + parse_manifest reads _fts_pack_hashes records
    + build_metadata_pack accepts fts_pack_hashes parameter and
      writes the new manifest record
  arborist/evict.py:push_pack
    + include_fts: bool = True parameter (CLI --no-fts opts out)
    + Phase B.5 emits the fts pack BEFORE Phase C (metadata pack)
      so its hash can be referenced in the metadata manifest

Consumer side:
  arborist/evict.py
    + _pull_fts_pack_into_targets() — pulls fts pack body, extracts
      embedded sqlite, ATTACHes into each target, INSERT OR IGNORE
      every shadow-table row. INSERT OR IGNORE protects against
      rowid collisions on other targets that don't own these chunks.
    + hydrate_from_metadata_pack_routed iterates fts_pack_hashes in
      full mode, calls _pull_fts_pack_into_targets per pack
    + _pull_pack_inner_routed returns fts_pack_hashes_referenced in
      its result dict (mirrors chunk_pack_hashes_referenced)

CLI / Makefile:
  arborist cold pack --no-fts                     (opt-out)
  make cold-hydrate                                (auto-detects: if
                                                   chunks_fts_data is
                                                   already populated
                                                   on shard 000 after
                                                   unpack, skip the
                                                   rebuild post-pass)
  make cold-hydrate HYDRATE_REBUILD_FTS=1          (force rebuild)
  make cold-hydrate HYDRATE_REBUILD_FTS=0          (skip rebuild)

Schema:
  cold_pending.kind CHECK extended to include 'fts'
  pack_key() accepts kind="fts" → packs/<hash>.fts.tar.zst

Expected wall-time impact on the 3090 genesis bench:
  with fts in packs:  no rebuild step → ~5-10 min total wall
  without fts:        rebuild post-pass needed → ~15-20 min

Trade-off: ~30-50% larger bucket (FTS shadow data per shard) for
~70-90% faster consumer hydrate. Producer flips the trade via
--no-fts. The fts pack is optional in the manifest (empty list →
consumer falls back to rebuild) so old bucket data without fts
packs continues to work unchanged.

34 cold-unpack-routed + migrate + planner tests pass.
This commit is contained in:
russell@unturf.com 2026-05-26 19:13:01 -04:00
parent cfb5666ef5
commit 2c11435f7f
No known key found for this signature in database
8 changed files with 3165 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

11
Makefile
View file

@ -961,9 +961,14 @@ cold-hydrate: bootstrap ## genesis a fresh peer from cloud, M-aware: pull every
--hydrate-shards-dir $(HYDRATE_DIR) \
--hydrate-M $(HYDRATE_M) \
$$MODE_FLAG
@if [ "$${HYDRATE_MODE:-full}" = "full" ] && [ "$${HYDRATE_REBUILD_FTS:-1}" = "1" ]; then \
echo ">> serial FTS5 rebuild (each shard gets full FS-journal bandwidth in turn)"; \
$(ARBORIST) cold rebuild-fts --shards-dir $(HYDRATE_DIR); \
@if [ "$${HYDRATE_MODE:-full}" = "full" ] && [ "$${HYDRATE_REBUILD_FTS:-auto}" != "0" ]; then \
has_fts=$$(python3 -c "import sqlite3; c=sqlite3.connect('$(HYDRATE_DIR)/000.db'); n=c.execute('SELECT COUNT(*) FROM chunks_fts_data').fetchone()[0]; print(n)" 2>/dev/null); \
if [ "$${HYDRATE_REBUILD_FTS:-auto}" = "1" ] || [ -z "$$has_fts" ] || [ "$$has_fts" = "0" ]; then \
echo ">> chunks_fts_data empty on shard 000 → serial FTS5 rebuild"; \
$(ARBORIST) cold rebuild-fts --shards-dir $(HYDRATE_DIR); \
else \
echo ">> chunks_fts_data already populated (fts pack restored) — skipping rebuild"; \
fi; \
fi
@echo ">> hydration complete:"
@ls -lh $(HYDRATE_DIR)/*.db 2>/dev/null