engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000067: defer FTS5 to serial post-pass + bulk-load SQLite tunings

Browse source 
Real-measured bottleneck during the 3090 genesis run (2026-05-26):
all 4 parallel workers sitting in jbd2_log_wait_commit /
do_get_write_access — fighting for ext4's single filesystem journal.
The reshard executor was fast because it ran FTS5 rebuild
SEQUENTIALLY (one process per target in turn); the parallel 4-way
unpack collapsed that into journal contention.

Two coupled changes:

1. arborist/evict.py — _pull_pack_inner_routed
   * Drop the _rebuild_fts_on_target call from phase 2c entirely.
     Parallel inner loop now does: download → metadata route →
     chunks fill. NO FTS5 writes during the hot parallel phase.
   * Bulk-load PRAGMA tuning on every target connection:
       synchronous=OFF      no fsync per commit (already had this)
       journal_mode=MEMORY  WAL in RAM, not on disk (was WAL)
       temp_store=MEMORY    sort scratch in RAM
       cache_size=-524288   512 MB page cache per connection
       mmap_size=536870912  512 MB read mmap
     Genesis crash recoverability = re-pull from bucket, so
     durability of intermediate state has no value — these tunings
     trade durability for throughput.

2. arborist/cli.py — new `arborist cold rebuild-fts --shards-dir DIR`
   subcommand. Sequentially rebuilds chunks_fts + documents_fts on
   every shard, one at a time. Each shard gets the full filesystem
   journal in its turn. Total wall = sum of single-shard FTS
   rebuild times, NOT 4× contention.

3. Makefile — `cold-hydrate` chains the FTS rebuild automatically
   after the parallel unpack (only when HYDRATE_MODE=full, since
   just-enough has no chunk bodies to index anyway). Operator can
   skip with HYDRATE_REBUILD_FTS=0 for a 2-pass workflow.

Expected wall time on 3090:
  parallel hydrate (download + chunks fill):  ~5-10 min
  serial FTS rebuild × 4 shards:              ~5-15 min total
  total:                                       ~15-25 min
vs the killed 2.5-hour run.

34 existing cold-unpack-routed + migrate + planner tests still pass
(the tests' FTS check runs against the reshard path, which still
calls _rebuild_fts_on_target inside the executor — that path is one
process, no parallel contention).
This commit is contained in:
russell@unturf.com 2026-05-26 18:49:53 -04:00
parent 939d3ced78
commit cfb5666ef5
No known key found for this signature in database
3 changed files with 87 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

4
Makefile
View file

@ -961,6 +961,10 @@ cold-hydrate: bootstrap ## genesis a fresh peer from cloud, M-aware: pull every
--hydrate-shards-dir $(HYDRATE_DIR) \
--hydrate-M $(HYDRATE_M) \
$$MODE_FLAG
@if [ "$${HYDRATE_MODE:-full}" = "full" ] && [ "$${HYDRATE_REBUILD_FTS:-1}" = "1" ]; then \
echo ">> serial FTS5 rebuild (each shard gets full FS-journal bandwidth in turn)"; \
$(ARBORIST) cold rebuild-fts --shards-dir $(HYDRATE_DIR); \
fi
@echo ">> hydration complete:"
@ls -lh $(HYDRATE_DIR)/*.db 2>/dev/null