wallet/fts-sidecar: real FTS5 in the cloud sidecar; delete custom BM25

Replaces the hand-rolled BM25 sidecar (arborist/wallet/sidecar.py, ~690
LOC) with a slim SQLite file that just COPIES the source shard's FTS5
shadow tables verbatim + minimal doc/chunk metadata. Cloud retrieval
then runs SQLite FTS5 bm25() on the same bytes the local shard uses —
bit-for-bit parity by construction. 5/5 source + audit_mode agreement
on the smoke fixture between local corpus-query and cloud-query against
the new manifest-fts.json.

Why
---
Custom binary sidecar was per-document BM25; main encyclopedia articles
got length-normalized so hard that on "why did the dinosaurs go extinct?"
"Edwina, the Dinosaur Who Didn't Know She Was Extinct" beat "Dinosaur"
(measured cloud-vs-local divergence). Local FTS5 indexes per-chunk so
each chunk is a moderate-length doc and the main article wins multiply.
Different granularity, not a tuning knob — fix is to use the same
indexer cloud-side.

What ships
----------
- arborist/wallet/fts_sidecar_build.py — builder. ATTACH source shard,
  copy documents (root/uri/title only), copy chunks (id/root/idx/leaf
  only, NO content), CREATE VIRTUAL TABLE chunks_fts/documents_fts with
  same DDL as source, bulk-copy the four shadow tables verbatim,
  VACUUM. 8.78 GB shard → 2.15 GB sidecar (24.5%) in ~45 s; full
  4-shard wiki corpus 37.4 GB → 8.1 GB (21.7%) in ~3 min.
- arborist/wallet/bucket.py: FtsSidecarShardClient — downloads slim
  sidecar once into ~/.arborist/sidecar-fts-cache/<hash>.idx.db, opens
  read-only sqlite3 (check_same_thread=False for parallel shard fan-
  out), runs FTS5 MATCH locally. Chunk content fetches via blobs/<hash>
  with HTTP-range big-shard fallback when blobs aren't published.
  MultiShardSidecarCorpus simplified to fts_sidecar_url ∨ bucket-direct
  (both are FTS5 backends; merge by raw bm25 MIN ascending).
- arborist/qa/corpus.py: SidecarBucketCorpus.higher_is_better=False
  (FTS5 bm25 is negative, lower=better). chunks_for_doc dispatches on
  fetch_chunk_body attr for the slim-FTS5 client. apply_title_boost
  imports tokenizer helpers from new arborist/qa/_text_norm.py.
- arborist/qa/_text_norm.py — fold_accents, numeral_expand,
  tokenize_text, STOPWORDS — extracted from the deleted sidecar.py so
  apply_title_boost keeps its lexical shape.
- arborist/cli.py: `arborist sidecar build-fts` subcommand; old
  `sidecar build`/`sidecar search` removed. cloud_query recognizes
  fts_sidecar_url + sidecar_url alike.
- Makefile: `sidecar-build-fts` + `sidecar-build-fts-all` targets;
  `sidecar-build` + `sidecar-search` removed.
- scripts/upload_fts_sidecars.py — boto3 producer: uploads slim
  sidecars to clones/sidecars-fts/<n>.idx.db, publishes
  clones/manifest-fts.json (4 wikipedia shards inherit existing
  shard_url for content fallback; ACL public-read; idempotent on
  size match). Existing manifest-sidecar.json untouched.
- tests/test_qa_corpus_functional.py + test_qa_corpus_integration.py
  converted from build_sidecar → build_fts_sidecar; 6 fixtures pass.
- bench/slim_fts_parity_bench.py — local 3-way bench
  (legacy/corpus/slim_fts) over the smoke fixture.

Validation
----------
- Per-shard FTS5 parity: slim sidecar returns IDENTICAL rowids + bm25
  scores to the source shard for top-10 of "dinosaurs extinct".
- 3-way bench (legacy local / corpus local / slim-FTS5 over real
  bucket fallback): 5/5 source agreement AND 5/5 audit_mode agreement
  between corpus and slim_fts. Q5 legacy disagreement (Edwina vs
  Dinosaur) is the pre-existing 2000-line query() retrieval quirk,
  unrelated.
- End-to-end cloud query against published manifest-fts.json (cold-
  start, ~149 s sidecar download once): STRICT · Dinosaur, every
  quote verified (2/2).
- Full pytest suite: 2737 passed, 28 skipped, 1 xfailed. One pre-
  existing failure (tests/test_doc_counts.py — claim_pack docs row-
  count drift) and one pre-existing cold_object failure, both reproduce
  on main HEAD.

Bucket state
------------
- s3://arborist/clones/sidecars-fts/00[0-3].idx.db (8.1 GB) — new
- s3://arborist/clones/manifest-fts.json — new
- s3://arborist/clones/manifest-sidecar.json — kept live (deprecated
  but still readable; downstream callers should switch to
  manifest-fts.json)
This commit is contained in:
russell@unturf.com 2026-05-31 10:43:03 -04:00
parent 2eea5b5655
commit d9fb6a9b69
No known key found for this signature in database
13 changed files with 1087 additions and 984 deletions

View file

@ -1604,11 +1604,6 @@ cloud-snapshot-root: bootstrap ## compute snapshot_root of the bucket-resident s
@echo "# shard: $(SHARD_URL)" >&2
@$(ARBORIST) cloud snapshot-root --shard-url "$(SHARD_URL)" --cache-mb $(or $(CACHE_MB),32)
sidecar-build: bootstrap ## build HTTP-optimized inverted-index sidecar [SHARD=path OUT=path]
@test -n "$(SHARD)" || { echo 'usage: make sidecar-build SHARD=/path/to/shard.db OUT=/path/to/sidecar.bin'; exit 2; }
@test -n "$(OUT)" || { echo 'OUT required'; exit 2; }
@$(ARBORIST) sidecar build --shard "$(SHARD)" --out "$(OUT)"
sidecar-build-fts: bootstrap ## build slim FTS5 sidecar (.idx.db) from a shard [SHARD=path OUT=path]
@test -n "$(SHARD)" || { echo 'usage: make sidecar-build-fts SHARD=/path/to/shard.db OUT=/path/to/shard.idx.db'; exit 2; }
@test -n "$(OUT)" || { echo 'OUT required'; exit 2; }
@ -1626,11 +1621,6 @@ sidecar-build-fts-all: bootstrap ## build slim FTS5 sidecar for every shard in S
done
@ls -lh "$$OUT_DIR"/*.idx.db 2>/dev/null || true
sidecar-search: bootstrap ## query a local sidecar.bin [Q="..." SIDECAR=/path/sidecar.bin LIMIT=N MODE=or|and]
@test -n "$(Q)" || { echo 'usage: make sidecar-search Q="your question" SIDECAR=/path/sidecar.bin [LIMIT=N] [MODE=or|and]'; exit 2; }
@test -n "$(SIDECAR)" || { echo 'SIDECAR=/path/sidecar.bin required'; exit 2; }
@$(ARBORIST) sidecar search '$(Q)' --sidecar "$(SIDECAR)" --limit $(or $(LIMIT),8) --mode $(or $(MODE),or)
cloud-fetch-chunk: bootstrap ## fetch + hash-verify one chunk from a bucket [LEAF_HASH=hex BLOB_BASE=https://...]
@test -n "$(LEAF_HASH)" || { echo 'usage: make cloud-fetch-chunk LEAF_HASH=hex BLOB_BASE=https://.../blobs'; exit 2; }
@test -n "$(BLOB_BASE)" || { echo 'BLOB_BASE required (per-chunk blobs/<hash> base URL)'; exit 2; }