wallet/fts-sidecar: real FTS5 in the cloud sidecar; delete custom BM25
Replaces the hand-rolled BM25 sidecar (arborist/wallet/sidecar.py, ~690 LOC) with a slim SQLite file that just COPIES the source shard's FTS5 shadow tables verbatim + minimal doc/chunk metadata. Cloud retrieval then runs SQLite FTS5 bm25() on the same bytes the local shard uses — bit-for-bit parity by construction. 5/5 source + audit_mode agreement on the smoke fixture between local corpus-query and cloud-query against the new manifest-fts.json. Why --- Custom binary sidecar was per-document BM25; main encyclopedia articles got length-normalized so hard that on "why did the dinosaurs go extinct?" "Edwina, the Dinosaur Who Didn't Know She Was Extinct" beat "Dinosaur" (measured cloud-vs-local divergence). Local FTS5 indexes per-chunk so each chunk is a moderate-length doc and the main article wins multiply. Different granularity, not a tuning knob — fix is to use the same indexer cloud-side. What ships ---------- - arborist/wallet/fts_sidecar_build.py — builder. ATTACH source shard, copy documents (root/uri/title only), copy chunks (id/root/idx/leaf only, NO content), CREATE VIRTUAL TABLE chunks_fts/documents_fts with same DDL as source, bulk-copy the four shadow tables verbatim, VACUUM. 8.78 GB shard → 2.15 GB sidecar (24.5%) in ~45 s; full 4-shard wiki corpus 37.4 GB → 8.1 GB (21.7%) in ~3 min. - arborist/wallet/bucket.py: FtsSidecarShardClient — downloads slim sidecar once into ~/.arborist/sidecar-fts-cache/<hash>.idx.db, opens read-only sqlite3 (check_same_thread=False for parallel shard fan- out), runs FTS5 MATCH locally. Chunk content fetches via blobs/<hash> with HTTP-range big-shard fallback when blobs aren't published. MultiShardSidecarCorpus simplified to fts_sidecar_url ∨ bucket-direct (both are FTS5 backends; merge by raw bm25 MIN ascending). - arborist/qa/corpus.py: SidecarBucketCorpus.higher_is_better=False (FTS5 bm25 is negative, lower=better). chunks_for_doc dispatches on fetch_chunk_body attr for the slim-FTS5 client. apply_title_boost imports tokenizer helpers from new arborist/qa/_text_norm.py. - arborist/qa/_text_norm.py — fold_accents, numeral_expand, tokenize_text, STOPWORDS — extracted from the deleted sidecar.py so apply_title_boost keeps its lexical shape. - arborist/cli.py: `arborist sidecar build-fts` subcommand; old `sidecar build`/`sidecar search` removed. cloud_query recognizes fts_sidecar_url + sidecar_url alike. - Makefile: `sidecar-build-fts` + `sidecar-build-fts-all` targets; `sidecar-build` + `sidecar-search` removed. - scripts/upload_fts_sidecars.py — boto3 producer: uploads slim sidecars to clones/sidecars-fts/<n>.idx.db, publishes clones/manifest-fts.json (4 wikipedia shards inherit existing shard_url for content fallback; ACL public-read; idempotent on size match). Existing manifest-sidecar.json untouched. - tests/test_qa_corpus_functional.py + test_qa_corpus_integration.py converted from build_sidecar → build_fts_sidecar; 6 fixtures pass. - bench/slim_fts_parity_bench.py — local 3-way bench (legacy/corpus/slim_fts) over the smoke fixture. Validation ---------- - Per-shard FTS5 parity: slim sidecar returns IDENTICAL rowids + bm25 scores to the source shard for top-10 of "dinosaurs extinct". - 3-way bench (legacy local / corpus local / slim-FTS5 over real bucket fallback): 5/5 source agreement AND 5/5 audit_mode agreement between corpus and slim_fts. Q5 legacy disagreement (Edwina vs Dinosaur) is the pre-existing 2000-line query() retrieval quirk, unrelated. - End-to-end cloud query against published manifest-fts.json (cold- start, ~149 s sidecar download once): STRICT · Dinosaur, every quote verified (2/2). - Full pytest suite: 2737 passed, 28 skipped, 1 xfailed. One pre- existing failure (tests/test_doc_counts.py — claim_pack docs row- count drift) and one pre-existing cold_object failure, both reproduce on main HEAD. Bucket state ------------ - s3://arborist/clones/sidecars-fts/00[0-3].idx.db (8.1 GB) — new - s3://arborist/clones/manifest-fts.json — new - s3://arborist/clones/manifest-sidecar.json — kept live (deprecated but still readable; downstream callers should switch to manifest-fts.json)
This commit is contained in:
parent
2eea5b5655
commit
d9fb6a9b69
13 changed files with 1087 additions and 984 deletions
10
Makefile
10
Makefile
|
|
@ -1604,11 +1604,6 @@ cloud-snapshot-root: bootstrap ## compute snapshot_root of the bucket-resident s
|
|||
@echo "# shard: $(SHARD_URL)" >&2
|
||||
@$(ARBORIST) cloud snapshot-root --shard-url "$(SHARD_URL)" --cache-mb $(or $(CACHE_MB),32)
|
||||
|
||||
sidecar-build: bootstrap ## build HTTP-optimized inverted-index sidecar [SHARD=path OUT=path]
|
||||
@test -n "$(SHARD)" || { echo 'usage: make sidecar-build SHARD=/path/to/shard.db OUT=/path/to/sidecar.bin'; exit 2; }
|
||||
@test -n "$(OUT)" || { echo 'OUT required'; exit 2; }
|
||||
@$(ARBORIST) sidecar build --shard "$(SHARD)" --out "$(OUT)"
|
||||
|
||||
sidecar-build-fts: bootstrap ## build slim FTS5 sidecar (.idx.db) from a shard [SHARD=path OUT=path]
|
||||
@test -n "$(SHARD)" || { echo 'usage: make sidecar-build-fts SHARD=/path/to/shard.db OUT=/path/to/shard.idx.db'; exit 2; }
|
||||
@test -n "$(OUT)" || { echo 'OUT required'; exit 2; }
|
||||
|
|
@ -1626,11 +1621,6 @@ sidecar-build-fts-all: bootstrap ## build slim FTS5 sidecar for every shard in S
|
|||
done
|
||||
@ls -lh "$$OUT_DIR"/*.idx.db 2>/dev/null || true
|
||||
|
||||
sidecar-search: bootstrap ## query a local sidecar.bin [Q="..." SIDECAR=/path/sidecar.bin LIMIT=N MODE=or|and]
|
||||
@test -n "$(Q)" || { echo 'usage: make sidecar-search Q="your question" SIDECAR=/path/sidecar.bin [LIMIT=N] [MODE=or|and]'; exit 2; }
|
||||
@test -n "$(SIDECAR)" || { echo 'SIDECAR=/path/sidecar.bin required'; exit 2; }
|
||||
@$(ARBORIST) sidecar search '$(Q)' --sidecar "$(SIDECAR)" --limit $(or $(LIMIT),8) --mode $(or $(MODE),or)
|
||||
|
||||
cloud-fetch-chunk: bootstrap ## fetch + hash-verify one chunk from a bucket [LEAF_HASH=hex BLOB_BASE=https://...]
|
||||
@test -n "$(LEAF_HASH)" || { echo 'usage: make cloud-fetch-chunk LEAF_HASH=hex BLOB_BASE=https://.../blobs'; exit 2; }
|
||||
@test -n "$(BLOB_BASE)" || { echo 'BLOB_BASE required (per-chunk blobs/<hash> base URL)'; exit 2; }
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue