wallet/fts-sidecar: real FTS5 in the cloud sidecar; delete custom BM25
Replaces the hand-rolled BM25 sidecar (arborist/wallet/sidecar.py, ~690 LOC) with a slim SQLite file that just COPIES the source shard's FTS5 shadow tables verbatim + minimal doc/chunk metadata. Cloud retrieval then runs SQLite FTS5 bm25() on the same bytes the local shard uses — bit-for-bit parity by construction. 5/5 source + audit_mode agreement on the smoke fixture between local corpus-query and cloud-query against the new manifest-fts.json. Why --- Custom binary sidecar was per-document BM25; main encyclopedia articles got length-normalized so hard that on "why did the dinosaurs go extinct?" "Edwina, the Dinosaur Who Didn't Know She Was Extinct" beat "Dinosaur" (measured cloud-vs-local divergence). Local FTS5 indexes per-chunk so each chunk is a moderate-length doc and the main article wins multiply. Different granularity, not a tuning knob — fix is to use the same indexer cloud-side. What ships ---------- - arborist/wallet/fts_sidecar_build.py — builder. ATTACH source shard, copy documents (root/uri/title only), copy chunks (id/root/idx/leaf only, NO content), CREATE VIRTUAL TABLE chunks_fts/documents_fts with same DDL as source, bulk-copy the four shadow tables verbatim, VACUUM. 8.78 GB shard → 2.15 GB sidecar (24.5%) in ~45 s; full 4-shard wiki corpus 37.4 GB → 8.1 GB (21.7%) in ~3 min. - arborist/wallet/bucket.py: FtsSidecarShardClient — downloads slim sidecar once into ~/.arborist/sidecar-fts-cache/<hash>.idx.db, opens read-only sqlite3 (check_same_thread=False for parallel shard fan- out), runs FTS5 MATCH locally. Chunk content fetches via blobs/<hash> with HTTP-range big-shard fallback when blobs aren't published. MultiShardSidecarCorpus simplified to fts_sidecar_url ∨ bucket-direct (both are FTS5 backends; merge by raw bm25 MIN ascending). - arborist/qa/corpus.py: SidecarBucketCorpus.higher_is_better=False (FTS5 bm25 is negative, lower=better). chunks_for_doc dispatches on fetch_chunk_body attr for the slim-FTS5 client. apply_title_boost imports tokenizer helpers from new arborist/qa/_text_norm.py. - arborist/qa/_text_norm.py — fold_accents, numeral_expand, tokenize_text, STOPWORDS — extracted from the deleted sidecar.py so apply_title_boost keeps its lexical shape. - arborist/cli.py: `arborist sidecar build-fts` subcommand; old `sidecar build`/`sidecar search` removed. cloud_query recognizes fts_sidecar_url + sidecar_url alike. - Makefile: `sidecar-build-fts` + `sidecar-build-fts-all` targets; `sidecar-build` + `sidecar-search` removed. - scripts/upload_fts_sidecars.py — boto3 producer: uploads slim sidecars to clones/sidecars-fts/<n>.idx.db, publishes clones/manifest-fts.json (4 wikipedia shards inherit existing shard_url for content fallback; ACL public-read; idempotent on size match). Existing manifest-sidecar.json untouched. - tests/test_qa_corpus_functional.py + test_qa_corpus_integration.py converted from build_sidecar → build_fts_sidecar; 6 fixtures pass. - bench/slim_fts_parity_bench.py — local 3-way bench (legacy/corpus/slim_fts) over the smoke fixture. Validation ---------- - Per-shard FTS5 parity: slim sidecar returns IDENTICAL rowids + bm25 scores to the source shard for top-10 of "dinosaurs extinct". - 3-way bench (legacy local / corpus local / slim-FTS5 over real bucket fallback): 5/5 source agreement AND 5/5 audit_mode agreement between corpus and slim_fts. Q5 legacy disagreement (Edwina vs Dinosaur) is the pre-existing 2000-line query() retrieval quirk, unrelated. - End-to-end cloud query against published manifest-fts.json (cold- start, ~149 s sidecar download once): STRICT · Dinosaur, every quote verified (2/2). - Full pytest suite: 2737 passed, 28 skipped, 1 xfailed. One pre- existing failure (tests/test_doc_counts.py — claim_pack docs row- count drift) and one pre-existing cold_object failure, both reproduce on main HEAD. Bucket state ------------ - s3://arborist/clones/sidecars-fts/00[0-3].idx.db (8.1 GB) — new - s3://arborist/clones/manifest-fts.json — new - s3://arborist/clones/manifest-sidecar.json — kept live (deprecated but still readable; downstream callers should switch to manifest-fts.json)
This commit is contained in:
parent
2eea5b5655
commit
d9fb6a9b69
13 changed files with 1087 additions and 984 deletions
|
|
@ -122,7 +122,8 @@ def test_sidecar_bucket_corpus_satisfies_protocol():
|
|||
corpus = SidecarBucketCorpus(_StubMulti())
|
||||
assert isinstance(corpus, Corpus)
|
||||
assert corpus.name == "sidecar-bucket"
|
||||
assert corpus.higher_is_better is True
|
||||
# FTS5 bm25 is negative; lower (more negative) = better.
|
||||
assert corpus.higher_is_better is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -53,7 +53,7 @@ from arborist.qa.verify import verify_claim_lattice
|
|||
from arborist.source import Source
|
||||
from arborist.store import connect
|
||||
from arborist.wallet.bucket import BucketManifest, MultiShardSidecarCorpus
|
||||
from arborist.wallet.sidecar import build_sidecar
|
||||
from arborist.wallet.fts_sidecar_build import build as build_fts_sidecar
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -101,14 +101,14 @@ def both_adapters(tmp_path: Path):
|
|||
c0.execute("PRAGMA wal_checkpoint(FULL)")
|
||||
c0.close()
|
||||
|
||||
# bucket layout + sidecar
|
||||
# bucket layout + slim FTS5 sidecar
|
||||
bucket = tmp_path / "bucket"
|
||||
clones = bucket / "clones" / "snap-1"
|
||||
clones.mkdir(parents=True)
|
||||
bucket_shard = clones / "000.db"
|
||||
shutil.copy(db, bucket_shard)
|
||||
sidecar_path = clones / "000.sidecar.bin"
|
||||
build_sidecar(bucket_shard, sidecar_path)
|
||||
sidecar_path = clones / "000.idx.db"
|
||||
build_fts_sidecar(str(bucket_shard), str(sidecar_path), verbose=False)
|
||||
|
||||
# in-process Range-aware server
|
||||
cwd_before = os.getcwd()
|
||||
|
|
@ -123,14 +123,14 @@ def both_adapters(tmp_path: Path):
|
|||
"url": f"{base_url}/clones/snap-1/000.db",
|
||||
"shard_idx": 0,
|
||||
"label": "test-shard",
|
||||
"sidecar_url": f"{base_url}/clones/snap-1/000.sidecar.bin",
|
||||
"fts_sidecar_url": f"{base_url}/clones/snap-1/000.idx.db",
|
||||
}],
|
||||
blob_base=f"{base_url}/blobs",
|
||||
snapshot_root=None,
|
||||
snapshot_ts=None,
|
||||
)
|
||||
multi = MultiShardSidecarCorpus(
|
||||
manifest, sidecar_cache_dir=str(tmp_path / "cache"),
|
||||
manifest, fts_sidecar_cache_dir=str(tmp_path / "cache"),
|
||||
)
|
||||
conn = connect(db)
|
||||
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ from arborist.wallet.bucket import (
|
|||
BucketManifest,
|
||||
MultiShardSidecarCorpus,
|
||||
)
|
||||
from arborist.wallet.sidecar import build_sidecar
|
||||
from arborist.wallet.fts_sidecar_build import build as build_fts_sidecar
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
|
@ -90,8 +90,8 @@ def served_corpus(tmp_path: Path):
|
|||
import shutil
|
||||
bucket_shard = clones / "000.db"
|
||||
shutil.copy(db, bucket_shard)
|
||||
sidecar_path = clones / "000.sidecar.bin"
|
||||
build_sidecar(bucket_shard, sidecar_path)
|
||||
sidecar_path = clones / "000.idx.db"
|
||||
build_fts_sidecar(str(bucket_shard), str(sidecar_path), verbose=False)
|
||||
|
||||
# Re-open the original for SqliteShardCorpus tests.
|
||||
conn = connect(db)
|
||||
|
|
@ -110,13 +110,13 @@ def served_corpus(tmp_path: Path):
|
|||
"url": f"{base_url}/clones/snap-1/000.db",
|
||||
"shard_idx": 0,
|
||||
"label": "test-shard",
|
||||
"sidecar_url": f"{base_url}/clones/snap-1/000.sidecar.bin",
|
||||
"fts_sidecar_url": f"{base_url}/clones/snap-1/000.idx.db",
|
||||
}],
|
||||
blob_base=f"{base_url}/blobs",
|
||||
snapshot_root=None,
|
||||
snapshot_ts=None,
|
||||
)
|
||||
multi = MultiShardSidecarCorpus(manifest, sidecar_cache_dir=str(tmp_path / "cache"))
|
||||
multi = MultiShardSidecarCorpus(manifest, fts_sidecar_cache_dir=str(tmp_path / "cache"))
|
||||
|
||||
try:
|
||||
yield conn, multi, base_url
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue