#52 fix: fts pack lands on owning target only (no cross-target leak)
v4 bench (2026-05-27 03:09 UTC) measured a corrupt outcome: chunks_fts=1,561,604 on EVERY target shard regardless of chunks count. _pull_fts_pack_into_targets had been iterating all M targets and INSERT'ing each fts pack's shadow tables into every one of them. Each fts pack's chunks_fts_docsize entries reference chunk_ids that were independently auto-assigned in its source producer shard (each source DB has chunk_ids 1..1.56M independently). Inserting all 4 packs into all 4 targets → 4× the per-target FTS rows, pointing at chunk_ids the target doesn't own. Body searches would return garbage. Fix: sample one id from the fts pack's chunks_fts_docsize, look it up in each target's chunks table, INSERT only into the target where it's found. The other M-1 targets stay untouched and receive their FTS data from their corresponding fts pack(s) in later iterations. In post-reshard production topology, each producer source shard's docs all hash to ONE consumer target, so this 1:1 mapping is exact. The test fixture is artificial (single-shard producer with docs hash-distributed across M=4 targets) but still validates the core property: only one target receives FTS data; the others stay empty. FTS5 shadow tables aren't subsettable per-row (segment data is opaque, mixed entries for many docs in one segment) so we can't filter FTS rows to "only chunks that exist on this target" — we copy all-or-nothing per pack. That's why the producer's post-reshard shape (each pack scoped to one target's docs) is the structural prerequisite for fts packs to make sense. New regression test (TestFtsPackRoutingRegression.test_fts_pack_only_on_owning_target): exactly 1 of M targets has chunks_fts_docsize > 0; the others must have 0. Pre-fix this asserted on all-4 targets having FTS data → failed. Post-fix passes. Returns now include owning_target_idx for forensic visibility into which target the fts pack landed on. 5 cold-unpack-routed tests pass.
This commit is contained in:
parent
45960d3909
commit
db18172f9e
2 changed files with 153 additions and 40 deletions
|
|
@ -191,6 +191,81 @@ class TestRoutedHydrate:
|
|||
c.close()
|
||||
|
||||
|
||||
class TestFtsPackRoutingRegression:
|
||||
"""Regression test for the v4 bench bug (#52): _pull_fts_pack_into_targets
|
||||
used to INSERT shadow tables into every target, cross-contaminating
|
||||
FTS indexes across shards. After the fix it should INSERT only into
|
||||
the ONE target that owns the chunks the fts pack references.
|
||||
|
||||
Assertion: per-shard chunks_fts row count equals per-shard chunks row
|
||||
count (no cross-target leakage).
|
||||
"""
|
||||
|
||||
def test_fts_pack_only_on_owning_target(
|
||||
self, producer_shard: Path, tmp_path: Path
|
||||
):
|
||||
from arborist.evict import push_pack
|
||||
backend = MemoryBackend()
|
||||
with sqlite3.connect(str(producer_shard)) as src_conn:
|
||||
src_conn.row_factory = sqlite3.Row
|
||||
push_result = push_pack(
|
||||
src_conn, backend,
|
||||
document_root=None,
|
||||
max_chunks=None,
|
||||
max_pack_bytes=10 * 1024 ** 2,
|
||||
allow_license_class="unknown",
|
||||
include_fts=True,
|
||||
)
|
||||
metadata_pack_hash = next(
|
||||
p["pack_hash"] for p in push_result["packs"]
|
||||
if p.get("kind") == "metadata"
|
||||
)
|
||||
|
||||
target_dir = tmp_path / "tgt"
|
||||
target_dir.mkdir()
|
||||
M = 4
|
||||
targets = _make_target_shards(target_dir, M)
|
||||
try:
|
||||
hydrate_from_metadata_pack_routed(
|
||||
targets, backend, metadata_pack_hash, M=M, mode="full",
|
||||
)
|
||||
finally:
|
||||
for t in targets:
|
||||
t.close()
|
||||
|
||||
# Regression check: the v4 bug INSERT'd the fts shadow tables
|
||||
# into EVERY target shard. After the fix, only ONE target gets
|
||||
# the FTS data (the target whose chunks the fts pack references).
|
||||
# The test fixture is single-shard so the producer's pack contains
|
||||
# FTS data for all docs, but only one consumer target hash-routes
|
||||
# to ownership of the chunks the FTS data references. The other
|
||||
# 3 targets must have ZERO chunks_fts data (no cross-contamination).
|
||||
per_shard = []
|
||||
for i in range(M):
|
||||
tdb = target_dir / f"{i:03d}.db"
|
||||
c = sqlite3.connect(f"file:{tdb}?mode=ro", uri=True)
|
||||
try:
|
||||
chunks = c.execute("SELECT COUNT(*) FROM chunks").fetchone()[0]
|
||||
fts_data = c.execute("SELECT COUNT(*) FROM chunks_fts_data").fetchone()[0]
|
||||
fts_docsize = c.execute("SELECT COUNT(*) FROM chunks_fts_docsize").fetchone()[0]
|
||||
per_shard.append({"chunks": chunks, "fts_data": fts_data, "fts_docsize": fts_docsize})
|
||||
finally:
|
||||
c.close()
|
||||
# Exactly one shard should hold FTS data; the rest must be empty.
|
||||
# (FTS5 shadow tables are segment-based — not subsettable per-row
|
||||
# — so the owning shard receives ALL of the fts pack's rows even
|
||||
# if its own chunks count is smaller.)
|
||||
with_fts = [s for s in per_shard if s["fts_docsize"] > 0]
|
||||
without_fts = [s for s in per_shard if s["fts_docsize"] == 0]
|
||||
assert len(with_fts) == 1, (
|
||||
f"expected exactly 1 target with FTS data; got {len(with_fts)} — "
|
||||
"v4 cross-target contamination bug back"
|
||||
)
|
||||
# chunks_fts_data has 1-2 default FTS5 internal config/structure
|
||||
# rows on a freshly-created virtual table; the per-document signal
|
||||
# lives in chunks_fts_docsize which is what we asserted above.
|
||||
|
||||
|
||||
class TestRoutedHydrateValidation:
|
||||
def test_M_mismatch_rejected(self, tmp_path: Path):
|
||||
target_dir = tmp_path / "tgt"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue