#52 fix: fts pack lands on owning target only (no cross-target leak)

v4 bench (2026-05-27 03:09 UTC) measured a corrupt outcome:
chunks_fts=1,561,604 on EVERY target shard regardless of chunks
count. _pull_fts_pack_into_targets had been iterating all M targets
and INSERT'ing each fts pack's shadow tables into every one of them.
Each fts pack's chunks_fts_docsize entries reference chunk_ids that
were independently auto-assigned in its source producer shard (each
source DB has chunk_ids 1..1.56M independently). Inserting all 4
packs into all 4 targets → 4× the per-target FTS rows, pointing at
chunk_ids the target doesn't own. Body searches would return garbage.

Fix: sample one id from the fts pack's chunks_fts_docsize, look it
up in each target's chunks table, INSERT only into the target where
it's found. The other M-1 targets stay untouched and receive their
FTS data from their corresponding fts pack(s) in later iterations.

In post-reshard production topology, each producer source shard's
docs all hash to ONE consumer target, so this 1:1 mapping is exact.
The test fixture is artificial (single-shard producer with docs
hash-distributed across M=4 targets) but still validates the core
property: only one target receives FTS data; the others stay empty.

FTS5 shadow tables aren't subsettable per-row (segment data is
opaque, mixed entries for many docs in one segment) so we can't
filter FTS rows to "only chunks that exist on this target" — we
copy all-or-nothing per pack. That's why the producer's post-reshard
shape (each pack scoped to one target's docs) is the structural
prerequisite for fts packs to make sense.

New regression test
(TestFtsPackRoutingRegression.test_fts_pack_only_on_owning_target):
exactly 1 of M targets has chunks_fts_docsize > 0; the others
must have 0. Pre-fix this asserted on all-4 targets having FTS
data → failed. Post-fix passes.

Returns now include owning_target_idx for forensic visibility into
which target the fts pack landed on.

5 cold-unpack-routed tests pass.
This commit is contained in:
russell@unturf.com 2026-05-27 05:18:25 -04:00
parent 45960d3909
commit db18172f9e
No known key found for this signature in database
2 changed files with 153 additions and 40 deletions

View file

@ -191,6 +191,81 @@ class TestRoutedHydrate:
c.close()
class TestFtsPackRoutingRegression:
"""Regression test for the v4 bench bug (#52): _pull_fts_pack_into_targets
used to INSERT shadow tables into every target, cross-contaminating
FTS indexes across shards. After the fix it should INSERT only into
the ONE target that owns the chunks the fts pack references.
Assertion: per-shard chunks_fts row count equals per-shard chunks row
count (no cross-target leakage).
"""
def test_fts_pack_only_on_owning_target(
self, producer_shard: Path, tmp_path: Path
):
from arborist.evict import push_pack
backend = MemoryBackend()
with sqlite3.connect(str(producer_shard)) as src_conn:
src_conn.row_factory = sqlite3.Row
push_result = push_pack(
src_conn, backend,
document_root=None,
max_chunks=None,
max_pack_bytes=10 * 1024 ** 2,
allow_license_class="unknown",
include_fts=True,
)
metadata_pack_hash = next(
p["pack_hash"] for p in push_result["packs"]
if p.get("kind") == "metadata"
)
target_dir = tmp_path / "tgt"
target_dir.mkdir()
M = 4
targets = _make_target_shards(target_dir, M)
try:
hydrate_from_metadata_pack_routed(
targets, backend, metadata_pack_hash, M=M, mode="full",
)
finally:
for t in targets:
t.close()
# Regression check: the v4 bug INSERT'd the fts shadow tables
# into EVERY target shard. After the fix, only ONE target gets
# the FTS data (the target whose chunks the fts pack references).
# The test fixture is single-shard so the producer's pack contains
# FTS data for all docs, but only one consumer target hash-routes
# to ownership of the chunks the FTS data references. The other
# 3 targets must have ZERO chunks_fts data (no cross-contamination).
per_shard = []
for i in range(M):
tdb = target_dir / f"{i:03d}.db"
c = sqlite3.connect(f"file:{tdb}?mode=ro", uri=True)
try:
chunks = c.execute("SELECT COUNT(*) FROM chunks").fetchone()[0]
fts_data = c.execute("SELECT COUNT(*) FROM chunks_fts_data").fetchone()[0]
fts_docsize = c.execute("SELECT COUNT(*) FROM chunks_fts_docsize").fetchone()[0]
per_shard.append({"chunks": chunks, "fts_data": fts_data, "fts_docsize": fts_docsize})
finally:
c.close()
# Exactly one shard should hold FTS data; the rest must be empty.
# (FTS5 shadow tables are segment-based — not subsettable per-row
# — so the owning shard receives ALL of the fts pack's rows even
# if its own chunks count is smaller.)
with_fts = [s for s in per_shard if s["fts_docsize"] > 0]
without_fts = [s for s in per_shard if s["fts_docsize"] == 0]
assert len(with_fts) == 1, (
f"expected exactly 1 target with FTS data; got {len(with_fts)} — "
"v4 cross-target contamination bug back"
)
# chunks_fts_data has 1-2 default FTS5 internal config/structure
# rows on a freshly-created virtual table; the per-document signal
# lives in chunks_fts_docsize which is what we asserted above.
class TestRoutedHydrateValidation:
def test_M_mismatch_rejected(self, tmp_path: Path):
target_dir = tmp_path / "tgt"