The bench-qa-smoke A/B against commit 416f956 surfaced a methodology
gap: the smoke fixture is structurally insensitive to retrieval-side
changes — every smoke query succeeds at full-AND chain 0 on every
shard, so cache_keys match byte-for-byte BEFORE/AFTER and any
STRICT-rate variance is pure LLM dice. Discovered when 5pp signal
floor at n=3 produced ±20pp swings with identical cache_keys.
This commit pins a fixture that ACTUALLY exercises the new code path:
- 5 PROGRESSIVE queries (full-AND fails on at least one shard;
_progressive_and_token_chains drops shortest-tokens-first). Includes
the canonical Gundremmingen case from commit 2b9d1f0.
- 2 DF_FILTER queries (every AND chain fails on at least one shard;
search lands in OR-mode where _filter_or_pool_by_df trims
high-DF tokens).
- 2 CONTROLs (full-AND succeeds on every shard; cache_keys must
remain identical BEFORE/AFTER for any retrieval-side change).
Which-chain-fires-on-which-shard was empirically verified at HEAD
against ~/.arborist/shards (4 wiki shards, 1.5M chunks each). Header
documents the chain that wins per query; re-probe if shards drift.
Baseline A/B (commit 416f956 ON vs OFF, n=3, --burn between samples,
fixture run via make bench-qa-progressive-and):
category cache_key parity STRICT-rate delta median latency delta
PROGRESSIVE 9/15 drifted +0.0pp (11/45 both) -3.6s
DF_FILTER 3/6 drifted +5.6pp (10/18 → 11/18) -3.9s
CONTROL 0/6 drifted +0.0pp (15/18 both) +0.0s
The CONTROL +0.0s latency delta is the load-bearing diagnostic —
when retrieval is byte-identical the search-side change cannot
affect total wall, and it doesn't. PROGRESSIVE / DF_FILTER cache_key
drift is the change actively redirecting retrieval, with the
expected latency win and no STRICT regression.
Future retrieval-side work (#000039 sqlite-vec backend, synonym
expansion changes, embedding rerank, hybrid scoring) should bench
against this fixture in addition to the full bench-qa scoreboard.
Hygiene:
- make test → 1623 passed, 28 skipped
- make chain-check-shards → 0 breaks across all 7 shards
russell@unturf.com
6ba833878d