russell@unturf.com
6ba833878d
The bench-qa-smoke A/B against commit416f956surfaced a methodology gap: the smoke fixture is structurally insensitive to retrieval-side changes — every smoke query succeeds at full-AND chain 0 on every shard, so cache_keys match byte-for-byte BEFORE/AFTER and any STRICT-rate variance is pure LLM dice. Discovered when 5pp signal floor at n=3 produced ±20pp swings with identical cache_keys. This commit pins a fixture that ACTUALLY exercises the new code path: - 5 PROGRESSIVE queries (full-AND fails on at least one shard; _progressive_and_token_chains drops shortest-tokens-first). Includes the canonical Gundremmingen case from commit2b9d1f0. - 2 DF_FILTER queries (every AND chain fails on at least one shard; search lands in OR-mode where _filter_or_pool_by_df trims high-DF tokens). - 2 CONTROLs (full-AND succeeds on every shard; cache_keys must remain identical BEFORE/AFTER for any retrieval-side change). Which-chain-fires-on-which-shard was empirically verified at HEAD against ~/.arborist/shards (4 wiki shards, 1.5M chunks each). Header documents the chain that wins per query; re-probe if shards drift. Baseline A/B (commit416f956ON vs OFF, n=3, --burn between samples, fixture run via make bench-qa-progressive-and): category cache_key parity STRICT-rate delta median latency delta PROGRESSIVE 9/15 drifted +0.0pp (11/45 both) -3.6s DF_FILTER 3/6 drifted +5.6pp (10/18 → 11/18) -3.9s CONTROL 0/6 drifted +0.0pp (15/18 both) +0.0s The CONTROL +0.0s latency delta is the load-bearing diagnostic — when retrieval is byte-identical the search-side change cannot affect total wall, and it doesn't. PROGRESSIVE / DF_FILTER cache_key drift is the change actively redirecting retrieval, with the expected latency win and no STRICT regression. Future retrieval-side work (#000039 sqlite-vec backend, synonym expansion changes, embedding rerank, hybrid scoring) should bench against this fixture in addition to the full bench-qa scoreboard. Hygiene: - make test → 1623 passed, 28 skipped - make chain-check-shards → 0 breaks across all 7 shards
67 lines
3.1 KiB
Text
67 lines
3.1 KiB
Text
# Progressive-AND / DF-filter bench fixture (2026-05-09).
|
|
#
|
|
# Why this fixture exists: bench/qa_questions_smoke.txt covers the
|
|
# happy-path retrieval where AND-mode succeeds on chain 0 across every
|
|
# shard. That fixture is structurally insensitive to retrieval-side
|
|
# changes — any A/B run produces byte-identical cache_keys and the only
|
|
# variance is LLM sampling noise. We discovered this the hard way when
|
|
# A/B-benching the progressive-AND fallback (commit 416f956): the smoke
|
|
# fixture's STRICT-rate moved ±20pp at n=3, and every single cell had
|
|
# matching cache_keys before and after. Pure dice.
|
|
#
|
|
# This fixture pins queries that EXERCISE the new code path:
|
|
#
|
|
# - PROGRESSIVE: full AND returns zero on at least one shard, so
|
|
# `_progressive_and_token_chains` drops one or more shortest tokens
|
|
# before AND succeeds. The Gundremmingen case is canonical.
|
|
# - DF_FILTER: every AND chain returns zero on at least one shard;
|
|
# search falls to OR-mode where `_filter_or_pool_by_df` drops
|
|
# high-DF tokens.
|
|
# - CONTROL: full AND succeeds on every shard. Used to verify the
|
|
# fixture itself doesn't drift retrieval — these cells should
|
|
# produce identical cache_keys before/after any retrieval-side
|
|
# change.
|
|
#
|
|
# Which-chain-fires per shard was empirically verified at commit
|
|
# 416f956 against ~/.arborist/shards (4 wiki shards, 1.5M chunks each).
|
|
# Re-run /tmp/probe_candidates.py if you suspect drift.
|
|
#
|
|
# Run with: make bench-qa-progressive-and (n=3 default, ~3-5 min wall).
|
|
# Future retrieval-side work (e.g. #000039 sqlite-vec backend) should
|
|
# bench against this fixture, not just the smoke fixture, to confirm
|
|
# the change actually moves retrieval rather than just LLM dice.
|
|
|
|
# === PROGRESSIVE-AND firers ===
|
|
# Two-entity geography query — canonical case from commit 2b9d1f0.
|
|
# Shards 002, 003: chain1/chain2 fires (drops "located"/"located"+"Bavaria").
|
|
where is Gundremmingen located? where is Bavaria?
|
|
|
|
# Two-entity geography, smaller German municipality.
|
|
# Shards 001, 003: chain3, chain4 fire (drops several tokens).
|
|
where is Hardthausen am Kocher located? where is Heilbronn?
|
|
|
|
# Yes/no near-relation — most chunks don't co-mention both entities.
|
|
# All shards: chain2 fires.
|
|
is Gundremmingen near Munich
|
|
|
|
# Open-ended "known for" — broad-quantifier shape, narrow topical anchor.
|
|
# Shard 003: chain1 fires (drops "for").
|
|
what is Gundremmingen known for
|
|
|
|
# Author-described-taxon shape — every chain falls except late ones.
|
|
# All shards: chain2 fires (drops "year"+"describe", keeps "klotzsch"+"sphagnum").
|
|
what year did Klotzsch describe sphagnum
|
|
|
|
# === DF-FILTER firers ===
|
|
# Two-entity query where every AND chain on shard 002 returns zero,
|
|
# so OR-fallback fires and the DF filter trims high-DF tokens.
|
|
where is Diedrichshagen located? where is Greifswald?
|
|
|
|
# Single-entity obscure-place query that lands in OR-mode on shard 002.
|
|
where is Diedrichshagen located
|
|
|
|
# === CONTROLS (FULL_AND on every shard) ===
|
|
# Identical cache_keys expected before/after any retrieval change.
|
|
# A drift here means the change ALSO affected the happy path.
|
|
who painted the mona lisa?
|
|
when did the soviet union dissolve?
|