bench: progressive-AND fixture + 2026-05-09 A/B baseline report
The bench-qa-smoke A/B against commit416f956surfaced a methodology gap: the smoke fixture is structurally insensitive to retrieval-side changes — every smoke query succeeds at full-AND chain 0 on every shard, so cache_keys match byte-for-byte BEFORE/AFTER and any STRICT-rate variance is pure LLM dice. Discovered when 5pp signal floor at n=3 produced ±20pp swings with identical cache_keys. This commit pins a fixture that ACTUALLY exercises the new code path: - 5 PROGRESSIVE queries (full-AND fails on at least one shard; _progressive_and_token_chains drops shortest-tokens-first). Includes the canonical Gundremmingen case from commit2b9d1f0. - 2 DF_FILTER queries (every AND chain fails on at least one shard; search lands in OR-mode where _filter_or_pool_by_df trims high-DF tokens). - 2 CONTROLs (full-AND succeeds on every shard; cache_keys must remain identical BEFORE/AFTER for any retrieval-side change). Which-chain-fires-on-which-shard was empirically verified at HEAD against ~/.arborist/shards (4 wiki shards, 1.5M chunks each). Header documents the chain that wins per query; re-probe if shards drift. Baseline A/B (commit416f956ON vs OFF, n=3, --burn between samples, fixture run via make bench-qa-progressive-and): category cache_key parity STRICT-rate delta median latency delta PROGRESSIVE 9/15 drifted +0.0pp (11/45 both) -3.6s DF_FILTER 3/6 drifted +5.6pp (10/18 → 11/18) -3.9s CONTROL 0/6 drifted +0.0pp (15/18 both) +0.0s The CONTROL +0.0s latency delta is the load-bearing diagnostic — when retrieval is byte-identical the search-side change cannot affect total wall, and it doesn't. PROGRESSIVE / DF_FILTER cache_key drift is the change actively redirecting retrieval, with the expected latency win and no STRICT regression. Future retrieval-side work (#000039 sqlite-vec backend, synonym expansion changes, embedding rerank, hybrid scoring) should bench against this fixture in addition to the full bench-qa scoreboard. Hygiene: - make test → 1623 passed, 28 skipped - make chain-check-shards → 0 breaks across all 7 shards
This commit is contained in:
parent
a5d666eba2
commit
6ba833878d
2 changed files with 222 additions and 0 deletions
67
bench/qa_questions_progressive_and.txt
Normal file
67
bench/qa_questions_progressive_and.txt
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
# Progressive-AND / DF-filter bench fixture (2026-05-09).
|
||||
#
|
||||
# Why this fixture exists: bench/qa_questions_smoke.txt covers the
|
||||
# happy-path retrieval where AND-mode succeeds on chain 0 across every
|
||||
# shard. That fixture is structurally insensitive to retrieval-side
|
||||
# changes — any A/B run produces byte-identical cache_keys and the only
|
||||
# variance is LLM sampling noise. We discovered this the hard way when
|
||||
# A/B-benching the progressive-AND fallback (commit 416f956): the smoke
|
||||
# fixture's STRICT-rate moved ±20pp at n=3, and every single cell had
|
||||
# matching cache_keys before and after. Pure dice.
|
||||
#
|
||||
# This fixture pins queries that EXERCISE the new code path:
|
||||
#
|
||||
# - PROGRESSIVE: full AND returns zero on at least one shard, so
|
||||
# `_progressive_and_token_chains` drops one or more shortest tokens
|
||||
# before AND succeeds. The Gundremmingen case is canonical.
|
||||
# - DF_FILTER: every AND chain returns zero on at least one shard;
|
||||
# search falls to OR-mode where `_filter_or_pool_by_df` drops
|
||||
# high-DF tokens.
|
||||
# - CONTROL: full AND succeeds on every shard. Used to verify the
|
||||
# fixture itself doesn't drift retrieval — these cells should
|
||||
# produce identical cache_keys before/after any retrieval-side
|
||||
# change.
|
||||
#
|
||||
# Which-chain-fires per shard was empirically verified at commit
|
||||
# 416f956 against ~/.arborist/shards (4 wiki shards, 1.5M chunks each).
|
||||
# Re-run /tmp/probe_candidates.py if you suspect drift.
|
||||
#
|
||||
# Run with: make bench-qa-progressive-and (n=3 default, ~3-5 min wall).
|
||||
# Future retrieval-side work (e.g. #000039 sqlite-vec backend) should
|
||||
# bench against this fixture, not just the smoke fixture, to confirm
|
||||
# the change actually moves retrieval rather than just LLM dice.
|
||||
|
||||
# === PROGRESSIVE-AND firers ===
|
||||
# Two-entity geography query — canonical case from commit 2b9d1f0.
|
||||
# Shards 002, 003: chain1/chain2 fires (drops "located"/"located"+"Bavaria").
|
||||
where is Gundremmingen located? where is Bavaria?
|
||||
|
||||
# Two-entity geography, smaller German municipality.
|
||||
# Shards 001, 003: chain3, chain4 fire (drops several tokens).
|
||||
where is Hardthausen am Kocher located? where is Heilbronn?
|
||||
|
||||
# Yes/no near-relation — most chunks don't co-mention both entities.
|
||||
# All shards: chain2 fires.
|
||||
is Gundremmingen near Munich
|
||||
|
||||
# Open-ended "known for" — broad-quantifier shape, narrow topical anchor.
|
||||
# Shard 003: chain1 fires (drops "for").
|
||||
what is Gundremmingen known for
|
||||
|
||||
# Author-described-taxon shape — every chain falls except late ones.
|
||||
# All shards: chain2 fires (drops "year"+"describe", keeps "klotzsch"+"sphagnum").
|
||||
what year did Klotzsch describe sphagnum
|
||||
|
||||
# === DF-FILTER firers ===
|
||||
# Two-entity query where every AND chain on shard 002 returns zero,
|
||||
# so OR-fallback fires and the DF filter trims high-DF tokens.
|
||||
where is Diedrichshagen located? where is Greifswald?
|
||||
|
||||
# Single-entity obscure-place query that lands in OR-mode on shard 002.
|
||||
where is Diedrichshagen located
|
||||
|
||||
# === CONTROLS (FULL_AND on every shard) ===
|
||||
# Identical cache_keys expected before/after any retrieval change.
|
||||
# A drift here means the change ALSO affected the happy path.
|
||||
who painted the mona lisa?
|
||||
when did the soviet union dissolve?
|
||||
155
bench/results/progressive-and-2026-05-09.md
Normal file
155
bench/results/progressive-and-2026-05-09.md
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
# Progressive-AND fallback A/B (commit 416f956)
|
||||
|
||||
**Date**: 2026-05-09
|
||||
**Change**: `arborist/search/fts5.py` — progressive-AND fallback +
|
||||
DF filter at OR-pool (commit
|
||||
[`416f956`](https://git.unturf.com/) ).
|
||||
**Fixture**: `bench/qa_questions_progressive_and.txt` (9 questions ×
|
||||
3 modes × n=3 = 81 samples per condition).
|
||||
**Hardware**: 4 wiki shards (~10 GB each, 1.5M chunks each), serial
|
||||
shard workers (`ARBORIST_SHARD_WORKERS=1`) for clean per-shard
|
||||
attribution; `bench-qa-progressive-and` uses the default (parallel)
|
||||
worker count for total-wall measurements.
|
||||
|
||||
## Why this fixture exists
|
||||
|
||||
`bench/qa_questions_smoke.txt` is structurally insensitive to
|
||||
retrieval-side changes — every smoke query succeeds at full-AND
|
||||
chain 0 on every shard, so no retrieval-side change can move its
|
||||
cache_keys. We discovered this hard way A/B-benching commit 416f956
|
||||
against the smoke fixture: STRICT-rate moved ±20pp at n=3 with every
|
||||
single cache_key matching byte-for-byte BEFORE and AFTER. Pure LLM
|
||||
sampling variance, no retrieval signal at all.
|
||||
|
||||
This fixture pins queries that actually exercise the new code path:
|
||||
|
||||
- **PROGRESSIVE** — full-AND returns zero on at least one shard;
|
||||
`_progressive_and_token_chains` drops shortest-tokens-first and
|
||||
retries.
|
||||
- **DF_FILTER** — every AND chain returns zero on at least one shard;
|
||||
search falls to OR-mode where `_filter_or_pool_by_df` drops
|
||||
high-DF tokens.
|
||||
- **CONTROL** — full-AND succeeds on every shard. Cache_keys must
|
||||
remain identical BEFORE/AFTER for any retrieval-side change.
|
||||
|
||||
Use this fixture for any future retrieval-side work (e.g. #000039
|
||||
sqlite-vec backend) — it's the only way to get a real signal.
|
||||
|
||||
## Methodology
|
||||
|
||||
```
|
||||
BEFORE = HEAD with arborist/search/fts5.py reverted to 416f956~1
|
||||
AFTER = HEAD (progressive-AND + DF filter active)
|
||||
```
|
||||
|
||||
n=3 samples per (question, mode) cell. `qa_sweep.py --n` burns the
|
||||
cached record between samples so Hermes nondeterminism becomes the
|
||||
variation source within a condition.
|
||||
|
||||
## Cache-key parity (the load-bearing diagnostic)
|
||||
|
||||
Cache_key matching ⇒ retrieval produced byte-identical
|
||||
`(source_root, model_profile, conversation, governance_policy, …)`
|
||||
input to the LLM. STRICT-rate variance under matching cache_keys is
|
||||
LLM sampling noise, not retrieval signal.
|
||||
|
||||
| category | cells | matching cks | drifted cks |
|
||||
|---|---|---|---|
|
||||
| PROGRESSIVE | 15 | 6 | **9** |
|
||||
| DF_FILTER | 6 | 3 | **3** |
|
||||
| CONTROL | 6 | **6** | 0 |
|
||||
|
||||
CONTROL = 0 drift confirms the change is inert on happy-path
|
||||
queries. PROGRESSIVE / DF_FILTER drift is the change actively
|
||||
producing different retrieval — which is what we wanted to verify.
|
||||
|
||||
## STRICT-rate aggregate by category
|
||||
|
||||
| category | BEFORE | AFTER | delta |
|
||||
|---|---|---|---|
|
||||
| PROGRESSIVE | 11/45 (24.4%) | 11/45 (24.4%) | +0.0pp |
|
||||
| DF_FILTER | 10/18 (55.6%) | 11/18 (61.1%) | +5.6pp |
|
||||
| CONTROL | 15/18 (83.3%) | 15/18 (83.3%) | +0.0pp |
|
||||
|
||||
PROGRESSIVE and CONTROL aggregate STRICT counts are byte-identical
|
||||
between BEFORE and AFTER. DF_FILTER gains one STRICT cell (within the
|
||||
~5pp noise floor). **No STRICT regression in any category.**
|
||||
|
||||
The PROGRESSIVE 24.4% absolute STRICT-rate is low because the
|
||||
fixture intentionally targets queries the system finds hard — these
|
||||
are obscure-place questions, multi-entity questions, and one
|
||||
specific-fact question (Klotzsch / sphagnum) the corpus may not have
|
||||
clean evidence for. The fixture's job is to exercise the code path,
|
||||
not to score well; the scoreboard is `bench/qa_questions.txt`.
|
||||
|
||||
## Median latency by category
|
||||
|
||||
| category | BEFORE median | AFTER median | delta |
|
||||
|---|---|---|---|
|
||||
| PROGRESSIVE | 10.3s | 6.7s | **−3.6s** |
|
||||
| DF_FILTER | 9.5s | 5.5s | **−3.9s** |
|
||||
| CONTROL | 8.1s | 8.1s | ±0.0s |
|
||||
|
||||
CONTROL latency delta of exactly zero is a beautiful noise-floor
|
||||
calibration: when retrieval input is identical, the LLM produces
|
||||
roughly the same wall regardless of the search code path. The
|
||||
−3.6s/−3.9s wins on PROGRESSIVE / DF_FILTER are real retrieval
|
||||
speedup translating into total wall, which on Hermes-3-8B is
|
||||
otherwise dominated by generation time.
|
||||
|
||||
## Key per-cell observations
|
||||
|
||||
- **`what year did Klotzsch describe sphagnum`**: BEFORE latency
|
||||
16-23s, AFTER 7-8s — this is the cleanest case where OR-fallback
|
||||
was hitting a high-DF token that progressive-AND now sidesteps.
|
||||
STRICT-rate is 0 in both conditions because the corpus doesn't
|
||||
have clean Klotzsch / sphagnum evidence; the test is about
|
||||
retrieval cost, not answer quality.
|
||||
- **`is Gundremmingen near Munich`**: AFTER quote mode picks up 1
|
||||
STRICT (vs 0 BEFORE) on n=3 — coincidence noise (cells differ in
|
||||
cache_keys, so retrieval is genuinely different, but the +1 is
|
||||
within the n=3 floor).
|
||||
- **`who painted the mona lisa? · claim_lattice_pointer`**:
|
||||
BEFORE 1/3, AFTER 0/3. cache_keys MATCH, so this is LLM dice. The
|
||||
pointer mode's 33% sample-to-sample variance on this question is
|
||||
observed in every smoke A/B too (see 2026-05-09 smoke A/B
|
||||
investigation).
|
||||
|
||||
## What this fixture is and isn't
|
||||
|
||||
**Is**: A retrieval-side regression detector. Any change to
|
||||
`arborist/search/`, `arborist/qa/query.py`'s retrieval pipeline, or
|
||||
the synonym/concept layer should bench against this fixture and
|
||||
expect:
|
||||
|
||||
- CONTROL cache_keys remain identical (the happy path didn't drift).
|
||||
- PROGRESSIVE / DF_FILTER cache_keys may differ (the changed path is
|
||||
doing something).
|
||||
- STRICT-rate aggregate within ±5pp on each category.
|
||||
- Latency: hopefully better on PROGRESSIVE / DF_FILTER, unchanged on
|
||||
CONTROL.
|
||||
|
||||
**Isn't**: A scoreboard for absolute retrieval quality. The
|
||||
PROGRESSIVE category is intentionally hard. The full bench
|
||||
(`bench/qa_questions.txt`) stays the scoreboard for
|
||||
prompt/verifier/lattice changes.
|
||||
|
||||
## Reproducing
|
||||
|
||||
```bash
|
||||
# AFTER (HEAD)
|
||||
make bench-qa-progressive-and BENCH_QA_OUT=bench/qa_results_progressive/after
|
||||
|
||||
# BEFORE (revert search/fts5.py temporarily)
|
||||
git show 416f956~1:arborist/search/fts5.py > arborist/search/fts5.py
|
||||
make bench-qa-progressive-and BENCH_QA_OUT=bench/qa_results_progressive/before
|
||||
git checkout HEAD -- arborist/search/fts5.py
|
||||
```
|
||||
|
||||
## Source files
|
||||
|
||||
- Fixture: `bench/qa_questions_progressive_and.txt`
|
||||
- Makefile target: `bench-qa-progressive-and` (n=3 default,
|
||||
`BENCH_PROGRESSIVE_N=N` to override)
|
||||
- BEFORE results: `bench/qa_results_progressive/before/`
|
||||
- AFTER results: `bench/qa_results_progressive/after/`
|
||||
Loading…
Add table
Add a link
Reference in a new issue