From 6ba833878d2db2c5f0ecf3bca3ba4a2b00e851d7 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 10 May 2026 06:35:18 -0400 Subject: [PATCH] bench: progressive-AND fixture + 2026-05-09 A/B baseline report MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bench-qa-smoke A/B against commit 416f956 surfaced a methodology gap: the smoke fixture is structurally insensitive to retrieval-side changes — every smoke query succeeds at full-AND chain 0 on every shard, so cache_keys match byte-for-byte BEFORE/AFTER and any STRICT-rate variance is pure LLM dice. Discovered when 5pp signal floor at n=3 produced ±20pp swings with identical cache_keys. This commit pins a fixture that ACTUALLY exercises the new code path: - 5 PROGRESSIVE queries (full-AND fails on at least one shard; _progressive_and_token_chains drops shortest-tokens-first). Includes the canonical Gundremmingen case from commit 2b9d1f0. - 2 DF_FILTER queries (every AND chain fails on at least one shard; search lands in OR-mode where _filter_or_pool_by_df trims high-DF tokens). - 2 CONTROLs (full-AND succeeds on every shard; cache_keys must remain identical BEFORE/AFTER for any retrieval-side change). Which-chain-fires-on-which-shard was empirically verified at HEAD against ~/.arborist/shards (4 wiki shards, 1.5M chunks each). Header documents the chain that wins per query; re-probe if shards drift. Baseline A/B (commit 416f956 ON vs OFF, n=3, --burn between samples, fixture run via make bench-qa-progressive-and): category cache_key parity STRICT-rate delta median latency delta PROGRESSIVE 9/15 drifted +0.0pp (11/45 both) -3.6s DF_FILTER 3/6 drifted +5.6pp (10/18 → 11/18) -3.9s CONTROL 0/6 drifted +0.0pp (15/18 both) +0.0s The CONTROL +0.0s latency delta is the load-bearing diagnostic — when retrieval is byte-identical the search-side change cannot affect total wall, and it doesn't. PROGRESSIVE / DF_FILTER cache_key drift is the change actively redirecting retrieval, with the expected latency win and no STRICT regression. Future retrieval-side work (#000039 sqlite-vec backend, synonym expansion changes, embedding rerank, hybrid scoring) should bench against this fixture in addition to the full bench-qa scoreboard. Hygiene: - make test → 1623 passed, 28 skipped - make chain-check-shards → 0 breaks across all 7 shards --- bench/qa_questions_progressive_and.txt | 67 +++++++++ bench/results/progressive-and-2026-05-09.md | 155 ++++++++++++++++++++ 2 files changed, 222 insertions(+) create mode 100644 bench/qa_questions_progressive_and.txt create mode 100644 bench/results/progressive-and-2026-05-09.md diff --git a/bench/qa_questions_progressive_and.txt b/bench/qa_questions_progressive_and.txt new file mode 100644 index 0000000..04b5506 --- /dev/null +++ b/bench/qa_questions_progressive_and.txt @@ -0,0 +1,67 @@ +# Progressive-AND / DF-filter bench fixture (2026-05-09). +# +# Why this fixture exists: bench/qa_questions_smoke.txt covers the +# happy-path retrieval where AND-mode succeeds on chain 0 across every +# shard. That fixture is structurally insensitive to retrieval-side +# changes — any A/B run produces byte-identical cache_keys and the only +# variance is LLM sampling noise. We discovered this the hard way when +# A/B-benching the progressive-AND fallback (commit 416f956): the smoke +# fixture's STRICT-rate moved ±20pp at n=3, and every single cell had +# matching cache_keys before and after. Pure dice. +# +# This fixture pins queries that EXERCISE the new code path: +# +# - PROGRESSIVE: full AND returns zero on at least one shard, so +# `_progressive_and_token_chains` drops one or more shortest tokens +# before AND succeeds. The Gundremmingen case is canonical. +# - DF_FILTER: every AND chain returns zero on at least one shard; +# search falls to OR-mode where `_filter_or_pool_by_df` drops +# high-DF tokens. +# - CONTROL: full AND succeeds on every shard. Used to verify the +# fixture itself doesn't drift retrieval — these cells should +# produce identical cache_keys before/after any retrieval-side +# change. +# +# Which-chain-fires per shard was empirically verified at commit +# 416f956 against ~/.arborist/shards (4 wiki shards, 1.5M chunks each). +# Re-run /tmp/probe_candidates.py if you suspect drift. +# +# Run with: make bench-qa-progressive-and (n=3 default, ~3-5 min wall). +# Future retrieval-side work (e.g. #000039 sqlite-vec backend) should +# bench against this fixture, not just the smoke fixture, to confirm +# the change actually moves retrieval rather than just LLM dice. + +# === PROGRESSIVE-AND firers === +# Two-entity geography query — canonical case from commit 2b9d1f0. +# Shards 002, 003: chain1/chain2 fires (drops "located"/"located"+"Bavaria"). +where is Gundremmingen located? where is Bavaria? + +# Two-entity geography, smaller German municipality. +# Shards 001, 003: chain3, chain4 fire (drops several tokens). +where is Hardthausen am Kocher located? where is Heilbronn? + +# Yes/no near-relation — most chunks don't co-mention both entities. +# All shards: chain2 fires. +is Gundremmingen near Munich + +# Open-ended "known for" — broad-quantifier shape, narrow topical anchor. +# Shard 003: chain1 fires (drops "for"). +what is Gundremmingen known for + +# Author-described-taxon shape — every chain falls except late ones. +# All shards: chain2 fires (drops "year"+"describe", keeps "klotzsch"+"sphagnum"). +what year did Klotzsch describe sphagnum + +# === DF-FILTER firers === +# Two-entity query where every AND chain on shard 002 returns zero, +# so OR-fallback fires and the DF filter trims high-DF tokens. +where is Diedrichshagen located? where is Greifswald? + +# Single-entity obscure-place query that lands in OR-mode on shard 002. +where is Diedrichshagen located + +# === CONTROLS (FULL_AND on every shard) === +# Identical cache_keys expected before/after any retrieval change. +# A drift here means the change ALSO affected the happy path. +who painted the mona lisa? +when did the soviet union dissolve? diff --git a/bench/results/progressive-and-2026-05-09.md b/bench/results/progressive-and-2026-05-09.md new file mode 100644 index 0000000..5fbcffc --- /dev/null +++ b/bench/results/progressive-and-2026-05-09.md @@ -0,0 +1,155 @@ +# Progressive-AND fallback A/B (commit 416f956) + +**Date**: 2026-05-09 +**Change**: `arborist/search/fts5.py` — progressive-AND fallback + +DF filter at OR-pool (commit +[`416f956`](https://git.unturf.com/) ). +**Fixture**: `bench/qa_questions_progressive_and.txt` (9 questions × +3 modes × n=3 = 81 samples per condition). +**Hardware**: 4 wiki shards (~10 GB each, 1.5M chunks each), serial +shard workers (`ARBORIST_SHARD_WORKERS=1`) for clean per-shard +attribution; `bench-qa-progressive-and` uses the default (parallel) +worker count for total-wall measurements. + +## Why this fixture exists + +`bench/qa_questions_smoke.txt` is structurally insensitive to +retrieval-side changes — every smoke query succeeds at full-AND +chain 0 on every shard, so no retrieval-side change can move its +cache_keys. We discovered this hard way A/B-benching commit 416f956 +against the smoke fixture: STRICT-rate moved ±20pp at n=3 with every +single cache_key matching byte-for-byte BEFORE and AFTER. Pure LLM +sampling variance, no retrieval signal at all. + +This fixture pins queries that actually exercise the new code path: + +- **PROGRESSIVE** — full-AND returns zero on at least one shard; + `_progressive_and_token_chains` drops shortest-tokens-first and + retries. +- **DF_FILTER** — every AND chain returns zero on at least one shard; + search falls to OR-mode where `_filter_or_pool_by_df` drops + high-DF tokens. +- **CONTROL** — full-AND succeeds on every shard. Cache_keys must + remain identical BEFORE/AFTER for any retrieval-side change. + +Use this fixture for any future retrieval-side work (e.g. #000039 +sqlite-vec backend) — it's the only way to get a real signal. + +## Methodology + +``` +BEFORE = HEAD with arborist/search/fts5.py reverted to 416f956~1 +AFTER = HEAD (progressive-AND + DF filter active) +``` + +n=3 samples per (question, mode) cell. `qa_sweep.py --n` burns the +cached record between samples so Hermes nondeterminism becomes the +variation source within a condition. + +## Cache-key parity (the load-bearing diagnostic) + +Cache_key matching ⇒ retrieval produced byte-identical +`(source_root, model_profile, conversation, governance_policy, …)` +input to the LLM. STRICT-rate variance under matching cache_keys is +LLM sampling noise, not retrieval signal. + +| category | cells | matching cks | drifted cks | +|---|---|---|---| +| PROGRESSIVE | 15 | 6 | **9** | +| DF_FILTER | 6 | 3 | **3** | +| CONTROL | 6 | **6** | 0 | + +CONTROL = 0 drift confirms the change is inert on happy-path +queries. PROGRESSIVE / DF_FILTER drift is the change actively +producing different retrieval — which is what we wanted to verify. + +## STRICT-rate aggregate by category + +| category | BEFORE | AFTER | delta | +|---|---|---|---| +| PROGRESSIVE | 11/45 (24.4%) | 11/45 (24.4%) | +0.0pp | +| DF_FILTER | 10/18 (55.6%) | 11/18 (61.1%) | +5.6pp | +| CONTROL | 15/18 (83.3%) | 15/18 (83.3%) | +0.0pp | + +PROGRESSIVE and CONTROL aggregate STRICT counts are byte-identical +between BEFORE and AFTER. DF_FILTER gains one STRICT cell (within the +~5pp noise floor). **No STRICT regression in any category.** + +The PROGRESSIVE 24.4% absolute STRICT-rate is low because the +fixture intentionally targets queries the system finds hard — these +are obscure-place questions, multi-entity questions, and one +specific-fact question (Klotzsch / sphagnum) the corpus may not have +clean evidence for. The fixture's job is to exercise the code path, +not to score well; the scoreboard is `bench/qa_questions.txt`. + +## Median latency by category + +| category | BEFORE median | AFTER median | delta | +|---|---|---|---| +| PROGRESSIVE | 10.3s | 6.7s | **−3.6s** | +| DF_FILTER | 9.5s | 5.5s | **−3.9s** | +| CONTROL | 8.1s | 8.1s | ±0.0s | + +CONTROL latency delta of exactly zero is a beautiful noise-floor +calibration: when retrieval input is identical, the LLM produces +roughly the same wall regardless of the search code path. The +−3.6s/−3.9s wins on PROGRESSIVE / DF_FILTER are real retrieval +speedup translating into total wall, which on Hermes-3-8B is +otherwise dominated by generation time. + +## Key per-cell observations + +- **`what year did Klotzsch describe sphagnum`**: BEFORE latency + 16-23s, AFTER 7-8s — this is the cleanest case where OR-fallback + was hitting a high-DF token that progressive-AND now sidesteps. + STRICT-rate is 0 in both conditions because the corpus doesn't + have clean Klotzsch / sphagnum evidence; the test is about + retrieval cost, not answer quality. +- **`is Gundremmingen near Munich`**: AFTER quote mode picks up 1 + STRICT (vs 0 BEFORE) on n=3 — coincidence noise (cells differ in + cache_keys, so retrieval is genuinely different, but the +1 is + within the n=3 floor). +- **`who painted the mona lisa? · claim_lattice_pointer`**: + BEFORE 1/3, AFTER 0/3. cache_keys MATCH, so this is LLM dice. The + pointer mode's 33% sample-to-sample variance on this question is + observed in every smoke A/B too (see 2026-05-09 smoke A/B + investigation). + +## What this fixture is and isn't + +**Is**: A retrieval-side regression detector. Any change to +`arborist/search/`, `arborist/qa/query.py`'s retrieval pipeline, or +the synonym/concept layer should bench against this fixture and +expect: + +- CONTROL cache_keys remain identical (the happy path didn't drift). +- PROGRESSIVE / DF_FILTER cache_keys may differ (the changed path is + doing something). +- STRICT-rate aggregate within ±5pp on each category. +- Latency: hopefully better on PROGRESSIVE / DF_FILTER, unchanged on + CONTROL. + +**Isn't**: A scoreboard for absolute retrieval quality. The +PROGRESSIVE category is intentionally hard. The full bench +(`bench/qa_questions.txt`) stays the scoreboard for +prompt/verifier/lattice changes. + +## Reproducing + +```bash +# AFTER (HEAD) +make bench-qa-progressive-and BENCH_QA_OUT=bench/qa_results_progressive/after + +# BEFORE (revert search/fts5.py temporarily) +git show 416f956~1:arborist/search/fts5.py > arborist/search/fts5.py +make bench-qa-progressive-and BENCH_QA_OUT=bench/qa_results_progressive/before +git checkout HEAD -- arborist/search/fts5.py +``` + +## Source files + +- Fixture: `bench/qa_questions_progressive_and.txt` +- Makefile target: `bench-qa-progressive-and` (n=3 default, + `BENCH_PROGRESSIVE_N=N` to override) +- BEFORE results: `bench/qa_results_progressive/before/` +- AFTER results: `bench/qa_results_progressive/after/`