# Progressive-AND fallback A/B (commit 416f956)

**Date**: 2026-05-09
**Change**: `arborist/search/fts5.py` — progressive-AND fallback +
DF filter at OR-pool (commit
[`416f956`](https://git.unturf.com/) ).
**Fixture**: `bench/qa_questions_progressive_and.txt` (9 questions ×
3 modes × n=3 = 81 samples per condition).
**Hardware**: 4 wiki shards (~10 GB each, 1.5M chunks each), serial
shard workers (`ARBORIST_SHARD_WORKERS=1`) for clean per-shard
attribution; `bench-qa-progressive-and` uses the default (parallel)
worker count for total-wall measurements.

## Why this fixture exists

`bench/qa_questions_smoke.txt` is structurally insensitive to
retrieval-side changes — every smoke query succeeds at full-AND
chain 0 on every shard, so no retrieval-side change can move its
cache_keys. We discovered this hard way A/B-benching commit 416f956
against the smoke fixture: STRICT-rate moved ±20pp at n=3 with every
single cache_key matching byte-for-byte BEFORE and AFTER. Pure LLM
sampling variance, no retrieval signal at all.

This fixture pins queries that actually exercise the new code path:

- **PROGRESSIVE** — full-AND returns zero on at least one shard;
  `_progressive_and_token_chains` drops shortest-tokens-first and
  retries.
- **DF_FILTER** — every AND chain returns zero on at least one shard;
  search falls to OR-mode where `_filter_or_pool_by_df` drops
  high-DF tokens.
- **CONTROL** — full-AND succeeds on every shard. Cache_keys must
  remain identical BEFORE/AFTER for any retrieval-side change.

Use this fixture for any future retrieval-side work (e.g. #000039
sqlite-vec backend) — it's the only way to get a real signal.

## Methodology

```
BEFORE = HEAD with arborist/search/fts5.py reverted to 416f956~1
AFTER  = HEAD (progressive-AND + DF filter active)
```

n=3 samples per (question, mode) cell. `qa_sweep.py --n` burns the
cached record between samples so Hermes nondeterminism becomes the
variation source within a condition.

## Cache-key parity (the load-bearing diagnostic)

Cache_key matching ⇒ retrieval produced byte-identical
`(source_root, model_profile, conversation, governance_policy, …)`
input to the LLM. STRICT-rate variance under matching cache_keys is
LLM sampling noise, not retrieval signal.

| category | cells | matching cks | drifted cks |
|---|---|---|---|
| PROGRESSIVE | 15 | 6 | **9** |
| DF_FILTER | 6 | 3 | **3** |
| CONTROL | 6 | **6** | 0 |

CONTROL = 0 drift confirms the change is inert on happy-path
queries. PROGRESSIVE / DF_FILTER drift is the change actively
producing different retrieval — which is what we wanted to verify.

## STRICT-rate aggregate by category

| category | BEFORE | AFTER | delta |
|---|---|---|---|
| PROGRESSIVE | 11/45 (24.4%) | 11/45 (24.4%) | +0.0pp |
| DF_FILTER | 10/18 (55.6%) | 11/18 (61.1%) | +5.6pp |
| CONTROL | 15/18 (83.3%) | 15/18 (83.3%) | +0.0pp |

PROGRESSIVE and CONTROL aggregate STRICT counts are byte-identical
between BEFORE and AFTER. DF_FILTER gains one STRICT cell (within the
~5pp noise floor). **No STRICT regression in any category.**

The PROGRESSIVE 24.4% absolute STRICT-rate is low because the
fixture intentionally targets queries the system finds hard — these
are obscure-place questions, multi-entity questions, and one
specific-fact question (Klotzsch / sphagnum) the corpus may not have
clean evidence for. The fixture's job is to exercise the code path,
not to score well; the scoreboard is `bench/qa_questions.txt`.

## Median latency by category

| category | BEFORE median | AFTER median | delta |
|---|---|---|---|
| PROGRESSIVE | 10.3s | 6.7s | **−3.6s** |
| DF_FILTER | 9.5s | 5.5s | **−3.9s** |
| CONTROL | 8.1s | 8.1s | ±0.0s |

CONTROL latency delta of exactly zero is a beautiful noise-floor
calibration: when retrieval input is identical, the LLM produces
roughly the same wall regardless of the search code path. The
−3.6s/−3.9s wins on PROGRESSIVE / DF_FILTER are real retrieval
speedup translating into total wall, which on Hermes-3-8B is
otherwise dominated by generation time.

## Key per-cell observations

- **`what year did Klotzsch describe sphagnum`**: BEFORE latency
  16-23s, AFTER 7-8s — this is the cleanest case where OR-fallback
  was hitting a high-DF token that progressive-AND now sidesteps.
  STRICT-rate is 0 in both conditions because the corpus doesn't
  have clean Klotzsch / sphagnum evidence; the test is about
  retrieval cost, not answer quality.
- **`is Gundremmingen near Munich`**: AFTER quote mode picks up 1
  STRICT (vs 0 BEFORE) on n=3 — coincidence noise (cells differ in
  cache_keys, so retrieval is genuinely different, but the +1 is
  within the n=3 floor).
- **`who painted the mona lisa? · claim_lattice_pointer`**:
  BEFORE 1/3, AFTER 0/3. cache_keys MATCH, so this is LLM dice. The
  pointer mode's 33% sample-to-sample variance on this question is
  observed in every smoke A/B too (see 2026-05-09 smoke A/B
  investigation).

## What this fixture is and isn't

**Is**: A retrieval-side regression detector. Any change to
`arborist/search/`, `arborist/qa/query.py`'s retrieval pipeline, or
the synonym/concept layer should bench against this fixture and
expect:

- CONTROL cache_keys remain identical (the happy path didn't drift).
- PROGRESSIVE / DF_FILTER cache_keys may differ (the changed path is
  doing something).
- STRICT-rate aggregate within ±5pp on each category.
- Latency: hopefully better on PROGRESSIVE / DF_FILTER, unchanged on
  CONTROL.

**Isn't**: A scoreboard for absolute retrieval quality. The
PROGRESSIVE category is intentionally hard. The full bench
(`bench/qa_questions.txt`) stays the scoreboard for
prompt/verifier/lattice changes.

## Reproducing

```bash
# AFTER (HEAD)
make bench-qa-progressive-and BENCH_QA_OUT=bench/qa_results_progressive/after

# BEFORE (revert search/fts5.py temporarily)
git show 416f956~1:arborist/search/fts5.py > arborist/search/fts5.py
make bench-qa-progressive-and BENCH_QA_OUT=bench/qa_results_progressive/before
git checkout HEAD -- arborist/search/fts5.py
```

## Source files

- Fixture: `bench/qa_questions_progressive_and.txt`
- Makefile target: `bench-qa-progressive-and` (n=3 default,
  `BENCH_PROGRESSIVE_N=N` to override)
- BEFORE results: `bench/qa_results_progressive/before/`
- AFTER results: `bench/qa_results_progressive/after/`
