#000050: add §2a — semantic-allusion fixture set (tracked cases)

Enumerates the concrete query-words-share-zero-tokens-with-target-title
cases the §2 gate's "semantic-allusion fixtures" must include, as a
running list: Orwell→Eastasia (genuine conceptual allusion — the case
that justifies the vec layer), "what is a CPU?"→Central processing
unit, "what is a GPU?"→Graphics processing unit (abbreviation→expansion
subclass — also fixable upstream by a concepts/ synonym edge; bench
records which fix closes each row). Field cases 2026-05-13, fox.
#000053 fixed the verifier's separate acronym blind spot but not this
retrieval gap.
This commit is contained in:
russell@unturf.com 2026-05-12 21:03:35 -04:00
parent d9760bc5bb
commit 4ec97bdfaf
No known key found for this signature in database
2 changed files with 32 additions and 4 deletions

File diff suppressed because one or more lines are too long

View file

@ -61,9 +61,10 @@ Phase 2 implementation does **not** open until both:
**#000051** — this prereq is "a vecpack exists and has been
imported on the bench box", not "fox embedded the corpus locally".
2. **A recall bench clears the gate.** Run **four conditions** on the
bench fixtures (bench-emergent semantic-allusion shapes + the
curated set + the adversarial semantic-neighbor pack from §3
below): **(A)** FTS5-only baseline, **(B)** vec-only, **(C)** FTS5
bench fixtures (the **§2a semantic-allusion fixture set** + the
bench-emergent shapes + the curated set + the adversarial
semantic-neighbor pack from §3 below): **(A)** FTS5-only baseline,
**(B)** vec-only, **(C)** FTS5
+ vec RRF hybrid, **(D)** FTS5 + vec candidate-union but the
*existing* reranker only (no RRF). D is the control that
distinguishes "RRF helped" from "more candidates helped" — if C
@ -87,6 +88,33 @@ Phase 2 implementation does **not** open until both:
(`--backend vec`), hybrid does NOT become the default, and the
~2.4 GB tax isn't paid for nothing.
## 2a. Semantic-allusion fixture set (the cases the §2 bench must include)
The §2 gate's "semantic-allusion fixtures" aren't a vague category —
this is the running list of concrete cases where the query's words
share *zero tokens* with the target article's title, so the four FTS5
routes (body BM25, title-LIKE, title-token, phrase-pattern) structurally
can't surface it, but a dense-vector route should. The vec-only bench
(condition B) must beat FTS5-only (A) by ≥ 5pp recall@K on **≥ 1** of
these *and* surface ≥ 1 correct candidate absent from A's top-K; the
hybrid bench (C) must lift the proof-quality-label rate on the set.
When a new field case of this shape shows up, add a row here.
| query | target article (title shares 0 tokens with query) | why FTS5 misses it |
|---|---|---|
| "what did Orwell call the country at war with Oceania?" | *Nineteen Eighty-Four* (→ Eastasia / Eurasia) | query tokens {orwell, country, war, oceania}; the article's distinctive content ("Eastasia", "always been at war with") isn't in the title, and the title shares no token with the query — only the verbatim *phrase* route ("always been at war") rescues a piece of it today (CLAUDE.md retrieval-pipeline §1) |
| "what is a CPU?" | *Central processing unit* | query token "CPU" ≠ the title "Central processing unit" — FTS5 pulls the "CPU *" satellites (CPU design, CPU socket, CPU time, CPU cache, CPU-Z, CPU multiplier, CPU disambiguation) where "CPU" is *in* the title; the canonical article never surfaces. (Field case 2026-05-13, fox. Note: #000053 fixed the *verifier*'s separate acronym blind spot — `EVIDENCE-WARRANTED` instead of a spurious `TITLE_MISMATCH` — but the answer is still grounded in "CPU design", not the real article; that's *this* gap.) |
| "what is a GPU?" | *Graphics processing unit* | identical shape to CPU — FTS5 returns GPU cluster / GPU (disambiguation) / Molecular modeling on GPU / Evergreen (GPU family) / etc.; observed 0/1 UNGROUNDED because the model recited the (modern) GPU-lead definition and *no* 2003-dump GPU-* satellite carried that text. (Field case 2026-05-13, fox.) |
(The abbreviation→expansion subclass — `CPU``Central processing unit`,
`GPU``Graphics processing unit` — is *also* addressable upstream by a
`concepts/` synonym edge, cheaper than vec; both fixes are
complementary and the bench should record which one (or both) closes
each row. The Orwell row is *not* an abbreviation case — it's genuine
conceptual allusion, the kind only dense vectors reach — so it's the
one that actually justifies the vec layer if the abbreviation cases
turn out to be all `concepts/` could-have-fixed.)
## 3. Implementation sketch (when the gate clears)
**RRF merge.** `arborist/qa/query.py`: after the four FTS5 routes