#000050: add §2a — semantic-allusion fixture set (tracked cases)
Enumerates the concrete query-words-share-zero-tokens-with-target-title cases the §2 gate's "semantic-allusion fixtures" must include, as a running list: Orwell→Eastasia (genuine conceptual allusion — the case that justifies the vec layer), "what is a CPU?"→Central processing unit, "what is a GPU?"→Graphics processing unit (abbreviation→expansion subclass — also fixable upstream by a concepts/ synonym edge; bench records which fix closes each row). Field cases 2026-05-13, fox. #000053 fixed the verifier's separate acronym blind spot but not this retrieval gap.
This commit is contained in:
parent
d9760bc5bb
commit
4ec97bdfaf
2 changed files with 32 additions and 4 deletions
File diff suppressed because one or more lines are too long
|
|
@ -61,9 +61,10 @@ Phase 2 implementation does **not** open until both:
|
|||
**#000051** — this prereq is "a vecpack exists and has been
|
||||
imported on the bench box", not "fox embedded the corpus locally".
|
||||
2. **A recall bench clears the gate.** Run **four conditions** on the
|
||||
bench fixtures (bench-emergent semantic-allusion shapes + the
|
||||
curated set + the adversarial semantic-neighbor pack from §3
|
||||
below): **(A)** FTS5-only baseline, **(B)** vec-only, **(C)** FTS5
|
||||
bench fixtures (the **§2a semantic-allusion fixture set** + the
|
||||
bench-emergent shapes + the curated set + the adversarial
|
||||
semantic-neighbor pack from §3 below): **(A)** FTS5-only baseline,
|
||||
**(B)** vec-only, **(C)** FTS5
|
||||
+ vec RRF hybrid, **(D)** FTS5 + vec candidate-union but the
|
||||
*existing* reranker only (no RRF). D is the control that
|
||||
distinguishes "RRF helped" from "more candidates helped" — if C
|
||||
|
|
@ -87,6 +88,33 @@ Phase 2 implementation does **not** open until both:
|
|||
(`--backend vec`), hybrid does NOT become the default, and the
|
||||
~2.4 GB tax isn't paid for nothing.
|
||||
|
||||
## 2a. Semantic-allusion fixture set (the cases the §2 bench must include)
|
||||
|
||||
The §2 gate's "semantic-allusion fixtures" aren't a vague category —
|
||||
this is the running list of concrete cases where the query's words
|
||||
share *zero tokens* with the target article's title, so the four FTS5
|
||||
routes (body BM25, title-LIKE, title-token, phrase-pattern) structurally
|
||||
can't surface it, but a dense-vector route should. The vec-only bench
|
||||
(condition B) must beat FTS5-only (A) by ≥ 5pp recall@K on **≥ 1** of
|
||||
these *and* surface ≥ 1 correct candidate absent from A's top-K; the
|
||||
hybrid bench (C) must lift the proof-quality-label rate on the set.
|
||||
When a new field case of this shape shows up, add a row here.
|
||||
|
||||
| query | target article (title shares 0 tokens with query) | why FTS5 misses it |
|
||||
|---|---|---|
|
||||
| "what did Orwell call the country at war with Oceania?" | *Nineteen Eighty-Four* (→ Eastasia / Eurasia) | query tokens {orwell, country, war, oceania}; the article's distinctive content ("Eastasia", "always been at war with") isn't in the title, and the title shares no token with the query — only the verbatim *phrase* route ("always been at war") rescues a piece of it today (CLAUDE.md retrieval-pipeline §1) |
|
||||
| "what is a CPU?" | *Central processing unit* | query token "CPU" ≠ the title "Central processing unit" — FTS5 pulls the "CPU *" satellites (CPU design, CPU socket, CPU time, CPU cache, CPU-Z, CPU multiplier, CPU disambiguation) where "CPU" is *in* the title; the canonical article never surfaces. (Field case 2026-05-13, fox. Note: #000053 fixed the *verifier*'s separate acronym blind spot — `EVIDENCE-WARRANTED` instead of a spurious `TITLE_MISMATCH` — but the answer is still grounded in "CPU design", not the real article; that's *this* gap.) |
|
||||
| "what is a GPU?" | *Graphics processing unit* | identical shape to CPU — FTS5 returns GPU cluster / GPU (disambiguation) / Molecular modeling on GPU / Evergreen (GPU family) / etc.; observed 0/1 UNGROUNDED because the model recited the (modern) GPU-lead definition and *no* 2003-dump GPU-* satellite carried that text. (Field case 2026-05-13, fox.) |
|
||||
|
||||
(The abbreviation→expansion subclass — `CPU`↔`Central processing unit`,
|
||||
`GPU`↔`Graphics processing unit` — is *also* addressable upstream by a
|
||||
`concepts/` synonym edge, cheaper than vec; both fixes are
|
||||
complementary and the bench should record which one (or both) closes
|
||||
each row. The Orwell row is *not* an abbreviation case — it's genuine
|
||||
conceptual allusion, the kind only dense vectors reach — so it's the
|
||||
one that actually justifies the vec layer if the abbreviation cases
|
||||
turn out to be all `concepts/` could-have-fixed.)
|
||||
|
||||
## 3. Implementation sketch (when the gate clears)
|
||||
|
||||
**RRF merge.** `arborist/qa/query.py`: after the four FTS5 routes
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue