russell@unturf.com
1b8677d3d5
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
169 lines
6.4 KiB
Markdown
169 lines
6.4 KiB
Markdown
# Reference-frame failure class — the Orwell case
|
|
|
|
**Date:** 2026-05-01
|
|
**Scope:** Worked example documenting a failure mode in retrieval +
|
|
phrase-pattern boost as the response.
|
|
**Audience:** fox + future blackops shifts.
|
|
**Status:** investigation log; the phrase-pattern route landed in
|
|
the same commit window. Not a ticket — this is a journal entry.
|
|
|
|
---
|
|
|
|
## What happened
|
|
|
|
Run 1 (no augmentation):
|
|
|
|
```
|
|
make query Q="has oceania always been at war with east asia"
|
|
HYBRID 2/16 via claim_lattice
|
|
sources: Oceania · Asia · Outline of Oceania · Far East · ...
|
|
```
|
|
|
|
The system selected the **literal geography frame** — answered as
|
|
if asked about real-world Oceania and East Asia regions. The
|
|
Nineteen Eighty-Four article was nowhere in the top-K despite
|
|
existing in shard `003.db`.
|
|
|
|
Run 2 (manual augmentation — `K=` flag):
|
|
|
|
```
|
|
make query Q="has oceania always been at war with east asia? do you understand what this reference"
|
|
STRICT 1/1 via claim_lattice
|
|
sources: Nineteen Eighty-Four (top result)
|
|
```
|
|
|
|
Adding the literal word "reference" to the question pushed the
|
|
retrieval into the right map. The user solved it themselves with
|
|
operator hints — but a high-quality system shouldn't need those
|
|
hints for diagnostic phrases.
|
|
|
|
## The diagnosis
|
|
|
|
The cause was retrieval-shape, not model behavior:
|
|
|
|
```
|
|
Q tokens (FTS5-tokenized, no stem): {has, oceania, always, been, at, war, with, east, asia}
|
|
Q content tokens (stopword-stripped): {oceania, war, east, asia}
|
|
```
|
|
|
|
For these content tokens:
|
|
|
|
| Article | spin | glass | modeling | tensors | distinct match |
|
|
|----------------------------|------|-------|----------|---------|----------------|
|
|
| Nineteen Eighty-Four | 0 | - | - | - | (no overlap on title-tokens at all) |
|
|
| Oceania | many | - | - | - | 1 token in title (oceania) |
|
|
| Foreign relations of Axis | - | - | - | - | 0 |
|
|
|
|
The **diagnostic signal** wasn't in the content-token overlap. It
|
|
was in the verbatim 5-token phrase `oceania always been at war`
|
|
which appears in the 1984 article's body but nowhere else.
|
|
|
|
Pre-2026-05-01 retrieval had three routes (body BM25, title-LIKE,
|
|
core-keyword) — none of them captured "find articles whose body
|
|
contains a verbatim multi-token sequence from the question." So
|
|
the 1984 article was effectively invisible to the search.
|
|
|
|
## The fix — phrase-pattern retrieval route
|
|
|
|
`aborist/qa/query.py:_search_phrases` adds a fourth retrieval
|
|
route that runs an FTS5 quoted-phrase MATCH for each n-gram
|
|
extracted from the question:
|
|
|
|
```python
|
|
_question_phrases("has oceania always been at war with east asia", n=5)
|
|
# → ["has oceania always been at",
|
|
# "oceania always been at war", ← diagnostic Orwell signal
|
|
# "always been at war with",
|
|
# "been at war with east",
|
|
# "at war with east asia"]
|
|
```
|
|
|
|
Two pass-throughs — n=6 (specificity 100) and n=5 (specificity 90).
|
|
4-grams were tried and rejected: "always been at war" matches generic
|
|
war-history articles too noisily for the rerank pipeline to separate.
|
|
5+ tokens trade recall for precision; most allusions survive at
|
|
length 5.
|
|
|
|
**`_filter_by_title_relevance` accept-path 4** lets phrase-route
|
|
hits bypass the title-token-overlap gate. The 1984 article's title
|
|
shares zero tokens with the question — without accept-path 4,
|
|
phrase-route candidates would be retrieved and immediately filtered
|
|
out before they could rerank into top-K.
|
|
|
|
## Result
|
|
|
|
After the fix:
|
|
|
|
```
|
|
make query Q="has oceania always been at war with east asia"
|
|
EVIDENCE-LINKED · via claim_lattice 1/1 11.5s (fresh)
|
|
- The text does not directly state that Oceania has always been at
|
|
war with East Asia. The passage describes a change in alliances,
|
|
where Oceania switched from being allies with Eastasia to being
|
|
allies with Eurasia, and the public was manipulated to accept
|
|
this change without realizing it.
|
|
[E13 | Nineteen Eighty-Four | 682f0a11: "...To hide such
|
|
contradictions, history is re-written to explain that the (new)
|
|
alliance always was so..."]
|
|
```
|
|
|
|
No operator augmentation needed. The Orwell frame surfaces from
|
|
phrase-route alone.
|
|
|
|
## Why not a "Reference Frame Router"?
|
|
|
|
An earlier sketch (in fox's review, 2026-05-01 Asia/Kuala_Lumpur)
|
|
proposed a hand-rolled `REFERENCE_PATTERNS` table mapping query
|
|
shapes ("has X always been at war with Y") to known references
|
|
(Orwell). That approach was rejected because:
|
|
|
|
- **Scaling**: allusions are long-tail. `winter is coming`,
|
|
`the cake is a lie`, `may the force be with you`, `to be or
|
|
not to be` — the catalog is open-ended.
|
|
- **Maintenance**: per-pattern code rots; new allusions need new
|
|
rules.
|
|
- **The corpus already knows**: 1984 article exists in the corpus.
|
|
The defect was retrieval not surfacing it. Fix retrieval, not
|
|
add a new pre-retrieval stage.
|
|
|
|
Phrase-pattern boost generalizes: any verbatim 5+ token sequence
|
|
from the question that appears in an article body lifts that
|
|
article into consideration, regardless of whether it's an Orwell
|
|
reference, a Star Wars quote, a Hamlet line, or a meme.
|
|
|
|
## Bench coverage
|
|
|
|
Added to `bench/qa_questions.txt` under `# allusion / reference
|
|
frame`:
|
|
|
|
```
|
|
has oceania always been at war with east asia?
|
|
who said may the force be with you?
|
|
what does winter is coming mean?
|
|
what is the meaning of rosebud?
|
|
who said to be or not to be that is the question?
|
|
what does the cake is a lie reference?
|
|
```
|
|
|
|
A future bench sweep will reveal whether phrase-pattern boost
|
|
generalizes across these allusions or if some need different
|
|
treatment.
|
|
|
|
## Related concerns NOT addressed here
|
|
|
|
- **Source-role taxonomy** — proposed roles like
|
|
`primary_reference_source` (cited reference work) vs
|
|
`literal_world_background` (geography article that's tangentially
|
|
related). Currently all retrieved sources land as
|
|
`background_source` in the rendered output; finer-grained roles
|
|
could improve answer framing. Out of scope for this fix.
|
|
- **Frame-assumption verifier field** — orthogonal to STRICT/HYBRID/
|
|
UNGROUNDED labels. Records "answered as Orwell reference" vs
|
|
"answered as geography." Useful for audit but requires verifier
|
|
schema work; deferred.
|
|
- **Reference-aware prompting** — system prompt could acknowledge
|
|
reference-frame ambiguity ("if the query reads as an allusion,
|
|
prefer the reference frame when retrieved evidence supports it").
|
|
Per fox's "less prompt engineering, more code-level discipline"
|
|
preference, deferred until phrase-pattern boost's empirical
|
|
performance is known.
|