arborist/docs/reference-frame-failure-class.md

Reference-frame failure class — the Orwell case

Date: 2026-05-01 Scope: Worked example documenting a failure mode in retrieval + phrase-pattern boost as the response. Audience: fox + future blackops shifts. Status: investigation log; the phrase-pattern route landed in the same commit window. Not a ticket — this is a journal entry.

---

What happened

Run 1 (no augmentation):

make query Q="has oceania always been at war with east asia"
  HYBRID 2/16  via claim_lattice
  sources: Oceania · Asia · Outline of Oceania · Far East · ...

The system selected the literal geography frame — answered as if asked about real-world Oceania and East Asia regions. The Nineteen Eighty-Four article was nowhere in the top-K despite existing in shard 003.db.

Run 2 (manual augmentation — K= flag):

make query Q="has oceania always been at war with east asia? do you understand what this reference"
  STRICT 1/1  via claim_lattice
  sources: Nineteen Eighty-Four (top result)

Adding the literal word "reference" to the question pushed the retrieval into the right map. The user solved it themselves with operator hints — but a high-quality system shouldn't need those hints for diagnostic phrases.

The diagnosis

The cause was retrieval-shape, not model behavior:

Q tokens (FTS5-tokenized, no stem): {has, oceania, always, been, at, war, with, east, asia}
Q content tokens (stopword-stripped): {oceania, war, east, asia}

For these content tokens:

Article	spin	glass	modeling	tensors	distinct match
Nineteen Eighty-Four	0	-	-	-	(no overlap on title-tokens at all)
Oceania	many	-	-	-	1 token in title (oceania)
Foreign relations of Axis	-	-	-	-	0

The diagnostic signal wasn't in the content-token overlap. It was in the verbatim 5-token phrase oceania always been at war which appears in the 1984 article's body but nowhere else.

Pre-2026-05-01 retrieval had three routes (body BM25, title-LIKE, core-keyword) — none of them captured "find articles whose body contains a verbatim multi-token sequence from the question." So the 1984 article was effectively invisible to the search.

The fix — phrase-pattern retrieval route

aborist/qa/query.py:_search_phrases adds a fourth retrieval route that runs an FTS5 quoted-phrase MATCH for each n-gram extracted from the question:

_question_phrases("has oceania always been at war with east asia", n=5)
# → ["has oceania always been at",
#    "oceania always been at war",  ← diagnostic Orwell signal
#    "always been at war with",
#    "been at war with east",
#    "at war with east asia"]

Two pass-throughs — n=6 (specificity 100) and n=5 (specificity 90). 4-grams were tried and rejected: "always been at war" matches generic war-history articles too noisily for the rerank pipeline to separate. 5+ tokens trade recall for precision; most allusions survive at length 5.

_filter_by_title_relevance accept-path 4 lets phrase-route hits bypass the title-token-overlap gate. The 1984 article's title shares zero tokens with the question — without accept-path 4, phrase-route candidates would be retrieved and immediately filtered out before they could rerank into top-K.

Result

After the fix:

make query Q="has oceania always been at war with east asia"
  EVIDENCE-LINKED · via claim_lattice  1/1  11.5s  (fresh)
  - The text does not directly state that Oceania has always been at
    war with East Asia. The passage describes a change in alliances,
    where Oceania switched from being allies with Eastasia to being
    allies with Eurasia, and the public was manipulated to accept
    this change without realizing it.
    [E13 | Nineteen Eighty-Four | 682f0a11: "...To hide such
    contradictions, history is re-written to explain that the (new)
    alliance always was so..."]

No operator augmentation needed. The Orwell frame surfaces from phrase-route alone.

Why not a "Reference Frame Router"?

An earlier sketch (in fox's review, 2026-05-01 Asia/Kuala_Lumpur) proposed a hand-rolled REFERENCE_PATTERNS table mapping query shapes ("has X always been at war with Y") to known references (Orwell). That approach was rejected because:

• Scaling: allusions are long-tail. winter is coming, the cake is a lie, may the force be with you, to be or not to be — the catalog is open-ended.
• Maintenance: per-pattern code rots; new allusions need new rules.
• The corpus already knows: 1984 article exists in the corpus. The defect was retrieval not surfacing it. Fix retrieval, not add a new pre-retrieval stage.

Phrase-pattern boost generalizes: any verbatim 5+ token sequence from the question that appears in an article body lifts that article into consideration, regardless of whether it's an Orwell reference, a Star Wars quote, a Hamlet line, or a meme.

Bench coverage

Added to bench/qa_questions.txt under # allusion / reference frame:

has oceania always been at war with east asia?
who said may the force be with you?
what does winter is coming mean?
what is the meaning of rosebud?
who said to be or not to be that is the question?
what does the cake is a lie reference?

A future bench sweep will reveal whether phrase-pattern boost generalizes across these allusions or if some need different treatment.

Related concerns NOT addressed here

• Source-role taxonomy — proposed roles like primary_reference_source (cited reference work) vs literal_world_background (geography article that's tangentially related). Currently all retrieved sources land as background_source in the rendered output; finer-grained roles could improve answer framing. Out of scope for this fix.
• Frame-assumption verifier field — orthogonal to STRICT/HYBRID/ UNGROUNDED labels. Records "answered as Orwell reference" vs "answered as geography." Useful for audit but requires verifier schema work; deferred.
• Reference-aware prompting — system prompt could acknowledge reference-frame ambiguity ("if the query reads as an allusion, prefer the reference frame when retrieved evidence supports it"). Per fox's "less prompt engineering, more code-level discipline" preference, deferred until phrase-pattern boost's empirical performance is known.