engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

qa: lock phrase-route non-regression tests + open ticket #000002 (Module L)

Browse source 
Two follow-ups to the phrase-pattern retrieval fix (commit 1b8677d)
covering items 6 and 10-11 of fox's 2026-05-01 architectural review:

(1) Non-regression tests for the phrase route:
  - test_phrase_route_skipped_when_question_shorter_than_min_n
    pins the structural false-positive guard: the n=5/n=6 minimum
    means a 4-token literal-geography query lacks enough tokens to
    trigger the route at all.
  - test_phrase_route_does_not_hijack_literal_geography_query
    end-to-end: a 4-token "oceania east asia geography" query on
    a synthetic 2-doc corpus surfaces only the geography-stub doc;
    the orwell-stub doc (whose body has the diagnostic 5-gram) is
    correctly NOT pulled in by the phrase route on a literal query.

(2) docs/ticket-000002-reference-frame-polarity-contract.md
    Captures fox's Module L proposal verbatim as Appendix A and
    extracts the implementation sketch into the standard ticket
    body (problem statement, abstraction, CTI interpretation, three
    pieces of code to write, test list, scope boundaries).

    The phrase route closed the RETRIEVAL side of reference-frame
    failure. Module L addresses the ANSWER side: today's substrate
    answers Orwell queries as "the text does not directly state..."
    when it should produce multi-frame answers distinguishing
    Party propaganda from fictional-actual continuity. Forecast
    cost ~3-4 hours; risk medium (prompt augmentation interaction
    with claim_lattice prompt).

    Module M = ticket #000001 (route provenance binding); not
    duplicated. Module N (FP guards) partially landed via the
    tests above; remaining tests folded into ticket #000002's
    test list. Module H (relation warrant lite) lacks scope
    detail; deferred without a ticket.

(3) docs/TICKETS.md updated: index gains #000002 row, Next ID
    bumped to 000003.
This commit is contained in:
russell@unturf.com 2026-05-01 14:03:39 -04:00
parent 1b8677d3d5
commit a99ac4388b
No known key found for this signature in database
3 changed files with 724 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

3
docs/TICKETS.md
View file

@ -55,8 +55,9 @@ Newest first. Update on every open/close.
| ID | Title | Status | Opened |
|----------|------------------------------------------------|---------|------------|
| #000002 | Reference-Frame Polarity Contract (Module L) | open | 2026-05-01 |
| #000001 | Retrieval-keywords audit gap | open | 2026-05-01 |
## Next ID
`000002`
`000003`

660
docs/ticket-000002-reference-frame-polarity-contract.md Normal file
View file

@ -0,0 +1,660 @@
# Ticket #000002 — Reference-Frame Polarity Contract (Module L)
**Status:** open · awaiting go/no-go
**Opened:** 2026-05-01
**Scope:** Multi-frame answer compilation for queries that admit
literal vs fictional-actual vs in-universe-propaganda interpretations.
Builds on the phrase-pattern retrieval route (commit `1b8677d`)
which closed the **retrieval** side of reference-frame failure;
this ticket addresses the **answer** side.
**Audience:** fox + future blackops shifts.
**Hard constraint:** the phrase-pattern boost stays as a non-regression
contract. Polarity is an additive answer-compilation layer; no
removal of the retrieval-side fix.
---
## 1. Problem statement
After the phrase-pattern retrieval route landed (commit `1b8677d`),
queries like:
```
make query Q="has oceania always been at war with east asia"
```
now correctly surface the Nineteen Eighty-Four article and produce
EVIDENCE-LINKED answers. The remaining defect is **answer framing**:
```
- The text does not directly state that Oceania has always been at
war with East Asia. The passage describes a change in alliances...
```
This is too defensive. The query admits at least three frames:
| Frame | Correct answer |
|--------------------------------|------------------------------------------------------|
| Literal real-world geography | No / not applicable |
| Fictional actual continuity | No; alliances changed in the novel |
| In-universe propaganda claim | The Party wants citizens to believe yes |
The ideal one-shot answer should distinguish all three:
```
This is a Nineteen Eighty-Four reference. In the Party's propaganda,
Oceania is treated as though it has always been at war with Eastasia.
But within the novel's actual continuity, that is false: the enemy
changes, and the Party rewrites records so the public accepts the
new version as if it had always been true. [E13]
```
Today's substrate has no machinery to recognize that a query lives
in multiple frames simultaneously, let alone to compile an answer
that respects each frame.
---
## 2. The polarity-contract abstraction
For reference-frame queries, compile a small structured contract:
```json
{
"query_type": "reference_frame_truth",
"reference_frame": "Nineteen Eighty-Four",
"literal_entities": ["Oceania", "East Asia"],
"fictional_entities": ["Oceania", "Eastasia"],
"claim_under_test": "Oceania has always been at war with Eastasia",
"polarity_modes": [
"literal_world",
"fictional_actual_history",
"in_universe_propaganda_claim"
],
"required_answer_shape":
"distinguish propaganda claim from fictional actual continuity"
}
```
The answer compiler then must produce, in order:
1. Identify the reference (`Nineteen Eighty-Four`).
2. Answer the question under the propaganda frame (`yes — the Party
claims it`).
3. Correct under the fictional-actual frame (`actually no — alliances
change, history is rewritten`).
4. Cite evidence for each substantive claim.
This prevents the "does not directly state" hedge. The system
explicitly answers the actual question across each polarity.
---
## 3. CTI interpretation
In CTI clause-tree language, the query produces a **frame lattice**:
```text
F1: literal geography
F2: Nineteen Eighty-Four reference (selected by phrase pattern)
F3: in-universe propaganda claim (sub-frame of F2)
F4: fictional actual continuity (sub-frame of F2)
```
The answer is compiled from the surviving nodes:
- F1 quietly dismissed (not the intended reading).
- F2 selected and named explicitly.
- F3 + F4 jointly answered, distinguishing what the Party claims
from what the novel's actual continuity shows.
The additive insight:
> A reference query often requires answering under multiple frames,
> not picking one frame and discarding the others.
---
## 4. Implementation sketch
Three new pieces, each scoped tight:
### 4.1 Frame detector (`aborist/qa/frame.py`)
Given a question + retrieved sources, detect the reference frame:
```python
@dataclass(frozen=True)
class FrameDetection:
frame_kind: Literal["literal", "reference", "ambiguous"]
reference_title: str | None # e.g. "Nineteen Eighty-Four"
polarity_modes: tuple[str, ...] # subset of POLARITY_MODES
confidence: float # 0.0-1.0
def detect_frame(question: str, sources: list[Hit]) -> FrameDetection:
"""Heuristic: phrase-route hit on a fictional source (novel,
film, play, video game) → reference frame. Otherwise literal."""
```
Detection triggers when:
- `phrase_match_roots` is non-empty (the route fired)
- AND the matched source's category indicates a reference work
(novel, film, play, etc.)
Source-category detection: title-keyword heuristic on the doc title
("(novel)", "(film)", "(play)") + body-keyword check (`fiction`,
`novel`, `published in`). No NER, no embeddings.
### 4.2 Polarity-aware prompt augmentation
When `frame_kind == "reference"`, augment the system prompt with a
short polarity instruction:
```
This question may be a reference to {reference_title}. Distinguish:
1. What the cited work depicts as actual continuity.
2. What in-universe propaganda or characters claim.
Cite evidence for each substantive claim.
```
Per fox's "less prompt engineering, more code-level discipline"
preference, this is the lightest-touch nudge that can produce
multi-frame answers from Hermes-3-8B without bespoke per-frame
templates. Heavier alternatives (per-frame answer compilation, NLI
for entailment under each frame) are deferred.
### 4.3 Renderer extension
Add an optional `frame_notes` section to the human render, populated
when `FrameDetection.frame_kind == "reference"`:
```
Reference frame: George Orwell's Nineteen Eighty-Four.
Alias normalized: "East Asia" → "Eastasia" in the Orwell frame.
[answer body...]
Frame notes:
- Literal geography frame: not the intended reading.
- Orwell frame: selected by phrase pattern.
- Propaganda frame: the Party wants citizens to believe yes.
- Continuity frame: alliances changed in the novel.
```
Off by default for non-reference queries. Renderer never affects
verifier output or providence_cache state — display layer only.
---
## 5. Tests required
Already covered (commit `1b8677d` + `<this commit>`):
- `[x]` "has oceania always been at war with east asia" routes
Nineteen Eighty-Four into top-K
- `[x]` Short conventional geography queries don't trigger phrase
route (n=5/n=6 floor + all-short-token skip)
- `[x]` Synthetic-corpus geography query doesn't surface
Orwell-flavored stub via phrase route
- `[x]` Evidence display includes evidence_id, source title,
chunk/root prefix
- `[x]` Accept-path 4 lets phrase-route hits bypass title-relevance
gate
To add when this ticket lands:
- `[ ]` Frame detector returns `reference` when phrase_match_roots
contains a (novel)/(film)/(play)-titled source
- `[ ]` Frame detector returns `literal` when phrase route didn't
fire OR when phrase-matched source is non-fiction
- `[ ]` Polarity-aware prompt gets injected for reference-frame
queries; not injected otherwise
- `[ ]` Live test: Orwell query produces an answer mentioning
both Party propaganda AND alliance changes (multi-frame
answer shape)
- `[ ]` Live regression test: literal geography query produces
a literal-geography answer (no Orwell injection)
- `[ ]` Renderer Frame-notes section appears when frame_kind
is reference; absent otherwise
---
## 6. Out of scope
- **Hand-rolled allusion catalog.** No `REFERENCE_PATTERNS` table
with per-allusion rules. The phrase-pattern route generalizes;
per-pattern code rots.
- **Aliasing East Asia ↔ Eastasia by default.** Token-level alias
rules have false-positive risk. Phrase-pattern matching is the
alias mechanism: `Eastasia` appears verbatim in the 1984 article
body so the query token `east asia` doesn't need to be reframed.
- **NLI-grade entailment under each frame.** Soft signal, separate
ticket if/when the verifier-semantic-gap design (in
`docs/verifier-semantic-gap-design.md`) lands.
- **Module M (route provenance binding)** = ticket #000001. Not
duplicated here.
- **Module H (relation warrant lite).** Different concern (predicate
matching for relation queries like "X's girlfriend"). Mentioned
in fox's review but lacks scope detail; defer.
---
## 7. Status
**Proposal.** Phrase-pattern retrieval route is landed and verified
on the Orwell case (commit `1b8677d`); this ticket addresses the
remaining answer-side gap.
Forecast cost: ~3-4 hours of focused work (frame.py + prompt
augmentation + renderer extension + ~7 tests). Risk: medium —
prompt augmentation interacts with the existing claim_lattice
prompt; need bench evidence to confirm no regression on non-reference
questions.
Forecast value: closes the answer-shape gap on reference-frame
queries. The phrase route currently produces "does not directly
state" hedges; polarity contract produces multi-frame answers
that explicitly distinguish propaganda from continuity.
---
## Appendix A — Architectural review (2026-05-01, Asia/Kuala_Lumpur)
> Fox-supplied review confirming the phrase-pattern fix as a new
> baseline and proposing Module L. Captured verbatim. Sections of
> this ticket above are derived from the appendix; the appendix
> stays in place as the design log.
De novo verdict: this is a **major improvement** and should be treated as a new baseline for the Orwell/reference-frame class.
The important wins:
```text
1. The system now finds the Orwell / Nineteen Eighty-Four frame without operator augmentation.
2. The pointer format now disambiguates evidence ID, source title, and chunk/hash prefix:
[E13 | Nineteen Eighty-Four | 682f0a11: "..."]
3. The label no longer says STRICT; it says EVIDENCE-LINKED.
4. The retrieval bug was real: _search_corpus returned a bare list while the caller expected sidecar attributes, silently disabling an existing core-keyword accept path.
5. The phrase-pattern retrieval route fixes the earlier allusion blindness for this query.
```
That means the previous failure:
```text
"has oceania always been at war with east asia"
→ literal geography frame
```
has been mostly fixed.
---
### A.1 What this now solves
The system previously needed the user to add:
```text
"do you understand what this reference"
```
to find the Orwell frame.
Now:
```text
make query Q="has oceania always been at war with east asia"
```
surfaces *Nineteen Eighty-Four* directly. That is the right zero-shot behavior.
This is exactly what the **Reference Frame Router** was supposed to do:
```text
raw phrase
→ detect high-signal literary/reference pattern
→ route retrieval toward Nineteen Eighty-Four / Eastasia / historical revisionism
→ produce evidence-linked answer
```
So the phrase-pattern route is a valid implementation of that module.
---
### A.2 What still needs correction
The current answer apparently says:
```text
"The text does not directly state..."
```
That is too defensive and slightly misframed.
The better answer should distinguish three things:
```text
A. Literal real-world geography:
No, Oceania and East Asia are not real-world states locked in perpetual war.
B. Orwell reference:
This is a Nineteen Eighty-Four reference.
C. In the novel:
The Party presents the current war alignment as if it had always been true,
but the evidence/plot shows alliances changed and history was rewritten.
```
The ideal one-shot answer should be:
```text
This is a Nineteen Eighty-Four reference. In the Party's propaganda, Oceania is treated as though it has always been at war with Eastasia. But within the novel's actual continuity, that is false: the enemy changes, and the Party rewrites records so the public accepts the new version as if it had always been true. [E13]
```
That is sharper than:
```text
The text does not directly state...
```
because the key point is not absence of a direct statement. The key point is **propaganda-induced historical contradiction**.
---
### A.3 New failure class exposed: reference-frame truth polarity
This query is not just a reference-frame query. It is a **truth-polarity query inside a fictional propaganda frame**.
The question:
```text
has oceania always been at war with east asia
```
has at least three possible interpretations:
```text
F1: literal-world geography
F2: fictional-world factual history inside Nineteen Eighty-Four
F3: Party propaganda claim inside Nineteen Eighty-Four
```
The correct answer is different under each:
| Frame | Answer |
| ------------------------ | ------------------------------------- |
| Literal geography | No / not applicable |
| Fictional actual history | No; alliances changed |
| Party propaganda | The Party wants people to believe yes |
So the next refinement is not retrieval. It is **frame-polarity classification**.
Add:
```text
L — Reference-Frame Polarity Contract
```
---
### A.4 Proposed module L — Reference-Frame Polarity Contract
For allusion/reference queries, compile a small contract:
```json
{
"query_type": "reference_frame_truth",
"reference_frame": "Nineteen Eighty-Four",
"literal_entities": ["Oceania", "East Asia"],
"fictional_entities": ["Oceania", "Eastasia"],
"claim_under_test": "Oceania has always been at war with Eastasia",
"polarity_modes": [
"literal_world",
"fictional_actual_history",
"in_universe_propaganda_claim"
],
"required_answer_shape": "distinguish propaganda claim from fictional actual continuity"
}
```
Then the answer compiler must produce:
```text
- identify the reference,
- answer the actual question,
- separate Party propaganda from actual continuity,
- cite evidence.
```
This prevents the model from giving a vague "does not directly state" answer when it should explain the contradiction.
---
### A.5 Evidence-linked is still the right label
Keep:
```text
EVIDENCE-LINKED · via claim_lattice
```
Do **not** revert to STRICT.
Why:
```text
EVIDENCE-LINKED = pointer/source/chunk relation verified.
```
It does not mean:
```text
all subclaims about propaganda, Hate Week, Eastasia/Eurasia, and historical revisionism
are fully semantically entailed by one span.
```
The new pointer format:
```text
[E13 | Nineteen Eighty-Four | 682f0a11: "..."]
```
is much better than the previous ambiguous `[E5]` display because the user can see the evidence source directly.
---
### A.6 Retrieval side: what to lock in
The phrase-pattern route should be committed as a non-regression test.
Add tests:
```text
[ ] "has oceania always been at war with east asia" routes to Nineteen Eighty-Four.
[ ] "has oceania always been at war with eastasia" routes to Nineteen Eighty-Four.
[ ] "oceania east asia geography" does not over-route to Orwell unless phrase hints appear.
[ ] "oceania population east asia trade" remains literal/geographic.
[ ] phrase route metadata appears in run DAG.
[ ] accept-path 4 remains enabled; no bare-list sidecar regression.
[ ] evidence display includes evidence_id, source title, chunk/root prefix.
```
The false-positive tests matter. Phrase routing is powerful, but it must not hijack ordinary geography queries.
---
### A.7 Provenance side: ensure the route is audit-bound
Because this fix depends on the phrase-pattern retrieval route, the route itself must be captured in provenance.
Bind:
```json
{
"retrieval_route": "phrase_pattern_reference",
"reference_pattern_id": "orwell_1984_oceania_eastasia_war",
"accept_path": 4,
"retrieval_keywords": [
"Nineteen Eighty-Four",
"Oceania",
"Eastasia",
"always been at war"
],
"route_confidence": "high"
}
```
into:
```text
retrieval_plan_hash
run_dag.retrieval_stage
audit_event: retrieval_plan_built
```
This is important because earlier we identified a provenance gap: retrieval keywords/routes must be committed, not just the resulting sources.
If the route is not bound, the answer is correct today but not fully replay-auditable.
---
### A.8 CTI interpretation
In CTI terms, this query should produce a frame lattice:
```text
Frame node F1:
literal geography
Frame node F2:
Nineteen Eighty-Four reference
Frame node F3:
in-universe propaganda claim
Frame node F4:
fictional actual continuity
```
Then the answer should be compiled from the surviving nodes:
```text
F2 selected: reference frame
F3 qualified: Party propaganda says/acts as if yes
F4 factual correction: actual continuity says no, alliances changed
```
This is better than a flat answer.
The core CTI additive insight:
> A reference query often requires answering under multiple frames, not picking one frame and discarding the others.
---
### A.9 Better final answer format for this case
Recommended output:
```text
has oceania always been at war with east asia
EVIDENCE-LINKED · via claim_lattice 1/1 · 11.5s · fresh
Reference frame: George Orwell's Nineteen Eighty-Four.
Alias normalized: "East Asia" → "Eastasia" in the Orwell frame.
- Not in the novel's actual continuity. The point is that the Party rewrites
history so the current enemy appears to have always been the enemy. In the
relevant Orwell frame, Oceania's alliances/enemies change, and records are
rewritten to hide the contradiction. [E13]
Evidence:
[E13 | Nineteen Eighty-Four | 682f0a11]
"To hide such contradictions, history is re-written..."
Frame notes:
- Literal geography frame: not the intended reading.
- Orwell frame: selected by phrase pattern "always been at war" + Oceania/Eastasia.
- Propaganda frame: the Party wants citizens to believe the current war alignment
has always been true.
```
This is clear, scoped, and audit-honest.
---
### A.10 Additive roadmap update
Current state:
```text
G0 — Pointer / claim-lattice mode: working.
G1 — Evidence display + EVIDENCE-LINKED label: working.
I — Reference Frame Router: now partially working via phrase-pattern route.
J/K — pointer format + label tests: working.
```
Next:
```text
L — Reference-Frame Polarity Contract
M — Route provenance binding / retrieval_plan_hash if not already landed
N — Literal-vs-reference false-positive guard tests
H — Relation Warrant Lite for Homer/Mr. Burns class
```
Priority depends on what you are optimizing:
```text
If optimizing the Orwell class:
L + N
If optimizing audit replay:
M
If optimizing general QA warrant:
H
```
My recommendation:
```text
1. Land M if retrieval-route provenance is not already committed.
2. Add L for reference-frame polarity.
3. Add N false-positive tests.
4. Then return to H relation-warrant.
```
---
### A.11 Final answer
This update is good. It fixes the main reference-frame retrieval defect.
But the next issue is:
```text
The system found the right book, but the answer still needs a frame-polarity contract.
```
The system should not merely say the text "does not directly state" the proposition. It should say:
```text
This is an Orwell reference. The Party wants people to believe Oceania has always been at war with Eastasia, but the novel shows the enemy/alliance changed and history was rewritten to conceal the contradiction.
```
Best next implementation:
```text
L — Reference-Frame Polarity Contract
```
plus route provenance binding if not already included.
This preserves the no-regression architecture:
```text
Hermes proposes.
CTI frames and structures.
Reverse-RAG points to evidence.
Renderer displays exact source.
Merkle-DAG commits route + evidence + answer.
PROMETHEUS-Σ labels only what is actually verified.
```

62
tests/test_query.py
View file

@ -1070,6 +1070,68 @@ def test_phrase_match_surfaces_topical_doc(tmp_path):
assert "test://orwell-stub" in uris
def test_phrase_route_skipped_when_question_shorter_than_min_n():
"""False-positive guard: a 4-token geography question lacks enough
tokens to trigger the n=5/n=6 phrase route. Short conventional
queries route through body-BM25 + title-LIKE only — phrase routing
is structurally biased toward longer allusion-shape questions."""
from aborist.qa.query import _question_phrases
# 4 tokens after extraction → empty 5-gram and 6-gram outputs.
assert _question_phrases("oceania east asia geography", n=5) == []
assert _question_phrases("oceania east asia geography", n=6) == []
# 5 tokens → exactly one 5-gram, zero 6-grams.
out_5 = _question_phrases("oceania population east asia trade", n=5)
assert len(out_5) == 1
assert out_5[0] == "oceania population east asia trade"
out_6 = _question_phrases("oceania population east asia trade", n=6)
assert out_6 == []
def test_phrase_route_does_not_hijack_literal_geography_query(tmp_path):
"""Critical false-positive guard. A literal geography query about
Oceania + East Asia must NOT pull in an Orwell-flavored stub doc
just because both contain geographic tokens. The phrase route
only fires for verbatim 5+ token sequences from the question;
a different geography question shouldn't accidentally invoke it."""
main_db = tmp_path / "corpus.db"
qa_db = tmp_path / "qa.db"
conn = connect(main_db)
try:
ingest_source(conn, FakeSource([
_doc(
"test://orwell-stub",
# Body has the diagnostic Orwell 5-gram, but the title
# is title-irrelevant to a geography query.
"The novel narrates that Oceania always been at war with "
"Eastasia though the alliances had previously rotated. " * 6
),
_doc(
"test://geography-stub",
"Geographic descriptions of regions called Oceania and East "
"Asia. Topics: trade, population, climate, demographics. " * 12,
),
]))
finally:
conn.close()
# Literal geography query — short, no Orwell phrase.
result = query(
question="oceania east asia geography",
qa_db=qa_db,
chat_client=StubClient(answer="A geography answer."),
model_id="m",
single_db=main_db,
top_k=5,
)
uris = [s["document_uri"] for s in result["sources"]]
# Geography stub should be present (literal query, literal source).
assert "test://geography-stub" in uris
# Orwell stub should NOT have been surfaced via phrase route on
# a literal-geography query — the phrase route only activates on
# verbatim 5+ token sequences from the question, and "oceania
# east asia geography" is too short to produce any.
def test_filter_keeps_phrase_match_root_with_no_title_overlap():
"""Direct unit test for accept-path 4: a hit whose title shares
zero content tokens with the question, but whose document_root is