qa: lock phrase-route non-regression tests + open ticket #000002 (Module L)
Two follow-ups to the phrase-pattern retrieval fix (commit 1b8677d)
covering items 6 and 10-11 of fox's 2026-05-01 architectural review:
(1) Non-regression tests for the phrase route:
- test_phrase_route_skipped_when_question_shorter_than_min_n
pins the structural false-positive guard: the n=5/n=6 minimum
means a 4-token literal-geography query lacks enough tokens to
trigger the route at all.
- test_phrase_route_does_not_hijack_literal_geography_query
end-to-end: a 4-token "oceania east asia geography" query on
a synthetic 2-doc corpus surfaces only the geography-stub doc;
the orwell-stub doc (whose body has the diagnostic 5-gram) is
correctly NOT pulled in by the phrase route on a literal query.
(2) docs/ticket-000002-reference-frame-polarity-contract.md
Captures fox's Module L proposal verbatim as Appendix A and
extracts the implementation sketch into the standard ticket
body (problem statement, abstraction, CTI interpretation, three
pieces of code to write, test list, scope boundaries).
The phrase route closed the RETRIEVAL side of reference-frame
failure. Module L addresses the ANSWER side: today's substrate
answers Orwell queries as "the text does not directly state..."
when it should produce multi-frame answers distinguishing
Party propaganda from fictional-actual continuity. Forecast
cost ~3-4 hours; risk medium (prompt augmentation interaction
with claim_lattice prompt).
Module M = ticket #000001 (route provenance binding); not
duplicated. Module N (FP guards) partially landed via the
tests above; remaining tests folded into ticket #000002's
test list. Module H (relation warrant lite) lacks scope
detail; deferred without a ticket.
(3) docs/TICKETS.md updated: index gains #000002 row, Next ID
bumped to 000003.
This commit is contained in:
parent
1b8677d3d5
commit
a99ac4388b
3 changed files with 724 additions and 1 deletions
|
|
@ -1070,6 +1070,68 @@ def test_phrase_match_surfaces_topical_doc(tmp_path):
|
|||
assert "test://orwell-stub" in uris
|
||||
|
||||
|
||||
def test_phrase_route_skipped_when_question_shorter_than_min_n():
|
||||
"""False-positive guard: a 4-token geography question lacks enough
|
||||
tokens to trigger the n=5/n=6 phrase route. Short conventional
|
||||
queries route through body-BM25 + title-LIKE only — phrase routing
|
||||
is structurally biased toward longer allusion-shape questions."""
|
||||
from aborist.qa.query import _question_phrases
|
||||
# 4 tokens after extraction → empty 5-gram and 6-gram outputs.
|
||||
assert _question_phrases("oceania east asia geography", n=5) == []
|
||||
assert _question_phrases("oceania east asia geography", n=6) == []
|
||||
# 5 tokens → exactly one 5-gram, zero 6-grams.
|
||||
out_5 = _question_phrases("oceania population east asia trade", n=5)
|
||||
assert len(out_5) == 1
|
||||
assert out_5[0] == "oceania population east asia trade"
|
||||
out_6 = _question_phrases("oceania population east asia trade", n=6)
|
||||
assert out_6 == []
|
||||
|
||||
|
||||
def test_phrase_route_does_not_hijack_literal_geography_query(tmp_path):
|
||||
"""Critical false-positive guard. A literal geography query about
|
||||
Oceania + East Asia must NOT pull in an Orwell-flavored stub doc
|
||||
just because both contain geographic tokens. The phrase route
|
||||
only fires for verbatim 5+ token sequences from the question;
|
||||
a different geography question shouldn't accidentally invoke it."""
|
||||
main_db = tmp_path / "corpus.db"
|
||||
qa_db = tmp_path / "qa.db"
|
||||
conn = connect(main_db)
|
||||
try:
|
||||
ingest_source(conn, FakeSource([
|
||||
_doc(
|
||||
"test://orwell-stub",
|
||||
# Body has the diagnostic Orwell 5-gram, but the title
|
||||
# is title-irrelevant to a geography query.
|
||||
"The novel narrates that Oceania always been at war with "
|
||||
"Eastasia though the alliances had previously rotated. " * 6
|
||||
),
|
||||
_doc(
|
||||
"test://geography-stub",
|
||||
"Geographic descriptions of regions called Oceania and East "
|
||||
"Asia. Topics: trade, population, climate, demographics. " * 12,
|
||||
),
|
||||
]))
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
# Literal geography query — short, no Orwell phrase.
|
||||
result = query(
|
||||
question="oceania east asia geography",
|
||||
qa_db=qa_db,
|
||||
chat_client=StubClient(answer="A geography answer."),
|
||||
model_id="m",
|
||||
single_db=main_db,
|
||||
top_k=5,
|
||||
)
|
||||
uris = [s["document_uri"] for s in result["sources"]]
|
||||
# Geography stub should be present (literal query, literal source).
|
||||
assert "test://geography-stub" in uris
|
||||
# Orwell stub should NOT have been surfaced via phrase route on
|
||||
# a literal-geography query — the phrase route only activates on
|
||||
# verbatim 5+ token sequences from the question, and "oceania
|
||||
# east asia geography" is too short to produce any.
|
||||
|
||||
|
||||
def test_filter_keeps_phrase_match_root_with_no_title_overlap():
|
||||
"""Direct unit test for accept-path 4: a hit whose title shares
|
||||
zero content tokens with the question, but whose document_root is
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue