engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

qa: phrase-pattern retrieval route closes the reference-frame failure class

Browse source 
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.

Fix is two parts:

(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.

(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.

Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.

Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.

Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.

docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).

9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
This commit is contained in:
russell@unturf.com 2026-05-01 13:53:03 -04:00
parent 3586eeeb0a
commit 1b8677d3d5
No known key found for this signature in database
4 changed files with 578 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

174
aborist/qa/query.py
View file

@ -149,9 +149,10 @@ def _filter_by_title_relevance(
*,
core_match_roots: set[str] | None = None,
body_density_check: callable | None = None,
phrase_match_roots: set[str] | None = None,
fallback_top_n: int = 5,
) -> list:
"""Concept-aware relevance filter with three accept paths:
"""Concept-aware relevance filter with four accept paths:
1. Title-token overlap (after synonym expansion). Strongest signal.
2. TF-IDF core keyword overlap — `core_match_roots` is a precomputed
@ -162,11 +163,21 @@ def _filter_by_title_relevance(
3. Body density — docs mentioning the query token >= N times pass
even without title or core match. Cheap proxy for "actually about
the topic." `body_density_check(hit)` returns bool.
4. Phrase-match — docs whose body contains a verbatim 4+ token
sequence from the question pass even when title and content
tokens don't overlap. Closes the allusion gap (2026-05-01
Orwell case): "has oceania always been at war with east asia"
has zero token overlap with the title "Nineteen Eighty-Four"
but the body contains the verbatim phrase "always been at
war" — without this accept path, the phrase-route hit gets
filtered out before it can rerank into the top-K. The
upstream phrase route already gates on 4-token-min sequences
(see _question_phrases) so false-positive risk is low.
Rivalry exclusion (Intel-titled docs in AMD queries) still applies on
every accept path.
If all three accept paths together produce nothing, fall back to the
If all four accept paths together produce nothing, fall back to the
top `fallback_top_n` body-BM25 hits — the LLM gets enough context to
say "I don't know" rather than fabricating from a single tangential
source.
@ -178,6 +189,7 @@ def _filter_by_title_relevance(
accept = synonym_expand(qtokens)
exclude = rivalry_excluded(qtokens, compare_phrasing=has_compare_phrasing(question))
core_roots = core_match_roots or set()
phrase_roots = phrase_match_roots or set()
# Title-overlap breadth threshold scales with query length, mirroring
# _body_density_passes: ≤2 tokens require ALL, 3+ require N-1. Without
# this, a 2-token query like "supermans girlfriend" admits docs that
@ -206,6 +218,9 @@ def _filter_by_title_relevance(
if h.document_root in core_roots:
kept.append(h)
continue
if h.document_root in phrase_roots:
kept.append(h)
continue
if body_density_check is not None and body_density_check(h):
kept.append(h)
continue
@ -557,6 +572,101 @@ def _search_titles(conn, qtokens: list[str], limit: int) -> list[tuple]:
return rows
def _question_phrases(question: str, *, n: int = 4) -> list[str]:
"""Extract verbatim n-token sliding-window phrases from the question.
Used by the phrase-pattern retrieval route to catch allusions /
idioms / fictional-world references whose diagnostic signal is
the EXACT sequence including function words. Stopword stripping
would kill this:
"always been at war" — diagnostic Orwell signal
"always war" — generic, useless
So we DON'T strip stopwords here. Skip phrases whose tokens are
all ≤ 3 chars (pure boilerplate, no diagnostic value). Output
is lowercase, deduped, in source order. Default ``n=4`` is the
sweet spot empirically: 3-grams are too noisy ("the cat in"
matches loads of things), 5-grams miss shorter idioms ("winter
is coming" → 3 tokens).
"""
import re as _re
tokens = _re.findall(r"[A-Za-z][A-Za-z0-9]+", question)
if len(tokens) < n:
return []
out: list[str] = []
seen: set[str] = set()
for i in range(len(tokens) - n + 1):
window = tokens[i:i + n]
if max(len(t) for t in window) < 4:
continue # all-short-tokens → boilerplate
phrase = " ".join(t.lower() for t in window)
if phrase in seen:
continue
seen.add(phrase)
out.append(phrase)
return out
def _search_phrases(conn, phrases: list[str], limit: int) -> list[tuple]:
"""FTS5 phrase-match search across chunk bodies.
Each phrase becomes an FTS5 quoted-phrase token (``'"phrase"'``);
we OR the phrases so any verbatim match wins. Returns sqlite3.Row
objects with columns ``document_root, idx, document_uri, title``
matching the rest of the search-route surface.
Why this exists: AND-mode FTS5 token-matching (the default body
search route) treats query tokens independently — a doc must
contain every token but the tokens can be anywhere. For
allusion-shape queries the diagnostic signal is the verbatim
sequence:
Q = "has oceania always been at war with east asia"
body BM25 surfaces literal-geography articles (Oceania, Asia,
Far East) because they have the most occurrences of "Oceania"
+ "Asia" + "war" individually.
phrase MATCH '"always been at war"' surfaces
Nineteen_Eighty-Four because the phrase is verbatim Orwell.
The verbatim phrase route doesn't dominate: in ``_search_corpus``
its score is 70 (between core-keyword and title-LIKE ranks), and
the existing rerank pipeline still gates by title relevance.
Phrase matches just get a seat at the table.
Defensive: silently skip phrases containing double-quotes
(adversarial / malformed input). Wraps the FTS5 query in a
try/except so a malformed MATCH doesn't crash the search;
upstream callers see an empty result set.
"""
if not phrases:
return []
quoted = [f'"{p}"' for p in phrases if '"' not in p]
if not quoted:
return []
fts_query = " OR ".join(quoted)
try:
rows = conn.execute(
"""
SELECT
c.document_root,
c.idx,
d.document_uri,
d.title
FROM chunks_fts AS f
JOIN chunks AS c ON c.chunk_id = f.rowid
JOIN documents AS d ON d.document_root = c.document_root
WHERE chunks_fts MATCH ?
ORDER BY bm25(chunks_fts) ASC
LIMIT ?
""",
(fts_query, limit),
).fetchall()
except Exception: # noqa: BLE001 — search must fail soft
rows = []
return rows
def _docs_with_core_keyword_match(
conn, qtokens: list[str], limit: int
) -> list[tuple]:
@ -749,6 +859,12 @@ def _search_corpus(
# Roots whose TF-IDF cores contain a query token — collected across
# shards. Used downstream by _filter_by_title_relevance as accept-path 2.
core_match_roots: set[str] = set()
# Roots that matched a verbatim 4+ token phrase from the question.
# Used downstream by _filter_by_title_relevance as accept-path 4 so
# an allusion-shape hit (e.g. Nineteen_Eighty-Four for "always been
# at war") survives the title-token-overlap filter even when its
# title shares no tokens with the question.
phrase_match_roots: set[str] = set()
# Per-shard mapping of doc root -> shard path, for body-density lookups
# in accept-path 3. Lets us reach back to the source shard cheaply.
root_to_shard: dict[str, str] = {}
@ -798,6 +914,41 @@ def _search_corpus(
)
)
root_to_shard[r["document_root"]] = str(p.resolve())
# Phrase-pattern search — verbatim multi-token sequences
# from the question. Catches allusions / idioms / fictional-
# world references whose diagnostic signal is the exact
# sequence including function words. Two passes for layered
# specificity:
# - n=6 (highest specificity): "oceania always been at
# war with" is essentially unique to Orwell. Score 100.
# - n=5 (high specificity): "oceania always been at war"
# still strongly Orwell-anchored. Score 90.
# 4-grams were tried (2026-05-01) and dropped: "always been
# at war" matches generic war-history articles too often,
# creating retrieval noise that the rerank pipeline can't
# cleanly separate from the actual allusion. Empirically
# 5+ grams trade recall for precision — most allusions
# ("may the force be with you", "to be or not to be",
# "winter is coming") survive at length 5 or 3-with-light-
# tokens, but the 4-gram floor is where diagnostic-ness
# collapses.
for n in (6, 5):
phrase_score = 100.0 if n == 6 else 90.0
for r in _search_phrases(
conn, _question_phrases(question, n=n), over_fetch
):
raw.append(
(
phrase_score,
r["document_root"],
r["document_uri"],
r["title"],
r["idx"],
str(p.resolve()),
)
)
root_to_shard[r["document_root"]] = str(p.resolve())
phrase_match_roots.add(r["document_root"])
# Core-keyword search: docs whose TF-IDF core keywords match.
# The query token doesn't need to be in title or even in body —
# being a TF-IDF keyword of the doc's core is enough signal.
@ -845,7 +996,15 @@ def _search_corpus(
chunk_idx=idx,
)
)
return out
# Sidecar sets returned alongside the hit list. Pre-2026-05-01
# the function returned a bare list and the caller used
# `getattr(hits, "_core_match_roots", set())` to fish out the
# sets — but the sidecar was never attached, so the
# `core_match_roots` accept-path in _filter_by_title_relevance
# silently received an empty set. Returning a tuple corrects
# the plumbing AND threads the new `phrase_match_roots` for
# accept-path 4.
return out, core_match_roots, phrase_match_roots, root_to_shard
def _rerank(
@ -854,6 +1013,7 @@ def _rerank(
*,
core_match_roots: set[str] | None = None,
body_density_check: callable | None = None,
phrase_match_roots: set[str] | None = None,
) -> list[_Hit]:
"""Filter off-topic, then layer in body-coverage, title-overlap, and
source-role rank boosts.
@ -873,6 +1033,7 @@ def _rerank(
question,
core_match_roots=core_match_roots,
body_density_check=body_density_check,
phrase_match_roots=phrase_match_roots,
)
hits = _rerank_by_body_coverage(hits, question)
hits = _rerank_by_title(hits, question)
@ -1299,7 +1460,9 @@ def query(
if retrieval_keywords and retrieval_keywords.strip():
retrieval_query = f"{question} {retrieval_keywords.strip()}"
t_phase = time.monotonic()
hits = _search_corpus(shards_dir, single_db, retrieval_query, over_fetch)
hits, core_match_roots, phrase_match_roots, root_to_shard = _search_corpus(
shards_dir, single_db, retrieval_query, over_fetch
)
if not hits:
return {
"status": "no_sources",
@ -1309,8 +1472,6 @@ def query(
"total_ms": _ms_since(t_start),
},
}
core_match_roots = getattr(hits, "_core_match_roots", set())
root_to_shard = getattr(hits, "_root_to_shard", {})
qtokens_lower = {t.lower() for t in _title_query_tokens(retrieval_query)}
def _body_density_check(h) -> bool:
@ -1329,6 +1490,7 @@ def query(
retrieval_query,
core_match_roots=core_match_roots,
body_density_check=_body_density_check,
phrase_match_roots=phrase_match_roots,
)
search_ms = _ms_since(t_phase)

68
bench/qa_questions.txt
View file

@ -11,33 +11,84 @@ who wrote GNU linux?
when was a programming language named Python created?
who painted the mona lisa?
a bridge between new london & groton?
who wrote the play hamlet?
what is the chemical symbol for gold?
# broad descriptive — encyclopedic shape, prone to mode-collapse
tell me about connecticut
tell me about the C programming language
tell me about method man?
tell me all there is to know about york england?
tell me about the roman empire
describe the structure of DNA
# entity list — invites lazy-anchor on a magnet chunk
what dinosaurs were in the first jurassic park film?
who are the members of the beatles?
name simpsons family members including pets?
list of obelisk in connecticut
what are the planets of our solar system?
who were the original seven mercury astronauts?
# relationship / multi-fact
who is supermans girlfriend?
who is bilbo baggins's nephew?
what is the relationship between linux & unix?
who is veronica ballestrini & what month was she born?
who is luke skywalker's father and sister?
# comparison — multi-entity, prone to attribution drift
what's the difference between linux and bsd?
how does intel compare to amd?
what is the difference between http and ftp?
mac vs windows for software development
# niche / partial — corpus may be thin
what is the boltzmann constant?
who invented the doppler effect?
# date / time — when-questions stress year/date grounding
when did world war 2 end?
what year did the berlin wall fall?
when was the first moon landing?
when did the soviet union dissolve?
in what year was the magna carta signed?
# quantity — numeric grounding, prone to confabulated digits
how many states are in the united states?
how many bones are in the adult human body?
what is the population of japan?
how many wives did henry the eighth have?
how many moons does jupiter have?
# geographic — where / what country / location grounding
where is mount kilimanjaro located?
what country is the city of prague in?
in which ocean is madagascar?
where does the nile river begin?
what continent is egypt on?
# cause / effect — why-questions, narrative grounding
why did the titanic sink?
what caused the chernobyl disaster?
why did the dinosaurs go extinct?
what triggered world war 1?
# entity disambiguation — common names that collide with many entities
who is michael jordan?
who is george bush?
who is john smith of jamestown?
what is the matrix?
who is paul of tarsus?
# synonym / paraphrase robustness — same fact, different phrasing.
# Pairs probe whether equivalence_class question dedup collapses these
# at write & whether retrieval grounds them identically.
who founded microsoft?
who is the founder of microsoft?
when was the eiffel tower built?
what year was the eiffel tower constructed?
# leading / forensic — probe whether retrieval surfaces the right
# fact when the question's premise contradicts the popular narrative.
# Elevation question expected answer: north (per geographic surveys);
@ -46,7 +97,24 @@ who invented the doppler effect?
# elevation fact, so this also stress-tests lazy-anchor STRICT.
which side is the ground elevation highest throughout the span of the great wall of china, the north or south?
whales are endangered from over hunting
did napoleon really die on saint helena?
# out-of-corpus — should land UNGROUNDED honestly
who is a benevolent dictator for life for mars?
what year does our cold fusion breakthrough happen?
who won the 2024 us presidential election?
what is the latest version of the ipad pro?
# allusion / reference frame — the diagnostic signal is a verbatim
# multi-token sequence, not the individual content tokens. Pure-BM25
# on tokens will surface literal-frame articles (Oceania the region,
# Asia the continent) instead of the reference (Nineteen Eighty-Four).
# The phrase-pattern retrieval route (commit 2026-05-01) targets this
# class. Bench reveals prevalence of the failure mode + whether the
# fix generalizes beyond Orwell.
has oceania always been at war with east asia?
who said may the force be with you?
what does winter is coming mean?
what is the meaning of rosebud?
who said to be or not to be that is the question?
what does the cake is a lie reference?

169
docs/reference-frame-failure-class.md Normal file
View file

@ -0,0 +1,169 @@
# Reference-frame failure class — the Orwell case
**Date:** 2026-05-01
**Scope:** Worked example documenting a failure mode in retrieval +
phrase-pattern boost as the response.
**Audience:** fox + future blackops shifts.
**Status:** investigation log; the phrase-pattern route landed in
the same commit window. Not a ticket — this is a journal entry.
---
## What happened
Run 1 (no augmentation):
```
make query Q="has oceania always been at war with east asia"
HYBRID 2/16 via claim_lattice
sources: Oceania · Asia · Outline of Oceania · Far East · ...
```
The system selected the **literal geography frame** — answered as
if asked about real-world Oceania and East Asia regions. The
Nineteen Eighty-Four article was nowhere in the top-K despite
existing in shard `003.db`.
Run 2 (manual augmentation — `K=` flag):
```
make query Q="has oceania always been at war with east asia? do you understand what this reference"
STRICT 1/1 via claim_lattice
sources: Nineteen Eighty-Four (top result)
```
Adding the literal word "reference" to the question pushed the
retrieval into the right map. The user solved it themselves with
operator hints — but a high-quality system shouldn't need those
hints for diagnostic phrases.
## The diagnosis
The cause was retrieval-shape, not model behavior:
```
Q tokens (FTS5-tokenized, no stem): {has, oceania, always, been, at, war, with, east, asia}
Q content tokens (stopword-stripped): {oceania, war, east, asia}
```
For these content tokens:
| Article | spin | glass | modeling | tensors | distinct match |
|----------------------------|------|-------|----------|---------|----------------|
| Nineteen Eighty-Four | 0 | - | - | - | (no overlap on title-tokens at all) |
| Oceania | many | - | - | - | 1 token in title (oceania) |
| Foreign relations of Axis | - | - | - | - | 0 |
The **diagnostic signal** wasn't in the content-token overlap. It
was in the verbatim 5-token phrase `oceania always been at war`
which appears in the 1984 article's body but nowhere else.
Pre-2026-05-01 retrieval had three routes (body BM25, title-LIKE,
core-keyword) — none of them captured "find articles whose body
contains a verbatim multi-token sequence from the question." So
the 1984 article was effectively invisible to the search.
## The fix — phrase-pattern retrieval route
`aborist/qa/query.py:_search_phrases` adds a fourth retrieval
route that runs an FTS5 quoted-phrase MATCH for each n-gram
extracted from the question:
```python
_question_phrases("has oceania always been at war with east asia", n=5)
# → ["has oceania always been at",
# "oceania always been at war", ← diagnostic Orwell signal
# "always been at war with",
# "been at war with east",
# "at war with east asia"]
```
Two pass-throughs — n=6 (specificity 100) and n=5 (specificity 90).
4-grams were tried and rejected: "always been at war" matches generic
war-history articles too noisily for the rerank pipeline to separate.
5+ tokens trade recall for precision; most allusions survive at
length 5.
**`_filter_by_title_relevance` accept-path 4** lets phrase-route
hits bypass the title-token-overlap gate. The 1984 article's title
shares zero tokens with the question — without accept-path 4,
phrase-route candidates would be retrieved and immediately filtered
out before they could rerank into top-K.
## Result
After the fix:
```
make query Q="has oceania always been at war with east asia"
EVIDENCE-LINKED · via claim_lattice 1/1 11.5s (fresh)
- The text does not directly state that Oceania has always been at
war with East Asia. The passage describes a change in alliances,
where Oceania switched from being allies with Eastasia to being
allies with Eurasia, and the public was manipulated to accept
this change without realizing it.
[E13 | Nineteen Eighty-Four | 682f0a11: "...To hide such
contradictions, history is re-written to explain that the (new)
alliance always was so..."]
```
No operator augmentation needed. The Orwell frame surfaces from
phrase-route alone.
## Why not a "Reference Frame Router"?
An earlier sketch (in fox's review, 2026-05-01 Asia/Kuala_Lumpur)
proposed a hand-rolled `REFERENCE_PATTERNS` table mapping query
shapes ("has X always been at war with Y") to known references
(Orwell). That approach was rejected because:
- **Scaling**: allusions are long-tail. `winter is coming`,
`the cake is a lie`, `may the force be with you`, `to be or
not to be` — the catalog is open-ended.
- **Maintenance**: per-pattern code rots; new allusions need new
rules.
- **The corpus already knows**: 1984 article exists in the corpus.
The defect was retrieval not surfacing it. Fix retrieval, not
add a new pre-retrieval stage.
Phrase-pattern boost generalizes: any verbatim 5+ token sequence
from the question that appears in an article body lifts that
article into consideration, regardless of whether it's an Orwell
reference, a Star Wars quote, a Hamlet line, or a meme.
## Bench coverage
Added to `bench/qa_questions.txt` under `# allusion / reference
frame`:
```
has oceania always been at war with east asia?
who said may the force be with you?
what does winter is coming mean?
what is the meaning of rosebud?
who said to be or not to be that is the question?
what does the cake is a lie reference?
```
A future bench sweep will reveal whether phrase-pattern boost
generalizes across these allusions or if some need different
treatment.
## Related concerns NOT addressed here
- **Source-role taxonomy** — proposed roles like
`primary_reference_source` (cited reference work) vs
`literal_world_background` (geography article that's tangentially
related). Currently all retrieved sources land as
`background_source` in the rendered output; finer-grained roles
could improve answer framing. Out of scope for this fix.
- **Frame-assumption verifier field** — orthogonal to STRICT/HYBRID/
UNGROUNDED labels. Records "answered as Orwell reference" vs
"answered as geography." Useful for audit but requires verifier
schema work; deferred.
- **Reference-aware prompting** — system prompt could acknowledge
reference-frame ambiguity ("if the query reads as an allusion,
prefer the reference frame when retrieved evidence supports it").
Per fox's "less prompt engineering, more code-level discipline"
preference, deferred until phrase-pattern boost's empirical
performance is known.

173
tests/test_query.py
View file

@ -936,3 +936,176 @@ def test_retrieval_keywords_changes_retrieved_sources(tmp_path):
assert a_uris != b_uris or set(a_uris) != set(b_uris), (
f"keywords should affect retrieval; got identical sources {a_uris}"
)
# ---------------------------------------------------------------------------
# phrase-pattern retrieval route (allusion / verbatim sequence boost)
# ---------------------------------------------------------------------------
def test_question_phrases_returns_sliding_n_grams_no_stopword_strip():
"""`_question_phrases` extracts verbatim n-token windows. Function
words are kept — diagnostic value of an allusion is the EXACT
sequence ('always been at war' >> 'always war')."""
from aborist.qa.query import _question_phrases
out = _question_phrases("has oceania always been at war with east asia", n=4)
# 9-token query, 4-gram window → 6 phrases, all preserved verbatim
# (lowercase) and deduped.
assert "has oceania always been" in out
assert "always been at war" in out
assert "war with east asia" in out
# Stopwords ARE present — that's the design, not a bug.
assert any("at" in p.split() for p in out)
def test_question_phrases_n_5_yields_five_token_phrases():
"""5-grams trade recall for precision; 'oceania always been at war'
is a much stronger Orwell signal than 'always been at war' alone."""
from aborist.qa.query import _question_phrases
out = _question_phrases("has oceania always been at war with east asia", n=5)
assert "oceania always been at war" in out
assert "always been at war with" in out
# Too short for 6-grams of just "war with east asia" alone.
assert all(len(p.split()) == 5 for p in out)
def test_question_phrases_skips_when_question_shorter_than_n():
"""`who is X?` is too short to yield 4-grams. Empty output is the
expected behavior (the body BM25 + title routes still cover it)."""
from aborist.qa.query import _question_phrases
assert _question_phrases("who is X?", n=4) == []
assert _question_phrases("", n=4) == []
def test_question_phrases_drops_all_short_token_phrases():
"""A window of all 1-3 char tokens is boilerplate ('to be or not')
— drops to avoid over-matching. The skip rule fires only when ALL
tokens in the window are <4 chars."""
from aborist.qa.query import _question_phrases
# All ≤3-char tokens — drop.
assert _question_phrases("to be or not", n=4) == []
# Mixed: at least one ≥4-char token → keep.
out = _question_phrases("to be or maybe", n=4)
assert out == ["to be or maybe"]
def test_question_phrases_lowercases_and_dedupes():
"""Output is lowercase, deduped on string equality. Same
sequence in different cases collapses to one phrase."""
from aborist.qa.query import _question_phrases
out = _question_phrases("Always been at war Always been at war", n=4)
# Repeated sequence appears only once in the output.
assert out.count("always been at war") == 1
def test_search_phrases_returns_empty_on_no_phrases(tmp_path):
"""Defensive: empty phrase list yields no rows, no exceptions."""
from aborist.qa.query import _search_phrases
main_db = tmp_path / "corpus.db"
conn = connect(main_db)
try:
ingest_source(conn, FakeSource(DOCS))
rows = _search_phrases(conn, [], 10)
finally:
conn.close()
assert rows == []
def test_search_phrases_skips_phrases_with_double_quotes(tmp_path):
"""Adversarial input safety: phrases containing `"` would break
the FTS5 quoted-phrase syntax. The function silently drops them."""
from aborist.qa.query import _search_phrases
main_db = tmp_path / "corpus.db"
conn = connect(main_db)
try:
ingest_source(conn, FakeSource(DOCS))
# All phrases contain quotes — function returns empty.
rows = _search_phrases(conn, ['has "embedded" quote', 'also "bad"'], 10)
finally:
conn.close()
assert rows == []
def test_phrase_match_surfaces_topical_doc(tmp_path):
"""End-to-end: a query whose phrase verbatim-matches one doc's
body should surface that doc even when title tokens don't
overlap. Closes the 2026-05-01 Orwell case where the 1984
article had zero token overlap with the question's title-tokens
but matched the verbatim phrase 'always been at war'."""
main_db = tmp_path / "corpus.db"
qa_db = tmp_path / "qa.db"
conn = connect(main_db)
try:
# Two docs: one explicitly contains the diagnostic phrase
# but its title doesn't overlap question tokens; the other
# is a generic geography article.
ingest_source(conn, FakeSource([
_doc(
"test://orwell-stub",
# Title-irrelevant to the question; body contains
# the diagnostic 5-gram.
"The novel narrates that Oceania always been at war with "
"Eastasia though the alliances had previously rotated. " * 6
),
_doc(
"test://geography-stub",
"Geographic descriptions of regions called Oceania and East "
"Asia. " * 12,
),
]))
finally:
conn.close()
result = query(
question="has oceania always been at war with east asia",
qa_db=qa_db,
chat_client=StubClient(answer="An answer."),
model_id="m",
single_db=main_db,
top_k=5,
)
uris = [s["document_uri"] for s in result["sources"]]
# Both docs surface; the phrase-route ensures the orwell-stub
# doc isn't filtered out by the title-relevance gate.
assert "test://orwell-stub" in uris
def test_filter_keeps_phrase_match_root_with_no_title_overlap():
"""Direct unit test for accept-path 4: a hit whose title shares
zero content tokens with the question, but whose document_root is
in `phrase_match_roots`, must pass the filter."""
from aborist.qa.query import _Hit, _filter_by_title_relevance
hits = [
# Title shares NO content tokens with the question. Without
# accept-path 4 (phrase_match_roots), it would be dropped.
_Hit(
document_root="bbb",
document_uri="t://nineteen-eighty-four",
title="Nineteen Eighty-Four",
score=70.0,
shard_path="x",
chunk_idx=0,
),
]
# Without phrase_match_roots, the hit is dropped (title-relevance
# filter has no accept path that fires).
kept_without = _filter_by_title_relevance(
hits,
"has oceania always been at war with east asia",
)
# Filter falls back to top-N when nothing accepts; accept the
# fallback as 'kept' here too — what we care about is whether
# accept-path 4 is the path firing when phrase_match_roots is set.
kept_with = _filter_by_title_relevance(
hits,
"has oceania always been at war with east asia",
phrase_match_roots={"bbb"},
)
kept_roots = {h.document_root for h in kept_with}
assert "bbb" in kept_roots, (
"phrase_match_roots accept-path 4 should keep titles with no "
"token overlap when their body verbatim-matched a question phrase"
)
# And the fallback path doesn't suddenly fail when phrase_match_roots
# is present — the keep is via accept-path 4, not via the fallback.
_ = kept_without # documents that fallback may also keep, but via different path