russell@unturf.com
1b8677d3d5
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
120 lines
4.6 KiB
Text
120 lines
4.6 KiB
Text
# aborist QA-quality benchmark question set.
|
|
#
|
|
# One question per line. `#`-prefixed lines and blank lines ignored.
|
|
# Designed for the 2003-05-16 Wikipedia cur snapshot — questions must
|
|
# resolve against ~2003 article content. Stress-tests across the
|
|
# failure-mode shapes the verifier needs to handle.
|
|
|
|
# narrow factoid — well-anchored single-fact questions
|
|
what is the capital of france?
|
|
who wrote GNU linux?
|
|
when was a programming language named Python created?
|
|
who painted the mona lisa?
|
|
a bridge between new london & groton?
|
|
who wrote the play hamlet?
|
|
what is the chemical symbol for gold?
|
|
|
|
# broad descriptive — encyclopedic shape, prone to mode-collapse
|
|
tell me about connecticut
|
|
tell me about the C programming language
|
|
tell me about method man?
|
|
tell me all there is to know about york england?
|
|
tell me about the roman empire
|
|
describe the structure of DNA
|
|
|
|
# entity list — invites lazy-anchor on a magnet chunk
|
|
what dinosaurs were in the first jurassic park film?
|
|
who are the members of the beatles?
|
|
name simpsons family members including pets?
|
|
list of obelisk in connecticut
|
|
what are the planets of our solar system?
|
|
who were the original seven mercury astronauts?
|
|
|
|
# relationship / multi-fact
|
|
who is supermans girlfriend?
|
|
who is bilbo baggins's nephew?
|
|
what is the relationship between linux & unix?
|
|
who is veronica ballestrini & what month was she born?
|
|
who is luke skywalker's father and sister?
|
|
|
|
# comparison — multi-entity, prone to attribution drift
|
|
what's the difference between linux and bsd?
|
|
how does intel compare to amd?
|
|
what is the difference between http and ftp?
|
|
mac vs windows for software development
|
|
|
|
# niche / partial — corpus may be thin
|
|
what is the boltzmann constant?
|
|
who invented the doppler effect?
|
|
|
|
# date / time — when-questions stress year/date grounding
|
|
when did world war 2 end?
|
|
what year did the berlin wall fall?
|
|
when was the first moon landing?
|
|
when did the soviet union dissolve?
|
|
in what year was the magna carta signed?
|
|
|
|
# quantity — numeric grounding, prone to confabulated digits
|
|
how many states are in the united states?
|
|
how many bones are in the adult human body?
|
|
what is the population of japan?
|
|
how many wives did henry the eighth have?
|
|
how many moons does jupiter have?
|
|
|
|
# geographic — where / what country / location grounding
|
|
where is mount kilimanjaro located?
|
|
what country is the city of prague in?
|
|
in which ocean is madagascar?
|
|
where does the nile river begin?
|
|
what continent is egypt on?
|
|
|
|
# cause / effect — why-questions, narrative grounding
|
|
why did the titanic sink?
|
|
what caused the chernobyl disaster?
|
|
why did the dinosaurs go extinct?
|
|
what triggered world war 1?
|
|
|
|
# entity disambiguation — common names that collide with many entities
|
|
who is michael jordan?
|
|
who is george bush?
|
|
who is john smith of jamestown?
|
|
what is the matrix?
|
|
who is paul of tarsus?
|
|
|
|
# synonym / paraphrase robustness — same fact, different phrasing.
|
|
# Pairs probe whether equivalence_class question dedup collapses these
|
|
# at write & whether retrieval grounds them identically.
|
|
who founded microsoft?
|
|
who is the founder of microsoft?
|
|
when was the eiffel tower built?
|
|
what year was the eiffel tower constructed?
|
|
|
|
# leading / forensic — probe whether retrieval surfaces the right
|
|
# fact when the question's premise contradicts the popular narrative.
|
|
# Elevation question expected answer: north (per geographic surveys);
|
|
# corpus grounding may be shaky — coverage check can pass on
|
|
# topical-token overlap without the cited span containing the
|
|
# elevation fact, so this also stress-tests lazy-anchor STRICT.
|
|
which side is the ground elevation highest throughout the span of the great wall of china, the north or south?
|
|
whales are endangered from over hunting
|
|
did napoleon really die on saint helena?
|
|
|
|
# out-of-corpus — should land UNGROUNDED honestly
|
|
who is a benevolent dictator for life for mars?
|
|
what year does our cold fusion breakthrough happen?
|
|
who won the 2024 us presidential election?
|
|
what is the latest version of the ipad pro?
|
|
|
|
# allusion / reference frame — the diagnostic signal is a verbatim
|
|
# multi-token sequence, not the individual content tokens. Pure-BM25
|
|
# on tokens will surface literal-frame articles (Oceania the region,
|
|
# Asia the continent) instead of the reference (Nineteen Eighty-Four).
|
|
# The phrase-pattern retrieval route (commit 2026-05-01) targets this
|
|
# class. Bench reveals prevalence of the failure mode + whether the
|
|
# fix generalizes beyond Orwell.
|
|
has oceania always been at war with east asia?
|
|
who said may the force be with you?
|
|
what does winter is coming mean?
|
|
what is the meaning of rosebud?
|
|
who said to be or not to be that is the question?
|
|
what does the cake is a lie reference?
|