russell@unturf.com
d24291bc8b
Caught by the 2026-05-03 dry-run distribution review across the
73-question bench set (§10.11.3 step 2):
intensity pre-fix post-fix
SINGULAR 61 (84%) 65 (89%)
MANY 4 ( 5%) 0 ( 0%) ← all 4 were `how many X?`
ALL 1 ( 1%) 1 ( 1%)
COMPREHENSIVE 1 ( 1%) 1 ( 1%)
OPEN_REQUEST 5 ( 7%) 5 ( 7%)
SMALL_NUM 1 ( 1%) 1 ( 1%)
Defect: `how many states are there?` matched the bare `\bmany\b`
pattern in MANY rung — wrong. `how many X?` is a count-question
SHAPE, asking for ONE numeric answer ("50"), not enumeration of
many things. Cap should be 1 (SINGULAR), not 8 (Hermes MANY).
Fix: count-question short-circuit in classify_question_quantifier()
that returns SINGULAR for `^\s*(?:and\s+|but\s+|so\s+)?how (?:many|much)\b`.
Anchored at start so buried `how many` (e.g. "list all the states;
how many are there?") doesn't suppress the rest of the question's
quantifier markers — the leading `list all` still wins.
9 new tests pin: count questions classify SINGULAR, leading
conjunctions don't break the short-circuit, buried `how many` does
NOT short-circuit (verifies anchor is leading-only).
Bonus — Finding 2 from the dry-run review: zero bounded universals
in bench fixture. Adds two:
name all members of the beatles
list all planets in the solar system
Both classify ALL · scope_bound_hint=bounded so the §10.1 bounded-
vs-unbounded distinction has live bench coverage. Without these,
--reject-broad correctness on bounded universals has no automated
test fixture.
915 tests passing (9 new); 36 skipped.
154 lines
6.4 KiB
Text
154 lines
6.4 KiB
Text
# aborist QA-quality benchmark question set.
|
|
#
|
|
# One question per line. `#`-prefixed lines and blank lines ignored.
|
|
# Designed for the 2003-05-16 Wikipedia cur snapshot — questions must
|
|
# resolve against ~2003 article content. Stress-tests across the
|
|
# failure-mode shapes the verifier needs to handle.
|
|
|
|
# narrow factoid — well-anchored single-fact questions
|
|
what is the capital of france?
|
|
who wrote GNU linux?
|
|
when was a programming language named Python created?
|
|
who painted the mona lisa?
|
|
a bridge between new london & groton?
|
|
who wrote the play hamlet?
|
|
what is the chemical symbol for gold?
|
|
|
|
# broad descriptive — encyclopedic shape, prone to mode-collapse
|
|
tell me about connecticut
|
|
tell me about the C programming language
|
|
tell me about method man?
|
|
tell me all there is to know about york england?
|
|
tell me about the roman empire
|
|
describe the structure of DNA
|
|
|
|
# under-specified "all" — the word "all" reads to Hermes-3-8B as
|
|
# license to enumerate every adjacent fact in training prior, which
|
|
# in claim_lattice_pointer mode degrades to free-form prose with zero
|
|
# `[E\d+]` tags (FORMAT_COLLAPSED soft-demote, 2026-05-02). Stronger
|
|
# models (Qwen / GPT-4 family) plausibly recover format discipline
|
|
# under the same prompt — bench coverage of this shape lets us
|
|
# measure cross-model resilience. Ticket #000008 (broad-quantifier
|
|
# preflight guard) proposes upstream classification + per-model
|
|
# claim ceiling.
|
|
winners of all major sports?
|
|
|
|
# bounded universals — finite, corpus-known answer sets. Ticket
|
|
# #000008 §10.1 splits broad universals into bounded vs unbounded.
|
|
# These should classify ALL but with `scope_bound_hint: "bounded"`,
|
|
# meaning --reject-broad does NOT reject and the cap is the natural
|
|
# bound. Without these fixtures, the bounded-vs-unbounded distinction
|
|
# has no live bench coverage.
|
|
name all members of the beatles
|
|
list all planets in the solar system
|
|
|
|
# entity list — invites lazy-anchor on a magnet chunk
|
|
what dinosaurs were in the first jurassic park film?
|
|
who are the members of the beatles?
|
|
name simpsons family members including pets?
|
|
list of obelisk in connecticut
|
|
what are the planets of our solar system?
|
|
who were the original seven mercury astronauts?
|
|
|
|
# relationship / multi-fact
|
|
who is supermans girlfriend?
|
|
who is bilbo baggins's nephew?
|
|
what is the relationship between linux & unix?
|
|
who is veronica ballestrini & what month was she born?
|
|
who is luke skywalker's father and sister?
|
|
|
|
# comparison — multi-entity, prone to attribution drift
|
|
what's the difference between linux and bsd?
|
|
how does intel compare to amd?
|
|
what is the difference between http and ftp?
|
|
mac vs windows for software development
|
|
|
|
# niche / partial — corpus may be thin
|
|
what is the boltzmann constant?
|
|
who invented the doppler effect?
|
|
|
|
# date / time — when-questions stress year/date grounding
|
|
when did world war 2 end?
|
|
what year did the berlin wall fall?
|
|
when was the first moon landing?
|
|
when did the soviet union dissolve?
|
|
in what year was the magna carta signed?
|
|
|
|
# quantity — numeric grounding, prone to confabulated digits
|
|
how many states are in the united states?
|
|
how many bones are in the adult human body?
|
|
what is the population of japan?
|
|
how many wives did henry the eighth have?
|
|
how many moons does jupiter have?
|
|
|
|
# geographic — where / what country / location grounding
|
|
where is mount kilimanjaro located?
|
|
what country is the city of prague in?
|
|
in which ocean is madagascar?
|
|
where does the nile river begin?
|
|
what continent is egypt on?
|
|
|
|
# cause / effect — why-questions, narrative grounding
|
|
why did the titanic sink?
|
|
what caused the chernobyl disaster?
|
|
why did the dinosaurs go extinct?
|
|
what triggered world war 1?
|
|
|
|
# entity disambiguation — common names that collide with many entities
|
|
who is michael jordan?
|
|
who is george bush?
|
|
who is john smith of jamestown?
|
|
what is the matrix?
|
|
who is paul of tarsus?
|
|
|
|
# synonym / paraphrase robustness — same fact, different phrasing.
|
|
# Pairs probe whether equivalence_class question dedup collapses these
|
|
# at write & whether retrieval grounds them identically.
|
|
who founded microsoft?
|
|
who is the founder of microsoft?
|
|
when was the eiffel tower built?
|
|
what year was the eiffel tower constructed?
|
|
|
|
# leading / forensic — probe whether retrieval surfaces the right
|
|
# fact when the question's premise contradicts the popular narrative.
|
|
# Elevation question expected answer: north (per geographic surveys);
|
|
# corpus grounding may be shaky — coverage check can pass on
|
|
# topical-token overlap without the cited span containing the
|
|
# elevation fact, so this also stress-tests lazy-anchor STRICT.
|
|
which side is the ground elevation highest throughout the span of the great wall of china, the north or south?
|
|
whales are endangered from over hunting
|
|
did napoleon really die on saint helena?
|
|
|
|
# out-of-corpus — should land UNGROUNDED honestly
|
|
who is a benevolent dictator for life for mars?
|
|
what year does our cold fusion breakthrough happen?
|
|
who won the 2024 us presidential election?
|
|
what is the latest version of the ipad pro?
|
|
|
|
# allusion / reference frame — the diagnostic signal is a verbatim
|
|
# multi-token sequence, not the individual content tokens. Pure-BM25
|
|
# on tokens will surface literal-frame articles (Oceania the region,
|
|
# Asia the continent) instead of the reference (Nineteen Eighty-Four).
|
|
# The phrase-pattern retrieval route (commit 2026-05-01) targets this
|
|
# class. Bench reveals prevalence of the failure mode + whether the
|
|
# fix generalizes beyond Orwell.
|
|
has oceania always been at war with east asia?
|
|
who said may the force be with you?
|
|
what does winter is coming mean?
|
|
what is the meaning of rosebud?
|
|
who said to be or not to be that is the question?
|
|
what does the cake is a lie reference?
|
|
|
|
# hyphenation / orthography — query and corpus use different hyphen
|
|
# conventions for the same concept. FTS5 `unicode61` splits on hyphen
|
|
# at index AND query time, so `bi-polar` (the orthography many
|
|
# English readers actually use) tokenizes to [bi, polar] while the
|
|
# Wikipedia article `Bipolar disorder` tokenizes to [bipolar]. Two
|
|
# non-overlapping token sets for the same medical condition. The
|
|
# query-layer hyphen-fold (Ticket #000007, 2026-05-02) closes this
|
|
# class by emitting joined-no-hyphen variants on the query side and
|
|
# adding a hyphen-anchor accept path in `_filter_by_title_relevance`.
|
|
# Pre-fix: only Bi-Polar (album) / Bi-Polar Blues / BI disambiguation
|
|
# surfaced; the medical-condition cluster never reached the LLM.
|
|
# Post-fix: `Bipolar disorder` lands in top-K and grounds the answer.
|
|
bi-polar is rare?
|