qa(#000008): classifier fix — count-question short-circuit + bounded fixtures
Caught by the 2026-05-03 dry-run distribution review across the
73-question bench set (§10.11.3 step 2):
intensity pre-fix post-fix
SINGULAR 61 (84%) 65 (89%)
MANY 4 ( 5%) 0 ( 0%) ← all 4 were `how many X?`
ALL 1 ( 1%) 1 ( 1%)
COMPREHENSIVE 1 ( 1%) 1 ( 1%)
OPEN_REQUEST 5 ( 7%) 5 ( 7%)
SMALL_NUM 1 ( 1%) 1 ( 1%)
Defect: `how many states are there?` matched the bare `\bmany\b`
pattern in MANY rung — wrong. `how many X?` is a count-question
SHAPE, asking for ONE numeric answer ("50"), not enumeration of
many things. Cap should be 1 (SINGULAR), not 8 (Hermes MANY).
Fix: count-question short-circuit in classify_question_quantifier()
that returns SINGULAR for `^\s*(?:and\s+|but\s+|so\s+)?how (?:many|much)\b`.
Anchored at start so buried `how many` (e.g. "list all the states;
how many are there?") doesn't suppress the rest of the question's
quantifier markers — the leading `list all` still wins.
9 new tests pin: count questions classify SINGULAR, leading
conjunctions don't break the short-circuit, buried `how many` does
NOT short-circuit (verifies anchor is leading-only).
Bonus — Finding 2 from the dry-run review: zero bounded universals
in bench fixture. Adds two:
name all members of the beatles
list all planets in the solar system
Both classify ALL · scope_bound_hint=bounded so the §10.1 bounded-
vs-unbounded distinction has live bench coverage. Without these,
--reject-broad correctness on bounded universals has no automated
test fixture.
915 tests passing (9 new); 36 skipped.
This commit is contained in:
parent
041e865132
commit
d24291bc8b
3 changed files with 93 additions and 0 deletions
|
|
@ -202,6 +202,22 @@ _PROPORTIONAL_PATTERNS = [
|
|||
r"\bthe lion's share of\b",
|
||||
]
|
||||
|
||||
# Count-question short-circuit. `how many` / `how much` at the
|
||||
# start of a question (or after a leading wh-clause like
|
||||
# "and how many...") is asking for a single numeric answer, not
|
||||
# enumeration. Without this short-circuit the bare `\bmany\b`
|
||||
# pattern below misfires.
|
||||
#
|
||||
# Anchored so we only short-circuit when the question is a count-
|
||||
# question SHAPE — `how many` further into the question (e.g.
|
||||
# "list the states; how many are there?") doesn't take precedence
|
||||
# over the rest of the question's quantifier markers.
|
||||
_COUNT_QUESTION_RE = re.compile(
|
||||
r"^\s*(?:and\s+|but\s+|so\s+)?how (?:many|much)\b",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
# ABSENT — universal-negation.
|
||||
_ABSENT_PATTERNS = [
|
||||
r"\bnone\b",
|
||||
|
|
@ -318,6 +334,31 @@ def classify_question_quantifier(question: str) -> dict:
|
|||
"classifier_version": CLASSIFIER_VERSION,
|
||||
}
|
||||
|
||||
# Count-question short-circuit (caught by 2026-05-03 dry-run
|
||||
# review across bench/qa_questions.txt). `how many X?` and
|
||||
# `how much X?` ask for a SINGLE numeric answer ("50 states",
|
||||
# "206 bones") — not enumeration. Without this short-circuit,
|
||||
# the bare `\bmany\b` pattern in _MANY_PATTERNS misfires on
|
||||
# `how many` and the question lands in MANY rung (cap 8 on
|
||||
# Hermes), which is wrong: a count question deserves cap 1
|
||||
# (SINGULAR), not 8.
|
||||
#
|
||||
# The same applies to `how often`, `how long`, `how big` —
|
||||
# all count/measurement questions with single-fact answers.
|
||||
# We catch the dominant `how many|much` shape here; the others
|
||||
# already classify SINGULAR by default.
|
||||
if _COUNT_QUESTION_RE.search(question):
|
||||
m = _COUNT_QUESTION_RE.search(question)
|
||||
return {
|
||||
"intensity": "SINGULAR",
|
||||
"matched_token": m.group(0),
|
||||
"explicit_count": None,
|
||||
"is_broad": False,
|
||||
"operational_shape": _OPERATIONAL_SHAPE["SINGULAR"],
|
||||
"scope_bound_hint": "unknown",
|
||||
"classifier_version": CLASSIFIER_VERSION,
|
||||
}
|
||||
|
||||
candidates: list[tuple[str, str, int | None]] = []
|
||||
|
||||
# Per-rung detection. Earlier rungs run first but rung selection
|
||||
|
|
|
|||
|
|
@ -33,6 +33,15 @@ describe the structure of DNA
|
|||
# claim ceiling.
|
||||
winners of all major sports?
|
||||
|
||||
# bounded universals — finite, corpus-known answer sets. Ticket
|
||||
# #000008 §10.1 splits broad universals into bounded vs unbounded.
|
||||
# These should classify ALL but with `scope_bound_hint: "bounded"`,
|
||||
# meaning --reject-broad does NOT reject and the cap is the natural
|
||||
# bound. Without these fixtures, the bounded-vs-unbounded distinction
|
||||
# has no live bench coverage.
|
||||
name all members of the beatles
|
||||
list all planets in the solar system
|
||||
|
||||
# entity list — invites lazy-anchor on a magnet chunk
|
||||
what dinosaurs were in the first jurassic park film?
|
||||
who are the members of the beatles?
|
||||
|
|
|
|||
|
|
@ -335,3 +335,46 @@ def test_simple_what_question_not_open_request():
|
|||
out = classify_question_quantifier("what dinosaurs were in the first jurassic park film?")
|
||||
assert out["intensity"] == "SINGULAR"
|
||||
assert out["is_broad"] is False
|
||||
|
||||
|
||||
# ----------------------------------------------------------- count-question short-circuit
|
||||
|
||||
# Caught by the 2026-05-03 dry-run distribution review across
|
||||
# bench/qa_questions.txt. `how many X` matched the bare `\bmany\b`
|
||||
# pattern in MANY rung — wrong: count questions ask for a SINGLE
|
||||
# numeric answer, not enumeration. Cap should be 1 (SINGULAR), not
|
||||
# 8 (Hermes MANY).
|
||||
|
||||
@pytest.mark.parametrize("q", [
|
||||
"how many states are in the united states?",
|
||||
"how many bones are in the adult human body?",
|
||||
"how many wives did henry the eighth have?",
|
||||
"how many moons does jupiter have?",
|
||||
"how many planets are there?",
|
||||
"how much does the earth weigh?",
|
||||
"how much water is in the ocean?",
|
||||
])
|
||||
def test_how_many_classifies_singular_not_many(q):
|
||||
out = classify_question_quantifier(q)
|
||||
assert out["intensity"] == "SINGULAR", f"{q} → {out}"
|
||||
assert out["is_broad"] is False
|
||||
assert out["matched_token"].lower().startswith("how ")
|
||||
|
||||
|
||||
def test_how_many_with_leading_conjunction_still_classifies_singular():
|
||||
"""`and how many X` still a count question — the conjunction
|
||||
doesn't change the shape."""
|
||||
out = classify_question_quantifier("and how many states are there?")
|
||||
assert out["intensity"] == "SINGULAR"
|
||||
|
||||
|
||||
def test_buried_how_many_does_not_short_circuit():
|
||||
"""`how many` in the middle of a longer multi-clause question
|
||||
is NOT necessarily count-question shape. The short-circuit
|
||||
only fires on leading `how many` / `how much`."""
|
||||
out = classify_question_quantifier(
|
||||
"list all the states; how many are there?"
|
||||
)
|
||||
# Leading `list all` → ALL fires. Don't short-circuit on the
|
||||
# buried "how many".
|
||||
assert out["intensity"] == "ALL"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue