qa(#000008): classifier fix — count-question short-circuit + bounded fixtures

Caught by the 2026-05-03 dry-run distribution review across the
73-question bench set (§10.11.3 step 2):

  intensity     pre-fix    post-fix
  SINGULAR      61 (84%)   65 (89%)
  MANY           4 ( 5%)    0 ( 0%)   ← all 4 were `how many X?`
  ALL            1 ( 1%)    1 ( 1%)
  COMPREHENSIVE  1 ( 1%)    1 ( 1%)
  OPEN_REQUEST   5 ( 7%)    5 ( 7%)
  SMALL_NUM      1 ( 1%)    1 ( 1%)

Defect: `how many states are there?` matched the bare `\bmany\b`
pattern in MANY rung — wrong. `how many X?` is a count-question
SHAPE, asking for ONE numeric answer ("50"), not enumeration of
many things. Cap should be 1 (SINGULAR), not 8 (Hermes MANY).

Fix: count-question short-circuit in classify_question_quantifier()
that returns SINGULAR for `^\s*(?:and\s+|but\s+|so\s+)?how (?:many|much)\b`.
Anchored at start so buried `how many` (e.g. "list all the states;
how many are there?") doesn't suppress the rest of the question's
quantifier markers — the leading `list all` still wins.

9 new tests pin: count questions classify SINGULAR, leading
conjunctions don't break the short-circuit, buried `how many` does
NOT short-circuit (verifies anchor is leading-only).

Bonus — Finding 2 from the dry-run review: zero bounded universals
in bench fixture. Adds two:

  name all members of the beatles
  list all planets in the solar system

Both classify ALL · scope_bound_hint=bounded so the §10.1 bounded-
vs-unbounded distinction has live bench coverage. Without these,
--reject-broad correctness on bounded universals has no automated
test fixture.

915 tests passing (9 new); 36 skipped.
This commit is contained in:
russell@unturf.com 2026-05-03 08:29:36 -04:00
parent 041e865132
commit d24291bc8b
No known key found for this signature in database
3 changed files with 93 additions and 0 deletions

View file

@ -202,6 +202,22 @@ _PROPORTIONAL_PATTERNS = [
r"\bthe lion's share of\b",
]
# Count-question short-circuit. `how many` / `how much` at the
# start of a question (or after a leading wh-clause like
# "and how many...") is asking for a single numeric answer, not
# enumeration. Without this short-circuit the bare `\bmany\b`
# pattern below misfires.
#
# Anchored so we only short-circuit when the question is a count-
# question SHAPE — `how many` further into the question (e.g.
# "list the states; how many are there?") doesn't take precedence
# over the rest of the question's quantifier markers.
_COUNT_QUESTION_RE = re.compile(
r"^\s*(?:and\s+|but\s+|so\s+)?how (?:many|much)\b",
re.IGNORECASE,
)
# ABSENT — universal-negation.
_ABSENT_PATTERNS = [
r"\bnone\b",
@ -318,6 +334,31 @@ def classify_question_quantifier(question: str) -> dict:
"classifier_version": CLASSIFIER_VERSION,
}
# Count-question short-circuit (caught by 2026-05-03 dry-run
# review across bench/qa_questions.txt). `how many X?` and
# `how much X?` ask for a SINGLE numeric answer ("50 states",
# "206 bones") — not enumeration. Without this short-circuit,
# the bare `\bmany\b` pattern in _MANY_PATTERNS misfires on
# `how many` and the question lands in MANY rung (cap 8 on
# Hermes), which is wrong: a count question deserves cap 1
# (SINGULAR), not 8.
#
# The same applies to `how often`, `how long`, `how big` —
# all count/measurement questions with single-fact answers.
# We catch the dominant `how many|much` shape here; the others
# already classify SINGULAR by default.
if _COUNT_QUESTION_RE.search(question):
m = _COUNT_QUESTION_RE.search(question)
return {
"intensity": "SINGULAR",
"matched_token": m.group(0),
"explicit_count": None,
"is_broad": False,
"operational_shape": _OPERATIONAL_SHAPE["SINGULAR"],
"scope_bound_hint": "unknown",
"classifier_version": CLASSIFIER_VERSION,
}
candidates: list[tuple[str, str, int | None]] = []
# Per-rung detection. Earlier rungs run first but rung selection

View file

@ -33,6 +33,15 @@ describe the structure of DNA
# claim ceiling.
winners of all major sports?
# bounded universals — finite, corpus-known answer sets. Ticket
# #000008 §10.1 splits broad universals into bounded vs unbounded.
# These should classify ALL but with `scope_bound_hint: "bounded"`,
# meaning --reject-broad does NOT reject and the cap is the natural
# bound. Without these fixtures, the bounded-vs-unbounded distinction
# has no live bench coverage.
name all members of the beatles
list all planets in the solar system
# entity list — invites lazy-anchor on a magnet chunk
what dinosaurs were in the first jurassic park film?
who are the members of the beatles?

View file

@ -335,3 +335,46 @@ def test_simple_what_question_not_open_request():
out = classify_question_quantifier("what dinosaurs were in the first jurassic park film?")
assert out["intensity"] == "SINGULAR"
assert out["is_broad"] is False
# ----------------------------------------------------------- count-question short-circuit
# Caught by the 2026-05-03 dry-run distribution review across
# bench/qa_questions.txt. `how many X` matched the bare `\bmany\b`
# pattern in MANY rung — wrong: count questions ask for a SINGLE
# numeric answer, not enumeration. Cap should be 1 (SINGULAR), not
# 8 (Hermes MANY).
@pytest.mark.parametrize("q", [
"how many states are in the united states?",
"how many bones are in the adult human body?",
"how many wives did henry the eighth have?",
"how many moons does jupiter have?",
"how many planets are there?",
"how much does the earth weigh?",
"how much water is in the ocean?",
])
def test_how_many_classifies_singular_not_many(q):
out = classify_question_quantifier(q)
assert out["intensity"] == "SINGULAR", f"{q}{out}"
assert out["is_broad"] is False
assert out["matched_token"].lower().startswith("how ")
def test_how_many_with_leading_conjunction_still_classifies_singular():
"""`and how many X` still a count question — the conjunction
doesn't change the shape."""
out = classify_question_quantifier("and how many states are there?")
assert out["intensity"] == "SINGULAR"
def test_buried_how_many_does_not_short_circuit():
"""`how many` in the middle of a longer multi-clause question
is NOT necessarily count-question shape. The short-circuit
only fires on leading `how many` / `how much`."""
out = classify_question_quantifier(
"list all the states; how many are there?"
)
# Leading `list all` → ALL fires. Don't short-circuit on the
# buried "how many".
assert out["intensity"] == "ALL"