russell@unturf.com
d24291bc8b
Caught by the 2026-05-03 dry-run distribution review across the
73-question bench set (§10.11.3 step 2):
intensity pre-fix post-fix
SINGULAR 61 (84%) 65 (89%)
MANY 4 ( 5%) 0 ( 0%) ← all 4 were `how many X?`
ALL 1 ( 1%) 1 ( 1%)
COMPREHENSIVE 1 ( 1%) 1 ( 1%)
OPEN_REQUEST 5 ( 7%) 5 ( 7%)
SMALL_NUM 1 ( 1%) 1 ( 1%)
Defect: `how many states are there?` matched the bare `\bmany\b`
pattern in MANY rung — wrong. `how many X?` is a count-question
SHAPE, asking for ONE numeric answer ("50"), not enumeration of
many things. Cap should be 1 (SINGULAR), not 8 (Hermes MANY).
Fix: count-question short-circuit in classify_question_quantifier()
that returns SINGULAR for `^\s*(?:and\s+|but\s+|so\s+)?how (?:many|much)\b`.
Anchored at start so buried `how many` (e.g. "list all the states;
how many are there?") doesn't suppress the rest of the question's
quantifier markers — the leading `list all` still wins.
9 new tests pin: count questions classify SINGULAR, leading
conjunctions don't break the short-circuit, buried `how many` does
NOT short-circuit (verifies anchor is leading-only).
Bonus — Finding 2 from the dry-run review: zero bounded universals
in bench fixture. Adds two:
name all members of the beatles
list all planets in the solar system
Both classify ALL · scope_bound_hint=bounded so the §10.1 bounded-
vs-unbounded distinction has live bench coverage. Without these,
--reject-broad correctness on bounded universals has no automated
test fixture.
915 tests passing (9 new); 36 skipped.
|
||
|---|---|---|
| .. | ||
| emergent_log.jsonl | ||
| qa_questions.txt | ||
| qa_questions_quantifier_baseline.txt | ||
| qa_questions_smoke.txt | ||
| qa_sweep.py | ||
| run.sh | ||