ticket(#000008): §12 dry-run bench findings + --policy harness flag
§12 captures the 2026-05-03 post-implementation bench cycle:
§12.1 — pre-bench classifier scan (free, no LLM). Distribution
across the 73-question bench: 65 SINGULAR, 5 OPEN_REQUEST,
1 ALL, 1 COMPREHENSIVE, 1 SMALL_NUM_EXPLICIT, 0 MANY.
Documents the `how many X` defect caught + fixed in
d24291b.
§12.2 — live bench on 9-question broad subset (3 modes × n=3 = 81
runs). Per-mode summary, per-question table, pointer-mode
violation distribution.
§12.3 — telemetry verification end-to-end. Sampled per-question
classifier output showing intensity / scope_bound_hint /
claim_cap_applied populated as designed.
§12.4 — §10.8 decision-tree implications. Cap-only unlikely to
clear 5pp gate (pointer is already 0 STRICT); NO_EVIDENCE_
POINTER (9/27) is the load-bearing failure → Phase 3
reminder is the strongest single-knob candidate.
§12.5 — next bench cycles checklist (reminder-only, cap-only,
cap+reminder).
Headline findings:
- JSON mode hits 3/3 STRICT on bounded universal `name all members
of the beatles`. Same model, same verifier — bounded vs unbounded
is empirically real (validates §10.1 split).
- Pointer mode 0/27 STRICT on broad subset. CITATION_MISMATCH(14),
TITLE_MISMATCH(10), NO_EVIDENCE_POINTER(9), TOO_MANY_CLAIMS(7)
dominate.
- Quote mode 0.56 strict-rate validates keeping it out of
quantifier_guard_modes default.
Bench harness extension:
bench/qa_sweep.py gains --policy KEY=VALUE flag (repeatable).
Values are json.loads-decoded so booleans/ints/lists/strings work.
Enables §10.8 A/B cycles without monkey-patching defaults.
Plumbed through _run_one via new policy_overrides kwarg.
bench/qa_questions_quantifier_subset.txt landed as the 9-question
A/B fixture for ticket #000008.
russell@unturf.com
002f84c5a4