russell@unturf.com
002f84c5a4
§12 captures the 2026-05-03 post-implementation bench cycle:
§12.1 — pre-bench classifier scan (free, no LLM). Distribution
across the 73-question bench: 65 SINGULAR, 5 OPEN_REQUEST,
1 ALL, 1 COMPREHENSIVE, 1 SMALL_NUM_EXPLICIT, 0 MANY.
Documents the `how many X` defect caught + fixed in
d24291b.
§12.2 — live bench on 9-question broad subset (3 modes × n=3 = 81
runs). Per-mode summary, per-question table, pointer-mode
violation distribution.
§12.3 — telemetry verification end-to-end. Sampled per-question
classifier output showing intensity / scope_bound_hint /
claim_cap_applied populated as designed.
§12.4 — §10.8 decision-tree implications. Cap-only unlikely to
clear 5pp gate (pointer is already 0 STRICT); NO_EVIDENCE_
POINTER (9/27) is the load-bearing failure → Phase 3
reminder is the strongest single-knob candidate.
§12.5 — next bench cycles checklist (reminder-only, cap-only,
cap+reminder).
Headline findings:
- JSON mode hits 3/3 STRICT on bounded universal `name all members
of the beatles`. Same model, same verifier — bounded vs unbounded
is empirically real (validates §10.1 split).
- Pointer mode 0/27 STRICT on broad subset. CITATION_MISMATCH(14),
TITLE_MISMATCH(10), NO_EVIDENCE_POINTER(9), TOO_MANY_CLAIMS(7)
dominate.
- Quote mode 0.56 strict-rate validates keeping it out of
quantifier_guard_modes default.
Bench harness extension:
bench/qa_sweep.py gains --policy KEY=VALUE flag (repeatable).
Values are json.loads-decoded so booleans/ints/lists/strings work.
Enables §10.8 A/B cycles without monkey-patching defaults.
Plumbed through _run_one via new policy_overrides kwarg.
bench/qa_questions_quantifier_subset.txt landed as the 9-question
A/B fixture for ticket #000008.
23 lines
863 B
Text
23 lines
863 B
Text
# Broad-quantifier bench subset — Ticket #000008 dry-run measurement.
|
|
# Runs the 9 questions that classify as broad (ALL/COMPREHENSIVE/
|
|
# OPEN_REQUEST) plus one bounded-universal fixture pair so we can
|
|
# compare bounded vs unbounded behavior under guard-on dry-run.
|
|
#
|
|
# Used 2026-05-03 to measure post-implementation classifier and
|
|
# cap-lookup output. apply_caps stays FALSE during this run per
|
|
# §10.11.3 dry-run discipline — telemetry only.
|
|
|
|
# unbounded universals
|
|
winners of all major sports?
|
|
tell me all there is to know about york england?
|
|
|
|
# OPEN_REQUEST — verb-driven
|
|
tell me about connecticut
|
|
tell me about the C programming language
|
|
tell me about method man?
|
|
tell me about the roman empire
|
|
describe the structure of DNA
|
|
|
|
# bounded universals (Finding 2 from 2026-05-03 review)
|
|
name all members of the beatles
|
|
list all planets in the solar system
|