qa/live: lock 3 bench standouts as STRICT-rate regression gates
Bench finding (2026-04-30, n=3 across 24 questions): JSON mode hit
S:3 H:0 U:0 on three questions where pointer mode landed S:0 H:3 U:0
or quote mode landed S:0 H:0 U:3 — clean modal divergence. Pinning
these as live fixtures gives us regression detection: a JSON-mode
quality drop on these questions reverts the STRICT signal first.
test_mona_lisa_strict_and_names_leonardo — was _grounded, tightened
to STRICT (bench: quote 0S, pointer 0S, JSON 3S)
test_supermans_girlfriend_strict_and_names_lois_lane (NEW)
— relationship-shape, JSON's strong suit (quote 0S, pointer 0S,
JSON 3S)
test_cold_fusion_breakthrough_year_strict_and_names_1989 (NEW)
— out-of-corpus rephrasing, atomic-claim rule unlocked grounding
(quote 0S 3U, pointer 3S, JSON 3S)
Per fox's "scoreboard + gates" docstring: bench gives aggregate
counts; these fixtures fail by name when the scoreboard moves.
This commit is contained in:
parent
5051f9538f
commit
9c47d5b97b
1 changed files with 38 additions and 2 deletions
|
|
@ -124,13 +124,49 @@ def test_linux_kernel_grounded_and_names_torvalds():
|
|||
f"answer missing 'Torvalds': {_answer_lower(r)[:200]}"
|
||||
|
||||
|
||||
def test_mona_lisa_grounded_and_names_leonardo():
|
||||
def test_mona_lisa_strict_and_names_leonardo():
|
||||
"""Bench standout (2026-04-30, n=3): JSON mode hit S:3 H:0 U:0
|
||||
while quote landed 0S and pointer 0S. Tightened from _grounded
|
||||
to STRICT to lock in the JSON-mode unique win as a regression
|
||||
gate — drop in JSON-mode quality on this question reverts the
|
||||
STRICT signal first."""
|
||||
r = _ask("who painted the mona lisa?")
|
||||
assert _grounded(r), f"audit_mode={r.get('audit_mode')!r}"
|
||||
assert r.get("audit_mode") == "STRICT", \
|
||||
f"expected STRICT (bench 3/3 STRICT in JSON), got {r.get('audit_mode')}"
|
||||
assert "leonardo" in _answer_lower(r), \
|
||||
f"answer missing 'Leonardo': {_answer_lower(r)[:200]}"
|
||||
|
||||
|
||||
def test_supermans_girlfriend_strict_and_names_lois_lane():
|
||||
"""Bench standout (2026-04-30, n=3): JSON mode hit S:3 H:0 U:0
|
||||
while pointer landed S:0 H:3 (every grounded answer demoted by
|
||||
lazy-anchor or pointer-saturation). The relationship-shape
|
||||
question 'who is X's girlfriend' is JSON's strong suit — atomic
|
||||
claims, single-name target. Gate: STRICT + 'lois' or 'lane'."""
|
||||
r = _ask("who is supermans girlfriend?")
|
||||
assert r.get("audit_mode") == "STRICT", \
|
||||
f"expected STRICT (bench 3/3 STRICT in JSON), got {r.get('audit_mode')}"
|
||||
txt = _answer_lower(r)
|
||||
assert "lois" in txt or "lane" in txt, \
|
||||
f"answer missing Lois/Lane: {txt[:200]}"
|
||||
|
||||
|
||||
def test_cold_fusion_breakthrough_year_strict_and_names_1989():
|
||||
"""Bench standout (2026-04-30, n=3): originally classified
|
||||
out-of-corpus but JSON mode hit S:3 H:0 U:0 — the corpus has
|
||||
Pons & Fleischmann's 1989 announcement covered. Quote mode
|
||||
landed 0S 0H 3U (UNGROUNDED) so this fixture also catches
|
||||
quote/JSON divergence. Gate: STRICT + a 1989-era marker
|
||||
(year, Pons, Fleischmann, or Utah)."""
|
||||
r = _ask("what year does our cold fusion breakthrough happen?")
|
||||
assert r.get("audit_mode") == "STRICT", \
|
||||
f"expected STRICT (bench 3/3 STRICT in JSON), got {r.get('audit_mode')}"
|
||||
txt = _answer_lower(r)
|
||||
cold_fusion_markers = ("1989", "pons", "fleischmann", "utah")
|
||||
assert any(m in txt for m in cold_fusion_markers), \
|
||||
f"answer missing 1989-era cold-fusion markers: {txt[:300]}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- retrieval disambiguation
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue