russell@unturf.com
9899a33b7b
Closes the 4 HYBRID_ENTITY over-grounds #000046 left in falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a multi-word proper noun matches the source — but "Insulin was discovered by Alexander Fleming" against "Penicillin was discovered by Alexander Fleming" matches on the shared "Alexander Fleming" while the swapped subject "Insulin" (the falsehood) is ignored. arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx) flags a >4-char Capitalized content token (stopword-filtered) or a digit-number in the answer absent from the source. _is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in verify_quotes' entity branch (proximity policy) in the weakest-grounding slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is what keeps a structured multi-claim summary untouched — the Matrix cast list (many entities, a tight cluster) and the TMNT answer (a numbered list with parenthetical nicknames the source omits): model-added accurate detail in a real summary isn't a contradiction, only the single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid entity-path regression tests still pass, pinned untouched. Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin / 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard pack around) and 5f-fal-live-028 — had expected_reason updated HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE recombinations (Mercury, Einstein — step 2.2) + 8 Formulate mis-segments (step 2.4). Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after (bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline (...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54, pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise floor. Per-row diff (675 common cells, 30 quote-mode rows changed audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity path — the gate fired on 0 legitimate QA answers in the whole bench. Every transition was LLM re-answer variance (verifier quote→quote with the verdict flipping); pointer/lattice deltas are noise too (the gate is in verify_quotes / quote mode, not the claim-lattice verifier). No regression — the gate is provably narrow on real traffic. Summarized in qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1. Tests: 4 new in test_verify.py (_is_single_sentence helper, _entity_salient_disagrees helper, swapped-subject → UNGROUNDED, gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated (positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5 dilution; ACCEPT via a degraded-parent sub-scenario). make test 2343 passed, 28 skipped. #000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md / Makefile / fixture _meta + notes / baseline JSON updated.
177 lines
4.8 KiB
JSON
177 lines
4.8 KiB
JSON
{
|
|
"schema_version": "bench-result-v1",
|
|
"results": [
|
|
{
|
|
"battery": "5f",
|
|
"sub_battery": "falsification",
|
|
"fixture_path": "bench/fixtures/5f/falsification-hard-v1.jsonl",
|
|
"fixture_digest": "8d21958a7e4218681678d1b2fa0ff7e551bbb2e2dfe0eb00c6ef947e1826ad0c",
|
|
"pass_count": 10,
|
|
"fail_count": 2,
|
|
"metrics": {
|
|
"error_detection_rate": 0.8333333333333334
|
|
},
|
|
"per_task": [
|
|
{
|
|
"task_id": "5f-fal-hard-001",
|
|
"passed": true,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"UNGROUNDED"
|
|
],
|
|
"detected": true,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-002",
|
|
"passed": true,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"UNGROUNDED"
|
|
],
|
|
"detected": true,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-003",
|
|
"passed": false,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"STRICT_PARAPHRASE"
|
|
],
|
|
"detected": false,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-004",
|
|
"passed": true,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"UNGROUNDED"
|
|
],
|
|
"detected": true,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-005",
|
|
"passed": false,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"STRICT_PARAPHRASE"
|
|
],
|
|
"detected": false,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-006",
|
|
"passed": true,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"UNGROUNDED"
|
|
],
|
|
"detected": true,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-007",
|
|
"passed": true,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"UNGROUNDED"
|
|
],
|
|
"detected": true,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-008",
|
|
"passed": true,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"UNGROUNDED",
|
|
"UNVERIFIED_QUOTE"
|
|
],
|
|
"detected": true,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-009",
|
|
"passed": true,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"UNGROUNDED"
|
|
],
|
|
"detected": true,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-010",
|
|
"passed": true,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"UNGROUNDED"
|
|
],
|
|
"detected": true,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-011",
|
|
"passed": true,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"UNGROUNDED"
|
|
],
|
|
"detected": true,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
},
|
|
{
|
|
"task_id": "5f-fal-hard-012",
|
|
"passed": true,
|
|
"detail": {
|
|
"source": "live",
|
|
"expected_reason": "UNGROUNDED",
|
|
"observed_violations": [
|
|
"UNGROUNDED"
|
|
],
|
|
"detected": true,
|
|
"verifier_method_root": "verify_quotes-v1"
|
|
}
|
|
}
|
|
],
|
|
"runtime_digest": "da5203214469cc5e682dd8c9edfcb0c95a758e1ba78567b4577b2c598c307951",
|
|
"timestamp": 1778519494
|
|
}
|
|
]
|
|
}
|