From 9899a33b7b8aa358ef450362307b82bc08075d17 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Mon, 11 May 2026 13:57:45 -0400 Subject: [PATCH] =?UTF-8?q?#000048=20step=202.1=20=E2=80=94=20verify=5Fquo?= =?UTF-8?q?tes=20entity=20salient-token-disagreement=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the 4 HYBRID_ENTITY over-grounds #000046 left in falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a multi-word proper noun matches the source — but "Insulin was discovered by Alexander Fleming" against "Penicillin was discovered by Alexander Fleming" matches on the shared "Alexander Fleming" while the swapped subject "Insulin" (the falsehood) is ignored. arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx) flags a >4-char Capitalized content token (stopword-filtered) or a digit-number in the answer absent from the source. _is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in verify_quotes' entity branch (proximity policy) in the weakest-grounding slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is what keeps a structured multi-claim summary untouched — the Matrix cast list (many entities, a tight cluster) and the TMNT answer (a numbered list with parenthetical nicknames the source omits): model-added accurate detail in a real summary isn't a contradiction, only the single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid entity-path regression tests still pass, pinned untouched. Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin / 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard pack around) and 5f-fal-live-028 — had expected_reason updated HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE recombinations (Mercury, Einstein — step 2.2) + 8 Formulate mis-segments (step 2.4). Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after (bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline (...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54, pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise floor. Per-row diff (675 common cells, 30 quote-mode rows changed audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity path — the gate fired on 0 legitimate QA answers in the whole bench. Every transition was LLM re-answer variance (verifier quote→quote with the verdict flipping); pointer/lattice deltas are noise too (the gate is in verify_quotes / quote mode, not the claim-lattice verifier). No regression — the gate is provably narrow on real traffic. Summarized in qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1. Tests: 4 new in test_verify.py (_is_single_sentence helper, _entity_salient_disagrees helper, swapped-subject → UNGROUNDED, gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated (positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5 dilution; ACCEPT via a degraded-parent sub-scenario). make test 2343 passed, 28 skipped. #000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md / Makefile / fixture _meta + notes / baseline JSON updated. --- Makefile | 16 ++-- arborist/qa/verify.py | 61 +++++++++++++++ bench/fixtures/5f/falsification-hard-v1.jsonl | 10 +-- bench/fixtures/5f/falsification-live-v1.jsonl | 4 +- .../results/baseline-falsification-hard.json | 34 ++++---- docs/TICKETS.md | 4 +- docs/qa-modes-bench.md | 39 ++++++++++ ...ket-000012-selection-consensus-protocol.md | 17 ++-- .../ticket-000046-harder-5sf-fixture-tier.md | 27 +++---- ...fier-upgrade-recombination-segmentation.md | 75 ++++++++++++++++-- tests/test_bench_batteries.py | 67 +++++++++------- tests/test_verify.py | 77 +++++++++++++++++++ 12 files changed, 340 insertions(+), 91 deletions(-) diff --git a/Makefile b/Makefile index 4cbd2c1..adad1cd 100644 --- a/Makefile +++ b/Makefile @@ -493,13 +493,15 @@ bench-5f-live: bench-5f-formulate-live bench-5f-feedback-loop-live bench-5f-func # #000046 Phase 1 — HARD live-path Falsification tier. Every fixture is # a near-miss the correct verdict is UNGROUNDED on; verify_quotes -# over-grounds 6 of 12 today → rate 6/12 = 0.5 at HEAD (the paraphrase -# numeric-agreement gate, #000046 closure, caught the other 2). A real -# below-ceiling baseline. NOT part of bench-5f / bench-fork-baseline. -# bench-fork-baseline-hard pins the rate so a future verifier upgrade -# shows up as a larger positive gamma*Delta5f term. +# over-grounds 2 of 12 today → rate 10/12 = 0.833 at HEAD (#000046's +# paraphrase numeric gate caught 2, #000048 step 2.1's entity +# salient-token gate caught 4 more; the last 2 are recombination +# STRICT_PARAPHRASE — #000048 step 2.2). A real below-ceiling baseline. +# NOT part of bench-5f / bench-fork-baseline. bench-fork-baseline-hard +# pins the rate so a future verifier upgrade shows up as a larger +# positive gamma*Delta5f term. FORK_PARENT_HARD ?= bench/results/baseline-falsification-hard.json -# The hard pack has its expected failures (6/12 over-grounds), so the +# The hard pack has its expected failures (2/12 over-grounds), so the # runner exits 1 — that's the below-ceiling baseline working as # designed, not a build error. `|| true` keeps the make target green; # the JSON / report is written regardless (the --out write precedes @@ -511,7 +513,7 @@ bench-fork-baseline-hard: bootstrap ## #000046 — pin the below-ceiling hard-Fa @mkdir -p bench/results PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5f --sub falsification \ --fixtures bench/fixtures/5f/falsification-hard-v1.jsonl --out $(FORK_PARENT_HARD) || true - @echo ">> below-ceiling parent pinned: $(FORK_PARENT_HARD) (6/12 over-grounds expected → runner exits 1; the JSON is written)" + @echo ">> below-ceiling parent pinned: $(FORK_PARENT_HARD) (2/12 over-grounds expected → runner exits 1; the JSON is written)" # #000046 Phase 2 — HARD Formulate tier. Prose that parse_pointer_claims # should segment a particular way; the line/bullet-based parser # mis-segments 8 of 12 (merges multi-claim lines, splits wrapped diff --git a/arborist/qa/verify.py b/arborist/qa/verify.py index 54461d3..3d4552e 100644 --- a/arborist/qa/verify.py +++ b/arborist/qa/verify.py @@ -392,6 +392,53 @@ def _numeric_signature(text: str) -> frozenset[str]: return frozenset(m.group(0).replace(",", "") for m in _DIGIT_RUN_RE.finditer(text)) +_SENTENCE_BREAK_RE = re.compile(r"[.!?](?=\s)") + + +def _is_single_sentence(text: str) -> bool: + """One sentence — no internal ``. `` / ``! `` / ``? `` break (a trailing + terminator with nothing after it is fine). A numbered/bulleted list + ("1. … 2. …") or a multi-sentence summary is not single; "Insulin + was discovered by Alexander Fleming." is. Conservative: a borderline + abbreviation ("8,849 m. tall.") reads as not-single, so the gate it + guards simply doesn't fire — the safe direction.""" + return not _SENTENCE_BREAK_RE.search(text.strip()) + + +def _entity_salient_disagrees(answer_text: str, norm_ctx: str) -> bool: + """Entity-strategy gate (ticket #000048): does the answer assert a + *salient near-proper-noun token* — a > 4-char Capitalized content + token, stopword-filtered — or a digit-number that the source lacks? + + Used in :func:`verify_quotes` *only* on the weakest entity grounding + — a single short sentence whose lone matching multi-word proper noun + is not part of a cluster (caller checks ``len(verified) <= 1`` and + :func:`_is_single_sentence`). There, "Insulin was discovered by + Alexander Fleming" against "Penicillin was discovered by Alexander + Fleming" matches on the shared "Alexander Fleming" while the + *swapped subject* ("Insulin", capitalized, absent from the source) + is exactly the falsehood — so we decline the grounding. The narrow + caller-gate is what keeps a structured multi-claim summary (the + Matrix cast list — many entities, a tight cluster — or the TMNT + answer — a numbered list with parenthetical nicknames the source + omits) untouched: model-added accurate detail in a real summary + isn't a contradiction, and only the single-sentence-one-weak-match + shape is. + """ + if not (_numeric_signature(_normalize(answer_text)) <= _numeric_signature(norm_ctx)): + return True + for raw in answer_text.split(): + core = raw.strip(_TOKEN_PUNCT_STRIP) + if len(core) <= 4 or not core[0].isupper(): + continue + low = core.lower() + if low in _ENGLISH_STOPWORDS: + continue + if low not in norm_ctx: + return True + return False + + # Common English stopwords of length >= 4 chars. Hand-curated rather # than imported from NLTK to keep arborist dependency-light and the # behavior pinned to a known set. Tokens are normalized form @@ -657,6 +704,20 @@ def verify_quotes( mode = "STRICT" elif cluster: mode = "HYBRID" + elif ( + # #000048: weakest grounding — a single short + # sentence whose lone matching multi-word proper + # noun isn't part of a cluster. If the claim + # asserts a salient capitalized token / number the + # source lacks (a swapped subject / city / ocean / + # year — "Insulin was discovered by Alexander + # Fleming" vs "Penicillin was…"), the shared + # proper noun doesn't ground it → UNGROUNDED. + len(verified) <= 1 + and _is_single_sentence(answer_text) + and _entity_salient_disagrees(answer_text, norm_ctx) + ): + mode = "UNGROUNDED" elif verified: mode = "HYBRID" else: diff --git a/bench/fixtures/5f/falsification-hard-v1.jsonl b/bench/fixtures/5f/falsification-hard-v1.jsonl index db84830..b61b2df 100644 --- a/bench/fixtures/5f/falsification-hard-v1.jsonl +++ b/bench/fixtures/5f/falsification-hard-v1.jsonl @@ -1,12 +1,12 @@ -{"_meta": {"battery": "5f", "sub_battery": "falsification", "version": "v1", "task_count": 12, "notes": "#000046 — HARD live-path Falsification tier. Every fixture is a near-miss: the claim is FALSE / unsupported, so the correct verdict is UNGROUNDED, and that is what `expected_reason` records. As of #000046 closure (2026-05-11), verify_quotes' paraphrase numeric-agreement gate catches the 2 magnitude/year over-grounds (50-vs-100, 300-vs-300,000) → those now correctly land UNGROUNDED. The remaining 6 still over-ground: 4 via the entity-proximity strategy (matches on a shared proper noun while the answer's other salient term is wrong → HYBRID_ENTITY) and 2 via paraphrase token-coverage on a recombined-no-number claim (\"Mercury is the largest\" vs source \"Jupiter is the largest; Mercury is the smallest\" — all content tokens present, just recombined → STRICT_PARAPHRASE). The other 4 the verifier always handled correctly. Rate at HEAD = 6/12 = 0.5 — a real below-ceiling baseline. A further verifier upgrade (order/dependency-aware match or a contradiction check, vs the lexical token-coverage that can't see recombination) lifts the rate toward 1.0 → a larger positive gamma*Delta5f term in fork_score. verify_quotes is pure-lexical / deterministic, so the rate is stable. NOT part of `make bench-5f`, `make bench-5s5t5f`, `bench-fork-baseline`, or `runner --all`; pinned separately via `make bench-fork-baseline-hard`. Pre-documented gap this builds around: 5f-fal-live-003 in falsification-live-v1.jsonl. See ticket #000046."}} -{"id": "5f-fal-hard-001", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Insulin was discovered by Alexander Fleming.", "context": "Penicillin was discovered by Alexander Fleming in 1928.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: HYBRID_ENTITY — substance swapped, but entity strategy matches on shared 'Alexander Fleming'."} -{"id": "5f-fal-hard-002", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Eiffel Tower is in Berlin.", "context": "The Eiffel Tower is in Paris, designed by Gustave Eiffel.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: HYBRID_ENTITY — city swapped, entity strategy matches on 'Eiffel Tower'/'Eiffel'."} +{"_meta": {"battery": "5f", "sub_battery": "falsification", "version": "v1", "task_count": 12, "notes": "#000046 — HARD live-path Falsification tier. Every fixture is a near-miss: the claim is FALSE / unsupported, so the correct verdict is UNGROUNDED, and that is what `expected_reason` records. As of #000048 step 2.1 (2026-05-11): #000046's paraphrase numeric-agreement gate catches the 2 magnitude/year over-grounds (50-vs-100, 300-vs-300,000) AND #000048's entity salient-token-disagreement gate catches the 4 HYBRID_ENTITY over-grounds (Insulin/Berlin/1889/Pacific — single short sentence, lone non-clustered proper-noun match, a >4-char capitalized token or number the source lacks) → all 6 now correctly land UNGROUNDED. The remaining 2 over-ground via paraphrase token-coverage on a recombined-no-number claim (\"Mercury is the largest\" vs source \"Jupiter is the largest; Mercury is the smallest\" — all content tokens present, just recombined → STRICT_PARAPHRASE; #000048 step 2.2 is the sequence-aware match that closes those). The other 4 the verifier always handled correctly. Rate at HEAD = 10/12 = 0.833 — a real below-ceiling baseline. verify_quotes is pure-lexical / deterministic, so the rate is stable. NOT part of `make bench-5f`, `make bench-5s5t5f`, `bench-fork-baseline`, or `runner --all`; pinned separately via `make bench-fork-baseline-hard`. Pre-documented gap this builds around: 5f-fal-live-003 in falsification-live-v1.jsonl. See tickets #000046 (closed) + #000048."}} +{"id": "5f-fal-hard-001", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Insulin was discovered by Alexander Fleming.", "context": "Penicillin was discovered by Alexander Fleming in 1928.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000048 step 2.1): entity match on shared 'Alexander Fleming' but the swapped subject 'Insulin' (capitalized, absent from source) trips the salient gate → UNGROUNDED. PASSES today."} +{"id": "5f-fal-hard-002", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Eiffel Tower is in Berlin.", "context": "The Eiffel Tower is in Paris, designed by Gustave Eiffel.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000048 step 2.1): entity match on 'Eiffel Tower' but the swapped city 'Berlin' (absent from source) trips the salient gate → UNGROUNDED. PASSES today."} {"id": "5f-fal-hard-003", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Mercury is the largest planet in the Solar System.", "context": "Jupiter is the largest planet in the Solar System; Mercury is the smallest.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: STRICT_PARAPHRASE — verifier claims STRICT on a false statement; paraphrase token-coverage matches 'largest planet in the Solar System'."} {"id": "5f-fal-hard-004", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Water boils at 50 degrees Celsius at sea level.", "context": "Water boils at 100 degrees Celsius at sea level.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000046 closure): the paraphrase numeric-agreement gate sees 50 ∉ {100} → UNGROUNDED. PASSES today."} {"id": "5f-fal-hard-005", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Albert Einstein developed the theory of evolution.", "context": "Albert Einstein developed the theory of relativity; Charles Darwin developed evolution.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: STRICT_PARAPHRASE — 'evolution' vs 'relativity' swapped; 'Albert Einstein developed the theory of' token-covers + 'evolution' appears in context (attributed to Darwin)."} -{"id": "5f-fal-hard-006", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The French Revolution began in 1889.", "context": "The French Revolution began in 1789, with the storming of the Bastille.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: HYBRID_ENTITY — year off by a century (1889 vs 1789); entity strategy matches on 'French Revolution'."} +{"id": "5f-fal-hard-006", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The French Revolution began in 1889.", "context": "The French Revolution began in 1789, with the storming of the Bastille.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000048 step 2.1): entity match on 'French Revolution' but the swapped year 1889 ∉ {1789} trips the salient gate's numeric arm → UNGROUNDED. PASSES today."} {"id": "5f-fal-hard-007", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The speed of light is approximately 300 kilometers per second.", "context": "The speed of light is approximately 300,000 kilometers per second in a vacuum.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000046 closure): the paraphrase numeric-agreement gate sees 300 ∉ {300000} (← '300,000' comma-stripped) → UNGROUNDED. PASSES today."} -{"id": "5f-fal-hard-008", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Titanic sank in the Pacific Ocean.", "context": "The Titanic sank in the North Atlantic Ocean in 1912.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: HYBRID_ENTITY — ocean swapped (Pacific vs North Atlantic); entity strategy matches on 'Titanic'."} +{"id": "5f-fal-hard-008", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Titanic sank in the Pacific Ocean.", "context": "The Titanic sank in the North Atlantic Ocean in 1912.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000048 step 2.1): entity match on 'The Titanic' but the swapped 'Pacific' (absent from source — it says 'North Atlantic Ocean') trips the salient gate → UNGROUNDED. PASSES today."} {"id": "5f-fal-hard-009", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Shakespeare wrote War and Peace.", "context": "Tolstoy wrote War and Peace; Shakespeare wrote Hamlet.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Correctly UNGROUNDED at HEAD — verifier does NOT over-ground this near-miss. PASSES today; kept as headroom marker (a regression that started over-grounding this would drop the rate)."} {"id": "5f-fal-hard-010", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The mitochondria is the powerhouse of the nucleus.", "context": "The mitochondria is the powerhouse of the cell.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Correctly UNGROUNDED at HEAD — 'nucleus' vs 'cell' swap is caught. PASSES today."} {"id": "5f-fal-hard-011", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The capital of Australia is Sydney.", "context": "The capital of Australia is Canberra; Sydney is the largest city.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Correctly UNGROUNDED at HEAD. PASSES today."} diff --git a/bench/fixtures/5f/falsification-live-v1.jsonl b/bench/fixtures/5f/falsification-live-v1.jsonl index 88d9830..84035be 100644 --- a/bench/fixtures/5f/falsification-live-v1.jsonl +++ b/bench/fixtures/5f/falsification-live-v1.jsonl @@ -1,7 +1,7 @@ {"_meta": {"battery": "5f", "sub_battery": "falsification", "version": "v1", "task_count": 50, "notes": "Phase 1b.2: answer_text + context routes through arborist.qa.verify.verify_quotes (real verifier surface). Live signals: UNGROUNDED, STRICT_, HYBRID_, UNVERIFIED_QUOTE. Phase 1d (2026-05-09): expanded 30→50."}} {"id": "5f-fal-live-001", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Random unsupported claim about nothing.", "context": "Some completely unrelated context.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"} {"id": "5f-fal-live-002", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "K2 is the tallest mountain.", "context": "Mount Everest is the tallest mountain at 8849 meters.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"} -{"id": "5f-fal-live-003", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Insulin was discovered by Alexander Fleming.", "context": "Penicillin was discovered by Alexander Fleming in 1928.", "expected_reason": "HYBRID_ENTITY", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "verify_quotes' entity strategy matches on shared 'Alexander Fleming' entity even though the substance is swapped. Documents a known soft-signal gap; the title-relevance hard check + claim-lattice verifier catch this case in production."} +{"id": "5f-fal-live-003", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Insulin was discovered by Alexander Fleming.", "context": "Penicillin was discovered by Alexander Fleming in 1928.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Was HYBRID_ENTITY — the entity strategy matched the shared 'Alexander Fleming' while ignoring the swapped subject 'Insulin' (absent from the source). #000048 step 2.1's entity salient-token-disagreement gate (single short sentence, lone non-clustered match, a >4-char capitalized token the source lacks) now declines that grounding → UNGROUNDED. This is the soft-signal gap #000046 built its falsification-hard pack around; #000048 closed it."} {"id": "5f-fal-live-004", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Water boils at 100 degrees Celsius at sea level.", "context": "Water boils at 100 degrees Celsius at sea level.", "expected_reason": "STRICT_SPAN", "verifier_method_root": "verify_quotes-v1", "expected": "pass"} {"id": "5f-fal-live-005", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Hydrogen has atomic number 1.", "context": "Hydrogen has atomic number 1 and is the lightest chemical element.", "expected_reason": "STRICT_SPAN", "verifier_method_root": "verify_quotes-v1", "expected": "pass"} {"id": "5f-fal-live-006", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "There are nine planets in the solar system.", "context": "There are eight planets in the solar system since Pluto's reclassification.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"} @@ -26,7 +26,7 @@ {"id": "5f-fal-live-025", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Pluto is a planet.", "context": "Pluto was reclassified as a dwarf planet in 2006.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"} {"id": "5f-fal-live-026", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Nile is short.", "context": "The Nile is one of the longest rivers in the world at about 6650 kilometers.", "expected_reason": "HYBRID_ENTITY", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Live verifier signal: verify_quotes returns HYBRID_ENTITY for this (answer, context) pair. Captured here as a behavior-pinning fixture; divergence from synthetic expectation is the value of live mode."} {"id": "5f-fal-live-027", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Sputnik 1 was launched in 1962.", "context": "Sputnik 1 was launched in 1957.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"} -{"id": "5f-fal-live-028", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Eiffel Tower is in London.", "context": "The Eiffel Tower is in Paris France.", "expected_reason": "HYBRID_ENTITY", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Live verifier signal: verify_quotes returns HYBRID_ENTITY for this (answer, context) pair. Captured here as a behavior-pinning fixture; divergence from synthetic expectation is the value of live mode."} +{"id": "5f-fal-live-028", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Eiffel Tower is in London.", "context": "The Eiffel Tower is in Paris France.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Was HYBRID_ENTITY — the entity strategy matched the shared 'Eiffel Tower' while ignoring the swapped city 'London' (absent from the source). #000048 step 2.1's entity salient-token-disagreement gate now declines that grounding → UNGROUNDED."} {"id": "5f-fal-live-029", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Pride and Prejudice was published in 1813.", "context": "Pride and Prejudice was published in 1813.", "expected_reason": "STRICT_SPAN", "verifier_method_root": "verify_quotes-v1", "expected": "pass"} {"id": "5f-fal-live-030", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The capital of France is Berlin.", "context": "The capital of France is Paris.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"} {"id": "5f-fal-live-031", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Random unsupported claim about nothing. (variant 31)", "context": "Some context.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"} diff --git a/bench/results/baseline-falsification-hard.json b/bench/results/baseline-falsification-hard.json index 037104c..bdba796 100644 --- a/bench/results/baseline-falsification-hard.json +++ b/bench/results/baseline-falsification-hard.json @@ -5,36 +5,36 @@ "battery": "5f", "sub_battery": "falsification", "fixture_path": "bench/fixtures/5f/falsification-hard-v1.jsonl", - "fixture_digest": "519524e6ceab694d14971832e4cd676c952f032bb05580637d9185c2b6e22096", - "pass_count": 6, - "fail_count": 6, + "fixture_digest": "8d21958a7e4218681678d1b2fa0ff7e551bbb2e2dfe0eb00c6ef947e1826ad0c", + "pass_count": 10, + "fail_count": 2, "metrics": { - "error_detection_rate": 0.5 + "error_detection_rate": 0.8333333333333334 }, "per_task": [ { "task_id": "5f-fal-hard-001", - "passed": false, + "passed": true, "detail": { "source": "live", "expected_reason": "UNGROUNDED", "observed_violations": [ - "HYBRID_ENTITY" + "UNGROUNDED" ], - "detected": false, + "detected": true, "verifier_method_root": "verify_quotes-v1" } }, { "task_id": "5f-fal-hard-002", - "passed": false, + "passed": true, "detail": { "source": "live", "expected_reason": "UNGROUNDED", "observed_violations": [ - "HYBRID_ENTITY" + "UNGROUNDED" ], - "detected": false, + "detected": true, "verifier_method_root": "verify_quotes-v1" } }, @@ -79,14 +79,14 @@ }, { "task_id": "5f-fal-hard-006", - "passed": false, + "passed": true, "detail": { "source": "live", "expected_reason": "UNGROUNDED", "observed_violations": [ - "HYBRID_ENTITY" + "UNGROUNDED" ], - "detected": false, + "detected": true, "verifier_method_root": "verify_quotes-v1" } }, @@ -105,15 +105,15 @@ }, { "task_id": "5f-fal-hard-008", - "passed": false, + "passed": true, "detail": { "source": "live", "expected_reason": "UNGROUNDED", "observed_violations": [ - "HYBRID_ENTITY", + "UNGROUNDED", "UNVERIFIED_QUOTE" ], - "detected": false, + "detected": true, "verifier_method_root": "verify_quotes-v1" } }, @@ -171,7 +171,7 @@ } ], "runtime_digest": "da5203214469cc5e682dd8c9edfcb0c95a758e1ba78567b4577b2c598c307951", - "timestamp": 1778507146 + "timestamp": 1778519494 } ] } diff --git a/docs/TICKETS.md b/docs/TICKETS.md index fbc00b8..cac3e1c 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -92,9 +92,9 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| -| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | open · awaiting go/no-go · doc-only; closes the headroom #000046 left (6 over-grounds in `falsification-hard` + 8 mis-segments in `formulate-hard`). Recommends: 2.1 entity salient-token-disagreement gate (mirrors #000046's numeric gate) → 2.4 `parse_pointer_claims` clause segmentation → 2.2 sequence-aware paraphrase; defer 2.3 mini-NLI. Each bench-gated. #000046 follow-up | 2026-05-11 | — | +| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | in progress · **step 2.1 landed 2026-05-11** — `verify_quotes` entity salient-token-disagreement gate (`_entity_salient_disagrees` + `_is_single_sentence`; demotes a single short sentence whose lone non-clustered proper-noun match co-occurs with a >4-char capitalized token / number the source lacks) → `falsification-hard` 6/12 → 10/12; bench-gated (`make bench-qa` n=3×75×3 — 0 gate-attributable QA shifts, deltas within the 5-pp floor); 2 live-pack fixtures' `expected_reason` updated HYBRID_ENTITY→UNGROUNDED. Steps 2.4 (`parse_pointer_claims` clause segmentation) + 2.2 (sequence-aware paraphrase) next; defer 2.3 mini-NLI. #000046 follow-up | 2026-05-11 | — | | #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — | -| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom (6 over-grounds + 8 mis-segments) = optional bigger-verifier-upgrade follow-up. `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed | 2026-05-11 | — | +| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom (2 over-grounds + 8 mis-segments after #000048 step 2.1 closed the 4 entity over-grounds) = #000048. `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 follow-up | 2026-05-11 | — | | #000045 | Prometheus-Σ Phase 3 sleep-sweep scheduler (gating ticket) | open · doc-only scaffold 2026-05-10; pins 8 governance parameters + 4 retrigger gates; opens implementation only after one retrigger fires | 2026-05-10 | — | | #000044 | AUTOCOUNT doc-drift discipline | closed · landed across `fc5ba50` / `03c0f6a` / `6c6defb` / `f5dbfab` / `3b30126` 2026-05-10 (mechanism + 4 metrics + 54 tags across 7 doc files; harness catches drift at test time, refresh is 60-second turnaround) | 2026-05-10 | — | | #000043 | Empirical tightening of T3 bound constants (#000036 Phase 2) | open · awaiting go/no-go (parks until v7 deployment surfaces measurement data) | 2026-05-10 | — | diff --git a/docs/qa-modes-bench.md b/docs/qa-modes-bench.md index 2dc09eb..6a95d55 100644 --- a/docs/qa-modes-bench.md +++ b/docs/qa-modes-bench.md @@ -442,3 +442,42 @@ Bench artifacts: `bench/qa_results/2026-05-11T13-42-38Z.{jsonl,md}` (before) · `bench/qa_results/2026-05-11T14-19-51Z.{jsonl,md}` (after). Full per-ticket detail: `docs/tickets/ticket-000046-harder-5sf-fixture-tier.md` §5 Phase 3. + + +### Addendum 6 — #000048 step 2.1 verifier entity-gate regression check (2026-05-11) + +`arborist/qa/verify.py` gained an entity salient-token-disagreement +gate (`_entity_salient_disagrees` + `_is_single_sentence`): in +`verify_quotes`' entity branch (proximity policy), the *weakest* +grounding — `not cluster AND len(verified) <= 1 AND single sentence` — +is declined (→ UNGROUNDED) when the answer asserts a > 4-char +capitalized content token (stopword-filtered) or a digit-number the +source lacks. Catches "Insulin was discovered by Alexander Fleming" +against "Penicillin was discovered by Alexander Fleming" (shared +"Alexander Fleming" matched; swapped subject "Insulin" trips the +gate). Lifts the `falsification-hard-v1.jsonl` rate 6/12 → 10/12 (the +4 HYBRID_ENTITY over-grounds: Insulin / Berlin / 1889 / Pacific). + +Before/after `make bench-qa` (n=3 × 75 questions × 3 modes = 675 +cells): + +| mode | STRICT-rate before → after | Δ | +|------|----------------------------|---| +| quote | 0.50 → 0.54 | +4pp | +| claim_lattice_pointer | 0.25 → 0.22 | −3pp | +| claim_lattice | 0.45 → 0.43 | −2pp | + +All within the 5-pp noise floor. Per-row diff (675 common cells, 30 +quote-mode rows changed audit_mode): **0 quote-mode rows demoted to +UNGROUNDED from the entity path** — the gate didn't fire on a single +legitimate QA answer in the whole bench. Every quote-mode transition +was LLM re-answer variance (`verifier quote→quote` with the verdict +flipping = a different answer text); the pointer/lattice deltas are +noise too (the gate is in `verify_quotes` / quote mode, not the +claim-lattice verifier). The gate is provably narrow on real traffic; +it ships. + +Bench artifacts: `bench/qa_results/2026-05-11T14-19-51Z.{jsonl,md}` +(before — HEAD's verify.py) · `bench/qa_results/2026-05-11T17-12-41Z.{jsonl,md}` +(after). Full per-ticket detail: `docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md` +§5 step 2.1. diff --git a/docs/tickets/ticket-000012-selection-consensus-protocol.md b/docs/tickets/ticket-000012-selection-consensus-protocol.md index a74bdd2..d5c1c9a 100644 --- a/docs/tickets/ticket-000012-selection-consensus-protocol.md +++ b/docs/tickets/ticket-000012-selection-consensus-protocol.md @@ -575,12 +575,13 @@ protocol must account for: `validator_diversity`, …), not by bench Δ-rate. The harder tier is **#000046 (closed 2026-05-11)**: two below-ceiling 5F packs (`falsification-hard-v1.jsonl`, `formulate-hard-v1.jsonl`) + - `verify_quotes`' paraphrase numeric-agreement gate, which lifted - the falsification-hard rate 4/12 → 6/12 on a *real* change - (bench-gated: `make bench-qa` n=3 before/after showed no - STRICT-rate regression on legit answers) → `fork_score`'s `γ·Δ5f` - went positive on it. So the bench Δ-rate *does* now carry signal + `verify_quotes`' paraphrase numeric-agreement gate + (#000048 step + 2.1) the entity salient-token-disagreement gate, which together + lifted the falsification-hard rate 4/12 → 10/12 on *real* changes + (each bench-gated: `make bench-qa` n=3 before/after showed no + STRICT-rate regression on legit answers — #000048 step 2.1's gate + fired on 0 QA answers in the whole bench) → `fork_score`'s `γ·Δ5f` + went positive on them. So the bench Δ-rate *does* now carry signal on the 5F/falsification axis, demonstrated end-to-end. Remaining - hard-pack headroom (6 over-grounds + 8 mis-segments) is an - optional bigger-verifier-upgrade follow-up, not a #000012 - blocker. + hard-pack headroom (now 2 over-grounds + 8 mis-segments — #000048 + steps 2.2 + 2.4) is an optional follow-up, not a #000012 blocker. diff --git a/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md b/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md index 63f5e74..a5c76e2 100644 --- a/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md +++ b/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md @@ -280,21 +280,18 @@ testable only once there's a below-ceiling baseline to bench against. ### Headroom (optional follow-up — not a #000046 blocker) -6 over-grounds remain in the falsification-hard pack (4 HYBRID_ENTITY -where the entity-proximity strategy matches on a shared proper noun -while the answer's other salient term is wrong; 2 STRICT_PARAPHRASE -where the false claim recombines source tokens into a different true -statement — "Mercury is the largest" — and lexical token-coverage -genuinely can't tell recombination from grounding), plus the 8 -mis-segments in the Formulate hard pack. Closing those needs a -bigger verifier upgrade — an order/dependency-aware match or a -contradiction check for `verify_quotes`, and sentence/clause -segmentation for `parse_pointer_claims` — **tracked as #000048** -(opened 2026-05-11; recommends an entity salient-token-disagreement -gate first, then `parse_pointer_claims` clause segmentation, then a -sequence-aware paraphrase match; each bench-gated). #000046 itself is -done: the below-ceiling baselines exist, a real surface improvement -lifted a rate, and ForkScore's bench-Δ went positive on it. +Tracked as **#000048** (opened 2026-05-11). As of #000048 step 2.1 +(2026-05-11) the falsification-hard pack is at 10/12 — the entity +salient-token-disagreement gate closed the 4 HYBRID_ENTITY +over-grounds. The remaining headroom: 2 STRICT_PARAPHRASE in the +falsification-hard pack (hard-003 Mercury, hard-005 Einstein — the +false claim recombines source tokens into a different true statement, +which lexical token-coverage can't tell from grounding; #000048 step +2.2 = sequence-aware paraphrase) + the 8 mis-segments in the Formulate +hard pack (line/bullet-only `parse_pointer_claims`; #000048 step 2.4 = +clause segmentation). #000046 itself is done: the below-ceiling +baselines exist, real surface improvements lifted a rate, and +ForkScore's bench-Δ went positive on them. --- diff --git a/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md b/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md index c7fe3a0..79545cd 100644 --- a/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md +++ b/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md @@ -1,6 +1,10 @@ # Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation -**Status:** open · awaiting go/no-go +**Status:** in progress · **step 2.1 landed 2026-05-11** (entity +salient-token-disagreement gate in `verify_quotes` — `_entity_salient_disagrees` ++ `_is_single_sentence`; lifts `falsification-hard` 6/12 → 10/12; +bench-gated, 0 gate-attributable QA shifts). Steps 2.4 (`parse_pointer_claims` +segmentation) + 2.2 (sequence-aware paraphrase) next per §3 / §5. **Opened:** 2026-05-11 **Scope:** Close the headroom #000046 left in the two below-ceiling 5F hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl` @@ -182,13 +186,68 @@ cost. ## 5. Status -**Open · awaiting go/no-go.** Doc-only proposal. Opens when fox picks -a starting point (recommendation: 2.1 / entity salient-token gate). -The implementation commit chain it gates is the §3 sequence: -2.1 → 2.4 → 2.2, each bench-gated, each re-pinning the hard-pack rate -+ documenting the `make bench-qa` (or Formulate-fixtures) before/after. -Closure: both `falsification-hard-v1.jsonl` and `formulate-hard-v1.jsonl` -at (or near) rate 1.0, or fox-decided "residue not worth it". +**In progress — step 2.1 landed 2026-05-11.** Steps 2.4 + 2.2 next +per §3. + +### Step 2.1 (landed 2026-05-11) — entity salient-token-disagreement gate + +- **Change:** `arborist/qa/verify.py` — `_entity_salient_disagrees(answer_text, + norm_ctx)` (a > 4-char Capitalized content token, stopword-filtered, + or a digit-number in the answer that's absent from the source) + + `_is_single_sentence(text)`. Gated in `verify_quotes`' entity branch + (proximity policy) in the *weakest-grounding* slot only — `not + cluster AND len(verified) <= 1 AND _is_single_sentence(answer_text) + AND _entity_salient_disagrees(...)` → `UNGROUNDED`. The narrow + caller-gate is what keeps a structured multi-claim summary (the + Matrix cast list — many entities, a tight cluster; the TMNT answer — + a numbered list with parenthetical nicknames the source omits) + untouched: model-added accurate detail in a real summary isn't a + contradiction, only the single-sentence-one-weak-match shape is. +- **Effect on the hard pack:** lifts `5f-fal-hard-001/002/006/008` + (Insulin / Berlin / 1889 / Pacific — the 4 HYBRID_ENTITY + over-grounds) to UNGROUNDED → **falsification-hard rate 6/12 → + 10/12 = 0.833**. The remaining 2 (hard-003 Mercury, hard-005 + Einstein — recombination STRICT_PARAPHRASE) are step 2.2's job. +- **Live-pack bookkeeping:** `5f-fal-live-003` (the *exact* gap #000046 + built its hard pack around — Insulin/Fleming) and `5f-fal-live-028` + (Eiffel/London) had `expected_reason` updated `HYBRID_ENTITY → + UNGROUNDED` — the live pack records what `verify_quotes` actually + does, and now it does the right thing. +- **Bench gate:** `make bench-qa` (n=3 × 75 questions × 3 modes = 675 + cells) after (`bench/qa_results/2026-05-11T17-12-41Z.md`) vs the + pre-step-2.1 baseline (`...T14-19-51Z.md`, which is HEAD's verify.py). + STRICT-rate: quote 0.50 → 0.54, pointer 0.25 → 0.22, lattice 0.45 → + 0.43 — all within the 5-pp noise floor. **Per-row diff: 0 quote-mode + rows demoted to UNGROUNDED from the entity path** — the gate didn't + fire on a single legitimate QA answer in the whole bench. Every + quote-mode `audit_mode` change was LLM re-answer variance + (`verifier quote→quote` with the verdict flipping = a different + answer); the pointer/lattice deltas are noise too (the gate is in + `verify_quotes` / quote mode, not the claim-lattice verifier). No + regression — the gate is provably narrow on real traffic. +- **Tests:** `tests/test_verify.py` — `test_is_single_sentence_helper`, + `test_entity_salient_disagrees_helper`, + `test_entity_path_swapped_subject_demoted_to_ungrounded`, + `test_entity_gate_narrow_multi_claim_summary_unaffected` (the + Matrix / TMNT / hybrid entity-path regression tests still pass — + pinned untouched); `tests/test_bench_batteries.py` + `test_5f_falsification_hard_pack_below_ceiling` re-pinned 6/12 → + 10/12; `test_fork_score_positive_gamma_5f_on_hard_falsification_improvement` + updated (positive γ·Δ5f on the real lift — possibly MARGINAL given + the ÷5 dilution; ACCEPT shown via a degraded-parent sub-scenario). + +### Still ahead + +- **Step 2.4** — `parse_pointer_claims` clause segmentation (closes + the 8 mis-segments in `formulate-hard-v1.jsonl`). Bench gate: the + Formulate fixtures (incl. `formulate-v1.jsonl` / `formulate-live-v1.jsonl`, + must not regress) + a QA smoke (`parse_pointer_claims` feeds the + claim-lattice modes). +- **Step 2.2** — sequence-aware paraphrase match (closes hard-003 + + hard-005). Bench-gated like 2.1; conservative threshold. +- **Closure:** both hard packs at (or near) rate 1.0, or fox-decided + "residue not worth it" — at which point this ticket closes (and + step 2.3 / mini-NLI stays a deferred maybe). --- diff --git a/tests/test_bench_batteries.py b/tests/test_bench_batteries.py index 888bf6f..a4e2059 100644 --- a/tests/test_bench_batteries.py +++ b/tests/test_bench_batteries.py @@ -701,26 +701,29 @@ def test_5f_formulate_hard_pack_below_ceiling(): def test_5f_falsification_hard_pack_below_ceiling(): """The hard pack is a deliberate below-ceiling baseline: 12 - near-misses whose correct verdict is UNGROUNDED. As of #000046 - closure (2026-05-11), verify_quotes' paraphrase numeric-agreement - gate catches the 2 magnitude/year over-grounds (50-vs-100, - 300-vs-300,000) → 6/12 pass; the remaining 6 over-grounds (4 - HYBRID_ENTITY, 2 STRICT_PARAPHRASE without a number) are the - headroom a future verifier upgrade closes. verify_quotes is + near-misses whose correct verdict is UNGROUNDED. As of #000048 + step 2.1 (2026-05-11): #000046's paraphrase numeric-agreement gate + catches the 2 magnitude/year over-grounds (50-vs-100, + 300-vs-300,000) + #000048's entity salient-token gate catches the + 4 HYBRID_ENTITY over-grounds (Insulin/Berlin/1889/Pacific) → 10/12 + pass; the remaining 2 are recombination STRICT_PARAPHRASE (hard-003 + Mercury, hard-005 Einstein — the false claim recombines source + tokens; lexical token-coverage can't tell recombination from + grounding), headroom for #000048 step 2.2. verify_quotes is deterministic, so this is stable; if it shifts, the pinned value here fires (loud signal — either the verifier got better/worse, or the pack drifted).""" res = b_5f.run_falsification(F5F / "falsification-hard-v1.jsonl") - assert res.pass_count == 6 - assert res.fail_count == 6 - assert res.metrics["error_detection_rate"] == pytest.approx(6 / 12) + assert res.pass_count == 10 + assert res.fail_count == 2 + assert res.metrics["error_detection_rate"] == pytest.approx(10 / 12) # All tasks route the live verifier; every fixture asserts UNGROUNDED. for t in res.per_task: assert t.detail["source"] == "live" assert t.detail["expected_reason"] == "UNGROUNDED" - # The 8 that fail do so by over-grounding (STRICT_/HYBRID_), never - # by the verifier saying UNGROUNDED — i.e. these are real - # over-ground misses, not "the verifier abstained". + # The 2 that fail do so by over-grounding (STRICT_/HYBRID_), never + # by the verifier saying UNGROUNDED — real over-ground misses, not + # "the verifier abstained". for t in res.per_task: if not t.passed: obs = set(t.detail["observed_violations"]) @@ -729,27 +732,37 @@ def test_5f_falsification_hard_pack_below_ceiling(): def test_fork_score_positive_gamma_5f_on_hard_falsification_improvement(): - """Worked example for #000046 — the loop: with a below-ceiling - parent (5f/falsification at the hard-pack rate) and a child that - lifts it to 1.0, fork_score's γ·Δ5f term goes strictly positive - and the verdict is ACCEPT. This is what a real verify_quotes - tightening would produce — the bench Δ-rate carrying signal it - can't carry while every pack is at ceiling.""" + """Worked example for #000046/#000048 — the loop: a real verifier + improvement that lifts the falsification-hard rate produces a + *positive* γ·Δ5f term in fork_score. Whether it clears SIGNAL_FLOOR + (→ ACCEPT) or not (→ MARGINAL) depends on the magnitude and the + ÷5 averaging dilution (#000047) — a small single-sub lift is + positive-but-MARGINAL by design; a big-enough one ACCEPTs. Both + shown here. The point: the bench Δ-rate carries signal it can't + carry while every pack is at ceiling.""" from arborist.substrate.fork_score import fork_score hard_rate = b_5f.run_falsification( F5F / "falsification-hard-v1.jsonl" ).metrics["error_detection_rate"] assert hard_rate < 1.0 - parent = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": hard_rate}}} - child = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": 1.0}}} - sf = fork_score(parent, child) - assert sf.breakdown["gamma_x_delta_5f"] > 0.0 - # Δ5f = (1.0 - hard_rate) averaged over the 5 5F subs (the other 4 - # subs absent ⇒ 0 - 0 = 0). With hard_rate ≈ 1/3 → Δ5f ≈ 0.133. - assert sf.breakdown["gamma_x_delta_5f"] == pytest.approx((1.0 - hard_rate) / 5) - assert sf.verdict == "ACCEPT" - assert not any(f.startswith(("REGRESSION_", "NEG_INF_")) for f in sf.flags) + # (a) A real lift from the current hard-pack rate to 1.0 — positive, + # possibly MARGINAL (the ÷5 dilution of a single-sub gain). + parent_a = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": hard_rate}}} + child_a = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": 1.0}}} + sf_a = fork_score(parent_a, child_a) + assert sf_a.breakdown["gamma_x_delta_5f"] > 0.0 + assert sf_a.breakdown["gamma_x_delta_5f"] == pytest.approx((1.0 - hard_rate) / 5) + assert sf_a.verdict in ("ACCEPT", "MARGINAL") + assert not any(f.startswith(("REGRESSION_", "NEG_INF_")) for f in sf_a.flags) + # (b) A bigger lift (degraded parent at 0.5 → child 1.0) clears the + # floor: Δ5f = 0.5/5 = 0.1 ≥ SIGNAL_FLOOR → ACCEPT. + parent_b = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": 0.5}}} + child_b = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": 1.0}}} + sf_b = fork_score(parent_b, child_b) + assert sf_b.breakdown["gamma_x_delta_5f"] == pytest.approx(0.5 / 5) + assert sf_b.verdict == "ACCEPT" + assert not any(f.startswith(("REGRESSION_", "NEG_INF_")) for f in sf_b.flags) def test_5f_falsification_covers_every_documented_motif(): diff --git a/tests/test_verify.py b/tests/test_verify.py index 0cb78bf..7ad2dfe 100644 --- a/tests/test_verify.py +++ b/tests/test_verify.py @@ -13,6 +13,9 @@ Tests cover the full trichotomy under each path plus regressions: from __future__ import annotations from arborist.qa.verify import ( + _entity_salient_disagrees, + _is_single_sentence, + _normalize, _numeric_signature, extract_claim_spans, extract_proper_nouns, @@ -615,3 +618,77 @@ def test_paraphrase_number_present_in_source_still_verifies(): # quote/span/entity may classify first; what matters is it's not # demoted to UNGROUNDED by a spurious numeric mismatch. assert v["audit_mode"] in ("STRICT", "HYBRID") + + +# --- #000048 step 2.1 — entity salient-token-disagreement gate ---- + + +def test_is_single_sentence_helper(): + assert _is_single_sentence("Insulin was discovered by Alexander Fleming.") + assert _is_single_sentence("The Eiffel Tower is in Berlin.") + assert _is_single_sentence("The Titanic sank in the Pacific Ocean.\n") # trailing ws stripped + assert not _is_single_sentence("One claim. Another claim.") + assert not _is_single_sentence("1. Leonardo\n2. Raphael\n3. Donatello") + assert not _is_single_sentence("Was it true? She asked.") + + +def test_entity_salient_disagrees_helper(): + # Swapped subject: "Insulin" is capitalized, >4 chars, absent from + # the source → disagrees. + assert _entity_salient_disagrees( + "Insulin was discovered by Alexander Fleming.", + _normalize("Penicillin was discovered by Alexander Fleming in 1928."), + ) + # Swapped city: "London" absent. + assert _entity_salient_disagrees( + "The Eiffel Tower is in London.", + _normalize("The Eiffel Tower is in Paris France."), + ) + # Swapped year: a digit-number absent. + assert _entity_salient_disagrees( + "The French Revolution began in 1889.", + _normalize("The French Revolution began in 1789, with the storming of the Bastille."), + ) + # Legit entity claim: every capitalized token is in the source → + # does NOT disagree (the prose differing — "stars in" vs "cast" — + # is fine; only capitalized tokens / numbers are checked). + assert not _entity_salient_disagrees( + "Keanu Reeves stars in The Matrix.", + _normalize("The Matrix cast: Keanu Reeves as Neo, Laurence Fishburne as Morpheus."), + ) + + +def test_entity_path_swapped_subject_demoted_to_ungrounded(): + """The entity strategy used to grant HYBRID for "Insulin was + discovered by Alexander Fleming" against "Penicillin was discovered + by Alexander Fleming" on the shared "Alexander Fleming". #000048's + salient gate (single short sentence, lone non-clustered match, a + capitalized token the source lacks) declines that grounding.""" + v = verify_quotes( + "Insulin was discovered by Alexander Fleming.", + "Penicillin was discovered by Alexander Fleming in 1928.", + ) + assert v["audit_mode"] == "UNGROUNDED" + v2 = verify_quotes( + "The Eiffel Tower is in Berlin.", + "The Eiffel Tower is in Paris, designed by Gustave Eiffel.", + ) + assert v2["audit_mode"] == "UNGROUNDED" + + +def test_entity_gate_narrow_multi_claim_summary_unaffected(): + """The gate fires only on the single-sentence-lone-match shape — a + structured multi-claim summary that the source partly grounds (a + cast list with model-added accurate detail) is NOT demoted. The + Matrix / TMNT regression tests above are the full version; this + pins the principle on a minimal case: two sentences, so even a + capitalized token the source lacks ("Morpheus") doesn't trip the + gate.""" + v = verify_quotes( + "The cast includes Keanu Reeves. Morpheus is played by Laurence Fishburne.", + "Cast of The Matrix: Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss.", + ) + # Not single-sentence → entity gate doesn't fire → still grounded + # at HYBRID (or STRICT if a cluster forms — either way, not + # UNGROUNDED-by-the-gate). + assert v["audit_mode"] in ("STRICT", "HYBRID")