#000048 step 2.1 — verify_quotes entity salient-token-disagreement gate
Closes the 4 HYBRID_ENTITY over-grounds #000046 left in falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a multi-word proper noun matches the source — but "Insulin was discovered by Alexander Fleming" against "Penicillin was discovered by Alexander Fleming" matches on the shared "Alexander Fleming" while the swapped subject "Insulin" (the falsehood) is ignored. arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx) flags a >4-char Capitalized content token (stopword-filtered) or a digit-number in the answer absent from the source. _is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in verify_quotes' entity branch (proximity policy) in the weakest-grounding slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is what keeps a structured multi-claim summary untouched — the Matrix cast list (many entities, a tight cluster) and the TMNT answer (a numbered list with parenthetical nicknames the source omits): model-added accurate detail in a real summary isn't a contradiction, only the single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid entity-path regression tests still pass, pinned untouched. Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin / 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard pack around) and 5f-fal-live-028 — had expected_reason updated HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE recombinations (Mercury, Einstein — step 2.2) + 8 Formulate mis-segments (step 2.4). Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after (bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline (...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54, pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise floor. Per-row diff (675 common cells, 30 quote-mode rows changed audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity path — the gate fired on 0 legitimate QA answers in the whole bench. Every transition was LLM re-answer variance (verifier quote→quote with the verdict flipping); pointer/lattice deltas are noise too (the gate is in verify_quotes / quote mode, not the claim-lattice verifier). No regression — the gate is provably narrow on real traffic. Summarized in qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1. Tests: 4 new in test_verify.py (_is_single_sentence helper, _entity_salient_disagrees helper, swapped-subject → UNGROUNDED, gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated (positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5 dilution; ACCEPT via a degraded-parent sub-scenario). make test 2343 passed, 28 skipped. #000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md / Makefile / fixture _meta + notes / baseline JSON updated.
This commit is contained in:
parent
06f5a11651
commit
9899a33b7b
12 changed files with 340 additions and 91 deletions
16
Makefile
16
Makefile
|
|
@ -493,13 +493,15 @@ bench-5f-live: bench-5f-formulate-live bench-5f-feedback-loop-live bench-5f-func
|
|||
|
||||
# #000046 Phase 1 — HARD live-path Falsification tier. Every fixture is
|
||||
# a near-miss the correct verdict is UNGROUNDED on; verify_quotes
|
||||
# over-grounds 6 of 12 today → rate 6/12 = 0.5 at HEAD (the paraphrase
|
||||
# numeric-agreement gate, #000046 closure, caught the other 2). A real
|
||||
# below-ceiling baseline. NOT part of bench-5f / bench-fork-baseline.
|
||||
# bench-fork-baseline-hard pins the rate so a future verifier upgrade
|
||||
# shows up as a larger positive gamma*Delta5f term.
|
||||
# over-grounds 2 of 12 today → rate 10/12 = 0.833 at HEAD (#000046's
|
||||
# paraphrase numeric gate caught 2, #000048 step 2.1's entity
|
||||
# salient-token gate caught 4 more; the last 2 are recombination
|
||||
# STRICT_PARAPHRASE — #000048 step 2.2). A real below-ceiling baseline.
|
||||
# NOT part of bench-5f / bench-fork-baseline. bench-fork-baseline-hard
|
||||
# pins the rate so a future verifier upgrade shows up as a larger
|
||||
# positive gamma*Delta5f term.
|
||||
FORK_PARENT_HARD ?= bench/results/baseline-falsification-hard.json
|
||||
# The hard pack has its expected failures (6/12 over-grounds), so the
|
||||
# The hard pack has its expected failures (2/12 over-grounds), so the
|
||||
# runner exits 1 — that's the below-ceiling baseline working as
|
||||
# designed, not a build error. `|| true` keeps the make target green;
|
||||
# the JSON / report is written regardless (the --out write precedes
|
||||
|
|
@ -511,7 +513,7 @@ bench-fork-baseline-hard: bootstrap ## #000046 — pin the below-ceiling hard-Fa
|
|||
@mkdir -p bench/results
|
||||
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5f --sub falsification \
|
||||
--fixtures bench/fixtures/5f/falsification-hard-v1.jsonl --out $(FORK_PARENT_HARD) || true
|
||||
@echo ">> below-ceiling parent pinned: $(FORK_PARENT_HARD) (6/12 over-grounds expected → runner exits 1; the JSON is written)"
|
||||
@echo ">> below-ceiling parent pinned: $(FORK_PARENT_HARD) (2/12 over-grounds expected → runner exits 1; the JSON is written)"
|
||||
# #000046 Phase 2 — HARD Formulate tier. Prose that parse_pointer_claims
|
||||
# should segment a particular way; the line/bullet-based parser
|
||||
# mis-segments 8 of 12 (merges multi-claim lines, splits wrapped
|
||||
|
|
|
|||
|
|
@ -392,6 +392,53 @@ def _numeric_signature(text: str) -> frozenset[str]:
|
|||
return frozenset(m.group(0).replace(",", "") for m in _DIGIT_RUN_RE.finditer(text))
|
||||
|
||||
|
||||
_SENTENCE_BREAK_RE = re.compile(r"[.!?](?=\s)")
|
||||
|
||||
|
||||
def _is_single_sentence(text: str) -> bool:
|
||||
"""One sentence — no internal ``. `` / ``! `` / ``? `` break (a trailing
|
||||
terminator with nothing after it is fine). A numbered/bulleted list
|
||||
("1. … 2. …") or a multi-sentence summary is not single; "Insulin
|
||||
was discovered by Alexander Fleming." is. Conservative: a borderline
|
||||
abbreviation ("8,849 m. tall.") reads as not-single, so the gate it
|
||||
guards simply doesn't fire — the safe direction."""
|
||||
return not _SENTENCE_BREAK_RE.search(text.strip())
|
||||
|
||||
|
||||
def _entity_salient_disagrees(answer_text: str, norm_ctx: str) -> bool:
|
||||
"""Entity-strategy gate (ticket #000048): does the answer assert a
|
||||
*salient near-proper-noun token* — a > 4-char Capitalized content
|
||||
token, stopword-filtered — or a digit-number that the source lacks?
|
||||
|
||||
Used in :func:`verify_quotes` *only* on the weakest entity grounding
|
||||
— a single short sentence whose lone matching multi-word proper noun
|
||||
is not part of a cluster (caller checks ``len(verified) <= 1`` and
|
||||
:func:`_is_single_sentence`). There, "Insulin was discovered by
|
||||
Alexander Fleming" against "Penicillin was discovered by Alexander
|
||||
Fleming" matches on the shared "Alexander Fleming" while the
|
||||
*swapped subject* ("Insulin", capitalized, absent from the source)
|
||||
is exactly the falsehood — so we decline the grounding. The narrow
|
||||
caller-gate is what keeps a structured multi-claim summary (the
|
||||
Matrix cast list — many entities, a tight cluster — or the TMNT
|
||||
answer — a numbered list with parenthetical nicknames the source
|
||||
omits) untouched: model-added accurate detail in a real summary
|
||||
isn't a contradiction, and only the single-sentence-one-weak-match
|
||||
shape is.
|
||||
"""
|
||||
if not (_numeric_signature(_normalize(answer_text)) <= _numeric_signature(norm_ctx)):
|
||||
return True
|
||||
for raw in answer_text.split():
|
||||
core = raw.strip(_TOKEN_PUNCT_STRIP)
|
||||
if len(core) <= 4 or not core[0].isupper():
|
||||
continue
|
||||
low = core.lower()
|
||||
if low in _ENGLISH_STOPWORDS:
|
||||
continue
|
||||
if low not in norm_ctx:
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# Common English stopwords of length >= 4 chars. Hand-curated rather
|
||||
# than imported from NLTK to keep arborist dependency-light and the
|
||||
# behavior pinned to a known set. Tokens are normalized form
|
||||
|
|
@ -657,6 +704,20 @@ def verify_quotes(
|
|||
mode = "STRICT"
|
||||
elif cluster:
|
||||
mode = "HYBRID"
|
||||
elif (
|
||||
# #000048: weakest grounding — a single short
|
||||
# sentence whose lone matching multi-word proper
|
||||
# noun isn't part of a cluster. If the claim
|
||||
# asserts a salient capitalized token / number the
|
||||
# source lacks (a swapped subject / city / ocean /
|
||||
# year — "Insulin was discovered by Alexander
|
||||
# Fleming" vs "Penicillin was…"), the shared
|
||||
# proper noun doesn't ground it → UNGROUNDED.
|
||||
len(verified) <= 1
|
||||
and _is_single_sentence(answer_text)
|
||||
and _entity_salient_disagrees(answer_text, norm_ctx)
|
||||
):
|
||||
mode = "UNGROUNDED"
|
||||
elif verified:
|
||||
mode = "HYBRID"
|
||||
else:
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
{"_meta": {"battery": "5f", "sub_battery": "falsification", "version": "v1", "task_count": 12, "notes": "#000046 — HARD live-path Falsification tier. Every fixture is a near-miss: the claim is FALSE / unsupported, so the correct verdict is UNGROUNDED, and that is what `expected_reason` records. As of #000046 closure (2026-05-11), verify_quotes' paraphrase numeric-agreement gate catches the 2 magnitude/year over-grounds (50-vs-100, 300-vs-300,000) → those now correctly land UNGROUNDED. The remaining 6 still over-ground: 4 via the entity-proximity strategy (matches on a shared proper noun while the answer's other salient term is wrong → HYBRID_ENTITY) and 2 via paraphrase token-coverage on a recombined-no-number claim (\"Mercury is the largest\" vs source \"Jupiter is the largest; Mercury is the smallest\" — all content tokens present, just recombined → STRICT_PARAPHRASE). The other 4 the verifier always handled correctly. Rate at HEAD = 6/12 = 0.5 — a real below-ceiling baseline. A further verifier upgrade (order/dependency-aware match or a contradiction check, vs the lexical token-coverage that can't see recombination) lifts the rate toward 1.0 → a larger positive gamma*Delta5f term in fork_score. verify_quotes is pure-lexical / deterministic, so the rate is stable. NOT part of `make bench-5f`, `make bench-5s5t5f`, `bench-fork-baseline`, or `runner --all`; pinned separately via `make bench-fork-baseline-hard`. Pre-documented gap this builds around: 5f-fal-live-003 in falsification-live-v1.jsonl. See ticket #000046."}}
|
||||
{"id": "5f-fal-hard-001", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Insulin was discovered by Alexander Fleming.", "context": "Penicillin was discovered by Alexander Fleming in 1928.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: HYBRID_ENTITY — substance swapped, but entity strategy matches on shared 'Alexander Fleming'."}
|
||||
{"id": "5f-fal-hard-002", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Eiffel Tower is in Berlin.", "context": "The Eiffel Tower is in Paris, designed by Gustave Eiffel.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: HYBRID_ENTITY — city swapped, entity strategy matches on 'Eiffel Tower'/'Eiffel'."}
|
||||
{"_meta": {"battery": "5f", "sub_battery": "falsification", "version": "v1", "task_count": 12, "notes": "#000046 — HARD live-path Falsification tier. Every fixture is a near-miss: the claim is FALSE / unsupported, so the correct verdict is UNGROUNDED, and that is what `expected_reason` records. As of #000048 step 2.1 (2026-05-11): #000046's paraphrase numeric-agreement gate catches the 2 magnitude/year over-grounds (50-vs-100, 300-vs-300,000) AND #000048's entity salient-token-disagreement gate catches the 4 HYBRID_ENTITY over-grounds (Insulin/Berlin/1889/Pacific — single short sentence, lone non-clustered proper-noun match, a >4-char capitalized token or number the source lacks) → all 6 now correctly land UNGROUNDED. The remaining 2 over-ground via paraphrase token-coverage on a recombined-no-number claim (\"Mercury is the largest\" vs source \"Jupiter is the largest; Mercury is the smallest\" — all content tokens present, just recombined → STRICT_PARAPHRASE; #000048 step 2.2 is the sequence-aware match that closes those). The other 4 the verifier always handled correctly. Rate at HEAD = 10/12 = 0.833 — a real below-ceiling baseline. verify_quotes is pure-lexical / deterministic, so the rate is stable. NOT part of `make bench-5f`, `make bench-5s5t5f`, `bench-fork-baseline`, or `runner --all`; pinned separately via `make bench-fork-baseline-hard`. Pre-documented gap this builds around: 5f-fal-live-003 in falsification-live-v1.jsonl. See tickets #000046 (closed) + #000048."}}
|
||||
{"id": "5f-fal-hard-001", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Insulin was discovered by Alexander Fleming.", "context": "Penicillin was discovered by Alexander Fleming in 1928.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000048 step 2.1): entity match on shared 'Alexander Fleming' but the swapped subject 'Insulin' (capitalized, absent from source) trips the salient gate → UNGROUNDED. PASSES today."}
|
||||
{"id": "5f-fal-hard-002", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Eiffel Tower is in Berlin.", "context": "The Eiffel Tower is in Paris, designed by Gustave Eiffel.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000048 step 2.1): entity match on 'Eiffel Tower' but the swapped city 'Berlin' (absent from source) trips the salient gate → UNGROUNDED. PASSES today."}
|
||||
{"id": "5f-fal-hard-003", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Mercury is the largest planet in the Solar System.", "context": "Jupiter is the largest planet in the Solar System; Mercury is the smallest.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: STRICT_PARAPHRASE — verifier claims STRICT on a false statement; paraphrase token-coverage matches 'largest planet in the Solar System'."}
|
||||
{"id": "5f-fal-hard-004", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Water boils at 50 degrees Celsius at sea level.", "context": "Water boils at 100 degrees Celsius at sea level.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000046 closure): the paraphrase numeric-agreement gate sees 50 ∉ {100} → UNGROUNDED. PASSES today."}
|
||||
{"id": "5f-fal-hard-005", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Albert Einstein developed the theory of evolution.", "context": "Albert Einstein developed the theory of relativity; Charles Darwin developed evolution.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: STRICT_PARAPHRASE — 'evolution' vs 'relativity' swapped; 'Albert Einstein developed the theory of' token-covers + 'evolution' appears in context (attributed to Darwin)."}
|
||||
{"id": "5f-fal-hard-006", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The French Revolution began in 1889.", "context": "The French Revolution began in 1789, with the storming of the Bastille.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: HYBRID_ENTITY — year off by a century (1889 vs 1789); entity strategy matches on 'French Revolution'."}
|
||||
{"id": "5f-fal-hard-006", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The French Revolution began in 1889.", "context": "The French Revolution began in 1789, with the storming of the Bastille.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000048 step 2.1): entity match on 'French Revolution' but the swapped year 1889 ∉ {1789} trips the salient gate's numeric arm → UNGROUNDED. PASSES today."}
|
||||
{"id": "5f-fal-hard-007", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The speed of light is approximately 300 kilometers per second.", "context": "The speed of light is approximately 300,000 kilometers per second in a vacuum.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000046 closure): the paraphrase numeric-agreement gate sees 300 ∉ {300000} (← '300,000' comma-stripped) → UNGROUNDED. PASSES today."}
|
||||
{"id": "5f-fal-hard-008", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Titanic sank in the Pacific Ocean.", "context": "The Titanic sank in the North Atlantic Ocean in 1912.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "OVER-GROUND at HEAD: HYBRID_ENTITY — ocean swapped (Pacific vs North Atlantic); entity strategy matches on 'Titanic'."}
|
||||
{"id": "5f-fal-hard-008", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Titanic sank in the Pacific Ocean.", "context": "The Titanic sank in the North Atlantic Ocean in 1912.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Caught (#000048 step 2.1): entity match on 'The Titanic' but the swapped 'Pacific' (absent from source — it says 'North Atlantic Ocean') trips the salient gate → UNGROUNDED. PASSES today."}
|
||||
{"id": "5f-fal-hard-009", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Shakespeare wrote War and Peace.", "context": "Tolstoy wrote War and Peace; Shakespeare wrote Hamlet.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Correctly UNGROUNDED at HEAD — verifier does NOT over-ground this near-miss. PASSES today; kept as headroom marker (a regression that started over-grounding this would drop the rate)."}
|
||||
{"id": "5f-fal-hard-010", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The mitochondria is the powerhouse of the nucleus.", "context": "The mitochondria is the powerhouse of the cell.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Correctly UNGROUNDED at HEAD — 'nucleus' vs 'cell' swap is caught. PASSES today."}
|
||||
{"id": "5f-fal-hard-011", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The capital of Australia is Sydney.", "context": "The capital of Australia is Canberra; Sydney is the largest city.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Correctly UNGROUNDED at HEAD. PASSES today."}
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
{"_meta": {"battery": "5f", "sub_battery": "falsification", "version": "v1", "task_count": 50, "notes": "Phase 1b.2: answer_text + context routes through arborist.qa.verify.verify_quotes (real verifier surface). Live signals: UNGROUNDED, STRICT_<method>, HYBRID_<method>, UNVERIFIED_QUOTE. Phase 1d (2026-05-09): expanded 30→50."}}
|
||||
{"id": "5f-fal-live-001", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Random unsupported claim about nothing.", "context": "Some completely unrelated context.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"}
|
||||
{"id": "5f-fal-live-002", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "K2 is the tallest mountain.", "context": "Mount Everest is the tallest mountain at 8849 meters.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"}
|
||||
{"id": "5f-fal-live-003", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Insulin was discovered by Alexander Fleming.", "context": "Penicillin was discovered by Alexander Fleming in 1928.", "expected_reason": "HYBRID_ENTITY", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "verify_quotes' entity strategy matches on shared 'Alexander Fleming' entity even though the substance is swapped. Documents a known soft-signal gap; the title-relevance hard check + claim-lattice verifier catch this case in production."}
|
||||
{"id": "5f-fal-live-003", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Insulin was discovered by Alexander Fleming.", "context": "Penicillin was discovered by Alexander Fleming in 1928.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Was HYBRID_ENTITY — the entity strategy matched the shared 'Alexander Fleming' while ignoring the swapped subject 'Insulin' (absent from the source). #000048 step 2.1's entity salient-token-disagreement gate (single short sentence, lone non-clustered match, a >4-char capitalized token the source lacks) now declines that grounding → UNGROUNDED. This is the soft-signal gap #000046 built its falsification-hard pack around; #000048 closed it."}
|
||||
{"id": "5f-fal-live-004", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Water boils at 100 degrees Celsius at sea level.", "context": "Water boils at 100 degrees Celsius at sea level.", "expected_reason": "STRICT_SPAN", "verifier_method_root": "verify_quotes-v1", "expected": "pass"}
|
||||
{"id": "5f-fal-live-005", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Hydrogen has atomic number 1.", "context": "Hydrogen has atomic number 1 and is the lightest chemical element.", "expected_reason": "STRICT_SPAN", "verifier_method_root": "verify_quotes-v1", "expected": "pass"}
|
||||
{"id": "5f-fal-live-006", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "There are nine planets in the solar system.", "context": "There are eight planets in the solar system since Pluto's reclassification.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"}
|
||||
|
|
@ -26,7 +26,7 @@
|
|||
{"id": "5f-fal-live-025", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Pluto is a planet.", "context": "Pluto was reclassified as a dwarf planet in 2006.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"}
|
||||
{"id": "5f-fal-live-026", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Nile is short.", "context": "The Nile is one of the longest rivers in the world at about 6650 kilometers.", "expected_reason": "HYBRID_ENTITY", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Live verifier signal: verify_quotes returns HYBRID_ENTITY for this (answer, context) pair. Captured here as a behavior-pinning fixture; divergence from synthetic expectation is the value of live mode."}
|
||||
{"id": "5f-fal-live-027", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Sputnik 1 was launched in 1962.", "context": "Sputnik 1 was launched in 1957.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"}
|
||||
{"id": "5f-fal-live-028", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Eiffel Tower is in London.", "context": "The Eiffel Tower is in Paris France.", "expected_reason": "HYBRID_ENTITY", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Live verifier signal: verify_quotes returns HYBRID_ENTITY for this (answer, context) pair. Captured here as a behavior-pinning fixture; divergence from synthetic expectation is the value of live mode."}
|
||||
{"id": "5f-fal-live-028", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Eiffel Tower is in London.", "context": "The Eiffel Tower is in Paris France.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass", "note": "Was HYBRID_ENTITY — the entity strategy matched the shared 'Eiffel Tower' while ignoring the swapped city 'London' (absent from the source). #000048 step 2.1's entity salient-token-disagreement gate now declines that grounding → UNGROUNDED."}
|
||||
{"id": "5f-fal-live-029", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Pride and Prejudice was published in 1813.", "context": "Pride and Prejudice was published in 1813.", "expected_reason": "STRICT_SPAN", "verifier_method_root": "verify_quotes-v1", "expected": "pass"}
|
||||
{"id": "5f-fal-live-030", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The capital of France is Berlin.", "context": "The capital of France is Paris.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"}
|
||||
{"id": "5f-fal-live-031", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Random unsupported claim about nothing. (variant 31)", "context": "Some context.", "expected_reason": "UNGROUNDED", "verifier_method_root": "verify_quotes-v1", "expected": "pass"}
|
||||
|
|
|
|||
|
|
@ -5,36 +5,36 @@
|
|||
"battery": "5f",
|
||||
"sub_battery": "falsification",
|
||||
"fixture_path": "bench/fixtures/5f/falsification-hard-v1.jsonl",
|
||||
"fixture_digest": "519524e6ceab694d14971832e4cd676c952f032bb05580637d9185c2b6e22096",
|
||||
"pass_count": 6,
|
||||
"fail_count": 6,
|
||||
"fixture_digest": "8d21958a7e4218681678d1b2fa0ff7e551bbb2e2dfe0eb00c6ef947e1826ad0c",
|
||||
"pass_count": 10,
|
||||
"fail_count": 2,
|
||||
"metrics": {
|
||||
"error_detection_rate": 0.5
|
||||
"error_detection_rate": 0.8333333333333334
|
||||
},
|
||||
"per_task": [
|
||||
{
|
||||
"task_id": "5f-fal-hard-001",
|
||||
"passed": false,
|
||||
"passed": true,
|
||||
"detail": {
|
||||
"source": "live",
|
||||
"expected_reason": "UNGROUNDED",
|
||||
"observed_violations": [
|
||||
"HYBRID_ENTITY"
|
||||
"UNGROUNDED"
|
||||
],
|
||||
"detected": false,
|
||||
"detected": true,
|
||||
"verifier_method_root": "verify_quotes-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"task_id": "5f-fal-hard-002",
|
||||
"passed": false,
|
||||
"passed": true,
|
||||
"detail": {
|
||||
"source": "live",
|
||||
"expected_reason": "UNGROUNDED",
|
||||
"observed_violations": [
|
||||
"HYBRID_ENTITY"
|
||||
"UNGROUNDED"
|
||||
],
|
||||
"detected": false,
|
||||
"detected": true,
|
||||
"verifier_method_root": "verify_quotes-v1"
|
||||
}
|
||||
},
|
||||
|
|
@ -79,14 +79,14 @@
|
|||
},
|
||||
{
|
||||
"task_id": "5f-fal-hard-006",
|
||||
"passed": false,
|
||||
"passed": true,
|
||||
"detail": {
|
||||
"source": "live",
|
||||
"expected_reason": "UNGROUNDED",
|
||||
"observed_violations": [
|
||||
"HYBRID_ENTITY"
|
||||
"UNGROUNDED"
|
||||
],
|
||||
"detected": false,
|
||||
"detected": true,
|
||||
"verifier_method_root": "verify_quotes-v1"
|
||||
}
|
||||
},
|
||||
|
|
@ -105,15 +105,15 @@
|
|||
},
|
||||
{
|
||||
"task_id": "5f-fal-hard-008",
|
||||
"passed": false,
|
||||
"passed": true,
|
||||
"detail": {
|
||||
"source": "live",
|
||||
"expected_reason": "UNGROUNDED",
|
||||
"observed_violations": [
|
||||
"HYBRID_ENTITY",
|
||||
"UNGROUNDED",
|
||||
"UNVERIFIED_QUOTE"
|
||||
],
|
||||
"detected": false,
|
||||
"detected": true,
|
||||
"verifier_method_root": "verify_quotes-v1"
|
||||
}
|
||||
},
|
||||
|
|
@ -171,7 +171,7 @@
|
|||
}
|
||||
],
|
||||
"runtime_digest": "da5203214469cc5e682dd8c9edfcb0c95a758e1ba78567b4577b2c598c307951",
|
||||
"timestamp": 1778507146
|
||||
"timestamp": 1778519494
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -92,9 +92,9 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | open · awaiting go/no-go · doc-only; closes the headroom #000046 left (6 over-grounds in `falsification-hard` + 8 mis-segments in `formulate-hard`). Recommends: 2.1 entity salient-token-disagreement gate (mirrors #000046's numeric gate) → 2.4 `parse_pointer_claims` clause segmentation → 2.2 sequence-aware paraphrase; defer 2.3 mini-NLI. Each bench-gated. #000046 follow-up | 2026-05-11 | — |
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | in progress · **step 2.1 landed 2026-05-11** — `verify_quotes` entity salient-token-disagreement gate (`_entity_salient_disagrees` + `_is_single_sentence`; demotes a single short sentence whose lone non-clustered proper-noun match co-occurs with a >4-char capitalized token / number the source lacks) → `falsification-hard` 6/12 → 10/12; bench-gated (`make bench-qa` n=3×75×3 — 0 gate-attributable QA shifts, deltas within the 5-pp floor); 2 live-pack fixtures' `expected_reason` updated HYBRID_ENTITY→UNGROUNDED. Steps 2.4 (`parse_pointer_claims` clause segmentation) + 2.2 (sequence-aware paraphrase) next; defer 2.3 mini-NLI. #000046 follow-up | 2026-05-11 | — |
|
||||
| #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom (6 over-grounds + 8 mis-segments) = optional bigger-verifier-upgrade follow-up. `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom (2 over-grounds + 8 mis-segments after #000048 step 2.1 closed the 4 entity over-grounds) = #000048. `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 follow-up | 2026-05-11 | — |
|
||||
| #000045 | Prometheus-Σ Phase 3 sleep-sweep scheduler (gating ticket) | open · doc-only scaffold 2026-05-10; pins 8 governance parameters + 4 retrigger gates; opens implementation only after one retrigger fires | 2026-05-10 | — |
|
||||
| #000044 | AUTOCOUNT doc-drift discipline | closed · landed across `fc5ba50` / `03c0f6a` / `6c6defb` / `f5dbfab` / `3b30126` 2026-05-10 (mechanism + 4 metrics + 54 tags across 7 doc files; harness catches drift at test time, refresh is 60-second turnaround) | 2026-05-10 | — |
|
||||
| #000043 | Empirical tightening of T3 bound constants (#000036 Phase 2) | open · awaiting go/no-go (parks until v7 deployment surfaces measurement data) | 2026-05-10 | — |
|
||||
|
|
|
|||
|
|
@ -442,3 +442,42 @@ Bench artifacts: `bench/qa_results/2026-05-11T13-42-38Z.{jsonl,md}`
|
|||
(before) · `bench/qa_results/2026-05-11T14-19-51Z.{jsonl,md}` (after).
|
||||
Full per-ticket detail: `docs/tickets/ticket-000046-harder-5sf-fixture-tier.md`
|
||||
§5 Phase 3.
|
||||
|
||||
|
||||
### Addendum 6 — #000048 step 2.1 verifier entity-gate regression check (2026-05-11)
|
||||
|
||||
`arborist/qa/verify.py` gained an entity salient-token-disagreement
|
||||
gate (`_entity_salient_disagrees` + `_is_single_sentence`): in
|
||||
`verify_quotes`' entity branch (proximity policy), the *weakest*
|
||||
grounding — `not cluster AND len(verified) <= 1 AND single sentence` —
|
||||
is declined (→ UNGROUNDED) when the answer asserts a > 4-char
|
||||
capitalized content token (stopword-filtered) or a digit-number the
|
||||
source lacks. Catches "Insulin was discovered by Alexander Fleming"
|
||||
against "Penicillin was discovered by Alexander Fleming" (shared
|
||||
"Alexander Fleming" matched; swapped subject "Insulin" trips the
|
||||
gate). Lifts the `falsification-hard-v1.jsonl` rate 6/12 → 10/12 (the
|
||||
4 HYBRID_ENTITY over-grounds: Insulin / Berlin / 1889 / Pacific).
|
||||
|
||||
Before/after `make bench-qa` (n=3 × 75 questions × 3 modes = 675
|
||||
cells):
|
||||
|
||||
| mode | STRICT-rate before → after | Δ |
|
||||
|------|----------------------------|---|
|
||||
| quote | 0.50 → 0.54 | +4pp |
|
||||
| claim_lattice_pointer | 0.25 → 0.22 | −3pp |
|
||||
| claim_lattice | 0.45 → 0.43 | −2pp |
|
||||
|
||||
All within the 5-pp noise floor. Per-row diff (675 common cells, 30
|
||||
quote-mode rows changed audit_mode): **0 quote-mode rows demoted to
|
||||
UNGROUNDED from the entity path** — the gate didn't fire on a single
|
||||
legitimate QA answer in the whole bench. Every quote-mode transition
|
||||
was LLM re-answer variance (`verifier quote→quote` with the verdict
|
||||
flipping = a different answer text); the pointer/lattice deltas are
|
||||
noise too (the gate is in `verify_quotes` / quote mode, not the
|
||||
claim-lattice verifier). The gate is provably narrow on real traffic;
|
||||
it ships.
|
||||
|
||||
Bench artifacts: `bench/qa_results/2026-05-11T14-19-51Z.{jsonl,md}`
|
||||
(before — HEAD's verify.py) · `bench/qa_results/2026-05-11T17-12-41Z.{jsonl,md}`
|
||||
(after). Full per-ticket detail: `docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md`
|
||||
§5 step 2.1.
|
||||
|
|
|
|||
|
|
@ -575,12 +575,13 @@ protocol must account for:
|
|||
`validator_diversity`, …), not by bench Δ-rate. The harder tier
|
||||
is **#000046 (closed 2026-05-11)**: two below-ceiling 5F packs
|
||||
(`falsification-hard-v1.jsonl`, `formulate-hard-v1.jsonl`) +
|
||||
`verify_quotes`' paraphrase numeric-agreement gate, which lifted
|
||||
the falsification-hard rate 4/12 → 6/12 on a *real* change
|
||||
(bench-gated: `make bench-qa` n=3 before/after showed no
|
||||
STRICT-rate regression on legit answers) → `fork_score`'s `γ·Δ5f`
|
||||
went positive on it. So the bench Δ-rate *does* now carry signal
|
||||
`verify_quotes`' paraphrase numeric-agreement gate + (#000048 step
|
||||
2.1) the entity salient-token-disagreement gate, which together
|
||||
lifted the falsification-hard rate 4/12 → 10/12 on *real* changes
|
||||
(each bench-gated: `make bench-qa` n=3 before/after showed no
|
||||
STRICT-rate regression on legit answers — #000048 step 2.1's gate
|
||||
fired on 0 QA answers in the whole bench) → `fork_score`'s `γ·Δ5f`
|
||||
went positive on them. So the bench Δ-rate *does* now carry signal
|
||||
on the 5F/falsification axis, demonstrated end-to-end. Remaining
|
||||
hard-pack headroom (6 over-grounds + 8 mis-segments) is an
|
||||
optional bigger-verifier-upgrade follow-up, not a #000012
|
||||
blocker.
|
||||
hard-pack headroom (now 2 over-grounds + 8 mis-segments — #000048
|
||||
steps 2.2 + 2.4) is an optional follow-up, not a #000012 blocker.
|
||||
|
|
|
|||
|
|
@ -280,21 +280,18 @@ testable only once there's a below-ceiling baseline to bench against.
|
|||
|
||||
### Headroom (optional follow-up — not a #000046 blocker)
|
||||
|
||||
6 over-grounds remain in the falsification-hard pack (4 HYBRID_ENTITY
|
||||
where the entity-proximity strategy matches on a shared proper noun
|
||||
while the answer's other salient term is wrong; 2 STRICT_PARAPHRASE
|
||||
where the false claim recombines source tokens into a different true
|
||||
statement — "Mercury is the largest" — and lexical token-coverage
|
||||
genuinely can't tell recombination from grounding), plus the 8
|
||||
mis-segments in the Formulate hard pack. Closing those needs a
|
||||
bigger verifier upgrade — an order/dependency-aware match or a
|
||||
contradiction check for `verify_quotes`, and sentence/clause
|
||||
segmentation for `parse_pointer_claims` — **tracked as #000048**
|
||||
(opened 2026-05-11; recommends an entity salient-token-disagreement
|
||||
gate first, then `parse_pointer_claims` clause segmentation, then a
|
||||
sequence-aware paraphrase match; each bench-gated). #000046 itself is
|
||||
done: the below-ceiling baselines exist, a real surface improvement
|
||||
lifted a rate, and ForkScore's bench-Δ went positive on it.
|
||||
Tracked as **#000048** (opened 2026-05-11). As of #000048 step 2.1
|
||||
(2026-05-11) the falsification-hard pack is at 10/12 — the entity
|
||||
salient-token-disagreement gate closed the 4 HYBRID_ENTITY
|
||||
over-grounds. The remaining headroom: 2 STRICT_PARAPHRASE in the
|
||||
falsification-hard pack (hard-003 Mercury, hard-005 Einstein — the
|
||||
false claim recombines source tokens into a different true statement,
|
||||
which lexical token-coverage can't tell from grounding; #000048 step
|
||||
2.2 = sequence-aware paraphrase) + the 8 mis-segments in the Formulate
|
||||
hard pack (line/bullet-only `parse_pointer_claims`; #000048 step 2.4 =
|
||||
clause segmentation). #000046 itself is done: the below-ceiling
|
||||
baselines exist, real surface improvements lifted a rate, and
|
||||
ForkScore's bench-Δ went positive on them.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,10 @@
|
|||
# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Status:** in progress · **step 2.1 landed 2026-05-11** (entity
|
||||
salient-token-disagreement gate in `verify_quotes` — `_entity_salient_disagrees`
|
||||
+ `_is_single_sentence`; lifts `falsification-hard` 6/12 → 10/12;
|
||||
bench-gated, 0 gate-attributable QA shifts). Steps 2.4 (`parse_pointer_claims`
|
||||
segmentation) + 2.2 (sequence-aware paraphrase) next per §3 / §5.
|
||||
**Opened:** 2026-05-11
|
||||
**Scope:** Close the headroom #000046 left in the two below-ceiling 5F
|
||||
hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl`
|
||||
|
|
@ -182,13 +186,68 @@ cost.
|
|||
|
||||
## 5. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Doc-only proposal. Opens when fox picks
|
||||
a starting point (recommendation: 2.1 / entity salient-token gate).
|
||||
The implementation commit chain it gates is the §3 sequence:
|
||||
2.1 → 2.4 → 2.2, each bench-gated, each re-pinning the hard-pack rate
|
||||
+ documenting the `make bench-qa` (or Formulate-fixtures) before/after.
|
||||
Closure: both `falsification-hard-v1.jsonl` and `formulate-hard-v1.jsonl`
|
||||
at (or near) rate 1.0, or fox-decided "residue not worth it".
|
||||
**In progress — step 2.1 landed 2026-05-11.** Steps 2.4 + 2.2 next
|
||||
per §3.
|
||||
|
||||
### Step 2.1 (landed 2026-05-11) — entity salient-token-disagreement gate
|
||||
|
||||
- **Change:** `arborist/qa/verify.py` — `_entity_salient_disagrees(answer_text,
|
||||
norm_ctx)` (a > 4-char Capitalized content token, stopword-filtered,
|
||||
or a digit-number in the answer that's absent from the source) +
|
||||
`_is_single_sentence(text)`. Gated in `verify_quotes`' entity branch
|
||||
(proximity policy) in the *weakest-grounding* slot only — `not
|
||||
cluster AND len(verified) <= 1 AND _is_single_sentence(answer_text)
|
||||
AND _entity_salient_disagrees(...)` → `UNGROUNDED`. The narrow
|
||||
caller-gate is what keeps a structured multi-claim summary (the
|
||||
Matrix cast list — many entities, a tight cluster; the TMNT answer —
|
||||
a numbered list with parenthetical nicknames the source omits)
|
||||
untouched: model-added accurate detail in a real summary isn't a
|
||||
contradiction, only the single-sentence-one-weak-match shape is.
|
||||
- **Effect on the hard pack:** lifts `5f-fal-hard-001/002/006/008`
|
||||
(Insulin / Berlin / 1889 / Pacific — the 4 HYBRID_ENTITY
|
||||
over-grounds) to UNGROUNDED → **falsification-hard rate 6/12 →
|
||||
10/12 = 0.833**. The remaining 2 (hard-003 Mercury, hard-005
|
||||
Einstein — recombination STRICT_PARAPHRASE) are step 2.2's job.
|
||||
- **Live-pack bookkeeping:** `5f-fal-live-003` (the *exact* gap #000046
|
||||
built its hard pack around — Insulin/Fleming) and `5f-fal-live-028`
|
||||
(Eiffel/London) had `expected_reason` updated `HYBRID_ENTITY →
|
||||
UNGROUNDED` — the live pack records what `verify_quotes` actually
|
||||
does, and now it does the right thing.
|
||||
- **Bench gate:** `make bench-qa` (n=3 × 75 questions × 3 modes = 675
|
||||
cells) after (`bench/qa_results/2026-05-11T17-12-41Z.md`) vs the
|
||||
pre-step-2.1 baseline (`...T14-19-51Z.md`, which is HEAD's verify.py).
|
||||
STRICT-rate: quote 0.50 → 0.54, pointer 0.25 → 0.22, lattice 0.45 →
|
||||
0.43 — all within the 5-pp noise floor. **Per-row diff: 0 quote-mode
|
||||
rows demoted to UNGROUNDED from the entity path** — the gate didn't
|
||||
fire on a single legitimate QA answer in the whole bench. Every
|
||||
quote-mode `audit_mode` change was LLM re-answer variance
|
||||
(`verifier quote→quote` with the verdict flipping = a different
|
||||
answer); the pointer/lattice deltas are noise too (the gate is in
|
||||
`verify_quotes` / quote mode, not the claim-lattice verifier). No
|
||||
regression — the gate is provably narrow on real traffic.
|
||||
- **Tests:** `tests/test_verify.py` — `test_is_single_sentence_helper`,
|
||||
`test_entity_salient_disagrees_helper`,
|
||||
`test_entity_path_swapped_subject_demoted_to_ungrounded`,
|
||||
`test_entity_gate_narrow_multi_claim_summary_unaffected` (the
|
||||
Matrix / TMNT / hybrid entity-path regression tests still pass —
|
||||
pinned untouched); `tests/test_bench_batteries.py`
|
||||
`test_5f_falsification_hard_pack_below_ceiling` re-pinned 6/12 →
|
||||
10/12; `test_fork_score_positive_gamma_5f_on_hard_falsification_improvement`
|
||||
updated (positive γ·Δ5f on the real lift — possibly MARGINAL given
|
||||
the ÷5 dilution; ACCEPT shown via a degraded-parent sub-scenario).
|
||||
|
||||
### Still ahead
|
||||
|
||||
- **Step 2.4** — `parse_pointer_claims` clause segmentation (closes
|
||||
the 8 mis-segments in `formulate-hard-v1.jsonl`). Bench gate: the
|
||||
Formulate fixtures (incl. `formulate-v1.jsonl` / `formulate-live-v1.jsonl`,
|
||||
must not regress) + a QA smoke (`parse_pointer_claims` feeds the
|
||||
claim-lattice modes).
|
||||
- **Step 2.2** — sequence-aware paraphrase match (closes hard-003 +
|
||||
hard-005). Bench-gated like 2.1; conservative threshold.
|
||||
- **Closure:** both hard packs at (or near) rate 1.0, or fox-decided
|
||||
"residue not worth it" — at which point this ticket closes (and
|
||||
step 2.3 / mini-NLI stays a deferred maybe).
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -701,26 +701,29 @@ def test_5f_formulate_hard_pack_below_ceiling():
|
|||
|
||||
def test_5f_falsification_hard_pack_below_ceiling():
|
||||
"""The hard pack is a deliberate below-ceiling baseline: 12
|
||||
near-misses whose correct verdict is UNGROUNDED. As of #000046
|
||||
closure (2026-05-11), verify_quotes' paraphrase numeric-agreement
|
||||
gate catches the 2 magnitude/year over-grounds (50-vs-100,
|
||||
300-vs-300,000) → 6/12 pass; the remaining 6 over-grounds (4
|
||||
HYBRID_ENTITY, 2 STRICT_PARAPHRASE without a number) are the
|
||||
headroom a future verifier upgrade closes. verify_quotes is
|
||||
near-misses whose correct verdict is UNGROUNDED. As of #000048
|
||||
step 2.1 (2026-05-11): #000046's paraphrase numeric-agreement gate
|
||||
catches the 2 magnitude/year over-grounds (50-vs-100,
|
||||
300-vs-300,000) + #000048's entity salient-token gate catches the
|
||||
4 HYBRID_ENTITY over-grounds (Insulin/Berlin/1889/Pacific) → 10/12
|
||||
pass; the remaining 2 are recombination STRICT_PARAPHRASE (hard-003
|
||||
Mercury, hard-005 Einstein — the false claim recombines source
|
||||
tokens; lexical token-coverage can't tell recombination from
|
||||
grounding), headroom for #000048 step 2.2. verify_quotes is
|
||||
deterministic, so this is stable; if it shifts, the pinned value
|
||||
here fires (loud signal — either the verifier got better/worse, or
|
||||
the pack drifted)."""
|
||||
res = b_5f.run_falsification(F5F / "falsification-hard-v1.jsonl")
|
||||
assert res.pass_count == 6
|
||||
assert res.fail_count == 6
|
||||
assert res.metrics["error_detection_rate"] == pytest.approx(6 / 12)
|
||||
assert res.pass_count == 10
|
||||
assert res.fail_count == 2
|
||||
assert res.metrics["error_detection_rate"] == pytest.approx(10 / 12)
|
||||
# All tasks route the live verifier; every fixture asserts UNGROUNDED.
|
||||
for t in res.per_task:
|
||||
assert t.detail["source"] == "live"
|
||||
assert t.detail["expected_reason"] == "UNGROUNDED"
|
||||
# The 8 that fail do so by over-grounding (STRICT_/HYBRID_), never
|
||||
# by the verifier saying UNGROUNDED — i.e. these are real
|
||||
# over-ground misses, not "the verifier abstained".
|
||||
# The 2 that fail do so by over-grounding (STRICT_/HYBRID_), never
|
||||
# by the verifier saying UNGROUNDED — real over-ground misses, not
|
||||
# "the verifier abstained".
|
||||
for t in res.per_task:
|
||||
if not t.passed:
|
||||
obs = set(t.detail["observed_violations"])
|
||||
|
|
@ -729,27 +732,37 @@ def test_5f_falsification_hard_pack_below_ceiling():
|
|||
|
||||
|
||||
def test_fork_score_positive_gamma_5f_on_hard_falsification_improvement():
|
||||
"""Worked example for #000046 — the loop: with a below-ceiling
|
||||
parent (5f/falsification at the hard-pack rate) and a child that
|
||||
lifts it to 1.0, fork_score's γ·Δ5f term goes strictly positive
|
||||
and the verdict is ACCEPT. This is what a real verify_quotes
|
||||
tightening would produce — the bench Δ-rate carrying signal it
|
||||
can't carry while every pack is at ceiling."""
|
||||
"""Worked example for #000046/#000048 — the loop: a real verifier
|
||||
improvement that lifts the falsification-hard rate produces a
|
||||
*positive* γ·Δ5f term in fork_score. Whether it clears SIGNAL_FLOOR
|
||||
(→ ACCEPT) or not (→ MARGINAL) depends on the magnitude and the
|
||||
÷5 averaging dilution (#000047) — a small single-sub lift is
|
||||
positive-but-MARGINAL by design; a big-enough one ACCEPTs. Both
|
||||
shown here. The point: the bench Δ-rate carries signal it can't
|
||||
carry while every pack is at ceiling."""
|
||||
from arborist.substrate.fork_score import fork_score
|
||||
|
||||
hard_rate = b_5f.run_falsification(
|
||||
F5F / "falsification-hard-v1.jsonl"
|
||||
).metrics["error_detection_rate"]
|
||||
assert hard_rate < 1.0
|
||||
parent = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": hard_rate}}}
|
||||
child = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": 1.0}}}
|
||||
sf = fork_score(parent, child)
|
||||
assert sf.breakdown["gamma_x_delta_5f"] > 0.0
|
||||
# Δ5f = (1.0 - hard_rate) averaged over the 5 5F subs (the other 4
|
||||
# subs absent ⇒ 0 - 0 = 0). With hard_rate ≈ 1/3 → Δ5f ≈ 0.133.
|
||||
assert sf.breakdown["gamma_x_delta_5f"] == pytest.approx((1.0 - hard_rate) / 5)
|
||||
assert sf.verdict == "ACCEPT"
|
||||
assert not any(f.startswith(("REGRESSION_", "NEG_INF_")) for f in sf.flags)
|
||||
# (a) A real lift from the current hard-pack rate to 1.0 — positive,
|
||||
# possibly MARGINAL (the ÷5 dilution of a single-sub gain).
|
||||
parent_a = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": hard_rate}}}
|
||||
child_a = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": 1.0}}}
|
||||
sf_a = fork_score(parent_a, child_a)
|
||||
assert sf_a.breakdown["gamma_x_delta_5f"] > 0.0
|
||||
assert sf_a.breakdown["gamma_x_delta_5f"] == pytest.approx((1.0 - hard_rate) / 5)
|
||||
assert sf_a.verdict in ("ACCEPT", "MARGINAL")
|
||||
assert not any(f.startswith(("REGRESSION_", "NEG_INF_")) for f in sf_a.flags)
|
||||
# (b) A bigger lift (degraded parent at 0.5 → child 1.0) clears the
|
||||
# floor: Δ5f = 0.5/5 = 0.1 ≥ SIGNAL_FLOOR → ACCEPT.
|
||||
parent_b = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": 0.5}}}
|
||||
child_b = {"5s": {}, "5t": {}, "5f": {"falsification": {"error_detection_rate": 1.0}}}
|
||||
sf_b = fork_score(parent_b, child_b)
|
||||
assert sf_b.breakdown["gamma_x_delta_5f"] == pytest.approx(0.5 / 5)
|
||||
assert sf_b.verdict == "ACCEPT"
|
||||
assert not any(f.startswith(("REGRESSION_", "NEG_INF_")) for f in sf_b.flags)
|
||||
|
||||
|
||||
def test_5f_falsification_covers_every_documented_motif():
|
||||
|
|
|
|||
|
|
@ -13,6 +13,9 @@ Tests cover the full trichotomy under each path plus regressions:
|
|||
from __future__ import annotations
|
||||
|
||||
from arborist.qa.verify import (
|
||||
_entity_salient_disagrees,
|
||||
_is_single_sentence,
|
||||
_normalize,
|
||||
_numeric_signature,
|
||||
extract_claim_spans,
|
||||
extract_proper_nouns,
|
||||
|
|
@ -615,3 +618,77 @@ def test_paraphrase_number_present_in_source_still_verifies():
|
|||
# quote/span/entity may classify first; what matters is it's not
|
||||
# demoted to UNGROUNDED by a spurious numeric mismatch.
|
||||
assert v["audit_mode"] in ("STRICT", "HYBRID")
|
||||
|
||||
|
||||
# --- #000048 step 2.1 — entity salient-token-disagreement gate ----
|
||||
|
||||
|
||||
def test_is_single_sentence_helper():
|
||||
assert _is_single_sentence("Insulin was discovered by Alexander Fleming.")
|
||||
assert _is_single_sentence("The Eiffel Tower is in Berlin.")
|
||||
assert _is_single_sentence("The Titanic sank in the Pacific Ocean.\n") # trailing ws stripped
|
||||
assert not _is_single_sentence("One claim. Another claim.")
|
||||
assert not _is_single_sentence("1. Leonardo\n2. Raphael\n3. Donatello")
|
||||
assert not _is_single_sentence("Was it true? She asked.")
|
||||
|
||||
|
||||
def test_entity_salient_disagrees_helper():
|
||||
# Swapped subject: "Insulin" is capitalized, >4 chars, absent from
|
||||
# the source → disagrees.
|
||||
assert _entity_salient_disagrees(
|
||||
"Insulin was discovered by Alexander Fleming.",
|
||||
_normalize("Penicillin was discovered by Alexander Fleming in 1928."),
|
||||
)
|
||||
# Swapped city: "London" absent.
|
||||
assert _entity_salient_disagrees(
|
||||
"The Eiffel Tower is in London.",
|
||||
_normalize("The Eiffel Tower is in Paris France."),
|
||||
)
|
||||
# Swapped year: a digit-number absent.
|
||||
assert _entity_salient_disagrees(
|
||||
"The French Revolution began in 1889.",
|
||||
_normalize("The French Revolution began in 1789, with the storming of the Bastille."),
|
||||
)
|
||||
# Legit entity claim: every capitalized token is in the source →
|
||||
# does NOT disagree (the prose differing — "stars in" vs "cast" —
|
||||
# is fine; only capitalized tokens / numbers are checked).
|
||||
assert not _entity_salient_disagrees(
|
||||
"Keanu Reeves stars in The Matrix.",
|
||||
_normalize("The Matrix cast: Keanu Reeves as Neo, Laurence Fishburne as Morpheus."),
|
||||
)
|
||||
|
||||
|
||||
def test_entity_path_swapped_subject_demoted_to_ungrounded():
|
||||
"""The entity strategy used to grant HYBRID for "Insulin was
|
||||
discovered by Alexander Fleming" against "Penicillin was discovered
|
||||
by Alexander Fleming" on the shared "Alexander Fleming". #000048's
|
||||
salient gate (single short sentence, lone non-clustered match, a
|
||||
capitalized token the source lacks) declines that grounding."""
|
||||
v = verify_quotes(
|
||||
"Insulin was discovered by Alexander Fleming.",
|
||||
"Penicillin was discovered by Alexander Fleming in 1928.",
|
||||
)
|
||||
assert v["audit_mode"] == "UNGROUNDED"
|
||||
v2 = verify_quotes(
|
||||
"The Eiffel Tower is in Berlin.",
|
||||
"The Eiffel Tower is in Paris, designed by Gustave Eiffel.",
|
||||
)
|
||||
assert v2["audit_mode"] == "UNGROUNDED"
|
||||
|
||||
|
||||
def test_entity_gate_narrow_multi_claim_summary_unaffected():
|
||||
"""The gate fires only on the single-sentence-lone-match shape — a
|
||||
structured multi-claim summary that the source partly grounds (a
|
||||
cast list with model-added accurate detail) is NOT demoted. The
|
||||
Matrix / TMNT regression tests above are the full version; this
|
||||
pins the principle on a minimal case: two sentences, so even a
|
||||
capitalized token the source lacks ("Morpheus") doesn't trip the
|
||||
gate."""
|
||||
v = verify_quotes(
|
||||
"The cast includes Keanu Reeves. Morpheus is played by Laurence Fishburne.",
|
||||
"Cast of The Matrix: Keanu Reeves, Laurence Fishburne, Carrie-Anne Moss.",
|
||||
)
|
||||
# Not single-sentence → entity gate doesn't fire → still grounded
|
||||
# at HYBRID (or STRICT if a cluster forms — either way, not
|
||||
# UNGROUNDED-by-the-gate).
|
||||
assert v["audit_mode"] in ("STRICT", "HYBRID")
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue