arborist/bench/results
russell@unturf.com 9899a33b7b
#000048 step 2.1 — verify_quotes entity salient-token-disagreement gate
Closes the 4 HYBRID_ENTITY over-grounds #000046 left in
falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a
multi-word proper noun matches the source — but "Insulin was
discovered by Alexander Fleming" against "Penicillin was discovered by
Alexander Fleming" matches on the shared "Alexander Fleming" while the
swapped subject "Insulin" (the falsehood) is ignored.

arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx)
flags a >4-char Capitalized content token (stopword-filtered) or a
digit-number in the answer absent from the source.
_is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in
verify_quotes' entity branch (proximity policy) in the weakest-grounding
slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence
AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is
what keeps a structured multi-claim summary untouched — the Matrix cast
list (many entities, a tight cluster) and the TMNT answer (a numbered
list with parenthetical nicknames the source omits): model-added
accurate detail in a real summary isn't a contradiction, only the
single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid
entity-path regression tests still pass, pinned untouched.

Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin
/ 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it
newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard
pack around) and 5f-fal-live-028 — had expected_reason updated
HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes
actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE
recombinations (Mercury, Einstein — step 2.2) + 8 Formulate
mis-segments (step 2.4).

Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after
(bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline
(...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54,
pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise
floor. Per-row diff (675 common cells, 30 quote-mode rows changed
audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity
path — the gate fired on 0 legitimate QA answers in the whole bench.
Every transition was LLM re-answer variance (verifier quote→quote with
the verdict flipping); pointer/lattice deltas are noise too (the gate
is in verify_quotes / quote mode, not the claim-lattice verifier). No
regression — the gate is provably narrow on real traffic. Summarized in
qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1.

Tests: 4 new in test_verify.py (_is_single_sentence helper,
_entity_salient_disagrees helper, swapped-subject → UNGROUNDED,
gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling
re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated
(positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5
dilution; ACCEPT via a degraded-parent sub-scenario).
make test 2343 passed, 28 skipped.

#000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes / baseline JSON updated.
2026-05-11 13:57:45 -04:00
..
5f-threshold-calibration-2026-05-11.md #000047 — close: delta_aggregator knob on ForkScore (Option D) 2026-05-11 08:27:38 -04:00
baseline-falsification-hard.json #000048 step 2.1 — verify_quotes entity salient-token-disagreement gate 2026-05-11 13:57:45 -04:00
citation-aliases-batch-2026-05-10.md citation-aliases batch: 18 -> 35 / 92 records resolve (38%) 2026-05-10 06:58:27 -04:00
claim-pack-retrieval-lift-2026-05-09.md bench/results: claim-pack retrieval lift measurement (2026-05-09) 2026-05-09 13:38:04 -04:00
full-bench-qa-2026-05-10.md bench/full-bench-qa: no regression after substrate+Phase 3 sprint (675 cells) 2026-05-10 13:42:12 -04:00
full-warrant-resolution-2026-05-10.md docs: refresh stale alias counts (residual 40/54 → 74) 2026-05-10 12:27:23 -04:00
grinstead-snell-and-cascade-tuning-2026-05-10.md Grinstead-Snell ingest + cascade tuning batch: 68 -> 77/92 (84%) 2026-05-10 08:03:11 -04:00
hilbert-pillar-iv-fully-resolved-2026-05-09.md ticket #000042: 4 geometry aliases resolve all 18 Hilbert pillar IV records 2026-05-09 20:18:58 -04:00
lambda-and-pm-additions-2026-05-10.md PLFA + SF-LF + PM Vol 1 ingest: 62 -> 68/92 (74%) — pillar IX nearly closed 2026-05-10 07:45:40 -04:00
medium-confidence-aliases-and-peano-2026-05-10.md medium-conf aliases + Peano ingest + title backfill: 35 -> 54/92 (59%) 2026-05-10 07:14:44 -04:00
phase3-live-validation-2026-05-10.md Phase 3 live validation: 6/7 probes reach EVIDENCE-WARRANTED 2026-05-10 09:46:10 -04:00
phase3-warrant-chain-A-B-2026-05-10.md bench: #000031 Phase 3 A/B finds mechanism dormant on warrant-targeted fixture 2026-05-10 09:37:27 -04:00
phase3-warrant-chain-A-B-aggressive-2026-05-10.md bench: aggressive warrant fixture confirms Phase 3 is rescue-only, not default-path 2026-05-10 09:53:21 -04:00
phase3-warrant-chain-paraphrase-investigation-2026-05-10.md bench: Phase 3 paraphrase fixture investigation — empirically dormant on current corpus 2026-05-10 10:06:23 -04:00
progressive-and-2026-05-09.md bench: progressive-AND fixture + 2026-05-09 A/B baseline report 2026-05-10 06:35:18 -04:00
prometheus-sigma-sweep-dryrun-2026-05-10.md #000037 §22: dry-run swap to sweep_weights() + §15.4 added 2026-05-10 19:22:03 -04:00
prometheus-sigma-triggers-2026-05-10-npower.md three-thread session output: stale TODOs, N-power probe, ForkScore Phase 1c 2026-05-10 07:46:35 -04:00
prometheus-sigma-triggers-2026-05-10.md #000037 §12 Trigger 2 fires empirically — Phase 1 gate satisfied 2026-05-10 17:55:19 -04:00
prometheus-sigma-triggers-2026-05-11.md #000012 Phase 1c follow-through: wire #000037 §12 Trigger 1 probe to fork_score_branches 2026-05-11 06:56:09 -04:00
real-shard-baseline.json modified: .gitignore 2026-05-08 16:38:09 -04:00
real-shard-baseline.md modified: .gitignore 2026-05-08 16:38:09 -04:00
textbook-base-knowledge-upgrade-2026-05-09.md textbook ingest: 4 base-knowledge additions for pillars I/II/III/IX 2026-05-09 20:38:32 -04:00
three-more-textbooks-2026-05-10.md 3 more textbooks (Russell PoM + Laplace + Dedekind): 54 -> 62/92 (67%) 2026-05-10 07:31:40 -04:00
warrant-chain-render-2026-05-09.md bench/results: warrant-chain render tail measurement (2026-05-09) 2026-05-09 18:00:36 -04:00
witness-sweep.json fan-out: witness automation · #000018 analysis · #000033 pillar VII (Claude) 2026-05-09 14:36:39 -04:00