arborist/bench
russell@unturf.com 9899a33b7b
#000048 step 2.1 — verify_quotes entity salient-token-disagreement gate
Closes the 4 HYBRID_ENTITY over-grounds #000046 left in
falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a
multi-word proper noun matches the source — but "Insulin was
discovered by Alexander Fleming" against "Penicillin was discovered by
Alexander Fleming" matches on the shared "Alexander Fleming" while the
swapped subject "Insulin" (the falsehood) is ignored.

arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx)
flags a >4-char Capitalized content token (stopword-filtered) or a
digit-number in the answer absent from the source.
_is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in
verify_quotes' entity branch (proximity policy) in the weakest-grounding
slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence
AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is
what keeps a structured multi-claim summary untouched — the Matrix cast
list (many entities, a tight cluster) and the TMNT answer (a numbered
list with parenthetical nicknames the source omits): model-added
accurate detail in a real summary isn't a contradiction, only the
single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid
entity-path regression tests still pass, pinned untouched.

Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin
/ 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it
newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard
pack around) and 5f-fal-live-028 — had expected_reason updated
HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes
actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE
recombinations (Mercury, Einstein — step 2.2) + 8 Formulate
mis-segments (step 2.4).

Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after
(bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline
(...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54,
pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise
floor. Per-row diff (675 common cells, 30 quote-mode rows changed
audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity
path — the gate fired on 0 legitimate QA answers in the whole bench.
Every transition was LLM re-answer variance (verifier quote→quote with
the verdict flipping); pointer/lattice deltas are noise too (the gate
is in verify_quotes / quote mode, not the claim-lattice verifier). No
regression — the gate is provably narrow on real traffic. Summarized in
qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1.

Tests: 4 new in test_verify.py (_is_single_sentence helper,
_entity_salient_disagrees helper, swapped-subject → UNGROUNDED,
gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling
re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated
(positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5
dilution; ACCEPT via a degraded-parent sub-scenario).
make test 2343 passed, 28 skipped.

#000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes / baseline JSON updated.
2026-05-11 13:57:45 -04:00
..
batteries #000025 §10.11 + §10.13 + §10.14 — close the 5F battery 2026-05-11 07:41:37 -04:00
fixtures #000048 step 2.1 — verify_quotes entity salient-token-disagreement gate 2026-05-11 13:57:45 -04:00
results #000048 step 2.1 — verify_quotes entity salient-token-disagreement gate 2026-05-11 13:57:45 -04:00
scripts #000047 — close: delta_aggregator knob on ForkScore (Option D) 2026-05-11 08:27:38 -04:00
emergent_log.jsonl ticket(#000006): 300-cycle update — zero false-positive STRICT post-hardening 2026-05-03 23:16:45 -04:00
prometheus_sigma_trigger_probe.py #000012 Phase 1c follow-through: wire #000037 §12 Trigger 1 probe to fork_score_branches 2026-05-11 06:56:09 -04:00
qa_questions.txt aborist/arborist 2026-05-07 09:31:49 -04:00
qa_questions_canonical_witness_npower.txt three-thread session output: stale TODOs, N-power probe, ForkScore Phase 1c 2026-05-10 07:46:35 -04:00
qa_questions_metacog_subset.txt qa(#000011 + 4 more): SOFT_PREFLIGHT_HINT impl + 5-task fan-out 2026-05-03 23:00:56 -04:00
qa_questions_progressive_and.txt bench: progressive-AND fixture + 2026-05-09 A/B baseline report 2026-05-10 06:35:18 -04:00
qa_questions_quantifier_baseline.txt bench(#000008): harness extension — FC rate, violation kinds, raw brackets 2026-05-02 18:35:08 -04:00
qa_questions_quantifier_subset.txt ticket(#000008): §12 dry-run bench findings + --policy harness flag 2026-05-03 08:39:20 -04:00
qa_questions_smoke.txt speed: pytest-xdist, bench smoke, concurrency default; UTF surrogate fix 2026-05-02 09:29:40 -04:00
qa_questions_warrant_chain_aggressive.txt bench: aggressive warrant fixture confirms Phase 3 is rescue-only, not default-path 2026-05-10 09:53:21 -04:00
qa_questions_warrant_chain_paraphrase.txt bench: Phase 3 paraphrase fixture investigation — empirically dormant on current corpus 2026-05-10 10:06:23 -04:00
qa_questions_warrant_chain_probe.txt bench: #000031 Phase 3 A/B finds mechanism dormant on warrant-targeted fixture 2026-05-10 09:37:27 -04:00
qa_sweep.py aborist/arborist 2026-05-07 09:31:49 -04:00
run.sh aborist/arborist 2026-05-07 09:31:49 -04:00