From f6ac72270c35eb204e9387695038eb8991012103 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Mon, 11 May 2026 17:27:43 -0400 Subject: [PATCH] =?UTF-8?q?#000048=20step=202.2=20=E2=80=94=20attempted,?= =?UTF-8?q?=20reverted=20(not=20feasible=20as=20a=20conservative=20lexical?= =?UTF-8?q?=20change)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tried a single-clause-containment check on the paraphrase path (and the entity path's weakest slot): a span is grounded only if a single source clause (;/. -delimited) covers >= paraphrase_coverage of its content tokens — a recombination spreads them across clauses. On the contrived fixtures it works (hard-003 Mercury, hard-005 Einstein → UNGROUNDED → falsification-hard 10/12 → 12/12). But it ALSO rejects legitimate cross-sentence summaries — the established fox-2026-04-29 Batman case ("Batman, who is the alias of Bruce Wayne, lives in Gotham City." against "Batman is the alias of Bruce Wayne. Batman lives in Gotham City.": spans two clauses, best clause covers 4/6 = 0.67 → demoted, but it's a correct paraphrase). And there's no threshold that separates the two directions — the recombination ("Mercury …": best clause covers 4/5 = 0.8) sits ABOVE the legit summary's 0.67, so any cut that rejects Mercury also rejects Batman. Distinguishing "recombined into a different statement" from "summarized two adjacent sentences" needs attribution / dependency parsing or a mini-NLI model (the §2.3 territory), not a conservative lexical proxy. Reverted — verify.py stays at its post-step-2.4 state; falsification- hard stays at 10/12 with hard-003 + hard-005 as documented residue. #000048 §5 records the attempt + the no-threshold-separates argument; recommendation: close #000048 at 2.1+2.4 (12 of 16 residual items closed), with the 2 recombination fixtures a marker for a future attribution-aware verifier (its own ticket). Awaiting fox's call. Doc-only — no code change in this commit. --- ...fier-upgrade-recombination-segmentation.md | 68 ++++++++++++++----- 1 file changed, 50 insertions(+), 18 deletions(-) diff --git a/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md b/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md index 3dd33d5..b691ca0 100644 --- a/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md +++ b/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md @@ -1,15 +1,18 @@ # Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation -**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11.** -2.1 — `verify_quotes` entity salient-token-disagreement gate -(`_entity_salient_disagrees` + `_is_single_sentence`; `falsification-hard` -6/12 → 10/12; bench-gated, 0 gate-attributable QA shifts). 2.4 — -`parse_pointer_claims` clause segmentation (`_SEGMENT_SEP_RE` + -`_segment_line` guard + wrapped-bullet join; `formulate-hard` 4/12 → -12/12 — that pack at ceiling now; bench-gated, 7 lattice cells -touched, both verdict changes correct). Step 2.2 (sequence-aware -paraphrase — closes `falsification-hard` 10/12 → 12/12, the last -residue) next per §3 / §5. +**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11; step +2.2 attempted + reverted (not feasible as a conservative lexical +change — see §5).** 2.1 — `verify_quotes` entity salient-token gate +(`falsification-hard` 6/12 → 10/12; bench-gated, 0 gate-attributable +QA shifts). 2.4 — `parse_pointer_claims` clause segmentation +(`formulate-hard` 4/12 → 12/12, at ceiling; bench-gated, 7 lattice +cells touched, both verdict changes correct). 2.2 — a single-clause- +containment paraphrase check catches the 2 recombination fixtures but +also rejects legit cross-sentence summaries with no threshold that +separates the two; reverted. **Recommendation: close at 2.1+2.4** — +the 2 residual `falsification-hard` recombination fixtures stand as a +documented marker; an attribution-aware verifier (§2.3) opens as its +own ticket if ever wanted. Awaiting fox's close-or-keep-open call. **Opened:** 2026-05-11 **Scope:** Close the headroom #000046 left in the two below-ceiling 5F hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl` @@ -286,15 +289,44 @@ per §3. tests pass untouched; `test_5f_formulate_hard_pack` re-pinned 4/12 → 12/12. -### Still ahead +### Step 2.2 (attempted 2026-05-11 — not feasible as a conservative lexical change) -- **Step 2.2** — sequence-aware paraphrase match (closes hard-003 - Mercury + hard-005 Einstein → falsification-hard 10/12 → 12/12). - Bench-gated like 2.1; conservative threshold. -- **Closure:** `falsification-hard-v1.jsonl` at (or near) rate 1.0 - (it's the only hard pack with residue — Formulate is already 12/12), - or fox-decided "residue not worth it" — at which point this ticket - closes (step 2.3 / mini-NLI stays a deferred maybe). +Attempt: a *single-clause-containment* check on the paraphrase path +(and the entity path's weakest slot) — a span is grounded only if a +single source clause (`;`/`. `-delimited) covers ≥ `paraphrase_coverage` +of its content tokens; a recombination spreads them across clauses. +On the contrived fixtures it works (hard-003 Mercury, hard-005 +Einstein → UNGROUNDED → `falsification-hard` 10/12 → 12/12). But it +**also rejects legitimate cross-sentence summaries** — the established +fox-2026-04-29 Batman case (`test_paraphrase_stopword_filter_does_not_inflate_coverage`): +"Batman, who is the alias of Bruce Wayne, lives in Gotham City." +against source "Batman is the alias of Bruce Wayne. Batman lives in +Gotham City." spans two source clauses (best clause covers 4/6 +tokens = 0.67) → demoted, but it's a correct paraphrase. And there's +no threshold that separates the two directions: the *recombination* +("Mercury …"; best clause covers 4/5 = 0.8) sits *above* the legit +summary's 0.67 — so any cut that rejects Mercury also rejects Batman. +Distinguishing "recombined into a *different* statement" from +"summarized two adjacent sentences" needs attribution / dependency +parsing or a mini-NLI model (the §2.3 territory), not a conservative +lexical proxy. **Reverted** — `verify.py` stays at its post-step-2.4 +state; `falsification-hard` stays at **10/12** with hard-003 + +hard-005 as documented residue (a marker of what an attribution-aware +verifier would close). + +### Closure + +Steps 2.1 + 2.4 landed (12 of the original 16 residual items: 4 +HYBRID_ENTITY over-grounds + 8 Formulate mis-segments closed → +`formulate-hard` 12/12, `falsification-hard` 10/12). Step 2.2 found +infeasible as a conservative lexical change (above); step 2.3 +(mini-NLI / attribution) is the only path to the last 2, and it's a +model-dependency / bigger-surface move out of #000048's "lexical, +deterministic, no-LLM-judge" scope. **Recommendation: close #000048 +at 2.1+2.4** — the 2 residual `falsification-hard` fixtures stand as +a documented marker; if an attribution-aware verifier is ever wanted, +it opens as its own ticket (the natural home for §2.3). Awaiting fox's +call (close at 2.1+2.4, or keep open for §2.3). ---