diff --git a/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md b/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md index 3dd33d5..b691ca0 100644 --- a/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md +++ b/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md @@ -1,15 +1,18 @@ # Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation -**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11.** -2.1 — `verify_quotes` entity salient-token-disagreement gate -(`_entity_salient_disagrees` + `_is_single_sentence`; `falsification-hard` -6/12 → 10/12; bench-gated, 0 gate-attributable QA shifts). 2.4 — -`parse_pointer_claims` clause segmentation (`_SEGMENT_SEP_RE` + -`_segment_line` guard + wrapped-bullet join; `formulate-hard` 4/12 → -12/12 — that pack at ceiling now; bench-gated, 7 lattice cells -touched, both verdict changes correct). Step 2.2 (sequence-aware -paraphrase — closes `falsification-hard` 10/12 → 12/12, the last -residue) next per §3 / §5. +**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11; step +2.2 attempted + reverted (not feasible as a conservative lexical +change — see §5).** 2.1 — `verify_quotes` entity salient-token gate +(`falsification-hard` 6/12 → 10/12; bench-gated, 0 gate-attributable +QA shifts). 2.4 — `parse_pointer_claims` clause segmentation +(`formulate-hard` 4/12 → 12/12, at ceiling; bench-gated, 7 lattice +cells touched, both verdict changes correct). 2.2 — a single-clause- +containment paraphrase check catches the 2 recombination fixtures but +also rejects legit cross-sentence summaries with no threshold that +separates the two; reverted. **Recommendation: close at 2.1+2.4** — +the 2 residual `falsification-hard` recombination fixtures stand as a +documented marker; an attribution-aware verifier (§2.3) opens as its +own ticket if ever wanted. Awaiting fox's close-or-keep-open call. **Opened:** 2026-05-11 **Scope:** Close the headroom #000046 left in the two below-ceiling 5F hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl` @@ -286,15 +289,44 @@ per §3. tests pass untouched; `test_5f_formulate_hard_pack` re-pinned 4/12 → 12/12. -### Still ahead +### Step 2.2 (attempted 2026-05-11 — not feasible as a conservative lexical change) -- **Step 2.2** — sequence-aware paraphrase match (closes hard-003 - Mercury + hard-005 Einstein → falsification-hard 10/12 → 12/12). - Bench-gated like 2.1; conservative threshold. -- **Closure:** `falsification-hard-v1.jsonl` at (or near) rate 1.0 - (it's the only hard pack with residue — Formulate is already 12/12), - or fox-decided "residue not worth it" — at which point this ticket - closes (step 2.3 / mini-NLI stays a deferred maybe). +Attempt: a *single-clause-containment* check on the paraphrase path +(and the entity path's weakest slot) — a span is grounded only if a +single source clause (`;`/`. `-delimited) covers ≥ `paraphrase_coverage` +of its content tokens; a recombination spreads them across clauses. +On the contrived fixtures it works (hard-003 Mercury, hard-005 +Einstein → UNGROUNDED → `falsification-hard` 10/12 → 12/12). But it +**also rejects legitimate cross-sentence summaries** — the established +fox-2026-04-29 Batman case (`test_paraphrase_stopword_filter_does_not_inflate_coverage`): +"Batman, who is the alias of Bruce Wayne, lives in Gotham City." +against source "Batman is the alias of Bruce Wayne. Batman lives in +Gotham City." spans two source clauses (best clause covers 4/6 +tokens = 0.67) → demoted, but it's a correct paraphrase. And there's +no threshold that separates the two directions: the *recombination* +("Mercury …"; best clause covers 4/5 = 0.8) sits *above* the legit +summary's 0.67 — so any cut that rejects Mercury also rejects Batman. +Distinguishing "recombined into a *different* statement" from +"summarized two adjacent sentences" needs attribution / dependency +parsing or a mini-NLI model (the §2.3 territory), not a conservative +lexical proxy. **Reverted** — `verify.py` stays at its post-step-2.4 +state; `falsification-hard` stays at **10/12** with hard-003 + +hard-005 as documented residue (a marker of what an attribution-aware +verifier would close). + +### Closure + +Steps 2.1 + 2.4 landed (12 of the original 16 residual items: 4 +HYBRID_ENTITY over-grounds + 8 Formulate mis-segments closed → +`formulate-hard` 12/12, `falsification-hard` 10/12). Step 2.2 found +infeasible as a conservative lexical change (above); step 2.3 +(mini-NLI / attribution) is the only path to the last 2, and it's a +model-dependency / bigger-surface move out of #000048's "lexical, +deterministic, no-LLM-judge" scope. **Recommendation: close #000048 +at 2.1+2.4** — the 2 residual `falsification-hard` fixtures stand as +a documented marker; if an attribution-aware verifier is ever wanted, +it opens as its own ticket (the natural home for §2.3). Awaiting fox's +call (close at 2.1+2.4, or keep open for §2.3). ---