#000048 step 2.2 — attempted, reverted (not feasible as a conservative lexical change)

Tried a single-clause-containment check on the paraphrase path (and
the entity path's weakest slot): a span is grounded only if a single
source clause (;/. -delimited) covers >= paraphrase_coverage of its
content tokens — a recombination spreads them across clauses. On the
contrived fixtures it works (hard-003 Mercury, hard-005 Einstein →
UNGROUNDED → falsification-hard 10/12 → 12/12). But it ALSO rejects
legitimate cross-sentence summaries — the established fox-2026-04-29
Batman case ("Batman, who is the alias of Bruce Wayne, lives in
Gotham City." against "Batman is the alias of Bruce Wayne. Batman
lives in Gotham City.": spans two clauses, best clause covers 4/6 =
0.67 → demoted, but it's a correct paraphrase). And there's no
threshold that separates the two directions — the recombination
("Mercury …": best clause covers 4/5 = 0.8) sits ABOVE the legit
summary's 0.67, so any cut that rejects Mercury also rejects Batman.
Distinguishing "recombined into a different statement" from
"summarized two adjacent sentences" needs attribution / dependency
parsing or a mini-NLI model (the §2.3 territory), not a conservative
lexical proxy.

Reverted — verify.py stays at its post-step-2.4 state; falsification-
hard stays at 10/12 with hard-003 + hard-005 as documented residue.
#000048 §5 records the attempt + the no-threshold-separates argument;
recommendation: close #000048 at 2.1+2.4 (12 of 16 residual items
closed), with the 2 recombination fixtures a marker for a future
attribution-aware verifier (its own ticket). Awaiting fox's call.
Doc-only — no code change in this commit.
This commit is contained in:
russell@unturf.com 2026-05-11 17:27:43 -04:00
parent 1908fba162
commit f6ac72270c
No known key found for this signature in database

View file

@ -1,15 +1,18 @@
# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation # Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation
**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11.** **Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11; step
2.1 — `verify_quotes` entity salient-token-disagreement gate 2.2 attempted + reverted (not feasible as a conservative lexical
(`_entity_salient_disagrees` + `_is_single_sentence`; `falsification-hard` change — see §5).** 2.1 — `verify_quotes` entity salient-token gate
6/12 → 10/12; bench-gated, 0 gate-attributable QA shifts). 2.4 — (`falsification-hard` 6/12 → 10/12; bench-gated, 0 gate-attributable
`parse_pointer_claims` clause segmentation (`_SEGMENT_SEP_RE` + QA shifts). 2.4 — `parse_pointer_claims` clause segmentation
`_segment_line` guard + wrapped-bullet join; `formulate-hard` 4/12 → (`formulate-hard` 4/12 → 12/12, at ceiling; bench-gated, 7 lattice
12/12 — that pack at ceiling now; bench-gated, 7 lattice cells cells touched, both verdict changes correct). 2.2 — a single-clause-
touched, both verdict changes correct). Step 2.2 (sequence-aware containment paraphrase check catches the 2 recombination fixtures but
paraphrase — closes `falsification-hard` 10/12 → 12/12, the last also rejects legit cross-sentence summaries with no threshold that
residue) next per §3 / §5. separates the two; reverted. **Recommendation: close at 2.1+2.4** —
the 2 residual `falsification-hard` recombination fixtures stand as a
documented marker; an attribution-aware verifier (§2.3) opens as its
own ticket if ever wanted. Awaiting fox's close-or-keep-open call.
**Opened:** 2026-05-11 **Opened:** 2026-05-11
**Scope:** Close the headroom #000046 left in the two below-ceiling 5F **Scope:** Close the headroom #000046 left in the two below-ceiling 5F
hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl` hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl`
@ -286,15 +289,44 @@ per §3.
tests pass untouched; `test_5f_formulate_hard_pack` re-pinned 4/12 tests pass untouched; `test_5f_formulate_hard_pack` re-pinned 4/12
→ 12/12. → 12/12.
### Still ahead ### Step 2.2 (attempted 2026-05-11 — not feasible as a conservative lexical change)
- **Step 2.2** — sequence-aware paraphrase match (closes hard-003 Attempt: a *single-clause-containment* check on the paraphrase path
Mercury + hard-005 Einstein → falsification-hard 10/12 → 12/12). (and the entity path's weakest slot) — a span is grounded only if a
Bench-gated like 2.1; conservative threshold. single source clause (`;`/`. `-delimited) covers ≥ `paraphrase_coverage`
- **Closure:** `falsification-hard-v1.jsonl` at (or near) rate 1.0 of its content tokens; a recombination spreads them across clauses.
(it's the only hard pack with residue — Formulate is already 12/12), On the contrived fixtures it works (hard-003 Mercury, hard-005
or fox-decided "residue not worth it" — at which point this ticket Einstein → UNGROUNDED → `falsification-hard` 10/12 → 12/12). But it
closes (step 2.3 / mini-NLI stays a deferred maybe). **also rejects legitimate cross-sentence summaries** — the established
fox-2026-04-29 Batman case (`test_paraphrase_stopword_filter_does_not_inflate_coverage`):
"Batman, who is the alias of Bruce Wayne, lives in Gotham City."
against source "Batman is the alias of Bruce Wayne. Batman lives in
Gotham City." spans two source clauses (best clause covers 4/6
tokens = 0.67) → demoted, but it's a correct paraphrase. And there's
no threshold that separates the two directions: the *recombination*
("Mercury …"; best clause covers 4/5 = 0.8) sits *above* the legit
summary's 0.67 — so any cut that rejects Mercury also rejects Batman.
Distinguishing "recombined into a *different* statement" from
"summarized two adjacent sentences" needs attribution / dependency
parsing or a mini-NLI model (the §2.3 territory), not a conservative
lexical proxy. **Reverted** — `verify.py` stays at its post-step-2.4
state; `falsification-hard` stays at **10/12** with hard-003 +
hard-005 as documented residue (a marker of what an attribution-aware
verifier would close).
### Closure
Steps 2.1 + 2.4 landed (12 of the original 16 residual items: 4
HYBRID_ENTITY over-grounds + 8 Formulate mis-segments closed →
`formulate-hard` 12/12, `falsification-hard` 10/12). Step 2.2 found
infeasible as a conservative lexical change (above); step 2.3
(mini-NLI / attribution) is the only path to the last 2, and it's a
model-dependency / bigger-surface move out of #000048's "lexical,
deterministic, no-LLM-judge" scope. **Recommendation: close #000048
at 2.1+2.4** — the 2 residual `falsification-hard` fixtures stand as
a documented marker; if an attribution-aware verifier is ever wanted,
it opens as its own ticket (the natural home for §2.3). Awaiting fox's
call (close at 2.1+2.4, or keep open for §2.3).
--- ---