#000048 step 2.2 — attempted, reverted (not feasible as a conservative lexical change)

Tried a single-clause-containment check on the paraphrase path (and
the entity path's weakest slot): a span is grounded only if a single
source clause (;/. -delimited) covers >= paraphrase_coverage of its
content tokens — a recombination spreads them across clauses. On the
contrived fixtures it works (hard-003 Mercury, hard-005 Einstein →
UNGROUNDED → falsification-hard 10/12 → 12/12). But it ALSO rejects
legitimate cross-sentence summaries — the established fox-2026-04-29
Batman case ("Batman, who is the alias of Bruce Wayne, lives in
Gotham City." against "Batman is the alias of Bruce Wayne. Batman
lives in Gotham City.": spans two clauses, best clause covers 4/6 =
0.67 → demoted, but it's a correct paraphrase). And there's no
threshold that separates the two directions — the recombination
("Mercury …": best clause covers 4/5 = 0.8) sits ABOVE the legit
summary's 0.67, so any cut that rejects Mercury also rejects Batman.
Distinguishing "recombined into a different statement" from
"summarized two adjacent sentences" needs attribution / dependency
parsing or a mini-NLI model (the §2.3 territory), not a conservative
lexical proxy.

Reverted — verify.py stays at its post-step-2.4 state; falsification-
hard stays at 10/12 with hard-003 + hard-005 as documented residue.
#000048 §5 records the attempt + the no-threshold-separates argument;
recommendation: close #000048 at 2.1+2.4 (12 of 16 residual items
closed), with the 2 recombination fixtures a marker for a future
attribution-aware verifier (its own ticket). Awaiting fox's call.
Doc-only — no code change in this commit.
This commit is contained in:
russell@unturf.com 2026-05-11 17:27:43 -04:00
parent 1908fba162
commit f6ac72270c
No known key found for this signature in database

View file

@ -1,15 +1,18 @@
# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation
**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11.**
2.1 — `verify_quotes` entity salient-token-disagreement gate
(`_entity_salient_disagrees` + `_is_single_sentence`; `falsification-hard`
6/12 → 10/12; bench-gated, 0 gate-attributable QA shifts). 2.4 —
`parse_pointer_claims` clause segmentation (`_SEGMENT_SEP_RE` +
`_segment_line` guard + wrapped-bullet join; `formulate-hard` 4/12 →
12/12 — that pack at ceiling now; bench-gated, 7 lattice cells
touched, both verdict changes correct). Step 2.2 (sequence-aware
paraphrase — closes `falsification-hard` 10/12 → 12/12, the last
residue) next per §3 / §5.
**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11; step
2.2 attempted + reverted (not feasible as a conservative lexical
change — see §5).** 2.1 — `verify_quotes` entity salient-token gate
(`falsification-hard` 6/12 → 10/12; bench-gated, 0 gate-attributable
QA shifts). 2.4 — `parse_pointer_claims` clause segmentation
(`formulate-hard` 4/12 → 12/12, at ceiling; bench-gated, 7 lattice
cells touched, both verdict changes correct). 2.2 — a single-clause-
containment paraphrase check catches the 2 recombination fixtures but
also rejects legit cross-sentence summaries with no threshold that
separates the two; reverted. **Recommendation: close at 2.1+2.4** —
the 2 residual `falsification-hard` recombination fixtures stand as a
documented marker; an attribution-aware verifier (§2.3) opens as its
own ticket if ever wanted. Awaiting fox's close-or-keep-open call.
**Opened:** 2026-05-11
**Scope:** Close the headroom #000046 left in the two below-ceiling 5F
hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl`
@ -286,15 +289,44 @@ per §3.
tests pass untouched; `test_5f_formulate_hard_pack` re-pinned 4/12
→ 12/12.
### Still ahead
### Step 2.2 (attempted 2026-05-11 — not feasible as a conservative lexical change)
- **Step 2.2** — sequence-aware paraphrase match (closes hard-003
Mercury + hard-005 Einstein → falsification-hard 10/12 → 12/12).
Bench-gated like 2.1; conservative threshold.
- **Closure:** `falsification-hard-v1.jsonl` at (or near) rate 1.0
(it's the only hard pack with residue — Formulate is already 12/12),
or fox-decided "residue not worth it" — at which point this ticket
closes (step 2.3 / mini-NLI stays a deferred maybe).
Attempt: a *single-clause-containment* check on the paraphrase path
(and the entity path's weakest slot) — a span is grounded only if a
single source clause (`;`/`. `-delimited) covers ≥ `paraphrase_coverage`
of its content tokens; a recombination spreads them across clauses.
On the contrived fixtures it works (hard-003 Mercury, hard-005
Einstein → UNGROUNDED → `falsification-hard` 10/12 → 12/12). But it
**also rejects legitimate cross-sentence summaries** — the established
fox-2026-04-29 Batman case (`test_paraphrase_stopword_filter_does_not_inflate_coverage`):
"Batman, who is the alias of Bruce Wayne, lives in Gotham City."
against source "Batman is the alias of Bruce Wayne. Batman lives in
Gotham City." spans two source clauses (best clause covers 4/6
tokens = 0.67) → demoted, but it's a correct paraphrase. And there's
no threshold that separates the two directions: the *recombination*
("Mercury …"; best clause covers 4/5 = 0.8) sits *above* the legit
summary's 0.67 — so any cut that rejects Mercury also rejects Batman.
Distinguishing "recombined into a *different* statement" from
"summarized two adjacent sentences" needs attribution / dependency
parsing or a mini-NLI model (the §2.3 territory), not a conservative
lexical proxy. **Reverted** — `verify.py` stays at its post-step-2.4
state; `falsification-hard` stays at **10/12** with hard-003 +
hard-005 as documented residue (a marker of what an attribution-aware
verifier would close).
### Closure
Steps 2.1 + 2.4 landed (12 of the original 16 residual items: 4
HYBRID_ENTITY over-grounds + 8 Formulate mis-segments closed →
`formulate-hard` 12/12, `falsification-hard` 10/12). Step 2.2 found
infeasible as a conservative lexical change (above); step 2.3
(mini-NLI / attribution) is the only path to the last 2, and it's a
model-dependency / bigger-surface move out of #000048's "lexical,
deterministic, no-LLM-judge" scope. **Recommendation: close #000048
at 2.1+2.4** — the 2 residual `falsification-hard` fixtures stand as
a documented marker; if an attribution-aware verifier is ever wanted,
it opens as its own ticket (the natural home for §2.3). Awaiting fox's
call (close at 2.1+2.4, or keep open for §2.3).
---