#000048 step 2.2 — attempted, reverted (not feasible as a conservative lexical change)
Tried a single-clause-containment check on the paraphrase path (and
the entity path's weakest slot): a span is grounded only if a single
source clause (;/. -delimited) covers >= paraphrase_coverage of its
content tokens — a recombination spreads them across clauses. On the
contrived fixtures it works (hard-003 Mercury, hard-005 Einstein →
UNGROUNDED → falsification-hard 10/12 → 12/12). But it ALSO rejects
legitimate cross-sentence summaries — the established fox-2026-04-29
Batman case ("Batman, who is the alias of Bruce Wayne, lives in
Gotham City." against "Batman is the alias of Bruce Wayne. Batman
lives in Gotham City.": spans two clauses, best clause covers 4/6 =
0.67 → demoted, but it's a correct paraphrase). And there's no
threshold that separates the two directions — the recombination
("Mercury …": best clause covers 4/5 = 0.8) sits ABOVE the legit
summary's 0.67, so any cut that rejects Mercury also rejects Batman.
Distinguishing "recombined into a different statement" from
"summarized two adjacent sentences" needs attribution / dependency
parsing or a mini-NLI model (the §2.3 territory), not a conservative
lexical proxy.
Reverted — verify.py stays at its post-step-2.4 state; falsification-
hard stays at 10/12 with hard-003 + hard-005 as documented residue.
#000048 §5 records the attempt + the no-threshold-separates argument;
recommendation: close #000048 at 2.1+2.4 (12 of 16 residual items
closed), with the 2 recombination fixtures a marker for a future
attribution-aware verifier (its own ticket). Awaiting fox's call.
Doc-only — no code change in this commit.
This commit is contained in:
parent
1908fba162
commit
f6ac72270c
1 changed files with 50 additions and 18 deletions
|
|
@ -1,15 +1,18 @@
|
|||
# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation
|
||||
|
||||
**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11.**
|
||||
2.1 — `verify_quotes` entity salient-token-disagreement gate
|
||||
(`_entity_salient_disagrees` + `_is_single_sentence`; `falsification-hard`
|
||||
6/12 → 10/12; bench-gated, 0 gate-attributable QA shifts). 2.4 —
|
||||
`parse_pointer_claims` clause segmentation (`_SEGMENT_SEP_RE` +
|
||||
`_segment_line` guard + wrapped-bullet join; `formulate-hard` 4/12 →
|
||||
12/12 — that pack at ceiling now; bench-gated, 7 lattice cells
|
||||
touched, both verdict changes correct). Step 2.2 (sequence-aware
|
||||
paraphrase — closes `falsification-hard` 10/12 → 12/12, the last
|
||||
residue) next per §3 / §5.
|
||||
**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11; step
|
||||
2.2 attempted + reverted (not feasible as a conservative lexical
|
||||
change — see §5).** 2.1 — `verify_quotes` entity salient-token gate
|
||||
(`falsification-hard` 6/12 → 10/12; bench-gated, 0 gate-attributable
|
||||
QA shifts). 2.4 — `parse_pointer_claims` clause segmentation
|
||||
(`formulate-hard` 4/12 → 12/12, at ceiling; bench-gated, 7 lattice
|
||||
cells touched, both verdict changes correct). 2.2 — a single-clause-
|
||||
containment paraphrase check catches the 2 recombination fixtures but
|
||||
also rejects legit cross-sentence summaries with no threshold that
|
||||
separates the two; reverted. **Recommendation: close at 2.1+2.4** —
|
||||
the 2 residual `falsification-hard` recombination fixtures stand as a
|
||||
documented marker; an attribution-aware verifier (§2.3) opens as its
|
||||
own ticket if ever wanted. Awaiting fox's close-or-keep-open call.
|
||||
**Opened:** 2026-05-11
|
||||
**Scope:** Close the headroom #000046 left in the two below-ceiling 5F
|
||||
hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl`
|
||||
|
|
@ -286,15 +289,44 @@ per §3.
|
|||
tests pass untouched; `test_5f_formulate_hard_pack` re-pinned 4/12
|
||||
→ 12/12.
|
||||
|
||||
### Still ahead
|
||||
### Step 2.2 (attempted 2026-05-11 — not feasible as a conservative lexical change)
|
||||
|
||||
- **Step 2.2** — sequence-aware paraphrase match (closes hard-003
|
||||
Mercury + hard-005 Einstein → falsification-hard 10/12 → 12/12).
|
||||
Bench-gated like 2.1; conservative threshold.
|
||||
- **Closure:** `falsification-hard-v1.jsonl` at (or near) rate 1.0
|
||||
(it's the only hard pack with residue — Formulate is already 12/12),
|
||||
or fox-decided "residue not worth it" — at which point this ticket
|
||||
closes (step 2.3 / mini-NLI stays a deferred maybe).
|
||||
Attempt: a *single-clause-containment* check on the paraphrase path
|
||||
(and the entity path's weakest slot) — a span is grounded only if a
|
||||
single source clause (`;`/`. `-delimited) covers ≥ `paraphrase_coverage`
|
||||
of its content tokens; a recombination spreads them across clauses.
|
||||
On the contrived fixtures it works (hard-003 Mercury, hard-005
|
||||
Einstein → UNGROUNDED → `falsification-hard` 10/12 → 12/12). But it
|
||||
**also rejects legitimate cross-sentence summaries** — the established
|
||||
fox-2026-04-29 Batman case (`test_paraphrase_stopword_filter_does_not_inflate_coverage`):
|
||||
"Batman, who is the alias of Bruce Wayne, lives in Gotham City."
|
||||
against source "Batman is the alias of Bruce Wayne. Batman lives in
|
||||
Gotham City." spans two source clauses (best clause covers 4/6
|
||||
tokens = 0.67) → demoted, but it's a correct paraphrase. And there's
|
||||
no threshold that separates the two directions: the *recombination*
|
||||
("Mercury …"; best clause covers 4/5 = 0.8) sits *above* the legit
|
||||
summary's 0.67 — so any cut that rejects Mercury also rejects Batman.
|
||||
Distinguishing "recombined into a *different* statement" from
|
||||
"summarized two adjacent sentences" needs attribution / dependency
|
||||
parsing or a mini-NLI model (the §2.3 territory), not a conservative
|
||||
lexical proxy. **Reverted** — `verify.py` stays at its post-step-2.4
|
||||
state; `falsification-hard` stays at **10/12** with hard-003 +
|
||||
hard-005 as documented residue (a marker of what an attribution-aware
|
||||
verifier would close).
|
||||
|
||||
### Closure
|
||||
|
||||
Steps 2.1 + 2.4 landed (12 of the original 16 residual items: 4
|
||||
HYBRID_ENTITY over-grounds + 8 Formulate mis-segments closed →
|
||||
`formulate-hard` 12/12, `falsification-hard` 10/12). Step 2.2 found
|
||||
infeasible as a conservative lexical change (above); step 2.3
|
||||
(mini-NLI / attribution) is the only path to the last 2, and it's a
|
||||
model-dependency / bigger-surface move out of #000048's "lexical,
|
||||
deterministic, no-LLM-judge" scope. **Recommendation: close #000048
|
||||
at 2.1+2.4** — the 2 residual `falsification-hard` fixtures stand as
|
||||
a documented marker; if an attribution-aware verifier is ever wanted,
|
||||
it opens as its own ticket (the natural home for §2.3). Awaiting fox's
|
||||
call (close at 2.1+2.4, or keep open for §2.3).
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue