#000048 step 2.2 — attempted, reverted (not feasible as a conservative lexical change)
Tried a single-clause-containment check on the paraphrase path (and
the entity path's weakest slot): a span is grounded only if a single
source clause (;/. -delimited) covers >= paraphrase_coverage of its
content tokens — a recombination spreads them across clauses. On the
contrived fixtures it works (hard-003 Mercury, hard-005 Einstein →
UNGROUNDED → falsification-hard 10/12 → 12/12). But it ALSO rejects
legitimate cross-sentence summaries — the established fox-2026-04-29
Batman case ("Batman, who is the alias of Bruce Wayne, lives in
Gotham City." against "Batman is the alias of Bruce Wayne. Batman
lives in Gotham City.": spans two clauses, best clause covers 4/6 =
0.67 → demoted, but it's a correct paraphrase). And there's no
threshold that separates the two directions — the recombination
("Mercury …": best clause covers 4/5 = 0.8) sits ABOVE the legit
summary's 0.67, so any cut that rejects Mercury also rejects Batman.
Distinguishing "recombined into a different statement" from
"summarized two adjacent sentences" needs attribution / dependency
parsing or a mini-NLI model (the §2.3 territory), not a conservative
lexical proxy.
Reverted — verify.py stays at its post-step-2.4 state; falsification-
hard stays at 10/12 with hard-003 + hard-005 as documented residue.
#000048 §5 records the attempt + the no-threshold-separates argument;
recommendation: close #000048 at 2.1+2.4 (12 of 16 residual items
closed), with the 2 recombination fixtures a marker for a future
attribution-aware verifier (its own ticket). Awaiting fox's call.
Doc-only — no code change in this commit.
This commit is contained in:
parent
1908fba162
commit
f6ac72270c
1 changed files with 50 additions and 18 deletions
|
|
@ -1,15 +1,18 @@
|
||||||
# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation
|
# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation
|
||||||
|
|
||||||
**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11.**
|
**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11; step
|
||||||
2.1 — `verify_quotes` entity salient-token-disagreement gate
|
2.2 attempted + reverted (not feasible as a conservative lexical
|
||||||
(`_entity_salient_disagrees` + `_is_single_sentence`; `falsification-hard`
|
change — see §5).** 2.1 — `verify_quotes` entity salient-token gate
|
||||||
6/12 → 10/12; bench-gated, 0 gate-attributable QA shifts). 2.4 —
|
(`falsification-hard` 6/12 → 10/12; bench-gated, 0 gate-attributable
|
||||||
`parse_pointer_claims` clause segmentation (`_SEGMENT_SEP_RE` +
|
QA shifts). 2.4 — `parse_pointer_claims` clause segmentation
|
||||||
`_segment_line` guard + wrapped-bullet join; `formulate-hard` 4/12 →
|
(`formulate-hard` 4/12 → 12/12, at ceiling; bench-gated, 7 lattice
|
||||||
12/12 — that pack at ceiling now; bench-gated, 7 lattice cells
|
cells touched, both verdict changes correct). 2.2 — a single-clause-
|
||||||
touched, both verdict changes correct). Step 2.2 (sequence-aware
|
containment paraphrase check catches the 2 recombination fixtures but
|
||||||
paraphrase — closes `falsification-hard` 10/12 → 12/12, the last
|
also rejects legit cross-sentence summaries with no threshold that
|
||||||
residue) next per §3 / §5.
|
separates the two; reverted. **Recommendation: close at 2.1+2.4** —
|
||||||
|
the 2 residual `falsification-hard` recombination fixtures stand as a
|
||||||
|
documented marker; an attribution-aware verifier (§2.3) opens as its
|
||||||
|
own ticket if ever wanted. Awaiting fox's close-or-keep-open call.
|
||||||
**Opened:** 2026-05-11
|
**Opened:** 2026-05-11
|
||||||
**Scope:** Close the headroom #000046 left in the two below-ceiling 5F
|
**Scope:** Close the headroom #000046 left in the two below-ceiling 5F
|
||||||
hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl`
|
hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl`
|
||||||
|
|
@ -286,15 +289,44 @@ per §3.
|
||||||
tests pass untouched; `test_5f_formulate_hard_pack` re-pinned 4/12
|
tests pass untouched; `test_5f_formulate_hard_pack` re-pinned 4/12
|
||||||
→ 12/12.
|
→ 12/12.
|
||||||
|
|
||||||
### Still ahead
|
### Step 2.2 (attempted 2026-05-11 — not feasible as a conservative lexical change)
|
||||||
|
|
||||||
- **Step 2.2** — sequence-aware paraphrase match (closes hard-003
|
Attempt: a *single-clause-containment* check on the paraphrase path
|
||||||
Mercury + hard-005 Einstein → falsification-hard 10/12 → 12/12).
|
(and the entity path's weakest slot) — a span is grounded only if a
|
||||||
Bench-gated like 2.1; conservative threshold.
|
single source clause (`;`/`. `-delimited) covers ≥ `paraphrase_coverage`
|
||||||
- **Closure:** `falsification-hard-v1.jsonl` at (or near) rate 1.0
|
of its content tokens; a recombination spreads them across clauses.
|
||||||
(it's the only hard pack with residue — Formulate is already 12/12),
|
On the contrived fixtures it works (hard-003 Mercury, hard-005
|
||||||
or fox-decided "residue not worth it" — at which point this ticket
|
Einstein → UNGROUNDED → `falsification-hard` 10/12 → 12/12). But it
|
||||||
closes (step 2.3 / mini-NLI stays a deferred maybe).
|
**also rejects legitimate cross-sentence summaries** — the established
|
||||||
|
fox-2026-04-29 Batman case (`test_paraphrase_stopword_filter_does_not_inflate_coverage`):
|
||||||
|
"Batman, who is the alias of Bruce Wayne, lives in Gotham City."
|
||||||
|
against source "Batman is the alias of Bruce Wayne. Batman lives in
|
||||||
|
Gotham City." spans two source clauses (best clause covers 4/6
|
||||||
|
tokens = 0.67) → demoted, but it's a correct paraphrase. And there's
|
||||||
|
no threshold that separates the two directions: the *recombination*
|
||||||
|
("Mercury …"; best clause covers 4/5 = 0.8) sits *above* the legit
|
||||||
|
summary's 0.67 — so any cut that rejects Mercury also rejects Batman.
|
||||||
|
Distinguishing "recombined into a *different* statement" from
|
||||||
|
"summarized two adjacent sentences" needs attribution / dependency
|
||||||
|
parsing or a mini-NLI model (the §2.3 territory), not a conservative
|
||||||
|
lexical proxy. **Reverted** — `verify.py` stays at its post-step-2.4
|
||||||
|
state; `falsification-hard` stays at **10/12** with hard-003 +
|
||||||
|
hard-005 as documented residue (a marker of what an attribution-aware
|
||||||
|
verifier would close).
|
||||||
|
|
||||||
|
### Closure
|
||||||
|
|
||||||
|
Steps 2.1 + 2.4 landed (12 of the original 16 residual items: 4
|
||||||
|
HYBRID_ENTITY over-grounds + 8 Formulate mis-segments closed →
|
||||||
|
`formulate-hard` 12/12, `falsification-hard` 10/12). Step 2.2 found
|
||||||
|
infeasible as a conservative lexical change (above); step 2.3
|
||||||
|
(mini-NLI / attribution) is the only path to the last 2, and it's a
|
||||||
|
model-dependency / bigger-surface move out of #000048's "lexical,
|
||||||
|
deterministic, no-LLM-judge" scope. **Recommendation: close #000048
|
||||||
|
at 2.1+2.4** — the 2 residual `falsification-hard` fixtures stand as
|
||||||
|
a documented marker; if an attribution-aware verifier is ever wanted,
|
||||||
|
it opens as its own ticket (the natural home for §2.3). Awaiting fox's
|
||||||
|
call (close at 2.1+2.4, or keep open for §2.3).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue