# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation

**Status:** **closed · 2026-05-12** — steps 2.1 + 2.4 landed
2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds +
8 Formulate mis-segments → `formulate-hard` 12/12, `falsification-hard`
10/12; each bench-gated, no STRICT-rate regression). Step 2.2 (a
single-clause-containment paraphrase check) was attempted + reverted
— it catches the 2 recombination fixtures but also rejects legit
cross-sentence summaries with no threshold separating the two
(recombination-vs-summary isn't lexical; §5 "What we learned"). The
attribution-aware path for those last 2 (and the broader recombination
class) moved to **#000049**; step 2.3 / mini-NLI lives there now, not
here. The 2 residual `falsification-hard` fixtures (hard-003 Mercury,
hard-005 Einstein) stand as a documented marker of where the lexical
verifier stops.
**Opened:** 2026-05-11
**Scope:** Close the headroom #000046 left in the two below-ceiling 5F
hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl`
(4 HYBRID_ENTITY + 2 recombined-no-number STRICT_PARAPHRASE) and the
8 mis-segments in `formulate-hard-v1.jsonl` — by upgrading the live
surfaces those packs exercise: `arborist/qa/verify.py` (the entity
and paraphrase strategies) and `arborist/qa/parse_claims.py`
(`parse_pointer_claims` segmentation). Doc-only proposal: options +
a recommended order; no code in this ticket. Each landing is
bench-gated.
**Audience:** fox + #000046 follow-up readers + maintainers of
`arborist/qa/verify.py` / `arborist/qa/parse_claims.py` + #000012's
v8 acceptance-protocol author (the hard-pack rate feeds `fork_score`'s
`γ·Δ5f`).
**Hard constraint:** every change is bench-gated. A `verify_quotes`
change runs `make bench-qa` (n=3 × all questions × 3 modes) before
*and* after — STRICT-rate must not regress on legitimate answers
(an honesty-driven demotion of an over-claim is allowed; demoting a
verbatim-grounded answer is not, and the gate can't do it anyway —
verbatim/span paths stay untouched). A `parse_pointer_claims` change
runs the Formulate fixtures + a QA smoke (`parse_pointer_claims`
feeds the claim-lattice modes). No schema / governance-hash /
canonicalization-version change. No LLM-as-judge in the verifier.
The existing `*-v1.jsonl` packs keep their pinned digests.

---

## 1. Problem statement

#000046 (closed 2026-05-11) built two below-ceiling 5F packs and
shipped the first surface improvement — `verify_quotes`' paraphrase
numeric-agreement gate, which lifted `falsification-hard-v1.jsonl`
4/12 → 6/12. What it deliberately *didn't* close:

**`falsification-hard-v1.jsonl` — 6 over-grounds remain (rate 6/12).**

- **4 × HYBRID_ENTITY** (`5f-fal-hard-001` Insulin/Fleming, `-002`
  Eiffel/Berlin, `-006` French Revolution 1889, `-008` Titanic/
  Pacific): the entity-proximity strategy grants HYBRID because a
  multi-word proper noun ("Alexander Fleming", "Eiffel Tower",
  "French Revolution", "Titanic") appears in the source — even though
  the answer's *other* salient term (the swapped substance / city /
  year / ocean) is wrong or absent. The strategy looks at the proper
  nouns that *match* and ignores the salient token that *doesn't*.
- **2 × STRICT_PARAPHRASE** (`5f-fal-hard-003` "Mercury is the
  largest planet" vs source "Jupiter is the largest planet …; Mercury
  is the smallest", `-005` "Einstein developed the theory of
  evolution" vs "Einstein developed relativity; Darwin developed
  evolution"): the false claim recombines source tokens into a
  *different true statement*. All content tokens of the false claim
  are present (just rearranged), token coverage is 100%, no digit
  involved — the numeric gate doesn't fire, and lexical
  token-coverage genuinely can't tell recombination from grounding.

**`formulate-hard-v1.jsonl` — 8 mis-segments (rate 4/12).**
`parse_pointer_claims` is line/bullet-based — one line ⇒ one claim,
`[E#]` tokens attach to it — so it merges and-/semicolon-/dash-joined
or `(1)(2)`-enumerated multi-claim lines into one claim with all the
pointers, and splits a wrapped bullet into two. The fixtures'
`expected_lattice` records the sentence/clause-segmented structure a
real claim-lattice parser would produce.

These are the headroom `fork_score`'s `γ·Δ5f` term has left to gain
on the 5F axis (#000012 §8 #4). Each is a distinct, scoped surface
upgrade; this ticket organizes them so they're not lost.

---

## 2. Options

### 2.1 `verify_quotes` — entity-strategy salient-token-disagreement gate (recommended first)

When the entity strategy is about to grant HYBRID for a claim because
a multi-word proper-noun cluster matches the source, also check the
claim's *other* salient tokens — its remaining proper nouns, numbers,
and high-information capitalized terms — don't *disagree* with the
source: a salient token in the claim that appears nowhere in the
source (or appears only attributed to a different subject) blocks the
HYBRID grant for that claim. This is the direct analogue of #000046's
numeric gate (which gated the paraphrase path on digit-numbers) — a
narrow, lexical, bench-safe tightening, no new deps. Catches all 4
HYBRID_ENTITY over-grounds ("Insulin" / "Berlin" / "1889" / "Pacific
Ocean" each absent-or-misattributed in their source). Cost: ~30–50
LOC + tests + a `make bench-qa` before/after (entity-path STRICT/
HYBRID classifications on the QA bench may shift slightly — the gate
only demotes, never promotes, and only on the entity fallback).

### 2.2 `verify_quotes` — sequence-aware paraphrase match (for the 2 recombinations) — ⚠ ATTEMPTED & REVERTED (see §5)

> **Outcome:** the implemented version (single-clause containment —
> the cleanest form of "the answer's tokens didn't recombine across
> source clauses") catches the 2 fixtures but also rejects legitimate
> cross-sentence summaries, with no threshold separating the two
> directions. Reverted. The design idea is preserved below as the
> design log; the finding + the general lesson are in §5 ("Step 2.2
> (attempted …)" + "What we learned").

Original idea: the verbatim/span strategies are substring-exact; the
paraphrase strategy is bag-of-tokens. Neither sees *order* or
*clause structure*. Add an order/structure signal to the paraphrase
path — a longest-common-subsequence ratio, a single-clause-containment
threshold, or "the claim's content-token order is a subsequence of
some N-token window of the source" — high enough to keep real
paraphrases (which preserve local order / stay within a clause)
verifying while rejecting recombinations ("Mercury is the largest"
isn't an ordered subsequence of, nor contained in any one clause of,
"Jupiter is the largest planet in the Solar System; Mercury is the
smallest"). Riskier than 2.1 — paraphrases do reorder and summarize
across sentences — so it needs the full `make bench-qa`, watched
closely. **In practice no threshold works** (§5): the recombination's
best-single-clause coverage sits *above* the legit cross-sentence
summary's, so any cut that rejects the one rejects the other.

### 2.3 `verify_quotes` — attribution-aware / mini-NLI contradiction check (the only path to the last 2; out of scope here)

With 2.1 landed and 2.2 found infeasible (above), this is the *only*
remaining way to close hard-003 + hard-005 — the recombination
fixtures whose distinguishing feature ("the source pairs these tokens
with a *different* subject/predicate than the answer does") is a
dependency / entailment question, not a lexical one (see §5 "What we
learned"). It needs either a small entailment/contradiction model
(adds a model dependency) or a dependency parser + an attribution
check ("for each content token in the answer, find its
subject/predicate in the source; flag if the answer attaches it
elsewhere") — both bigger surface than #000048's "lexical,
deterministic, no-LLM-judge" scope. **Moved to #000049** (fox,
2026-05-12 — its own ticket); the 2 residual fixtures are a
documented marker of where #000049 would pick up.

### 2.4 `parse_pointer_claims` — sentence/clause segmentation

Replace the line/bullet-only segmentation with: within a line, split
on sentence boundaries (`. ` followed by a capital), `;`, a comma-
list after a colon, subordinating conjunctions ("because",
"although"), and inline enumerations `(1) … (2) …`; join a line that
is an indented continuation of the preceding bullet. Each `[E#]`
attaches to the *segment* it appears in. Closes the 8 Formulate
mis-segments. `parse_pointer_claims` feeds the claim-lattice modes,
so the bench gate is the Formulate fixtures + a QA smoke (watch the
claim-lattice STRICT-rate and the `claim_count` distribution). Cost:
~60–100 LOC (the segmenter) + tests + bench; medium.

---

## 3. Recommendation (original — superseded by what actually happened; see §5)

The original plan was **2.1 → 2.4 → 2.2; defer 2.3**. Outcome
(2026-05-11):

1. **2.1 (entity salient-token gate)** — **LANDED.** Narrowest change,
   the bench-safe absence-signal pattern #000046 validated. `falsification-hard`
   6/12 → 10/12; bench-gated, 0 gate-attributable QA shifts.
2. **2.4 (`parse_pointer_claims` segmentation)** — **LANDED.**
   Independent of the verifier work; closed the whole Formulate pack
   `formulate-hard` 4/12 → 12/12 (at ceiling). Bench-gated, 7 lattice
   cells touched, both verdict changes correct.
3. **2.2 (sequence-aware paraphrase)** — **ATTEMPTED & REVERTED** —
   the recombination-vs-summary distinction isn't lexical (see §5 +
   the "What we learned" subsection there). `falsification-hard`
   stays 10/12; hard-003 + hard-005 are documented residue.
4. **2.3 (mini-NLI / attribution)** — the *only* remaining path to
   the last 2 fixtures, and a model-dependency / bigger-surface move
   outside this ticket's lexical-verifier scope. Opens as its own
   ticket if/when those 2 fixtures are worth it. Not done.

Each landed step did: re-pin the hard-pack rate test, update the
fixture `_meta` + per-fixture notes, regenerate
`bench/results/baseline-falsification-hard.json`, add a `make bench-qa`
before/after to `qa-modes-bench.md` as a new Addendum (Addenda 6 +
7), and update the cross-refs in #000046 / #000012 §8 / TICKETS.md.
**Recommended closure: at 2.1+2.4** (see §5 "Closure").

---

## 4. Out of scope

- **The hard packs themselves** — #000046 built them; this ticket
  upgrades the surfaces they exercise, not the fixtures.
- **`fork_score` / the aggregator** — #000047 shipped the
  `delta_aggregator` knob; the bench-Δ term mechanics don't change
  here, only the *rate* it reads.
- **A new harder fixture tier** — if the upgrades make the current
  12-fixture packs trivial again, a Phase 4 of #000046 (or a new
  ticket) adds harder fixtures; not this ticket.
- **Extending the hard tier to 5S/5T sub-batteries** (a retrieval-
  backed sub, say) — #000046 §2 noted it; separate ticket if/when
  wanted.
- **Replacing the lexical verifier with an embedding/LLM one** — a
  much bigger architectural move; this ticket stays within the
  "lexical, deterministic, no-LLM-judge" verifier discipline.

---

## 5. Status

**In progress — step 2.1 landed 2026-05-11.** Steps 2.4 + 2.2 next
per §3.

### Step 2.1 (landed 2026-05-11) — entity salient-token-disagreement gate

- **Change:** `arborist/qa/verify.py` — `_entity_salient_disagrees(answer_text,
  norm_ctx)` (a > 4-char Capitalized content token, stopword-filtered,
  or a digit-number in the answer that's absent from the source) +
  `_is_single_sentence(text)`. Gated in `verify_quotes`' entity branch
  (proximity policy) in the *weakest-grounding* slot only — `not
  cluster AND len(verified) <= 1 AND _is_single_sentence(answer_text)
  AND _entity_salient_disagrees(...)` → `UNGROUNDED`. The narrow
  caller-gate is what keeps a structured multi-claim summary (the
  Matrix cast list — many entities, a tight cluster; the TMNT answer —
  a numbered list with parenthetical nicknames the source omits)
  untouched: model-added accurate detail in a real summary isn't a
  contradiction, only the single-sentence-one-weak-match shape is.
- **Effect on the hard pack:** lifts `5f-fal-hard-001/002/006/008`
  (Insulin / Berlin / 1889 / Pacific — the 4 HYBRID_ENTITY
  over-grounds) to UNGROUNDED → **falsification-hard rate 6/12 →
  10/12 = 0.833**. The remaining 2 (hard-003 Mercury, hard-005
  Einstein — recombination STRICT_PARAPHRASE) are step 2.2's job.
- **Live-pack bookkeeping:** `5f-fal-live-003` (the *exact* gap #000046
  built its hard pack around — Insulin/Fleming) and `5f-fal-live-028`
  (Eiffel/London) had `expected_reason` updated `HYBRID_ENTITY →
  UNGROUNDED` — the live pack records what `verify_quotes` actually
  does, and now it does the right thing.
- **Bench gate:** `make bench-qa` (n=3 × 75 questions × 3 modes = 675
  cells) after (`bench/qa_results/2026-05-11T17-12-41Z.md`) vs the
  pre-step-2.1 baseline (`...T14-19-51Z.md`, which is HEAD's verify.py).
  STRICT-rate: quote 0.50 → 0.54, pointer 0.25 → 0.22, lattice 0.45 →
  0.43 — all within the 5-pp noise floor. **Per-row diff: 0 quote-mode
  rows demoted to UNGROUNDED from the entity path** — the gate didn't
  fire on a single legitimate QA answer in the whole bench. Every
  quote-mode `audit_mode` change was LLM re-answer variance
  (`verifier quote→quote` with the verdict flipping = a different
  answer); the pointer/lattice deltas are noise too (the gate is in
  `verify_quotes` / quote mode, not the claim-lattice verifier). No
  regression — the gate is provably narrow on real traffic.
- **Tests:** `tests/test_verify.py` — `test_is_single_sentence_helper`,
  `test_entity_salient_disagrees_helper`,
  `test_entity_path_swapped_subject_demoted_to_ungrounded`,
  `test_entity_gate_narrow_multi_claim_summary_unaffected` (the
  Matrix / TMNT / hybrid entity-path regression tests still pass —
  pinned untouched); `tests/test_bench_batteries.py`
  `test_5f_falsification_hard_pack_below_ceiling` re-pinned 6/12 →
  10/12; `test_fork_score_positive_gamma_5f_on_hard_falsification_improvement`
  updated (positive γ·Δ5f on the real lift — possibly MARGINAL given
  the ÷5 dilution; ACCEPT shown via a degraded-parent sub-scenario).

### Step 2.4 (landed 2026-05-11) — `parse_pointer_claims` clause segmentation

- **Change:** `arborist/qa/parse_claims.py` — `_SEGMENT_SEP_RE`
  (split a line on `;`, sentence boundaries `. `/`! `/`? ` then a
  Capital, spaced dashes ` - `/` — `/` – `, ` and `/` or `/` because
  `/` although `/` since `/` while `, inline `(N)` enumeration
  markers, and commas — with `(?![^\[]*\])` so a comma *inside* a
  `[E1, E2]` bracket never splits it). `_segment_line` keeps the
  split **only if every resulting non-empty segment is a well-pointered
  claim** — a legit single claim ("The cat is black and white [E1].",
  "The cast: A, B, C [E1].") is never broken because splitting it
  would manufacture pointer-less prose fragments → guard rejects; a
  *leading* colon-terminated header with no pointer ("Two facts:",
  "Key points:") is allowed and dropped. Plus a wrapped-bullet join:
  a continuation line (leading whitespace then a lowercase letter, no
  bullet glyph) folds its text + pointers into the previous claim.
- **Effect on the hard pack:** closes all 8 mis-segments in
  `formulate-hard-v1.jsonl` → **rate 4/12 → 12/12** (the pack is now
  at ceiling — a harder Formulate tier would re-open below-ceiling
  headroom; a #000046 follow-up, out of #000048's scope).
- **Bench gate:** `make bench-qa` (n=3 × 75 × 3 = 675 cells;
  `parse_pointer_claims` feeds the 450 `claim_lattice_pointer` +
  `claim_lattice` cells) after (`bench/qa_results/2026-05-11T20-26-37Z.md`)
  vs the pre-step-2.4 baseline (`...T17-12-41Z.md` = HEAD's
  parse_claims.py). STRICT-rate: quote 0.54 → 0.55, pointer 0.22 →
  0.22, lattice 0.43 → 0.45 — all within the 5-pp noise floor.
  Per-row diff: the segmenter changed the parsed-claim count on the
  *same answer text* for **7 of the 450 lattice cells** (0 in
  `claim_lattice`, 7 in `claim_lattice_pointer`); of those, 2 caused
  an `audit_mode` change — both **correct**: a wrap-join recovered an
  answer's intended structure (4 claims, 2 of which were pointer-less
  wrap-fragments → HYBRID) into 2 well-pointered claims → STRICT; and
  a crammed-one-line-blob (1 monolithic claim, all pointers → STRICT)
  split into 8 claims, some of which don't individually verify →
  HYBRID, the honest verdict (false-positive STRICT was the
  corruption). Every other lattice/quote delta is LLM re-answer
  variance (`answer_chars` changed). No regression — the segmenter's
  only visible effects on real traffic are honest improvements.
- **Tests:** 8 new in `tests/test_claim_lattice.py` (semicolon /
  sentence / conjunction splits; the pointerless-fragment + cast-list
  guards; leading-colon-header drop; wrapped-bullet join;
  pointer-order / multi-pointer); the existing `parse_pointer_claims`
  tests pass untouched; `test_5f_formulate_hard_pack` re-pinned 4/12
  → 12/12.

### Step 2.2 (attempted 2026-05-11 — not feasible as a conservative lexical change)

Attempt: a *single-clause-containment* check on the paraphrase path
(and the entity path's weakest slot) — a span is grounded only if a
single source clause (`;`/`. `-delimited) covers ≥ `paraphrase_coverage`
of its content tokens; a recombination spreads them across clauses.
On the contrived fixtures it works (hard-003 Mercury, hard-005
Einstein → UNGROUNDED → `falsification-hard` 10/12 → 12/12). But it
**also rejects legitimate cross-sentence summaries** — the established
fox-2026-04-29 Batman case (`test_paraphrase_stopword_filter_does_not_inflate_coverage`):
"Batman, who is the alias of Bruce Wayne, lives in Gotham City."
against source "Batman is the alias of Bruce Wayne. Batman lives in
Gotham City." spans two source clauses (best clause covers 4/6
tokens = 0.67) → demoted, but it's a correct paraphrase. And there's
no threshold that separates the two directions: the *recombination*
("Mercury …"; best clause covers 4/5 = 0.8) sits *above* the legit
summary's 0.67 — so any cut that rejects Mercury also rejects Batman.
Distinguishing "recombined into a *different* statement" from
"summarized two adjacent sentences" needs attribution / dependency
parsing or a mini-NLI model (the §2.3 territory), not a conservative
lexical proxy. **Reverted** — `verify.py` stays at its post-step-2.4
state; `falsification-hard` stays at **10/12** with hard-003 +
hard-005 as documented residue (a marker of what an attribution-aware
verifier would close).

### What we learned — the lexical verifier's recombination boundary

The step-2.2 dead end is worth stating as a general principle, not
just a single failed attempt:

1. **A "recombination" and a "cross-sentence summary" are the same
   shape to a lexical verifier.** Both have the answer's content
   tokens *present in the source but scattered across clauses*:
   - recombination — "Mercury is the largest planet …" reuses
     {largest, planet, solar, system} from one source clause and
     {mercury} from another, pairing them into a *false* statement;
   - summary — "Batman, who is the alias of Bruce Wayne, lives in
     Gotham City." reuses {batman, alias, bruce, wayne} from one
     source sentence and {batman, lives, gotham} from the next,
     pairing them into a *true* one.
   Token-coverage (bag-of-words), clause-containment (best-single-
   clause coverage), even bigram-coverage all measure "how scattered"
   — not "scattered into something the source supports vs.
   contradicts". And the orderings don't even sort right: the
   recombination's best-single-clause coverage (Mercury: 4/5 = 0.8)
   sits *above* the legit summary's (Batman: 4/6 = 0.67), so no
   threshold on any of these signals separates them in the safe
   direction.

2. **The discriminating thing is *attribution*: in the source, are
   the answer's paired tokens attached to the same subject/predicate
   the answer attaches them to?** "largest" is the source's predicate
   for *Jupiter*, not Mercury; "evolution" is the source's object for
   *Darwin developed*, not "Einstein developed the theory of". That's
   a dependency / NLI question — it needs parsing or a model, not a
   string metric.

3. **So this is the boundary of the deterministic, no-LLM-judge
   lexical verifier.** #000046's numeric-agreement gate and #000048
   step 2.1's entity salient-token gate work because their signal is
   *absence* ("a digit / a capitalized token the source doesn't
   contain at all") — a clean lexical fact. Step 2.4's clause
   segmenter works because its signal is *structure of the model's
   own output* (one line, several pointered clauses) — also lexical.
   "The source contradicts this pairing" is not lexical, and trying
   to proxy it with a coverage threshold trades a small win on
   contrived fixtures for honest demotions of real cross-sentence
   summaries — a net loss against `docs/bench-maxing.md`'s 5-pp
   floor. (Same wall step 2.1's *first* draft hit: an un-narrowed
   "any absent capitalized token" gate over-fired on the Matrix /
   TMNT answers, which add accurate detail the source omits — the
   fix there was a *narrowing* heuristic, single-sentence-lone-match,
   not a threshold; no analogous narrowing exists for recombination
   because the whole point is that the answer's prose is heavily
   rephrased.)

### Closure

**Closed 2026-05-12 at 2.1+2.4.** Steps 2.1 + 2.4 landed (12 of the
original 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate
mis-segments closed → `formulate-hard` 12/12, `falsification-hard`
10/12; each bench-gated, no STRICT-rate regression). Step 2.2 found
infeasible as a conservative lexical change (above). The
attribution-aware path for the last 2 — step 2.3 / mini-NLI /
dependency-attribution, the only thing that can distinguish a
recombination from a legit cross-sentence summary — **moved to
#000049** (fox's call, 2026-05-12: that capability is its own
ticket, not a #000048 phase). The 2 residual `falsification-hard`
fixtures (hard-003 Mercury, hard-005 Einstein) stand as a documented
marker of exactly where the lexical verifier stops; #000049 weighs
whether to cross that boundary and at what cost.

---

## 6. References

- `docs/tickets/ticket-000046-harder-5sf-fixture-tier.md` (closed) —
  parent; §5 Phase 1/2/3 built the hard packs + the numeric gate;
  the "Headroom" section is what this ticket picks up.
- `bench/fixtures/5f/falsification-hard-v1.jsonl` — the 6 remaining
  over-grounds (`5f-fal-hard-001/002/006/008` HYBRID_ENTITY,
  `-003/-005` STRICT_PARAPHRASE).
- `bench/fixtures/5f/formulate-hard-v1.jsonl` — the 8 mis-segments
  (`5f-form-hard-001..008`).
- `arborist/qa/verify.py` — `_has_entity_cluster` / the entity
  strategy (2.1 target), `_check_each_with_paraphrase` /
  `_token_coverage` / `_numeric_signature` (2.2 target; the 2.1 gate
  mirrors the `_numeric_signature` pattern).
- `arborist/qa/parse_claims.py` — `parse_pointer_claims` (2.4 target).
- `docs/qa-modes-bench.md` Addendum 5 — the #000046 Phase 3 bench
  gate; each #000048 verifier step adds a sibling Addendum.
- `docs/bench-maxing.md` — the bench discipline every step honors
  (n=3, 5-pp floor, honest verdicts over optimistic).
- `docs/tickets/ticket-000012-selection-consensus-protocol.md` §8 #4
  — why this matters: the hard-pack rate is the bench-Δ signal
  `fork_score`'s `γ·Δ5f` carries.
