From 9559c2ed31e4882f6d4d17e301e9f60629f288da Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Mon, 11 May 2026 12:53:01 -0400 Subject: [PATCH] =?UTF-8?q?#000048=20=E2=80=94=20open:=20verifier=20upgrad?= =?UTF-8?q?e=20for=20the=20#000046=20hard-pack=20headroom?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tracks the headroom #000046 (closed) left: the 6 over-grounds still in falsification-hard-v1.jsonl (4 HYBRID_ENTITY where the entity-proximity strategy matches a shared proper noun while the answer's other salient term is wrong; 2 STRICT_PARAPHRASE where the false claim recombines source tokens into a different true statement — "Mercury is the largest" — and lexical token-coverage can't tell recombination from grounding) + the 8 mis-segments in formulate-hard-v1.jsonl (line/bullet-only parse_pointer_claims merges multi-claim lines / splits wrapped bullets). Options + recommended order: 2.1 verify_quotes entity salient-token-disagreement gate (the direct analogue of #000046's numeric gate — narrow, lexical, bench-safe; catches the 4 HYBRID_ENTITY) → 2.4 parse_pointer_claims sentence/clause segmentation (closes the Formulate pack) → 2.2 sequence-aware paraphrase match (conservative threshold; catches the 2 recombinations); defer 2.3 mini-NLI (heavier; only if 2.1+2.2 leave a residue worth a model dep). Each landing bench-gated — verify_quotes changes run `make bench-qa` before/after, parse_pointer_claims changes run the Formulate fixtures + a QA smoke. Doc-only proposal; no code in this ticket. Next ID 000048 → 000049; #000046 "Headroom" section + TICKETS.md index updated to point at #000048. --- docs/TICKETS.md | 3 +- .../ticket-000046-harder-5sf-fixture-tier.md | 10 +- ...fier-upgrade-recombination-segmentation.md | 216 ++++++++++++++++++ 3 files changed, 224 insertions(+), 5 deletions(-) create mode 100644 docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 1f8bd56..fbc00b8 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -92,6 +92,7 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| +| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | open · awaiting go/no-go · doc-only; closes the headroom #000046 left (6 over-grounds in `falsification-hard` + 8 mis-segments in `formulate-hard`). Recommends: 2.1 entity salient-token-disagreement gate (mirrors #000046's numeric gate) → 2.4 `parse_pointer_claims` clause segmentation → 2.2 sequence-aware paraphrase; defer 2.3 mini-NLI. Each bench-gated. #000046 follow-up | 2026-05-11 | — | | #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — | | #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom (6 over-grounds + 8 mis-segments) = optional bigger-verifier-upgrade follow-up. `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed | 2026-05-11 | — | | #000045 | Prometheus-Σ Phase 3 sleep-sweep scheduler (gating ticket) | open · doc-only scaffold 2026-05-10; pins 8 governance parameters + 4 retrigger gates; opens implementation only after one retrigger fires | 2026-05-10 | — | @@ -142,4 +143,4 @@ Newest first. Update on every open/close. ## Next ID -`000048` +`000049` diff --git a/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md b/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md index f566867..63f5e74 100644 --- a/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md +++ b/docs/tickets/ticket-000046-harder-5sf-fixture-tier.md @@ -289,10 +289,12 @@ genuinely can't tell recombination from grounding), plus the 8 mis-segments in the Formulate hard pack. Closing those needs a bigger verifier upgrade — an order/dependency-aware match or a contradiction check for `verify_quotes`, and sentence/clause -segmentation for `parse_pointer_claims` — each its own bench-gated -ticket if/when fox wants the headroom closed. #000046 itself is done: -the below-ceiling baselines exist, a real surface improvement lifted -a rate, and ForkScore's bench-Δ went positive on it. +segmentation for `parse_pointer_claims` — **tracked as #000048** +(opened 2026-05-11; recommends an entity salient-token-disagreement +gate first, then `parse_pointer_claims` clause segmentation, then a +sequence-aware paraphrase match; each bench-gated). #000046 itself is +done: the below-ceiling baselines exist, a real surface improvement +lifted a rate, and ForkScore's bench-Δ went positive on it. --- diff --git a/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md b/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md new file mode 100644 index 0000000..c7fe3a0 --- /dev/null +++ b/docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md @@ -0,0 +1,216 @@ +# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation + +**Status:** open · awaiting go/no-go +**Opened:** 2026-05-11 +**Scope:** Close the headroom #000046 left in the two below-ceiling 5F +hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl` +(4 HYBRID_ENTITY + 2 recombined-no-number STRICT_PARAPHRASE) and the +8 mis-segments in `formulate-hard-v1.jsonl` — by upgrading the live +surfaces those packs exercise: `arborist/qa/verify.py` (the entity +and paraphrase strategies) and `arborist/qa/parse_claims.py` +(`parse_pointer_claims` segmentation). Doc-only proposal: options + +a recommended order; no code in this ticket. Each landing is +bench-gated. +**Audience:** fox + #000046 follow-up readers + maintainers of +`arborist/qa/verify.py` / `arborist/qa/parse_claims.py` + #000012's +v8 acceptance-protocol author (the hard-pack rate feeds `fork_score`'s +`γ·Δ5f`). +**Hard constraint:** every change is bench-gated. A `verify_quotes` +change runs `make bench-qa` (n=3 × all questions × 3 modes) before +*and* after — STRICT-rate must not regress on legitimate answers +(an honesty-driven demotion of an over-claim is allowed; demoting a +verbatim-grounded answer is not, and the gate can't do it anyway — +verbatim/span paths stay untouched). A `parse_pointer_claims` change +runs the Formulate fixtures + a QA smoke (`parse_pointer_claims` +feeds the claim-lattice modes). No schema / governance-hash / +canonicalization-version change. No LLM-as-judge in the verifier. +The existing `*-v1.jsonl` packs keep their pinned digests. + +--- + +## 1. Problem statement + +#000046 (closed 2026-05-11) built two below-ceiling 5F packs and +shipped the first surface improvement — `verify_quotes`' paraphrase +numeric-agreement gate, which lifted `falsification-hard-v1.jsonl` +4/12 → 6/12. What it deliberately *didn't* close: + +**`falsification-hard-v1.jsonl` — 6 over-grounds remain (rate 6/12).** + +- **4 × HYBRID_ENTITY** (`5f-fal-hard-001` Insulin/Fleming, `-002` + Eiffel/Berlin, `-006` French Revolution 1889, `-008` Titanic/ + Pacific): the entity-proximity strategy grants HYBRID because a + multi-word proper noun ("Alexander Fleming", "Eiffel Tower", + "French Revolution", "Titanic") appears in the source — even though + the answer's *other* salient term (the swapped substance / city / + year / ocean) is wrong or absent. The strategy looks at the proper + nouns that *match* and ignores the salient token that *doesn't*. +- **2 × STRICT_PARAPHRASE** (`5f-fal-hard-003` "Mercury is the + largest planet" vs source "Jupiter is the largest planet …; Mercury + is the smallest", `-005` "Einstein developed the theory of + evolution" vs "Einstein developed relativity; Darwin developed + evolution"): the false claim recombines source tokens into a + *different true statement*. All content tokens of the false claim + are present (just rearranged), token coverage is 100%, no digit + involved — the numeric gate doesn't fire, and lexical + token-coverage genuinely can't tell recombination from grounding. + +**`formulate-hard-v1.jsonl` — 8 mis-segments (rate 4/12).** +`parse_pointer_claims` is line/bullet-based — one line ⇒ one claim, +`[E#]` tokens attach to it — so it merges and-/semicolon-/dash-joined +or `(1)(2)`-enumerated multi-claim lines into one claim with all the +pointers, and splits a wrapped bullet into two. The fixtures' +`expected_lattice` records the sentence/clause-segmented structure a +real claim-lattice parser would produce. + +These are the headroom `fork_score`'s `γ·Δ5f` term has left to gain +on the 5F axis (#000012 §8 #4). Each is a distinct, scoped surface +upgrade; this ticket organizes them so they're not lost. + +--- + +## 2. Options + +### 2.1 `verify_quotes` — entity-strategy salient-token-disagreement gate (recommended first) + +When the entity strategy is about to grant HYBRID for a claim because +a multi-word proper-noun cluster matches the source, also check the +claim's *other* salient tokens — its remaining proper nouns, numbers, +and high-information capitalized terms — don't *disagree* with the +source: a salient token in the claim that appears nowhere in the +source (or appears only attributed to a different subject) blocks the +HYBRID grant for that claim. This is the direct analogue of #000046's +numeric gate (which gated the paraphrase path on digit-numbers) — a +narrow, lexical, bench-safe tightening, no new deps. Catches all 4 +HYBRID_ENTITY over-grounds ("Insulin" / "Berlin" / "1889" / "Pacific +Ocean" each absent-or-misattributed in their source). Cost: ~30–50 +LOC + tests + a `make bench-qa` before/after (entity-path STRICT/ +HYBRID classifications on the QA bench may shift slightly — the gate +only demotes, never promotes, and only on the entity fallback). + +### 2.2 `verify_quotes` — sequence-aware paraphrase match (for the 2 recombinations) + +The verbatim/span strategies are substring-exact; the paraphrase +strategy is bag-of-tokens. Neither sees *order*. Add a token- +*sequence* signal to the paraphrase path — a longest-common- +subsequence ratio, or "the claim's content-token order is a +subsequence of some N-token window of the source", with a threshold +high enough to keep real paraphrases (which preserve local order) +verifying while rejecting recombinations ("Mercury is the largest" +isn't an ordered subsequence of any window of "Jupiter is the largest +planet in the Solar System; Mercury is the smallest" — "mercury" +comes *after* "largest" in the source). Riskier than 2.1 — paraphrases +do reorder clauses sometimes — so it needs a *wider* bench (the full +`make bench-qa`, watched closely for STRICT-rate loss on legit +paraphrases), and probably a conservative threshold that catches only +the egregious recombinations. Cost: ~40–60 LOC + tests + bench; +medium risk. + +### 2.3 `verify_quotes` — mini-NLI contradiction check (heavier; not recommended now) + +A small entailment/contradiction model (or a hand-rolled "subject X +paired with predicate-value P in the claim, but P′ ≠ P near X in the +source" heuristic) catches the residue 2.1+2.2 miss. Adds a dep (a +model) or a brittle heuristic; bigger surface; out of scope until +2.1+2.2 land and the remaining headroom is small enough to justify +it. Listed as the eventual ceiling, not a near-term move. + +### 2.4 `parse_pointer_claims` — sentence/clause segmentation + +Replace the line/bullet-only segmentation with: within a line, split +on sentence boundaries (`. ` followed by a capital), `;`, a comma- +list after a colon, subordinating conjunctions ("because", +"although"), and inline enumerations `(1) … (2) …`; join a line that +is an indented continuation of the preceding bullet. Each `[E#]` +attaches to the *segment* it appears in. Closes the 8 Formulate +mis-segments. `parse_pointer_claims` feeds the claim-lattice modes, +so the bench gate is the Formulate fixtures + a QA smoke (watch the +claim-lattice STRICT-rate and the `claim_count` distribution). Cost: +~60–100 LOC (the segmenter) + tests + bench; medium. + +--- + +## 3. Recommendation + +**Land in order: 2.1 → 2.4 → 2.2; defer 2.3.** + +1. **2.1 (entity salient-token gate)** first — biggest single bucket + (4 over-grounds), narrowest change, the bench-safe pattern + #000046 already validated. Lifts `falsification-hard` 6/12 → 10/12. +2. **2.4 (`parse_pointer_claims` segmentation)** — independent of the + verifier work, closes the whole Formulate pack (4/12 → 12/12 if + the segmenter is good; some fixtures may need iteration). Lighter + bench gate (no live LLM — Formulate fixtures + a QA smoke). +3. **2.2 (sequence-aware paraphrase)** — last of the in-scope three, + highest risk, smallest remaining bucket (2 over-grounds). A + conservative threshold lifts `falsification-hard` 10/12 → 12/12. +4. **2.3 (mini-NLI)** — only if 2.1+2.2 leave a residue worth a model + dependency. Probably never needed for these 12 fixtures; revisit + if a bigger, harder pack surfaces. + +Each step: re-pin the hard-pack rate test +(`test_5f_falsification_hard_pack_below_ceiling` / +`test_5f_formulate_hard_pack_below_ceiling`), update the fixture +`_meta`, regenerate `bench/results/baseline-falsification-hard.json`, +add the worked-example note to `ticket-000046` §5 / `qa-modes-bench.md` +showing `fork_score`'s `γ·Δ5f` on the real lift, and (for verifier +changes) the `make bench-qa` before/after in `qa-modes-bench.md` as +a new Addendum. The ticket closes when both hard packs are at (or +near) 1.0 — or when fox decides the remaining residue isn't worth the +cost. + +--- + +## 4. Out of scope + +- **The hard packs themselves** — #000046 built them; this ticket + upgrades the surfaces they exercise, not the fixtures. +- **`fork_score` / the aggregator** — #000047 shipped the + `delta_aggregator` knob; the bench-Δ term mechanics don't change + here, only the *rate* it reads. +- **A new harder fixture tier** — if the upgrades make the current + 12-fixture packs trivial again, a Phase 4 of #000046 (or a new + ticket) adds harder fixtures; not this ticket. +- **Extending the hard tier to 5S/5T sub-batteries** (a retrieval- + backed sub, say) — #000046 §2 noted it; separate ticket if/when + wanted. +- **Replacing the lexical verifier with an embedding/LLM one** — a + much bigger architectural move; this ticket stays within the + "lexical, deterministic, no-LLM-judge" verifier discipline. + +--- + +## 5. Status + +**Open · awaiting go/no-go.** Doc-only proposal. Opens when fox picks +a starting point (recommendation: 2.1 / entity salient-token gate). +The implementation commit chain it gates is the §3 sequence: +2.1 → 2.4 → 2.2, each bench-gated, each re-pinning the hard-pack rate ++ documenting the `make bench-qa` (or Formulate-fixtures) before/after. +Closure: both `falsification-hard-v1.jsonl` and `formulate-hard-v1.jsonl` +at (or near) rate 1.0, or fox-decided "residue not worth it". + +--- + +## 6. References + +- `docs/tickets/ticket-000046-harder-5sf-fixture-tier.md` (closed) — + parent; §5 Phase 1/2/3 built the hard packs + the numeric gate; + the "Headroom" section is what this ticket picks up. +- `bench/fixtures/5f/falsification-hard-v1.jsonl` — the 6 remaining + over-grounds (`5f-fal-hard-001/002/006/008` HYBRID_ENTITY, + `-003/-005` STRICT_PARAPHRASE). +- `bench/fixtures/5f/formulate-hard-v1.jsonl` — the 8 mis-segments + (`5f-form-hard-001..008`). +- `arborist/qa/verify.py` — `_has_entity_cluster` / the entity + strategy (2.1 target), `_check_each_with_paraphrase` / + `_token_coverage` / `_numeric_signature` (2.2 target; the 2.1 gate + mirrors the `_numeric_signature` pattern). +- `arborist/qa/parse_claims.py` — `parse_pointer_claims` (2.4 target). +- `docs/qa-modes-bench.md` Addendum 5 — the #000046 Phase 3 bench + gate; each #000048 verifier step adds a sibling Addendum. +- `docs/bench-maxing.md` — the bench discipline every step honors + (n=3, 5-pp floor, honest verdicts over optimistic). +- `docs/tickets/ticket-000012-selection-consensus-protocol.md` §8 #4 + — why this matters: the hard-pack rate is the bench-Δ signal + `fork_score`'s `γ·Δ5f` carries.