Tracks the headroom #000046 (closed) left: the 6 over-grounds still in falsification-hard-v1.jsonl (4 HYBRID_ENTITY where the entity-proximity strategy matches a shared proper noun while the answer's other salient term is wrong; 2 STRICT_PARAPHRASE where the false claim recombines source tokens into a different true statement — "Mercury is the largest" — and lexical token-coverage can't tell recombination from grounding) + the 8 mis-segments in formulate-hard-v1.jsonl (line/bullet-only parse_pointer_claims merges multi-claim lines / splits wrapped bullets). Options + recommended order: 2.1 verify_quotes entity salient-token-disagreement gate (the direct analogue of #000046's numeric gate — narrow, lexical, bench-safe; catches the 4 HYBRID_ENTITY) → 2.4 parse_pointer_claims sentence/clause segmentation (closes the Formulate pack) → 2.2 sequence-aware paraphrase match (conservative threshold; catches the 2 recombinations); defer 2.3 mini-NLI (heavier; only if 2.1+2.2 leave a residue worth a model dep). Each landing bench-gated — verify_quotes changes run `make bench-qa` before/after, parse_pointer_claims changes run the Formulate fixtures + a QA smoke. Doc-only proposal; no code in this ticket. Next ID 000048 → 000049; #000046 "Headroom" section + TICKETS.md index updated to point at #000048.
This commit is contained in:
parent
724f433790
commit
9559c2ed31
3 changed files with 224 additions and 5 deletions
|
|
@ -92,6 +92,7 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | open · awaiting go/no-go · doc-only; closes the headroom #000046 left (6 over-grounds in `falsification-hard` + 8 mis-segments in `formulate-hard`). Recommends: 2.1 entity salient-token-disagreement gate (mirrors #000046's numeric gate) → 2.4 `parse_pointer_claims` clause segmentation → 2.2 sequence-aware paraphrase; defer 2.3 mini-NLI. Each bench-gated. #000046 follow-up | 2026-05-11 | — |
|
||||
| #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom (6 over-grounds + 8 mis-segments) = optional bigger-verifier-upgrade follow-up. `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed | 2026-05-11 | — |
|
||||
| #000045 | Prometheus-Σ Phase 3 sleep-sweep scheduler (gating ticket) | open · doc-only scaffold 2026-05-10; pins 8 governance parameters + 4 retrigger gates; opens implementation only after one retrigger fires | 2026-05-10 | — |
|
||||
|
|
@ -142,4 +143,4 @@ Newest first. Update on every open/close.
|
|||
|
||||
## Next ID
|
||||
|
||||
`000048`
|
||||
`000049`
|
||||
|
|
|
|||
|
|
@ -289,10 +289,12 @@ genuinely can't tell recombination from grounding), plus the 8
|
|||
mis-segments in the Formulate hard pack. Closing those needs a
|
||||
bigger verifier upgrade — an order/dependency-aware match or a
|
||||
contradiction check for `verify_quotes`, and sentence/clause
|
||||
segmentation for `parse_pointer_claims` — each its own bench-gated
|
||||
ticket if/when fox wants the headroom closed. #000046 itself is done:
|
||||
the below-ceiling baselines exist, a real surface improvement lifted
|
||||
a rate, and ForkScore's bench-Δ went positive on it.
|
||||
segmentation for `parse_pointer_claims` — **tracked as #000048**
|
||||
(opened 2026-05-11; recommends an entity salient-token-disagreement
|
||||
gate first, then `parse_pointer_claims` clause segmentation, then a
|
||||
sequence-aware paraphrase match; each bench-gated). #000046 itself is
|
||||
done: the below-ceiling baselines exist, a real surface improvement
|
||||
lifted a rate, and ForkScore's bench-Δ went positive on it.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,216 @@
|
|||
# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-11
|
||||
**Scope:** Close the headroom #000046 left in the two below-ceiling 5F
|
||||
hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl`
|
||||
(4 HYBRID_ENTITY + 2 recombined-no-number STRICT_PARAPHRASE) and the
|
||||
8 mis-segments in `formulate-hard-v1.jsonl` — by upgrading the live
|
||||
surfaces those packs exercise: `arborist/qa/verify.py` (the entity
|
||||
and paraphrase strategies) and `arborist/qa/parse_claims.py`
|
||||
(`parse_pointer_claims` segmentation). Doc-only proposal: options +
|
||||
a recommended order; no code in this ticket. Each landing is
|
||||
bench-gated.
|
||||
**Audience:** fox + #000046 follow-up readers + maintainers of
|
||||
`arborist/qa/verify.py` / `arborist/qa/parse_claims.py` + #000012's
|
||||
v8 acceptance-protocol author (the hard-pack rate feeds `fork_score`'s
|
||||
`γ·Δ5f`).
|
||||
**Hard constraint:** every change is bench-gated. A `verify_quotes`
|
||||
change runs `make bench-qa` (n=3 × all questions × 3 modes) before
|
||||
*and* after — STRICT-rate must not regress on legitimate answers
|
||||
(an honesty-driven demotion of an over-claim is allowed; demoting a
|
||||
verbatim-grounded answer is not, and the gate can't do it anyway —
|
||||
verbatim/span paths stay untouched). A `parse_pointer_claims` change
|
||||
runs the Formulate fixtures + a QA smoke (`parse_pointer_claims`
|
||||
feeds the claim-lattice modes). No schema / governance-hash /
|
||||
canonicalization-version change. No LLM-as-judge in the verifier.
|
||||
The existing `*-v1.jsonl` packs keep their pinned digests.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
#000046 (closed 2026-05-11) built two below-ceiling 5F packs and
|
||||
shipped the first surface improvement — `verify_quotes`' paraphrase
|
||||
numeric-agreement gate, which lifted `falsification-hard-v1.jsonl`
|
||||
4/12 → 6/12. What it deliberately *didn't* close:
|
||||
|
||||
**`falsification-hard-v1.jsonl` — 6 over-grounds remain (rate 6/12).**
|
||||
|
||||
- **4 × HYBRID_ENTITY** (`5f-fal-hard-001` Insulin/Fleming, `-002`
|
||||
Eiffel/Berlin, `-006` French Revolution 1889, `-008` Titanic/
|
||||
Pacific): the entity-proximity strategy grants HYBRID because a
|
||||
multi-word proper noun ("Alexander Fleming", "Eiffel Tower",
|
||||
"French Revolution", "Titanic") appears in the source — even though
|
||||
the answer's *other* salient term (the swapped substance / city /
|
||||
year / ocean) is wrong or absent. The strategy looks at the proper
|
||||
nouns that *match* and ignores the salient token that *doesn't*.
|
||||
- **2 × STRICT_PARAPHRASE** (`5f-fal-hard-003` "Mercury is the
|
||||
largest planet" vs source "Jupiter is the largest planet …; Mercury
|
||||
is the smallest", `-005` "Einstein developed the theory of
|
||||
evolution" vs "Einstein developed relativity; Darwin developed
|
||||
evolution"): the false claim recombines source tokens into a
|
||||
*different true statement*. All content tokens of the false claim
|
||||
are present (just rearranged), token coverage is 100%, no digit
|
||||
involved — the numeric gate doesn't fire, and lexical
|
||||
token-coverage genuinely can't tell recombination from grounding.
|
||||
|
||||
**`formulate-hard-v1.jsonl` — 8 mis-segments (rate 4/12).**
|
||||
`parse_pointer_claims` is line/bullet-based — one line ⇒ one claim,
|
||||
`[E#]` tokens attach to it — so it merges and-/semicolon-/dash-joined
|
||||
or `(1)(2)`-enumerated multi-claim lines into one claim with all the
|
||||
pointers, and splits a wrapped bullet into two. The fixtures'
|
||||
`expected_lattice` records the sentence/clause-segmented structure a
|
||||
real claim-lattice parser would produce.
|
||||
|
||||
These are the headroom `fork_score`'s `γ·Δ5f` term has left to gain
|
||||
on the 5F axis (#000012 §8 #4). Each is a distinct, scoped surface
|
||||
upgrade; this ticket organizes them so they're not lost.
|
||||
|
||||
---
|
||||
|
||||
## 2. Options
|
||||
|
||||
### 2.1 `verify_quotes` — entity-strategy salient-token-disagreement gate (recommended first)
|
||||
|
||||
When the entity strategy is about to grant HYBRID for a claim because
|
||||
a multi-word proper-noun cluster matches the source, also check the
|
||||
claim's *other* salient tokens — its remaining proper nouns, numbers,
|
||||
and high-information capitalized terms — don't *disagree* with the
|
||||
source: a salient token in the claim that appears nowhere in the
|
||||
source (or appears only attributed to a different subject) blocks the
|
||||
HYBRID grant for that claim. This is the direct analogue of #000046's
|
||||
numeric gate (which gated the paraphrase path on digit-numbers) — a
|
||||
narrow, lexical, bench-safe tightening, no new deps. Catches all 4
|
||||
HYBRID_ENTITY over-grounds ("Insulin" / "Berlin" / "1889" / "Pacific
|
||||
Ocean" each absent-or-misattributed in their source). Cost: ~30–50
|
||||
LOC + tests + a `make bench-qa` before/after (entity-path STRICT/
|
||||
HYBRID classifications on the QA bench may shift slightly — the gate
|
||||
only demotes, never promotes, and only on the entity fallback).
|
||||
|
||||
### 2.2 `verify_quotes` — sequence-aware paraphrase match (for the 2 recombinations)
|
||||
|
||||
The verbatim/span strategies are substring-exact; the paraphrase
|
||||
strategy is bag-of-tokens. Neither sees *order*. Add a token-
|
||||
*sequence* signal to the paraphrase path — a longest-common-
|
||||
subsequence ratio, or "the claim's content-token order is a
|
||||
subsequence of some N-token window of the source", with a threshold
|
||||
high enough to keep real paraphrases (which preserve local order)
|
||||
verifying while rejecting recombinations ("Mercury is the largest"
|
||||
isn't an ordered subsequence of any window of "Jupiter is the largest
|
||||
planet in the Solar System; Mercury is the smallest" — "mercury"
|
||||
comes *after* "largest" in the source). Riskier than 2.1 — paraphrases
|
||||
do reorder clauses sometimes — so it needs a *wider* bench (the full
|
||||
`make bench-qa`, watched closely for STRICT-rate loss on legit
|
||||
paraphrases), and probably a conservative threshold that catches only
|
||||
the egregious recombinations. Cost: ~40–60 LOC + tests + bench;
|
||||
medium risk.
|
||||
|
||||
### 2.3 `verify_quotes` — mini-NLI contradiction check (heavier; not recommended now)
|
||||
|
||||
A small entailment/contradiction model (or a hand-rolled "subject X
|
||||
paired with predicate-value P in the claim, but P′ ≠ P near X in the
|
||||
source" heuristic) catches the residue 2.1+2.2 miss. Adds a dep (a
|
||||
model) or a brittle heuristic; bigger surface; out of scope until
|
||||
2.1+2.2 land and the remaining headroom is small enough to justify
|
||||
it. Listed as the eventual ceiling, not a near-term move.
|
||||
|
||||
### 2.4 `parse_pointer_claims` — sentence/clause segmentation
|
||||
|
||||
Replace the line/bullet-only segmentation with: within a line, split
|
||||
on sentence boundaries (`. ` followed by a capital), `;`, a comma-
|
||||
list after a colon, subordinating conjunctions ("because",
|
||||
"although"), and inline enumerations `(1) … (2) …`; join a line that
|
||||
is an indented continuation of the preceding bullet. Each `[E#]`
|
||||
attaches to the *segment* it appears in. Closes the 8 Formulate
|
||||
mis-segments. `parse_pointer_claims` feeds the claim-lattice modes,
|
||||
so the bench gate is the Formulate fixtures + a QA smoke (watch the
|
||||
claim-lattice STRICT-rate and the `claim_count` distribution). Cost:
|
||||
~60–100 LOC (the segmenter) + tests + bench; medium.
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommendation
|
||||
|
||||
**Land in order: 2.1 → 2.4 → 2.2; defer 2.3.**
|
||||
|
||||
1. **2.1 (entity salient-token gate)** first — biggest single bucket
|
||||
(4 over-grounds), narrowest change, the bench-safe pattern
|
||||
#000046 already validated. Lifts `falsification-hard` 6/12 → 10/12.
|
||||
2. **2.4 (`parse_pointer_claims` segmentation)** — independent of the
|
||||
verifier work, closes the whole Formulate pack (4/12 → 12/12 if
|
||||
the segmenter is good; some fixtures may need iteration). Lighter
|
||||
bench gate (no live LLM — Formulate fixtures + a QA smoke).
|
||||
3. **2.2 (sequence-aware paraphrase)** — last of the in-scope three,
|
||||
highest risk, smallest remaining bucket (2 over-grounds). A
|
||||
conservative threshold lifts `falsification-hard` 10/12 → 12/12.
|
||||
4. **2.3 (mini-NLI)** — only if 2.1+2.2 leave a residue worth a model
|
||||
dependency. Probably never needed for these 12 fixtures; revisit
|
||||
if a bigger, harder pack surfaces.
|
||||
|
||||
Each step: re-pin the hard-pack rate test
|
||||
(`test_5f_falsification_hard_pack_below_ceiling` /
|
||||
`test_5f_formulate_hard_pack_below_ceiling`), update the fixture
|
||||
`_meta`, regenerate `bench/results/baseline-falsification-hard.json`,
|
||||
add the worked-example note to `ticket-000046` §5 / `qa-modes-bench.md`
|
||||
showing `fork_score`'s `γ·Δ5f` on the real lift, and (for verifier
|
||||
changes) the `make bench-qa` before/after in `qa-modes-bench.md` as
|
||||
a new Addendum. The ticket closes when both hard packs are at (or
|
||||
near) 1.0 — or when fox decides the remaining residue isn't worth the
|
||||
cost.
|
||||
|
||||
---
|
||||
|
||||
## 4. Out of scope
|
||||
|
||||
- **The hard packs themselves** — #000046 built them; this ticket
|
||||
upgrades the surfaces they exercise, not the fixtures.
|
||||
- **`fork_score` / the aggregator** — #000047 shipped the
|
||||
`delta_aggregator` knob; the bench-Δ term mechanics don't change
|
||||
here, only the *rate* it reads.
|
||||
- **A new harder fixture tier** — if the upgrades make the current
|
||||
12-fixture packs trivial again, a Phase 4 of #000046 (or a new
|
||||
ticket) adds harder fixtures; not this ticket.
|
||||
- **Extending the hard tier to 5S/5T sub-batteries** (a retrieval-
|
||||
backed sub, say) — #000046 §2 noted it; separate ticket if/when
|
||||
wanted.
|
||||
- **Replacing the lexical verifier with an embedding/LLM one** — a
|
||||
much bigger architectural move; this ticket stays within the
|
||||
"lexical, deterministic, no-LLM-judge" verifier discipline.
|
||||
|
||||
---
|
||||
|
||||
## 5. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Doc-only proposal. Opens when fox picks
|
||||
a starting point (recommendation: 2.1 / entity salient-token gate).
|
||||
The implementation commit chain it gates is the §3 sequence:
|
||||
2.1 → 2.4 → 2.2, each bench-gated, each re-pinning the hard-pack rate
|
||||
+ documenting the `make bench-qa` (or Formulate-fixtures) before/after.
|
||||
Closure: both `falsification-hard-v1.jsonl` and `formulate-hard-v1.jsonl`
|
||||
at (or near) rate 1.0, or fox-decided "residue not worth it".
|
||||
|
||||
---
|
||||
|
||||
## 6. References
|
||||
|
||||
- `docs/tickets/ticket-000046-harder-5sf-fixture-tier.md` (closed) —
|
||||
parent; §5 Phase 1/2/3 built the hard packs + the numeric gate;
|
||||
the "Headroom" section is what this ticket picks up.
|
||||
- `bench/fixtures/5f/falsification-hard-v1.jsonl` — the 6 remaining
|
||||
over-grounds (`5f-fal-hard-001/002/006/008` HYBRID_ENTITY,
|
||||
`-003/-005` STRICT_PARAPHRASE).
|
||||
- `bench/fixtures/5f/formulate-hard-v1.jsonl` — the 8 mis-segments
|
||||
(`5f-form-hard-001..008`).
|
||||
- `arborist/qa/verify.py` — `_has_entity_cluster` / the entity
|
||||
strategy (2.1 target), `_check_each_with_paraphrase` /
|
||||
`_token_coverage` / `_numeric_signature` (2.2 target; the 2.1 gate
|
||||
mirrors the `_numeric_signature` pattern).
|
||||
- `arborist/qa/parse_claims.py` — `parse_pointer_claims` (2.4 target).
|
||||
- `docs/qa-modes-bench.md` Addendum 5 — the #000046 Phase 3 bench
|
||||
gate; each #000048 verifier step adds a sibling Addendum.
|
||||
- `docs/bench-maxing.md` — the bench discipline every step honors
|
||||
(n=3, 5-pp floor, honest verdicts over optimistic).
|
||||
- `docs/tickets/ticket-000012-selection-consensus-protocol.md` §8 #4
|
||||
— why this matters: the hard-pack rate is the bench-Δ signal
|
||||
`fork_score`'s `γ·Δ5f` carries.
|
||||
Loading…
Add table
Add a link
Reference in a new issue