#000048 — fold the step-2.2 finding through the ticket + the cross-refs
What we learned from the (reverted) step-2.2 attempt, stated as a general principle in #000048 §5 "What we learned": A recombination ("Mercury is the largest planet …" reusing the source's "largest planet …" with its "Mercury") and a legitimate cross-sentence summary ("Batman, who is the alias of Bruce Wayne, lives in Gotham City." reusing two adjacent source sentences) are the same SHAPE to any lexical signal — both scatter the answer's content tokens across source clauses, and the recombination's best-single-clause coverage (4/5 = 0.8) sits ABOVE the legit summary's (4/6 = 0.67), so no token-coverage / clause-containment / bigram threshold separates them in the safe direction. The discriminating thing is *attribution* — in the source, are these tokens attached to the same subject/predicate the answer attaches them to? — which is a dependency / NLI question, not a string metric. That's the boundary of the deterministic, no-LLM-judge lexical verifier: absence signals (#000046 numeric gate, #000048 step 2.1 entity gate) and structure-of-the-model's-own-output signals (step 2.4 segmenter) are lexical and work; "the source contradicts this pairing" is not, and proxying it with a coverage cut trades a small contrived-fixture win for honest demotions of real summaries — a net loss against bench-maxing's 5-pp floor. Updated: #000048 §2.2 (the attempted idea kept as design log + the no-threshold-separates finding), §2.3 (now framed as the only path to the last 2 — attribution / mini-NLI, its own ticket if ever), §3 (original plan annotated with the LANDED / ATTEMPTED+REVERTED / NOT DONE outcome), §5 (the step-2.2 receipt + the "What we learned" subsection + the closure recommendation). Stale cross-refs fixed: TICKETS.md #000048 + #000046 rows, #000046 ticket Headroom section, #000012 §8 #4 — all of which said "#000048 step 2.2 closes the last 2", now corrected to "step 2.2 attempted + reverted; the 2 recombination fixtures stand as documented residue; #000048 §2.3 is the path if ever wanted". Recommendation unchanged: close #000048 at 2.1+2.4 (12 of 16 residual items closed — formulate-hard 12/12, falsification-hard 10/12). Doc- only — no code change.
This commit is contained in:
parent
bbfd2ddc17
commit
5428b3b1d6
4 changed files with 135 additions and 60 deletions
|
|
@ -92,9 +92,9 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | in progress · **steps 2.1 + 2.4 landed 2026-05-11** — 2.1: `verify_quotes` entity salient-token-disagreement gate (`_entity_salient_disagrees` + `_is_single_sentence`) → `falsification-hard` 6/12 → 10/12; 2.4: `parse_pointer_claims` clause segmentation (`_SEGMENT_SEP_RE` split — `;`/sentence/dash/conjunction/`(N)`/comma — kept only when every segment is well-pointered; leading colon-header dropped; wrapped-bullet join) → `formulate-hard` 4/12 → 12/12 (that pack at ceiling now). Each bench-gated (`make bench-qa` n=3×75×3 — 2.1: 0 gate-attributable QA shifts; 2.4: 7 lattice cells touched, both verdict changes correct; all deltas within the 5-pp floor). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED. Step 2.2 (sequence-aware paraphrase — `falsification-hard` 10/12 → 12/12, the last residue) next; defer 2.3 mini-NLI. #000046 follow-up | 2026-05-11 | — |
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | in progress · **steps 2.1 + 2.4 landed 2026-05-11; step 2.2 attempted + reverted.** 2.1: `verify_quotes` entity salient-token-disagreement gate (`_entity_salient_disagrees` + `_is_single_sentence`) → `falsification-hard` 6/12 → 10/12. 2.4: `parse_pointer_claims` clause segmentation (`_SEGMENT_SEP_RE` split kept only when every segment is well-pointered; leading colon-header dropped; wrapped-bullet join) → `formulate-hard` 4/12 → 12/12 (at ceiling). Each bench-gated (`make bench-qa` n=3×75×3 — 0 / 7-lattice-cells-touched-both-correct; all deltas within the 5-pp floor). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED. 2.2 (single-clause-containment paraphrase check): catches the 2 recombination fixtures but also rejects legit cross-sentence summaries (the Batman case) with no threshold separating the two — recombination-vs-summary isn't lexical; reverted. **Recommendation: close at 2.1+2.4** (12 of 16 residual items closed; the 2 `falsification-hard` recombination fixtures a documented marker; an attribution-aware verifier = §2.3, its own ticket if ever). #000046 follow-up; awaiting fox's close-or-keep-open call | 2026-05-11 | — |
|
||||
| #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom now down to 2 falsification-hard over-grounds (#000048 step 2.1 closed the 4 entity over-grounds; step 2.4 closed the 8 Formulate mis-segments → that pack 12/12; step 2.2 closes the last 2). `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 follow-up | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom now down to 2 falsification-hard over-grounds (#000048 step 2.1 closed the 4 entity over-grounds; step 2.4 closed the 8 Formulate mis-segments → that pack 12/12; step 2.2 attempted + reverted — the last 2 recombination fixtures need an attribution-aware verifier, #000048 §2.3, and stand as documented residue). `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 follow-up | 2026-05-11 | — |
|
||||
| #000045 | Prometheus-Σ Phase 3 sleep-sweep scheduler (gating ticket) | open · doc-only scaffold 2026-05-10; pins 8 governance parameters + 4 retrigger gates; opens implementation only after one retrigger fires | 2026-05-10 | — |
|
||||
| #000044 | AUTOCOUNT doc-drift discipline | closed · landed across `fc5ba50` / `03c0f6a` / `6c6defb` / `f5dbfab` / `3b30126` 2026-05-10 (mechanism + 4 metrics + 54 tags across 7 doc files; harness catches drift at test time, refresh is 60-second turnaround) | 2026-05-10 | — |
|
||||
| #000043 | Empirical tightening of T3 bound constants (#000036 Phase 2) | open · awaiting go/no-go (parks until v7 deployment surfaces measurement data) | 2026-05-10 | — |
|
||||
|
|
|
|||
|
|
@ -585,5 +585,9 @@ protocol must account for:
|
|||
450 lattice cells, both verdict changes correct) → `fork_score`'s
|
||||
`γ·Δ5f` went positive on them. So the bench Δ-rate *does* now carry
|
||||
signal on the 5F/falsification axis, demonstrated end-to-end.
|
||||
Remaining headroom: 2 falsification-hard over-grounds (#000048 step
|
||||
2.2) — an optional follow-up, not a #000012 blocker.
|
||||
Remaining headroom: 2 `falsification-hard` recombination
|
||||
over-grounds — #000048 step 2.2 (a lexical clause-containment
|
||||
check) was attempted and reverted (can't tell recombination from a
|
||||
legit cross-sentence summary; #000048 §5), so closing those needs
|
||||
an attribution-aware verifier (#000048 §2.3, its own ticket if
|
||||
wanted) — an optional follow-up, not a #000012 blocker.
|
||||
|
|
|
|||
|
|
@ -287,11 +287,16 @@ HYBRID_ENTITY over-grounds → `falsification-hard` 10/12; step 2.4
|
|||
mis-segments → `formulate-hard` 12/12 (that pack at ceiling). The
|
||||
last residue: 2 STRICT_PARAPHRASE in `falsification-hard` (hard-003
|
||||
Mercury, hard-005 Einstein — the false claim recombines source tokens
|
||||
into a different true statement, which lexical token-coverage can't
|
||||
tell from grounding; #000048 step 2.2 = sequence-aware paraphrase
|
||||
closes those, then #000048 itself closes). #000046 is done: the
|
||||
below-ceiling baselines exist, real surface improvements lifted rates,
|
||||
and ForkScore's bench-Δ went positive on them.
|
||||
into a different true statement). #000048 step 2.2 (a single-clause-
|
||||
containment lexical check) was *attempted and reverted* — it can't
|
||||
distinguish that recombination from a legitimate cross-sentence
|
||||
summary, and no threshold separates them (#000048 §5 "What we
|
||||
learned"); closing those 2 needs an attribution-aware verifier
|
||||
(#000048 §2.3 — its own ticket if ever wanted), so they stand as
|
||||
documented residue. #000046 is done: the below-ceiling baselines
|
||||
exist, real surface improvements (numeric gate, entity gate, clause
|
||||
segmenter) lifted rates, and ForkScore's bench-Δ went positive on
|
||||
them.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -100,32 +100,47 @@ LOC + tests + a `make bench-qa` before/after (entity-path STRICT/
|
|||
HYBRID classifications on the QA bench may shift slightly — the gate
|
||||
only demotes, never promotes, and only on the entity fallback).
|
||||
|
||||
### 2.2 `verify_quotes` — sequence-aware paraphrase match (for the 2 recombinations)
|
||||
### 2.2 `verify_quotes` — sequence-aware paraphrase match (for the 2 recombinations) — ⚠ ATTEMPTED & REVERTED (see §5)
|
||||
|
||||
The verbatim/span strategies are substring-exact; the paraphrase
|
||||
strategy is bag-of-tokens. Neither sees *order*. Add a token-
|
||||
*sequence* signal to the paraphrase path — a longest-common-
|
||||
subsequence ratio, or "the claim's content-token order is a
|
||||
subsequence of some N-token window of the source", with a threshold
|
||||
high enough to keep real paraphrases (which preserve local order)
|
||||
> **Outcome:** the implemented version (single-clause containment —
|
||||
> the cleanest form of "the answer's tokens didn't recombine across
|
||||
> source clauses") catches the 2 fixtures but also rejects legitimate
|
||||
> cross-sentence summaries, with no threshold separating the two
|
||||
> directions. Reverted. The design idea is preserved below as the
|
||||
> design log; the finding + the general lesson are in §5 ("Step 2.2
|
||||
> (attempted …)" + "What we learned").
|
||||
|
||||
Original idea: the verbatim/span strategies are substring-exact; the
|
||||
paraphrase strategy is bag-of-tokens. Neither sees *order* or
|
||||
*clause structure*. Add an order/structure signal to the paraphrase
|
||||
path — a longest-common-subsequence ratio, a single-clause-containment
|
||||
threshold, or "the claim's content-token order is a subsequence of
|
||||
some N-token window of the source" — high enough to keep real
|
||||
paraphrases (which preserve local order / stay within a clause)
|
||||
verifying while rejecting recombinations ("Mercury is the largest"
|
||||
isn't an ordered subsequence of any window of "Jupiter is the largest
|
||||
planet in the Solar System; Mercury is the smallest" — "mercury"
|
||||
comes *after* "largest" in the source). Riskier than 2.1 — paraphrases
|
||||
do reorder clauses sometimes — so it needs a *wider* bench (the full
|
||||
`make bench-qa`, watched closely for STRICT-rate loss on legit
|
||||
paraphrases), and probably a conservative threshold that catches only
|
||||
the egregious recombinations. Cost: ~40–60 LOC + tests + bench;
|
||||
medium risk.
|
||||
isn't an ordered subsequence of, nor contained in any one clause of,
|
||||
"Jupiter is the largest planet in the Solar System; Mercury is the
|
||||
smallest"). Riskier than 2.1 — paraphrases do reorder and summarize
|
||||
across sentences — so it needs the full `make bench-qa`, watched
|
||||
closely. **In practice no threshold works** (§5): the recombination's
|
||||
best-single-clause coverage sits *above* the legit cross-sentence
|
||||
summary's, so any cut that rejects the one rejects the other.
|
||||
|
||||
### 2.3 `verify_quotes` — mini-NLI contradiction check (heavier; not recommended now)
|
||||
### 2.3 `verify_quotes` — attribution-aware / mini-NLI contradiction check (the only path to the last 2; out of scope here)
|
||||
|
||||
A small entailment/contradiction model (or a hand-rolled "subject X
|
||||
paired with predicate-value P in the claim, but P′ ≠ P near X in the
|
||||
source" heuristic) catches the residue 2.1+2.2 miss. Adds a dep (a
|
||||
model) or a brittle heuristic; bigger surface; out of scope until
|
||||
2.1+2.2 land and the remaining headroom is small enough to justify
|
||||
it. Listed as the eventual ceiling, not a near-term move.
|
||||
With 2.1 landed and 2.2 found infeasible (above), this is the *only*
|
||||
remaining way to close hard-003 + hard-005 — the recombination
|
||||
fixtures whose distinguishing feature ("the source pairs these tokens
|
||||
with a *different* subject/predicate than the answer does") is a
|
||||
dependency / entailment question, not a lexical one (see §5 "What we
|
||||
learned"). It needs either a small entailment/contradiction model
|
||||
(adds a model dependency) or a dependency parser + an attribution
|
||||
check ("for each content token in the answer, find its
|
||||
subject/predicate in the source; flag if the answer attaches it
|
||||
elsewhere") — both bigger surface than #000048's "lexical,
|
||||
deterministic, no-LLM-judge" scope, and both warrant their own
|
||||
ticket. **Not in scope for #000048**; the 2 residual fixtures are a
|
||||
documented marker of where this would pick up.
|
||||
|
||||
### 2.4 `parse_pointer_claims` — sentence/clause segmentation
|
||||
|
||||
|
|
@ -142,34 +157,33 @@ claim-lattice STRICT-rate and the `claim_count` distribution). Cost:
|
|||
|
||||
---
|
||||
|
||||
## 3. Recommendation
|
||||
## 3. Recommendation (original — superseded by what actually happened; see §5)
|
||||
|
||||
**Land in order: 2.1 → 2.4 → 2.2; defer 2.3.**
|
||||
The original plan was **2.1 → 2.4 → 2.2; defer 2.3**. Outcome
|
||||
(2026-05-11):
|
||||
|
||||
1. **2.1 (entity salient-token gate)** first — biggest single bucket
|
||||
(4 over-grounds), narrowest change, the bench-safe pattern
|
||||
#000046 already validated. Lifts `falsification-hard` 6/12 → 10/12.
|
||||
2. **2.4 (`parse_pointer_claims` segmentation)** — independent of the
|
||||
verifier work, closes the whole Formulate pack (4/12 → 12/12 if
|
||||
the segmenter is good; some fixtures may need iteration). Lighter
|
||||
bench gate (no live LLM — Formulate fixtures + a QA smoke).
|
||||
3. **2.2 (sequence-aware paraphrase)** — last of the in-scope three,
|
||||
highest risk, smallest remaining bucket (2 over-grounds). A
|
||||
conservative threshold lifts `falsification-hard` 10/12 → 12/12.
|
||||
4. **2.3 (mini-NLI)** — only if 2.1+2.2 leave a residue worth a model
|
||||
dependency. Probably never needed for these 12 fixtures; revisit
|
||||
if a bigger, harder pack surfaces.
|
||||
1. **2.1 (entity salient-token gate)** — **LANDED.** Narrowest change,
|
||||
the bench-safe absence-signal pattern #000046 validated. `falsification-hard`
|
||||
6/12 → 10/12; bench-gated, 0 gate-attributable QA shifts.
|
||||
2. **2.4 (`parse_pointer_claims` segmentation)** — **LANDED.**
|
||||
Independent of the verifier work; closed the whole Formulate pack
|
||||
`formulate-hard` 4/12 → 12/12 (at ceiling). Bench-gated, 7 lattice
|
||||
cells touched, both verdict changes correct.
|
||||
3. **2.2 (sequence-aware paraphrase)** — **ATTEMPTED & REVERTED** —
|
||||
the recombination-vs-summary distinction isn't lexical (see §5 +
|
||||
the "What we learned" subsection there). `falsification-hard`
|
||||
stays 10/12; hard-003 + hard-005 are documented residue.
|
||||
4. **2.3 (mini-NLI / attribution)** — the *only* remaining path to
|
||||
the last 2 fixtures, and a model-dependency / bigger-surface move
|
||||
outside this ticket's lexical-verifier scope. Opens as its own
|
||||
ticket if/when those 2 fixtures are worth it. Not done.
|
||||
|
||||
Each step: re-pin the hard-pack rate test
|
||||
(`test_5f_falsification_hard_pack_below_ceiling` /
|
||||
`test_5f_formulate_hard_pack_below_ceiling`), update the fixture
|
||||
`_meta`, regenerate `bench/results/baseline-falsification-hard.json`,
|
||||
add the worked-example note to `ticket-000046` §5 / `qa-modes-bench.md`
|
||||
showing `fork_score`'s `γ·Δ5f` on the real lift, and (for verifier
|
||||
changes) the `make bench-qa` before/after in `qa-modes-bench.md` as
|
||||
a new Addendum. The ticket closes when both hard packs are at (or
|
||||
near) 1.0 — or when fox decides the remaining residue isn't worth the
|
||||
cost.
|
||||
Each landed step did: re-pin the hard-pack rate test, update the
|
||||
fixture `_meta` + per-fixture notes, regenerate
|
||||
`bench/results/baseline-falsification-hard.json`, add a `make bench-qa`
|
||||
before/after to `qa-modes-bench.md` as a new Addendum (Addenda 6 +
|
||||
7), and update the cross-refs in #000046 / #000012 §8 / TICKETS.md.
|
||||
**Recommended closure: at 2.1+2.4** (see §5 "Closure").
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -314,6 +328,57 @@ state; `falsification-hard` stays at **10/12** with hard-003 +
|
|||
hard-005 as documented residue (a marker of what an attribution-aware
|
||||
verifier would close).
|
||||
|
||||
### What we learned — the lexical verifier's recombination boundary
|
||||
|
||||
The step-2.2 dead end is worth stating as a general principle, not
|
||||
just a single failed attempt:
|
||||
|
||||
1. **A "recombination" and a "cross-sentence summary" are the same
|
||||
shape to a lexical verifier.** Both have the answer's content
|
||||
tokens *present in the source but scattered across clauses*:
|
||||
- recombination — "Mercury is the largest planet …" reuses
|
||||
{largest, planet, solar, system} from one source clause and
|
||||
{mercury} from another, pairing them into a *false* statement;
|
||||
- summary — "Batman, who is the alias of Bruce Wayne, lives in
|
||||
Gotham City." reuses {batman, alias, bruce, wayne} from one
|
||||
source sentence and {batman, lives, gotham} from the next,
|
||||
pairing them into a *true* one.
|
||||
Token-coverage (bag-of-words), clause-containment (best-single-
|
||||
clause coverage), even bigram-coverage all measure "how scattered"
|
||||
— not "scattered into something the source supports vs.
|
||||
contradicts". And the orderings don't even sort right: the
|
||||
recombination's best-single-clause coverage (Mercury: 4/5 = 0.8)
|
||||
sits *above* the legit summary's (Batman: 4/6 = 0.67), so no
|
||||
threshold on any of these signals separates them in the safe
|
||||
direction.
|
||||
|
||||
2. **The discriminating thing is *attribution*: in the source, are
|
||||
the answer's paired tokens attached to the same subject/predicate
|
||||
the answer attaches them to?** "largest" is the source's predicate
|
||||
for *Jupiter*, not Mercury; "evolution" is the source's object for
|
||||
*Darwin developed*, not "Einstein developed the theory of". That's
|
||||
a dependency / NLI question — it needs parsing or a model, not a
|
||||
string metric.
|
||||
|
||||
3. **So this is the boundary of the deterministic, no-LLM-judge
|
||||
lexical verifier.** #000046's numeric-agreement gate and #000048
|
||||
step 2.1's entity salient-token gate work because their signal is
|
||||
*absence* ("a digit / a capitalized token the source doesn't
|
||||
contain at all") — a clean lexical fact. Step 2.4's clause
|
||||
segmenter works because its signal is *structure of the model's
|
||||
own output* (one line, several pointered clauses) — also lexical.
|
||||
"The source contradicts this pairing" is not lexical, and trying
|
||||
to proxy it with a coverage threshold trades a small win on
|
||||
contrived fixtures for honest demotions of real cross-sentence
|
||||
summaries — a net loss against `docs/bench-maxing.md`'s 5-pp
|
||||
floor. (Same wall step 2.1's *first* draft hit: an un-narrowed
|
||||
"any absent capitalized token" gate over-fired on the Matrix /
|
||||
TMNT answers, which add accurate detail the source omits — the
|
||||
fix there was a *narrowing* heuristic, single-sentence-lone-match,
|
||||
not a threshold; no analogous narrowing exists for recombination
|
||||
because the whole point is that the answer's prose is heavily
|
||||
rephrased.)
|
||||
|
||||
### Closure
|
||||
|
||||
Steps 2.1 + 2.4 landed (12 of the original 16 residual items: 4
|
||||
|
|
@ -323,10 +388,11 @@ infeasible as a conservative lexical change (above); step 2.3
|
|||
(mini-NLI / attribution) is the only path to the last 2, and it's a
|
||||
model-dependency / bigger-surface move out of #000048's "lexical,
|
||||
deterministic, no-LLM-judge" scope. **Recommendation: close #000048
|
||||
at 2.1+2.4** — the 2 residual `falsification-hard` fixtures stand as
|
||||
a documented marker; if an attribution-aware verifier is ever wanted,
|
||||
it opens as its own ticket (the natural home for §2.3). Awaiting fox's
|
||||
call (close at 2.1+2.4, or keep open for §2.3).
|
||||
at 2.1+2.4** — the 2 residual `falsification-hard` fixtures (hard-003
|
||||
Mercury, hard-005 Einstein) stand as a documented marker of exactly
|
||||
where the lexical verifier stops; if an attribution-aware verifier is
|
||||
ever wanted, it opens as its own ticket (the natural home for §2.3).
|
||||
Awaiting fox's call (close at 2.1+2.4, or keep open for §2.3).
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue