#000048 step 2.1 — verify_quotes entity salient-token-disagreement gate
Closes the 4 HYBRID_ENTITY over-grounds #000046 left in falsification-hard-v1.jsonl. The entity strategy grants HYBRID when a multi-word proper noun matches the source — but "Insulin was discovered by Alexander Fleming" against "Penicillin was discovered by Alexander Fleming" matches on the shared "Alexander Fleming" while the swapped subject "Insulin" (the falsehood) is ignored. arborist/qa/verify.py: _entity_salient_disagrees(answer_text, norm_ctx) flags a >4-char Capitalized content token (stopword-filtered) or a digit-number in the answer absent from the source. _is_single_sentence(text) — no internal '. '/'! '/'? ' break. Gated in verify_quotes' entity branch (proximity policy) in the weakest-grounding slot only: not cluster AND len(verified) <= 1 AND _is_single_sentence AND _entity_salient_disagrees → UNGROUNDED. The narrow caller-gate is what keeps a structured multi-claim summary untouched — the Matrix cast list (many entities, a tight cluster) and the TMNT answer (a numbered list with parenthetical nicknames the source omits): model-added accurate detail in a real summary isn't a contradiction, only the single-sentence-one-weak-match shape is. The Matrix/TMNT/hybrid entity-path regression tests still pass, pinned untouched. Effect: falsification-hard rate 6/12 → 10/12 = 0.833 (Insulin / Berlin / 1889 / Pacific now correctly UNGROUNDED). The 2 live-pack fixtures it newly demotes — 5f-fal-live-003 (the exact gap #000046 built its hard pack around) and 5f-fal-live-028 — had expected_reason updated HYBRID_ENTITY → UNGROUNDED (the live pack records what verify_quotes actually does). Remaining hard-pack headroom: 2 STRICT_PARAPHRASE recombinations (Mercury, Einstein — step 2.2) + 8 Formulate mis-segments (step 2.4). Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells) after (bench/qa_results/2026-05-11T17-12-41Z) vs the pre-step-2.1 baseline (...T14-19-51Z = HEAD's verify.py). STRICT-rate quote 0.50→0.54, pointer 0.25→0.22, lattice 0.45→0.43 — all within the 5-pp noise floor. Per-row diff (675 common cells, 30 quote-mode rows changed audit_mode): 0 quote-mode rows demoted to UNGROUNDED from the entity path — the gate fired on 0 legitimate QA answers in the whole bench. Every transition was LLM re-answer variance (verifier quote→quote with the verdict flipping); pointer/lattice deltas are noise too (the gate is in verify_quotes / quote mode, not the claim-lattice verifier). No regression — the gate is provably narrow on real traffic. Summarized in qa-modes-bench.md Addendum 6 + ticket-000048 §5 step 2.1. Tests: 4 new in test_verify.py (_is_single_sentence helper, _entity_salient_disagrees helper, swapped-subject → UNGROUNDED, gate-narrow-on-multi-claim); test_5f_falsification_hard_pack_below_ceiling re-pinned 6/12 → 10/12; test_fork_score_positive_gamma_5f_... updated (positive γ·Δ5f on the real lift — possibly MARGINAL given the ÷5 dilution; ACCEPT via a degraded-parent sub-scenario). make test 2343 passed, 28 skipped. #000048 → step 2.1 landed; #000046 / #000012 §8 / TICKETS.md / Makefile / fixture _meta + notes / baseline JSON updated.
This commit is contained in:
parent
06f5a11651
commit
9899a33b7b
12 changed files with 340 additions and 91 deletions
|
|
@ -92,9 +92,9 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | open · awaiting go/no-go · doc-only; closes the headroom #000046 left (6 over-grounds in `falsification-hard` + 8 mis-segments in `formulate-hard`). Recommends: 2.1 entity salient-token-disagreement gate (mirrors #000046's numeric gate) → 2.4 `parse_pointer_claims` clause segmentation → 2.2 sequence-aware paraphrase; defer 2.3 mini-NLI. Each bench-gated. #000046 follow-up | 2026-05-11 | — |
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | in progress · **step 2.1 landed 2026-05-11** — `verify_quotes` entity salient-token-disagreement gate (`_entity_salient_disagrees` + `_is_single_sentence`; demotes a single short sentence whose lone non-clustered proper-noun match co-occurs with a >4-char capitalized token / number the source lacks) → `falsification-hard` 6/12 → 10/12; bench-gated (`make bench-qa` n=3×75×3 — 0 gate-attributable QA shifts, deltas within the 5-pp floor); 2 live-pack fixtures' `expected_reason` updated HYBRID_ENTITY→UNGROUNDED. Steps 2.4 (`parse_pointer_claims` clause segmentation) + 2.2 (sequence-aware paraphrase) next; defer 2.3 mini-NLI. #000046 follow-up | 2026-05-11 | — |
|
||||
| #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom (6 over-grounds + 8 mis-segments) = optional bigger-verifier-upgrade follow-up. `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom (2 over-grounds + 8 mis-segments after #000048 step 2.1 closed the 4 entity over-grounds) = #000048. `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 follow-up | 2026-05-11 | — |
|
||||
| #000045 | Prometheus-Σ Phase 3 sleep-sweep scheduler (gating ticket) | open · doc-only scaffold 2026-05-10; pins 8 governance parameters + 4 retrigger gates; opens implementation only after one retrigger fires | 2026-05-10 | — |
|
||||
| #000044 | AUTOCOUNT doc-drift discipline | closed · landed across `fc5ba50` / `03c0f6a` / `6c6defb` / `f5dbfab` / `3b30126` 2026-05-10 (mechanism + 4 metrics + 54 tags across 7 doc files; harness catches drift at test time, refresh is 60-second turnaround) | 2026-05-10 | — |
|
||||
| #000043 | Empirical tightening of T3 bound constants (#000036 Phase 2) | open · awaiting go/no-go (parks until v7 deployment surfaces measurement data) | 2026-05-10 | — |
|
||||
|
|
|
|||
|
|
@ -442,3 +442,42 @@ Bench artifacts: `bench/qa_results/2026-05-11T13-42-38Z.{jsonl,md}`
|
|||
(before) · `bench/qa_results/2026-05-11T14-19-51Z.{jsonl,md}` (after).
|
||||
Full per-ticket detail: `docs/tickets/ticket-000046-harder-5sf-fixture-tier.md`
|
||||
§5 Phase 3.
|
||||
|
||||
|
||||
### Addendum 6 — #000048 step 2.1 verifier entity-gate regression check (2026-05-11)
|
||||
|
||||
`arborist/qa/verify.py` gained an entity salient-token-disagreement
|
||||
gate (`_entity_salient_disagrees` + `_is_single_sentence`): in
|
||||
`verify_quotes`' entity branch (proximity policy), the *weakest*
|
||||
grounding — `not cluster AND len(verified) <= 1 AND single sentence` —
|
||||
is declined (→ UNGROUNDED) when the answer asserts a > 4-char
|
||||
capitalized content token (stopword-filtered) or a digit-number the
|
||||
source lacks. Catches "Insulin was discovered by Alexander Fleming"
|
||||
against "Penicillin was discovered by Alexander Fleming" (shared
|
||||
"Alexander Fleming" matched; swapped subject "Insulin" trips the
|
||||
gate). Lifts the `falsification-hard-v1.jsonl` rate 6/12 → 10/12 (the
|
||||
4 HYBRID_ENTITY over-grounds: Insulin / Berlin / 1889 / Pacific).
|
||||
|
||||
Before/after `make bench-qa` (n=3 × 75 questions × 3 modes = 675
|
||||
cells):
|
||||
|
||||
| mode | STRICT-rate before → after | Δ |
|
||||
|------|----------------------------|---|
|
||||
| quote | 0.50 → 0.54 | +4pp |
|
||||
| claim_lattice_pointer | 0.25 → 0.22 | −3pp |
|
||||
| claim_lattice | 0.45 → 0.43 | −2pp |
|
||||
|
||||
All within the 5-pp noise floor. Per-row diff (675 common cells, 30
|
||||
quote-mode rows changed audit_mode): **0 quote-mode rows demoted to
|
||||
UNGROUNDED from the entity path** — the gate didn't fire on a single
|
||||
legitimate QA answer in the whole bench. Every quote-mode transition
|
||||
was LLM re-answer variance (`verifier quote→quote` with the verdict
|
||||
flipping = a different answer text); the pointer/lattice deltas are
|
||||
noise too (the gate is in `verify_quotes` / quote mode, not the
|
||||
claim-lattice verifier). The gate is provably narrow on real traffic;
|
||||
it ships.
|
||||
|
||||
Bench artifacts: `bench/qa_results/2026-05-11T14-19-51Z.{jsonl,md}`
|
||||
(before — HEAD's verify.py) · `bench/qa_results/2026-05-11T17-12-41Z.{jsonl,md}`
|
||||
(after). Full per-ticket detail: `docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md`
|
||||
§5 step 2.1.
|
||||
|
|
|
|||
|
|
@ -575,12 +575,13 @@ protocol must account for:
|
|||
`validator_diversity`, …), not by bench Δ-rate. The harder tier
|
||||
is **#000046 (closed 2026-05-11)**: two below-ceiling 5F packs
|
||||
(`falsification-hard-v1.jsonl`, `formulate-hard-v1.jsonl`) +
|
||||
`verify_quotes`' paraphrase numeric-agreement gate, which lifted
|
||||
the falsification-hard rate 4/12 → 6/12 on a *real* change
|
||||
(bench-gated: `make bench-qa` n=3 before/after showed no
|
||||
STRICT-rate regression on legit answers) → `fork_score`'s `γ·Δ5f`
|
||||
went positive on it. So the bench Δ-rate *does* now carry signal
|
||||
`verify_quotes`' paraphrase numeric-agreement gate + (#000048 step
|
||||
2.1) the entity salient-token-disagreement gate, which together
|
||||
lifted the falsification-hard rate 4/12 → 10/12 on *real* changes
|
||||
(each bench-gated: `make bench-qa` n=3 before/after showed no
|
||||
STRICT-rate regression on legit answers — #000048 step 2.1's gate
|
||||
fired on 0 QA answers in the whole bench) → `fork_score`'s `γ·Δ5f`
|
||||
went positive on them. So the bench Δ-rate *does* now carry signal
|
||||
on the 5F/falsification axis, demonstrated end-to-end. Remaining
|
||||
hard-pack headroom (6 over-grounds + 8 mis-segments) is an
|
||||
optional bigger-verifier-upgrade follow-up, not a #000012
|
||||
blocker.
|
||||
hard-pack headroom (now 2 over-grounds + 8 mis-segments — #000048
|
||||
steps 2.2 + 2.4) is an optional follow-up, not a #000012 blocker.
|
||||
|
|
|
|||
|
|
@ -280,21 +280,18 @@ testable only once there's a below-ceiling baseline to bench against.
|
|||
|
||||
### Headroom (optional follow-up — not a #000046 blocker)
|
||||
|
||||
6 over-grounds remain in the falsification-hard pack (4 HYBRID_ENTITY
|
||||
where the entity-proximity strategy matches on a shared proper noun
|
||||
while the answer's other salient term is wrong; 2 STRICT_PARAPHRASE
|
||||
where the false claim recombines source tokens into a different true
|
||||
statement — "Mercury is the largest" — and lexical token-coverage
|
||||
genuinely can't tell recombination from grounding), plus the 8
|
||||
mis-segments in the Formulate hard pack. Closing those needs a
|
||||
bigger verifier upgrade — an order/dependency-aware match or a
|
||||
contradiction check for `verify_quotes`, and sentence/clause
|
||||
segmentation for `parse_pointer_claims` — **tracked as #000048**
|
||||
(opened 2026-05-11; recommends an entity salient-token-disagreement
|
||||
gate first, then `parse_pointer_claims` clause segmentation, then a
|
||||
sequence-aware paraphrase match; each bench-gated). #000046 itself is
|
||||
done: the below-ceiling baselines exist, a real surface improvement
|
||||
lifted a rate, and ForkScore's bench-Δ went positive on it.
|
||||
Tracked as **#000048** (opened 2026-05-11). As of #000048 step 2.1
|
||||
(2026-05-11) the falsification-hard pack is at 10/12 — the entity
|
||||
salient-token-disagreement gate closed the 4 HYBRID_ENTITY
|
||||
over-grounds. The remaining headroom: 2 STRICT_PARAPHRASE in the
|
||||
falsification-hard pack (hard-003 Mercury, hard-005 Einstein — the
|
||||
false claim recombines source tokens into a different true statement,
|
||||
which lexical token-coverage can't tell from grounding; #000048 step
|
||||
2.2 = sequence-aware paraphrase) + the 8 mis-segments in the Formulate
|
||||
hard pack (line/bullet-only `parse_pointer_claims`; #000048 step 2.4 =
|
||||
clause segmentation). #000046 itself is done: the below-ceiling
|
||||
baselines exist, real surface improvements lifted a rate, and
|
||||
ForkScore's bench-Δ went positive on them.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,10 @@
|
|||
# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Status:** in progress · **step 2.1 landed 2026-05-11** (entity
|
||||
salient-token-disagreement gate in `verify_quotes` — `_entity_salient_disagrees`
|
||||
+ `_is_single_sentence`; lifts `falsification-hard` 6/12 → 10/12;
|
||||
bench-gated, 0 gate-attributable QA shifts). Steps 2.4 (`parse_pointer_claims`
|
||||
segmentation) + 2.2 (sequence-aware paraphrase) next per §3 / §5.
|
||||
**Opened:** 2026-05-11
|
||||
**Scope:** Close the headroom #000046 left in the two below-ceiling 5F
|
||||
hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl`
|
||||
|
|
@ -182,13 +186,68 @@ cost.
|
|||
|
||||
## 5. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Doc-only proposal. Opens when fox picks
|
||||
a starting point (recommendation: 2.1 / entity salient-token gate).
|
||||
The implementation commit chain it gates is the §3 sequence:
|
||||
2.1 → 2.4 → 2.2, each bench-gated, each re-pinning the hard-pack rate
|
||||
+ documenting the `make bench-qa` (or Formulate-fixtures) before/after.
|
||||
Closure: both `falsification-hard-v1.jsonl` and `formulate-hard-v1.jsonl`
|
||||
at (or near) rate 1.0, or fox-decided "residue not worth it".
|
||||
**In progress — step 2.1 landed 2026-05-11.** Steps 2.4 + 2.2 next
|
||||
per §3.
|
||||
|
||||
### Step 2.1 (landed 2026-05-11) — entity salient-token-disagreement gate
|
||||
|
||||
- **Change:** `arborist/qa/verify.py` — `_entity_salient_disagrees(answer_text,
|
||||
norm_ctx)` (a > 4-char Capitalized content token, stopword-filtered,
|
||||
or a digit-number in the answer that's absent from the source) +
|
||||
`_is_single_sentence(text)`. Gated in `verify_quotes`' entity branch
|
||||
(proximity policy) in the *weakest-grounding* slot only — `not
|
||||
cluster AND len(verified) <= 1 AND _is_single_sentence(answer_text)
|
||||
AND _entity_salient_disagrees(...)` → `UNGROUNDED`. The narrow
|
||||
caller-gate is what keeps a structured multi-claim summary (the
|
||||
Matrix cast list — many entities, a tight cluster; the TMNT answer —
|
||||
a numbered list with parenthetical nicknames the source omits)
|
||||
untouched: model-added accurate detail in a real summary isn't a
|
||||
contradiction, only the single-sentence-one-weak-match shape is.
|
||||
- **Effect on the hard pack:** lifts `5f-fal-hard-001/002/006/008`
|
||||
(Insulin / Berlin / 1889 / Pacific — the 4 HYBRID_ENTITY
|
||||
over-grounds) to UNGROUNDED → **falsification-hard rate 6/12 →
|
||||
10/12 = 0.833**. The remaining 2 (hard-003 Mercury, hard-005
|
||||
Einstein — recombination STRICT_PARAPHRASE) are step 2.2's job.
|
||||
- **Live-pack bookkeeping:** `5f-fal-live-003` (the *exact* gap #000046
|
||||
built its hard pack around — Insulin/Fleming) and `5f-fal-live-028`
|
||||
(Eiffel/London) had `expected_reason` updated `HYBRID_ENTITY →
|
||||
UNGROUNDED` — the live pack records what `verify_quotes` actually
|
||||
does, and now it does the right thing.
|
||||
- **Bench gate:** `make bench-qa` (n=3 × 75 questions × 3 modes = 675
|
||||
cells) after (`bench/qa_results/2026-05-11T17-12-41Z.md`) vs the
|
||||
pre-step-2.1 baseline (`...T14-19-51Z.md`, which is HEAD's verify.py).
|
||||
STRICT-rate: quote 0.50 → 0.54, pointer 0.25 → 0.22, lattice 0.45 →
|
||||
0.43 — all within the 5-pp noise floor. **Per-row diff: 0 quote-mode
|
||||
rows demoted to UNGROUNDED from the entity path** — the gate didn't
|
||||
fire on a single legitimate QA answer in the whole bench. Every
|
||||
quote-mode `audit_mode` change was LLM re-answer variance
|
||||
(`verifier quote→quote` with the verdict flipping = a different
|
||||
answer); the pointer/lattice deltas are noise too (the gate is in
|
||||
`verify_quotes` / quote mode, not the claim-lattice verifier). No
|
||||
regression — the gate is provably narrow on real traffic.
|
||||
- **Tests:** `tests/test_verify.py` — `test_is_single_sentence_helper`,
|
||||
`test_entity_salient_disagrees_helper`,
|
||||
`test_entity_path_swapped_subject_demoted_to_ungrounded`,
|
||||
`test_entity_gate_narrow_multi_claim_summary_unaffected` (the
|
||||
Matrix / TMNT / hybrid entity-path regression tests still pass —
|
||||
pinned untouched); `tests/test_bench_batteries.py`
|
||||
`test_5f_falsification_hard_pack_below_ceiling` re-pinned 6/12 →
|
||||
10/12; `test_fork_score_positive_gamma_5f_on_hard_falsification_improvement`
|
||||
updated (positive γ·Δ5f on the real lift — possibly MARGINAL given
|
||||
the ÷5 dilution; ACCEPT shown via a degraded-parent sub-scenario).
|
||||
|
||||
### Still ahead
|
||||
|
||||
- **Step 2.4** — `parse_pointer_claims` clause segmentation (closes
|
||||
the 8 mis-segments in `formulate-hard-v1.jsonl`). Bench gate: the
|
||||
Formulate fixtures (incl. `formulate-v1.jsonl` / `formulate-live-v1.jsonl`,
|
||||
must not regress) + a QA smoke (`parse_pointer_claims` feeds the
|
||||
claim-lattice modes).
|
||||
- **Step 2.2** — sequence-aware paraphrase match (closes hard-003 +
|
||||
hard-005). Bench-gated like 2.1; conservative threshold.
|
||||
- **Closure:** both hard packs at (or near) rate 1.0, or fox-decided
|
||||
"residue not worth it" — at which point this ticket closes (and
|
||||
step 2.3 / mini-NLI stays a deferred maybe).
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue