fox 2026-05-12: the attribution-aware path (#000048's deferred §2.3 — closing the 2 recombination over-grounds in falsification-hard) is its own ticket, not a #000048 phase. So: #000048 → closed (at 2.1 + 2.4). Steps 2.1 + 2.4 landed 2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate mis-segments → formulate-hard 12/12, falsification-hard 10/12; each bench-gated, no STRICT-rate regression). Step 2.2 (single-clause- containment paraphrase check) attempted + reverted — recombination- vs-summary isn't lexical (§5 "What we learned"). The 2 residual falsification-hard fixtures (hard-003 Mercury, hard-005 Einstein) stand as a documented marker of where the lexical verifier stops. Header + §5 Closure + §2.3 updated; cross-refs in #000046 / #000012 §8 / TICKETS.md repointed from "#000048 §2.3" to "#000049". #000049 opened (doc-only, awaiting go/no-go) — "Attribution-aware grounding check (the recombination boundary)". The recombination class needs an attribution / dependency-parse or mini-NLI check (distinguishing "Mercury is the largest" against "Jupiter is the largest; Mercury is the smallest" from a legit cross-sentence summary). Options: 2.1 hand-rolled dependency-attribution heuristic (no model, brittle — same threshold-can't-separate problem one rung up); 2.2 small purpose-built NLI model ([nli] extra, policy-gated, off-by-default, bench-gated — the right capability, but forces the "is a fixed NLI model an LLM-judge?" discipline call + a model dependency + a non-determinism surface to pin); 2.3 do nothing (the 2 fixtures are a boundary marker, no observed real-traffic harm). Recommends 2.3 until real-traffic recombination over-grounds show up, then 2.2 *if* fox rules a fixed NLI model is acceptable in the proof-adjacent path; the first decision the ticket needs is that discipline question. Next ID 000049 → 000050. #000048 follow-up. Doc-only — no code change.
This commit is contained in:
parent
2ad2dceada
commit
20f6061f83
5 changed files with 253 additions and 34 deletions
|
|
@ -92,9 +92,10 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | in progress · **steps 2.1 + 2.4 landed 2026-05-11; step 2.2 attempted + reverted.** 2.1: `verify_quotes` entity salient-token-disagreement gate (`_entity_salient_disagrees` + `_is_single_sentence`) → `falsification-hard` 6/12 → 10/12. 2.4: `parse_pointer_claims` clause segmentation (`_SEGMENT_SEP_RE` split kept only when every segment is well-pointered; leading colon-header dropped; wrapped-bullet join) → `formulate-hard` 4/12 → 12/12 (at ceiling). Each bench-gated (`make bench-qa` n=3×75×3 — 0 / 7-lattice-cells-touched-both-correct; all deltas within the 5-pp floor). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED. 2.2 (single-clause-containment paraphrase check): catches the 2 recombination fixtures but also rejects legit cross-sentence summaries (the Batman case) with no threshold separating the two — recombination-vs-summary isn't lexical; reverted. **Recommendation: close at 2.1+2.4** (12 of 16 residual items closed; the 2 `falsification-hard` recombination fixtures a documented marker; an attribution-aware verifier = §2.3, its own ticket if ever). #000046 follow-up; awaiting fox's close-or-keep-open call | 2026-05-11 | — |
|
||||
| #000049 | Attribution-aware grounding check (the recombination boundary) | open · awaiting go/no-go · doc-only; the home for #000048's deferred §2.3 — closing the 2 recombination over-grounds in `falsification-hard` (hard-003 Mercury / hard-005 Einstein) needs an attribution / dependency-parse or mini-NLI check, which is *not lexical* (#000048 §5). First decision = the discipline question: may a small fixed purpose-built NLI/entailment *model* influence `audit_mode`? (vs the "no LLM-as-judge" rule). Recommends Option 2.3 (do nothing — the 2 fixtures are a boundary marker) until real-traffic recombination-over-grounds show up, then Option 2.2 (`[nli]` extra, policy-gated, off-by-default, bench-gated) *if* fox rules a fixed NLI model is acceptable. #000048 follow-up | 2026-05-12 | — |
|
||||
| #000048 | Verifier upgrade — recombination-aware grounding + clause segmentation | **closed · 2026-05-12** — steps 2.1 + 2.4 landed 2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate mis-segments → `formulate-hard` 12/12, `falsification-hard` 10/12; each bench-gated, no STRICT-rate regression — 2.1's gate fired on 0 QA answers, 2.4's segmenter touched 7 of 450 lattice cells both verdict changes correct). Step 2.2 (single-clause-containment paraphrase check) attempted + reverted — catches the 2 recombination fixtures but also rejects legit cross-sentence summaries with no threshold separating the two; recombination-vs-summary isn't lexical (§5 "What we learned"). The attribution-aware path moved to **#000049** (fox 2026-05-12). 2 live-pack `expected_reason` updated HYBRID_ENTITY→UNGROUNDED; 12+ tests; `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`. #000046 follow-up; #000047 closed | 2026-05-11 | — |
|
||||
| #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom now down to 2 falsification-hard over-grounds (#000048 step 2.1 closed the 4 entity over-grounds; step 2.4 closed the 8 Formulate mis-segments → that pack 12/12; step 2.2 attempted + reverted — the last 2 recombination fixtures need an attribution-aware verifier, #000048 §2.3, and stand as documented residue). `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 follow-up | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | **closed · 2026-05-11** — Phase 1 `falsification-hard-v1.jsonl` (12 near-misses) + Phase 2 `formulate-hard-v1.jsonl` (12 mis-segments, rate 4/12) + Phase 3 `verify_quotes` paraphrase numeric-agreement gate (`_numeric_signature`; demotes a token-covering span asserting a digit-number the source lacks modulo thousands-comma) → falsification-hard rate 4/12 → 6/12 on a real change; bench-gated (`make bench-qa` n=3×75×3 before/after — no STRICT-rate regression on legit answers; only gate-caused QA shift was correctly demoting a fictional-year claim STRICT→HYBRID); `fork_score` γ·Δ5f went positive on it. Headroom now down to 2 falsification-hard over-grounds (#000048 step 2.1 closed the 4 entity over-grounds; step 2.4 closed the 8 Formulate mis-segments → that pack 12/12; step 2.2 attempted + reverted — the last 2 recombination fixtures need an attribution-aware verifier, now tracked as **#000049**, and stand as documented residue). `make bench-5f-falsification-hard` / `bench-5f-formulate-hard` / `bench-fork-baseline-hard`; 7+ tests. #000025 §10.14 follow-up; #000047 closed; #000048 closed | 2026-05-11 | — |
|
||||
| #000045 | Prometheus-Σ Phase 3 sleep-sweep scheduler (gating ticket) | open · doc-only scaffold 2026-05-10; pins 8 governance parameters + 4 retrigger gates; opens implementation only after one retrigger fires | 2026-05-10 | — |
|
||||
| #000044 | AUTOCOUNT doc-drift discipline | closed · landed across `fc5ba50` / `03c0f6a` / `6c6defb` / `f5dbfab` / `3b30126` 2026-05-10 (mechanism + 4 metrics + 54 tags across 7 doc files; harness catches drift at test time, refresh is 60-second turnaround) | 2026-05-10 | — |
|
||||
| #000043 | Empirical tightening of T3 bound constants (#000036 Phase 2) | open · awaiting go/no-go (parks until v7 deployment surfaces measurement data) | 2026-05-10 | — |
|
||||
|
|
@ -143,4 +144,4 @@ Newest first. Update on every open/close.
|
|||
|
||||
## Next ID
|
||||
|
||||
`000049`
|
||||
`000050`
|
||||
|
|
|
|||
|
|
@ -589,5 +589,5 @@ protocol must account for:
|
|||
over-grounds — #000048 step 2.2 (a lexical clause-containment
|
||||
check) was attempted and reverted (can't tell recombination from a
|
||||
legit cross-sentence summary; #000048 §5), so closing those needs
|
||||
an attribution-aware verifier (#000048 §2.3, its own ticket if
|
||||
an attribution-aware verifier (#000049, its own ticket if
|
||||
wanted) — an optional follow-up, not a #000012 blocker.
|
||||
|
|
|
|||
|
|
@ -291,8 +291,8 @@ into a different true statement). #000048 step 2.2 (a single-clause-
|
|||
containment lexical check) was *attempted and reverted* — it can't
|
||||
distinguish that recombination from a legitimate cross-sentence
|
||||
summary, and no threshold separates them (#000048 §5 "What we
|
||||
learned"); closing those 2 needs an attribution-aware verifier
|
||||
(#000048 §2.3 — its own ticket if ever wanted), so they stand as
|
||||
learned"); closing those 2 needs an attribution-aware verifier —
|
||||
its own ticket, **#000049** — so they stand as
|
||||
documented residue. #000046 is done: the below-ceiling baselines
|
||||
exist, real surface improvements (numeric gate, entity gate, clause
|
||||
segmenter) lifted rates, and ForkScore's bench-Δ went positive on
|
||||
|
|
|
|||
|
|
@ -1,18 +1,18 @@
|
|||
# Ticket #000048 — Verifier upgrade: recombination-aware grounding + clause segmentation
|
||||
|
||||
**Status:** in progress · **steps 2.1 + 2.4 landed 2026-05-11; step
|
||||
2.2 attempted + reverted (not feasible as a conservative lexical
|
||||
change — see §5).** 2.1 — `verify_quotes` entity salient-token gate
|
||||
(`falsification-hard` 6/12 → 10/12; bench-gated, 0 gate-attributable
|
||||
QA shifts). 2.4 — `parse_pointer_claims` clause segmentation
|
||||
(`formulate-hard` 4/12 → 12/12, at ceiling; bench-gated, 7 lattice
|
||||
cells touched, both verdict changes correct). 2.2 — a single-clause-
|
||||
containment paraphrase check catches the 2 recombination fixtures but
|
||||
also rejects legit cross-sentence summaries with no threshold that
|
||||
separates the two; reverted. **Recommendation: close at 2.1+2.4** —
|
||||
the 2 residual `falsification-hard` recombination fixtures stand as a
|
||||
documented marker; an attribution-aware verifier (§2.3) opens as its
|
||||
own ticket if ever wanted. Awaiting fox's close-or-keep-open call.
|
||||
**Status:** **closed · 2026-05-12** — steps 2.1 + 2.4 landed
|
||||
2026-05-11 (12 of 16 residual items: 4 HYBRID_ENTITY over-grounds +
|
||||
8 Formulate mis-segments → `formulate-hard` 12/12, `falsification-hard`
|
||||
10/12; each bench-gated, no STRICT-rate regression). Step 2.2 (a
|
||||
single-clause-containment paraphrase check) was attempted + reverted
|
||||
— it catches the 2 recombination fixtures but also rejects legit
|
||||
cross-sentence summaries with no threshold separating the two
|
||||
(recombination-vs-summary isn't lexical; §5 "What we learned"). The
|
||||
attribution-aware path for those last 2 (and the broader recombination
|
||||
class) moved to **#000049**; step 2.3 / mini-NLI lives there now, not
|
||||
here. The 2 residual `falsification-hard` fixtures (hard-003 Mercury,
|
||||
hard-005 Einstein) stand as a documented marker of where the lexical
|
||||
verifier stops.
|
||||
**Opened:** 2026-05-11
|
||||
**Scope:** Close the headroom #000046 left in the two below-ceiling 5F
|
||||
hard packs — the 6 over-grounds still in `falsification-hard-v1.jsonl`
|
||||
|
|
@ -138,9 +138,9 @@ learned"). It needs either a small entailment/contradiction model
|
|||
check ("for each content token in the answer, find its
|
||||
subject/predicate in the source; flag if the answer attaches it
|
||||
elsewhere") — both bigger surface than #000048's "lexical,
|
||||
deterministic, no-LLM-judge" scope, and both warrant their own
|
||||
ticket. **Not in scope for #000048**; the 2 residual fixtures are a
|
||||
documented marker of where this would pick up.
|
||||
deterministic, no-LLM-judge" scope. **Moved to #000049** (fox,
|
||||
2026-05-12 — its own ticket); the 2 residual fixtures are a
|
||||
documented marker of where #000049 would pick up.
|
||||
|
||||
### 2.4 `parse_pointer_claims` — sentence/clause segmentation
|
||||
|
||||
|
|
@ -381,18 +381,19 @@ just a single failed attempt:
|
|||
|
||||
### Closure
|
||||
|
||||
Steps 2.1 + 2.4 landed (12 of the original 16 residual items: 4
|
||||
HYBRID_ENTITY over-grounds + 8 Formulate mis-segments closed →
|
||||
`formulate-hard` 12/12, `falsification-hard` 10/12). Step 2.2 found
|
||||
infeasible as a conservative lexical change (above); step 2.3
|
||||
(mini-NLI / attribution) is the only path to the last 2, and it's a
|
||||
model-dependency / bigger-surface move out of #000048's "lexical,
|
||||
deterministic, no-LLM-judge" scope. **Recommendation: close #000048
|
||||
at 2.1+2.4** — the 2 residual `falsification-hard` fixtures (hard-003
|
||||
Mercury, hard-005 Einstein) stand as a documented marker of exactly
|
||||
where the lexical verifier stops; if an attribution-aware verifier is
|
||||
ever wanted, it opens as its own ticket (the natural home for §2.3).
|
||||
Awaiting fox's call (close at 2.1+2.4, or keep open for §2.3).
|
||||
**Closed 2026-05-12 at 2.1+2.4.** Steps 2.1 + 2.4 landed (12 of the
|
||||
original 16 residual items: 4 HYBRID_ENTITY over-grounds + 8 Formulate
|
||||
mis-segments closed → `formulate-hard` 12/12, `falsification-hard`
|
||||
10/12; each bench-gated, no STRICT-rate regression). Step 2.2 found
|
||||
infeasible as a conservative lexical change (above). The
|
||||
attribution-aware path for the last 2 — step 2.3 / mini-NLI /
|
||||
dependency-attribution, the only thing that can distinguish a
|
||||
recombination from a legit cross-sentence summary — **moved to
|
||||
#000049** (fox's call, 2026-05-12: that capability is its own
|
||||
ticket, not a #000048 phase). The 2 residual `falsification-hard`
|
||||
fixtures (hard-003 Mercury, hard-005 Einstein) stand as a documented
|
||||
marker of exactly where the lexical verifier stops; #000049 weighs
|
||||
whether to cross that boundary and at what cost.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
217
docs/tickets/ticket-000049-attribution-aware-grounding-check.md
Normal file
217
docs/tickets/ticket-000049-attribution-aware-grounding-check.md
Normal file
|
|
@ -0,0 +1,217 @@
|
|||
# Ticket #000049 — Attribution-aware grounding check (the recombination boundary)
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-12
|
||||
**Scope:** Decide whether — and if so how — to add a verifier check
|
||||
that catches a *recombination*: a claim whose content tokens are all
|
||||
present in the source but **paired with different subjects/predicates
|
||||
than the answer asserts**, so the answer is false even though it
|
||||
"covers" the source lexically. Doc-only proposal: the problem, the
|
||||
design options (dependency-attribution heuristic vs. mini-NLI model),
|
||||
the discipline trade-off, a recommendation. No code in this ticket.
|
||||
**Audience:** fox + maintainers of `arborist/qa/verify.py` + #000048
|
||||
follow-up readers + anyone weighing "do we put an ML model in the
|
||||
verifier path?".
|
||||
**Hard constraint:** whatever lands here is bench-gated (`make
|
||||
bench-qa` n=3, before/after — STRICT-rate must not regress on legit
|
||||
answers). It MUST NOT break the existing layered verifier
|
||||
(quote→span→entity→paraphrase, first-to-classify); a contradiction
|
||||
signal *demotes*, never promotes. **The "no LLM-as-judge in the
|
||||
verifier" rule still holds** — a small purpose-built NLI/entailment
|
||||
*model* is a discipline question this ticket has to settle explicitly
|
||||
(it is not the same as "ask the answering LLM to grade itself", but
|
||||
it is an ML model in the proof-adjacent path); a dependency-parse
|
||||
heuristic sidesteps that but is brittle. No schema / governance-hash /
|
||||
canonicalization-version change.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
`#000048` (closed 2026-05-12) closed 12 of the 16 residual
|
||||
over-grounds/mis-segments via *lexical* surface improvements: the
|
||||
numeric-agreement gate (`#000046` Phase 3), the entity
|
||||
salient-token-disagreement gate (`#000048` step 2.1 — an *absence*
|
||||
signal: a digit / capitalized token the source doesn't contain), and
|
||||
the `parse_pointer_claims` clause segmenter (`#000048` step 2.4 — a
|
||||
*structure-of-the-model's-own-output* signal). What it could **not**
|
||||
close — and `#000048` §5 "What we learned" spells out why — is the
|
||||
**recombination** class:
|
||||
|
||||
- `5f-fal-hard-003`: "Mercury is the largest planet in the Solar
|
||||
System." against source "Jupiter is the largest planet in the
|
||||
Solar System; **Mercury is the smallest**." — the answer reuses
|
||||
the source's "largest planet in the Solar System" (which is the
|
||||
source's predicate for *Jupiter*) and the source's "Mercury"
|
||||
(whose predicate is *smallest*), pairing them into a falsehood.
|
||||
Token-coverage = 1.0 (every content token is *somewhere* in the
|
||||
source); the entity gate doesn't fire (no token is *absent*);
|
||||
`verify_quotes` returns STRICT_PARAPHRASE or (post-#000048 step
|
||||
2.1, since the paraphrase path now falls through) HYBRID_ENTITY.
|
||||
- `5f-fal-hard-005`: "Albert Einstein developed the theory of
|
||||
evolution." against "Albert Einstein developed the theory of
|
||||
relativity; **Charles Darwin developed evolution**." — same
|
||||
shape: the source's "Albert Einstein developed the theory of"
|
||||
paired with the source's "evolution" (whose subject is *Darwin*).
|
||||
|
||||
`#000048` §5 established that this is **not a lexical question**:
|
||||
a recombination and a legitimate cross-sentence summary ("Batman,
|
||||
who is the alias of Bruce Wayne, lives in Gotham City." over "Batman
|
||||
is the alias of Bruce Wayne. Batman lives in Gotham City.") have the
|
||||
*same* lexical profile (tokens scattered across source clauses), and
|
||||
the recombination's best-single-clause coverage (4/5) sits *above*
|
||||
the legit summary's (4/6) — no token-coverage / clause-containment /
|
||||
bigram threshold separates them in the safe direction. The
|
||||
discriminating fact is **attribution**: in the source, is the
|
||||
answer's token X attached to the same subject/predicate the answer
|
||||
attaches it to? That requires either parsing the source's
|
||||
dependency/predicate structure or an entailment judgment — i.e., a
|
||||
capability the lexical verifier deliberately doesn't have. This
|
||||
ticket is the home for "do we want that capability, and at what
|
||||
cost?".
|
||||
|
||||
(Out-of-scope-of-#000048 was the conclusion; here it's the subject.)
|
||||
|
||||
---
|
||||
|
||||
## 2. Design options
|
||||
|
||||
### 2.1 — dependency-attribution heuristic (no model)
|
||||
|
||||
A small hand-rolled check, run on the paraphrase path (and the entity
|
||||
path's weakest slot) when a span token-covers the source but spans
|
||||
multiple source clauses: for each *content token* in the answer span,
|
||||
find its occurrence(s) in the source; identify the nearest preceding
|
||||
*subject-like* token (a capitalized noun phrase) and the nearest
|
||||
*predicate-like* head; if the answer pairs token X with subject S but
|
||||
in the source X's nearest subject is S′ ≠ S, flag a contradiction →
|
||||
the span is not grounded. No new dependency: just regex/heuristic
|
||||
parsing of "S … (verb) … X" patterns. **Pro:** stays within the
|
||||
"no-model" verifier discipline; deterministic. **Con:** brittle —
|
||||
"nearest capitalized noun before X" is a crude proxy for "X's
|
||||
syntactic subject"; passive voice, appositives, coordination, and
|
||||
list structure all break it; tuning it without over-firing on legit
|
||||
paraphrases (which reorder freely *within* a clause) is the same
|
||||
threshold-can't-separate problem one rung up the abstraction ladder.
|
||||
Likely catches the 2 contrived fixtures with hand-tuning but provides
|
||||
little assurance on the general class.
|
||||
|
||||
### 2.2 — small purpose-built NLI / entailment model (a model dependency)
|
||||
|
||||
Run a tiny sentence-pair entailment model (a few-hundred-MB DeBERTa-
|
||||
or RoBERTa-NLI checkpoint, CPU-fast) on `(source_clause, answer_span)`
|
||||
pairs the lexical path is unsure about; if the best-matching source
|
||||
clause *contradicts* (not just "doesn't entail") the answer span,
|
||||
demote. **Pro:** this is the *right* capability for the recombination
|
||||
class — "Mercury is the largest" vs "Mercury is the smallest" is a
|
||||
textbook contradiction; "Batman lives in Gotham" vs "Batman is the
|
||||
alias of Bruce Wayne" is neutral/entailed, not contradicted. **Con:**
|
||||
(a) **a model in the verifier path** — even though it's *not* "the
|
||||
answering LLM grading itself" (it's a fixed, purpose-built classifier
|
||||
with no shared parameters), it is an ML model whose output influences
|
||||
`audit_mode`, which the "no LLM-as-judge" rule was written to keep
|
||||
out; settling whether a small fixed NLI model is allowed is the
|
||||
*first* decision this ticket needs. (b) a new optional dependency
|
||||
(`transformers` + a checkpoint) — keep it behind a `[nli]` extra so a
|
||||
fresh checkout still needs only python+venv+sqlite3, and gate it
|
||||
behind `entity_policy`-style policy so it's off-by-default until
|
||||
benched. (c) non-determinism risk — pin the checkpoint + run on CPU
|
||||
with a fixed dtype; entailment classifiers are deterministic given
|
||||
that, but it's a thing to verify and pin. (d) latency — only run it
|
||||
on the small subset the lexical path can't resolve.
|
||||
|
||||
### 2.3 — do nothing; the 2 fixtures stand as a marker
|
||||
|
||||
`falsification-hard` at 10/12 is already a real below-ceiling
|
||||
baseline; the 2 recombination fixtures *are* the documentation of
|
||||
where the lexical verifier stops (#000048 §5). Closing them buys
|
||||
2/12 on a contrived pack at the cost of (heuristic brittleness, or a
|
||||
model dependency). If no real QA-bench evidence shows recombination
|
||||
over-grounds happening at scale on actual traffic, this is the
|
||||
honest choice. (No bench evidence of that exists today — the #000048
|
||||
step-2.1/2.4 per-row diffs found 0 gate-attributable and 0
|
||||
segmenter-attributable *legit* harms; they didn't surface
|
||||
recombination-over-grounds in real answers either, because the
|
||||
paraphrase path is a last resort that rarely fires on real Wikipedia
|
||||
QA.)
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommendation
|
||||
|
||||
**Option 2.3 (do nothing) for now — but settle the Option-2.2
|
||||
discipline question if/when recombination over-grounds show up on
|
||||
real traffic.** Rationale:
|
||||
|
||||
1. The lexical verifier is honest about its boundary (the 2 fixtures
|
||||
document it); there's no observed real-traffic harm.
|
||||
2. Option 2.1 (heuristic) is the same threshold-can't-separate
|
||||
problem dressed up — low assurance, real over-fire risk on the
|
||||
QA bench.
|
||||
3. Option 2.2 (NLI model) is the right *capability* but the wrong
|
||||
*cost* until there's evidence it's needed: it forces the
|
||||
"is a fixed NLI model an LLM-judge?" decision, adds a dependency,
|
||||
and adds a non-determinism surface to pin — all for a contrived
|
||||
2/12.
|
||||
4. The trigger to revisit: a `make bench-qa` per-row diff (or a
|
||||
field report) showing a *legitimate-looking but false* answer
|
||||
getting STRICT/HYBRID via the paraphrase/entity path because it
|
||||
recombined source tokens. At that point this ticket reopens with
|
||||
real data, and Option 2.2 (behind a `[nli]` extra, policy-gated,
|
||||
off by default, bench-gated) becomes the path — *if* fox rules a
|
||||
fixed purpose-built NLI model is acceptable in the proof-adjacent
|
||||
path (it's a genuine discipline call; the safe default is "no",
|
||||
in which case 2.3 stands permanently and the 2 fixtures are just
|
||||
permanent boundary markers).
|
||||
|
||||
---
|
||||
|
||||
## 4. Out of scope
|
||||
|
||||
- **The `falsification-hard` / `formulate-hard` packs themselves** —
|
||||
#000046 built them; this ticket would change the verifier surface,
|
||||
not the fixtures.
|
||||
- **`fork_score` / the aggregator / the bench Δ-rate mechanics** —
|
||||
unchanged; this only affects which `audit_mode` `verify_quotes`
|
||||
returns on a recombination span, hence `falsification-hard`'s rate.
|
||||
- **Replacing the lexical verifier wholesale with an embedding/LLM
|
||||
one** — a far bigger architectural move; this ticket is narrowly
|
||||
about an *added* contradiction check, lexical path untouched as the
|
||||
first-line.
|
||||
- **A harder fixture tier** — #000046's territory if the current
|
||||
packs go saturated again.
|
||||
|
||||
---
|
||||
|
||||
## 5. Status
|
||||
|
||||
**Open · awaiting go/no-go.** Doc-only. The first thing fox decides
|
||||
is the **discipline question** in §2.2's con (a): may a small, fixed,
|
||||
purpose-built NLI/entailment model influence `audit_mode`? — `yes` →
|
||||
Option 2.2 becomes viable when triggered (§3 #4); `no` → Option 2.3
|
||||
stands and the 2 recombination fixtures are permanent boundary
|
||||
markers. The second decision (only if `yes` and triggered) is the
|
||||
implementation: a `[nli]` extra + a pinned CPU checkpoint + policy-
|
||||
gated off-by-default + the `make bench-qa` before/after gate. Until
|
||||
then: nothing lands; `falsification-hard` stays 10/12.
|
||||
|
||||
---
|
||||
|
||||
## 6. References
|
||||
|
||||
- `docs/tickets/ticket-000048-verifier-upgrade-recombination-segmentation.md`
|
||||
(closed) — §5 "What we learned" is the argument that recombination
|
||||
isn't lexical; §2.3 there sketched this; this ticket is its home.
|
||||
- `docs/tickets/ticket-000046-harder-5sf-fixture-tier.md` (closed) —
|
||||
built `falsification-hard` (`5f-fal-hard-003` Mercury, `-005`
|
||||
Einstein are the 2 residual recombination fixtures this ticket
|
||||
would close).
|
||||
- `bench/fixtures/5f/falsification-hard-v1.jsonl` — the 2 fixtures.
|
||||
- `arborist/qa/verify.py` — the layered verifier; a contradiction
|
||||
check would attach to the paraphrase path + the entity path's
|
||||
weakest slot (the same places #000046 / #000048 step 2.1's gates
|
||||
attach).
|
||||
- `docs/bench-maxing.md` — the bench discipline (n=3, 5-pp floor,
|
||||
honest verdicts over optimistic) every step here honors.
|
||||
- `docs/seven-point-program.md` D1 ("no LLM in the verifier") — the
|
||||
rule §2.2's con (a) tests against.
|
||||
Loading…
Add table
Add a link
Reference in a new issue