#000049 §7 #27: recall-side realism check — synthetic 28/28 was a 4x over-estimate; bart-large-mnli/k=12/max/θc=0.999 is the new fp=0 leader on real haystacks at 0.48 recall (vs §7 #26's claimed 28/28 → 0.24 on real)
Built bench/fixtures/5f/recombination-realcontext-v1.jsonl: 25 hand-built recombination claims (deliberate re-attributions within real bench-qa 30 KB Wikipedia haystacks the pipeline produced — Mount Kenya into a Kilimanjaro answer, Galileo onto all 63 Jupiter moons, Joshua Reynolds onto the Mona Lisa, etc.). Ran the full 7-model × full-grid sweep vs the 808 pooled STRICT. Result: - bart-large-mnli / k=12 / max / θc=0.999 → 12/25 = 0.48 catch · 0/808 FP (the real-haystack fp=0 leader) - deberta-large-mnli / k=3 / margin / θc=0.995 → 6/25 = 0.24 (§7 #26's 'settled' config — 28/28 synthetic, 0.24 real-haystack: 4× over-estimate) - roberta-large 0.12, MiniLM 0.08, deberta-base 0.04 So §7 #26's 'boundary closed' walks back to 'boundary PARTIALLY closed' on real haystacks. The bottleneck is architectural: top-k by token overlap misses the contradicting clause when it shares few subject-area tokens with the answer (e.g. the Mount Kenya clause only shares 'Kenya' with a Kilimanjaro claim — ranked low, NLI never sees it). Threshold tuning doesn't lift the ceiling; a SEMANTIC candidate selector (vec-driven, sibling of #000050/#000051's hybrid retrieval) does. bart's pareto above fp=0: fp=0.011 catch=0.52, fp=0.057 catch=0.84, fp=0.068 catch=0.92 — permissive operating points are on the menu if fox+dav1d sign off. recommended_operating_point updated to bart-large-mnli/k=12/max/θc=0.999; deberta-large/margin kept as the synthetic-eval reference. Sixth meta-lesson instance: clean synthetic eval doesn't predict bench-qa precision OR recall — neither contrived dataset axis is load-bearing, only the real pipeline shape is. Production verifier unchanged; falsification-hard stays 10/12. Still SHADOW; runtime promotion fox+dav1d-decides.
This commit is contained in:
parent
d57e5c1435
commit
4352b84508
5 changed files with 9234 additions and 58 deletions
|
|
@ -20,19 +20,27 @@
|
|||
},
|
||||
"thresholds_provenance": "the bench-validated operating point — arborist-nli-bench commit a1cb28d, ticket #000049 §7 #18; on the 28+26-case synth set this point gives 28/28 catch, 0/26 false positives, and the entailment guard is what filters the spurious competing-superlative contradictions",
|
||||
"recommended_operating_point": {
|
||||
"_comment": "ticket #000049 §7 #26 — confirmed across n=3=275 AND the pooled n=1+n=3+n=5 = 808 real STRICT cells (the §7 #25 'agg=max @ θc=0.96' was a small-sample artifact — the n=5 444-cell set lacked the high-confidence token-collision spurious hits the pooled set has; on 808 cells θc must go back to ~0.995, and at θc=0.995 'margin' is the ONLY aggregation that still catches 28/28 — 'max' gets 27/28, 'mean' ≤0.75). The {model × k × agg × guard × θc × θe} mega-grid: microsoft/deberta-large-mnli + k=3 + agg='margin' (= max over the top-3 candidate clauses of (p_contra - p_entail) of THAT clause — a single threshold that discriminates a real recombination [source genuinely contradicts → that clause's entail≈0 → margin≈0.996-0.998] from a spurious token-collision FP [the source ALSO contains an entailing clause → its margin drags to ~0.95-0.99]) + θc=0.995 → catches all 28 synthetic recombinations (incl. both 5f-fal-hard fixtures), 0 false positives on 808 real STRICT answers, 0 on the 26 synthetic legit summaries. This is the ONLY config (across the 7-model × full-grid sweep) that hits 1.0/0.0 on the 808-cell denominator. Mining the 808 real STRICT answers found ZERO natural recombination errors — the boundary is theoretical-in-practice; the failure mode the mining surfaced is the candidate selector (top-k by token overlap) pulling different-entity same-subject-area clauses (Mount Kenya into a Kilimanjaro answer, Dalí into da Vinci, Donovan into Superman), which 'margin' survives but a Phase-3 verifier-matched-clause hook would eliminate. Still NOT the active default — pending (a) a synthetic-recombination-vs-real-bench-qa-context set (~20-30 hand-built fixtures: real haystack, deliberate re-attribution — the recall-side realism check), (b) fox+dav1d sign-off. When promoted: switch nli_model_version/hf_repo, wire 'aggregation'='margin'+'guard' into ShadowNLI.check, set max_candidate_clauses=3 + thresholds, and fold nli_policy_hash (model+k+agg+guard+θc) into governance_policy_hash before any audit_mode effect (§7 #2).",
|
||||
"nli_model_version": "nli-shadow-v1-deberta-large-mnli",
|
||||
"hf_repo": "microsoft/deberta-large-mnli",
|
||||
"max_candidate_clauses": 3,
|
||||
"aggregation": "margin",
|
||||
"guard": "max_entail (informational only — 'margin' folds the guard into the score, so θe is N/A)",
|
||||
"thresholds": {"contradiction_minus_entailment_margin_veto": 0.995},
|
||||
"simpler_alternate": {"config": "k=2/agg=max/guard=max_entail/θc=0.995/θe=0.9", "catch_pooled808": "27/28", "note": "the two-threshold variant — one recombination short of margin, but simpler; also 0/808 FP"},
|
||||
"model_alternates_proper_n": [
|
||||
{"hf_repo": "roberta-large-mnli", "best_fp0_pooled808": "21/28 = 0.75", "note": "355M; second-best on the pooled set — does NOT reach 28/28 there (the n=5-only 28/28 was a small-sample read)"},
|
||||
{"hf_repo": "facebook/bart-large-mnli", "best_fp0_pooled808": "~20/28", "note": "407M; bigger ≠ better — it's the specific checkpoint (and the right aggregation)"},
|
||||
{"hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli", "best_fp0_pooled808": "11/28", "note": "184M; collapses at proper-pooled-n — do not use"},
|
||||
{"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768", "best_fp0_pooled808": "14/28", "note": "82M; the §7 #18 'cost-pick' is overturned — caps low once you demand 0 FP at a real denominator"}
|
||||
"_comment": "ticket #000049 §7 #27 — REVISED at recall-side realism. The §7 #26 'deberta-large-mnli + margin@0.995 → 28/28 · 0/808 settled' was a synthetic-recall artifact: on the 25 hand-built real-context recombination fixtures (bench/fixtures/5f/recombination-realcontext-v1.jsonl — false re-attributions inside real bench-qa 30 KB haystacks) the same config catches only 6/25 = 0.24. The clean 28-case eval inflated recall 4x. The new fp=0 leader on real haystacks is bart-large-mnli + k=12 + agg=max + θc=0.999 → 12/25 = 0.48 recall, 0/808 STRICT FP. Diagnosis: the candidate-clause selector (top-k by content-token overlap) misses the contradicting clause when it isn't lexically close to the answer's subject (e.g. for 'Kilimanjaro is in central Kenya' against the real Kilimanjaro article, the Mount Kenya clause only shares 'Kenya' and gets ranked low). The architectural fix that lifts the ceiling is a SEMANTIC candidate selector (vec-driven, sibling of #000050/#000051's hybrid retrieval — embedding similarity finds meaning-similar clauses, not just token-similar) — not threshold tuning. Caveats: still SHADOW; the 25 fixtures are hand-built (representative but not exhaustive); the deberta-large-mnli/margin@0.995 config IS still 28/28 · 0/808 on the synthetic-eval side AND its real-context pareto at fp=0.001 (1 STRICT FP) jumps to catch=0.56 — so a 'near-fp=0' permissive variant is on the menu if dav1d signs off. Runtime promotion is fox+dav1d-decides.",
|
||||
"nli_model_version": "nli-shadow-v1-bart-large-mnli",
|
||||
"hf_repo": "facebook/bart-large-mnli",
|
||||
"max_candidate_clauses": 12,
|
||||
"aggregation": "max",
|
||||
"guard": "max_entail",
|
||||
"thresholds": {"contradiction_veto": 0.999, "entailment_block_veto": 0.5},
|
||||
"real_context_realism_n25_n808": {"catch": "12/25 = 0.480", "fp": "0/808", "note": "the fp=0 frontier point on real-haystack recombinations; bart's score distribution is more spread, so it stays FP-safe at k=12 while other models can't use k=12 without false-firing"},
|
||||
"pareto_above_fp0": [
|
||||
{"config": "bart k=12/max/θc=0.998", "catch": "0.52", "fp": "0.011"},
|
||||
{"config": "bart k=8/top2/θc=0.98", "catch": "0.60", "fp": "0.022"},
|
||||
{"config": "bart k=12/max/θc=0.97", "catch": "0.84", "fp": "0.057"},
|
||||
{"config": "deberta-large k=8/top2/θc=0.95", "catch": "0.56", "fp": "0.001 (1 STRICT FP)", "note": "deberta-large's near-fp=0 knee — single false-demote in 808 buys recall from 0.24 to 0.56"}
|
||||
],
|
||||
"synthetic_eval_alternate": {"hf_repo": "microsoft/deberta-large-mnli", "config": "k=3/agg=margin/θc=0.995", "catch_synth28": "28/28 = 1.000", "catch_realctx25": "6/25 = 0.240", "fp_pooled808": "0/808", "note": "the §7 #26 'settled' config — synthetic-side perfect, real-haystack only 0.24; kept here as a 'best on synth + 0 FP' reference but NOT the runtime recommendation"},
|
||||
"model_science_summary": [
|
||||
{"hf_repo": "facebook/bart-large-mnli", "realctx_recall_fp0": "0.48", "note": "407M; the real-haystack fp=0 winner — uses k=12 cleanly because its score distribution is more spread"},
|
||||
{"hf_repo": "microsoft/deberta-large-mnli", "realctx_recall_fp0": "0.24", "note": "400M; synthetic-set winner but only ~half of bart on real haystacks at fp=0"},
|
||||
{"hf_repo": "roberta-large-mnli", "realctx_recall_fp0": "0.12", "note": "355M"},
|
||||
{"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768", "realctx_recall_fp0": "0.08", "note": "82M; small models cap low on real haystacks too"},
|
||||
{"hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli", "realctx_recall_fp0": "0.04", "note": "184M; collapses on real-haystack recall"}
|
||||
]
|
||||
},
|
||||
"alternates": [
|
||||
|
|
|
|||
26
bench/fixtures/5f/recombination-realcontext-v1.jsonl
Normal file
26
bench/fixtures/5f/recombination-realcontext-v1.jsonl
Normal file
File diff suppressed because one or more lines are too long
9029
bench/results/nli-shadow-grid-realctx25-vs-pooled808.json
Normal file
9029
bench/results/nli-shadow-grid-realctx25-vs-pooled808.json
Normal file
File diff suppressed because it is too large
Load diff
File diff suppressed because one or more lines are too long
|
|
@ -23,50 +23,55 @@ on a 4090) + a `{model × k × agg × guard × θc × θe}` grid sweep
|
|||
(§7 #23–#26, `bench/scripts/nli_shadow_grid.py`, run against bench-qa
|
||||
STRICT samples of growing size: n=1=89 → n=3=275 → n=5=444 → pooled
|
||||
n=1+3+5 = 808) + a mining pass over the 808 real STRICT answers for
|
||||
natural recombinations landed; the verdict moved **five times then
|
||||
settled**: §7 #22 "fails" = default-config artifact; §7 #23
|
||||
"deberta-base passes" = small-FP-sample artifact; §7 #24 found a large
|
||||
model passes at n=3 with `margin`; §7 #25 said "no, at n=5 plain
|
||||
`max`@0.96 works" — but §7 #26 (pooled 808, the biggest real FP
|
||||
denominator) showed *that* was the small-sample artifact: on 808 cells
|
||||
θc must go back to ~0.995 and at θc=0.995 only `margin` still catches
|
||||
28/28. **Settled: `microsoft/deberta-large-mnli` / k=3 / agg=`margin`
|
||||
(= max over top-3 candidate clauses of `p_contra − p_entail`) / θc=0.995
|
||||
→ catches all 28 synthetic recombinations (incl. both `5f-fal-hard`
|
||||
fixtures) · 0/808 pooled real STRICT FP · 0/26 synthetic legit FP** —
|
||||
the *only* config in the 7-model × full-grid sweep that hits 1.0/0.0
|
||||
on 808 cells, and it held at n=3 too (the simpler two-threshold variant
|
||||
k=2/`max`/θc=0.995 gets 27/28 · 0/808). Findings: `margin` is the right
|
||||
score-shape (discriminates a real recombination — source genuinely
|
||||
contradicts, that clause's entail≈0, margin≈0.997 — from a spurious
|
||||
token-collision FP — source also has an entailing clause, margin dragged
|
||||
to ~0.95–0.99); the specific *large* checkpoint matters (deberta-large-mnli
|
||||
wins; bart-large/deberta-base/MiniLM all collapse at proper-pooled-n,
|
||||
so the §7 #18 "MiniLM cost-pick" is overturned); int8-ONNX costs ≈1
|
||||
catch. **Mining the 808 real STRICT answers found ZERO natural
|
||||
recombination errors** — the boundary is theoretical-in-practice; the
|
||||
37 would-fires at lowered θc are all token-collision FPs (Mount Kenya
|
||||
pulled into a Kilimanjaro answer, Dalí into da Vinci, Donovan into
|
||||
Superman) — i.e. the candidate selector (top-k by token overlap) pulls
|
||||
different-entity same-subject-area clauses; `margin@0.995` survives it,
|
||||
a Phase-3 verifier-matched-clause hook would eliminate it. Remaining
|
||||
caveats: the 28/28 recall is a 28-case denominator and (since there are
|
||||
no natural recombinations to mine) the realistic next check is ~20–30
|
||||
hand-built synthetic-recombination-claim-vs-real-bench-qa-context
|
||||
fixtures (real haystack, deliberate re-attribution); still SHADOW —
|
||||
runtime promotion is fox+dav1d-decides (then `nli_policy_hash` =
|
||||
model+k+agg+guard+θc folds into `governance_policy_hash` per §7 #2);
|
||||
deberta-large is ~400M (GPU-fast). Manifest active defaults stay
|
||||
k=6/θc=0.5; `recommended_operating_point` = the `margin`@0.995 config.
|
||||
Standing lesson, sharpened five times (codified in CLAUDE.md
|
||||
bench-maxing): the clean synthetic eval doesn't predict bench-qa
|
||||
precision (§7 #18→#20), the default config doesn't predict the best
|
||||
config (§7 #22→#23), **and a small FP-side sample doesn't predict the
|
||||
large-sample FP rate — and a *bigger* sample can vindicate a config a
|
||||
smaller one made look unnecessary (§7 #23→#24→#25→#26)** — every gate
|
||||
number is provisional until the denominators are big enough, and
|
||||
the sweep has to be wide enough to include the config that survives
|
||||
them. Production verifier unchanged; `falsification-hard` stays 10/12.
|
||||
natural recombinations + a recall-side realism check (§7 #27 — 25
|
||||
hand-built recombination claims against real bench-qa 30 KB haystacks)
|
||||
landed; the verdict moved **six times**: §7 #22 "fails" = default-
|
||||
config artifact; §7 #23 "deberta-base passes" = small-FP-sample
|
||||
artifact; §7 #24/#25/#26 = three more samples relocating the
|
||||
configuration target (large model + `margin`@0.995 settled at pooled
|
||||
808 STRICT); **§7 #27 shows even §7 #26's "settled" was a synthetic-
|
||||
*recall* artifact** — the same deberta-large-mnli/`margin`@0.995 config
|
||||
that gets 28/28 on the 28-case synthetic set gets **6/25 = 0.24** on
|
||||
the 25-case real-haystack set. **Honest current state: the lexical-
|
||||
candidate NLI veto *partially* closes the boundary** — the new fp=0
|
||||
leader on real haystacks is `bart-large-mnli / k=12 / agg=max / θc=0.999`
|
||||
→ **12/25 = 0.48 recall · 0/808 STRICT FP**. *Strictly better than
|
||||
the current 0%* but not the §7 #26 "full closure." The remaining gap
|
||||
is **architectural, not threshold-tuning**: the candidate-clause
|
||||
selector (top-k by content-token overlap) misses the contradicting
|
||||
clause when it shares few subject-area tokens with the answer (e.g.
|
||||
the Mount Kenya clause only shares "Kenya" with a "Kilimanjaro is in
|
||||
central Kenya" claim against the real Kilimanjaro article — gets
|
||||
ranked low, NLI never sees it, can't fire). The Phase-3 lever that
|
||||
lifts the ceiling is **a semantic candidate selector** (vec-driven,
|
||||
sibling of #000050/#000051's hybrid retrieval — embedding similarity
|
||||
finds *meaning-similar* clauses, not just *token-similar*); raising
|
||||
threshold doesn't help. Other findings: `bart-large-mnli` is the new
|
||||
real-context winner (k=12/max/θc=0.999 stays FP-safe because its
|
||||
score distribution is more spread) — both the §7 #18 "MiniLM
|
||||
cost-pick" *and* the §7 #26 "deberta-large + margin" are overturned;
|
||||
the FP-side analysis (`margin@0.995` discriminates spurious from real
|
||||
on the *FP* side) is unaffected; mining the 808 STRICT answers for
|
||||
*natural* recombinations found zero (the boundary is contrived-in-
|
||||
practice on factual-Wikipedia QA, but the synthetic-vs-real-haystack
|
||||
gap on the recall side is what matters for any *future* recombination
|
||||
that emerges in production). Caveats: still SHADOW; the 25 real-
|
||||
context fixtures are hand-built (representative but not exhaustive);
|
||||
the pareto above fp=0 buys substantial recall for small FP cost
|
||||
(bart-large at fp=0.057 catches 0.84; deberta-large at *single* STRICT
|
||||
FP buys catch 0.56) — a permissive operating point is on the menu if
|
||||
fox+dav1d sign off. Runtime promotion is fox+dav1d-decides (then
|
||||
`nli_policy_hash` = model+k+agg+guard+θc folds into
|
||||
`governance_policy_hash` per §7 #2). Manifest active defaults stay
|
||||
k=6/θc=0.5; `recommended_operating_point` updated to
|
||||
`bart-large-mnli / k=12 / max / θc=0.999` (the real-haystack fp=0
|
||||
leader); the deberta-large/margin config kept as a synthetic-eval
|
||||
reference. Standing lesson, sharpened *six* times (codified in
|
||||
CLAUDE.md bench-maxing): clean synthetic eval doesn't predict bench-qa
|
||||
**precision *or* recall** — neither contrived dataset axis is
|
||||
load-bearing; only the real pipeline shape is (§7 #18→#20→#22→#23→
|
||||
#24→#25→#26→**#27**). Production verifier unchanged; `falsification-hard`
|
||||
stays 10/12.
|
||||
**Opened:** 2026-05-12
|
||||
**Scope:** Decide whether — and if so how — to add a verifier check
|
||||
that catches a *recombination*: a claim whose content tokens are all
|
||||
|
|
@ -1255,3 +1260,111 @@ The honest discipline: don't trust *any* denominator until it's the
|
|||
biggest you can afford, and re-confirm the *config choice* (not just
|
||||
the threshold) every time the denominator grows. Pooled 808 STRICT ×
|
||||
the full grid was ~4 min on the 4090.
|
||||
|
||||
**27. The real-context recombination set — synthetic recall *also*
|
||||
inflated; the candidate-selector is the recall-side bottleneck
|
||||
(2026-05-13).** §7 #26's "the boundary is closeable; deberta-large-mnli
|
||||
+ margin@0.995 → 28/28 · 0/808" had one remaining caveat: the 28/28
|
||||
recall was on a *28-case synthetic* set with 1–3-sentence sources. To
|
||||
test recall against a *real* haystack, built `bench/fixtures/5f/recombination-realcontext-v1.jsonl`
|
||||
— 25 hand-crafted recombination claims, each a deliberate
|
||||
*re-attribution within a real bench-qa context* (8–64 KB Wikipedia
|
||||
haystack the pipeline actually produced for the curated bench
|
||||
questions; every claim FALSE; every content token present in the
|
||||
source somewhere): "Mount Kilimanjaro is in central Kenya, just south
|
||||
of the equator" (against the real Kilimanjaro article — re-attributes
|
||||
Mount Kenya's location); "Galileo Galilei discovered all 63 of
|
||||
Jupiter's confirmed moons" (against the real "Moons of Jupiter"
|
||||
article — Galileo discovered 4); "Joshua Reynolds painted the Mona
|
||||
Lisa" (against the real Mona Lisa article — Reynolds is mentioned
|
||||
there as a critic); and so on across 25 questions × domains. Then ran
|
||||
the full 7-model × full-grid sweep against these 25 (recall) vs the
|
||||
808 pooled real STRICT (FP):
|
||||
|
||||
| NLI model (size) | best `fp=0` config | recall on *real-context* recombinations (25) | FP (808 pooled STRICT) |
|
||||
|---|---|---|---|
|
||||
| **`bart-large-mnli` (~407M)** | **k=12, agg=max, θc=0.999, θe=0.5** | **12/25 = 0.480** | **0/808** |
|
||||
| `microsoft/deberta-large-mnli` (~400M) | k=2/3, agg=max, θc=0.995 | 6/25 = 0.240 | 0/808 |
|
||||
| `roberta-large-mnli` (~355M) | k=8, agg=max, θc=0.998 | 3/25 = 0.120 | 0/808 |
|
||||
| MiniLM-82M | k=4, agg=top3, θc=0.975 | 2/25 = 0.080 | 0/808 |
|
||||
| `deberta-base-184M` | k=3, agg=mean, θc=0.995 | 1/25 = 0.040 | 0/808 |
|
||||
|
||||
**The §7 #26 synthetic 28/28 was a 4× over-estimate of real-haystack
|
||||
recall.** Same model (`deberta-large-mnli`), same config (k=3/margin/
|
||||
θc=0.995) — synthetic recall 28/28 = 1.000; real-context recall 6/25
|
||||
= **0.240**. **The clean synthetic eval didn't predict bench-qa
|
||||
precision *or* real-haystack recall** — sixth instance of the
|
||||
denominator/realism lesson (instances 1–5 were all on the FP side;
|
||||
this one is the recall-side mirror).
|
||||
|
||||
**Diagnosis: the candidate-clause selector is the recall-side
|
||||
bottleneck.** Top-k by content-token overlap picks clauses *similar to
|
||||
the answer's subject area*; for a recombination claim like
|
||||
"Kilimanjaro is in central Kenya", the overlap-ranked top-k of the
|
||||
30 KB Kilimanjaro article are about the mountain itself (height,
|
||||
climbing routes, climate, glaciers) — clauses that share many tokens
|
||||
with the claim but *don't contradict it* — while the *Mount Kenya*
|
||||
clause (which would contradict by re-attribution) shares only "Kenya"
|
||||
and gets ranked low. So NLI never sees the contradicting clause and
|
||||
can't fire. This is the *recall mirror* of §7 #26's FP failure mode
|
||||
(the selector pulling subject-area-sharing-but-different-entity
|
||||
clauses): the same lexical proxy that pulls in spurious entities
|
||||
*also fails to pull in genuinely-contradicting ones*. Token overlap
|
||||
is the wrong tool for both.
|
||||
|
||||
**Model-science update at proper recall-side realism:** `bart-large-mnli`
|
||||
is **the new fp=0 leader on real haystacks** (12/25 at k=12/max/θc=0.999)
|
||||
— a complete reversal of §7 #24's "bart is the threshold-robust pick"
|
||||
narrative and §7 #26's "deberta-large-mnli is the winner." Why bart
|
||||
wins on real haystacks: its score distribution is more spread, so at
|
||||
very high θc (0.999) it stays FP-safe even at k=12 — and k=12 is what
|
||||
*finds* the contradicting clause when token overlap doesn't rank it
|
||||
top-2. The other large models can't use k=12 without false-firing.
|
||||
The §7 #18 "MiniLM cost-pick" remains overturned; the §7 #26
|
||||
"deberta-large is the winner" is now *also* overturned at recall-side
|
||||
realism.
|
||||
|
||||
**The precision/recall frontier on real haystacks** (bart-large-mnli
|
||||
pareto): fp=0.000 catch 0.48; fp=0.011 catch 0.52; fp=0.022 catch 0.60;
|
||||
fp=0.035 catch 0.64; fp=0.057 catch 0.80; fp=0.067 catch 0.88;
|
||||
fp=0.068 catch 0.92. So the *gate-passable* (fp=0) operating point
|
||||
catches ~half of real-haystack recombinations; pushing to 80–90%
|
||||
recall costs 3–7% false-demote — that's not gate-passable.
|
||||
deberta-large-mnli's pareto is similar: fp=0 catch 0.24 → fp=0.001
|
||||
catch 0.56 (a single STRICT FP buys huge recall via k=8) → fp=0.013
|
||||
catch 0.72.
|
||||
|
||||
**Revised verdict (the honest one for dav1d).** The lexical-candidate
|
||||
NLI veto **partially closes the recombination boundary on real
|
||||
haystacks** — ~48% recall at strictly 0 FP on 808 real STRICT cells,
|
||||
with `bart-large-mnli / k=12 / agg=max / θc=0.999`. That is *strictly
|
||||
better than the current verifier's 0%* but well short of the §7 #26
|
||||
"settled, 28/28" claim, which was a synthetic-set artifact. The
|
||||
remaining gap is **architectural, not threshold-tuning**: the
|
||||
candidate-clause selector (top-k by token overlap) misses the
|
||||
contradicting clause for the majority of real-haystack recombinations.
|
||||
The Phase-3 lever that lifts the ceiling is **a semantic candidate
|
||||
selector** (e.g. vec-driven, sibling of #000050/#000051's hybrid
|
||||
retrieval — embedding similarity finds *meaning-similar* clauses, not
|
||||
just *token-similar*); a verifier-matched-clause hook *also* doesn't
|
||||
help recall directly (for a false claim the verifier may not match
|
||||
cleanly) — but a better selector does. Caveats: still SHADOW; the 25
|
||||
real-context fixtures are hand-built (representative but not
|
||||
exhaustive); runtime promotion is fox+dav1d-decides. Manifest active
|
||||
defaults stay k=6/θc=0.5; `recommended_operating_point` updated to
|
||||
`bart-large-mnli / k=12 / max / θc=0.999` (the fp=0 leader on real
|
||||
haystacks), with the synthetic-set deberta-large config kept as a
|
||||
documented alternate.
|
||||
|
||||
*Meta-lesson, instance six (and the one that closes the loop):*
|
||||
the discipline isn't only "don't trust small *FP* denominators" — it
|
||||
is **"don't trust *any* contrived dataset on *either* axis until it
|
||||
has been re-confirmed on the real pipeline shape."** Synthetic eval
|
||||
predicts neither real precision (§7 #18→#20) nor real recall (§7
|
||||
#26→#27). The 28 synthetic recombinations had short 1–3-sentence
|
||||
sources where every clause was a candidate; the 25 real-context
|
||||
recombinations have 30 KB sources where the contradicting clause is
|
||||
buried — those are *different* problems, and only the second one is
|
||||
the production problem. The 4090 made this measurable in ~4 min
|
||||
(7 models × full grid × 833 records); without it we would have
|
||||
shipped a 24%-recall config under a "1.000" label.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue