engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000049 §7 #23: grid sweep flips the verdict — the lexical-candidate NLI veto PASSES the gate

Browse source 
The §7 #22 'fails the gate' was the verdict for the DEFAULT config
(k=6/θc=0.5/θe=0.9, tuned on the clean synthetic set), not the
approach. A {model × candidate-cap k × aggregation × θc × θe} grid
sweep (bench/scripts/nli_shadow_grid.py — NLI runs once per
(model,record) over the top-12 candidate clauses, the k/agg/θ grid is
then arithmetic on cached scores; ~10s on the 4090 for 4 models) finds
clean passing configs: on the 89 real STRICT cells (n=1 bench-qa),
deberta-base-184M / k=2 / agg=max / θc=0.99 / θe=0.9 → 27/28 synthetic
recombinations caught (incl. both 5f-fal-hard fixtures), 0/89 STRICT
FP. MiniLM-82M passes too (24/28 · 0/89). Model science: 184M > 82M >
407M for fp=0 recombination recall; int8-ONNX costs ~1 catch vs fp32.

Caveats: FP side is n=1 (BENCH_QA_N=3 run in flight); recall is on the
synthetic set; flipping to a runtime demotion-only veto is fox-decides
(then nli_policy_hash folds into governance_policy_hash per §7 #2).
Manifest active defaults stay k=6/θc=0.5; recommended_operating_point
(deberta-base, k=2, agg=max, θc=0.99, θe=0.9) documented in the
manifest. 3 grid result JSONs committed. Standing lesson: neither the
clean synthetic eval NOR the default config predicts bench-qa precision
— you have to sweep. Production verifier unchanged; falsification-hard
stays 10/12.
This commit is contained in:
russell@unturf.com 2026-05-12 18:12:36 -04:00
parent 9f528b4ac2
commit 421b2c5fd5
No known key found for this signature in database
6 changed files with 2718 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

10
arborist/qa/nli/manifest.json
View file

@ -19,6 +19,16 @@
"entailment_block_veto": 0.9
},
"thresholds_provenance": "the bench-validated operating point — arborist-nli-bench commit a1cb28d, ticket #000049 §7 #18; on the 28+26-case synth set this point gives 28/28 catch, 0/26 false positives, and the entailment guard is what filters the spurious competing-superlative contradictions",
"recommended_operating_point": {
"_comment": "ticket #000049 §7 #23 — the {model × k × agg × θc × θe} grid sweep found this clears the §7 #12 gate on bench-qa traffic: catches 27/28 synthetic recombinations (incl. both 5f-fal-hard fixtures), 0 false positives on 89 real STRICT answers (n=1 bench-qa). NOT yet the active default — pending confirmation on a BENCH_QA_N=3 STRICT sample (~250 cells). When promoted: switch nli_model_version/hf_repo to this, set thresholds + max_candidate_clauses + aggregation accordingly, and fold nli_policy_hash (model+k+agg+θc+θe) into governance_policy_hash before any audit_mode effect (§7 #2).",
"nli_model_version": "nli-shadow-v1-deberta-v3-base",
"hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli",
"pinned_revision": "6f5cf0a2b59cabb106aca4c287eed12e357e90eb",
"max_candidate_clauses": 2,
"aggregation": "max",
"thresholds": {"contradiction_veto": 0.99, "entailment_block_veto": 0.9},
"fallback_small_model": {"nli_model_version": "nli-shadow-v1-minilm2-l6-h768", "note": "82M; passes too at 24/28 catch · 0/89 FP with k=2/agg=max/θc=0.95/θe=0.9 — the cost-pick if the 184M latency matters"}
},
"alternates": [
{"nli_model_version": "nli-shadow-v1-minilm-l6-mnli", "hf_repo": "cross-encoder/nli-MiniLM2-L6-H768", "pinned_revision": "b95119ce93d3e065de6214e38cd4a97b0f2f2c6d", "license": "Apache-2.0", "approx_mb": 90, "note": "= the main checkpoint; listed here too so a grid sweep includes 'small' explicitly"},
{"nli_model_version": "nli-shadow-v1-deberta-v3-base", "hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli", "pinned_revision": "6f5cf0a2b59cabb106aca4c287eed12e357e90eb", "license": "MIT", "approx_mb": 440, "note": "184M; the threshold-equivalent base candidate (mid)"},

871
bench/results/nli-shadow-grid-n1-fixtures-vs-realstrict.json Normal file
View file

@ -0,0 +1,871 @@
{
"generated_at": "2026-05-12T22:06:14Z",
"should_demote_file": "bench/fixtures/5f/falsification-hard-v1.jsonl",
"n_should_demote": 12,
"should_not_demote_file": "/home/fox/nli-data/2026-05-12T20-53-11Z.jsonl",
"n_should_not_demote": 89,
"k_grid": [
1,
2,
3,
4,
6,
12
],
"aggs": [
"max",
"mean",
"top2"
],
"theta_c_grid": [
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.85,
0.9,
0.93,
0.95,
0.97,
0.99
],
"theta_e_grid": [
0.5,
0.7,
0.9,
1.01
],
"kmax": 12,
"models": {
"nli-shadow-v1-minilm2-l6-h768": {
"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 1085,
"infer_seconds": 1.24,
"configs_ranked": [
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.5833,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.5833,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.5833,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.5833,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
}
]
},
"nli-shadow-v1-minilm-l6-mnli": {
"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 1085,
"infer_seconds": 1.02,
"configs_ranked": [
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.5833,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.5833,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.5833,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.5833,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
}
]
},
"nli-shadow-v1-deberta-v3-base": {
"hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 1085,
"infer_seconds": 3.73,
"configs_ranked": [
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.8,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.8,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.85,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.5,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.5,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.1667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.1667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.1667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.8333,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.8333,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0225
},
"best_kind": "fp<=0.05"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0337
},
"best_kind": "fp<=0.05"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.4,
"theta_e": 0.5,
"catch": 0.3333,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.6,
"theta_e": 0.5,
"catch": 0.3333,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
}
]
},
"nli-shadow-v1-bart-large-mnli": {
"hf_repo": "facebook/bart-large-mnli",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 1085,
"infer_seconds": 5.69,
"configs_ranked": [
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.6667,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.85,
"theta_e": 0.5,
"catch": 0.9167,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.85,
"theta_e": 0.5,
"catch": 0.9167,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.8333,
"fp": 0.0337
},
"best_kind": "fp<=0.05"
},
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.8333,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.7,
"theta_e": 0.5,
"catch": 0.8333,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.8333,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.8333,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.8,
"theta_e": 0.5,
"catch": 0.3333,
"fp": 0.0337
},
"best_kind": "fp<=0.05"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.5,
"theta_e": 0.5,
"catch": 0.3333,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0225
},
"best_kind": "fp<=0.05"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.25,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.8333,
"fp": 0.0562
},
"best_kind": "knee"
}
]
}
}
}

871
bench/results/nli-shadow-grid-n1-synth-vs-realstrict.json Normal file
View file

@ -0,0 +1,871 @@
{
"generated_at": "2026-05-12T22:09:07Z",
"should_demote_file": "/home/fox/nli-data/recombination.jsonl",
"n_should_demote": 28,
"should_not_demote_file": "/home/fox/nli-data/2026-05-12T20-53-11Z.jsonl",
"n_should_not_demote": 89,
"k_grid": [
1,
2,
3,
4,
6,
12
],
"aggs": [
"max",
"mean",
"top2"
],
"theta_c_grid": [
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.85,
0.9,
0.93,
0.95,
0.97,
0.99
],
"theta_e_grid": [
0.5,
0.7,
0.9,
1.01
],
"kmax": 12,
"models": {
"nli-shadow-v1-minilm2-l6-h768": {
"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 1125,
"infer_seconds": 1.16,
"configs_ranked": [
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.95,
"theta_e": 0.9,
"catch": 0.8571,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.8214,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.8214,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.9,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.9,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.9,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.9,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.5357,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.5,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.4643,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.4643,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.4643,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
}
]
},
"nli-shadow-v1-minilm-l6-mnli": {
"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 1125,
"infer_seconds": 1.17,
"configs_ranked": [
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.95,
"theta_e": 0.9,
"catch": 0.8571,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.8214,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.8214,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.9,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.9,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.9,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.9,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.5357,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.93,
"theta_e": 0.5,
"catch": 0.5,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.4643,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.4643,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.4643,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
}
]
},
"nli-shadow-v1-deberta-v3-base": {
"hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 1125,
"infer_seconds": 3.02,
"configs_ranked": [
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.9,
"catch": 0.9643,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.8,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.8,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.85,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.6071,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.5714,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.4643,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.4643,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0337
},
"best_kind": "fp<=0.05"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0337
},
"best_kind": "fp<=0.05"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.93,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.9643,
"fp": 0.0337
},
"best_kind": "fp<=0.05"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.4,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.6,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
}
]
},
"nli-shadow-v1-bart-large-mnli": {
"hf_repo": "facebook/bart-large-mnli",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 1125,
"infer_seconds": 5.67,
"configs_ranked": [
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.95,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.6786,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.95,
"theta_e": 1.01,
"catch": 0.9643,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 1.01,
"catch": 0.9643,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0337
},
"best_kind": "fp<=0.05"
},
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.85,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.85,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.8,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0337
},
"best_kind": "fp<=0.05"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.9,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.7,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.5,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0449
},
"best_kind": "fp<=0.05"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.99,
"theta_e": 0.5,
"catch": 0.6429,
"fp": 0.0225
},
"best_kind": "fp<=0.05"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.97,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0899
},
"best_kind": "knee"
}
]
}
}
}

871
bench/results/nli-shadow-grid-n1-synth-vs-synth.json Normal file
View file

@ -0,0 +1,871 @@
{
"generated_at": "2026-05-12T22:08:31Z",
"should_demote_file": "/home/fox/nli-data/recombination.jsonl",
"n_should_demote": 28,
"should_not_demote_file": "/home/fox/nli-data/legit_summary.jsonl",
"n_should_not_demote": 26,
"k_grid": [
1,
2,
3,
4,
6,
12
],
"aggs": [
"max",
"mean",
"top2"
],
"theta_c_grid": [
0.3,
0.4,
0.5,
0.6,
0.7,
0.8,
0.85,
0.9,
0.93,
0.95,
0.97,
0.99
],
"theta_e_grid": [
0.5,
0.7,
0.9,
1.01
],
"kmax": 12,
"models": {
"nli-shadow-v1-minilm2-l6-h768": {
"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 110,
"infer_seconds": 0.23,
"configs_ranked": [
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.6786,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
}
]
},
"nli-shadow-v1-minilm-l6-mnli": {
"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 110,
"infer_seconds": 0.09,
"configs_ranked": [
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.6786,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
}
]
},
"nli-shadow-v1-deberta-v3-base": {
"hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 110,
"infer_seconds": 0.73,
"configs_ranked": [
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.9,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
}
]
},
"nli-shadow-v1-bart-large-mnli": {
"hf_repo": "facebook/bart-large-mnli",
"backend": "torch",
"device": "cuda",
"n_nli_pairs": 110,
"infer_seconds": 0.39,
"configs_ranked": [
{
"k": 2,
"agg": "max",
"best": {
"theta_c": 0.5,
"theta_e": 1.01,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 1.01,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "max",
"best": {
"theta_c": 0.5,
"theta_e": 1.01,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 1.01,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "max",
"best": {
"theta_c": 0.5,
"theta_e": 1.01,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 1.01,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "max",
"best": {
"theta_c": 0.5,
"theta_e": 1.01,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 1.01,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "max",
"best": {
"theta_c": 0.5,
"theta_e": 1.01,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 1.01,
"catch": 1.0,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "max",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "mean",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.9286,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 3,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 4,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 6,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 12,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.75,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 2,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.7143,
"fp": 0.0
},
"best_kind": "fp=0"
},
{
"k": 1,
"agg": "top2",
"best": {
"theta_c": 0.3,
"theta_e": 0.5,
"catch": 0.0,
"fp": 0.0
},
"best_kind": "fp=0"
}
]
}
}
}

2
docs/TICKETS.md
View file

File diff suppressed because one or more lines are too long

106
docs/tickets/ticket-000049-attribution-aware-grounding-check.md
View file

@ -18,18 +18,26 @@ the next Phase-2 step) must restrict the NLI call to the clauses the
lexical verifier actually matched and/or a deterministic recombination-risk
trigger; do NOT enable runtime NLI demotion on the current scaffold.
Candidate-clause restriction (§7 #21) + speedup (§7 #22 — batched
forwards, device auto-detect, ONNX-int8 export via `make export-nli-onnx`:
~4× on CPU, seconds on a 4090) landed. **Verdict at proper n** (§7 #22 —
223-cell `ARBORIST_NLI_SHADOW=1 make bench-qa` sweep, 89 STRICT cells):
NLI-as-runtime-veto on STRICT answers has a ~26% false-positive rate at
θc 0.5, ~8% at θc 0.90, ~0% only at θc 0.99 — but θc 0.99 gives up most
recombination recall (hard synthetic recombinations bottom out ~0.76).
**Fails the §7 #12 gate on this design.** Only untried path that might
pass: a Phase-3 runtime hook running NLI on the clauses the lexical
verifier actually matched (1–3, the right ones), not top-6-by-overlap —
a verifier-side change. Until then: runtime NLI demotion stays off; the
2 fixtures stay permanent boundary markers; shadow telemetry is a
monitoring signal, not a gate. Production verifier unchanged;
forwards, device auto-detect, ONNX-int8 export, ~4× on CPU / seconds
on a 4090) landed; a grid sweep over `{model × k × agg × θc × θe}`
(§7 #23, `bench/scripts/nli_shadow_grid.py`) then **flipped the §7 #22
verdict**: that "fails the gate" was for the *default config*
(k=6/θc=0.5/θe=0.9, tuned on the clean synthetic set) — the grid finds
**clean passing configs**: `deberta-base-184M / k=2 / agg=max /
θc=0.99 / θe=0.9` catches **27 of 28 synthetic recombinations** (incl.
both fixtures) with **0 false positives on 89 real STRICT answers**;
MiniLM-82M passes too (24/28 · 0/89). Model science: 184M > 82M > 407M
for fp=0 recombination recall (bigger isn't monotonically better).
Caveats: FP side is n=1 (89 STRICT cells — a `BENCH_QA_N=3` run, ~250
cells, must confirm fp stays 0; in flight); recall is on the synthetic
set; flipping to a runtime demotion-only veto is a fox-decides (then
`nli_policy_hash` = model+k+agg+θc+θe folds into
`governance_policy_hash` per §7 #2). Manifest active defaults stay
k=6/θc=0.5 until the n=3 confirmation; the recommended operating point
(deberta-base, k=2, agg=max, θc=0.99, θe=0.9) is documented in the
manifest. Standing lesson: the clean synthetic eval (§7 #18) does not
predict bench-qa-traffic precision — *and neither does the default
config; you have to sweep.* Production verifier unchanged;
`falsification-hard` stays 10/12.
**Opened:** 2026-05-12
**Scope:** Decide whether — and if so how — to add a verifier check
@ -901,3 +909,77 @@ change, not a standalone proxy. Until that's built and re-measured:
boundary markers; the shadow telemetry is a monitoring signal, not a
gate.** θc stays 0.5 in the manifest. Production verifier unchanged;
`falsification-hard` stays 10/12.
**23. Grid sweep flips the verdict — the §7 #22 "fails" was config-
specific, not approach-specific (2026-05-12).** §7 #22's "fails the
gate" was the verdict for the *default config* (`k=6, θc=0.5, θe=0.9`)
— which was tuned on the clean synthetic set, where everything is
≈0.99-vs-≈0.005, and is *not* the right config for bench-qa traffic.
A grid sweep over `{model × candidate-clause-cap k ∈ {1,2,3,4,6,12} ×
aggregation ∈ {max, mean, top2} × θc × θe}` (`bench/scripts/nli_shadow_grid.py`
— NLI runs once per (model, record) over the top-12 candidate clauses;
the k/agg/θ grid is then pure arithmetic on the cached per-clause
scores; the full 4-model grid is ~10 s on the 4090 vs ~3 min on CPU)
finds clean passing configs. Three sweeps (n=1 bench-qa = 89 STRICT
cells; results in `bench/results/nli-shadow-grid-n1-*.json`):
- *Sweep B — synthetic-28 recombination recall vs synthetic-26 legit
FP*: every model, **catch 1.00 · FP 0.00** at `k≥2`, low θc — the
clean set is trivially solved (reconfirms §7 #18; says nothing about
real traffic).
- *Sweep A — the 12 `falsification-hard` false-claims (incl. both
recombination fixtures) recall vs the 89 real STRICT cells FP*:
best fp=0 configs catch **8-9 / 12** — MiniLM-82M `k=1/θc=0.93` →
0.75, deberta-base-184M `k=2/θc=0.99` → 0.75, bart-large-407M → 0.67;
both `5f-fal-hard-003` (Mercury, contra 0.98) and `-005` (Einstein,
contra 0.98) fire — **the 2 fixtures #000049 exists for are caught
at 0 real-STRICT FP**.
- *Sweep C — the actual gate question: synthetic-28 recombination
recall vs the 89 real STRICT cells FP*:
| NLI model | best `fp=0` config | recombination catch | real-STRICT FP |
|---|---|---|---|
| MiniLM-82M | `k=2, agg=max, θc=0.95, θe=0.9` | **24/28 = 0.857** | **0/89** |
| **deberta-base-184M** | **`k=2, agg=max, θc=0.99, θe=0.9`** | **27/28 = 0.964** | **0/89** |
| bart-large-407M | `k=3, agg=mean, θc=0.95, θe=0.5` | 20/28 = 0.714 | 0/89 |
| bart-large-407M (relaxed) | `k=2, agg=max, θc=0.95, θe=1.01` | 27/28 = 0.964 | 4/89 = 0.045 |
**`deberta-base-184M` at `k=2 / agg=max / θc=0.99 / θe=0.9` catches 27
of 28 synthetic recombinations with ZERO false positives on 89 real
STRICT answers — that is a passing config.** MiniLM-82M passes too at
24/28. The structure that works: **k=2** (two candidate clauses — k=1
misses cases where the best-token-overlap clause isn't the
contradicting one; k≥3 with `max` reintroduces haystack; `mean` over
more clauses dilutes), **`max` aggregation**, **θc ≈ 0.95-0.99** (not
0.5), **θe = 0.9** (the manifest default for the guard is fine; the
tighter 0.5 isn't needed once θc is high). Model-size science: **184M
> 82M > 407M for fp=0 recombination recall** — bigger isn't
monotonically better, bart-large's contradiction scores are more
spread so it needs a lower θc that costs FP; deberta-base is the sweet
spot (much better recall than MiniLM at the same fp=0 bar, ~3 s for
~1100 pairs on GPU). int8-ONNX quantization costs ≈1 catch vs fp32
(MiniLM 0.667 int8 vs 0.75 fp32 on Sweep A) — a small but real tax.
**Revised verdict.** The lexical-candidate NLI veto **clears the §7
#12 gate** (catch the recombinations incl. both fixtures, ≈0 false-
demote on confidently-grounded answers) — at **~86-96% recombination
recall depending on model**, with `deberta-base / k=2 / agg=max /
θc=0.99 / θe=0.9` the recommended operating point. The Phase-3
verifier-matched-clause hook would only push recall higher (a refined
candidate selector), not rescue a failing approach. **Caveats before
this is final:** (a) the FP side is n=1 = 89 STRICT cells; a full
`ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=3` run (~250 STRICT
cells, in flight) must confirm fp stays 0 at the chosen θc — until
then the manifest defaults stay `k=2`-untuned (`max_candidate_clauses=6`,
θc 0.5); (b) the "27/28" / "24/28" recall is on the *synthetic*
recombination set — a bench-qa-derived recombination set (real false
answers that took the paraphrase/entity path) would be more
load-bearing, but those are rare in real traffic (the whole point of
the boundary); (c) this is still SHADOW — flipping it to a runtime
demotion-only veto is a fox-decides (`MODEL_ASSISTED_DEMOTION`, and
`nli_policy_hash` — model + k + agg + θc + θe — folds into
`governance_policy_hash` at that point, per §7 #2). What changes
*now*: the manifest gains `agg` + the k=2/θc=0.99 candidate config as
the recommended-but-not-yet-default operating point, pending the n=3
confirmation. Production verifier unchanged; `falsification-hard`
stays 10/12.