From d9760bc5bb8663c35def1f8a74baa0e4ae9a5422 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Tue, 12 May 2026 19:42:55 -0400 Subject: [PATCH] =?UTF-8?q?#000049=20=C2=A77=20#25:=20n=3D5=20confirmation?= =?UTF-8?q?=20(444=20STRICT=20cells)=20=E2=80=94=20verdict=20settles;=20th?= =?UTF-8?q?e=20large=20models=20are=20the=20robust=20ones,=20MiniLM-cost-p?= =?UTF-8?q?ick=20overturned?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ARBORIST_NLI_SHADOW=1 make bench-qa BENCH_QA_N=5 → 1125 cells, 444 real STRICT (5x n=1, 1.6x n=3). Re-ran the 7-model mega-grid: the lexical- candidate NLI veto robustly clears the §7 #12 gate with microsoft/deberta-large-mnli / k=2 / agg=max / guard=max_entail / θc≈0.96 / θe=0.9 → 28/28 synthetic recombinations (incl. both fixtures) · 0/444 real STRICT FP · 0/26 synthetic legit FP, ~4 pts θc headroom; roberta-large-mnli equally good (k=2/max/θc=0.95). Resolved: agg=max + max_entail guard is the robust score-shape (§7 #24's margin win was a sample tie); the large checkpoints (~350-400M deberta-large-mnli / roberta-large-mnli) hit 1.0/0.0, bart-large (similar size) only ~0.71, deberta-base at the cliff (27/28 here, 11/28 at n=3), small models (MiniLM-82M, deberta-v3-small) cap at ~0.82 — so §7 #18's 'MiniLM is the cost-pick' is OVERTURNED by the proper-n evidence; k=2 consistent winner; int8-ONNX costs ~1 catch. recommended_operating_point updated. Remaining: a bench-qa-derived recombination set (the one check not done); n=9 if dav1d wants more; still SHADOW; runtime promotion is fox+dav1d-decides. Production verifier unchanged; falsification-hard stays 10/12. This run is the worked example behind CLAUDE.md's new bench-maxing line. --- arborist/qa/nli/manifest.json | 19 +- ...li-shadow-grid-n5-synth-vs-realstrict.json | 9982 +++++++++++++++++ ...00049-attribution-aware-grounding-check.md | 137 +- 3 files changed, 10098 insertions(+), 40 deletions(-) create mode 100644 bench/results/nli-shadow-grid-n5-synth-vs-realstrict.json diff --git a/arborist/qa/nli/manifest.json b/arborist/qa/nli/manifest.json index 7d49f8a..04a9a5d 100644 --- a/arborist/qa/nli/manifest.json +++ b/arborist/qa/nli/manifest.json @@ -20,16 +20,19 @@ }, "thresholds_provenance": "the bench-validated operating point — arborist-nli-bench commit a1cb28d, ticket #000049 §7 #18; on the 28+26-case synth set this point gives 28/28 catch, 0/26 false positives, and the entailment guard is what filters the spurious competing-superlative contradictions", "recommended_operating_point": { - "_comment": "ticket #000049 §7 #24 — n=3-confirmed (275 real STRICT cells). The {model × candidate-cap k × aggregation × guard × θc × θe} mega-grid found THIS clears the §7 #12 gate: microsoft/deberta-large-mnli + k=3 + agg='margin' (= max over clauses of p_contra - p_entail, a single threshold folding the entailment guard in) + θc=0.995 → catches all 28 synthetic recombinations (incl. both 5f-fal-hard fixtures), 0 false positives on 275 real STRICT answers, 0 on the 26 synthetic legit summaries. NOTE: the §7 #23 candidate (deberta-base k=2 θc=0.99) did NOT survive the n=3 sample — it only catches 11/28 at θc=0.995, which is what the larger STRICT sample forces; the 89-cell n=1 sample under-counted the spurious-contradiction tail. Still NOT the active default — pending (a) an even larger STRICT sample / real-traffic check and (b) a bigger recombination set than 28. When promoted: switch nli_model_version/hf_repo, wire 'aggregation'='margin' into ShadowNLI.check, set max_candidate_clauses + thresholds accordingly, and fold nli_policy_hash (model+k+agg+θc) into governance_policy_hash before any audit_mode effect (§7 #2).", + "_comment": "ticket #000049 §7 #25 — n=5-confirmed (444 real STRICT cells; also held at n=3=275). The {model × candidate-cap k × aggregation × guard × θc × θe} mega-grid found THIS robustly clears the §7 #12 gate: microsoft/deberta-large-mnli + k=2 + agg='max' (over the top-2 candidate clauses) + guard='max_entail' (max p_entail over those clauses) → catches all 28 synthetic recombinations (incl. both 5f-fal-hard fixtures), 0 false positives on 444 real STRICT answers, 0 on the 26 synthetic legit summaries — with ~4 pts of θc headroom (it stays 0 FP up to ~θc 0.99 on this sample; the config below uses 0.96 with margin). roberta-large-mnli is an equally-good alternate (k=2/max/θc=0.95). NOTE the journey: the §7 #23 candidate (deberta-base k=2 θc=0.99) was a small-FP-sample artifact — it's 27/28 here but the §7 #24 n=3 sample drove it to 11/28; it sits at the cliff. The §7 #18 'MiniLM-82M is the cost-pick' is overturned — at the fp=0 bar MiniLM caps at ~0.82, the large model gets 1.0; cost was the wrong axis. Still NOT the active default — pending (a) a bench-qa-derived recombination set (real false answers, not synthetic), (b) optionally n=9 STRICT (~825 cells), (c) fox+dav1d sign-off. When promoted: switch nli_model_version/hf_repo, wire 'aggregation'+'guard' into ShadowNLI.check, set max_candidate_clauses + thresholds, and fold nli_policy_hash (model+k+agg+guard+θc) into governance_policy_hash before any audit_mode effect (§7 #2).", "nli_model_version": "nli-shadow-v1-deberta-large-mnli", "hf_repo": "microsoft/deberta-large-mnli", - "max_candidate_clauses": 3, - "aggregation": "margin", - "thresholds": {"contradiction_minus_entailment_margin_veto": 0.995}, - "alternates_at_fp0_n3": [ - {"hf_repo": "roberta-large-mnli", "config": "k=2/agg=max/θc=0.998", "catch": "21/28", "note": "355M; second-best, ~0.75 recall"}, - {"hf_repo": "facebook/bart-large-mnli", "config": "k=3/agg=mean/θc=0.94", "catch": "20/28", "note": "407M; ~0.71 recall"}, - {"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768", "config": "k=4/agg=mean/θc=0.96", "catch": "14/28", "note": "82M; only ~0.50 recall at fp=0 on the n=3 sample — NOT the cost-pick after all once you demand 0 FP at proper n"} + "max_candidate_clauses": 2, + "aggregation": "max", + "guard": "max_entail", + "thresholds": {"contradiction_veto": 0.96, "entailment_block_veto": 0.9}, + "thresholds_headroom_note": "0 FP held up to θc≈0.99 on the 444-cell n=5 STRICT sample; 0.96 is the conservative-recall choice (still 28/28). At n=3 the same model was 1.0/0.0 with agg='margin' @ θc=0.995 — both score-shapes work; 'max'+'max_entail' is the simpler/more-robust pick at n=5.", + "alternates_at_fp0_proper_n": [ + {"hf_repo": "roberta-large-mnli", "config": "k=2/agg=max/guard=max_entail/θc=0.95", "catch_n5": "28/28", "note": "355M; equally good as the primary at n=5; not separately validated at n=3"}, + {"hf_repo": "facebook/bart-large-mnli", "config": "k=3/agg=mean/θc=0.94", "catch_n5": "20/28", "note": "407M; only ~0.71 recall at fp=0 — bigger ≠ better, it's the specific checkpoint"}, + {"hf_repo": "cross-encoder/nli-MiniLM2-L6-H768", "config": "k=2/agg=max/θc=0.98", "catch_n5": "23/28", "note": "82M; ~0.82 recall at fp=0 at proper n — NOT the cost-pick once you demand 0 FP"}, + {"hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli", "config": "k=2/agg=max/θc=0.99", "catch_n5": "27/28", "note": "184M; sits at the cliff — 27/28 here, 11/28 on the n=3 sample; sample-sensitive, do not use"} ] }, "alternates": [ diff --git a/bench/results/nli-shadow-grid-n5-synth-vs-realstrict.json b/bench/results/nli-shadow-grid-n5-synth-vs-realstrict.json new file mode 100644 index 0000000..fad0a5c --- /dev/null +++ b/bench/results/nli-shadow-grid-n5-synth-vs-realstrict.json @@ -0,0 +1,9982 @@ +{ + "generated_at": "2026-05-12T23:34:15Z", + "should_demote_file": "/home/fox/nli-data/recombination.jsonl", + "n_should_demote": 28, + "should_not_demote_file": "/home/fox/nli-data/benchqa-n5.jsonl", + "n_should_not_demote": 444, + "k_grid": [ + 1, + 2, + 3, + 4, + 6, + 8, + 12 + ], + "aggs": [ + "max", + "mean", + "top2", + "top3", + "meanTop2", + "meanTop3", + "margin" + ], + "guards": [ + "max_entail", + "paired_entail" + ], + "theta_c_grid": [ + 0.3, + 0.4, + 0.5, + 0.6, + 0.7, + 0.8, + 0.85, + 0.9, + 0.92, + 0.94, + 0.95, + 0.96, + 0.97, + 0.975, + 0.98, + 0.985, + 0.99, + 0.995, + 0.998, + 0.999 + ], + "theta_e_grid": [ + 0.3, + 0.5, + 0.7, + 0.9, + 0.95, + 1.01 + ], + "kmax": 12, + "models": { + "nli-shadow-v1-minilm2-l6-h768": { + "hf_repo": "cross-encoder/nli-MiniLM2-L6-H768", + "backend": "torch", + "device": "cuda", + "n_nli_pairs": 5383, + "infer_seconds": 5.21, + "global_best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.98, + "theta_e": 0.9, + "catch": 0.8214, + "fp": 0.0 + }, + "pareto": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.98, + "theta_e": 0.9, + "catch": 0.8214, + "fp": 0.0 + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.9, + "catch": 0.8571, + "fp": 0.0045 + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.9, + "catch": 0.8929, + "fp": 0.0248 + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.9, + "catch": 0.9286, + "fp": 0.0293 + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.9, + "catch": 0.9643, + "fp": 0.0495 + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.7, + "theta_e": 0.9, + "catch": 1.0, + "fp": 0.0923 + } + ], + "configs_ranked": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.98, + "theta_e": 0.9, + "catch": 0.8214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.98, + "theta_e": 0.3, + "catch": 0.8214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.975, + "theta_e": 1.01, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 1.01, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + } + ] + }, + "nli-shadow-v1-minilm-l6-mnli": { + "hf_repo": "cross-encoder/nli-MiniLM2-L6-H768", + "backend": "torch", + "device": "cuda", + "n_nli_pairs": 5383, + "infer_seconds": 5.17, + "global_best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.98, + "theta_e": 0.9, + "catch": 0.8214, + "fp": 0.0 + }, + "pareto": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.98, + "theta_e": 0.9, + "catch": 0.8214, + "fp": 0.0 + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.9, + "catch": 0.8571, + "fp": 0.0045 + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.9, + "catch": 0.8929, + "fp": 0.0248 + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.9, + "catch": 0.9286, + "fp": 0.0293 + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.9, + "catch": 0.9643, + "fp": 0.0495 + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.7, + "theta_e": 0.9, + "catch": 1.0, + "fp": 0.0923 + } + ], + "configs_ranked": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.98, + "theta_e": 0.9, + "catch": 0.8214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.98, + "theta_e": 0.3, + "catch": 0.8214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.975, + "theta_e": 1.01, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 1.01, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + } + ] + }, + "nli-shadow-v1-deberta-v3-base": { + "hf_repo": "MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli", + "backend": "torch", + "device": "cuda", + "n_nli_pairs": 5383, + "infer_seconds": 13.6, + "global_best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.9643, + "fp": 0.0 + }, + "pareto": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.9643, + "fp": 0.0 + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.975, + "theta_e": 0.9, + "catch": 1.0, + "fp": 0.0045 + } + ], + "configs_ranked": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.9, + "catch": 0.9643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.9643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.9643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.98, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.985, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.985, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.1786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.1786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.1786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.1786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.1786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.1786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.1786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.1786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.1786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.1786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.1429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.1429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.1429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.1429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.0714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.0714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.98, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.98, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 1.0, + "fp": 0.0428 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.97, + "theta_e": 1.01, + "catch": 1.0, + "fp": 0.0428 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 1.0, + "fp": 0.0495 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 1.01, + "catch": 1.0, + "fp": 0.0495 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 1.0, + "fp": 0.0495 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 1.01, + "catch": 1.0, + "fp": 0.0495 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.9643, + "fp": 0.0405 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 1.01, + "catch": 0.9643, + "fp": 0.0405 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.045 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 1.01, + "catch": 0.8929, + "fp": 0.045 + }, + "best_kind": "fp<=0.05" + } + ] + }, + "nli-shadow-v1-bart-large-mnli": { + "hf_repo": "facebook/bart-large-mnli", + "backend": "torch", + "device": "cuda", + "n_nli_pairs": 5383, + "infer_seconds": 27.6, + "global_best": { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "pareto": [ + { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0113 + }, + { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 1.01, + "catch": 0.9286, + "fp": 0.0158 + }, + { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 1.01, + "catch": 0.9643, + "fp": 0.027 + }, + { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.6, + "theta_e": 1.01, + "catch": 1.0, + "fp": 0.1194 + } + ], + "configs_ranked": [ + { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.98, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.98, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 1.01, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 1.01, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 1.01, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 1.01, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 1.01, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 1.01, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.4643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 1.01, + "catch": 0.2857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + } + ] + }, + "extra:nli-deberta-v3-small": { + "hf_repo": "cross-encoder/nli-deberta-v3-small", + "backend": "torch", + "device": "cuda", + "n_nli_pairs": 5383, + "infer_seconds": 7.63, + "global_best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.95, + "catch": 0.8214, + "fp": 0.0 + }, + "pareto": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.95, + "catch": 0.8214, + "fp": 0.0 + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.95, + "catch": 0.9286, + "fp": 0.0023 + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.9643, + "fp": 0.018 + }, + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.95, + "catch": 1.0, + "fp": 0.0293 + } + ], + "configs_ranked": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.95, + "catch": 0.8214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.8214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 1.01, + "catch": 0.8214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 1.01, + "catch": 0.7857, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.985, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.985, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.4286, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.3929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.3214, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.6, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.6, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.95, + "catch": 1.0, + "fp": 0.045 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 4, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.95, + "catch": 1.0, + "fp": 0.0495 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 6, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.9643, + "fp": 0.045 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 8, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.9643, + "fp": 0.0473 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.9286, + "fp": 0.0248 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 1.01, + "catch": 0.9286, + "fp": 0.0248 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.9286, + "fp": 0.0248 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 1.01, + "catch": 0.9286, + "fp": 0.0248 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 1.01, + "catch": 0.9286, + "fp": 0.0383 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 12, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.027 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.8214, + "fp": 0.0225 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 1.01, + "catch": 0.8214, + "fp": 0.0315 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 1.01, + "catch": 0.8214, + "fp": 0.0315 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.8214, + "fp": 0.0428 + }, + "best_kind": "fp<=0.05" + }, + { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.8214, + "fp": 0.0428 + }, + "best_kind": "fp<=0.05" + } + ] + }, + "extra:roberta-large-mnli": { + "hf_repo": "roberta-large-mnli", + "backend": "torch", + "device": "cuda", + "n_nli_pairs": 5383, + "infer_seconds": 22.8, + "global_best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.9, + "catch": 1.0, + "fp": 0.0 + }, + "pareto": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.9, + "catch": 1.0, + "fp": 0.0 + } + ], + "configs_ranked": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.9, + "catch": 1.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 1.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 1.01, + "catch": 1.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 1.01, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.7, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.7, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.7, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.7, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 0.7, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 1.01, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 1.01, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 1.01, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 1.01, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 1.01, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.975, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.6071, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.5714, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + } + ] + }, + "extra:deberta-large-mnli": { + "hf_repo": "microsoft/deberta-large-mnli", + "backend": "torch", + "device": "cuda", + "n_nli_pairs": 5383, + "infer_seconds": 37.74, + "global_best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.9, + "catch": 1.0, + "fp": 0.0 + }, + "pareto": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.9, + "catch": 1.0, + "fp": 0.0 + } + ], + "configs_ranked": [ + { + "k": 2, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.9, + "catch": 1.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 1.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 1.01, + "catch": 1.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 1.01, + "catch": 1.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 1.01, + "catch": 1.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 1.01, + "catch": 1.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 1.01, + "catch": 1.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.5, + "catch": 0.9643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.5, + "catch": 0.9643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.5, + "catch": 0.9643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.5, + "catch": 0.9643, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 1.01, + "catch": 0.8929, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.75, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "margin", + "guard": "max_entail", + "theta_c": 0.998, + "theta_e": 1.01, + "catch": 0.7143, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.92, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "max_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "mean", + "guard": "paired_entail", + "theta_c": 0.85, + "theta_e": 0.3, + "catch": 0.6786, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.94, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "max_entail", + "theta_c": 0.96, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop3", + "guard": "paired_entail", + "theta_c": 0.97, + "theta_e": 0.3, + "catch": 0.6429, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.99, + "theta_e": 0.3, + "catch": 0.5357, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "max_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.5, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.995, + "theta_e": 0.3, + "catch": 0.3571, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "max_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.25, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "max", + "guard": "paired_entail", + "theta_c": 0.999, + "theta_e": 0.3, + "catch": 0.25, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "meanTop2", + "guard": "paired_entail", + "theta_c": 0.998, + "theta_e": 0.3, + "catch": 0.25, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top2", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 1, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 2, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.3, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 3, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.8, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 4, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 6, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 8, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.9, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "max_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + }, + { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "best": { + "k": 12, + "agg": "top3", + "guard": "paired_entail", + "theta_c": 0.95, + "theta_e": 0.3, + "catch": 0.0, + "fp": 0.0 + }, + "best_kind": "fp=0" + } + ] + } + } +} \ No newline at end of file diff --git a/docs/tickets/ticket-000049-attribution-aware-grounding-check.md b/docs/tickets/ticket-000049-attribution-aware-grounding-check.md index c8f5f49..fa16266 100644 --- a/docs/tickets/ticket-000049-attribution-aware-grounding-check.md +++ b/docs/tickets/ticket-000049-attribution-aware-grounding-check.md @@ -20,38 +20,44 @@ trigger; do NOT enable runtime NLI demotion on the current scaffold. Candidate-clause restriction (§7 #21) + speedup (§7 #22 — batched forwards, device auto-detect, ONNX-int8 export, ~4× on CPU / seconds on a 4090) + a `{model × k × agg × guard × θc × θe}` grid sweep -(§7 #23–#24, `bench/scripts/nli_shadow_grid.py`) landed; the verdict -moved **twice**: §7 #22's "fails the gate" was a *default-config* -artifact (k=6/θc=0.5/θe=0.9); §7 #23's "deberta-base k=2/θc=0.99 → -27/28 · 0/89 passes" was a *small-FP-sample* artifact (it collapses -to 11/28 on the n=3 275-STRICT sample, which forces θc up to ~0.995); -the n=3-confirmed truth (§7 #24, the "no stone unturned" mega-grid — -7 models incl. 4 extra xsmall→large, the new `margin` aggregation = -max over clauses of `p_contra − p_entail`): **`microsoft/deberta-large-mnli` -/ k=3 / agg=`margin` / θc=0.995 → catches all 28 synthetic -recombinations (incl. both `5f-fal-hard` fixtures) + 8/12 of the -`falsification-hard` pack, with 0 false positives on 275 real STRICT -answers and 0 on the 26 synthetic legit summaries** — a passing config -at proper n. Findings: (1) `margin` is the right score-shape (single -threshold, folds the entailment guard in); (2) the *specific -checkpoint* matters more than parameter count — `microsoft/deberta-large-mnli` -wins clean, `deberta-base` collapses, `roberta-large`/`bart-large` -(similar size) only reach ~0.71–0.75, there's no "bigger is better" -law. Caveats: still SHADOW; the 28/28 recall is a 28-case denominator -(100%-of-28 ≠ 100%-in-general — a bench-qa-derived recombination set -is the next check), and θc=0.995/0 FP is "on the largest STRICT sample -we have (275 cells)" not "0 FP, period" — a bigger STRICT sample -should be the other next check before runtime promotion; flipping to -a runtime demotion-only veto is fox+dav1d-decides (then `nli_policy_hash` -= model+k+agg+θc folds into `governance_policy_hash` per §7 #2); -deberta-large is ~400M (GPU-fast, slow on CPU). Manifest active -defaults stay k=6/θc=0.5; the `recommended_operating_point` -(deberta-large-mnli / k=3 / margin / θc=0.995) is documented in the -manifest. Standing lesson, sharpened twice: the clean synthetic eval -doesn't predict bench-qa precision (§7 #18→#20), the default config -doesn't predict the best config (§7 #22→#23), **and a small FP-side -sample doesn't predict the large-sample FP rate (§7 #23→#24)** — every -gate number is provisional until the denominators are big enough, and +(§7 #23–#25, `bench/scripts/nli_shadow_grid.py`, run against bench-qa +STRICT samples of growing size n=1=89 → n=3=275 → n=5=444) landed; +the verdict moved **three times then settled**: §7 #22's "fails the +gate" was a *default-config* artifact (k=6/θc=0.5/θe=0.9); §7 #23's +"deberta-base k=2/θc=0.99 passes" was a *small-FP-sample* artifact +(27/28 at n=5 but the n=3 sample drove it to 11/28 — sits at the +cliff); §7 #24 (n=3) found a large model passes; **§7 #25 +(n=5-confirmed, 444 real STRICT cells)**: the lexical-candidate NLI +veto **robustly clears the §7 #12 gate** — **`microsoft/deberta-large-mnli` +/ k=2 / agg=max / guard=max_entail / θc≈0.96 / θe=0.9 → catches all +28 synthetic recombinations (incl. both `5f-fal-hard` fixtures) · +0/444 real STRICT FP · 0/26 synthetic legit FP, ~4 pts of θc +headroom**; `roberta-large-mnli` is an equally-good alternate +(k=2/max/θc=0.95). Resolved findings: `agg=max + max_entail guard` is +the robust score-shape across both proper-n samples (§7 #24's `margin` +win was a sample tie); the *specific large checkpoint* (~350–400M: +deberta-large-mnli / roberta-large-mnli) is what hits 1.0/0.0 — +`bart-large` (similar size) only ~0.71, `deberta-base-184M` at the +cliff, the small models (MiniLM-82M, deberta-v3-small) cap at ~0.82, +so the §7 #18 "MiniLM is the cost-pick" is **overturned by the +proper-n evidence**; k=2 is the consistent winner; int8-ONNX costs +≈1 catch vs fp32. Remaining caveats (narrow now): the 28/28 recall is +a *28-case* denominator — a bench-qa-derived recombination set (real +false answers that took the paraphrase/entity path) is the one +load-bearing check not yet done; the FP side is 444 cells at 0 FP +(solid); n=9 (~825 cells) would add confidence but, given the ~4–5 pt +θc margin, is "if dav1d wants more"; still SHADOW — flipping to a +runtime demotion-only veto is fox+dav1d-decides (then `nli_policy_hash` += model+k+agg+guard+θc folds into `governance_policy_hash` per §7 #2); +deberta-large is ~400M (GPU-fast). Manifest active defaults stay +k=6/θc=0.5; the `recommended_operating_point` (deberta-large-mnli / +k=2 / max / θc=0.96 / θe=0.9) is documented in the manifest. Standing +lesson, sharpened thrice (codified in CLAUDE.md bench-maxing): the +clean synthetic eval doesn't predict bench-qa precision (§7 #18→#20), +the default config doesn't predict the best config (§7 #22→#23), +**and a small FP-side sample doesn't predict the large-sample FP rate +(§7 #23→#24→#25)** — every gate number is provisional until the +denominators are big enough, and the sweep has to be wide enough to include the config that survives them. Production verifier unchanged; `falsification-hard` stays 10/12. **Opened:** 2026-05-12 @@ -1082,3 +1088,70 @@ sweep has to be wide enough to include the config that survives them. The 4090 is what makes that affordable: this whole §7 #24 run — 7 models × the full {k × agg × guard × θc × θe} grid × 303 records — was ~3.5 min. + +**25. The n=5 confirmation — it firms up, and the right config + model +crystallize (2026-05-12).** `ARBORIST_NLI_SHADOW=1 make bench-qa +BENCH_QA_N=5` → 1125 cells, **444 STRICT** (5× the n=1 sample, 1.6× +the n=3). Re-ran the full 7-model mega-grid against the synthetic-28 +recombination set vs the 444 real STRICT cells +(`bench/results/nli-shadow-grid-n5-synth-vs-realstrict.json`): + +| NLI model (size) | best `fp=0` config | recombination catch | real-STRICT FP (n=5, 444 cells) | θc margin | +|---|---|---|---|---| +| `roberta-large-mnli` (~355M) | k=2, agg=max, θc=0.95 | **28/28 = 1.000** | **0/444** | ~5 pts | +| `microsoft/deberta-large-mnli` (~400M) | k=2, agg=max, θc=0.96 | **28/28 = 1.000** | **0/444** | ~4 pts | +| `deberta-base-184M` | k=2, agg=max, θc=0.99 | 27/28 = 0.964 | 0/444 | **edge** (28/28 needs θc=0.975 → fp 0.004) | +| MiniLM-82M | k=2, agg=max, θc=0.98 | 23/28 = 0.821 | 0/444 | — | +| `nli-deberta-v3-small` (~140M) | k=2, agg=max, θc=0.998 | 23/28 = 0.821 | 0/444 | — | +| `bart-large-mnli` (~407M) | k=3, agg=mean, θc=0.94 | 20/28 = 0.714 | 0/444 | — | + +**What resolved:** +- *Model.* The **large models (~350–400M) are the robust ones**: + `roberta-large-mnli` and `microsoft/deberta-large-mnli` both hit + **28/28 · 0/444 at modest θc (0.95–0.96)** — i.e. ~4–5 points of θc + margin before they'd start false-demoting. `deberta-large-mnli` is + the more-validated pick (1.0/0.0 at *both* n=3 [θc=0.995, margin + agg] and n=5 [θc=0.96, plain max]). **`deberta-base-184M` sits right + at the cliff** — 27/28 at θc=0.99 on this sample, but the §7 #24 n=3 + sample drove it to 11/28; its config is sample-sensitive. The small + models cap at ~0.82. So the §7 #18 "MiniLM-82M is the cost-pick" is + **overturned by the proper-n evidence** — at the fp=0 bar MiniLM + caps at 0.82, the large model gets 1.0; cost was the wrong axis to + optimize once you demand 0 false-demote. +- *Aggregation.* `agg=max` + the `max_entail` guard + θc≈0.95 is the + robust config across *all* models and both proper-n samples; §7 #24's + `margin` win was a sample-specific tie — keep `max`. +- *k.* `k=2` is the consistent winner everywhere. + +**Verdict (n=5-confirmed).** The lexical-candidate NLI veto **passes +the §7 #12 gate, robustly**, with **`microsoft/deberta-large-mnli` / +k=2 / agg=max / guard=max_entail / θc≈0.96 / θe=0.9** → 28/28 synthetic +recombinations (incl. both `5f-fal-hard` fixtures) · 0/444 real STRICT +FP, ~4 pts of θc headroom; `roberta-large-mnli` is an equally-good +alternate. The remaining caveats are now narrow: (a) the 28/28 recall +is still a *28-case* denominator — a bench-qa-derived recombination set +(real false answers that took the paraphrase/entity path) is the one +load-bearing check not yet done (those are rare in real traffic — the +whole reason this is a "boundary"); (b) the FP side is 444 cells at 0 +FP, a solid denominator — n=9 (~825 cells) would add confidence but, +given the ~4–5 pt θc margin, is unlikely to move the verdict (n=9 is +"if dav1d wants more"); (c) still SHADOW — flipping to a runtime +demotion-only veto is fox+dav1d-decides, then `nli_policy_hash` = +model+k+agg+guard+θc folds into `governance_policy_hash` (§7 #2); +deberta-large is ~400M (GPU-fast; ~200–300 ms/pair on CPU on the now- +small k=2 candidate set; ONNX-int8 would cost a catch or two). The +manifest's `recommended_operating_point` is updated to this config; +active defaults stay k=6/θc=0.5 until fox promotes. Production verifier +unchanged; `falsification-hard` stays 10/12. + +**This `make bench-qa BENCH_QA_N=5` run (~50 min Hermes) + the 7-model +mega-grid on it (~3 min on the 4090) is the worked example behind +CLAUDE.md's new bench-maxing line** ("when researching a config/model +decision, sweep wide+deep on real-traffic-sized samples; the GPU makes +it cheap; burn the electricity"). The n=1→n=3→n=5 sequence is exactly +why: n=1 said MiniLM/deberta-base pass at high recall; n=3 said +deberta-base collapses; n=5 said the *large* models are the robust +ones and the small ones cap at 0.82 — three samples, three sharper +answers, ~2 hours of GPU+LLM total. A guessed config (MiniLM, per the +n=1 read) would have shipped a 0.82-recall veto when a 1.0-recall one +was available.