#000052 §3.2.1 round-2: extend size spectrum to 7 models (30MB-560MB, ~20×) — real model science from a wider sweep
7 models on the 4090 against the 26-pair candidate-bench: bge-reranker-large 560MB +6.285 ← new primary (most cushion) ms-marco-electra-base 110MB +5.349 (cost alternate — 5× smaller, 87% the margin) MiniLM-L-4-v2 50MB +3.187 bge-reranker-base 280MB +3.076 MiniLM-L-6-v2 80MB +3.053 (was primary before §3.2.1) MiniLM-L-12-v2 130MB +2.663 (deeper ≠ better — same pattern as §7 #24) MiniLM-L-2-v2 30MB -1.970 ← capacity floor (NOT separable, 10/12 catch) Findings: - Biggest is best WITHIN a family (bge-large > bge-base, 2× size → 2× margin). - Across families: electra-base (110MB) beats bge-base (280MB) — 60% smaller, 75% more margin. Architecture/training corpus > parameter count. - Depth non-monotonic within MS-MARCO MiniLM: L-4 > L-6 > L-12. - Capacity floor between L-2 (30MB, fails) and L-4 (50MB, separates). - 6 of 7 clean-separate the candidate-bench. The 'reranker catches Zionist-style mis-cites' claim is a property of competent rerankers as a class — once above the floor. Manifest primary moves to bge-reranker-large for cushion. demote_below_score STAYS null — clean candidate-bench doesn't predict real-pipeline behavior (§7 #18→#27 = 6 verdict flips). Expect a walk-back at §3.2.2 step 2.
This commit is contained in:
parent
9ad224695e
commit
da5af2dfaa
3 changed files with 456 additions and 43 deletions
|
|
@ -1,10 +1,10 @@
|
|||
{
|
||||
"_comment": "Pinned aboutness/relevance reranker for the #000052 §3.2 SHADOW path. SHADOW MODE ONLY: this manifest does NOT yet fold into governance_policy_hash because shadow output cannot touch audit_mode (#000049 §7 #2 discipline inherited verbatim). If/when promoted to a runtime demotion-only veto, add a single-blob checkpoint_sha256 + tokenizer_sha256 + an explicit relevance_policy_hash that folds into governance_policy_hash before any audit_mode effect. Cross-encoder MS-MARCO reranker — single relevance score per (query, document) pair, NOT entailment (that's NLI / arborist.qa.nli; the boundary failure mode this catches is *topic-collision recombination* — answer/claim is on a different topic than its cited source despite shared tokens, the 'Zionist entity' case — which NLI returns neutral on and the lexical title-relevance sidecar passes because the cited title shares a polysemous token). Reuses the [nli] extra (transformers+torch+optimum); the checkpoint is loaded as AutoModelForSequenceClassification (single regression head emitting a relevance logit), no sentence-transformers dependency. PRIMARY = ms-marco-electra-base picked from the candidate-bench (#000052 §3.2.1 — sep margin +5.35 vs +3.05/+3.08/+2.66 for the others). NOTE: the candidate-bench is contrived; the §7 #18→#27 history says clean-eval doesn't predict real-traffic precision OR recall — this primary is provisional pending §3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) + step 3 (recall-side realism on real-context fixtures). Expect a walk-back.",
|
||||
"relevance_model_version": "relevance-shadow-v1-ms-marco-electra-base",
|
||||
"hf_repo": "cross-encoder/ms-marco-electra-base",
|
||||
"license": "Apache-2.0",
|
||||
"source_url": "https://huggingface.co/cross-encoder/ms-marco-electra-base",
|
||||
"approx_mb": 110,
|
||||
"_comment": "Pinned aboutness/relevance reranker for the #000052 §3.2 SHADOW path. SHADOW MODE ONLY: this manifest does NOT yet fold into governance_policy_hash because shadow output cannot touch audit_mode (#000049 §7 #2 discipline inherited verbatim). If/when promoted to a runtime demotion-only veto, add a single-blob checkpoint_sha256 + tokenizer_sha256 + an explicit relevance_policy_hash that folds into governance_policy_hash before any audit_mode effect. Cross-encoder reranker — single relevance score per (query, document) pair, NOT entailment (that's NLI / arborist.qa.nli; the boundary failure mode this catches is *topic-collision recombination* — answer/claim is on a different topic than its cited source despite shared tokens, the 'Zionist entity' case — which NLI returns neutral on and the lexical title-relevance sidecar passes because the cited title shares a polysemous token). Reuses the [nli] extra (transformers+torch+optimum). PRIMARY = BAAI/bge-reranker-large picked from the §3.2.1 round-2 candidate-bench (7 models across a 20× size spectrum 30MB→560MB; bge-large wins on separation margin at +6.285). NOTE: the candidate-bench is contrived; the §7 #18→#27 history is six verdict flips on the NLI side — clean candidate-bench doesn't predict real-traffic precision OR recall. This primary is provisional pending §3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) + step 3 (recall-side realism on real-context fixtures). Expect a walk-back when the denominator grows.",
|
||||
"relevance_model_version": "relevance-shadow-v1-bge-reranker-large",
|
||||
"hf_repo": "BAAI/bge-reranker-large",
|
||||
"license": "MIT",
|
||||
"source_url": "https://huggingface.co/BAAI/bge-reranker-large",
|
||||
"approx_mb": 560,
|
||||
"score_shape": "single logit per (query, document) pair, NOT a softmax — higher = more relevant; trained on MS MARCO passage-ranking, so the score is well-calibrated for 'is this doc relevant to this query?' and uncalibrated for any other notion of similarity",
|
||||
"max_length": 256,
|
||||
"truncation_policy": "query_then_document@v1",
|
||||
|
|
@ -19,16 +19,27 @@
|
|||
{"relevance_model_version": "relevance-shadow-v1-ms-marco-MiniLM-L-4-v2", "hf_repo": "cross-encoder/ms-marco-MiniLM-L-4-v2", "license": "Apache-2.0", "approx_mb": 50, "note": "SMALL end — added §3.2.1 round-2 (does L-4 still separate the candidate-bench?)"},
|
||||
{"relevance_model_version": "relevance-shadow-v1-ms-marco-MiniLM-L-2-v2", "hf_repo": "cross-encoder/ms-marco-MiniLM-L-2-v2", "license": "Apache-2.0", "approx_mb": 30, "note": "TINY end — added §3.2.1 round-2 (the smallest reranker; floor of the size spectrum)"}
|
||||
],
|
||||
"candidate_bench_results_2026_05_13": {
|
||||
"candidate_bench_results_2026_05_13_round2": {
|
||||
"fixtures": "bench/fixtures/5f/relevance-aboutness-v1.jsonl (14 POS + 12 NEG)",
|
||||
"grid_output": "bench/results/relevance-aboutness-candidatebench.json",
|
||||
"grid_output": "bench/results/relevance-aboutness-candidatebench-round2.json",
|
||||
"size_spectrum_mb": "30 → 50 → 80 → 110 → 130 → 280 → 560 (~20×)",
|
||||
"rank_by_sep_margin": [
|
||||
{"model": "ms-marco-electra-base", "margin": 5.349, "min_pos": 0.050, "max_neg": -5.300, "fp0_theta": 0.050, "catch_neg": "12/12"},
|
||||
{"model": "bge-reranker-base", "margin": 3.076, "min_pos": 2.395, "max_neg": -0.681, "fp0_theta": 2.395, "catch_neg": "12/12"},
|
||||
{"model": "ms-marco-MiniLM-L-6-v2", "margin": 3.053, "min_pos": 6.754, "max_neg": 3.701, "fp0_theta": 6.754, "catch_neg": "12/12"},
|
||||
{"model": "ms-marco-MiniLM-L-12-v2","margin": 2.663, "min_pos": 7.138, "max_neg": 4.475, "fp0_theta": 7.138, "catch_neg": "12/12"}
|
||||
{"model": "bge-reranker-large", "approx_mb": 560, "margin": 6.285, "min_pos": 4.643, "max_neg": -1.641, "fp0_theta": 4.643, "catch_neg": "12/12"},
|
||||
{"model": "ms-marco-electra-base", "approx_mb": 110, "margin": 5.349, "min_pos": 0.050, "max_neg": -5.300, "fp0_theta": 0.050, "catch_neg": "12/12"},
|
||||
{"model": "ms-marco-MiniLM-L-4-v2", "approx_mb": 50, "margin": 3.187, "min_pos": 6.736, "max_neg": 3.549, "fp0_theta": 6.736, "catch_neg": "12/12"},
|
||||
{"model": "bge-reranker-base", "approx_mb": 280, "margin": 3.076, "min_pos": 2.395, "max_neg": -0.681, "fp0_theta": 2.395, "catch_neg": "12/12"},
|
||||
{"model": "ms-marco-MiniLM-L-6-v2", "approx_mb": 80, "margin": 3.053, "min_pos": 6.754, "max_neg": 3.701, "fp0_theta": 6.754, "catch_neg": "12/12"},
|
||||
{"model": "ms-marco-MiniLM-L-12-v2","approx_mb": 130, "margin": 2.663, "min_pos": 7.138, "max_neg": 4.475, "fp0_theta": 7.138, "catch_neg": "12/12"},
|
||||
{"model": "ms-marco-MiniLM-L-2-v2", "approx_mb": 30, "margin": -1.970, "min_pos": 3.371, "max_neg": 5.341, "fp0_theta": 3.371, "catch_neg": "10/12", "note": "NOT clean-separable — floor of the size spectrum; some NEG outscore some POS"}
|
||||
],
|
||||
"note": "All 4 candidates clean-separate on this candidate-bench. Picked electra-base as primary by margin (per §7 #18: rank by separation, not raw score). The thresholds above are CANDIDATE-BENCH ONLY — they will move when the denominator grows (§7 #25→#26 happened three times on the NLI side; assume the same here). demote_below_score stays null until §3.2.2 step 2 (real-traffic shadow sweep) sets it."
|
||||
"findings": {
|
||||
"biggest_is_best_within_a_family": "bge-large (560MB) +6.285 > bge-base (280MB) +3.076 — within BGE family, doubling size doubles margin",
|
||||
"but_not_across_families": "electra-base (110MB) +5.349 beats bge-base (280MB) +3.076 and beats MiniLM-L-12 (130MB) +2.663 — architecture/training-corpus matters more than parameter count in this range",
|
||||
"deeper_is_not_better_within_MS_MARCO_MiniLM": "L-4 (50MB, +3.187) > L-6 (80MB, +3.053) > L-12 (130MB, +2.663) — adding depth from L-6 to L-12 HURTS; same 'deeper ≠ better' pattern as #000049 §7 #24's bart-large-vs-deberta-large",
|
||||
"capacity_floor_between_L_2_and_L_4": "L-2 (30MB) NOT clean-separable (margin -1.970, 10/12 catch); L-4 (50MB) clean-separable (+3.187, 12/12) — the floor is between them",
|
||||
"primary_pick": "bge-reranker-large for the most cushion (+6.285), trading 5× size vs electra-base for ~17% more margin — picks the model most likely to survive a denominator growth per §7 #25's lesson; electra-base is the cost alternate if real-traffic-FP shadow sweep at step 2 shows bge-large isn't worth the size"
|
||||
},
|
||||
"note": "All 6 except L-2 clean-separate on this 26-pair candidate-bench. The thresholds above are CANDIDATE-BENCH ONLY — they WILL move when the denominator grows (§7 #25→#26 walked back twice on the NLI side; the same pattern is expected here). demote_below_score stays null until §3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) sets it."
|
||||
},
|
||||
"rationale_for_cross_encoder_not_bi_encoder": "A bi-encoder (sentence-transformers, vec-style) gives an unsupervised cosine similarity that's good for retrieval *ranking* but not calibrated for the 'is this on-topic?' question we need here. A cross-encoder reranker is trained on (query, doc) pairs with relevance labels — its single logit IS the 'is this doc relevant to this query?' score we want. The architectural cross-link to #000050/#000051's vec hybrid: that work uses *bi-encoders* (fastembed, bge-small) for retrieval recall; this work uses a *cross-encoder* for downstream aboutness scoring. They are complementary — vec finds candidates, the cross-encoder scores them."
|
||||
}
|
||||
|
|
|
|||
377
bench/results/relevance-aboutness-candidatebench-round2.json
Normal file
377
bench/results/relevance-aboutness-candidatebench-round2.json
Normal file
|
|
@ -0,0 +1,377 @@
|
|||
{
|
||||
"generated_at": "2026-05-13T13:58:45Z",
|
||||
"fixtures_file": "/home/fox/nli-data/relevance-aboutness-v1.jsonl",
|
||||
"n_pos": 14,
|
||||
"n_neg": 12,
|
||||
"models": {
|
||||
"relevance-shadow-v1-ms-marco-electra-base": {
|
||||
"hf_repo": "cross-encoder/ms-marco-electra-base",
|
||||
"backend": "torch",
|
||||
"device": "cuda",
|
||||
"n_pairs": 26,
|
||||
"infer_seconds": 0.14,
|
||||
"pos_scores": {
|
||||
"pos-qa-001": 3.448,
|
||||
"pos-qa-002": 2.3775,
|
||||
"pos-qa-003": 3.8591,
|
||||
"pos-qa-004": 1.7753,
|
||||
"pos-qa-005": 3.4742,
|
||||
"pos-qa-006": 1.5494,
|
||||
"pos-qa-007": 3.3069,
|
||||
"pos-cs-001": 0.0497,
|
||||
"pos-cs-002": 2.3681,
|
||||
"pos-cs-003": 4.3109,
|
||||
"pos-cs-004": 2.4378,
|
||||
"pos-cs-005": 2.0715,
|
||||
"pos-cs-006": 5.0473,
|
||||
"pos-cs-007": 4.9582
|
||||
},
|
||||
"neg_scores": {
|
||||
"neg-qa-001": -5.7487,
|
||||
"neg-qa-002": -8.6513,
|
||||
"neg-qa-003": -6.7965,
|
||||
"neg-qa-004": -10.2494,
|
||||
"neg-qa-005": -10.6582,
|
||||
"neg-qa-006": -10.3879,
|
||||
"neg-cs-001": -10.4413,
|
||||
"neg-cs-002": -5.9059,
|
||||
"neg-cs-003": -5.2999,
|
||||
"neg-cs-004": -10.5911,
|
||||
"neg-cs-005": -9.6781,
|
||||
"neg-cs-006": -10.8435
|
||||
},
|
||||
"frontier": {
|
||||
"separation_margin": 5.3495,
|
||||
"fp0_best": {
|
||||
"theta": 0.0497,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
},
|
||||
"pareto": [
|
||||
{
|
||||
"theta": 0.0497,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"relevance-shadow-v1-bge-reranker-large": {
|
||||
"hf_repo": "BAAI/bge-reranker-large",
|
||||
"backend": "torch",
|
||||
"device": "cuda",
|
||||
"n_pairs": 26,
|
||||
"infer_seconds": 0.13,
|
||||
"pos_scores": {
|
||||
"pos-qa-001": 6.6537,
|
||||
"pos-qa-002": 6.5452,
|
||||
"pos-qa-003": 6.0493,
|
||||
"pos-qa-004": 5.2356,
|
||||
"pos-qa-005": 7.0453,
|
||||
"pos-qa-006": 7.1394,
|
||||
"pos-qa-007": 5.8083,
|
||||
"pos-cs-001": 4.6429,
|
||||
"pos-cs-002": 5.5175,
|
||||
"pos-cs-003": 6.8305,
|
||||
"pos-cs-004": 7.1814,
|
||||
"pos-cs-005": 7.122,
|
||||
"pos-cs-006": 9.3351,
|
||||
"pos-cs-007": 6.0121
|
||||
},
|
||||
"neg_scores": {
|
||||
"neg-qa-001": -2.8404,
|
||||
"neg-qa-002": -4.4815,
|
||||
"neg-qa-003": -3.3392,
|
||||
"neg-qa-004": -1.6424,
|
||||
"neg-qa-005": -7.585,
|
||||
"neg-qa-006": -5.5973,
|
||||
"neg-cs-001": -7.0224,
|
||||
"neg-cs-002": -5.5504,
|
||||
"neg-cs-003": -5.5888,
|
||||
"neg-cs-004": -5.1806,
|
||||
"neg-cs-005": -6.5948,
|
||||
"neg-cs-006": -6.2856
|
||||
},
|
||||
"frontier": {
|
||||
"separation_margin": 6.2853,
|
||||
"fp0_best": {
|
||||
"theta": 4.6429,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
},
|
||||
"pareto": [
|
||||
{
|
||||
"theta": 4.6429,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"relevance-shadow-v1-bge-reranker-base": {
|
||||
"hf_repo": "BAAI/bge-reranker-base",
|
||||
"backend": "torch",
|
||||
"device": "cuda",
|
||||
"n_pairs": 26,
|
||||
"infer_seconds": 0.05,
|
||||
"pos_scores": {
|
||||
"pos-qa-001": 8.0599,
|
||||
"pos-qa-002": 7.8094,
|
||||
"pos-qa-003": 6.7217,
|
||||
"pos-qa-004": 7.1199,
|
||||
"pos-qa-005": 8.3108,
|
||||
"pos-qa-006": 7.723,
|
||||
"pos-qa-007": 6.2936,
|
||||
"pos-cs-001": 2.395,
|
||||
"pos-cs-002": 2.7944,
|
||||
"pos-cs-003": 4.9885,
|
||||
"pos-cs-004": 4.7798,
|
||||
"pos-cs-005": 6.3818,
|
||||
"pos-cs-006": 8.417,
|
||||
"pos-cs-007": 5.2747
|
||||
},
|
||||
"neg_scores": {
|
||||
"neg-qa-001": -0.7671,
|
||||
"neg-qa-002": -3.387,
|
||||
"neg-qa-003": -1.9937,
|
||||
"neg-qa-004": -0.6807,
|
||||
"neg-qa-005": -6.0444,
|
||||
"neg-qa-006": -7.8323,
|
||||
"neg-cs-001": -6.4084,
|
||||
"neg-cs-002": -3.9995,
|
||||
"neg-cs-003": -5.247,
|
||||
"neg-cs-004": -4.4321,
|
||||
"neg-cs-005": -7.8685,
|
||||
"neg-cs-006": -6.9767
|
||||
},
|
||||
"frontier": {
|
||||
"separation_margin": 3.0757,
|
||||
"fp0_best": {
|
||||
"theta": 2.395,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
},
|
||||
"pareto": [
|
||||
{
|
||||
"theta": 2.395,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"relevance-shadow-v1-ms-marco-MiniLM-L-6-v2": {
|
||||
"hf_repo": "cross-encoder/ms-marco-MiniLM-L-6-v2",
|
||||
"backend": "torch",
|
||||
"device": "cuda",
|
||||
"n_pairs": 26,
|
||||
"infer_seconds": 0.03,
|
||||
"pos_scores": {
|
||||
"pos-qa-001": 10.3561,
|
||||
"pos-qa-002": 7.6135,
|
||||
"pos-qa-003": 10.2615,
|
||||
"pos-qa-004": 9.6113,
|
||||
"pos-qa-005": 10.761,
|
||||
"pos-qa-006": 10.6969,
|
||||
"pos-qa-007": 10.189,
|
||||
"pos-cs-001": 8.6417,
|
||||
"pos-cs-002": 6.754,
|
||||
"pos-cs-003": 10.2053,
|
||||
"pos-cs-004": 9.1802,
|
||||
"pos-cs-005": 10.58,
|
||||
"pos-cs-006": 9.8569,
|
||||
"pos-cs-007": 9.986
|
||||
},
|
||||
"neg_scores": {
|
||||
"neg-qa-001": 3.7006,
|
||||
"neg-qa-002": -0.0128,
|
||||
"neg-qa-003": 3.1369,
|
||||
"neg-qa-004": 0.2058,
|
||||
"neg-qa-005": -5.2781,
|
||||
"neg-qa-006": -9.0515,
|
||||
"neg-cs-001": -8.1389,
|
||||
"neg-cs-002": -2.9362,
|
||||
"neg-cs-003": -5.963,
|
||||
"neg-cs-004": -6.7567,
|
||||
"neg-cs-005": -5.9802,
|
||||
"neg-cs-006": -9.8485
|
||||
},
|
||||
"frontier": {
|
||||
"separation_margin": 3.0533,
|
||||
"fp0_best": {
|
||||
"theta": 6.754,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
},
|
||||
"pareto": [
|
||||
{
|
||||
"theta": 6.754,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"relevance-shadow-v1-ms-marco-MiniLM-L-12-v2": {
|
||||
"hf_repo": "cross-encoder/ms-marco-MiniLM-L-12-v2",
|
||||
"backend": "torch",
|
||||
"device": "cuda",
|
||||
"n_pairs": 26,
|
||||
"infer_seconds": 0.05,
|
||||
"pos_scores": {
|
||||
"pos-qa-001": 10.1692,
|
||||
"pos-qa-002": 8.139,
|
||||
"pos-qa-003": 10.1767,
|
||||
"pos-qa-004": 9.3511,
|
||||
"pos-qa-005": 10.5464,
|
||||
"pos-qa-006": 10.0394,
|
||||
"pos-qa-007": 9.4536,
|
||||
"pos-cs-001": 8.6255,
|
||||
"pos-cs-002": 7.1379,
|
||||
"pos-cs-003": 9.4853,
|
||||
"pos-cs-004": 9.6927,
|
||||
"pos-cs-005": 10.1209,
|
||||
"pos-cs-006": 9.9942,
|
||||
"pos-cs-007": 10.2065
|
||||
},
|
||||
"neg_scores": {
|
||||
"neg-qa-001": 4.4752,
|
||||
"neg-qa-002": -0.2735,
|
||||
"neg-qa-003": 3.1724,
|
||||
"neg-qa-004": -0.3941,
|
||||
"neg-qa-005": -5.2772,
|
||||
"neg-qa-006": -10.3666,
|
||||
"neg-cs-001": -6.9534,
|
||||
"neg-cs-002": -2.7428,
|
||||
"neg-cs-003": -6.2688,
|
||||
"neg-cs-004": -6.3582,
|
||||
"neg-cs-005": -7.7344,
|
||||
"neg-cs-006": -10.1564
|
||||
},
|
||||
"frontier": {
|
||||
"separation_margin": 2.6627,
|
||||
"fp0_best": {
|
||||
"theta": 7.1379,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
},
|
||||
"pareto": [
|
||||
{
|
||||
"theta": 7.1379,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"relevance-shadow-v1-ms-marco-MiniLM-L-4-v2": {
|
||||
"hf_repo": "cross-encoder/ms-marco-MiniLM-L-4-v2",
|
||||
"backend": "torch",
|
||||
"device": "cuda",
|
||||
"n_pairs": 26,
|
||||
"infer_seconds": 0.02,
|
||||
"pos_scores": {
|
||||
"pos-qa-001": 10.4093,
|
||||
"pos-qa-002": 8.3202,
|
||||
"pos-qa-003": 9.753,
|
||||
"pos-qa-004": 8.5922,
|
||||
"pos-qa-005": 10.3538,
|
||||
"pos-qa-006": 9.6334,
|
||||
"pos-qa-007": 8.9049,
|
||||
"pos-cs-001": 7.4864,
|
||||
"pos-cs-002": 6.736,
|
||||
"pos-cs-003": 9.3345,
|
||||
"pos-cs-004": 9.2342,
|
||||
"pos-cs-005": 10.1494,
|
||||
"pos-cs-006": 9.879,
|
||||
"pos-cs-007": 9.8858
|
||||
},
|
||||
"neg_scores": {
|
||||
"neg-qa-001": 3.5486,
|
||||
"neg-qa-002": -0.0804,
|
||||
"neg-qa-003": 1.3674,
|
||||
"neg-qa-004": -1.9597,
|
||||
"neg-qa-005": -7.1108,
|
||||
"neg-qa-006": -10.4729,
|
||||
"neg-cs-001": -10.0115,
|
||||
"neg-cs-002": -5.1121,
|
||||
"neg-cs-003": -6.8413,
|
||||
"neg-cs-004": -7.3554,
|
||||
"neg-cs-005": -8.4252,
|
||||
"neg-cs-006": -10.3512
|
||||
},
|
||||
"frontier": {
|
||||
"separation_margin": 3.1874,
|
||||
"fp0_best": {
|
||||
"theta": 6.736,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
},
|
||||
"pareto": [
|
||||
{
|
||||
"theta": 6.736,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"relevance-shadow-v1-ms-marco-MiniLM-L-2-v2": {
|
||||
"hf_repo": "cross-encoder/ms-marco-MiniLM-L-2-v2",
|
||||
"backend": "torch",
|
||||
"device": "cuda",
|
||||
"n_pairs": 26,
|
||||
"infer_seconds": 0.02,
|
||||
"pos_scores": {
|
||||
"pos-qa-001": 10.1813,
|
||||
"pos-qa-002": 7.5824,
|
||||
"pos-qa-003": 10.007,
|
||||
"pos-qa-004": 9.9509,
|
||||
"pos-qa-005": 8.9993,
|
||||
"pos-qa-006": 10.4307,
|
||||
"pos-qa-007": 9.0209,
|
||||
"pos-cs-001": 7.6376,
|
||||
"pos-cs-002": 3.3707,
|
||||
"pos-cs-003": 9.1434,
|
||||
"pos-cs-004": 9.0668,
|
||||
"pos-cs-005": 9.4988,
|
||||
"pos-cs-006": 8.6096,
|
||||
"pos-cs-007": 9.6396
|
||||
},
|
||||
"neg_scores": {
|
||||
"neg-qa-001": 5.3408,
|
||||
"neg-qa-002": -0.2153,
|
||||
"neg-qa-003": 4.4499,
|
||||
"neg-qa-004": -1.3149,
|
||||
"neg-qa-005": -6.088,
|
||||
"neg-qa-006": -11.2537,
|
||||
"neg-cs-001": -11.3311,
|
||||
"neg-cs-002": -9.3536,
|
||||
"neg-cs-003": -8.6388,
|
||||
"neg-cs-004": -11.478,
|
||||
"neg-cs-005": -9.13,
|
||||
"neg-cs-006": -9.0799
|
||||
},
|
||||
"frontier": {
|
||||
"separation_margin": -1.97,
|
||||
"fp0_best": {
|
||||
"theta": 3.3707,
|
||||
"catch_neg": 0.8333,
|
||||
"fp_on_pos": 0.0
|
||||
},
|
||||
"pareto": [
|
||||
{
|
||||
"theta": 3.3707,
|
||||
"catch_neg": 0.8333,
|
||||
"fp_on_pos": 0.0
|
||||
},
|
||||
{
|
||||
"theta": 7.5824,
|
||||
"catch_neg": 1.0,
|
||||
"fp_on_pos": 0.0714
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -254,41 +254,66 @@ structure verbatim. Pieces:
|
|||
- `tests/test_relevance_shadow.py` — <!--AUTOCOUNT:tests:tests/test_relevance_shadow.py-->13<!--/AUTOCOUNT--> tests (pure-Python + graceful
|
||||
degradation + the Zionist sanity check); all green.
|
||||
|
||||
**Candidate-bench (2026-05-13)** — 26-pair hand-built fixture set
|
||||
(`bench/fixtures/5f/relevance-aboutness-v1.jsonl`: 14 on-topic POS +
|
||||
12 off-topic / topic-collision NEG, incl. the Zionist field case),
|
||||
all 4 manifest candidates swept on the 4090 (104 GPU NLI pairs total,
|
||||
~1 s per model; `bench/results/relevance-aboutness-candidatebench.json`).
|
||||
**Candidate-bench round 2 (2026-05-13)** — 26-pair hand-built fixture
|
||||
set (`bench/fixtures/5f/relevance-aboutness-v1.jsonl`: 14 on-topic POS
|
||||
+ 12 off-topic / topic-collision NEG, incl. the Zionist field case),
|
||||
**7 candidates spanning ~20× in size (30 MB → 560 MB)** swept on the
|
||||
4090 (~1 s per model; `bench/results/relevance-aboutness-candidatebench-round2.json`).
|
||||
Ranked by separation margin (= `min(POS_score) − max(NEG_score)`;
|
||||
>0 = clean-separable):
|
||||
|
||||
| model (size) | min(POS) | max(NEG) | **sep margin** | fp=0 θ | catch on NEG |
|
||||
| model | size | min(POS) | max(NEG) | **sep margin** | catch on NEG @ fp=0 |
|
||||
|---|---|---|---|---|---|
|
||||
| **`cross-encoder/ms-marco-electra-base`** (~110 MB) | +0.050 | −5.300 | **+5.349** | 0.050 | **12/12 = 1.000** |
|
||||
| `BAAI/bge-reranker-base` (~280 MB) | +2.395 | −0.681 | +3.076 | 2.395 | 12/12 = 1.000 |
|
||||
| `cross-encoder/ms-marco-MiniLM-L-6-v2` (~80 MB) | +6.754 | +3.701 | +3.053 | 6.754 | 12/12 = 1.000 |
|
||||
| `cross-encoder/ms-marco-MiniLM-L-12-v2` (~130 MB) | +7.138 | +4.475 | +2.663 | 7.138 | 12/12 = 1.000 |
|
||||
| **`BAAI/bge-reranker-large`** | 560 MB | +4.64 | −1.64 | **+6.285** | 12/12 |
|
||||
| `cross-encoder/ms-marco-electra-base` | 110 MB | +0.05 | −5.30 | +5.349 | 12/12 |
|
||||
| `cross-encoder/ms-marco-MiniLM-L-4-v2` | 50 MB | +6.74 | +3.55 | +3.187 | 12/12 |
|
||||
| `BAAI/bge-reranker-base` | 280 MB | +2.40 | −0.68 | +3.076 | 12/12 |
|
||||
| `cross-encoder/ms-marco-MiniLM-L-6-v2` | 80 MB | +6.75 | +3.70 | +3.053 | 12/12 |
|
||||
| `cross-encoder/ms-marco-MiniLM-L-12-v2` | 130 MB | +7.14 | +4.48 | +2.663 | 12/12 |
|
||||
| **`cross-encoder/ms-marco-MiniLM-L-2-v2`** | **30 MB** | +3.37 | +5.34 | **−1.970** | **10/12 (NOT separable)** |
|
||||
|
||||
**Read carefully:**
|
||||
- **All 4 candidates clean-separate** the candidate-bench — every model
|
||||
has a θ that catches all 12 off-topic NEG (incl. the Zionist field
|
||||
case) with 0 FP on the 14 POS. So "the reranker discriminates the
|
||||
Zionist-style failure mode" is *not a property of any specific
|
||||
checkpoint*; it's a property of MS-MARCO-trained rerankers (and
|
||||
BGE) as a class on contrived data.
|
||||
- **The deepest MiniLM (L-12) is the *worst* by margin** (+2.66 vs
|
||||
+5.35 for electra-base). Same pattern as #000049 §7 #24's "bart-large
|
||||
is worse than deberta-large" — *deeper ≠ better*; specific
|
||||
fine-tune and architecture dominate. The §3.2 manifest primary
|
||||
moves to `ms-marco-electra-base` for the most cushion (~75% more
|
||||
margin than the next-best, ~100% more than the smallest L-6).
|
||||
- **None of this predicts real-traffic behavior.** The
|
||||
#000049 §7 #18→#27 history is the receipts: clean candidate-bench
|
||||
→ real-bench-qa precision flipped six times on the NLI side; assume
|
||||
the same here. The thresholds above (electra-base at θ=0.05, L-6 at
|
||||
θ=6.75) are *candidate-bench numbers* and will move when the
|
||||
denominator grows. The manifest's `demote_below_score` stays `null`
|
||||
until the real-traffic shadow sweep (§3.2.2 step 2) sets it.
|
||||
**Real findings from extending the spectrum** (the 4-model round-1
|
||||
table earlier in this section was the cost-pick story; round-2 is
|
||||
the *science* story):
|
||||
|
||||
- **Biggest is best within a family** — `bge-reranker-large` (560 MB)
|
||||
has +6.285 margin vs `bge-reranker-base` (280 MB) at +3.076.
|
||||
Doubling size doubled margin within BGE. **But not across families**:
|
||||
`electra-base` (110 MB) at +5.349 *beats* `bge-reranker-base`
|
||||
(280 MB) at +3.076 — 60% smaller, ~75% more margin. *Specific
|
||||
checkpoint / architecture / training corpus dominates parameter
|
||||
count in this size range.*
|
||||
- **Within the MS-MARCO MiniLM family, depth is non-monotonic**:
|
||||
**L-4 (50 MB, +3.187) > L-6 (80 MB, +3.053) > L-12 (130 MB, +2.663)**.
|
||||
Adding depth from L-6 to L-12 *hurts*. Same "deeper ≠ better"
|
||||
pattern as #000049 §7 #24's bart-large-vs-deberta-large finding,
|
||||
reproduced on a different model family.
|
||||
- **Capacity floor is between L-2 and L-4**. `MiniLM-L-2-v2` (30 MB)
|
||||
is the *only* model that fails — margin **−1.970**, NOT
|
||||
clean-separable. Some NEG outscore some POS. At its best fp=0
|
||||
threshold it catches 10/12, not 12/12. L-4 (50 MB) separates
|
||||
cleanly. So somewhere around 40–50 MB / ≥4 layers there's a phase
|
||||
transition for this task.
|
||||
- **All 6 non-floor candidates clean-separate** the 26-pair
|
||||
candidate-bench. So the "reranker catches Zionist-style topic
|
||||
mis-cites" claim is a *property of competent rerankers as a class*,
|
||||
not the specific checkpoint — once you're above the capacity floor.
|
||||
|
||||
**Manifest primary moves to `BAAI/bge-reranker-large`** for the most
|
||||
cushion (+6.285 — the model most likely to survive a denominator
|
||||
growth per #000049 §7 #25's lesson), with `ms-marco-electra-base` as
|
||||
the cost alternate (5× smaller, +5.349 margin — 87% of bge-large's
|
||||
cushion at 20% of the size; if the real-traffic shadow sweep at
|
||||
§3.2.2 step 2 shows bge-large isn't worth the size, walk back).
|
||||
|
||||
**None of this predicts real-traffic behavior.** The #000049 §7
|
||||
#18→#27 history is six verdict flips on the NLI side from candidate-
|
||||
bench → real-bench-qa-pipeline; assume the same here. The thresholds
|
||||
above (bge-large at θ=4.64, electra-base at θ=0.05) are
|
||||
*candidate-bench numbers* and *will* move when the denominator
|
||||
grows. The manifest's `demote_below_score` stays `null` until the
|
||||
real-traffic shadow sweep (§3.2.2 step 2) sets it. **Expect a
|
||||
walk-back.**
|
||||
|
||||
**What the candidate-bench does NOT establish** (preserved from the
|
||||
earlier 6-pair sanity check — the architectural split remains the
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue