russell@unturf.com
902e676d4d
Built bench/fixtures/5f/relevance-miscite-realcontext-v1.jsonl —
20 hand-crafted (claim, source) mis-cite pairs: claim about X, source
about Y, X≠Y but shared tokens. Each pair survives the lexical
title-relevance + verifier sidecars by construction. This is the
Zionist-entity failure mode (claim about a different entity than the
cited source, both lexically related).
Examples:
- 'Mercury is the smallest planet' / Roman-god Mercury source
- 'Java is a programming language' / Java-the-island source
- 'Apple Inc. was co-founded by Steve Jobs' / apple-the-fruit source
- 'The Eiffel Tower is in Paris' / Gustave-Eiffel-person source
- 'Mozart composed The Magic Flute' / Mozart-effect-theory source
Headline (bge-reranker-large + cleaned + θ ≤ -2.42):
Mis-cite catch: 20/20 = 100% ← FULL COVERAGE of motivating shape
Deflection catch: 11/20 = 55%
Combined NEG: 31/40 = 78%
Real-STRICT FP: 0/808 = 0% ← strictly safe
bge-large mis-cite scores: -9.37 to -3.81 (max). STRICT min: -2.42.
Mis-cite is STRICTLY SEPARABLE from real STRICT — there's a 1.4-pt
gap with no overlap. (Deflection harder; some overlap with weak STRICT.)
MiniLM-L-6 cost-pick (5× smaller, cleaned, θ=+3):
100% mis-cite + 65% deflect + 0.4% STRICT FP
Mis-cite is structurally MUCH easier than deflection — both models
hit 95-100% mis-cite catch at modest θ; deflection is harder because
'answer doesn't quite address question' can look like a weak STRICT.
That's appropriate: mis-cite is 'wrong topic entirely'; deflection
is 'right topic, not answering'.
Manifest:
- runtime_viability flipped (step 2 → step 3): NOT VIABLE → VIABLE
at the bge-large fp=0 operating point.
- demote_below_score still null pending fox+dav1d sign-off (setting
it folds relevance_policy_hash into governance_policy_hash per
#000049 §7 #2).
- PRIMARY RECOMMENDATION: bge-reranker-large + cleaned +
θ ≤ -2.42 → 100% mis-cite, 55% deflect, 0% STRICT FP.
Production verifier unchanged; still SHADOW. The §3.2 arc went:
candidate-bench (overclaim) → real STRICT step 2 (over-pessimistic
NOT VIABLE) → real-context deflection step 3 (positive reversal,
55% at fp=0) → real-context mis-cite step 3 part B (full closure
at fp=0 for the motivating failure mode). Eight meta-lesson
instances over the §000049 + §000052 arc, with the sharpest one
yet: clean candidate-bench can mis-predict in BOTH directions —
real-data fixtures on BOTH precision and recall axes are the
only load-bearing measurement.
67 lines
12 KiB
JSON
67 lines
12 KiB
JSON
{
|
||
"_comment": "Pinned aboutness/relevance reranker for the #000052 §3.2 SHADOW path. SHADOW MODE ONLY — and §3.2.2 step 2 (2026-05-13, bench/results/relevance-shadow-sweep-pooled808.json) showed the candidate-bench θ values DO NOT survive real bench-qa traffic: at the candidate-bench fp=0 θ, every one of 6 swept rerankers false-fires on 4.5–21.8% of pooled bench-qa STRICT answers (bge-large worst at 21.8%, MiniLM-L-2 best at 4.5%). The smallest θ that yields fp=0 on real STRICT is BELOW the candidate-bench NEG max for every model — meaning at the runtime-safe θ, catch on the 12 candidate-bench NEG = 0/12. Structural reason: real STRICT answers have a much wider score distribution (bge-large: min -2.20, p10 +2.99, p50 +5.80) than the tight contrived candidate-bench POS band, so the distributions overlap heavily and no θ separates them. The relevance reranker CANNOT be promoted to a runtime demotion-only veto on this design (joins #000049 §7 #27 in 'partial-on-candidate-bench, fails the real-traffic gate'). The PRIMARY below is the candidate-bench winner kept for reference; the operating point is null because no runtime threshold is viable. The score remains useful as: (a) an advisory soft signal (render-tail `· relevance: X.Y`, no thresholding); (b) one factor in a multi-signal advisory (relevance + title-mismatch + coherence + deflection); (c) contrastive use (Q→A vs claim→source delta, or relative ranking within an answer's claims). Reuses [nli] extra (transformers+torch+optimum).",
|
||
"relevance_model_version": "relevance-shadow-v1-bge-reranker-large",
|
||
"hf_repo": "BAAI/bge-reranker-large",
|
||
"license": "MIT",
|
||
"source_url": "https://huggingface.co/BAAI/bge-reranker-large",
|
||
"approx_mb": 560,
|
||
"score_shape": "single logit per (query, document) pair, NOT a softmax — higher = more relevant; trained on MS MARCO passage-ranking, so the score is well-calibrated for 'is this doc relevant to this query?' and uncalibrated for any other notion of similarity",
|
||
"max_length": 256,
|
||
"truncation_policy": "query_then_document@v1",
|
||
"pair_serialization": "relevance_pair@v1",
|
||
"demote_below_score": null,
|
||
"demote_below_score_comment": "STAYS NULL pending fox+dav1d sign-off, but step 3 part B (2026-05-13 — real-context deflection [n=20] + mis-cite [n=20] fixtures) confirmed VIABLE: bge-reranker-large + cleaned + θ ≤ -2.42 catches 100% of real-context MIS-CITE (the Zionist-shape motivating failure) AND 55% of real-context DEFLECTION, with strictly 0/808 real-STRICT FP. Mis-cite max bge-score is -3.81 vs STRICT min -2.42 — strictly separable for mis-cite. Deflection harder (some deflections score similarly to weak STRICT). MiniLM-L-6 + cleaned + θ=+3: 100% mis-cite + 65% deflect at 0.4% STRICT FP — cost-pick. Setting demote_below_score requires the runtime promotion decision per #000049 §7 #2 — at that point relevance_policy_hash (model+max_length+cleaning+θ+aggregation) folds into governance_policy_hash.",
|
||
"runtime_viability": {
|
||
"as_runtime_demotion_veto": "VIABLE at low-to-moderate FP — REVISED 2026-05-13 §3.2.2 step 3. The step-2 'NOT VIABLE' verdict was an artifact of using candidate-bench NEG as the recall denominator (tight contrived NEG band overlapped real STRICT). With REAL-CONTEXT deflection fixtures as the recall side (bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl, 20 hand-crafted Q→A deflections against real bench-qa questions), bge-reranker-large catches 11/20 = 55% at strictly 0% real-STRICT FP (θ ≤ -2.42), and pushes to ~95% at ~2% FP. MiniLM-L-6 (cleaned): 65% catch at 0.4% FP (θ=+3), 90% at 1.9% FP (θ=+5). These ARE usable runtime soft-veto operating points.",
|
||
"step3_pareto_minilm_l_6_cleaned_2026_05_13_partB": [
|
||
{"theta": -5.29, "miscite_catch": "14/20 = 0.70", "deflect_catch": "2/20 = 0.10", "combined": "16/40 = 0.40", "strict_fp": "0/808 = 0.000", "note": "fp=0 floor"},
|
||
{"theta": 0.0, "miscite_catch": "19/20 = 0.95", "deflect_catch": "7/20 = 0.35", "combined": "26/40 = 0.65", "strict_fp": "2/808 = 0.003"},
|
||
{"theta": 2.0, "miscite_catch": "19/20 = 0.95", "deflect_catch": "11/20 = 0.55", "combined": "30/40 = 0.75", "strict_fp": "2/808 = 0.003"},
|
||
{"theta": 3.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "13/20 = 0.65", "combined": "33/40 = 0.82", "strict_fp": "3/808 = 0.004", "note": "cost-pick — 100% mis-cite + 65% deflect at sub-1% FP"},
|
||
{"theta": 5.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "18/20 = 0.90", "combined": "38/40 = 0.95", "strict_fp": "15/808 = 0.019"},
|
||
{"theta": 7.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "20/20 = 1.00", "combined": "40/40 = 1.00", "strict_fp": "84/808 = 0.104", "note": "too FP for runtime"}
|
||
],
|
||
"step3_pareto_bge_large_cleaned_2026_05_13_partB": [
|
||
{"theta": -2.42, "miscite_catch": "20/20 = 1.00", "deflect_catch": "11/20 = 0.55", "combined": "31/40 = 0.78", "strict_fp": "0/808 = 0.000", "note": "PRIMARY RECOMMENDATION — 100% mis-cite + 55% deflect at strictly 0% FP. Zionist-shape motivating failure fully covered."},
|
||
{"theta": 0.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "17/20 = 0.85", "combined": "37/40 = 0.93", "strict_fp": "15/808 = 0.019"},
|
||
{"theta": 2.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "18/20 = 0.90", "combined": "38/40 = 0.95", "strict_fp": "36/808 = 0.045"},
|
||
{"theta": 3.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "19/20 = 0.95", "combined": "39/40 = 0.97", "strict_fp": "71/808 = 0.088"}
|
||
],
|
||
"step3_finding_miscite_vs_deflection": "Mis-cite is structurally much easier than deflection — bge-large mis-cite scores range -9.37 to -3.81 (max), strictly below STRICT min (-2.42). Deflection range -5.57 to +5.01 overlaps STRICT bottom. So the runtime veto is FULLY effective on mis-cite (the Zionist failure mode) and PARTIALLY effective on deflection. That's appropriate to the structural difference: mis-cite is 'wrong topic entirely', deflection is 'right topic, not answering'.",
|
||
"as_advisory_soft_signal": "viable — surface the raw score on the render tail or as bench telemetry; no threshold needed",
|
||
"as_multi_signal_factor": "viable — combine with title-relevance, deflection, coherence sidecars for a weighted advisory",
|
||
"as_contrastive_signal": "TESTED 2026-05-13 — DOES NOT SEPARATE on real bench-qa data: 97% of STRICT-fires at cb-θ have BOTH Q→A and A→ctx_lead low (co-varying); only 3% match the 'high A→ctx, low Q→A' deflection signature. Contrastive rescue path closed — but it's no longer needed since real-context deflections (step 3) DID separate cleanly with absolute thresholds.",
|
||
"preprocessing_cleaning_is_free_improvement": "claim-lattice metadata [E\\d+ | … | hash: '…'] depresses scores by ~3-6 pts on cleaned vs raw; clean_for_relevance is baked into check_* by default. Universal win across all 6 swept models (-0.5 to -5.6 pts FP)."
|
||
},
|
||
"alternates": [
|
||
{"relevance_model_version": "relevance-shadow-v1-bge-reranker-large", "hf_repo": "BAAI/bge-reranker-large", "license": "MIT", "approx_mb": 560, "note": "LARGE end of the spectrum — added §3.2.1 round-2 (does bigger BGE beat bge-base?)"},
|
||
{"relevance_model_version": "relevance-shadow-v1-bge-reranker-base", "hf_repo": "BAAI/bge-reranker-base", "license": "MIT", "approx_mb": 280, "candidate_bench_margin": 3.076, "note": "BAAI general-purpose reranker; second-best candidate-bench margin after electra-base (round 1)"},
|
||
{"relevance_model_version": "relevance-shadow-v1-ms-marco-MiniLM-L-6-v2", "hf_repo": "cross-encoder/ms-marco-MiniLM-L-6-v2", "license": "Apache-2.0", "approx_mb": 80, "candidate_bench_margin": 3.053, "note": "the original primary before §3.2.1 candidate-bench"},
|
||
{"relevance_model_version": "relevance-shadow-v1-ms-marco-MiniLM-L-12-v2", "hf_repo": "cross-encoder/ms-marco-MiniLM-L-12-v2", "license": "Apache-2.0", "approx_mb": 130, "candidate_bench_margin": 2.663, "note": "deeper L-12 MiniLM; round-1 worst — deeper ≠ better here"},
|
||
{"relevance_model_version": "relevance-shadow-v1-ms-marco-MiniLM-L-4-v2", "hf_repo": "cross-encoder/ms-marco-MiniLM-L-4-v2", "license": "Apache-2.0", "approx_mb": 50, "note": "SMALL end — added §3.2.1 round-2 (does L-4 still separate the candidate-bench?)"},
|
||
{"relevance_model_version": "relevance-shadow-v1-ms-marco-MiniLM-L-2-v2", "hf_repo": "cross-encoder/ms-marco-MiniLM-L-2-v2", "license": "Apache-2.0", "approx_mb": 30, "note": "TINY end — added §3.2.1 round-2 (the smallest reranker; floor of the size spectrum)"}
|
||
],
|
||
"candidate_bench_results_2026_05_13_round2": {
|
||
"fixtures": "bench/fixtures/5f/relevance-aboutness-v1.jsonl (14 POS + 12 NEG)",
|
||
"grid_output": "bench/results/relevance-aboutness-candidatebench-round2.json",
|
||
"size_spectrum_mb": "30 → 50 → 80 → 110 → 130 → 280 → 560 (~20×)",
|
||
"rank_by_sep_margin": [
|
||
{"model": "bge-reranker-large", "approx_mb": 560, "margin": 6.285, "min_pos": 4.643, "max_neg": -1.641, "fp0_theta": 4.643, "catch_neg": "12/12"},
|
||
{"model": "ms-marco-electra-base", "approx_mb": 110, "margin": 5.349, "min_pos": 0.050, "max_neg": -5.300, "fp0_theta": 0.050, "catch_neg": "12/12"},
|
||
{"model": "ms-marco-MiniLM-L-4-v2", "approx_mb": 50, "margin": 3.187, "min_pos": 6.736, "max_neg": 3.549, "fp0_theta": 6.736, "catch_neg": "12/12"},
|
||
{"model": "bge-reranker-base", "approx_mb": 280, "margin": 3.076, "min_pos": 2.395, "max_neg": -0.681, "fp0_theta": 2.395, "catch_neg": "12/12"},
|
||
{"model": "ms-marco-MiniLM-L-6-v2", "approx_mb": 80, "margin": 3.053, "min_pos": 6.754, "max_neg": 3.701, "fp0_theta": 6.754, "catch_neg": "12/12"},
|
||
{"model": "ms-marco-MiniLM-L-12-v2","approx_mb": 130, "margin": 2.663, "min_pos": 7.138, "max_neg": 4.475, "fp0_theta": 7.138, "catch_neg": "12/12"},
|
||
{"model": "ms-marco-MiniLM-L-2-v2", "approx_mb": 30, "margin": -1.970, "min_pos": 3.371, "max_neg": 5.341, "fp0_theta": 3.371, "catch_neg": "10/12", "note": "NOT clean-separable — floor of the size spectrum; some NEG outscore some POS"}
|
||
],
|
||
"findings": {
|
||
"biggest_is_best_within_a_family": "bge-large (560MB) +6.285 > bge-base (280MB) +3.076 — within BGE family, doubling size doubles margin",
|
||
"but_not_across_families": "electra-base (110MB) +5.349 beats bge-base (280MB) +3.076 and beats MiniLM-L-12 (130MB) +2.663 — architecture/training-corpus matters more than parameter count in this range",
|
||
"deeper_is_not_better_within_MS_MARCO_MiniLM": "L-4 (50MB, +3.187) > L-6 (80MB, +3.053) > L-12 (130MB, +2.663) — adding depth from L-6 to L-12 HURTS; same 'deeper ≠ better' pattern as #000049 §7 #24's bart-large-vs-deberta-large",
|
||
"capacity_floor_between_L_2_and_L_4": "L-2 (30MB) NOT clean-separable (margin -1.970, 10/12 catch); L-4 (50MB) clean-separable (+3.187, 12/12) — the floor is between them",
|
||
"primary_pick": "bge-reranker-large for the most cushion (+6.285), trading 5× size vs electra-base for ~17% more margin — picks the model most likely to survive a denominator growth per §7 #25's lesson; electra-base is the cost alternate if real-traffic-FP shadow sweep at step 2 shows bge-large isn't worth the size"
|
||
},
|
||
"note": "All 6 except L-2 clean-separate on this 26-pair candidate-bench. The thresholds above are CANDIDATE-BENCH ONLY — they WILL move when the denominator grows (§7 #25→#26 walked back twice on the NLI side; the same pattern is expected here). demote_below_score stays null until §3.2.2 step 2 (real-traffic shadow sweep on pooled bench-qa STRICT) sets it."
|
||
},
|
||
"rationale_for_cross_encoder_not_bi_encoder": "A bi-encoder (sentence-transformers, vec-style) gives an unsupervised cosine similarity that's good for retrieval *ranking* but not calibrated for the 'is this on-topic?' question we need here. A cross-encoder reranker is trained on (query, doc) pairs with relevance labels — its single logit IS the 'is this doc relevant to this query?' score we want. The architectural cross-link to #000050/#000051's vec hybrid: that work uses *bi-encoders* (fastembed, bge-small) for retrieval recall; this work uses a *cross-encoder* for downstream aboutness scoring. They are complementary — vec finds candidates, the cross-encoder scores them."
|
||
}
|