From 6f3a04bc8d6f6888179e22cc88fa4a223d686dcc Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Wed, 13 May 2026 15:06:04 -0400 Subject: [PATCH] =?UTF-8?q?#000052=20=C2=A73.2.2.b:=20contrastive=20signal?= =?UTF-8?q?=20experiment=20=E2=80=94=20DOES=20NOT=20SEPARATE=20on=20real?= =?UTF-8?q?=20bench-qa=20(97%=20of=20STRICT-fires=20have=20both=20Q?= =?UTF-8?q?=E2=86=92A=20and=20A=E2=86=92ctx=20low;=20only=203%=20match=20t?= =?UTF-8?q?he=20deflection=20signature).=20Rescue=20path=20closed;=20soft-?= =?UTF-8?q?signal=20role=20stands;=20runtime=20veto=20conclusively=20parke?= =?UTF-8?q?d.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tested Q→A score vs A→context_lead score on 808 pooled STRICT pairs with cleaned MiniLM-L-6. The hypothesis was: deflected answers should have HIGH A→ctx_lead but LOW Q→A (still grounded but off-topic). Reality: Q→A quantiles: p10 +6.90 p50 +9.32 p90 +10.57 A→ctx quantiles: p10 -6.73 p50 +1.51 p90 +8.68 Δ=Q→A-Actx: p10 +0.90 p50 +6.95 p90 +14.62 Of the 73 cleaned STRICT-fires at cb-θ=6.754: - 71 (97%) have BOTH axes low (co-varying — not deflection) - 2 (3%) match the deflection signature; both are the same 'who wrote GNU linux?' replica Why it doesn't separate: the A→ctx axis isn't measuring what the hypothesis assumed. A focused-claim against a 30KB topic-broad Wikipedia haystack scores LOW by default — that IS the normal STRICT shape (the answer is one clause in a sprawling document). The cross-encoder expects the document to be ABOUT the query (MS-MARCO retrieval shape); a STRICT (answer, full-haystack) pair violates that. So both axes co-vary and the delta is a poor discriminator. This closes the third precision-side rescue path I'd left open after step 2's verdict: ✗ absolute threshold — universal walk-back across 6 models ✗ cleaning preprocessing — helps but doesn't separate ✗ contrastive Q→A vs A→ctx — doesn't separate either The relevance reranker conclusively cannot be a runtime demotion-only veto on this design. The soft-signal uses (render-tail, multi-signal advisory, etc.) remain viable. runtime_viability.as_contrastive_signal updated from 'worth measuring' to 'tested, doesn't separate'. Production verifier unchanged; no audit_mode effect. --- arborist/qa/relevance/manifest.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arborist/qa/relevance/manifest.json b/arborist/qa/relevance/manifest.json index 41b180a..83532b3 100644 --- a/arborist/qa/relevance/manifest.json +++ b/arborist/qa/relevance/manifest.json @@ -15,7 +15,8 @@ "as_runtime_demotion_veto": "NOT VIABLE on this design (§3.2.2 step 2 measurement)", "as_advisory_soft_signal": "viable — surface the raw score on the render tail or as bench telemetry; no threshold needed", "as_multi_signal_factor": "viable — combine with title-relevance, deflection, coherence sidecars for a weighted advisory", - "as_contrastive_signal": "viable and worth measuring — relative score (Q→A vs claim→source, or within an answer's claims) may discriminate where absolute threshold does not" + "as_contrastive_signal": "TESTED 2026-05-13 — DOES NOT SEPARATE on real bench-qa data: 97% of STRICT-fires at cb-θ have BOTH Q→A and A→ctx_lead low (co-varying); only 3% match the 'high A→ctx, low Q→A' deflection signature. The A→ctx axis isn't measuring what was hoped — a focused-claim against a 30KB topic-broad context scores low by default (that's the normal STRICT shape), so the delta doesn't discriminate. Contrastive rescue path is closed.", + "preprocessing_cleaning_is_free_improvement": "claim-lattice metadata [E\\d+ | … | hash: '…'] depresses scores by ~3-6 pts on cleaned vs raw; clean_for_relevance is baked into check_* by default. Universal win across all 6 swept models (-0.5 to -5.6 pts FP)." }, "alternates": [ {"relevance_model_version": "relevance-shadow-v1-bge-reranker-large", "hf_repo": "BAAI/bge-reranker-large", "license": "MIT", "approx_mb": 560, "note": "LARGE end of the spectrum — added §3.2.1 round-2 (does bigger BGE beat bge-base?)"},