engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000052 §3.2.2.b: contrastive signal experiment — DOES NOT SEPARATE on real bench-qa (97% of STRICT-fires have both Q→A and A→ctx low; only 3% match the deflection signature). Rescue path closed; soft-signal role stands; runtime veto conclusively parked.

Browse source 
Tested Q→A score vs A→context_lead score on 808 pooled STRICT pairs
with cleaned MiniLM-L-6. The hypothesis was: deflected answers
should have HIGH A→ctx_lead but LOW Q→A (still grounded but
off-topic). Reality:

  Q→A      quantiles: p10 +6.90  p50 +9.32  p90 +10.57
  A→ctx    quantiles: p10 -6.73  p50 +1.51  p90 +8.68
  Δ=Q→A-Actx: p10 +0.90  p50 +6.95  p90 +14.62

Of the 73 cleaned STRICT-fires at cb-θ=6.754:
  - 71 (97%) have BOTH axes low (co-varying — not deflection)
  - 2 (3%) match the deflection signature; both are the same
    'who wrote GNU linux?' replica

Why it doesn't separate: the A→ctx axis isn't measuring what the
hypothesis assumed. A focused-claim against a 30KB topic-broad
Wikipedia haystack scores LOW by default — that IS the normal
STRICT shape (the answer is one clause in a sprawling document).
The cross-encoder expects the document to be ABOUT the query
(MS-MARCO retrieval shape); a STRICT (answer, full-haystack) pair
violates that. So both axes co-vary and the delta is a poor
discriminator.

This closes the third precision-side rescue path I'd left open
after step 2's verdict:
  ✗ absolute threshold — universal walk-back across 6 models
  ✗ cleaning preprocessing — helps but doesn't separate
  ✗ contrastive Q→A vs A→ctx — doesn't separate either

The relevance reranker conclusively cannot be a runtime
demotion-only veto on this design. The soft-signal uses
(render-tail, multi-signal advisory, etc.) remain viable.
runtime_viability.as_contrastive_signal updated from 'worth
measuring' to 'tested, doesn't separate'. Production verifier
unchanged; no audit_mode effect.
This commit is contained in:
russell@unturf.com 2026-05-13 15:06:04 -04:00
parent e4cc3293b5
commit 6f3a04bc8d
No known key found for this signature in database
1 changed files with 2 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

3
arborist/qa/relevance/manifest.json
View file

@ -15,7 +15,8 @@
"as_runtime_demotion_veto": "NOT VIABLE on this design (§3.2.2 step 2 measurement)",
"as_advisory_soft_signal": "viable — surface the raw score on the render tail or as bench telemetry; no threshold needed",
"as_multi_signal_factor": "viable — combine with title-relevance, deflection, coherence sidecars for a weighted advisory",
"as_contrastive_signal": "viable and worth measuring — relative score (Q→A vs claim→source, or within an answer's claims) may discriminate where absolute threshold does not"
"as_contrastive_signal": "TESTED 2026-05-13 — DOES NOT SEPARATE on real bench-qa data: 97% of STRICT-fires at cb-θ have BOTH Q→A and A→ctx_lead low (co-varying); only 3% match the 'high A→ctx, low Q→A' deflection signature. The A→ctx axis isn't measuring what was hoped — a focused-claim against a 30KB topic-broad context scores low by default (that's the normal STRICT shape), so the delta doesn't discriminate. Contrastive rescue path is closed.",
"preprocessing_cleaning_is_free_improvement": "claim-lattice metadata [E\\d+ | … | hash: '…'] depresses scores by ~3-6 pts on cleaned vs raw; clean_for_relevance is baked into check_* by default. Universal win across all 6 swept models (-0.5 to -5.6 pts FP)."
},
"alternates": [
{"relevance_model_version": "relevance-shadow-v1-bge-reranker-large", "hf_repo": "BAAI/bge-reranker-large", "license": "MIT", "approx_mb": 560, "note": "LARGE end of the spectrum — added §3.2.1 round-2 (does bigger BGE beat bge-base?)"},