Built bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl —
20 hand-crafted deflection answers against real bench-qa questions
(each: coherent well-formed answer using context-tokens, but NOT
addressing the question — the kind of failure §3.2 was built for).
Real-context deflection scores (cleaned, MiniLM-L-6):
min -6.71, median +1.56, max +6.34
Real-context deflection scores (cleaned, bge-reranker-large):
min -5.57, median -3.02, max +5.01
vs STRICT (cleaned):
MiniLM-L-6: min -5.29, p10 +6.90, median +9.32
bge-large: min -2.42, p10 +3.56, median +6.48
Distributions are CLEAN-separable on real data — 0% of MiniLM
deflections score above p10(STRICT); bge-large median is -3.02 vs
STRICT median +6.48.
Pareto frontier (MiniLM-L-6, cleaned):
θ=-5.29: catch 10% / fp 0% (fp=0 floor, low signal but safe)
θ=+3: catch 65% / fp 0.4% ← strong runtime soft-veto
θ=+5: catch 90% / fp 1.9% ← aggressive soft-veto
θ=+7: catch 100% / fp 10.4% (too FP for runtime)
bge-reranker-large:
θ=-2.42: catch 55% / fp 0% ← VIABLE runtime soft-veto at strict fp=0
The step-2 'NOT VIABLE' verdict was an artifact of using the
candidate-bench NEG distribution (tight contrived band, overlapped
real STRICT) as the recall denominator. Real-context deflections sit
in a much lower band (median around -3 for bge-large) than real
STRICT, so absolute thresholds DO separate them cleanly. The
relevance reranker IS a viable runtime soft-veto on this design —
the candidate-bench-only step 2 measurement misled us.
Manifest's runtime_viability.as_runtime_demotion_veto flipped from
'NOT VIABLE' to 'VIABLE at low-to-moderate FP', with both per-model
Pareto frontiers recorded. demote_below_score stays null pending
fox+dav1d sign-off; setting it folds relevance_policy_hash into
governance_policy_hash per #000049 §7 #2 discipline.
Still SHADOW; production verifier unchanged; no audit_mode effect.
Eighth instance of the meta-lesson, with the lesson sharpening AGAIN:
clean candidate-bench can MIS-PREDICT in BOTH directions — over-
optimistic on threshold (step 2) AND over-pessimistic on viability
(step 3 reversal). Real-data fixtures are the only load-bearing
denominators on either axis.
russell@unturf.com
7b1a57ade9