russell@unturf.com
7910440e50
Ran diagnose_deflection and diagnose_coherence on the 20+20 NEG fixtures from step 3 parts A+B, plus all 808 STRICT. Result: 0/40 NEG fire on diagnose_deflection (because the fixtures are token-overlap-correct by construction — the question's subject appears in the answer; that's exactly the failure mode #000052 was built to catch beyond lexical). 0/40 on diagnose_coherence (the fixtures are well-formed sentences). So: UNION (relevance OR deflection) = relevance alone (no lift) INTERSECT (relevance AND deflection) = 0/40 Multi-signal combination doesn't help on these failure shapes. But the architectural finding is positive in a different way: the sidecars and the relevance reranker cover NON-OVERLAPPING failure shapes cleanly: - diagnose_coherence: structural breakage (word-salad, vacuous, phrase-component-reuse). Owns the 'incoherent answer' slice. - diagnose_deflection: token-overlap mismatch (subject anchor not in answer). Owns the 'wholesale topic drift' slice. - bge-large relevance: semantic aboutness mismatch despite shared tokens. Owns the 'topic-collision / mis-cite / on-topic-but- not-answering' slice — what §3.2 was built for. Each signal owns its own slice; combining is redundant on these cases. That's the architectural validation of the §3.1 + §3.2 + existing-lexical-sidecars split as orthogonal, not overlapping. Manifest runtime_viability.as_multi_signal_factor updated from 'viable' to 'TESTED — does not lift; sidecars are complementary not combinatorial on these shapes'. Cleaned up stale nli-shadow-grid-n1-minilm.json. Also cleaned up a stale nli-shadow-grid JSON. The §3.2 arc is now complete: step 1 candidate-bench (overclaim, contrived data) step 2 real STRICT FP (over-pessimistic 'NOT VIABLE') step 3A real-context deflection (positive reversal, 55% at fp=0) step 3B real-context mis-cite (100% at fp=0, motivating failure covered) step 3C multi-signal combination (no lift; clean architectural split) The recommended operating point holds: bge-reranker-large + cleaned + θ ≤ -2.42 → 100% mis-cite, 55% deflect, 0% STRICT FP. |
||
|---|---|---|
| .. | ||
| batteries | ||
| fixtures | ||
| results | ||
| scripts | ||
| emergent_log.jsonl | ||
| prometheus_sigma_trigger_probe.py | ||
| qa_questions.txt | ||
| qa_questions_canonical_witness_npower.txt | ||
| qa_questions_metacog_subset.txt | ||
| qa_questions_progressive_and.txt | ||
| qa_questions_quantifier_baseline.txt | ||
| qa_questions_quantifier_subset.txt | ||
| qa_questions_smoke.txt | ||
| qa_questions_warrant_chain_aggressive.txt | ||
| qa_questions_warrant_chain_paraphrase.txt | ||
| qa_questions_warrant_chain_probe.txt | ||
| qa_sweep.py | ||
| run.sh | ||