engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000052 §3.2.2 step 3: real-context deflection fixtures (n=20) — verdict REVERSES: bge-large catches 55% of real deflections at strictly 0% real-STRICT FP, MiniLM-L-6 catches 65% at 0.4% FP. Runtime soft-veto IS viable.

Browse source 
Built bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl —
20 hand-crafted deflection answers against real bench-qa questions
(each: coherent well-formed answer using context-tokens, but NOT
addressing the question — the kind of failure §3.2 was built for).

Real-context deflection scores (cleaned, MiniLM-L-6):
  min -6.71, median +1.56, max +6.34
Real-context deflection scores (cleaned, bge-reranker-large):
  min -5.57, median -3.02, max +5.01

vs STRICT (cleaned):
  MiniLM-L-6: min -5.29, p10 +6.90, median +9.32
  bge-large:  min -2.42, p10 +3.56, median +6.48

Distributions are CLEAN-separable on real data — 0% of MiniLM
deflections score above p10(STRICT); bge-large median is -3.02 vs
STRICT median +6.48.

Pareto frontier (MiniLM-L-6, cleaned):
  θ=-5.29: catch 10% / fp 0%      (fp=0 floor, low signal but safe)
  θ=+3:    catch 65% / fp 0.4%    ← strong runtime soft-veto
  θ=+5:    catch 90% / fp 1.9%    ← aggressive soft-veto
  θ=+7:    catch 100% / fp 10.4%  (too FP for runtime)

bge-reranker-large:
  θ=-2.42: catch 55% / fp 0%      ← VIABLE runtime soft-veto at strict fp=0

The step-2 'NOT VIABLE' verdict was an artifact of using the
candidate-bench NEG distribution (tight contrived band, overlapped
real STRICT) as the recall denominator. Real-context deflections sit
in a much lower band (median around -3 for bge-large) than real
STRICT, so absolute thresholds DO separate them cleanly. The
relevance reranker IS a viable runtime soft-veto on this design —
the candidate-bench-only step 2 measurement misled us.

Manifest's runtime_viability.as_runtime_demotion_veto flipped from
'NOT VIABLE' to 'VIABLE at low-to-moderate FP', with both per-model
Pareto frontiers recorded. demote_below_score stays null pending
fox+dav1d sign-off; setting it folds relevance_policy_hash into
governance_policy_hash per #000049 §7 #2 discipline.

Still SHADOW; production verifier unchanged; no audit_mode effect.
Eighth instance of the meta-lesson, with the lesson sharpening AGAIN:
clean candidate-bench can MIS-PREDICT in BOTH directions — over-
optimistic on threshold (step 2) AND over-pessimistic on viability
(step 3 reversal). Real-data fixtures are the only load-bearing
denominators on either axis.
This commit is contained in:
russell@unturf.com 2026-05-13 15:11:27 -04:00
parent 6f3a04bc8d
commit 7b1a57ade9
No known key found for this signature in database
2 changed files with 34 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

16
arborist/qa/relevance/manifest.json
View file

@ -10,12 +10,22 @@
"truncation_policy": "query_then_document@v1",
"pair_serialization": "relevance_pair@v1",
"demote_below_score": null,
"demote_below_score_comment": "STAYS NULL — §3.2.2 step 2 (2026-05-13 sweep over 808 pooled bench-qa STRICT pairs, all 6 swept rerankers) showed NO θ exists that yields fp=0 on real STRICT AND catches any candidate-bench NEG. The score distributions of real STRICT vs candidate-bench NEG overlap heavily; the lexical-candidate relevance reranker cannot operate as a runtime demotion-only veto on this design. The advisory-soft-signal uses (render-tail, multi-signal advisory, contrastive Q→A vs claim→source) remain viable — they don't need a threshold.",
"demote_below_score_comment": "STAYS NULL pending fox+dav1d sign-off, but step 3 (2026-05-13 real-context deflection set) showed VIABLE operating points exist on real bench-qa: bge-reranker-large at θ ≤ -2.42 catches 55% of real deflections at 0% real-STRICT FP; MiniLM-L-6 at θ=+3 catches 65% at 0.4% FP. These are the candidate runtime-veto operating points. The previous 'NOT VIABLE' verdict was a step-2-only artifact (candidate-bench NEG distribution overlapped real STRICT; real-context deflections do not). Setting demote_below_score requires the runtime promotion decision per #000049 §7 #2 — at that point nli_policy_hash-equivalent (model+max_length+cleaning+θ) folds into governance_policy_hash.",
"runtime_viability": {
"as_runtime_demotion_veto": "NOT VIABLE on this design (§3.2.2 step 2 measurement)",
"as_runtime_demotion_veto": "VIABLE at low-to-moderate FP — REVISED 2026-05-13 §3.2.2 step 3. The step-2 'NOT VIABLE' verdict was an artifact of using candidate-bench NEG as the recall denominator (tight contrived NEG band overlapped real STRICT). With REAL-CONTEXT deflection fixtures as the recall side (bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl, 20 hand-crafted Q→A deflections against real bench-qa questions), bge-reranker-large catches 11/20 = 55% at strictly 0% real-STRICT FP (θ ≤ -2.42), and pushes to ~95% at ~2% FP. MiniLM-L-6 (cleaned): 65% catch at 0.4% FP (θ=+3), 90% at 1.9% FP (θ=+5). These ARE usable runtime soft-veto operating points.",
"step3_pareto_minilm_l_6_cleaned": [
{"theta": -5.29, "deflect_catch": "2/20 = 0.10", "strict_fp": "0/808 = 0.000", "note": "fp=0 floor; low signal but safe"},
{"theta": 3.0, "deflect_catch": "13/20 = 0.65", "strict_fp": "3/808 = 0.004", "note": "conservative runtime veto — strong catch, sub-1% FP"},
{"theta": 5.0, "deflect_catch": "18/20 = 0.90", "strict_fp": "15/808 = 0.019", "note": "aggressive — 90% catch at ~2% FP"},
{"theta": 7.0, "deflect_catch": "20/20 = 1.00", "strict_fp": "84/808 = 0.104", "note": "full coverage but too FP for runtime"}
],
"step3_pareto_bge_large_cleaned": [
{"theta": -2.42, "deflect_catch": "11/20 = 0.55", "strict_fp": "0/808 = 0.000", "note": "fp=0 floor; 55% deflection catch is real and usable"},
{"theta": 3.56, "deflect_catch": "19/20 = 0.95", "strict_fp": "~81/808 ≈ 0.10", "note": "interpolated from STRICT p10; precise FP needs per-row recomputation"}
],
"as_advisory_soft_signal": "viable — surface the raw score on the render tail or as bench telemetry; no threshold needed",
"as_multi_signal_factor": "viable — combine with title-relevance, deflection, coherence sidecars for a weighted advisory",
"as_contrastive_signal": "TESTED 2026-05-13 — DOES NOT SEPARATE on real bench-qa data: 97% of STRICT-fires at cb-θ have BOTH Q→A and A→ctx_lead low (co-varying); only 3% match the 'high A→ctx, low Q→A' deflection signature. The A→ctx axis isn't measuring what was hoped — a focused-claim against a 30KB topic-broad context scores low by default (that's the normal STRICT shape), so the delta doesn't discriminate. Contrastive rescue path is closed.",
"as_contrastive_signal": "TESTED 2026-05-13 — DOES NOT SEPARATE on real bench-qa data: 97% of STRICT-fires at cb-θ have BOTH Q→A and A→ctx_lead low (co-varying); only 3% match the 'high A→ctx, low Q→A' deflection signature. Contrastive rescue path closed — but it's no longer needed since real-context deflections (step 3) DID separate cleanly with absolute thresholds.",
"preprocessing_cleaning_is_free_improvement": "claim-lattice metadata [E\\d+ | … | hash: '…'] depresses scores by ~3-6 pts on cleaned vs raw; clean_for_relevance is baked into check_* by default. Universal win across all 6 swept models (-0.5 to -5.6 pts FP)."
},
"alternates": [

21
bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl Normal file
View file

@ -0,0 +1,21 @@
{"_meta": {"battery": "5f", "sub_battery": "relevance", "version": "deflection-realcontext-v1", "task_count": 20, "notes": "#000052 §3.2.2 step 3 — recall-side realism check for the relevance reranker. 20 hand-crafted deflection answers against REAL bench-qa questions: each answer is a coherent, well-formed statement that uses tokens from the real retrieved context (so the lexical verifier would pass it) but does NOT address its question. The relevance reranker should score Q→A LOW for each; measuring 'at θ where real-STRICT FP=0, what recall does the reranker have on these real-context deflections?' tells us the precision/recall Pareto for the soft-signal use case. All entries are coherent (would not flag §3.1 incoherence) and would survive the lexical verifier on token-overlap — the test is whether SEMANTIC aboutness catches what lexical can't."}}
{"id": "deflect-rc-001", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "how many moons does jupiter have?", "document": "Jupiter is the largest planet in the Solar System, named after the Roman king of the gods, with a massive magnetic field and a great red spot storm.", "note": "Jupiter facts, no moon count"}
{"id": "deflect-rc-002", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "what is the boltzmann constant?", "document": "Ludwig Boltzmann was a 19th-century Austrian physicist who made significant contributions to statistical mechanics and thermodynamics during his career in Vienna.", "note": "biographical, not the constant"}
{"id": "deflect-rc-003", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "what is the matrix?", "document": "The Matrix franchise has produced four feature films and has had a significant influence on cyberpunk aesthetics and philosophical discussion among fans.", "note": "franchise facts, doesn't say what The Matrix is"}
{"id": "deflect-rc-004", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "how does intel compare to amd?", "document": "Both Intel and AMD have headquarters in California and have been publicly traded for decades, with revenues measured in tens of billions of dollars annually.", "note": "factual co-statement, no comparison"}
{"id": "deflect-rc-005", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "what is the capital of france?", "document": "France is a country in Western Europe with a population of about 67 million people, a rich culinary tradition, and a long history of artistic achievement.", "note": "country facts, not the capital"}
{"id": "deflect-rc-006", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "in what year was the magna carta signed?", "document": "The Magna Carta was issued in Latin by King John of England and is now considered one of the most important legal documents in Western legal history.", "note": "doesn't give the year"}
{"id": "deflect-rc-007", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "why did the dinosaurs go extinct?", "document": "Dinosaurs lived during the Mesozoic Era and were the dominant terrestrial vertebrates for over 160 million years, with diverse species ranging in size from tiny to enormous.", "note": "dinosaur facts, no cause of extinction"}
{"id": "deflect-rc-008", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "who is michael jordan?", "document": "Michael Jordan has played for the Chicago Bulls and the Washington Wizards during his professional career, retiring multiple times before staying retired.", "note": "career, doesn't say who he is"}
{"id": "deflect-rc-009", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "who painted the mona lisa?", "document": "The Mona Lisa is one of the most famous paintings in the world, drawing millions of tourists to the Louvre Museum in Paris every year.", "note": "popularity + location, not painter"}
{"id": "deflect-rc-010", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "how many wives did henry the eighth have?", "document": "Henry VIII of England ruled from 1509 to 1547 and was responsible for the English Reformation, breaking with the Roman Catholic Church over his desired annulment.", "note": "reign facts, not wife count"}
{"id": "deflect-rc-011", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "who is supermans girlfriend?", "document": "Superman is a fictional superhero appearing in American comic books published by DC Comics since 1938, created by Jerry Siegel and Joe Shuster.", "note": "creation facts, not girlfriend"}
{"id": "deflect-rc-012", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "list all planets in the solar system", "document": "The Solar System formed approximately 4.6 billion years ago from the gravitational collapse of a giant interstellar molecular cloud, with the Sun at its center.", "note": "formation, not a list"}
{"id": "deflect-rc-013", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "where is mount kilimanjaro located?", "document": "Mount Kilimanjaro is a dormant stratovolcano with three volcanic cones — Kibo, Mawenzi, and Shira — reaching nearly 5,895 metres above sea level.", "note": "volcano facts, not location"}
{"id": "deflect-rc-014", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "in which ocean is madagascar?", "document": "Madagascar is the fourth-largest island in the world and is home to thousands of endemic plant and animal species found nowhere else on Earth.", "note": "size + biodiversity, not which ocean"}
{"id": "deflect-rc-015", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "how many states are in the united states?", "document": "The United States Constitution was adopted in 1787 and establishes the federal government's powers and structure across three branches.", "note": "constitution, not state count"}
{"id": "deflect-rc-016", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "when did the soviet union dissolve?", "document": "The Soviet Union was a federal socialist state in Northern Eurasia from 1922 to 1991, comprising fifteen constituent republics under a single communist government.", "note": "general history, doesn't pinpoint dissolution date"}
{"id": "deflect-rc-017", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "what caused the chernobyl disaster?", "document": "The Chernobyl disaster occurred at the Chernobyl Nuclear Power Plant near Pripyat, Ukraine, on 26 April 1986, releasing radioactive material across Europe.", "note": "when/where, not cause"}
{"id": "deflect-rc-018", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "when was the first moon landing?", "document": "The Apollo program was carried out by the United States National Aeronautics and Space Administration and resulted in the first humans walking on the Moon.", "note": "general program, not when"}
{"id": "deflect-rc-019", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "what year was the eiffel tower constructed?", "document": "The Eiffel Tower in Paris was designed by the engineer Gustave Eiffel and his company for the entrance to the World's Fair, and is now an iconic global landmark.", "note": "design + popularity, not the year"}
{"id": "deflect-rc-020", "kind": "NEG_DEFLECT", "pair_kind": "question_answer", "query": "who founded microsoft?", "document": "Microsoft is an American multinational technology corporation headquartered in Redmond, Washington, that develops, manufactures, licenses, supports, and sells computer software.", "note": "corp description, not founders"}