engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000052 §3.2.2 step 3 part B: real-context MIS-CITE fixtures (n=20) — bge-large catches 100% of mis-cite at strictly 0% real-STRICT FP (the Zionist failure mode is fully covered with margin)

Browse source 
Built bench/fixtures/5f/relevance-miscite-realcontext-v1.jsonl —
20 hand-crafted (claim, source) mis-cite pairs: claim about X, source
about Y, X≠Y but shared tokens. Each pair survives the lexical
title-relevance + verifier sidecars by construction. This is the
Zionist-entity failure mode (claim about a different entity than the
cited source, both lexically related).

Examples:
  - 'Mercury is the smallest planet' / Roman-god Mercury source
  - 'Java is a programming language' / Java-the-island source
  - 'Apple Inc. was co-founded by Steve Jobs' / apple-the-fruit source
  - 'The Eiffel Tower is in Paris' / Gustave-Eiffel-person source
  - 'Mozart composed The Magic Flute' / Mozart-effect-theory source

Headline (bge-reranker-large + cleaned + θ ≤ -2.42):
  Mis-cite catch:    20/20 = 100%   ← FULL COVERAGE of motivating shape
  Deflection catch:  11/20 =  55%
  Combined NEG:      31/40 =  78%
  Real-STRICT FP:    0/808 = 0%     ← strictly safe

bge-large mis-cite scores: -9.37 to -3.81 (max). STRICT min: -2.42.
Mis-cite is STRICTLY SEPARABLE from real STRICT — there's a 1.4-pt
gap with no overlap. (Deflection harder; some overlap with weak STRICT.)

MiniLM-L-6 cost-pick (5× smaller, cleaned, θ=+3):
  100% mis-cite + 65% deflect + 0.4% STRICT FP

Mis-cite is structurally MUCH easier than deflection — both models
hit 95-100% mis-cite catch at modest θ; deflection is harder because
'answer doesn't quite address question' can look like a weak STRICT.
That's appropriate: mis-cite is 'wrong topic entirely'; deflection
is 'right topic, not answering'.

Manifest:
  - runtime_viability flipped (step 2 → step 3): NOT VIABLE → VIABLE
    at the bge-large fp=0 operating point.
  - demote_below_score still null pending fox+dav1d sign-off (setting
    it folds relevance_policy_hash into governance_policy_hash per
    #000049 §7 #2).
  - PRIMARY RECOMMENDATION: bge-reranker-large + cleaned +
    θ ≤ -2.42 → 100% mis-cite, 55% deflect, 0% STRICT FP.

Production verifier unchanged; still SHADOW. The §3.2 arc went:
candidate-bench (overclaim) → real STRICT step 2 (over-pessimistic
NOT VIABLE) → real-context deflection step 3 (positive reversal,
55% at fp=0) → real-context mis-cite step 3 part B (full closure
at fp=0 for the motivating failure mode). Eight meta-lesson
instances over the §000049 + §000052 arc, with the sharpest one
yet: clean candidate-bench can mis-predict in BOTH directions —
real-data fixtures on BOTH precision and recall axes are the
only load-bearing measurement.
This commit is contained in:
russell@unturf.com 2026-05-13 15:27:58 -04:00
parent 7b1a57ade9
commit 902e676d4d
No known key found for this signature in database
2 changed files with 35 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

23
arborist/qa/relevance/manifest.json
View file

@ -10,19 +10,24 @@
"truncation_policy": "query_then_document@v1",
"pair_serialization": "relevance_pair@v1",
"demote_below_score": null,
"demote_below_score_comment": "STAYS NULL pending fox+dav1d sign-off, but step 3 (2026-05-13 real-context deflection set) showed VIABLE operating points exist on real bench-qa: bge-reranker-large at θ ≤ -2.42 catches 55% of real deflections at 0% real-STRICT FP; MiniLM-L-6 at θ=+3 catches 65% at 0.4% FP. These are the candidate runtime-veto operating points. The previous 'NOT VIABLE' verdict was a step-2-only artifact (candidate-bench NEG distribution overlapped real STRICT; real-context deflections do not). Setting demote_below_score requires the runtime promotion decision per #000049 §7 #2 — at that point nli_policy_hash-equivalent (model+max_length+cleaning+θ) folds into governance_policy_hash.",
"demote_below_score_comment": "STAYS NULL pending fox+dav1d sign-off, but step 3 part B (2026-05-13 — real-context deflection [n=20] + mis-cite [n=20] fixtures) confirmed VIABLE: bge-reranker-large + cleaned + θ ≤ -2.42 catches 100% of real-context MIS-CITE (the Zionist-shape motivating failure) AND 55% of real-context DEFLECTION, with strictly 0/808 real-STRICT FP. Mis-cite max bge-score is -3.81 vs STRICT min -2.42 — strictly separable for mis-cite. Deflection harder (some deflections score similarly to weak STRICT). MiniLM-L-6 + cleaned + θ=+3: 100% mis-cite + 65% deflect at 0.4% STRICT FP — cost-pick. Setting demote_below_score requires the runtime promotion decision per #000049 §7 #2 — at that point relevance_policy_hash (model+max_length+cleaning+θ+aggregation) folds into governance_policy_hash.",
"runtime_viability": {
"as_runtime_demotion_veto": "VIABLE at low-to-moderate FP — REVISED 2026-05-13 §3.2.2 step 3. The step-2 'NOT VIABLE' verdict was an artifact of using candidate-bench NEG as the recall denominator (tight contrived NEG band overlapped real STRICT). With REAL-CONTEXT deflection fixtures as the recall side (bench/fixtures/5f/relevance-deflection-realcontext-v1.jsonl, 20 hand-crafted Q→A deflections against real bench-qa questions), bge-reranker-large catches 11/20 = 55% at strictly 0% real-STRICT FP (θ ≤ -2.42), and pushes to ~95% at ~2% FP. MiniLM-L-6 (cleaned): 65% catch at 0.4% FP (θ=+3), 90% at 1.9% FP (θ=+5). These ARE usable runtime soft-veto operating points.",
"step3_pareto_minilm_l_6_cleaned": [
{"theta": -5.29, "deflect_catch": "2/20 = 0.10", "strict_fp": "0/808 = 0.000", "note": "fp=0 floor; low signal but safe"},
{"theta": 3.0, "deflect_catch": "13/20 = 0.65", "strict_fp": "3/808 = 0.004", "note": "conservative runtime veto — strong catch, sub-1% FP"},
{"theta": 5.0, "deflect_catch": "18/20 = 0.90", "strict_fp": "15/808 = 0.019", "note": "aggressive — 90% catch at ~2% FP"},
{"theta": 7.0, "deflect_catch": "20/20 = 1.00", "strict_fp": "84/808 = 0.104", "note": "full coverage but too FP for runtime"}
"step3_pareto_minilm_l_6_cleaned_2026_05_13_partB": [
{"theta": -5.29, "miscite_catch": "14/20 = 0.70", "deflect_catch": "2/20 = 0.10", "combined": "16/40 = 0.40", "strict_fp": "0/808 = 0.000", "note": "fp=0 floor"},
{"theta": 0.0, "miscite_catch": "19/20 = 0.95", "deflect_catch": "7/20 = 0.35", "combined": "26/40 = 0.65", "strict_fp": "2/808 = 0.003"},
{"theta": 2.0, "miscite_catch": "19/20 = 0.95", "deflect_catch": "11/20 = 0.55", "combined": "30/40 = 0.75", "strict_fp": "2/808 = 0.003"},
{"theta": 3.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "13/20 = 0.65", "combined": "33/40 = 0.82", "strict_fp": "3/808 = 0.004", "note": "cost-pick — 100% mis-cite + 65% deflect at sub-1% FP"},
{"theta": 5.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "18/20 = 0.90", "combined": "38/40 = 0.95", "strict_fp": "15/808 = 0.019"},
{"theta": 7.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "20/20 = 1.00", "combined": "40/40 = 1.00", "strict_fp": "84/808 = 0.104", "note": "too FP for runtime"}
],
"step3_pareto_bge_large_cleaned": [
{"theta": -2.42, "deflect_catch": "11/20 = 0.55", "strict_fp": "0/808 = 0.000", "note": "fp=0 floor; 55% deflection catch is real and usable"},
{"theta": 3.56, "deflect_catch": "19/20 = 0.95", "strict_fp": "~81/808 ≈ 0.10", "note": "interpolated from STRICT p10; precise FP needs per-row recomputation"}
"step3_pareto_bge_large_cleaned_2026_05_13_partB": [
{"theta": -2.42, "miscite_catch": "20/20 = 1.00", "deflect_catch": "11/20 = 0.55", "combined": "31/40 = 0.78", "strict_fp": "0/808 = 0.000", "note": "PRIMARY RECOMMENDATION — 100% mis-cite + 55% deflect at strictly 0% FP. Zionist-shape motivating failure fully covered."},
{"theta": 0.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "17/20 = 0.85", "combined": "37/40 = 0.93", "strict_fp": "15/808 = 0.019"},
{"theta": 2.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "18/20 = 0.90", "combined": "38/40 = 0.95", "strict_fp": "36/808 = 0.045"},
{"theta": 3.0, "miscite_catch": "20/20 = 1.00", "deflect_catch": "19/20 = 0.95", "combined": "39/40 = 0.97", "strict_fp": "71/808 = 0.088"}
],
"step3_finding_miscite_vs_deflection": "Mis-cite is structurally much easier than deflection — bge-large mis-cite scores range -9.37 to -3.81 (max), strictly below STRICT min (-2.42). Deflection range -5.57 to +5.01 overlaps STRICT bottom. So the runtime veto is FULLY effective on mis-cite (the Zionist failure mode) and PARTIALLY effective on deflection. That's appropriate to the structural difference: mis-cite is 'wrong topic entirely', deflection is 'right topic, not answering'.",
"as_advisory_soft_signal": "viable — surface the raw score on the render tail or as bench telemetry; no threshold needed",
"as_multi_signal_factor": "viable — combine with title-relevance, deflection, coherence sidecars for a weighted advisory",
"as_contrastive_signal": "TESTED 2026-05-13 — DOES NOT SEPARATE on real bench-qa data: 97% of STRICT-fires at cb-θ have BOTH Q→A and A→ctx_lead low (co-varying); only 3% match the 'high A→ctx, low Q→A' deflection signature. Contrastive rescue path closed — but it's no longer needed since real-context deflections (step 3) DID separate cleanly with absolute thresholds.",

21
bench/fixtures/5f/relevance-miscite-realcontext-v1.jsonl Normal file
View file

@ -0,0 +1,21 @@
{"_meta": {"battery": "5f", "sub_battery": "relevance", "version": "miscite-realcontext-v1", "task_count": 20, "notes": "#000052 §3.2.2 step 3 part B — mis-cite recall fixtures. Each (claim, source) pair: the claim is well-formed and on-topic for ONE thing, the source is well-formed about a DIFFERENT thing, and they share content tokens (so the lexical title-relevance + verifier sidecars wave them through). This is the Zionist-entity failure mode — where the claim's subject and the cited source's subject are different entities that share a polysemous token. The reranker should score claim→source LOW for each. Hand-built; pair_kind='claim_source'."}}
{"id": "miscite-rc-001", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Mercury is the smallest planet in the Solar System.", "document": "Mercury was the Roman god of commerce, messengers, and travelers, the son of Jupiter and Maia, equivalent to the Greek god Hermes.", "note": "planet vs god, shared 'Mercury' + 'Jupiter'"}
{"id": "miscite-rc-002", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Java is an object-oriented programming language developed by Sun Microsystems in 1995.", "document": "Java is an island in Indonesia with a population of over 140 million, the most populous island in the world and the political center of the Republic.", "note": "language vs island, shared 'Java'"}
{"id": "miscite-rc-003", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Python is a high-level interpreted programming language.", "document": "Pythons are a family of large non-venomous snakes found in tropical and subtropical regions of the Eastern Hemisphere, constricting their prey.", "note": "language vs snake, shared 'Python'"}
{"id": "miscite-rc-004", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Apple Inc. was co-founded by Steve Jobs in 1976.", "document": "Apples are pomaceous fruits grown worldwide on the apple tree, with thousands of cultivars including Gala, Fuji, and Granny Smith.", "note": "company vs fruit, shared 'Apple'"}
{"id": "miscite-rc-005", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Einstein developed the theory of general relativity in 1915.", "document": "Albert Einstein College of Medicine is a private medical school located in the Morris Park neighborhood of the Bronx in New York City.", "note": "physicist vs medical school, shared 'Einstein'"}
{"id": "miscite-rc-006", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Boltzmann constant relates the average particle energy of a gas to its absolute temperature.", "document": "Ludwig Boltzmann was born in Vienna in 1844 to a tax official; he studied at the University of Vienna under Josef Stefan.", "note": "constant vs biography, shared 'Boltzmann'"}
{"id": "miscite-rc-007", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Shakespeare wrote the tragedy Hamlet around the year 1600.", "document": "Shakespeare's Globe was a London playhouse built in 1599 and destroyed by fire in 1613 during a performance of Henry VIII.", "note": "play authorship vs theatre, shared 'Shakespeare'"}
{"id": "miscite-rc-008", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Eiffel Tower in Paris was completed in 1889.", "document": "Gustave Eiffel was a French civil engineer who also designed the metal framework of the Statue of Liberty in New York Harbor.", "note": "tower vs person, shared 'Eiffel'"}
{"id": "miscite-rc-009", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "DNA is the molecule that carries genetic information in all known living organisms.", "document": "Daniel Nathans Award, formally the DNA award, is given by the Genetics Society of America for outstanding contributions to genetics research.", "note": "molecule vs award, shared 'DNA'"}
{"id": "miscite-rc-010", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Tokyo is the capital and largest city of Japan.", "document": "Tokyo Disneyland is a 47-hectare theme park located in Urayasu, Chiba, near Tokyo, opened in 1983 as the first Disney park outside the United States.", "note": "capital city vs theme park, shared 'Tokyo'"}
{"id": "miscite-rc-011", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Marie Curie won the Nobel Prize in Chemistry in 1911 for the discovery of radium and polonium.", "document": "Pierre Curie was a French physicist who shared the 1903 Nobel Prize in Physics with his wife Marie and with Henri Becquerel.", "note": "1911 Chemistry vs 1903 Physics, shared 'Curie'"}
{"id": "miscite-rc-012", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The cheetah is the fastest land animal in the world, reaching speeds of around 110 kilometres per hour.", "document": "Cheetos is a popular cheese-flavored cornmeal snack food brand produced by Frito-Lay, a subsidiary of PepsiCo, since 1948.", "note": "animal vs snack, shared 'cheet-'"}
{"id": "miscite-rc-013", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Bill Gates is the co-founder of Microsoft Corporation.", "document": "The Bill & Melinda Gates Foundation is an American private foundation founded in 2000 by Bill Gates and Melinda French Gates, focused on global health.", "note": "Microsoft founder vs foundation, shared 'Bill Gates'"}
{"id": "miscite-rc-014", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Amazon River in South America discharges more water than any other river in the world.", "document": "Amazon.com, Inc. is an American multinational technology company based in Seattle, Washington, founded by Jeff Bezos in 1994.", "note": "river vs company, shared 'Amazon'"}
{"id": "miscite-rc-015", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Great Wall of China is a series of fortifications built over many centuries.", "document": "Pink Floyd's The Wall is a 1979 rock opera double album, the band's eleventh studio album, conceived and largely written by Roger Waters.", "note": "wall (fortification) vs album, shared 'Wall'"}
{"id": "miscite-rc-016", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Mount Everest is the highest mountain in the world above sea level.", "document": "Mount Everest College was a private college that operated branches across the United States and Canada from 1979 until its closure in 2014.", "note": "mountain vs college, shared 'Mount Everest'"}
{"id": "miscite-rc-017", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Mozart composed The Magic Flute, his last opera, in 1791.", "document": "The Mozart effect is a popular but largely disputed theory that listening to Mozart's music can boost spatial reasoning and intelligence.", "note": "composer-opera vs disputed-theory, shared 'Mozart'"}
{"id": "miscite-rc-018", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Albert Einstein developed the theory of special relativity in 1905.", "document": "Charles Darwin developed the theory of evolution by natural selection, presented in his 1859 work On the Origin of Species.", "note": "Einstein-relativity vs Darwin-evolution, shared 'theory of'"}
{"id": "miscite-rc-019", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Pacific Ocean is the largest and deepest ocean on Earth.", "document": "Pacific Rim is a 2013 American science fiction monster film directed by Guillermo del Toro about humanity defending Earth from giant sea creatures.", "note": "ocean vs film, shared 'Pacific'"}
{"id": "miscite-rc-020", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Roman Empire was a vast civilization that ruled over much of Europe, North Africa, and the Middle East.", "document": "Roman Polanski is a Polish-French film director, producer, writer, and actor, whose 1968 film Rosemary's Baby was an early career success.", "note": "empire vs filmmaker, shared 'Roman'"}