engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000052 §3.2.2 step 3 part B: real-context MIS-CITE fixtures (n=20) — bge-large catches 100% of mis-cite at strictly 0% real-STRICT FP (the Zionist failure mode is fully covered with margin)

Browse source 
Built bench/fixtures/5f/relevance-miscite-realcontext-v1.jsonl —
20 hand-crafted (claim, source) mis-cite pairs: claim about X, source
about Y, X≠Y but shared tokens. Each pair survives the lexical
title-relevance + verifier sidecars by construction. This is the
Zionist-entity failure mode (claim about a different entity than the
cited source, both lexically related).

Examples:
  - 'Mercury is the smallest planet' / Roman-god Mercury source
  - 'Java is a programming language' / Java-the-island source
  - 'Apple Inc. was co-founded by Steve Jobs' / apple-the-fruit source
  - 'The Eiffel Tower is in Paris' / Gustave-Eiffel-person source
  - 'Mozart composed The Magic Flute' / Mozart-effect-theory source

Headline (bge-reranker-large + cleaned + θ ≤ -2.42):
  Mis-cite catch:    20/20 = 100%   ← FULL COVERAGE of motivating shape
  Deflection catch:  11/20 =  55%
  Combined NEG:      31/40 =  78%
  Real-STRICT FP:    0/808 = 0%     ← strictly safe

bge-large mis-cite scores: -9.37 to -3.81 (max). STRICT min: -2.42.
Mis-cite is STRICTLY SEPARABLE from real STRICT — there's a 1.4-pt
gap with no overlap. (Deflection harder; some overlap with weak STRICT.)

MiniLM-L-6 cost-pick (5× smaller, cleaned, θ=+3):
  100% mis-cite + 65% deflect + 0.4% STRICT FP

Mis-cite is structurally MUCH easier than deflection — both models
hit 95-100% mis-cite catch at modest θ; deflection is harder because
'answer doesn't quite address question' can look like a weak STRICT.
That's appropriate: mis-cite is 'wrong topic entirely'; deflection
is 'right topic, not answering'.

Manifest:
  - runtime_viability flipped (step 2 → step 3): NOT VIABLE → VIABLE
    at the bge-large fp=0 operating point.
  - demote_below_score still null pending fox+dav1d sign-off (setting
    it folds relevance_policy_hash into governance_policy_hash per
    #000049 §7 #2).
  - PRIMARY RECOMMENDATION: bge-reranker-large + cleaned +
    θ ≤ -2.42 → 100% mis-cite, 55% deflect, 0% STRICT FP.

Production verifier unchanged; still SHADOW. The §3.2 arc went:
candidate-bench (overclaim) → real STRICT step 2 (over-pessimistic
NOT VIABLE) → real-context deflection step 3 (positive reversal,
55% at fp=0) → real-context mis-cite step 3 part B (full closure
at fp=0 for the motivating failure mode). Eight meta-lesson
instances over the §000049 + §000052 arc, with the sharpest one
yet: clean candidate-bench can mis-predict in BOTH directions —
real-data fixtures on BOTH precision and recall axes are the
only load-bearing measurement.
This commit is contained in:
russell@unturf.com 2026-05-13 15:27:58 -04:00
parent 7b1a57ade9
commit 902e676d4d
No known key found for this signature in database
2 changed files with 35 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

21
bench/fixtures/5f/relevance-miscite-realcontext-v1.jsonl Normal file
View file

@ -0,0 +1,21 @@
{"_meta": {"battery": "5f", "sub_battery": "relevance", "version": "miscite-realcontext-v1", "task_count": 20, "notes": "#000052 §3.2.2 step 3 part B — mis-cite recall fixtures. Each (claim, source) pair: the claim is well-formed and on-topic for ONE thing, the source is well-formed about a DIFFERENT thing, and they share content tokens (so the lexical title-relevance + verifier sidecars wave them through). This is the Zionist-entity failure mode — where the claim's subject and the cited source's subject are different entities that share a polysemous token. The reranker should score claim→source LOW for each. Hand-built; pair_kind='claim_source'."}}
{"id": "miscite-rc-001", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Mercury is the smallest planet in the Solar System.", "document": "Mercury was the Roman god of commerce, messengers, and travelers, the son of Jupiter and Maia, equivalent to the Greek god Hermes.", "note": "planet vs god, shared 'Mercury' + 'Jupiter'"}
{"id": "miscite-rc-002", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Java is an object-oriented programming language developed by Sun Microsystems in 1995.", "document": "Java is an island in Indonesia with a population of over 140 million, the most populous island in the world and the political center of the Republic.", "note": "language vs island, shared 'Java'"}
{"id": "miscite-rc-003", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Python is a high-level interpreted programming language.", "document": "Pythons are a family of large non-venomous snakes found in tropical and subtropical regions of the Eastern Hemisphere, constricting their prey.", "note": "language vs snake, shared 'Python'"}
{"id": "miscite-rc-004", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Apple Inc. was co-founded by Steve Jobs in 1976.", "document": "Apples are pomaceous fruits grown worldwide on the apple tree, with thousands of cultivars including Gala, Fuji, and Granny Smith.", "note": "company vs fruit, shared 'Apple'"}
{"id": "miscite-rc-005", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Einstein developed the theory of general relativity in 1915.", "document": "Albert Einstein College of Medicine is a private medical school located in the Morris Park neighborhood of the Bronx in New York City.", "note": "physicist vs medical school, shared 'Einstein'"}
{"id": "miscite-rc-006", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Boltzmann constant relates the average particle energy of a gas to its absolute temperature.", "document": "Ludwig Boltzmann was born in Vienna in 1844 to a tax official; he studied at the University of Vienna under Josef Stefan.", "note": "constant vs biography, shared 'Boltzmann'"}
{"id": "miscite-rc-007", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Shakespeare wrote the tragedy Hamlet around the year 1600.", "document": "Shakespeare's Globe was a London playhouse built in 1599 and destroyed by fire in 1613 during a performance of Henry VIII.", "note": "play authorship vs theatre, shared 'Shakespeare'"}
{"id": "miscite-rc-008", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Eiffel Tower in Paris was completed in 1889.", "document": "Gustave Eiffel was a French civil engineer who also designed the metal framework of the Statue of Liberty in New York Harbor.", "note": "tower vs person, shared 'Eiffel'"}
{"id": "miscite-rc-009", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "DNA is the molecule that carries genetic information in all known living organisms.", "document": "Daniel Nathans Award, formally the DNA award, is given by the Genetics Society of America for outstanding contributions to genetics research.", "note": "molecule vs award, shared 'DNA'"}
{"id": "miscite-rc-010", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Tokyo is the capital and largest city of Japan.", "document": "Tokyo Disneyland is a 47-hectare theme park located in Urayasu, Chiba, near Tokyo, opened in 1983 as the first Disney park outside the United States.", "note": "capital city vs theme park, shared 'Tokyo'"}
{"id": "miscite-rc-011", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Marie Curie won the Nobel Prize in Chemistry in 1911 for the discovery of radium and polonium.", "document": "Pierre Curie was a French physicist who shared the 1903 Nobel Prize in Physics with his wife Marie and with Henri Becquerel.", "note": "1911 Chemistry vs 1903 Physics, shared 'Curie'"}
{"id": "miscite-rc-012", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The cheetah is the fastest land animal in the world, reaching speeds of around 110 kilometres per hour.", "document": "Cheetos is a popular cheese-flavored cornmeal snack food brand produced by Frito-Lay, a subsidiary of PepsiCo, since 1948.", "note": "animal vs snack, shared 'cheet-'"}
{"id": "miscite-rc-013", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Bill Gates is the co-founder of Microsoft Corporation.", "document": "The Bill & Melinda Gates Foundation is an American private foundation founded in 2000 by Bill Gates and Melinda French Gates, focused on global health.", "note": "Microsoft founder vs foundation, shared 'Bill Gates'"}
{"id": "miscite-rc-014", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Amazon River in South America discharges more water than any other river in the world.", "document": "Amazon.com, Inc. is an American multinational technology company based in Seattle, Washington, founded by Jeff Bezos in 1994.", "note": "river vs company, shared 'Amazon'"}
{"id": "miscite-rc-015", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Great Wall of China is a series of fortifications built over many centuries.", "document": "Pink Floyd's The Wall is a 1979 rock opera double album, the band's eleventh studio album, conceived and largely written by Roger Waters.", "note": "wall (fortification) vs album, shared 'Wall'"}
{"id": "miscite-rc-016", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Mount Everest is the highest mountain in the world above sea level.", "document": "Mount Everest College was a private college that operated branches across the United States and Canada from 1979 until its closure in 2014.", "note": "mountain vs college, shared 'Mount Everest'"}
{"id": "miscite-rc-017", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Mozart composed The Magic Flute, his last opera, in 1791.", "document": "The Mozart effect is a popular but largely disputed theory that listening to Mozart's music can boost spatial reasoning and intelligence.", "note": "composer-opera vs disputed-theory, shared 'Mozart'"}
{"id": "miscite-rc-018", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "Albert Einstein developed the theory of special relativity in 1905.", "document": "Charles Darwin developed the theory of evolution by natural selection, presented in his 1859 work On the Origin of Species.", "note": "Einstein-relativity vs Darwin-evolution, shared 'theory of'"}
{"id": "miscite-rc-019", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Pacific Ocean is the largest and deepest ocean on Earth.", "document": "Pacific Rim is a 2013 American science fiction monster film directed by Guillermo del Toro about humanity defending Earth from giant sea creatures.", "note": "ocean vs film, shared 'Pacific'"}
{"id": "miscite-rc-020", "kind": "NEG_MISCITE", "pair_kind": "claim_source", "query": "The Roman Empire was a vast civilization that ruled over much of Europe, North Africa, and the Middle East.", "document": "Roman Polanski is a Polish-French film director, producer, writer, and actor, whose 1968 film Rosemary's Baby was an early career success.", "note": "empire vs filmmaker, shared 'Roman'"}