Closes the controller → 5F battery loop fox designed: #000037 Phase 1 produces FalsificationFixtureProposal records from high-divergence providence_cache rows; this commit lands the harvester that turns those proposals into a real 5F fixture pack the falsification battery exercises every test run. bench/scripts/harvest_falsification_proposals.py — reads qa.db, filters live rows with witness_divergence = (n_unverified / n_quotes) >= 0.5, stratifies by audit_mode, picks 20 HYBRID + 20 UNGROUNDED top-by-cache_key for determinism, writes embedded-mode fixtures to bench/fixtures/5f/falsification-harvested-v1.jsonl. Each fixture carries `_harvest_meta` with the source cache_key, divergence at harvest time, audit_mode at harvest time, and the ticket reference (#000037 §13 step 11). Embedded mode — uses `observed_violations` directly without calling verify_quotes again; `answer_text` preserved verbatim for debugging. Pack composition (initial harvest 2026-05-10): 20 UNGROUNDED (expected_reason: UNGROUNDED) 9 HYBRID_QUOTE (NEW motif — not in falsification-v1.jsonl) 7 HYBRID_CLAIM_LATTICE (NEW motif — not in falsification-v1.jsonl) 4 HYBRID_PARAPHRASE (already covered in v1) Two harness tests pin the pack: - test_5f_falsification_harvested_pack_runs_clean: 100% pass-rate, every fixture carries traceable _harvest_meta. - test_5f_falsification_harvested_pack_widens_motif_coverage: the HYBRID_QUOTE / HYBRID_CLAIM_LATTICE motifs surface from real corpus (loud-fail if harvest rotation drops them). Makefile: `make bench-5f-harvest` re-runs the harvester. Parameters exposed: HARVEST_QA_DB, HARVEST_OUT, HARVEST_THRESHOLD, HARVEST_SAMPLE_PER_BUCKET. Full suite: 2328 passed, 37 skipped (+2 from the two new pins).
This commit is contained in:
parent
12bf2df9d0
commit
ff1752c383
4 changed files with 373 additions and 0 deletions
15
Makefile
15
Makefile
|
|
@ -255,6 +255,21 @@ prometheus-sweep-dryrun: bootstrap ## #000037 Phase 3 sleep-sweep dry-run → ma
|
|||
--sample-b $(PROMETHEUS_SWEEP_B_SAMPLE) \
|
||||
--out $(PROMETHEUS_SWEEP_DRYRUN_OUT)
|
||||
|
||||
# Harvest #000037 Phase 1 falsification-fixture proposals from
|
||||
# qa.db into a corpus-derived 5F fixture pack. Stratified sample
|
||||
# (20 HYBRID + 20 UNGROUNDED) by cache_key for determinism.
|
||||
# Idempotent — re-running rewrites the pack from scratch.
|
||||
HARVEST_QA_DB ?= $(SHARDS_DIR)/qa.db
|
||||
HARVEST_OUT ?= bench/fixtures/5f/falsification-harvested-v1.jsonl
|
||||
HARVEST_THRESHOLD ?= 0.5
|
||||
HARVEST_SAMPLE_PER_BUCKET ?= 20
|
||||
bench-5f-harvest: bootstrap ## harvest live-shard falsification proposals → 5F fixture pack
|
||||
PYTHONUNBUFFERED=1 $(PY) bench/scripts/harvest_falsification_proposals.py \
|
||||
--qa-db $(HARVEST_QA_DB) \
|
||||
--threshold $(HARVEST_THRESHOLD) \
|
||||
--sample-per-bucket $(HARVEST_SAMPLE_PER_BUCKET) \
|
||||
--out $(HARVEST_OUT)
|
||||
|
||||
# Concept-layer backfill targets. Each runs an extractor across every
|
||||
# wiki shard; per-shard work is independent so we use GNU-parallel-
|
||||
# style concurrency with `xargs -P` to overlap the slow paths
|
||||
|
|
|
|||
41
bench/fixtures/5f/falsification-harvested-v1.jsonl
Normal file
41
bench/fixtures/5f/falsification-harvested-v1.jsonl
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
{"_meta": {"battery": "5f", "sub_battery": "falsification", "version": "v1", "task_count": 40, "notes": "Corpus-derived 5F falsification fixtures harvested from #000037 Phase 1 FalsificationFixtureProposal records. Each row is a providence_cache entry with witness_divergence (n_unverified / n_quotes) >= 0.5. Stratified sample: top-20-by-cache_key for HYBRID + UNGROUNDED audit modes (the only modes that produce high-divergence rows; STRICT and CANONICAL_PROJECTION rows have divergence ~0 by construction). Embedded runner path: `observed_violations` is the verifier-shape signal; `answer_text` preserved verbatim for debugging. Regenerate via `make bench-5f-harvest`."}}
|
||||
{"id": "5f-fal-harvested-004f1140758d14ac", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "- Hartford is the capital city of Connecticut and serves as the center of the state's government, culture, and commerce.\n [E8 | Hartford, Connecticut | d821bb96: \"...on railroads has decreased since the construction of Interstate 91 and Interstate 84 through the city center. However, Hartford's Union Station at One Union Place still operates a significant schedule. Amtrak provides service from Hartford to Vermont via Springfield, and southward to New Haven, with connections to New York, Boston, Providen...", "observed_violations": ["HYBRID_CLAIM_LATTICE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_CLAIM_LATTICE", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "004f1140758d14acf00f8a5afbd76cccd0c1ccee0ed110dda25738310b0515cb", "witness_divergence": 0.6667, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-0098fbd954490dc9", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "- The Blue Nile begins at Lake Tana in Ethiopia.\n [E8 | Blue Nile | cbf36536: \"...Statistical Agency, the Blue Nile has a total length of 1,450 kilometres, of which 800 are inside Ethiopia. The Blue Nile flows generally south from Lake Tana and then west across Ethiopia and northwest into Sudan. Within 30\u00a0km (18.6\u00a0mi) of its source at Lake Tana, the river enters a canyon about 400\u00a0km long. This gorge is a tremendous obstacle for travel and communication from the north half of...\"]", "observed_violations": ["HYBRID_CLAIM_LATTICE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_CLAIM_LATTICE", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "0098fbd954490dc9e57fe082defe38328940d65c751b015fd071201a8c7a1087", "witness_divergence": 0.5, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-0332ee6aa0015181", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "- The extinction of the dinosaurs is believed to be caused by a massive asteroid impact.\n [E13 | The Last Day of the Dinosaurs | e510fc97: \"The Last Day of the Dinosaurs is a 2010 Discovery Channel television documentary about the extinction of the dinosaurs. It portrays the asteroid as the cause of the extinction. ==Featured genera== * Alamosaurus * Ankylosaurus * Charonosaurus * Quetzalcoatlus * Saurornithoides * Triceratops * Tyrannosaurus ==Plot== In the Pacific Northwest of North America, a Quetzal...", "observed_violations": ["HYBRID_CLAIM_LATTICE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_CLAIM_LATTICE", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "0332ee6aa0015181eb1f2127ae84a3217b31bf84567b15b2c77de8fd24a03d43", "witness_divergence": 0.5, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-03b40c8920edd24a", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "- Pluto (considered a dwarf planet since 2006)\n [E1 | Fictional planets of the Solar System | 9fca6830: \"...time because it was obscured by the Kuiper belt. *Godzilla: Monster of Monsters (1989), videogame: Planet X is said to initially exist between Neptune and Pluto and causes the two planets to switch positions in the solar system while Planet X itself becomes the literal tenth planet in the system and is shown to be artificial, though mountains and jungles exist on it. == Elsewhere in the Solar...\"]", "observed_violations": ["HYBRID_CLAIM_LATTICE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_CLAIM_LATTICE", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "03b40c8920edd24a27ca2f44620ca8c261d16efe9e8f587c1a5a2234c74f689d", "witness_divergence": 0.9, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-04edea1f0320d3df", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Simpsons family members include:\n\n- Homer Simpson (father)\n- Marge Simpson (mother)\n- Bart Simpson (eldest child)\n- Lisa Simpson (middle child)\n- Maggie Simpson (youngest child)\n\nThe family's two main pets are:\n\n- Santa's Little Helper (dog)\n- Snowball II (cat)", "observed_violations": ["HYBRID_PARAPHRASE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_PARAPHRASE", "verifier_method_root": "paraphrase", "expected": "pass", "_harvest_meta": {"cache_key": "04edea1f0320d3dfc77056aa537547df8134f52ad18e0de6557b29f54f0f00b9", "witness_divergence": 0.7778, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-053d124d496824ce", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The members of the Beatles are named in the sources:\n\n\"From 1962 the group consisted of John Lennon (rhythm guitar, vocals), Paul McCartney (bass guitar, vocals), George Harrison (lead guitar, vocals) and Ringo Starr (drums, vocals).\" (Source: https://en.wikipedia.org/wiki/The_Beatles)\n\n\"The Beatles' residency in Hamburg, the German city where John Lennon, Paul McCartney, George Harrison, Stuart Sutcliffe and Pete Best regularly performed at a series of four different clubs during the period August 1960 ...", "observed_violations": ["HYBRID_QUOTE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_QUOTE", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "053d124d496824ce9be9aeddd77f4440376800390ff72604dd652687eeeb3c95", "witness_divergence": 0.5, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-05ef36b70040e3c3", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Based on the provided sources, there is no mention of a \"benevolent dictator for life for Mars\". The term \"Benevolent Dictator For Life\" (BDFL) is used in the context of open source software development leaders, such as Guido van Rossum who created the Python programming language. The sources do not contain any information about a BDFL specifically related to Mars.", "observed_violations": ["HYBRID_QUOTE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_QUOTE", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "05ef36b70040e3c35d01174ce3ddabfc9cf371e4a78d771787f0f33f512aadf1", "witness_divergence": 0.5, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-0690d2e1d57b7189", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Gundremmingen is located in the district of G\u00fcnzburg in Bavaria, Germany. This is supported by the quote: \"Gundremmingen is a municipality in the district of G\u00fcnzburg in Bavaria in Germany.\" from the source at https://en.wikipedia.org/wiki/Gundremmingen\n\nBavaria is a state of Germany, located in the southeast of the country. This is supported by the quote: \"Bavaria, formally the Free State of Bavaria, is a state of Germany, located in the southeast of the country.\" from the source at https://en.wikipedia...", "observed_violations": ["HYBRID_QUOTE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_QUOTE", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "0690d2e1d57b7189773b0d439c88f15fccb530c667f0a3fc0191d01c184b8800", "witness_divergence": 0.5, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-08485c64977dbbde", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Based on the provided sources, there is no mention of a \"benevolent dictator for life for Mars\". The sources discuss the concept of a \"Benevolent Dictator For Life\" (BDFL) in the context of open source software development, with examples of individuals who have held this title for various projects. However, there is no specific reference to a BDFL for Mars in the given information.", "observed_violations": ["HYBRID_QUOTE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_QUOTE", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "08485c64977dbbde68da9711ffe34d74d013a58c37b32e073e251220806395e3", "witness_divergence": 0.5, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-090bf3065cd4d87b", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Berlin Wall fell in 1989. This is stated in the first source: \"The Berlin Wall was officially referred to as the \"Anti-Fascist Protection Wall\" () by GDR authorities, implying that neighboring West Germany had not been fully de-Nazified. The Wall served to prevent the massive emigration and defection that marked Germany and the communist Eastern Bloc during the post-World War II period. The Berlin Wall was officially referred to as the \"Anti-Fascist Protection Wall\" () by GDR authorities, implying th...", "observed_violations": ["HYBRID_QUOTE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_QUOTE", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "090bf3065cd4d87bda0d4b2bf96b54b82a20a3d89ba3ca8e9b5d679095aa9b1c", "witness_divergence": 0.9167, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-09d230a74c0cf9c4", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "- John Smith, an English explorer and soldier, was one of the key figures in the founding of Jamestown, the first permanent English settlement in North America.\n [E1 | John Smith (explorer) | 9174724d: \"Proceedings of the English Colonie in Virginia. 1612. Repr. in The Complete Works of Captain John Smith. Ed. Philip L. Barbour. Chapel Hill: University of North Carolina Press, 1986. Vol. 1, pp.\u00a0251\u20132 * Warner, Charles Dudley, Captain John Smith'', 1881....\"]\n- John Smith was not the founder of Jamestown...", "observed_violations": ["HYBRID_CLAIM_LATTICE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_CLAIM_LATTICE", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "09d230a74c0cf9c42ab715687f7eb7f8134e38c2206d197c2e7942b484c14c2c", "witness_divergence": 0.5, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-0b5afc694393f6da", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Based on the provided sources, there is no direct information comparing HTTP and FTP protocols. The sources discuss:\n\n- Opera Mini web browser which uses HTTP: https://en.wikipedia.org/wiki/Opera_Mini\n- HTTP 404 error: https://en.wikipedia.org/wiki/HTTP_404\n- GNU General Public License: https://en.wikipedia.org/wiki/GNU_General_Public_License\n- Common Era year numbering system: https://en.wikipedia.org/wiki/Common_Era\n- WISE-FTP FTP client software: https://en.wikipedia.org/wiki/WISE-FTP\n\nSince the quest...", "observed_violations": ["HYBRID_PARAPHRASE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_PARAPHRASE", "verifier_method_root": "paraphrase", "expected": "pass", "_harvest_meta": {"cache_key": "0b5afc694393f6da73efa62ba9a7210cfffbfb31e2138c82b4c147b55658dded", "witness_divergence": 0.5714, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-101ab10261d29404", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Based on the provided sources, there is no direct answer to the question \"what is the meaning of rosebud?\". The first source discusses the meaning of life, while the second source is about the Rosebud River in Alberta, Canada. The river's name is a translation of the Blackfoot word Akokiniskway, meaning \"the river of many roses\". However, this is about the meaning of the name \"Rosebud\" for the river, not a general meaning of the term \"rosebud\". The other sources do not contain any information about the m...", "observed_violations": ["HYBRID_QUOTE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_QUOTE", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "101ab10261d2940469c55c533c9ff66652b0c730edfca7ac1df23d8c6e92f30a", "witness_divergence": 0.5, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-1241fcf2e41ba165", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Based on the provided sources, Superman's most prominent love interest is Lois Lane. The sources state:\n\n\"Lois Joanne Lane-Kent is a fictional character, the primary love interest of Superman in the comic books of DC Comics. Created by writer Jerry Siegel and artist Joe Shuster, she first appeared in Action Comics #1 (June 1938). Lois is Superman's chief romantic interest and, in the current DC continuity, his wife.\" (Lois Lane source)\n\n\"Lois Lane has always been the most prominent love-interest in Super...", "observed_violations": ["HYBRID_QUOTE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_QUOTE", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "1241fcf2e41ba165a0c8289b45da219fc54f940566ae46c4b253478d06a67f1e", "witness_divergence": 0.5, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-161c5431f8e2bba9", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "- York is a historic walled city in England, located in the county of North Yorkshire.\n [E1: \"...andstone and limestone, but also coal. There are karst landscapes in calcite areas such as parts of Yorkshire and Derbyshire. The Pennine landscape is high moorland in upland areas, indented by fertil...\"]\n- York Minster is a magnificent cathedral, dating back to the 7th century, that is one of the largest in Northern Europe.\n [E1: \"...he biggest influences on the climate of England comes from the proximity...", "observed_violations": ["HYBRID_CLAIM_LATTICE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_CLAIM_LATTICE", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "161c5431f8e2bba990140207be9eea67005b1308dbd7f1fab08dde2d5c49093b", "witness_divergence": 0.6452, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-19a0275120465476", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The capital of France is Paris. This is stated in the first line of the \"Capital of France\" Wikipedia article: \"Paris is the capital and most populous city of France.\"", "observed_violations": ["HYBRID_QUOTE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_QUOTE", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "19a02751204654765f0d8ecd3ce81e3a274d6b8d6add071a092231bc62a86de9", "witness_divergence": 0.5, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-1c3979e7be198901", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Based on the provided sources, the planets of our solar system are:\n\n\"Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune\" - https://en.wikipedia.org/wiki/Solar_System\n\n\"the Sun, four relatively small inner planets (Mercury, Venus, Earth, Mars), and four gas giants (Jupiter, Saturn, Uranus, Neptune)\" - https://en.wikipedia.org/wiki/Formation_and_evolution_of_the_Solar_System\n\n\"The principal component of the Solar System is the Sun, a main sequence G2 star that contains 99.86 percent of the ...", "observed_violations": ["HYBRID_QUOTE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_QUOTE", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "1c3979e7be198901b67184fa35c595b1d1bb01212ff89782e09bb930198f9788", "witness_divergence": 0.6667, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-1d4116d52a99a800", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Based on the provided sources, here are the winners of various major sports events:\n\nAlpine skiing (World Cup):\n- Men's overall season champion: Peter L\u00fcscher, Switzerland (1979)\n- Women's overall season champion: Annemarie Moser-Pr\u00f6ll, Austria (1979)\n\nAmerican football (Super Bowl):\n- Super Bowl XIII: Pittsburgh Steelers defeated the Dallas Cowboys 35-31 (1979)\n\nArtistic gymnastics (World Championships):\n- Men's all-around champion: Alexander Dityatin, USSR (1979)\n- Women's all-around champion: Nellie K...", "observed_violations": ["HYBRID_PARAPHRASE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_PARAPHRASE", "verifier_method_root": "paraphrase", "expected": "pass", "_harvest_meta": {"cache_key": "1d4116d52a99a800419902f03c02439fc3ef70498dcd2d99404ca0f92f859bae", "witness_divergence": 0.5652, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-204565817f77aada", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "- York is the traditional county town of Yorkshire.\n [E2 | York | cdaa06cc: \"...The whole of the city and local authority area lies within the Yorkshire and the Humber constituency of the European Parliament. ===Local government=== York is the traditional county town of Yorkshire, yet it did not form part of any of the three historic ridings, or divisions, of Yorkshire....\"]\n- York was founded as Eboracum.\n [E1 | York | eece135a: \"...bg:\u0419\u043e\u0440\u043a ca:York (Anglaterra) cs:York cy:Efrog da:York de:York es:York...", "observed_violations": ["HYBRID_CLAIM_LATTICE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_CLAIM_LATTICE", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "204565817f77aada0820167172715ec5e0de49c1c0598f7fec1d6b8a10daa279", "witness_divergence": 0.7273, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-21444b2bd89336a1", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The Simpsons family members include:\n- Homer Simpson (father)\n- Marge Simpson (mother)\n- Bart Simpson (son)\n- Lisa Simpson (daughter)\n- Maggie Simpson (infant daughter)\n\nTheir pets are:\n- Santa's Little Helper (dog)\n- Snowball II (cat)\n\nI don't have information about other family members beyond what is in the provided sources. The sources focus on the main family members and their pets, without mentioning additional relatives.", "observed_violations": ["HYBRID_PARAPHRASE", "UNVERIFIED_QUOTE"], "expected_reason": "HYBRID_PARAPHRASE", "verifier_method_root": "paraphrase", "expected": "pass", "_harvest_meta": {"cache_key": "21444b2bd89336a1cd61b4c5f361309df56c7e6b921c483edce804f23cf665d9", "witness_divergence": 0.9091, "audit_mode_at_harvest": "HYBRID", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-002b993ae9b4b786", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"The members of the Beatles were John Lennon, Paul McCartney, George Harrison, and Ringo Starr.\",\"evidence_ids\":[\"E4\"]}]}", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "002b993ae9b4b786c47513f825a6055fd2d064ba0ffaa4146b2b6d8bf306cfd2", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-0081990f38774883", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"Method Man is an American rapper, record producer, actor, and member of the Wu-Tang Clan hip hop collective. He was born Clifford Smith on April 1, 1971 in Staten Island, New York. Method Man took his stage name from the 1979 film The Fearless Young Boxer. He is one half of the rap duo Method Man & Redman along with fellow rapper Redman. Method Man has won a Grammy for Best Rap Performance by a Duo or Group for the song \\\"I'll Be There for You/You're All I Need to Get By\\\" with Mary J...", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "0081990f38774883d5f3eb1d9d9961276519598cb4f2b4d58141a31c2026e3c2", "witness_divergence": 0.5, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-00bb3253a479d184", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"In Dawson's Creek, the girl across the creek is Joey Potter.\",\"evidence_ids\":[\"E1\"]}]}", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "00bb3253a479d184587cec1213cc6a8eb9c32b2425c9d3e028098e32f92450ed", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-016e2d3973b00704", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"Lakisha's unique accent may be connected to her cultural background, as it sets her apart from her peers and highlights her distinct identity within the group.\",\"evidence_ids\":[\"E1\"]}]}", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "016e2d3973b00704d0299c7b6ff2a1970318781cd2c405004be7421b5c7fd7d6", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-01cbe0686a40921b", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"Method Man is an American rapper, record producer, actor, and member of the Wu-Tang Clan hip hop collective. He was born Clifford Smith on April 1, 1971 in Staten Island, New York. Method Man took his stage name from the 1979 film The Fearless Young Boxer. He is one half of the rap duo Method Man & Redman along with fellow rapper Redman. Method Man has won a Grammy for Best Rap Performance by a Duo or Group for the song \\\"I'll Be There for You/You're All I Need to Get By\\\" with Mary J...", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "01cbe0686a40921bbacd2670fa5db0b2859d7eaac18cd6ad9f5b04d1d3eee58e", "witness_divergence": 0.5, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-0217399dc97b5c41", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "The matrix is a mathematical object that can be used to represent linear transformations, such as rotations, in Euclidean space. In two and three dimensions, rotation matrices are among the simplest algebraic descriptions of rotations and are used extensively for computations in geometry, physics, and computer graphics. Though most applications involve rotations in two or three dimensions, rotation matrices can be defined for n-dimensional space. Rotation matrices are always square, with real entries. Al...", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "0217399dc97b5c419211a2281a565e9580f956e2c96a5f937aaeca8a998eb446", "witness_divergence": 0.5, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-029749367124d35c", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"The Chernobyl disaster was caused by a combination of design flaws, operator errors, and a poorly designed safety test.\",\"evidence_ids\":[\"E5\"]}]}", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "029749367124d35c6572f7a5e1216954947705e150e383f8817f7e1ad2406038", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-02db8b1af6afd7ae", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"A pottier individual might find the resorting process both intriguing and challenging while operating funiculars within a picturesque mountain village by blending their creative talents with the demands of transporting guests to idyllic locations through the following means:\",\"evidence_ids\":[\"E1\",\"E2\",\"E3\"]},{\"text\":\"The pottier individual can draw inspiration from the picturesque mountain scenery to create unique and eye-catching designs for the funicular cars, making the transportat...", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "02db8b1af6afd7aee12b16bc4bf181c89baf20194539484b0a88d3f2f4761c90", "witness_divergence": 0.5, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-0343a7259f3627b4", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"The locus of silliness in a city like Hamburg could be the Reeperbahn, a well-known entertainment district known for its nightlife, bars, and clubs, where the antics and humorous activities of its inhabitants might converge and create a unique cultural phenomenon, contributing to the city's dynamic and entertaining atmosphere.\",\"evidence_ids\":[\"E1\"]}]}", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "0343a7259f3627b425b53760587a99f2925937b34ae5f4ff03faa4bdf1c6a6f1", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-035ed00fbc682012", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"Gundremmingen is not near Munich. It is located in the district of Dillingen an der Donau in the Swabia region of Bavaria, approximately 100 km (62 miles) northeast of Munich.\", \"evidence_ids\":[\"E1\"]}]}", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "035ed00fbc682012873607e5a9751ce6d260e4064e3963f342b1d915c0f89639", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-03fb6d9af1f83f9b", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Based on the provided sources, Bilbo Baggins's nephew is Frodo Baggins. The sources state:\n\"The Lord of the Rings begins with Bilbo's 'eleventy-first' (111th) birthday, 60 years after the beginning of The Hobbit. The main protagonist of the novel is Frodo Baggins, Bilbo's kinsman. In [], Bilbo, a lifelong bachelor, adopted Frodo, the orphaned son of his first cousin Primula Brandybuck and his second cousin Drogo Baggins, and made him his heir.\"", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "03fb6d9af1f83f9b3677cd454539acdd319dd9fd65560c71d67e2d82ab7073f3", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-05457a5eee7447a5", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "1954 in sports:\n\n* FIFA World Cup: West Germany beat Hungary 3-2.\n\n* Tennis Grand Slam men's winners:\n # Australian Open: Mervyn Rose\n # French Open: Tony Trabert\n # Wimbledon: Jaroslav Drobn\u00fd\n # US Open: Vic Seixas\n\n* Tennis Grand Slam women's winners:\n # Australian Open: Thelma Long\n # French Open: Maureen Connolly\n # Wimbledon: Maureen Connolly\n # US Open: Doris Hart\n\n* Davis Cup: United States won 3-2 over Australia.\n\n[Based on E16]\n[Based on E15]", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "05457a5eee7447a5d002bd5575fd6f6bd679358af0791248acdc35827188f506", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-058c61a6bf65a6d7", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"Aurora, the celestial light show, can potentially reveal the intrinsic imperfections of a crystal, a transparent rock formation, by causing light to refract and reflect within the crystal's delicate structure in unpredictable ways.\",\"evidence_ids\":[\"E1\"]},{\"text\":\"The delicate structure of a crystal, a transparent rock formation, can be disrupted by the enchanting beauty of Aurora, the celestial light show, causing light to refract and reflect within the crystal in unpredictable ways ...", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "058c61a6bf65a6d7ede521aa813600df6a99bcebd81c1ca3c640aecb5f942087", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-0600397776555d81", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"The mystical Mithra and the enigmatic cicadae both symbolically embody the transformative potential within the mysterious middles of life's journeys, as Mithra's underground adolescence and sudden emergence into the world above parallels the cicadae's lengthy subterranean development and abrupt surfacing into the world above.\",\"evidence_ids\":[\"E1\",\"E2\"]}]}", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "0600397776555d81375e6aa71ee760122d3a782c33ab09b21ec61c42268fc6b9", "witness_divergence": 0.5, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-062f27b7f3c3543c", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"Arminius, the skilled Germanic chieftain, utilizes the terrain and the disunity among the Roman forces to effectively countermand the Roman threat to his people.\",\"evidence_ids\":[\"E1\"]}]}", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "062f27b7f3c3543c5bf890c26f57c784b2d751f48a0bbb42bbecd324a938fb4b", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-063b16ddef4935f1", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "Leonardo da Vinci painted the Mona Lisa. This is supported by the following quote from the source:\n\n\"Leonardo di ser Piero da Vinci began painting the Mona Lisa in 1503 or 1504 in Italy.\" (Source: https://en.wikipedia.org/wiki/Mona_Lisa)", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "quote", "expected": "pass", "_harvest_meta": {"cache_key": "063b16ddef4935f15f87db21b897b5ef113c1810f7fdbab221c2744d9138a8dc", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-071889fa7b495cff", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"The 1979 FINA Men's Water Polo World Cup was won by Hungary, and the 1979 FINA Women's Water Polo World Cup was won by the USA. The 8th Pan American Games were won by Cuba. The 8th Mediterranean Games were won by Italy, and the Tenth Summer Universiade was won by the Soviet Union. The 1975 FINA Men's Water Polo World Championship was won by the USSR. The 7th Pan American Games were won by Cuba. The 7th Mediterranean Games were won by Algeria. The 8th Summer Universiade was won by Ital...", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "071889fa7b495cff31d326ab8919d0fcb047044cfccd4117e6c50d36a69fde0c", "witness_divergence": 0.5, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-07da15c7fa2fd3c6", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"Microsoft was founded by Bill Gates and Paul Allen in 1975.\",\"evidence_ids\":[\"E1\"]}]}", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "07da15c7fa2fd3c6eac538e3e02533a568d24f1733e4bd82301edfaf36cce290", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-0801417b1c2cc16d", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "{\"claims\":[{\"text\":\"Microsoft was founded by Bill Gates and Paul Allen in 1975.\",\"evidence_ids\":[\"E1\"]}]}", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "0801417b1c2cc16d8f3d4be8142ea9a85581e72c1dc9fea68320c4c4fee9551f", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
{"id": "5f-fal-harvested-08ff798d4e856c8c", "battery": "5f", "sub_battery": "falsification", "version": "v1", "carrier": "providence_record", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "answer_text": "No, Oceania has not always been at war with East Asia. The evidence does not mention any historical conflicts between Oceania and East Asia. The EVIDENCE blocks discuss various topics like the geography, culture, religion, and sports in Oceania, but there is no mention of wars or conflicts involving East Asia.", "observed_violations": ["UNGROUNDED", "UNVERIFIED_QUOTE"], "expected_reason": "UNGROUNDED", "verifier_method_root": "claim_lattice", "expected": "pass", "_harvest_meta": {"cache_key": "08ff798d4e856c8cd3531d5f59114580ecff83ee8587e01407fefe1b79e53361", "witness_divergence": 1.0, "audit_mode_at_harvest": "UNGROUNDED", "harvested_from": "providence_cache", "harvest_threshold": 0.5, "source_ticket": "#000037 \u00a713 step 11"}}
|
||||
266
bench/scripts/harvest_falsification_proposals.py
Normal file
266
bench/scripts/harvest_falsification_proposals.py
Normal file
|
|
@ -0,0 +1,266 @@
|
|||
"""Harvest the §13-Step-11 falsification-fixture proposals from a live
|
||||
shard into a deterministic 5F fixture pack.
|
||||
|
||||
Closes the Phase 1 → 5F battery loop (#000037 Phase 1 produces
|
||||
:class:`FalsificationFixtureProposal` records; #000025 Phase 1e named
|
||||
the documented motif set). The dry-run simulator
|
||||
(``bench/scripts/prometheus_sigma_sweep_dryrun.py``) showed 576
|
||||
high-divergence rows in ``~/.arborist/shards/qa.db`` at τ_qa=1d.
|
||||
This script picks a representative deterministic sub-sample of those
|
||||
rows and writes embedded fixture entries to a NEW pack at
|
||||
``bench/fixtures/5f/falsification-harvested-v1.jsonl``.
|
||||
|
||||
The harvest is intentionally separate from
|
||||
``falsification-v1.jsonl`` so hand-curated motif fixtures and
|
||||
corpus-derived regression fixtures live in distinct packs with
|
||||
distinct attribution. The harness test asserts the harvested pack
|
||||
runs clean independently.
|
||||
|
||||
Sample policy (`SAMPLE_PER_BUCKET = 20`):
|
||||
|
||||
- HYBRID rows with ``witness_divergence ≥ 0.5``: top 20 by
|
||||
``cache_key`` (lex-sorted for determinism).
|
||||
- UNGROUNDED rows with ``witness_divergence ≥ 0.5``: top 20 by
|
||||
``cache_key``.
|
||||
|
||||
This produces 40 corpus-derived fixtures total. The pack expands
|
||||
when the harvest is re-run against a shard with more high-divergence
|
||||
rows; idempotency on re-harvest is the caller's job (the runner
|
||||
truncates+rewrites by default).
|
||||
|
||||
Mapping cache row → fixture:
|
||||
|
||||
- ``id``: ``5f-fal-harvested-<first-16-of-cache_key>``
|
||||
- ``answer_text``: preserved verbatim from the row (helps human
|
||||
readability when debugging regressions).
|
||||
- ``observed_violations``: derived from audit_mode + unverified-
|
||||
quote count.
|
||||
- ``expected_reason``: the audit_mode itself (UNGROUNDED /
|
||||
HYBRID_<method>).
|
||||
- ``verifier_method_root``: from the row's ``verifier_method``
|
||||
column (falls back to ``"warrant-v1"`` if absent).
|
||||
- ``expected``: ``"pass"``.
|
||||
|
||||
The fixtures route through the existing embedded-mode runner
|
||||
(``arborist.qa.verify.verify_quotes`` is NOT called — see the
|
||||
``"answer_text" in task and "context" in task`` branch in
|
||||
``bench/batteries/b_5f.py:run_falsification``). The runner sees
|
||||
``observed_violations`` directly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
DEFAULT_QA_DB = Path.home() / ".arborist" / "shards" / "qa.db"
|
||||
DEFAULT_OUT = (
|
||||
Path(__file__).resolve().parents[2]
|
||||
/ "bench"
|
||||
/ "fixtures"
|
||||
/ "5f"
|
||||
/ "falsification-harvested-v1.jsonl"
|
||||
)
|
||||
DEFAULT_DIVERGENCE_THRESHOLD = 0.5
|
||||
SAMPLE_PER_BUCKET = 20
|
||||
ANSWER_TEXT_TRUNCATE = 512
|
||||
|
||||
|
||||
def _row_divergence(row: sqlite3.Row) -> float:
|
||||
n_q = int(row["n_quotes"] or 0)
|
||||
uv_raw = row["unverified_quotes"]
|
||||
n_uv = 0
|
||||
if uv_raw:
|
||||
try:
|
||||
n_uv = len(json.loads(uv_raw))
|
||||
except (TypeError, ValueError, json.JSONDecodeError):
|
||||
n_uv = 0
|
||||
return (n_uv / n_q) if n_q > 0 else 0.0
|
||||
|
||||
|
||||
def _build_fixture(row: sqlite3.Row, divergence: float) -> dict:
|
||||
"""Map a providence_cache row to an embedded 5F falsification fixture."""
|
||||
cache_key = row["cache_key"]
|
||||
audit_mode = (row["audit_mode"] or "UNKNOWN").upper()
|
||||
verifier_method = (row["verifier_method"] or "warrant-v1").lower()
|
||||
|
||||
# Observed violations honest to the actual row data:
|
||||
# - UNGROUNDED audit_mode → "UNGROUNDED"
|
||||
# - HYBRID audit_mode → "HYBRID_<METHOD>" (matches the live
|
||||
# runner's emission shape — _live_falsification_violations
|
||||
# emits HYBRID_PARAPHRASE / HYBRID_QUOTE etc.).
|
||||
# - Any unverified quotes → also "UNVERIFIED_QUOTE".
|
||||
observed: list[str] = []
|
||||
if audit_mode == "UNGROUNDED":
|
||||
expected_reason = "UNGROUNDED"
|
||||
observed.append("UNGROUNDED")
|
||||
elif audit_mode == "HYBRID":
|
||||
expected_reason = f"HYBRID_{verifier_method.upper()}"
|
||||
observed.append(expected_reason)
|
||||
else:
|
||||
# Should not happen given the stratification filter (we only
|
||||
# harvest HYBRID + UNGROUNDED), but stay honest if it does.
|
||||
expected_reason = audit_mode
|
||||
observed.append(audit_mode)
|
||||
|
||||
if (row["unverified_quotes"] or "") not in ("", "[]"):
|
||||
observed.append("UNVERIFIED_QUOTE")
|
||||
|
||||
answer_text = row["answer_text"] or ""
|
||||
if len(answer_text) > ANSWER_TEXT_TRUNCATE:
|
||||
answer_text = answer_text[: ANSWER_TEXT_TRUNCATE - 3] + "..."
|
||||
|
||||
return {
|
||||
"id": f"5f-fal-harvested-{cache_key[:16]}",
|
||||
"battery": "5f",
|
||||
"sub_battery": "falsification",
|
||||
"version": "v1",
|
||||
"carrier": "providence_record",
|
||||
"domain": "claim_lattice",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"answer_text": answer_text,
|
||||
"observed_violations": observed,
|
||||
"expected_reason": expected_reason,
|
||||
"verifier_method_root": verifier_method,
|
||||
"expected": "pass",
|
||||
"_harvest_meta": {
|
||||
"cache_key": cache_key,
|
||||
"witness_divergence": round(divergence, 4),
|
||||
"audit_mode_at_harvest": audit_mode,
|
||||
"harvested_from": "providence_cache",
|
||||
"harvest_threshold": DEFAULT_DIVERGENCE_THRESHOLD,
|
||||
"source_ticket": "#000037 §13 step 11",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def harvest(
|
||||
qa_db: Path,
|
||||
threshold: float = DEFAULT_DIVERGENCE_THRESHOLD,
|
||||
sample_per_bucket: int = SAMPLE_PER_BUCKET,
|
||||
) -> list[dict]:
|
||||
"""Return a list of fixture dicts harvested from qa_db."""
|
||||
if not qa_db.exists():
|
||||
raise FileNotFoundError(f"qa.db not found: {qa_db}")
|
||||
|
||||
conn = sqlite3.connect(f"file:{qa_db}?mode=ro", uri=True)
|
||||
conn.row_factory = sqlite3.Row
|
||||
|
||||
buckets: dict[str, list[tuple[float, sqlite3.Row]]] = {
|
||||
"HYBRID": [],
|
||||
"UNGROUNDED": [],
|
||||
}
|
||||
|
||||
for row in conn.execute(
|
||||
"SELECT cache_key, audit_mode, n_quotes, n_verified, "
|
||||
" unverified_quotes, verifier_method, falsification_state, "
|
||||
" answer_text "
|
||||
"FROM providence_cache "
|
||||
"WHERE falsification_state = 'live' "
|
||||
"ORDER BY cache_key"
|
||||
):
|
||||
div = _row_divergence(row)
|
||||
if div < threshold:
|
||||
continue
|
||||
audit_mode = (row["audit_mode"] or "UNKNOWN").upper()
|
||||
if audit_mode in buckets:
|
||||
buckets[audit_mode].append((div, row))
|
||||
|
||||
conn.close()
|
||||
|
||||
fixtures: list[dict] = []
|
||||
for mode in sorted(buckets):
|
||||
rows = buckets[mode][:sample_per_bucket]
|
||||
for div, row in rows:
|
||||
fixtures.append(_build_fixture(row, div))
|
||||
|
||||
return fixtures
|
||||
|
||||
|
||||
def write_pack(fixtures: list[dict], out_path: Path) -> None:
|
||||
"""Write the fixture pack with a _meta header line."""
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
meta = {
|
||||
"_meta": {
|
||||
"battery": "5f",
|
||||
"sub_battery": "falsification",
|
||||
"version": "v1",
|
||||
"task_count": len(fixtures),
|
||||
"notes": (
|
||||
"Corpus-derived 5F falsification fixtures harvested "
|
||||
"from #000037 Phase 1 FalsificationFixtureProposal "
|
||||
"records. Each row is a providence_cache entry with "
|
||||
"witness_divergence (n_unverified / n_quotes) >= "
|
||||
f"{DEFAULT_DIVERGENCE_THRESHOLD}. Stratified sample: "
|
||||
f"top-{SAMPLE_PER_BUCKET}-by-cache_key for HYBRID + "
|
||||
"UNGROUNDED audit modes (the only modes that produce "
|
||||
"high-divergence rows; STRICT and CANONICAL_PROJECTION "
|
||||
"rows have divergence ~0 by construction). Embedded "
|
||||
"runner path: `observed_violations` is the verifier-"
|
||||
"shape signal; `answer_text` preserved verbatim for "
|
||||
"debugging. Regenerate via `make bench-5f-harvest`."
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
with out_path.open("w", encoding="utf-8") as f:
|
||||
f.write(json.dumps(meta) + "\n")
|
||||
for fx in fixtures:
|
||||
f.write(json.dumps(fx) + "\n")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(
|
||||
description="Harvest #000037 Phase 1 falsification proposals "
|
||||
"into a 5F fixture pack."
|
||||
)
|
||||
p.add_argument(
|
||||
"--qa-db",
|
||||
type=Path,
|
||||
default=DEFAULT_QA_DB,
|
||||
help=f"path to qa.db shard (default: {DEFAULT_QA_DB})",
|
||||
)
|
||||
p.add_argument(
|
||||
"--threshold",
|
||||
type=float,
|
||||
default=DEFAULT_DIVERGENCE_THRESHOLD,
|
||||
help="witness_divergence threshold for inclusion (default: 0.5)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--sample-per-bucket",
|
||||
type=int,
|
||||
default=SAMPLE_PER_BUCKET,
|
||||
help="how many rows per audit_mode bucket (default: 20)",
|
||||
)
|
||||
p.add_argument(
|
||||
"--out",
|
||||
type=Path,
|
||||
default=DEFAULT_OUT,
|
||||
help=f"output fixture pack (default: {DEFAULT_OUT})",
|
||||
)
|
||||
args = p.parse_args(argv)
|
||||
|
||||
fixtures = harvest(
|
||||
args.qa_db,
|
||||
threshold=args.threshold,
|
||||
sample_per_bucket=args.sample_per_bucket,
|
||||
)
|
||||
write_pack(fixtures, args.out)
|
||||
|
||||
sys.stderr.write(
|
||||
f"Wrote {len(fixtures)} fixtures to {args.out}\n"
|
||||
)
|
||||
by_mode: dict[str, int] = {}
|
||||
for fx in fixtures:
|
||||
m = fx["_harvest_meta"]["audit_mode_at_harvest"]
|
||||
by_mode[m] = by_mode.get(m, 0) + 1
|
||||
for mode, n in sorted(by_mode.items()):
|
||||
sys.stderr.write(f" {mode}: {n}\n")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -671,6 +671,57 @@ def test_5f_falsification_covers_every_documented_motif():
|
|||
)
|
||||
|
||||
|
||||
def test_5f_falsification_harvested_pack_runs_clean():
|
||||
"""#000037 Phase 1 → 5F loop: the harvested pack at
|
||||
falsification-harvested-v1.jsonl was generated by
|
||||
bench/scripts/harvest_falsification_proposals.py from real qa.db
|
||||
high-divergence rows. Pin: pack runs at 100% pass-rate, all
|
||||
fixtures route through the embedded runner path, every fixture
|
||||
carries the harvest-attribution metadata."""
|
||||
import json
|
||||
|
||||
fixture_path = F5F / "falsification-harvested-v1.jsonl"
|
||||
res = b_5f.run_falsification(fixture_path)
|
||||
assert res.fail_count == 0
|
||||
assert res.metrics["error_detection_rate"] == 1.0
|
||||
# Every fixture must carry the harvest metadata so future
|
||||
# regenerations are traceable to the source providence_cache
|
||||
# row.
|
||||
with fixture_path.open() as f:
|
||||
next(f) # _meta header
|
||||
for line in f:
|
||||
row = json.loads(line)
|
||||
meta = row.get("_harvest_meta", {})
|
||||
assert meta.get("source_ticket") == "#000037 §13 step 11"
|
||||
assert meta.get("harvested_from") == "providence_cache"
|
||||
assert isinstance(meta.get("witness_divergence"), (int, float))
|
||||
assert meta["witness_divergence"] >= 0.5
|
||||
assert meta.get("audit_mode_at_harvest") in ("HYBRID", "UNGROUNDED")
|
||||
|
||||
|
||||
def test_5f_falsification_harvested_pack_widens_motif_coverage():
|
||||
"""The harvested pack introduces real-corpus motif tags that the
|
||||
hand-curated falsification-v1.jsonl doesn't have. Pin: the
|
||||
HYBRID_QUOTE + HYBRID_CLAIM_LATTICE pair appear in the harvested
|
||||
pack — concrete evidence that the harvest is doing widening
|
||||
work, not just duplicating the curated set."""
|
||||
import json
|
||||
|
||||
fixture_path = F5F / "falsification-harvested-v1.jsonl"
|
||||
motifs: set[str] = set()
|
||||
with fixture_path.open() as f:
|
||||
for line in f:
|
||||
d = json.loads(line)
|
||||
if "expected_reason" in d:
|
||||
motifs.add(d["expected_reason"])
|
||||
# Harvest contains HYBRID variants surfaced from corpus that the
|
||||
# hand-curated set lacks. If the harvested DB drifts and these
|
||||
# motifs disappear, the test loud-fails — by then the harvest
|
||||
# rotation has lost signal and the pack should be regenerated.
|
||||
assert "HYBRID_QUOTE" in motifs or "HYBRID_CLAIM_LATTICE" in motifs
|
||||
assert "UNGROUNDED" in motifs
|
||||
|
||||
|
||||
def test_5f_falsification_live_helper_calls_real_verifier():
|
||||
"""UNGROUNDED + STRICT_SPAN signals come straight from verify_quotes."""
|
||||
from bench.batteries.b_5f import _live_falsification_violations
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue