russell@unturf.com
0c1bad8494
Extends the #000046 hard tier to the Formulate sub-battery. bench/fixtures/5f/formulate-hard-v1.jsonl — 12 prose inputs that arborist.qa.parse_claims.parse_pointer_claims SHOULD segment into a particular claim lattice (recorded in expected_lattice). The parser is line/bullet-based — one line ⇒ one claim, [E#] tokens attach to it — so 8 of 12 it mis-segments: merges and-/semicolon-/dash-joined or (1)(2)-enumerated multi-claim lines into one claim with all the pointers, or splits a wrapped bullet into two. Those 8 fail at HEAD on claim-count mismatch; the other 4 are well-formed bullet/numbered lists / single claims the parser handles right. Rate at HEAD = 4/12 = 0.333, stable (parse_pointer_claims is deterministic). A claim-lattice parser that does sentence/clause segmentation (split on '. ', ';', subordinating conjunctions, inline enumerations) + joins wrapped bullets lifts the rate toward 1.0 → positive γ·Δ5f for that child fork. make bench-5f-formulate-hard runs it (|| true past the runner's nonzero-on-failures exit). Not in `make bench-5f` / `runner --all`. tests/test_bench_batteries.py — test_5f_formulate_hard_pack_below_ceiling (pins rate 4/12, source=live, the 8 fails are claim-count mis-segments). #000046 → "Phase 1 + Phase 2 landed"; two below-ceiling 5F subs now exist (falsification, formulate). Closure still pending an actual surface improvement (verify_quotes tightening — bench-gated — or parse_pointer_claims segmentation) that lifts a rate. §5 + §6 + TICKETS.md row updated. (Makefile also carries an uncommitted chain-check-SQL improvement from the concurrent #000039 session — NOT included in this commit; staged only the bench-5f-formulate-hard hunk + the .PHONY line.) |
||
|---|---|---|
| .. | ||
| batteries | ||
| fixtures | ||
| results | ||
| scripts | ||
| emergent_log.jsonl | ||
| prometheus_sigma_trigger_probe.py | ||
| qa_questions.txt | ||
| qa_questions_canonical_witness_npower.txt | ||
| qa_questions_metacog_subset.txt | ||
| qa_questions_progressive_and.txt | ||
| qa_questions_quantifier_baseline.txt | ||
| qa_questions_quantifier_subset.txt | ||
| qa_questions_smoke.txt | ||
| qa_questions_warrant_chain_aggressive.txt | ||
| qa_questions_warrant_chain_paraphrase.txt | ||
| qa_questions_warrant_chain_probe.txt | ||
| qa_sweep.py | ||
| run.sh | ||