engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000048 step 2.4 — parse_pointer_claims clause segmentation

Browse source 
Closes the 8 mis-segments #000046 left in formulate-hard-v1.jsonl.
The parser was line/bullet-only — one line ⇒ one claim — so a line
that crammed several pointered claims onto one row ("Water is wet
[E1]; fire is hot [E2]", "X happened [E1]. Y followed [E2]") became
one monolithic claim with all the pointers, and a wrapped bullet
became two.

arborist/qa/parse_claims.py: _SEGMENT_SEP_RE splits a line on ';',
sentence boundaries ('. '/'! '/'? ' then a Capital), spaced dashes
(' - '/' — '/' – '), ' and '/' or '/' because '/' although '/' since
'/' while ', inline '(N)' enumeration markers, and commas — with
'(?![^\[]*\])' so a comma inside a [E1, E2] bracket never splits it.
_segment_line keeps the split ONLY IF every resulting non-empty
segment is a well-pointered claim — a legit single claim ("The cat
is black and white [E1].", "The cast: A, B, C [E1].") is never
broken because splitting it would manufacture pointer-less prose
fragments → guard rejects; a leading colon-terminated header with no
pointer ("Two facts:", "Key points:") is allowed and dropped. Plus a
wrapped-bullet join: a continuation line (leading whitespace then a
lowercase letter, no bullet glyph) folds its text + pointers into the
previous claim.

Effect: formulate-hard rate 4/12 → 12/12 (the pack is now at ceiling
— a harder Formulate tier would re-open below-ceiling headroom; a
#000046 follow-up). Remaining #000048 headroom: 2 STRICT_PARAPHRASE
recombinations in falsification-hard (Mercury, Einstein — step 2.2).

Bench gate: make bench-qa (n=3 × 75 × 3 = 675 cells; parse_pointer_claims
feeds the 450 claim_lattice_pointer + claim_lattice cells) after
(bench/qa_results/2026-05-11T20-26-37Z) vs the pre-step-2.4 baseline
(...T17-12-41Z = HEAD's parse_claims.py). STRICT-rate quote 0.54→0.55,
pointer 0.22→0.22, lattice 0.43→0.45 — all within the 5-pp noise
floor. Per-row diff: the segmenter changed the parsed-claim count on
the SAME answer text for 7 of the 450 lattice cells (0 in
claim_lattice, 7 in claim_lattice_pointer); of those, 2 caused an
audit_mode change — both correct: a wrap-join recovered an answer's
intended structure (4 claims, 2 pointer-less wrap-fragments → HYBRID)
into 2 well-pointered claims → STRICT; and a crammed-one-line blob (1
monolithic claim, all pointers → STRICT) split into 8 claims, some
not individually verifying → HYBRID (the honest verdict — false-
positive STRICT was the corruption). Every other lattice/quote delta
is LLM re-answer variance. No regression — the segmenter's only
visible effects on real traffic are honest improvements. Summarized
in qa-modes-bench.md Addendum 7 + ticket-000048 §5 step 2.4.

Tests: 8 new in test_claim_lattice.py (semicolon/sentence/conjunction
splits; pointerless-fragment + cast-list guards; leading-colon-header
drop; wrapped-bullet join; pointer-order/multi-pointer); existing
parse_pointer_claims tests pass untouched; test_5f_formulate_hard_pack
re-pinned 4/12 → 12/12. make test 2358 passed, 28 skipped.

#000048 → steps 2.1 + 2.4 landed; #000046 / #000012 §8 / TICKETS.md /
Makefile / fixture _meta + notes updated.
This commit is contained in:
russell@unturf.com 2026-05-11 17:09:06 -04:00
parent 4b85eb1fcf
commit 7bb11ed62f
No known key found for this signature in database
9 changed files with 349 additions and 107 deletions
Download patch file Download diff file Expand all files Collapse all files

18
bench/fixtures/5f/formulate-hard-v1.jsonl
View file

@ -1,12 +1,12 @@
{"_meta": {"battery": "5f", "sub_battery": "formulate", "version": "v1", "task_count": 12, "notes": "#000046 Phase 2 — HARD live-path Formulate tier. Every input is prose that arborist.qa.parse_claims.parse_pointer_claims SHOULD segment into a particular claim lattice; `expected_lattice` records that correct structure. The parser is line/bullet-based — one line ⇒ one claim, [E#] tokens on that line attach to it — so 8 of 12 it mis-segments: it merges multiple sentence/semicolon/clause claims that share a line into ONE claim with all the pointers, or splits a wrapped bullet into two. Those 8 fail at HEAD by design (count mismatch + pointer-set mismatch). The other 4 are well-formed bullet/numbered lists / single claims the parser handles correctly. Rate at HEAD = 4/12 = 0.333... — a real below-ceiling baseline; parse_pointer_claims is deterministic, so it's stable. A claim-lattice parser that does sentence/clause segmentation (split on '. ', ';', subordinating conjunctions, inline enumerations) and joins wrapped bullets would lift the rate toward 1.0 → a positive gamma*Delta5f term for that child fork. NOT part of `make bench-5f` / `runner --all` / `bench-fork-baseline`; pinned via `make bench-5f-formulate-hard` / shares `make bench-fork-baseline-hard` is falsification-only — formulate-hard is its own target. See ticket #000046 §5."}}
{"id": "5f-form-hard-001", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Water is wet [E1] and fire is hot [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Water is wet.", "pointer_ids": ["E1"]}, {"claim_text": "Fire is hot.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: two and-joined sentence claims on one line → parser merges to 1 claim with [E1,E2]; correct is 2."}
{"id": "5f-form-hard-002", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Water is wet [E1]; fire is hot [E2]; ice is cold [E3].", "expected_lattice": {"claim_count": 3, "claims": [{"claim_text": "Water is wet.", "pointer_ids": ["E1"]}, {"claim_text": "Fire is hot.", "pointer_ids": ["E2"]}, {"claim_text": "Ice is cold.", "pointer_ids": ["E3"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: three semicolon-separated claims on one line → parser merges to 1 claim with [E1,E2,E3]; correct is 3."}
{"id": "5f-form-hard-003", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- The cell respires aerobically\n to produce ATP efficiently. [E1]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "The cell respires aerobically to produce ATP efficiently.", "pointer_ids": ["E1"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: a wrapped bullet → parser splits into 2 claims, pointer on the wrong half; correct is 1 joined claim with [E1]."}
{"id": "5f-form-hard-004", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Water boils at 100 C [E1]. It freezes at 0 C [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Water boils at 100 C.", "pointer_ids": ["E1"]}, {"claim_text": "It freezes at 0 C.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: two sentences on one line (no bullets) → parser merges to 1 claim with [E1,E2]; correct is 2."}
{"id": "5f-form-hard-005", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "The engine runs hot [E1] because the coolant leaked [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "The engine runs hot.", "pointer_ids": ["E1"]}, {"claim_text": "The coolant leaked.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: main clause + 'because' sub-claim on one line → parser merges to 1 claim with [E1,E2]; correct is 2 (the cause is its own claim)."}
{"id": "5f-form-hard-006", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Two facts: (1) water is wet [E1] (2) fire is hot [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Water is wet.", "pointer_ids": ["E1"]}, {"claim_text": "Fire is hot.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: an inline (1)(2) enumeration on one line → parser merges to 1 claim with [E1,E2]; correct is 2."}
{"id": "5f-form-hard-007", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Key points: clarity matters [E1], brevity matters [E2], accuracy matters [E3].", "expected_lattice": {"claim_count": 3, "claims": [{"claim_text": "Clarity matters.", "pointer_ids": ["E1"]}, {"claim_text": "Brevity matters.", "pointer_ids": ["E2"]}, {"claim_text": "Accuracy matters.", "pointer_ids": ["E3"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: a comma-separated list after a colon on one line → parser merges to 1 claim with [E1,E2,E3]; correct is 3."}
{"id": "5f-form-hard-008", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "The sky is blue [E1] - the grass is green [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "The sky is blue.", "pointer_ids": ["E1"]}, {"claim_text": "The grass is green.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: two dash-joined clauses on one line → parser merges to 1 claim with [E1,E2]; correct is 2."}
{"_meta": {"battery": "5f", "sub_battery": "formulate", "version": "v1", "task_count": 12, "notes": "#000046 Phase 2 — HARD live-path Formulate tier. Every input is prose that arborist.qa.parse_claims.parse_pointer_claims SHOULD segment into a particular claim lattice; `expected_lattice` records that correct structure. Was 4/12 at HEAD — the original line/bullet-only parser merged multiple sentence/semicolon/clause claims sharing a line into ONE claim with all the pointers, and split a wrapped bullet into two. #000048 step 2.4 (2026-05-11) added a clause segmenter: within a line, split on ';', sentence boundaries ('. '/'! '/'? ' then a Capital), spaced dashes, ' and '/' or '/' because '/' although '/' since '/' while ', inline '(N)' enumeration markers, and commas — but the split is KEPT only if every resulting segment is itself a well-pointered claim (a leading colon-terminated header with no pointer — 'Two facts:', 'Key points:' — is allowed and dropped), so a legitimate single claim ('The cat is black and white [E1].', 'The cast: A, B, C [E1].') is never broken; plus a wrapped-bullet join (a leading-whitespace lowercase continuation line folds into the previous claim). Closes all 8 → rate now 12/12 = 1.0 (the pack is at ceiling; a harder Formulate tier would re-open below-ceiling headroom — a #000046 follow-up). parse_pointer_claims is deterministic, so the rate is stable. NOT part of `make bench-5f` / `runner --all` / `bench-fork-baseline`; pinned via `make bench-5f-formulate-hard`. See tickets #000046 §5 + #000048 §5 step 2.4."}}
{"id": "5f-form-hard-001", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Water is wet [E1] and fire is hot [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Water is wet.", "pointer_ids": ["E1"]}, {"claim_text": "Fire is hot.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "Caught (#000048 step 2.4): two and-joined sentence claims on one line → parser merges to 1 claim with [E1,E2]; correct is 2."}
{"id": "5f-form-hard-002", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Water is wet [E1]; fire is hot [E2]; ice is cold [E3].", "expected_lattice": {"claim_count": 3, "claims": [{"claim_text": "Water is wet.", "pointer_ids": ["E1"]}, {"claim_text": "Fire is hot.", "pointer_ids": ["E2"]}, {"claim_text": "Ice is cold.", "pointer_ids": ["E3"]}]}, "expected": "pass", "note": "Caught (#000048 step 2.4): three semicolon-separated claims on one line → parser merges to 1 claim with [E1,E2,E3]; correct is 3."}
{"id": "5f-form-hard-003", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- The cell respires aerobically\n to produce ATP efficiently. [E1]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "The cell respires aerobically to produce ATP efficiently.", "pointer_ids": ["E1"]}]}, "expected": "pass", "note": "Caught (#000048 step 2.4): a wrapped bullet → parser splits into 2 claims, pointer on the wrong half; correct is 1 joined claim with [E1]."}
{"id": "5f-form-hard-004", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Water boils at 100 C [E1]. It freezes at 0 C [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Water boils at 100 C.", "pointer_ids": ["E1"]}, {"claim_text": "It freezes at 0 C.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "Caught (#000048 step 2.4): two sentences on one line (no bullets) → parser merges to 1 claim with [E1,E2]; correct is 2."}
{"id": "5f-form-hard-005", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "The engine runs hot [E1] because the coolant leaked [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "The engine runs hot.", "pointer_ids": ["E1"]}, {"claim_text": "The coolant leaked.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "Caught (#000048 step 2.4): main clause + 'because' sub-claim on one line → parser merges to 1 claim with [E1,E2]; correct is 2 (the cause is its own claim)."}
{"id": "5f-form-hard-006", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Two facts: (1) water is wet [E1] (2) fire is hot [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Water is wet.", "pointer_ids": ["E1"]}, {"claim_text": "Fire is hot.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "Caught (#000048 step 2.4): an inline (1)(2) enumeration on one line → parser merges to 1 claim with [E1,E2]; correct is 2."}
{"id": "5f-form-hard-007", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Key points: clarity matters [E1], brevity matters [E2], accuracy matters [E3].", "expected_lattice": {"claim_count": 3, "claims": [{"claim_text": "Clarity matters.", "pointer_ids": ["E1"]}, {"claim_text": "Brevity matters.", "pointer_ids": ["E2"]}, {"claim_text": "Accuracy matters.", "pointer_ids": ["E3"]}]}, "expected": "pass", "note": "Caught (#000048 step 2.4): a comma-separated list after a colon on one line → parser merges to 1 claim with [E1,E2,E3]; correct is 3."}
{"id": "5f-form-hard-008", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "The sky is blue [E1] - the grass is green [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "The sky is blue.", "pointer_ids": ["E1"]}, {"claim_text": "The grass is green.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "Caught (#000048 step 2.4): two dash-joined clauses on one line → parser merges to 1 claim with [E1,E2]; correct is 2."}
{"id": "5f-form-hard-009", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- Sky is blue. [E1]\n- Grass is green. [E2]", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Sky is blue.", "pointer_ids": ["E1"]}, {"claim_text": "Grass is green.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "Correctly segmented at HEAD — well-formed 2-bullet list. PASSES today; kept as a headroom marker (a parser regression that started merging bullets would drop the rate)."}
{"id": "5f-form-hard-010", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "1. Sky is blue [E1]\n2. Grass is green [E2]\n3. Sun is bright [E3]", "expected_lattice": {"claim_count": 3, "claims": [{"claim_text": "Sky is blue.", "pointer_ids": ["E1"]}, {"claim_text": "Grass is green.", "pointer_ids": ["E2"]}, {"claim_text": "Sun is bright.", "pointer_ids": ["E3"]}]}, "expected": "pass", "note": "Correctly segmented at HEAD — well-formed 3-item numbered list. PASSES today."}
{"id": "5f-form-hard-011", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "The sky is blue. [E1]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "The sky is blue.", "pointer_ids": ["E1"]}]}, "expected": "pass", "note": "Correctly segmented at HEAD — single claim, single pointer. PASSES today."}