#000046 Phase 2 — HARD Formulate tier (second below-ceiling pack)
Extends the #000046 hard tier to the Formulate sub-battery. bench/fixtures/5f/formulate-hard-v1.jsonl — 12 prose inputs that arborist.qa.parse_claims.parse_pointer_claims SHOULD segment into a particular claim lattice (recorded in expected_lattice). The parser is line/bullet-based — one line ⇒ one claim, [E#] tokens attach to it — so 8 of 12 it mis-segments: merges and-/semicolon-/dash-joined or (1)(2)-enumerated multi-claim lines into one claim with all the pointers, or splits a wrapped bullet into two. Those 8 fail at HEAD on claim-count mismatch; the other 4 are well-formed bullet/numbered lists / single claims the parser handles right. Rate at HEAD = 4/12 = 0.333, stable (parse_pointer_claims is deterministic). A claim-lattice parser that does sentence/clause segmentation (split on '. ', ';', subordinating conjunctions, inline enumerations) + joins wrapped bullets lifts the rate toward 1.0 → positive γ·Δ5f for that child fork. make bench-5f-formulate-hard runs it (|| true past the runner's nonzero-on-failures exit). Not in `make bench-5f` / `runner --all`. tests/test_bench_batteries.py — test_5f_formulate_hard_pack_below_ceiling (pins rate 4/12, source=live, the 8 fails are claim-count mis-segments). #000046 → "Phase 1 + Phase 2 landed"; two below-ceiling 5F subs now exist (falsification, formulate). Closure still pending an actual surface improvement (verify_quotes tightening — bench-gated — or parse_pointer_claims segmentation) that lifts a rate. §5 + §6 + TICKETS.md row updated. (Makefile also carries an uncommitted chain-check-SQL improvement from the concurrent #000039 session — NOT included in this commit; staged only the bench-5f-formulate-hard hunk + the .PHONY line.)
This commit is contained in:
parent
af870bbf23
commit
0c1bad8494
5 changed files with 110 additions and 25 deletions
10
Makefile
10
Makefile
|
|
@ -35,7 +35,7 @@ SEARCH_Q ?= computer
|
|||
recrawl-check bench-qa bench-qa-smoke bench-qa-progressive-and \
|
||||
prometheus-trigger-probe bench-5f-threshold-calibration \
|
||||
bench-5f-selfmodel-snapshot bench-5f-finetuning-shardchain \
|
||||
bench-5f-falsification-hard bench-fork-baseline-hard \
|
||||
bench-5f-falsification-hard bench-fork-baseline-hard bench-5f-formulate-hard \
|
||||
bootstrap-math clean clean-db clean-data help \
|
||||
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
|
||||
crawl-textbooks crawl-textbooks-stats textbook textbook-list
|
||||
|
|
@ -511,6 +511,14 @@ bench-fork-baseline-hard: bootstrap ## #000046 — pin the below-ceiling hard-Fa
|
|||
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5f --sub falsification \
|
||||
--fixtures bench/fixtures/5f/falsification-hard-v1.jsonl --out $(FORK_PARENT_HARD) || true
|
||||
@echo ">> below-ceiling parent pinned: $(FORK_PARENT_HARD) (8/12 over-grounds expected → runner exits 1; the JSON is written)"
|
||||
# #000046 Phase 2 — HARD Formulate tier. Prose that parse_pointer_claims
|
||||
# should segment a particular way; the line/bullet-based parser
|
||||
# mis-segments 8 of 12 (merges multi-claim lines, splits wrapped
|
||||
# bullets) → rate 4/12 at HEAD. `|| true` past the runner's
|
||||
# nonzero-on-failures exit (8 fail by design).
|
||||
bench-5f-formulate-hard: bootstrap ## #000046 Phase 2 — HARD Formulate mis-segment pack (rate < 1.0 at HEAD by design)
|
||||
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5f --sub formulate \
|
||||
--fixtures bench/fixtures/5f/formulate-hard-v1.jsonl || true
|
||||
|
||||
# #000025 §10.11 — persistent SelfModel-chain lineage. Each run of
|
||||
# bench-5f-selfmodel-snapshot runs the 5S/5T/5F embedded packs and
|
||||
|
|
|
|||
13
bench/fixtures/5f/formulate-hard-v1.jsonl
Normal file
13
bench/fixtures/5f/formulate-hard-v1.jsonl
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
{"_meta": {"battery": "5f", "sub_battery": "formulate", "version": "v1", "task_count": 12, "notes": "#000046 Phase 2 — HARD live-path Formulate tier. Every input is prose that arborist.qa.parse_claims.parse_pointer_claims SHOULD segment into a particular claim lattice; `expected_lattice` records that correct structure. The parser is line/bullet-based — one line ⇒ one claim, [E#] tokens on that line attach to it — so 8 of 12 it mis-segments: it merges multiple sentence/semicolon/clause claims that share a line into ONE claim with all the pointers, or splits a wrapped bullet into two. Those 8 fail at HEAD by design (count mismatch + pointer-set mismatch). The other 4 are well-formed bullet/numbered lists / single claims the parser handles correctly. Rate at HEAD = 4/12 = 0.333... — a real below-ceiling baseline; parse_pointer_claims is deterministic, so it's stable. A claim-lattice parser that does sentence/clause segmentation (split on '. ', ';', subordinating conjunctions, inline enumerations) and joins wrapped bullets would lift the rate toward 1.0 → a positive gamma*Delta5f term for that child fork. NOT part of `make bench-5f` / `runner --all` / `bench-fork-baseline`; pinned via `make bench-5f-formulate-hard` / shares `make bench-fork-baseline-hard` is falsification-only — formulate-hard is its own target. See ticket #000046 §5."}}
|
||||
{"id": "5f-form-hard-001", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Water is wet [E1] and fire is hot [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Water is wet.", "pointer_ids": ["E1"]}, {"claim_text": "Fire is hot.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: two and-joined sentence claims on one line → parser merges to 1 claim with [E1,E2]; correct is 2."}
|
||||
{"id": "5f-form-hard-002", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Water is wet [E1]; fire is hot [E2]; ice is cold [E3].", "expected_lattice": {"claim_count": 3, "claims": [{"claim_text": "Water is wet.", "pointer_ids": ["E1"]}, {"claim_text": "Fire is hot.", "pointer_ids": ["E2"]}, {"claim_text": "Ice is cold.", "pointer_ids": ["E3"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: three semicolon-separated claims on one line → parser merges to 1 claim with [E1,E2,E3]; correct is 3."}
|
||||
{"id": "5f-form-hard-003", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- The cell respires aerobically\n to produce ATP efficiently. [E1]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "The cell respires aerobically to produce ATP efficiently.", "pointer_ids": ["E1"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: a wrapped bullet → parser splits into 2 claims, pointer on the wrong half; correct is 1 joined claim with [E1]."}
|
||||
{"id": "5f-form-hard-004", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Water boils at 100 C [E1]. It freezes at 0 C [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Water boils at 100 C.", "pointer_ids": ["E1"]}, {"claim_text": "It freezes at 0 C.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: two sentences on one line (no bullets) → parser merges to 1 claim with [E1,E2]; correct is 2."}
|
||||
{"id": "5f-form-hard-005", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "The engine runs hot [E1] because the coolant leaked [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "The engine runs hot.", "pointer_ids": ["E1"]}, {"claim_text": "The coolant leaked.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: main clause + 'because' sub-claim on one line → parser merges to 1 claim with [E1,E2]; correct is 2 (the cause is its own claim)."}
|
||||
{"id": "5f-form-hard-006", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Two facts: (1) water is wet [E1] (2) fire is hot [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Water is wet.", "pointer_ids": ["E1"]}, {"claim_text": "Fire is hot.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: an inline (1)(2) enumeration on one line → parser merges to 1 claim with [E1,E2]; correct is 2."}
|
||||
{"id": "5f-form-hard-007", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "Key points: clarity matters [E1], brevity matters [E2], accuracy matters [E3].", "expected_lattice": {"claim_count": 3, "claims": [{"claim_text": "Clarity matters.", "pointer_ids": ["E1"]}, {"claim_text": "Brevity matters.", "pointer_ids": ["E2"]}, {"claim_text": "Accuracy matters.", "pointer_ids": ["E3"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: a comma-separated list after a colon on one line → parser merges to 1 claim with [E1,E2,E3]; correct is 3."}
|
||||
{"id": "5f-form-hard-008", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "The sky is blue [E1] - the grass is green [E2].", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "The sky is blue.", "pointer_ids": ["E1"]}, {"claim_text": "The grass is green.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "MIS-SEGMENT at HEAD: two dash-joined clauses on one line → parser merges to 1 claim with [E1,E2]; correct is 2."}
|
||||
{"id": "5f-form-hard-009", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "- Sky is blue. [E1]\n- Grass is green. [E2]", "expected_lattice": {"claim_count": 2, "claims": [{"claim_text": "Sky is blue.", "pointer_ids": ["E1"]}, {"claim_text": "Grass is green.", "pointer_ids": ["E2"]}]}, "expected": "pass", "note": "Correctly segmented at HEAD — well-formed 2-bullet list. PASSES today; kept as a headroom marker (a parser regression that started merging bullets would drop the rate)."}
|
||||
{"id": "5f-form-hard-010", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "1. Sky is blue [E1]\n2. Grass is green [E2]\n3. Sun is bright [E3]", "expected_lattice": {"claim_count": 3, "claims": [{"claim_text": "Sky is blue.", "pointer_ids": ["E1"]}, {"claim_text": "Grass is green.", "pointer_ids": ["E2"]}, {"claim_text": "Sun is bright.", "pointer_ids": ["E3"]}]}, "expected": "pass", "note": "Correctly segmented at HEAD — well-formed 3-item numbered list. PASSES today."}
|
||||
{"id": "5f-form-hard-011", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "The sky is blue. [E1]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "The sky is blue.", "pointer_ids": ["E1"]}]}, "expected": "pass", "note": "Correctly segmented at HEAD — single claim, single pointer. PASSES today."}
|
||||
{"id": "5f-form-hard-012", "battery": "5f", "sub_battery": "formulate", "version": "v1", "carrier": "text", "domain": "claim_lattice", "pi_star_ref": "claim-lattice@v1", "input_text": "The sky is blue and clear. [E1, E2]", "expected_lattice": {"claim_count": 1, "claims": [{"claim_text": "The sky is blue and clear.", "pointer_ids": ["E1", "E2"]}]}, "expected": "pass", "note": "Correctly segmented at HEAD — single claim citing two pointers. PASSES today."}
|
||||
|
|
@ -93,7 +93,7 @@ Newest first. Update on every open/close.
|
|||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000047 | ForkScore `_delta_*` aggregator (mean vs max vs sum) | **closed · 2026-05-11** — Option D: `WeightSet.delta_aggregator` ∈ {`mean`,`max`,`sum`} (default `mean` unchanged → no `ESTIMATOR_VERSION` bump), `fork_score._delta_5{s,t,f}` dispatch via `_aggregate`, recorded in `ScoredFork.weights`, per-sub `HARD_REGRESSION_FLOOR` flags aggregator-independent; bench data behind keeping `mean` in `5f-threshold-calibration-2026-05-11.md` §5; 8+1 tests. #000012-revision / #000025 §10.14 follow-up | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | in progress · **Phase 1 landed 2026-05-11** — `falsification-hard-v1.jsonl` (12 near-misses, rate 4/12 at HEAD; `verify_quotes` over-grounds 8 via paraphrase/entity matching), `make bench-5f-falsification-hard` / `bench-fork-baseline-hard`, worked-example test (`fork_score` γ·Δ5f → positive on a lift to 1.0). Closure pending an actual `verify_quotes` tightening that lifts the rate; Phase 2 (extend to Formulate / a retrieval-backed sub) optional. #000025 §10.14 follow-up; gates #000047 | 2026-05-11 | — |
|
||||
| #000046 | Harder 5S/5T/5F fixture tier (below-ceiling baselines) | in progress · **Phase 1 + Phase 2 landed 2026-05-11** — `falsification-hard-v1.jsonl` (12 near-misses, `verify_quotes` over-grounds 8 → rate 4/12; `make bench-5f-falsification-hard` / `bench-fork-baseline-hard`, worked-example test `fork_score` γ·Δ5f → positive) + `formulate-hard-v1.jsonl` (12 mis-segment inputs, `parse_pointer_claims` mis-segments 8 → rate 4/12; `make bench-5f-formulate-hard`). Two below-ceiling 5F subs now exist. Closure pending an actual surface improvement (`verify_quotes` tightening — bench-gated — or `parse_pointer_claims` segmentation) that lifts a rate. #000025 §10.14 follow-up; #000047 closed | 2026-05-11 | — |
|
||||
| #000045 | Prometheus-Σ Phase 3 sleep-sweep scheduler (gating ticket) | open · doc-only scaffold 2026-05-10; pins 8 governance parameters + 4 retrigger gates; opens implementation only after one retrigger fires | 2026-05-10 | — |
|
||||
| #000044 | AUTOCOUNT doc-drift discipline | closed · landed across `fc5ba50` / `03c0f6a` / `6c6defb` / `f5dbfab` / `3b30126` 2026-05-10 (mechanism + 4 metrics + 54 tags across 7 doc files; harness catches drift at test time, refresh is 60-second turnaround) | 2026-05-10 | — |
|
||||
| #000043 | Empirical tightening of T3 bound constants (#000036 Phase 2) | open · awaiting go/no-go (parks until v7 deployment surfaces measurement data) | 2026-05-10 | — |
|
||||
|
|
|
|||
|
|
@ -1,12 +1,19 @@
|
|||
# Ticket #000046 — Harder 5S/5T/5F fixture tier (below-ceiling baselines)
|
||||
|
||||
**Status:** in progress · **Phase 1 landed 2026-05-11** (Option A,
|
||||
narrow start — 5F Falsification hard pack: `bench/fixtures/5f/
|
||||
falsification-hard-v1.jsonl`, rate 4/12 ≈ 0.333 at HEAD, stable;
|
||||
`make bench-5f-falsification-hard` / `make bench-fork-baseline-hard`;
|
||||
worked-example test shows `fork_score`'s γ·Δ5f going positive on a
|
||||
lift to 1.0). Closure pending an actual `verify_quotes` tightening
|
||||
that lifts the rate (a separate, larger task) — see §5 + §7.
|
||||
**Status:** in progress · **Phase 1 + Phase 2 landed 2026-05-11.**
|
||||
Phase 1 (Option A) — 5F Falsification hard pack
|
||||
`bench/fixtures/5f/falsification-hard-v1.jsonl` (12 near-misses, rate
|
||||
4/12 ≈ 0.333 at HEAD, stable; `make bench-5f-falsification-hard` /
|
||||
`make bench-fork-baseline-hard`; worked-example test:
|
||||
`fork_score`'s γ·Δ5f goes positive on a lift to 1.0). Phase 2 —
|
||||
5F Formulate hard pack `bench/fixtures/5f/formulate-hard-v1.jsonl`
|
||||
(12 mis-segment inputs, rate 4/12 at HEAD, stable; `make
|
||||
bench-5f-formulate-hard`). Two below-ceiling 5F sub-batteries now
|
||||
exist. **Closure pending an actual surface improvement** that lifts
|
||||
either rate — `verify_quotes` tightening (falsification) or
|
||||
`parse_pointer_claims` sentence/clause segmentation (formulate) —
|
||||
each a separate, larger task touching the live surface, not the
|
||||
bench. See §5 + §7.
|
||||
**Opened:** 2026-05-11
|
||||
**Scope:** Give the 5S/5T/5F bench packs *headroom* — a fixture tier
|
||||
the runtime does **not** pass at rate 1.0 — so the ForkScore Δ-rate
|
||||
|
|
@ -161,8 +168,9 @@ testable only once there's a below-ceiling baseline to bench against.
|
|||
|
||||
## 5. Status
|
||||
|
||||
**In progress — Phase 1 landed 2026-05-11 (Option A, 5F Falsification
|
||||
hard pack).** Receipt:
|
||||
**In progress — Phase 1 + Phase 2 landed 2026-05-11.**
|
||||
|
||||
### Phase 1 (Option A) — 5F Falsification hard pack
|
||||
|
||||
- **Hard pack:** `bench/fixtures/5f/falsification-hard-v1.jsonl` — 12
|
||||
near-misses, each a *false / unsupported* claim whose correct
|
||||
|
|
@ -191,18 +199,44 @@ hard pack).** Receipt:
|
|||
bench-fork-baseline` / `runner --all` — the hard pack is a
|
||||
separate, deliberately-failing artifact pinned on its own.
|
||||
|
||||
**Still open (the actual closure):** Phase 1 *proved the loop* with a
|
||||
synthetic child at rate 1.0. Closing the ticket needs a **real
|
||||
`verify_quotes` tightening** that lifts the hard-pack rate above
|
||||
0.333 (stricter entity strategy / higher paraphrase coverage
|
||||
threshold / wider title-relevance check) — a separate, larger task
|
||||
(it touches the live verifier surface, not the bench). When that
|
||||
lands, this ticket closes with the rate-improvement → ForkScore Δ
|
||||
demonstrated on an *actual* change, and Phase 2 (extend the hard
|
||||
tier to Formulate / a retrieval-backed 5S/5T sub) opens if the loop
|
||||
holds. Until then: Phase 1 stands, the hard pack pins the gap, and
|
||||
the next-step menu is (a) tighten `verify_quotes`, or (b) extend the
|
||||
hard tier to Formulate now without waiting for the verifier work.
|
||||
### Phase 2 — 5F Formulate hard pack
|
||||
|
||||
- **Hard pack:** `bench/fixtures/5f/formulate-hard-v1.jsonl` — 12
|
||||
prose inputs that `arborist.qa.parse_claims.parse_pointer_claims`
|
||||
*should* segment into a particular claim lattice (recorded in
|
||||
`expected_lattice`). The parser is line/bullet-based — one line ⇒
|
||||
one claim, `[E#]` tokens attach to it — so 8 of 12 it mis-segments:
|
||||
merges and-/semicolon-/dash-joined or `(1)(2)`-enumerated multi-claim
|
||||
lines into one claim with all the pointers, or splits a wrapped
|
||||
bullet into two. Those 8 fail at HEAD on claim-count mismatch
|
||||
(+ pointer-set mismatch); the other 4 are well-formed bullet/
|
||||
numbered lists / single claims the parser handles right. **Rate at
|
||||
HEAD = 4/12 = 0.333…**, stable (`parse_pointer_claims` is
|
||||
deterministic). A claim-lattice parser that does sentence/clause
|
||||
segmentation (split on `. `, `;`, subordinating conjunctions, inline
|
||||
enumerations) + joins wrapped bullets lifts the rate toward 1.0 →
|
||||
positive `γ·Δ5f` for that child fork.
|
||||
- **Make target:** `make bench-5f-formulate-hard` (runs it; `|| true`
|
||||
past the runner's nonzero-on-failures exit). Not in `make bench-5f`
|
||||
/ `runner --all`.
|
||||
- **Test:** `tests/test_bench_batteries.py` —
|
||||
`test_5f_formulate_hard_pack_below_ceiling` (pins rate 4/12,
|
||||
source=live, the 8 fails are claim-count mis-segments).
|
||||
|
||||
### Still open (the actual closure)
|
||||
|
||||
Both phases *built the below-ceiling baselines* and proved the loop
|
||||
with synthetic children at rate 1.0. Closing the ticket needs a
|
||||
**real surface improvement** that lifts one of the rates: a
|
||||
`verify_quotes` tightening (stricter entity strategy / higher
|
||||
paraphrase coverage threshold / wider title-relevance check —
|
||||
falsification) or `parse_pointer_claims` sentence/clause segmentation
|
||||
(formulate) — each a separate, larger task touching the live surface,
|
||||
not the bench, and (for `verify_quotes`) a bench-gated change (a QA
|
||||
sweep must show STRICT-rate doesn't regress on legitimate answers).
|
||||
When one lands, this ticket closes with the rate-improvement →
|
||||
ForkScore Δ demonstrated on an *actual* change. Until then: Phases 1
|
||||
+ 2 stand, the two hard packs pin the gaps.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -212,9 +246,15 @@ hard tier to Formulate now without waiting for the verifier work.
|
|||
pack (12 near-misses, rate 4/12 at HEAD). `make
|
||||
bench-5f-falsification-hard` runs it; `make bench-fork-baseline-hard`
|
||||
pins it to `bench/results/baseline-falsification-hard.json`.
|
||||
- `bench/fixtures/5f/formulate-hard-v1.jsonl` — the Phase-2 hard pack
|
||||
(12 mis-segment inputs, rate 4/12 at HEAD). `make
|
||||
bench-5f-formulate-hard` runs it.
|
||||
- `arborist/qa/parse_claims.py` (`parse_pointer_claims`) — the
|
||||
line/bullet-based parser the Phase-2 pack exercises.
|
||||
- `tests/test_bench_batteries.py` —
|
||||
`test_5f_falsification_hard_pack_below_ceiling` (pins the rate +
|
||||
the over-ground/abstention distinction) +
|
||||
`test_5f_falsification_hard_pack_below_ceiling`,
|
||||
`test_5f_formulate_hard_pack_below_ceiling` (pin the rates +
|
||||
the over-ground / mis-segment distinctions) +
|
||||
`test_fork_score_positive_gamma_5f_on_hard_falsification_improvement`
|
||||
(the worked example).
|
||||
- `bench/scripts/fivef_threshold_calibration.py` +
|
||||
|
|
|
|||
|
|
@ -672,6 +672,30 @@ def test_5f_falsification_embedded_path_still_works():
|
|||
assert t.detail["source"] == "embedded"
|
||||
|
||||
|
||||
# --- #000046 Phase 2 — HARD live-path Formulate tier --------------
|
||||
|
||||
|
||||
def test_5f_formulate_hard_pack_below_ceiling():
|
||||
"""Below-ceiling baseline for the Formulate sub-battery: 12 inputs
|
||||
that parse_pointer_claims should segment a particular way; the
|
||||
line/bullet-based parser mis-segments 8 of 12 (merges multi-claim
|
||||
lines, splits wrapped bullets) → rate 4/12. parse_pointer_claims
|
||||
is deterministic, so this is stable; the pinned value fires if it
|
||||
shifts."""
|
||||
res = b_5f.run_formulate(F5F / "formulate-hard-v1.jsonl")
|
||||
assert res.pass_count == 4
|
||||
assert res.fail_count == 8
|
||||
assert res.metrics["structural_match_rate"] == pytest.approx(4 / 12)
|
||||
for t in res.per_task:
|
||||
assert t.detail["source"] == "live"
|
||||
# The 8 that fail do so on claim-count mismatch (the parser put
|
||||
# multiple claims on one line, or split one across lines) — that's
|
||||
# the mis-segmentation, not a noisy pointer/text near-miss.
|
||||
for t in res.per_task:
|
||||
if not t.passed:
|
||||
assert t.detail["count_ok"] is False
|
||||
|
||||
|
||||
# --- #000046 Phase 1 — HARD live-path Falsification tier ----------
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue