5f: Phase 1b.2 — Formulate runner wires to arborist.qa.parse_claims (live)
First sub-battery to bridge from synthetic gold output to actual
organism behavior. run_formulate now supports two fixture modes
selected per-task:
- Embedded (Phase 1a): produced_lattice in the fixture. 10 seed
fixtures continue to pass via this path.
- Live (Phase 1b.2): only input_text in the fixture; runner calls
arborist.qa.parse_claims.parse_pointer_claims(input_text) and
matches the live output against expected_lattice.
Embedded takes precedence if both fields are present. Per-task
detail.source ("embedded" | "live") surfaces in bench output so
synthetic vs live signal is distinguishable.
Surface:
- bench/batteries/b_5f.py — _live_produced_lattice helper +
two-mode dispatch in run_formulate
- bench/fixtures/5f/formulate-live-v1.jsonl — 15 live-mode fixtures
with input_text + expected_lattice (no produced_lattice)
- Makefile: bench-5f-formulate-live target
- Tests: 4 new in tests/test_bench_batteries.py
- live path routes through real parser, all 15 pass
- embedded path still works (10 Phase-1a fixtures)
- _live_produced_lattice helper directly verifies parse output
- fixture missing both fields fails cleanly with explanatory reason
Phase 1a fixture digests unchanged. _DEFAULT_FIXTURES still points
at formulate-v1.jsonl so `runner --all` behavior is identical;
live-mode fixtures invoked via explicit --fixtures path.
Full suite: 1196 passed, 36 skipped.
Pattern set. Function/Finetuning/Falsification/Feedback Loop
follow in subsequent commits.
This commit is contained in:
parent
6e20c792c4
commit
ba653755e4
5 changed files with 141 additions and 5 deletions
|
|
@ -413,6 +413,61 @@ def test_capital_cost_delta_handles_missing_budget():
|
|||
}) == 5.0
|
||||
|
||||
|
||||
# --- 5F Phase 1b.2 — Formulate live wire-up ---------------------
|
||||
|
||||
|
||||
def test_5f_formulate_live_path_routes_through_parse_claims():
|
||||
"""The new formulate-live-v1.jsonl fixtures use input_text only;
|
||||
runner derives produced_lattice via parse_pointer_claims and
|
||||
matches against expected_lattice."""
|
||||
res = b_5f.run_formulate(F5F / "formulate-live-v1.jsonl")
|
||||
assert res.pass_count == 15
|
||||
assert res.metrics["structural_match_rate"] == 1.0
|
||||
# Every task ran through the live path.
|
||||
for t in res.per_task:
|
||||
assert t.detail["source"] == "live"
|
||||
|
||||
|
||||
def test_5f_formulate_embedded_path_still_works():
|
||||
"""Phase 1a fixtures (embedded produced_lattice) keep working
|
||||
after the Phase 1b.2 wire-up. Backward compat invariant."""
|
||||
res = b_5f.run_formulate(F5F / "formulate-v1.jsonl")
|
||||
assert res.pass_count == 10
|
||||
for t in res.per_task:
|
||||
assert t.detail["source"] == "embedded"
|
||||
|
||||
|
||||
def test_5f_formulate_live_helper_uses_real_arborist_parser(tmp_path):
|
||||
"""The live path actually calls
|
||||
arborist.qa.parse_claims.parse_pointer_claims — not a stub."""
|
||||
from bench.batteries.b_5f import _live_produced_lattice
|
||||
|
||||
out = _live_produced_lattice("- Hello world. [E1]\n- Second one. [E2]")
|
||||
assert out["claims"][0]["claim_text"] == "Hello world."
|
||||
assert out["claims"][0]["pointer_ids"] == ["E1"]
|
||||
assert out["claims"][1]["claim_text"] == "Second one."
|
||||
assert out["claims"][1]["pointer_ids"] == ["E2"]
|
||||
|
||||
|
||||
def test_5f_formulate_rejects_fixture_with_neither_field(tmp_path):
|
||||
"""Task without produced_lattice OR input_text → fails cleanly."""
|
||||
p = tmp_path / "bad.jsonl"
|
||||
p.write_text(
|
||||
json.dumps({"_meta": {"battery": "5f", "sub_battery": "formulate", "version": "v1"}}) + "\n" +
|
||||
json.dumps({
|
||||
"id": "test",
|
||||
"carrier": "text",
|
||||
"domain": "claim_lattice",
|
||||
"pi_star_ref": "claim-lattice@v1",
|
||||
"expected_lattice": {"claim_count": 0, "claims": []},
|
||||
}) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
res = b_5f.run_formulate(p)
|
||||
assert res.fail_count == 1
|
||||
assert "produced_lattice" in res.per_task[0].detail["reason"]
|
||||
|
||||
|
||||
# --- 5R Phase 2 (#000021) ----------------------------------------
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue