5f: Phase 1b.2 — Formulate runner wires to arborist.qa.parse_claims (live)

First sub-battery to bridge from synthetic gold output to actual
organism behavior. run_formulate now supports two fixture modes
selected per-task:

- Embedded (Phase 1a): produced_lattice in the fixture. 10 seed
  fixtures continue to pass via this path.
- Live (Phase 1b.2): only input_text in the fixture; runner calls
  arborist.qa.parse_claims.parse_pointer_claims(input_text) and
  matches the live output against expected_lattice.

Embedded takes precedence if both fields are present. Per-task
detail.source ("embedded" | "live") surfaces in bench output so
synthetic vs live signal is distinguishable.

Surface:

- bench/batteries/b_5f.py — _live_produced_lattice helper +
  two-mode dispatch in run_formulate
- bench/fixtures/5f/formulate-live-v1.jsonl — 15 live-mode fixtures
  with input_text + expected_lattice (no produced_lattice)
- Makefile: bench-5f-formulate-live target
- Tests: 4 new in tests/test_bench_batteries.py
  - live path routes through real parser, all 15 pass
  - embedded path still works (10 Phase-1a fixtures)
  - _live_produced_lattice helper directly verifies parse output
  - fixture missing both fields fails cleanly with explanatory reason

Phase 1a fixture digests unchanged. _DEFAULT_FIXTURES still points
at formulate-v1.jsonl so `runner --all` behavior is identical;
live-mode fixtures invoked via explicit --fixtures path.

Full suite: 1196 passed, 36 skipped.

Pattern set. Function/Finetuning/Falsification/Feedback Loop
follow in subsequent commits.
This commit is contained in:
russell@unturf.com 2026-05-08 08:20:51 -04:00
parent 6e20c792c4
commit ba653755e4
No known key found for this signature in database
5 changed files with 141 additions and 5 deletions

View file

@ -413,6 +413,61 @@ def test_capital_cost_delta_handles_missing_budget():
}) == 5.0
# --- 5F Phase 1b.2 — Formulate live wire-up ---------------------
def test_5f_formulate_live_path_routes_through_parse_claims():
"""The new formulate-live-v1.jsonl fixtures use input_text only;
runner derives produced_lattice via parse_pointer_claims and
matches against expected_lattice."""
res = b_5f.run_formulate(F5F / "formulate-live-v1.jsonl")
assert res.pass_count == 15
assert res.metrics["structural_match_rate"] == 1.0
# Every task ran through the live path.
for t in res.per_task:
assert t.detail["source"] == "live"
def test_5f_formulate_embedded_path_still_works():
"""Phase 1a fixtures (embedded produced_lattice) keep working
after the Phase 1b.2 wire-up. Backward compat invariant."""
res = b_5f.run_formulate(F5F / "formulate-v1.jsonl")
assert res.pass_count == 10
for t in res.per_task:
assert t.detail["source"] == "embedded"
def test_5f_formulate_live_helper_uses_real_arborist_parser(tmp_path):
"""The live path actually calls
arborist.qa.parse_claims.parse_pointer_claims not a stub."""
from bench.batteries.b_5f import _live_produced_lattice
out = _live_produced_lattice("- Hello world. [E1]\n- Second one. [E2]")
assert out["claims"][0]["claim_text"] == "Hello world."
assert out["claims"][0]["pointer_ids"] == ["E1"]
assert out["claims"][1]["claim_text"] == "Second one."
assert out["claims"][1]["pointer_ids"] == ["E2"]
def test_5f_formulate_rejects_fixture_with_neither_field(tmp_path):
"""Task without produced_lattice OR input_text → fails cleanly."""
p = tmp_path / "bad.jsonl"
p.write_text(
json.dumps({"_meta": {"battery": "5f", "sub_battery": "formulate", "version": "v1"}}) + "\n" +
json.dumps({
"id": "test",
"carrier": "text",
"domain": "claim_lattice",
"pi_star_ref": "claim-lattice@v1",
"expected_lattice": {"claim_count": 0, "claims": []},
}) + "\n",
encoding="utf-8",
)
res = b_5f.run_formulate(p)
assert res.fail_count == 1
assert "produced_lattice" in res.per_task[0].detail["reason"]
# --- 5R Phase 2 (#000021) ----------------------------------------