5f: Phase 1b.2 — Formulate runner wires to arborist.qa.parse_claims (live)
First sub-battery to bridge from synthetic gold output to actual
organism behavior. run_formulate now supports two fixture modes
selected per-task:
- Embedded (Phase 1a): produced_lattice in the fixture. 10 seed
fixtures continue to pass via this path.
- Live (Phase 1b.2): only input_text in the fixture; runner calls
arborist.qa.parse_claims.parse_pointer_claims(input_text) and
matches the live output against expected_lattice.
Embedded takes precedence if both fields are present. Per-task
detail.source ("embedded" | "live") surfaces in bench output so
synthetic vs live signal is distinguishable.
Surface:
- bench/batteries/b_5f.py — _live_produced_lattice helper +
two-mode dispatch in run_formulate
- bench/fixtures/5f/formulate-live-v1.jsonl — 15 live-mode fixtures
with input_text + expected_lattice (no produced_lattice)
- Makefile: bench-5f-formulate-live target
- Tests: 4 new in tests/test_bench_batteries.py
- live path routes through real parser, all 15 pass
- embedded path still works (10 Phase-1a fixtures)
- _live_produced_lattice helper directly verifies parse output
- fixture missing both fields fails cleanly with explanatory reason
Phase 1a fixture digests unchanged. _DEFAULT_FIXTURES still points
at formulate-v1.jsonl so `runner --all` behavior is identical;
live-mode fixtures invoked via explicit --fixtures path.
Full suite: 1196 passed, 36 skipped.
Pattern set. Function/Finetuning/Falsification/Feedback Loop
follow in subsequent commits.
This commit is contained in:
parent
6e20c792c4
commit
ba653755e4
5 changed files with 141 additions and 5 deletions
|
|
@ -517,6 +517,26 @@ test_5f_feedback_loop_observation_affects_downstream_snapshot
|
|||
14. Threshold calibration for v8 selection acceptance handed off
|
||||
to ticket #000012.
|
||||
|
||||
### Phase 1b.2 — first live wire-up landed 2026-05-08
|
||||
|
||||
`run_formulate` now supports two fixture modes:
|
||||
|
||||
- **Embedded** (Phase 1a): fixture provides `produced_lattice`
|
||||
directly. The 10 seed fixtures continue to pass.
|
||||
- **Live**: fixture provides `input_text` only; runner calls
|
||||
`arborist.qa.parse_claims.parse_pointer_claims` on it and
|
||||
matches the live output against `expected_lattice`.
|
||||
|
||||
`bench/fixtures/5f/formulate-live-v1.jsonl` ships 15 live-mode
|
||||
fixtures. `make bench-5f-formulate-live` runs them. Per-task
|
||||
`detail.source` reports `"embedded"` or `"live"` so bench output
|
||||
distinguishes synthetic from live signal.
|
||||
|
||||
This is the **first sub-battery to bridge from synthetic gold to
|
||||
actual organism behavior** — proves the Phase 1b.2 pattern. The
|
||||
remaining sub-batteries (Function, Finetuning, Falsification,
|
||||
Feedback Loop) follow the same pattern in subsequent commits.
|
||||
|
||||
---
|
||||
|
||||
## 11. Status
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue