engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

5f: Phase 1b.2 — Feedback Loop wires to live arborist memory + audit chain

Browse source 
Second live wire-up (after Formulate). run_feedback_loop now supports
both modes:

- Embedded (Phase 1a): chain of (operation, observation) string pairs;
  runner aggregates observations and string-matches expected_delta.
- Live (Phase 1b.2): live_chain of typed ops applied to a fresh temp
  arborist shard via real append_audit + memory.snapshot +
  selfmodel.snapshot. expected_delta is a dict of predicates against
  the resulting audit_events / memory_branch_summaries.

The live helper _live_feedback_chain creates a tempfile-backed shard,
runs the chain through real arborist surfaces, and queries the final
state. Every audit event chains via the production append_audit, so
the audit chain is re-verifiable after live execution (see new test
test_5f_live_feedback_chain_audit_chain_intact).

Three predicate types in expected_delta:

- audit_event_type_present: named event_type appears in audit chain
- memory_branch_present: named branch_id in memory_branch_summaries
- body_substring_present: substring appears in any audit body JSON

Surface:

- bench/batteries/b_5f.py — _live_feedback_chain helper +
  _live_delta_satisfied predicate checker; two-mode dispatch in
  run_feedback_loop
- bench/fixtures/5f/feedback-loop-live-v1.jsonl — 12 live fixtures
  exercising providence_write, providence_repair, memory_snapshot,
  selfmodel_snapshot ops. Includes 2 negative fixtures testing the
  predicate checker (expected_delta absent → expected:fail).
- Makefile: bench-5f-feedback-loop-live + bench-5f-live aggregate
  for all 5F Phase-1b.2 live wire-ups.
- Tests: 5 new in tests/test_bench_batteries.py
  - live path runs all 12 fixtures
  - embedded path still works (10 Phase-1a fixtures)
  - helper directly tests audit-event write
  - rejects unknown live op type
  - audit chain re-verifies after live ops

Two of the five 5F sub-batteries now bridge synthetic → live
(Formulate + Feedback Loop). Function/Finetuning/Falsification
follow in subsequent commits.

Full suite: 1201 passed, 36 skipped.
This commit is contained in:
russell@unturf.com 2026-05-08 08:30:04 -04:00
parent ba653755e4
commit 92b3a34b7a
No known key found for this signature in database
4 changed files with 283 additions and 31 deletions
Download patch file Download diff file Expand all files Collapse all files

6
Makefile
View file

@ -270,6 +270,12 @@ bench-5f-formulate-live: bootstrap ## 5F Formulate via live arborist.qa.parse_cl
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5f --sub formulate \
--fixtures bench/fixtures/5f/formulate-live-v1.jsonl
bench-5f-feedback-loop-live: bootstrap ## 5F Feedback Loop via live arborist memory + audit chain (Phase 1b.2)
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5f --sub feedback-loop \
--fixtures bench/fixtures/5f/feedback-loop-live-v1.jsonl
bench-5f-live: bench-5f-formulate-live bench-5f-feedback-loop-live ## all 5F Phase-1b.2 live wire-ups
bench-5r: bootstrap ## 5R battery (React+Rearrange+Restore+Replicate+Resonate)
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5r --sub react --fixtures bench/fixtures/5r/react-v1.jsonl
PYTHONUNBUFFERED=1 $(PY) -m bench.batteries.runner --battery 5r --sub rearrange --fixtures bench/fixtures/5r/rearrange-v1.jsonl

218
bench/batteries/b_5f.py
View file

@ -439,13 +439,132 @@ def run_formulate(fixtures_path: Path) -> BatteryResult:
# ---------------------------------------------------------------------
def _live_feedback_chain(live_chain: list[dict]) -> dict:
"""Apply ``live_chain`` ops against a fresh temp arborist shard;
return the final state for the runner to assert against.
Supported ops:
- ``append_audit`` — calls :func:`arborist.store.append_audit`
with the fixture-supplied ``event_type``, ``subject_root``,
and ``body``.
- ``memory_snapshot`` — calls
:func:`arborist.memory.snapshot` +
:func:`arborist.memory.store_snapshot` to persist a memory
snapshot.
- ``selfmodel_snapshot`` — same for SelfModel.
Returns ``{"audit_event_types": [...], "memory_branch_ids":
[...], "memory_event_bodies": [...]}`` — enough for the runner
to check whether `expected_delta` propagated into downstream
state.
"""
import os
import tempfile
import json as _json
from arborist.memory import (
snapshot as memory_snapshot,
store_snapshot as memory_store,
)
from arborist.selfmodel import (
snapshot as selfmodel_snapshot,
store_snapshot as selfmodel_store,
)
from arborist.store import append_audit, connect, transaction
tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
tmp.close()
try:
conn = connect(tmp.name)
try:
with transaction(conn):
for step in live_chain:
op = step["op"]
if op == "append_audit":
append_audit(
conn,
event_type=step["event_type"],
subject_root=step.get("subject_root"),
body=step.get("body", {}),
)
elif op == "memory_snapshot":
ms = memory_snapshot(conn)
memory_store(conn, ms)
elif op == "selfmodel_snapshot":
sm = selfmodel_snapshot(conn)
selfmodel_store(conn, sm)
else:
raise ValueError(f"unknown live op: {op!r}")
# Read final state.
audit_types = [
row["event_type"] for row in conn.execute(
"SELECT event_type FROM audit_events ORDER BY seq"
)
]
memory_branches = [
row["branch_id"] for row in conn.execute(
"SELECT DISTINCT branch_id FROM memory_branch_summaries"
)
]
event_bodies = [
row["body"] for row in conn.execute(
"SELECT body FROM audit_events ORDER BY seq"
)
]
return {
"audit_event_types": audit_types,
"memory_branch_ids": memory_branches,
"audit_event_bodies": event_bodies,
}
finally:
conn.close()
finally:
try:
os.unlink(tmp.name)
except OSError:
pass
def _live_delta_satisfied(state: dict, expected: dict) -> tuple[bool, str]:
"""Check expected_delta predicates against live final state.
Supported predicates:
- ``audit_event_type_present`` — the named event_type appears at
least once in the audit chain.
- ``memory_branch_present`` — the named branch_id appears in
memory_branch_summaries.
- ``body_substring_present`` — the substring appears in at least
one audit event body (loose check).
"""
for et in expected.get("audit_event_type_present", []):
if et not in state["audit_event_types"]:
return False, f"expected audit event_type {et!r} absent"
for bid in expected.get("memory_branch_present", []):
if bid not in state["memory_branch_ids"]:
return False, f"expected memory branch {bid!r} absent"
for needle in expected.get("body_substring_present", []):
if not any(needle in body for body in state["audit_event_bodies"]):
return False, f"substring {needle!r} absent from audit bodies"
return True, "all live deltas satisfied"
def run_feedback_loop(fixtures_path: Path) -> BatteryResult:
"""Integration coverage + feedback_efficiency metric.
Each task carries a chain of (operation, observation) pairs. The
last item lists ``expected_delta`` — content the post-chain
state must contain. Pass = expected_delta appears in the
aggregated observation feed.
Two fixture modes (selected per-task):
- **Embedded** (Phase 1a): fixture provides a ``chain`` of
``(operation, observation)`` pairs; runner aggregates the
observation text and checks ``expected_delta`` (a string
marker) appears in it.
- **Live** (Phase 1b.2): fixture provides ``live_chain`` —
a sequence of ops applied to a fresh temp arborist shard via
:func:`_live_feedback_chain` — plus an ``expected_delta``
dict with predicates against the resulting audit chain /
memory branches. Tests whether observations actually
propagate into downstream arborist state.
Per #000025 §5.5 and the 2026-05-08 fbd99a8 review:
``feedback_efficiency = downstream_effect_count / Δcapital_cost``
@ -467,33 +586,70 @@ def run_feedback_loop(fixtures_path: Path) -> BatteryResult:
)
continue
try:
chain = task["chain"]
total_obs += len(chain)
observed_text = " ".join(
step.get("observation", "") for step in chain
).lower()
final = chain[-1]
delta_marker = (final.get("expected_delta") or "").lower()
integrated_ok = bool(delta_marker) and delta_marker in observed_text
if integrated_ok:
integrated += 1
expected = task.get("expected", "pass")
observed = "pass" if integrated_ok else "fail"
passed = observed == expected
# Per-task feedback_efficiency: one downstream effect per
# successful chain, divided by chain-length-as-cost-proxy.
downstream_effect = 1.0 if integrated_ok else 0.0
cost = float(len(chain))
efficiency = _efficiency(downstream_effect, cost)
efficiency_values.append(efficiency)
if efficiency not in (EFFICIENCY_INFINITE, -EFFICIENCY_INFINITE):
finite_efficiency_values.append(efficiency)
detail = {
"integrated": integrated_ok,
"delta_marker": delta_marker,
"chain_length": len(chain),
"feedback_efficiency": efficiency,
}
if "live_chain" in task:
source = "live"
live_chain = task["live_chain"]
state = _live_feedback_chain(live_chain)
ok_live, why = _live_delta_satisfied(
state, task.get("expected_delta", {}) or {}
)
integrated_ok = ok_live
chain_length = len(live_chain)
total_obs += chain_length
if integrated_ok:
integrated += 1
expected_outcome = task.get("expected", "pass")
observed = "pass" if integrated_ok else "fail"
passed = observed == expected_outcome
downstream_effect = 1.0 if integrated_ok else 0.0
cost = float(chain_length)
efficiency = _efficiency(downstream_effect, cost)
efficiency_values.append(efficiency)
if efficiency not in (
EFFICIENCY_INFINITE, -EFFICIENCY_INFINITE
):
finite_efficiency_values.append(efficiency)
detail = {
"source": source,
"integrated": integrated_ok,
"reason": why,
"chain_length": chain_length,
"feedback_efficiency": efficiency,
"live_state": {
"audit_event_count": len(state["audit_event_types"]),
"memory_branch_count": len(state["memory_branch_ids"]),
},
}
else:
source = "embedded"
chain = task["chain"]
total_obs += len(chain)
observed_text = " ".join(
step.get("observation", "") for step in chain
).lower()
final = chain[-1]
delta_marker = (final.get("expected_delta") or "").lower()
integrated_ok = bool(delta_marker) and delta_marker in observed_text
if integrated_ok:
integrated += 1
expected_outcome = task.get("expected", "pass")
observed = "pass" if integrated_ok else "fail"
passed = observed == expected_outcome
downstream_effect = 1.0 if integrated_ok else 0.0
cost = float(len(chain))
efficiency = _efficiency(downstream_effect, cost)
efficiency_values.append(efficiency)
if efficiency not in (
EFFICIENCY_INFINITE, -EFFICIENCY_INFINITE
):
finite_efficiency_values.append(efficiency)
detail = {
"source": source,
"integrated": integrated_ok,
"delta_marker": delta_marker,
"chain_length": len(chain),
"feedback_efficiency": efficiency,
}
except Exception as exc: # noqa: BLE001
passed = False
detail = {"reason": f"{type(exc).__name__}: {exc}"}

13
bench/fixtures/5f/feedback-loop-live-v1.jsonl Normal file
View file

@ -0,0 +1,13 @@
{"_meta":{"battery":"5f","sub_battery":"feedback-loop","version":"v1","task_count":12,"notes":"Phase 1b.2: live_chain ops applied to a fresh temp arborist shard via append_audit + memory.snapshot + selfmodel.snapshot. Tests whether observations actually propagate into downstream audit_events / memory_branch_summaries / selfmodel_records."}}
{"id":"5f-fb-live-001","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"append_audit","event_type":"providence_write","body":{"audit_mode":"STRICT","violations":[]}},{"op":"memory_snapshot"}],"expected_delta":{"audit_event_type_present":["providence_write","memory_snapshot_landed"],"memory_branch_present":["audit-mode-distribution","falsification-state","failure-motif:violations"]},"expected":"pass"}
{"id":"5f-fb-live-002","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"append_audit","event_type":"providence_write","body":{"audit_mode":"HYBRID","violations":["TITLE_MISMATCH"]}},{"op":"memory_snapshot"}],"expected_delta":{"audit_event_type_present":["providence_write","memory_snapshot_landed"],"body_substring_present":["TITLE_MISMATCH"]},"expected":"pass"}
{"id":"5f-fb-live-003","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"append_audit","event_type":"providence_write","body":{"audit_mode":"STRICT","violations":[]}},{"op":"append_audit","event_type":"providence_write","body":{"audit_mode":"HYBRID","violations":["WARRANT_MISSING"]}},{"op":"memory_snapshot"}],"expected_delta":{"audit_event_type_present":["providence_write","memory_snapshot_landed"],"body_substring_present":["WARRANT_MISSING","STRICT","HYBRID"]},"expected":"pass"}
{"id":"5f-fb-live-004","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"selfmodel_snapshot","domain":"capability_transition","pi_star_ref":"pi_selfmodel_v1","live_chain":[{"op":"selfmodel_snapshot"}],"expected_delta":{"audit_event_type_present":["selfmodel_snapshot_landed"]},"expected":"pass"}
{"id":"5f-fb-live-005","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"append_audit","event_type":"providence_write","body":{"audit_mode":"UNGROUNDED","violations":[]}},{"op":"memory_snapshot"},{"op":"selfmodel_snapshot"}],"expected_delta":{"audit_event_type_present":["providence_write","memory_snapshot_landed","selfmodel_snapshot_landed"]},"expected":"pass"}
{"id":"5f-fb-live-006","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"memory_snapshot"}],"expected_delta":{"audit_event_type_present":["memory_snapshot_landed"],"memory_branch_present":["audit-mode-distribution"]},"expected":"pass"}
{"id":"5f-fb-live-007","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"append_audit","event_type":"providence_repair","body":{"changes":[{"kind":"strip_citation"}]}},{"op":"memory_snapshot"}],"expected_delta":{"audit_event_type_present":["providence_repair","memory_snapshot_landed"]},"expected":"pass"}
{"id":"5f-fb-live-008","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"append_audit","event_type":"providence_write","body":{"audit_mode":"STRICT","violations":["LAZY_ANCHOR_DEMOTED"]}},{"op":"memory_snapshot"}],"expected_delta":{"body_substring_present":["LAZY_ANCHOR_DEMOTED"]},"expected":"pass"}
{"id":"5f-fb-live-009","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"append_audit","event_type":"providence_write","body":{"audit_mode":"STRICT","violations":[]}}],"expected_delta":{"audit_event_type_present":["providence_write"],"memory_branch_present":[]},"expected":"pass"}
{"id":"5f-fb-live-010","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"memory_snapshot"},{"op":"append_audit","event_type":"providence_write","body":{"audit_mode":"HYBRID","violations":["TOO_MANY_CLAIMS"]}},{"op":"memory_snapshot"}],"expected_delta":{"audit_event_type_present":["memory_snapshot_landed","providence_write"],"body_substring_present":["TOO_MANY_CLAIMS"]},"expected":"pass"}
{"id":"5f-fb-live-011","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"append_audit","event_type":"providence_write","body":{"audit_mode":"STRICT","violations":[]}},{"op":"memory_snapshot"}],"expected_delta":{"audit_event_type_present":["nonexistent_event_type"]},"expected":"fail"}
{"id":"5f-fb-live-012","battery":"5f","sub_battery":"feedback-loop","version":"v1","carrier":"memory_snapshot","domain":"memory_root","pi_star_ref":"pi_memory_v1","live_chain":[{"op":"memory_snapshot"}],"expected_delta":{"body_substring_present":["NEVER_PRESENT_STRING"]},"expected":"fail"}

77
tests/test_bench_batteries.py
View file

@ -468,6 +468,83 @@ def test_5f_formulate_rejects_fixture_with_neither_field(tmp_path):
assert "produced_lattice" in res.per_task[0].detail["reason"]
# --- 5F Phase 1b.2 — Feedback Loop live wire-up -----------------
def test_5f_feedback_loop_live_path_writes_real_audit_events():
"""live_chain ops applied to a fresh temp shard via append_audit
+ memory.snapshot; expected_delta predicates verified against
the resulting audit_events / memory_branch_summaries."""
res = b_5f.run_feedback_loop(F5F / "feedback-loop-live-v1.jsonl")
assert res.pass_count == 12
# Every passing task must report source=live.
for t in res.per_task:
if t.passed:
assert t.detail["source"] == "live"
def test_5f_feedback_loop_embedded_path_still_works():
res = b_5f.run_feedback_loop(F5F / "feedback-loop-v1.jsonl")
assert res.pass_count == 10
for t in res.per_task:
assert t.detail["source"] == "embedded"
def test_5f_live_feedback_chain_helper_appends_audit_event(tmp_path):
"""The live helper actually writes an audit event to a real
arborist shard."""
from bench.batteries.b_5f import _live_feedback_chain
state = _live_feedback_chain([
{"op": "append_audit", "event_type": "test_event", "body": {"k": "v"}},
{"op": "memory_snapshot"},
])
assert "test_event" in state["audit_event_types"]
assert "memory_snapshot_landed" in state["audit_event_types"]
assert "audit-mode-distribution" in state["memory_branch_ids"]
def test_5f_live_feedback_rejects_unknown_op(tmp_path):
from bench.batteries.b_5f import _live_feedback_chain
with pytest.raises(ValueError, match="unknown live op"):
_live_feedback_chain([{"op": "fake_op"}])
def test_5f_live_feedback_chain_audit_chain_intact():
"""Live helper exercises the real audit chain; chain should
re-verify after the helper completes."""
import hashlib
import os
import tempfile
from arborist.store import append_audit, connect, transaction
tmp = tempfile.NamedTemporaryFile(suffix=".db", delete=False)
tmp.close()
try:
conn = connect(tmp.name)
with transaction(conn):
append_audit(conn, event_type="x", subject_root=None, body={"a": 1})
append_audit(conn, event_type="y", subject_root=None, body={"b": 2})
rows = conn.execute(
"SELECT event_hash, prev_event_hash, body FROM audit_events ORDER BY seq"
).fetchall()
prev = None
for row in rows:
h = hashlib.sha256()
if prev is not None:
h.update(bytes.fromhex(prev))
h.update(row["body"].encode("utf-8", errors="surrogatepass"))
assert h.hexdigest() == row["event_hash"]
prev = row["event_hash"]
conn.close()
finally:
try:
os.unlink(tmp.name)
except OSError:
pass
# --- 5R Phase 2 (#000021) ----------------------------------------