#000025 §10.11 + §10.13 + §10.14 — close the 5F battery

Closes the three open Phase-1b items of #000025; every §10 closure
criterion is now met, so the ticket flips to closed.

§10.14 — ForkScore threshold-calibration handoff to #000012.
bench/scripts/fivef_threshold_calibration.py (make bench-5f-threshold-
calibration) runs the canonical 5S/5T/5F packs + the 5F live packs and
reports baseline rates, observability granularity (1/n), and fork_score
verdicts on the parent vs synthetic child perturbations →
bench/results/5f-threshold-calibration-2026-05-11.md. Findings written
into ticket-000012 §8: keep SIGNAL_FLOOR / HARD_REGRESSION_FLOOR at
0.05; the small 5S packs (syntax n=10, semantics n=8) are coarser than
the floors so any regression there trips hard-reject (intended zero-
tolerance); the 5x averaging dilution in _delta_*; ceiling saturation
(every pack at 1.0 -> delta-rate terms <= 0). No constant change
shipped. 6 tests in tests/test_fivef_threshold_calibration.py.

§10.13 — feedback latency / efficiency on real workload.
run_feedback_loop now computes feedback_latency (listed in §5.5 since
Phase 1a, never implemented) — wall-clock seconds to apply a live
chain against its temp shard, surfaced per-task
(feedback_latency_seconds) + battery (feedback_latency_mean_seconds,
feedback_live_task_count). For live chains feedback_efficiency's cost
denominator switched from len(chain) (count of requested ops) to the
persisted footprint _persisted_cost = audit-event rows the chain
actually wrote + their body bytes / 1e6. Embedded chains keep
len(chain) and report feedback_latency_seconds = None. Latency is a
wall-clock field (run-to-run variable, like BatteryResult.timestamp)
and is not a fork_score input. 3 tests in tests/test_bench_batteries.py.

§10.11 — real selfmodel finetuning chains.
bench/scripts/selfmodel_chain_snapshot.py (make bench-5f-selfmodel-
snapshot) appends one chained SelfModel snapshot per run to a
persistent shard (~/.arborist/shards/selfmodel-chain.db, override via
ARBORIST_SELFMODEL_CHAIN_DB) with one CapabilityClaim per sub-battery
(metric = "5S-syntax" etc., measured_value = that pack's rate,
eval_digest = the pack's fixture digest, threshold = SIGNAL_FLOOR).
snapshot() auto-parents, so each snapshot is a distinct root and the
lineage grows by one per run. run_finetuning gains a third dispatch
mode — shard-chain (gated on a task's selfmodel_shard key) — via
_chain_finetuning_measure: reads the two most-recent snapshots
(latest() = child, its parent_selfmodel_root = parent) and measures
improvement on target_capability between them. This is the real
lineage replacing Phase-1a's synthetic parent->child pairs; the
chained delta reflects genuine cross-run drift (0.0 today — the
embedded packs are at ceiling). Operator pack
bench/fixtures/5f/finetuning-shardchain-v1.jsonl (6 tasks) + make
bench-5f-finetuning-shardchain; not in `make bench-5f`, `make test`,
or a fresh checkout (a missing/too-short chain fails honestly). The
real chain shard was bootstrapped 2-deep on 2026-05-11; make
chain-check-shards reports 0 breaks on it (and all other shards).
10 tests in tests/test_selfmodel_chain.py.

Full suite: 2311 passed, 28 skipped.

tests/test_bench_batteries.py

@@ -547,6 +547,55 @@ def test_5f_live_feedback_chain_audit_chain_intact():
             pass
 
 
+# --- 5F §10.13 — feedback latency / real-workload efficiency --------
+
+
+def test_5f_persisted_cost_helper():
+    """_persisted_cost = audit-row count + body bytes / 1e6 (the real
+    footprint a live chain wrote, not the count of requested ops)."""
+    from bench.batteries.b_5f import _persisted_cost
+
+    assert _persisted_cost({}) == 0.0
+    state = {
+        "audit_event_types": ["a", "b", "c"],
+        "audit_event_bodies": ['{"x":1}', '{"y":2}', ""],  # 7 + 7 + 0 bytes
+    }
+    assert _persisted_cost(state) == pytest.approx(3.0 + 14 / 1e6)
+
+
+def test_5f_feedback_loop_embedded_reports_no_latency():
+    res = b_5f.run_feedback_loop(F5F / "feedback-loop-v1.jsonl")
+    assert res.metrics["feedback_latency_mean_seconds"] == 0.0
+    assert res.metrics["feedback_live_task_count"] == 0.0
+    for t in res.per_task:
+        assert t.detail.get("feedback_latency_seconds") is None
+        # Embedded cost-proxy stays len(chain) → integer-ratio efficiency.
+        assert "persisted_cost" not in t.detail
+
+
+def test_5f_feedback_loop_live_reports_latency_and_persisted_cost():
+    res = b_5f.run_feedback_loop(F5F / "feedback-loop-live-v1.jsonl")
+    assert res.metrics["feedback_live_task_count"] == 50.0  # Phase 1d
+    assert res.metrics["feedback_latency_mean_seconds"] > 0.0
+    for t in res.per_task:
+        if not t.passed:
+            continue
+        lat = t.detail["feedback_latency_seconds"]
+        assert isinstance(lat, float) and lat >= 0.0
+        # Real footprint: every live chain has >= 1 append_audit op.
+        assert t.detail["persisted_audit_rows"] >= 1
+        assert t.detail["persisted_cost"] >= 1.0
+        # feedback_efficiency now = downstream_effect / persisted_cost,
+        # not / len(chain). downstream_effect is 1.0 when the chain
+        # integrated, else 0.0 (an expected-"fail" task still passes).
+        if t.detail["integrated"]:
+            assert t.detail["feedback_efficiency"] == pytest.approx(
+                1.0 / t.detail["persisted_cost"]
+            )
+        else:
+            assert t.detail["feedback_efficiency"] == 0.0
+
+
 # --- 5F Phase 1b.2 — Function / Finetuning / Falsification live -
 
 

tests/test_fivef_threshold_calibration.py

@@ -0,0 +1,86 @@
+"""Tests for the #000025 §10.14 ForkScore threshold-calibration handoff
+script (``bench/scripts/fivef_threshold_calibration.py``).
+
+The script is pure measurement over the canonical 5S/5T/5F packs +
+`fork_score`, so its markdown report is deterministic given fixed
+fixtures + code. These tests pin the structural anchors and the
+floor-constant sanity verdicts so the handoff doesn't drift silently.
+
+The live 5F packs (``*-live-v1.jsonl``) spin up temp shards and add
+~15s per run; the calibration *logic* (granularity, fork-score sanity)
+doesn't depend on them, so the tests stub ``_LIVE_PACKS`` to empty.
+The live runners are exercised by ``tests/test_bench_batteries.py``
+and ``make bench-5f-live``.
+"""
+from __future__ import annotations
+
+import pytest
+
+from bench.scripts import fivef_threshold_calibration as calib
+
+
+@pytest.fixture(scope="module", autouse=True)
+def _no_live_packs():
+    saved = calib._LIVE_PACKS
+    calib._LIVE_PACKS = {}
+    try:
+        yield
+    finally:
+        calib._LIVE_PACKS = saved
+
+
+@pytest.fixture(scope="module")
+def report() -> str:
+    return calib.build_report()
+
+
+def test_report_is_deterministic():
+    a = calib.build_report()
+    b = calib.build_report()
+
+    def _strip_ts(s: str) -> str:
+        return "\n".join(ln for ln in s.splitlines() if not ln.startswith("**Date:**"))
+
+    assert _strip_ts(a) == _strip_ts(b)
+
+
+def test_report_has_structural_anchors(report: str):
+    assert report.startswith("# 5S/5T/5F → v8 ForkScore threshold-calibration handoff")
+    assert "#000025 §10.14" in report
+    assert "## 1. Baseline rates + granularity" in report
+    assert "## 2. Identity-fork verdict (no change)" in report
+    assert "## 3. Floor-constant sanity checks" in report
+    assert "## 4. Recommendation for #000012" in report
+    for sub in (
+        "syntax", "semantics", "syllogism", "synthesis", "semiotics",
+        "transfer-learning", "triangulation", "truthtables", "transitivity", "time",
+        "function", "finetuning", "falsification", "formulate", "feedback-loop",
+    ):
+        assert f"| {sub} |" in report
+
+
+def test_identity_fork_is_marginal(report: str):
+    # Packs are at ceiling at HEAD → every Δ-rate term is 0 → MARGINAL.
+    assert "`fork_score(parent, parent)` → **MARGINAL (score +0.0000)**" in report
+
+
+def test_floor_sanity_verdicts_present(report: str):
+    assert "child recovers **all 5** 5F subs by +0.06 | ACCEPT" in report
+    assert "child recovers **only 1** 5F sub by +0.06 | MARGINAL" in report
+    assert "child drops one 5F sub by −0.05 | REJECT" in report
+    assert "REGRESSION_5F:5f/falsification" in report
+
+
+def test_recommendation_keeps_floors_and_flags_dilution(report: str):
+    assert "Keep `SIGNAL_FLOOR = 0.05` and `HARD_REGRESSION_FLOOR = 0.05`" in report
+    assert "5× averaging dilution" in report
+    assert "Ceiling saturation" in report
+    assert "No constant change shipped" in report
+
+
+def test_cli_main_writes_report(tmp_path):
+    out = tmp_path / "calib.md"
+    rc = calib.main(["--out", str(out)])
+    assert rc == 0
+    text = out.read_text(encoding="utf-8")
+    assert text.startswith("# 5S/5T/5F → v8 ForkScore threshold-calibration handoff")

tests/test_selfmodel_chain.py

@@ -0,0 +1,134 @@
+"""Tests for the #000025 §10.11 SelfModel snapshot-chain — the
+persistent lineage that replaces Phase-1a's synthetic parent→child
+pairs in the Finetuning sub-battery.
+
+Two surfaces:
+  - ``bench/scripts/selfmodel_chain_snapshot.py`` — the bootstrapper
+    that appends one chained snapshot per run (capability claims =
+    the 5S/5T/5F battery rates).
+  - ``bench.batteries.b_5f._chain_finetuning_measure`` + the
+    ``run_finetuning`` "shard-chain" dispatch — reads the two latest
+    snapshots and measures improvement on ``target_capability``.
+
+All against temp shards (``tmp_path`` / ``tmp_path_factory``) — never
+the real ``~/.arborist/shards/selfmodel-chain.db``.
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from bench.batteries import b_5f
+from bench.scripts.selfmodel_chain_snapshot import append_snapshot
+
+_F5F = Path(__file__).parent.parent / "bench" / "fixtures" / "5f"
+_SHARDCHAIN_PACK = _F5F / "finetuning-shardchain-v1.jsonl"
+
+
+@pytest.fixture(scope="module")
+def chain_shard(tmp_path_factory) -> Path:
+    """A persistent 2-deep SelfModel chain on a temp shard."""
+    shard = tmp_path_factory.mktemp("smchain") / "chain.db"
+    s1 = append_snapshot(shard, ts=1_700_000_000)
+    s2 = append_snapshot(shard, ts=1_700_000_001)
+    assert s1["depth"] == 1 and s1["parent_root"] is None
+    assert s2["depth"] == 2 and s2["parent_root"] == s1["root"]
+    return shard
+
+
+# ---- bootstrapper -------------------------------------------------------
+
+
+def test_bootstrapper_grows_chain_one_per_run(tmp_path):
+    shard = tmp_path / "c.db"
+    a = append_snapshot(shard, ts=1_700_000_000)
+    assert a["depth"] == 1
+    assert a["parent_root"] is None
+    # 15 canonical sub-batteries → 15 capability claims.
+    assert len(a["claims"]) == 15
+    assert all(v == 1.0 for v in a["claims"].values())  # packs at ceiling
+    b = append_snapshot(shard, ts=1_700_000_001)
+    assert b["depth"] == 2
+    assert b["parent_root"] == a["root"]
+    assert b["root"] != a["root"]  # distinct parent ⇒ distinct root
+
+
+def test_bootstrapper_claims_carry_real_metric_names(chain_shard: Path):
+    a = append_snapshot(chain_shard, ts=1_700_000_002)  # depth 3
+    assert a["depth"] == 3
+    assert {"5S-syntax", "5T-time", "5F-feedback-loop"} <= set(a["claims"])
+
+
+# ---- _chain_finetuning_measure -----------------------------------------
+
+
+def test_chain_measure_reads_two_latest(chain_shard: Path):
+    out = b_5f._chain_finetuning_measure(
+        {"target_capability": "5F-feedback-loop", "selfmodel_shard": str(chain_shard)}
+    )
+    assert out["parent_measured"] == 1.0
+    assert out["child_measured"] == 1.0
+    assert out["parent_root"] != out["child_root"]
+
+
+def test_chain_measure_env_var_overrides_fixture(chain_shard: Path, monkeypatch):
+    monkeypatch.setenv("ARBORIST_SELFMODEL_CHAIN_DB", str(chain_shard))
+    out = b_5f._chain_finetuning_measure(
+        {"target_capability": "5S-syntax", "selfmodel_shard": "/nonexistent/ignored.db"}
+    )
+    assert out["child_measured"] == 1.0
+
+
+def test_chain_measure_absent_shard_raises(tmp_path):
+    with pytest.raises(FileNotFoundError, match="chain shard not found"):
+        b_5f._chain_finetuning_measure(
+            {"target_capability": "5S-syntax", "selfmodel_shard": str(tmp_path / "nope.db")}
+        )
+
+
+def test_chain_measure_single_snapshot_raises(tmp_path):
+    shard = tmp_path / "one.db"
+    append_snapshot(shard, ts=1_700_000_000)  # depth 1
+    with pytest.raises(ValueError, match="only one snapshot"):
+        b_5f._chain_finetuning_measure(
+            {"target_capability": "5S-syntax", "selfmodel_shard": str(shard)}
+        )
+
+
+def test_chain_measure_unknown_metric_raises(chain_shard: Path):
+    with pytest.raises(ValueError, match="no claim for"):
+        b_5f._chain_finetuning_measure(
+            {"target_capability": "NOT-A-REAL-METRIC", "selfmodel_shard": str(chain_shard)}
+        )
+
+
+# ---- run_finetuning shard-chain dispatch -------------------------------
+
+
+def test_run_finetuning_shardchain_pack_passes(chain_shard: Path, monkeypatch):
+    monkeypatch.setenv("ARBORIST_SELFMODEL_CHAIN_DB", str(chain_shard))
+    res = b_5f.run_finetuning(_SHARDCHAIN_PACK)
+    assert res.pass_count == 6
+    assert res.fail_count == 0
+    for t in res.per_task:
+        assert t.detail["source"] == "shard-chain"
+        assert t.detail["improvement"] == 0.0  # packs at ceiling ⇒ no drift
+        assert t.detail["chain_parent_root"] != t.detail["chain_child_root"]
+
+
+def test_run_finetuning_shardchain_pack_fails_honestly_without_chain(tmp_path, monkeypatch):
+    monkeypatch.setenv("ARBORIST_SELFMODEL_CHAIN_DB", str(tmp_path / "absent.db"))
+    res = b_5f.run_finetuning(_SHARDCHAIN_PACK)
+    assert res.pass_count == 0
+    assert res.fail_count == 6
+    assert "FileNotFoundError" in res.per_task[0].detail["reason"]
+
+
+def test_embedded_and_live_finetuning_unaffected():
+    """The shard-chain dispatch is gated on ``selfmodel_shard``; the
+    embedded + live packs still route through their original paths."""
+    emb = b_5f.run_finetuning(_F5F / "finetuning-v1.jsonl")
+    assert emb.pass_count == 50
+    for t in emb.per_task:
+        assert t.detail["source"] == "embedded"