russell@unturf.com
d78dccc8ed
Closes the three open Phase-1b items of #000025; every §10 closure criterion is now met, so the ticket flips to closed. §10.14 — ForkScore threshold-calibration handoff to #000012. bench/scripts/fivef_threshold_calibration.py (make bench-5f-threshold- calibration) runs the canonical 5S/5T/5F packs + the 5F live packs and reports baseline rates, observability granularity (1/n), and fork_score verdicts on the parent vs synthetic child perturbations → bench/results/5f-threshold-calibration-2026-05-11.md. Findings written into ticket-000012 §8: keep SIGNAL_FLOOR / HARD_REGRESSION_FLOOR at 0.05; the small 5S packs (syntax n=10, semantics n=8) are coarser than the floors so any regression there trips hard-reject (intended zero- tolerance); the 5x averaging dilution in _delta_*; ceiling saturation (every pack at 1.0 -> delta-rate terms <= 0). No constant change shipped. 6 tests in tests/test_fivef_threshold_calibration.py. §10.13 — feedback latency / efficiency on real workload. run_feedback_loop now computes feedback_latency (listed in §5.5 since Phase 1a, never implemented) — wall-clock seconds to apply a live chain against its temp shard, surfaced per-task (feedback_latency_seconds) + battery (feedback_latency_mean_seconds, feedback_live_task_count). For live chains feedback_efficiency's cost denominator switched from len(chain) (count of requested ops) to the persisted footprint _persisted_cost = audit-event rows the chain actually wrote + their body bytes / 1e6. Embedded chains keep len(chain) and report feedback_latency_seconds = None. Latency is a wall-clock field (run-to-run variable, like BatteryResult.timestamp) and is not a fork_score input. 3 tests in tests/test_bench_batteries.py. §10.11 — real selfmodel finetuning chains. bench/scripts/selfmodel_chain_snapshot.py (make bench-5f-selfmodel- snapshot) appends one chained SelfModel snapshot per run to a persistent shard (~/.arborist/shards/selfmodel-chain.db, override via ARBORIST_SELFMODEL_CHAIN_DB) with one CapabilityClaim per sub-battery (metric = "5S-syntax" etc., measured_value = that pack's rate, eval_digest = the pack's fixture digest, threshold = SIGNAL_FLOOR). snapshot() auto-parents, so each snapshot is a distinct root and the lineage grows by one per run. run_finetuning gains a third dispatch mode — shard-chain (gated on a task's selfmodel_shard key) — via _chain_finetuning_measure: reads the two most-recent snapshots (latest() = child, its parent_selfmodel_root = parent) and measures improvement on target_capability between them. This is the real lineage replacing Phase-1a's synthetic parent->child pairs; the chained delta reflects genuine cross-run drift (0.0 today — the embedded packs are at ceiling). Operator pack bench/fixtures/5f/finetuning-shardchain-v1.jsonl (6 tasks) + make bench-5f-finetuning-shardchain; not in `make bench-5f`, `make test`, or a fresh checkout (a missing/too-short chain fails honestly). The real chain shard was bootstrapped 2-deep on 2026-05-11; make chain-check-shards reports 0 breaks on it (and all other shards). 10 tests in tests/test_selfmodel_chain.py. Full suite: 2311 passed, 28 skipped.
86 lines
3.1 KiB
Python
86 lines
3.1 KiB
Python
"""Tests for the #000025 §10.14 ForkScore threshold-calibration handoff
|
||
script (``bench/scripts/fivef_threshold_calibration.py``).
|
||
|
||
The script is pure measurement over the canonical 5S/5T/5F packs +
|
||
`fork_score`, so its markdown report is deterministic given fixed
|
||
fixtures + code. These tests pin the structural anchors and the
|
||
floor-constant sanity verdicts so the handoff doesn't drift silently.
|
||
|
||
The live 5F packs (``*-live-v1.jsonl``) spin up temp shards and add
|
||
~15s per run; the calibration *logic* (granularity, fork-score sanity)
|
||
doesn't depend on them, so the tests stub ``_LIVE_PACKS`` to empty.
|
||
The live runners are exercised by ``tests/test_bench_batteries.py``
|
||
and ``make bench-5f-live``.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import pytest
|
||
|
||
from bench.scripts import fivef_threshold_calibration as calib
|
||
|
||
|
||
@pytest.fixture(scope="module", autouse=True)
|
||
def _no_live_packs():
|
||
saved = calib._LIVE_PACKS
|
||
calib._LIVE_PACKS = {}
|
||
try:
|
||
yield
|
||
finally:
|
||
calib._LIVE_PACKS = saved
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def report() -> str:
|
||
return calib.build_report()
|
||
|
||
|
||
def test_report_is_deterministic():
|
||
a = calib.build_report()
|
||
b = calib.build_report()
|
||
|
||
def _strip_ts(s: str) -> str:
|
||
return "\n".join(ln for ln in s.splitlines() if not ln.startswith("**Date:**"))
|
||
|
||
assert _strip_ts(a) == _strip_ts(b)
|
||
|
||
|
||
def test_report_has_structural_anchors(report: str):
|
||
assert report.startswith("# 5S/5T/5F → v8 ForkScore threshold-calibration handoff")
|
||
assert "#000025 §10.14" in report
|
||
assert "## 1. Baseline rates + granularity" in report
|
||
assert "## 2. Identity-fork verdict (no change)" in report
|
||
assert "## 3. Floor-constant sanity checks" in report
|
||
assert "## 4. Recommendation for #000012" in report
|
||
for sub in (
|
||
"syntax", "semantics", "syllogism", "synthesis", "semiotics",
|
||
"transfer-learning", "triangulation", "truthtables", "transitivity", "time",
|
||
"function", "finetuning", "falsification", "formulate", "feedback-loop",
|
||
):
|
||
assert f"| {sub} |" in report
|
||
|
||
|
||
def test_identity_fork_is_marginal(report: str):
|
||
# Packs are at ceiling at HEAD → every Δ-rate term is 0 → MARGINAL.
|
||
assert "`fork_score(parent, parent)` → **MARGINAL (score +0.0000)**" in report
|
||
|
||
|
||
def test_floor_sanity_verdicts_present(report: str):
|
||
assert "child recovers **all 5** 5F subs by +0.06 | ACCEPT" in report
|
||
assert "child recovers **only 1** 5F sub by +0.06 | MARGINAL" in report
|
||
assert "child drops one 5F sub by −0.05 | REJECT" in report
|
||
assert "REGRESSION_5F:5f/falsification" in report
|
||
|
||
|
||
def test_recommendation_keeps_floors_and_flags_dilution(report: str):
|
||
assert "Keep `SIGNAL_FLOOR = 0.05` and `HARD_REGRESSION_FLOOR = 0.05`" in report
|
||
assert "5× averaging dilution" in report
|
||
assert "Ceiling saturation" in report
|
||
assert "No constant change shipped" in report
|
||
|
||
|
||
def test_cli_main_writes_report(tmp_path):
|
||
out = tmp_path / "calib.md"
|
||
rc = calib.main(["--out", str(out)])
|
||
assert rc == 0
|
||
text = out.read_text(encoding="utf-8")
|
||
assert text.startswith("# 5S/5T/5F → v8 ForkScore threshold-calibration handoff")
|