#000025 §10.11 + §10.13 + §10.14 — close the 5F battery
Closes the three open Phase-1b items of #000025; every §10 closure criterion is now met, so the ticket flips to closed. §10.14 — ForkScore threshold-calibration handoff to #000012. bench/scripts/fivef_threshold_calibration.py (make bench-5f-threshold- calibration) runs the canonical 5S/5T/5F packs + the 5F live packs and reports baseline rates, observability granularity (1/n), and fork_score verdicts on the parent vs synthetic child perturbations → bench/results/5f-threshold-calibration-2026-05-11.md. Findings written into ticket-000012 §8: keep SIGNAL_FLOOR / HARD_REGRESSION_FLOOR at 0.05; the small 5S packs (syntax n=10, semantics n=8) are coarser than the floors so any regression there trips hard-reject (intended zero- tolerance); the 5x averaging dilution in _delta_*; ceiling saturation (every pack at 1.0 -> delta-rate terms <= 0). No constant change shipped. 6 tests in tests/test_fivef_threshold_calibration.py. §10.13 — feedback latency / efficiency on real workload. run_feedback_loop now computes feedback_latency (listed in §5.5 since Phase 1a, never implemented) — wall-clock seconds to apply a live chain against its temp shard, surfaced per-task (feedback_latency_seconds) + battery (feedback_latency_mean_seconds, feedback_live_task_count). For live chains feedback_efficiency's cost denominator switched from len(chain) (count of requested ops) to the persisted footprint _persisted_cost = audit-event rows the chain actually wrote + their body bytes / 1e6. Embedded chains keep len(chain) and report feedback_latency_seconds = None. Latency is a wall-clock field (run-to-run variable, like BatteryResult.timestamp) and is not a fork_score input. 3 tests in tests/test_bench_batteries.py. §10.11 — real selfmodel finetuning chains. bench/scripts/selfmodel_chain_snapshot.py (make bench-5f-selfmodel- snapshot) appends one chained SelfModel snapshot per run to a persistent shard (~/.arborist/shards/selfmodel-chain.db, override via ARBORIST_SELFMODEL_CHAIN_DB) with one CapabilityClaim per sub-battery (metric = "5S-syntax" etc., measured_value = that pack's rate, eval_digest = the pack's fixture digest, threshold = SIGNAL_FLOOR). snapshot() auto-parents, so each snapshot is a distinct root and the lineage grows by one per run. run_finetuning gains a third dispatch mode — shard-chain (gated on a task's selfmodel_shard key) — via _chain_finetuning_measure: reads the two most-recent snapshots (latest() = child, its parent_selfmodel_root = parent) and measures improvement on target_capability between them. This is the real lineage replacing Phase-1a's synthetic parent->child pairs; the chained delta reflects genuine cross-run drift (0.0 today — the embedded packs are at ceiling). Operator pack bench/fixtures/5f/finetuning-shardchain-v1.jsonl (6 tasks) + make bench-5f-finetuning-shardchain; not in `make bench-5f`, `make test`, or a fresh checkout (a missing/too-short chain fails honestly). The real chain shard was bootstrapped 2-deep on 2026-05-11; make chain-check-shards reports 0 breaks on it (and all other shards). 10 tests in tests/test_selfmodel_chain.py. Full suite: 2311 passed, 28 skipped.
This commit is contained in:
parent
1dc47e01b3
commit
d78dccc8ed
12 changed files with 1169 additions and 18 deletions
349
bench/scripts/fivef_threshold_calibration.py
Normal file
349
bench/scripts/fivef_threshold_calibration.py
Normal file
|
|
@ -0,0 +1,349 @@
|
|||
"""5S/5T/5F → v8 ForkScore threshold-calibration handoff (#000025 §10.14).
|
||||
|
||||
Closure deliverable for ticket #000025 §10.14 ("threshold calibration for
|
||||
v8 selection acceptance handed off to #000012"). Pure measurement — runs
|
||||
the canonical 5S / 5T / 5F sub-batteries (embedded packs + the 5F live
|
||||
packs), builds the parent/child metrics bundle the ForkScore reads
|
||||
(:func:`arborist.substrate.fork_score.bench_result_to_metrics` shape),
|
||||
and reports:
|
||||
|
||||
1. Per-pack baseline rate (the ``_BATTERY_RATE_KEYS`` metric the scorer
|
||||
consumes) + fixture count + 1/n observability granularity.
|
||||
2. The identity-fork verdict — ``fork_score(parent, parent)`` — which
|
||||
tells #000012 what "no change" scores to (it's MARGINAL, not
|
||||
ACCEPT, because every Δ-rate term is 0).
|
||||
3. Synthetic-perturbation verdicts: a single +``SIGNAL_FLOOR`` bump on
|
||||
one 5F sub-battery → ACCEPT; a single −``HARD_REGRESSION_FLOOR``
|
||||
drop → REJECT. Confirms the floor constants do what the docstrings
|
||||
claim.
|
||||
4. A recommendation block: are ``SIGNAL_FLOOR`` / ``HARD_REGRESSION_FLOOR``
|
||||
(both 0.05 today) appropriate given the observed pack granularity?
|
||||
|
||||
No mutation, no LLM call, no schema / governance-hash / canonicalization
|
||||
change. The markdown report is the deliverable; #000012 cites it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import statistics
|
||||
import sys
|
||||
from dataclasses import asdict
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# bench/ is on sys.path when run as a module (python -m); when run as a
|
||||
# script, add the repo root so `import bench...` and `import arborist...`
|
||||
# resolve.
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(_REPO_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(_REPO_ROOT))
|
||||
|
||||
from bench.batteries import b_5f, b_5s, b_5t # noqa: E402
|
||||
from bench.batteries.base import BatteryResult # noqa: E402
|
||||
from arborist.substrate.fork_score import ( # noqa: E402
|
||||
HARD_REGRESSION_FLOOR,
|
||||
SIGNAL_FLOOR,
|
||||
_BATTERY_RATE_KEYS,
|
||||
bench_result_to_metrics,
|
||||
fork_score,
|
||||
)
|
||||
|
||||
_FIX = _REPO_ROOT / "bench" / "fixtures"
|
||||
|
||||
# The canonical sub-battery → fixture-pack map the ForkScore Δ-rate terms
|
||||
# read. Mirrors bench.batteries.runner._DEFAULT_FIXTURES but restricted
|
||||
# to the keys present in fork_score._BATTERY_RATE_KEYS (so legacy 5t
|
||||
# "transfer" is excluded — the scorer drops it). The "live" 5F packs are
|
||||
# included as a separate column because their runners exercise real
|
||||
# arborist surfaces on temp shards (Phase 1b.2).
|
||||
_EMBEDDED_PACKS: dict[tuple[str, str], Path] = {
|
||||
("5s", "syntax"): _FIX / "5s" / "syntax-v1.jsonl",
|
||||
("5s", "semantics"): _FIX / "5s" / "semantics-v1.jsonl",
|
||||
("5s", "syllogism"): _FIX / "5s" / "syllogism-v1.jsonl",
|
||||
("5s", "synthesis"): _FIX / "5s" / "synthesis-v1.jsonl",
|
||||
("5s", "semiotics"): _FIX / "5s" / "semiotics-v1.jsonl",
|
||||
("5t", "transfer-learning"): _FIX / "5t" / "transfer-learning-v2.jsonl",
|
||||
("5t", "triangulation"): _FIX / "5t" / "triangulation-v1.jsonl",
|
||||
("5t", "truthtables"): _FIX / "5t" / "truthtables-v1.jsonl",
|
||||
("5t", "transitivity"): _FIX / "5t" / "transitivity-v1.jsonl",
|
||||
("5t", "time"): _FIX / "5t" / "time-v1.jsonl",
|
||||
("5f", "function"): _FIX / "5f" / "function-v1.jsonl",
|
||||
("5f", "finetuning"): _FIX / "5f" / "finetuning-v1.jsonl",
|
||||
("5f", "falsification"): _FIX / "5f" / "falsification-v1.jsonl",
|
||||
("5f", "formulate"): _FIX / "5f" / "formulate-v1.jsonl",
|
||||
("5f", "feedback-loop"): _FIX / "5f" / "feedback-loop-v1.jsonl",
|
||||
}
|
||||
|
||||
_LIVE_PACKS: dict[tuple[str, str], Path] = {
|
||||
("5f", "function"): _FIX / "5f" / "function-live-v1.jsonl",
|
||||
("5f", "finetuning"): _FIX / "5f" / "finetuning-live-v1.jsonl",
|
||||
("5f", "falsification"): _FIX / "5f" / "falsification-live-v1.jsonl",
|
||||
("5f", "formulate"): _FIX / "5f" / "formulate-live-v1.jsonl",
|
||||
("5f", "feedback-loop"): _FIX / "5f" / "feedback-loop-live-v1.jsonl",
|
||||
}
|
||||
|
||||
_SUB_RUNNERS = {"5s": b_5s.SUB_BATTERIES, "5t": b_5t.SUB_BATTERIES, "5f": b_5f.SUB_BATTERIES}
|
||||
|
||||
|
||||
def _run_pack(battery: str, sub: str, path: Path) -> BatteryResult:
|
||||
return _SUB_RUNNERS[battery][sub](path)
|
||||
|
||||
|
||||
def _results_payload(packs: dict[tuple[str, str], Path]) -> dict:
|
||||
"""Run every pack and shape the output like bench.batteries.runner."""
|
||||
results = []
|
||||
for (battery, sub), path in packs.items():
|
||||
r = _run_pack(battery, sub, path)
|
||||
results.append(asdict(r))
|
||||
return {"schema_version": "bench-result-v1", "results": results}
|
||||
|
||||
|
||||
def _rate_of(metrics: dict, battery: str, sub: str) -> float | None:
|
||||
key = _BATTERY_RATE_KEYS.get(battery, {}).get(sub)
|
||||
if key is None:
|
||||
return None
|
||||
v = metrics.get(key)
|
||||
return float(v) if v is not None else None
|
||||
|
||||
|
||||
def _deepcopy_bundle(metrics_bundle: dict) -> dict:
|
||||
return {b: {s: dict(m) for s, m in subs.items()} for b, subs in metrics_bundle.items()}
|
||||
|
||||
|
||||
def _set_rate(bundle: dict, battery: str, sub: str, value: float) -> None:
|
||||
key = _BATTERY_RATE_KEYS[battery][sub]
|
||||
bundle.setdefault(battery, {}).setdefault(sub, {})[key] = max(0.0, min(1.0, value))
|
||||
|
||||
|
||||
def _shift_rate(bundle: dict, battery: str, sub: str, delta: float) -> None:
|
||||
key = _BATTERY_RATE_KEYS[battery][sub]
|
||||
base = float(bundle.get(battery, {}).get(sub, {}).get(key, 0.0))
|
||||
_set_rate(bundle, battery, sub, base + delta)
|
||||
|
||||
|
||||
def _degraded_parent(parent: dict, knock_to: float = 0.90) -> dict:
|
||||
"""A synthetic parent with every 5F sub-battery rate knocked down to
|
||||
``knock_to`` — gives a child headroom to improve. The production
|
||||
parent is at ceiling (rate 1.0 on every pack), so threshold
|
||||
mechanics can only be exercised against a below-ceiling baseline."""
|
||||
out = _deepcopy_bundle(parent)
|
||||
for sub in _BATTERY_RATE_KEYS["5f"]:
|
||||
_set_rate(out, "5f", sub, knock_to)
|
||||
return out
|
||||
|
||||
|
||||
def _fmt_verdict(sf) -> str:
|
||||
flags = f" — flags: {', '.join(sf.flags)}" if sf.flags else ""
|
||||
return f"{sf.verdict} (score {sf.score:+.4f}){flags}"
|
||||
|
||||
|
||||
def build_report() -> str:
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
embedded = _results_payload(_EMBEDDED_PACKS)
|
||||
live = _results_payload(_LIVE_PACKS)
|
||||
parent = bench_result_to_metrics(embedded)
|
||||
live_metrics = bench_result_to_metrics(live)
|
||||
|
||||
# ---- per-pack table rows -------------------------------------------------
|
||||
rows = []
|
||||
granularities: list[float] = []
|
||||
for r in embedded["results"]:
|
||||
battery, sub = r["battery"], r["sub_battery"]
|
||||
n = r["pass_count"] + r["fail_count"]
|
||||
rate = _rate_of(r["metrics"], battery, sub)
|
||||
gran = (1.0 / n) if n else float("nan")
|
||||
if n:
|
||||
granularities.append(gran)
|
||||
live_r = next(
|
||||
(lr for lr in live["results"] if lr["battery"] == battery and lr["sub_battery"] == sub),
|
||||
None,
|
||||
)
|
||||
live_rate = _rate_of(live_r["metrics"], battery, sub) if live_r else None
|
||||
rows.append((battery, sub, n, rate, gran, live_rate))
|
||||
|
||||
coarsest = max(granularities) if granularities else float("nan")
|
||||
finest = min(granularities) if granularities else float("nan")
|
||||
|
||||
# ---- fork-score sanity verdicts ------------------------------------------
|
||||
# The production parent is at ceiling (rate 1.0 everywhere), so a
|
||||
# "+SIGNAL_FLOOR" child clamps right back to 1.0 and shows nothing.
|
||||
# Exercise the threshold mechanics against a synthetic below-ceiling
|
||||
# parent (every 5F sub knocked to 0.90).
|
||||
identity = fork_score(parent, parent)
|
||||
knock = 0.90
|
||||
degraded = _degraded_parent(parent, knock)
|
||||
|
||||
# Child recovers EVERY 5F sub uniformly by 0.06 (> SIGNAL_FLOOR each)
|
||||
# → mean Δ5F = 0.06 → score = γ·0.06 = 0.06 ≥ SIGNAL_FLOOR → ACCEPT.
|
||||
child_uniform = _deepcopy_bundle(degraded)
|
||||
for sub in _BATTERY_RATE_KEYS["5f"]:
|
||||
_shift_rate(child_uniform, "5f", sub, 0.06)
|
||||
accept_uniform = fork_score(degraded, child_uniform)
|
||||
|
||||
# Child recovers only ONE 5F sub by 0.06 → mean Δ5F = 0.06/5 = 0.012
|
||||
# → score 0.012 < SIGNAL_FLOOR → MARGINAL. This is the 5× averaging
|
||||
# dilution: a single-sub gain is worth a fifth of its face value.
|
||||
child_single = _deepcopy_bundle(degraded)
|
||||
_shift_rate(child_single, "5f", "function", 0.06)
|
||||
marginal_single = fork_score(degraded, child_single)
|
||||
|
||||
# Child drops one 5F sub by HARD_REGRESSION_FLOOR from the degraded
|
||||
# parent → hard regression → REJECT regardless of the other terms.
|
||||
child_regress = _deepcopy_bundle(degraded)
|
||||
_shift_rate(child_regress, "5f", "falsification", -HARD_REGRESSION_FLOOR)
|
||||
reject_regress = fork_score(degraded, child_regress)
|
||||
|
||||
# ---- emit ----------------------------------------------------------------
|
||||
out: list[str] = []
|
||||
out.append("# 5S/5T/5F → v8 ForkScore threshold-calibration handoff")
|
||||
out.append("")
|
||||
out.append(f"**Date:** {now}")
|
||||
out.append("**Ticket:** #000025 §10.14 (closure deliverable) — handoff to #000012.")
|
||||
out.append(
|
||||
"**Method:** ran the canonical 5S/5T/5F sub-batteries (the packs "
|
||||
"`fork_score._BATTERY_RATE_KEYS` reads) plus the 5F live packs; "
|
||||
"computed baseline rates, observability granularity (1/n), and "
|
||||
"ran `fork_score` on the parent vs three synthetic child "
|
||||
"perturbations. Pure measurement."
|
||||
)
|
||||
out.append("")
|
||||
out.append("## 1. Baseline rates + granularity")
|
||||
out.append("")
|
||||
out.append(
|
||||
"Each row's *rate* is the single metric `fork_score` consumes for "
|
||||
"that sub-battery (`_BATTERY_RATE_KEYS`). *Granularity* = 1/n — "
|
||||
"the smallest rate change a single fixture flip can produce, i.e. "
|
||||
"the finest Δ the scorer could ever observe on that pack."
|
||||
)
|
||||
out.append("")
|
||||
out.append("| battery | sub-battery | fixtures (n) | embedded rate | live rate | granularity (1/n) |")
|
||||
out.append("|---|---|---|---|---|---|")
|
||||
for battery, sub, n, rate, gran, live_rate in rows:
|
||||
rate_s = f"{rate:.4f}" if rate is not None else "—"
|
||||
live_s = f"{live_rate:.4f}" if live_rate is not None else "—"
|
||||
out.append(f"| {battery} | {sub} | {n} | {rate_s} | {live_s} | {gran:.4f} ({gran*100:.1f}pp) |")
|
||||
out.append("")
|
||||
# Per-battery mean rate.
|
||||
for battery in ("5s", "5t", "5f"):
|
||||
vals = [rate for b, s, n, rate, g, lr in rows if b == battery and rate is not None]
|
||||
if vals:
|
||||
out.append(f"- **{battery.upper()} mean baseline rate:** {statistics.fmean(vals):.4f}")
|
||||
out.append(f"- **Coarsest pack granularity:** {coarsest:.4f} ({coarsest*100:.1f}pp)")
|
||||
out.append(f"- **Finest pack granularity:** {finest:.4f} ({finest*100:.1f}pp)")
|
||||
out.append("")
|
||||
out.append("## 2. Identity-fork verdict (no change)")
|
||||
out.append("")
|
||||
out.append(f"`fork_score(parent, parent)` → **{_fmt_verdict(identity)}**")
|
||||
out.append("")
|
||||
out.append(
|
||||
"Every 5S/5T/5F pack is at ceiling (rate 1.0) at HEAD, so every "
|
||||
"Δ-rate term is exactly 0 and the identity fork lands in the "
|
||||
"`[0, SIGNAL_FLOOR)` band → **MARGINAL**, not ACCEPT. Takeaway for "
|
||||
"#000012: an unchanged child is *not* auto-accepted; positive "
|
||||
"score has to come from a Δ-rate gain (needs harder fixtures, see "
|
||||
"§4), an efficiency-bonus increase (`adaptation_efficiency` / "
|
||||
"`feedback_efficiency`), or one of the non-bench terms "
|
||||
"(`selfmodel_calibration_gain`, `audit_completeness`, …)."
|
||||
)
|
||||
out.append("")
|
||||
out.append("## 3. Floor-constant sanity checks")
|
||||
out.append("")
|
||||
out.append(
|
||||
f"`SIGNAL_FLOOR = {SIGNAL_FLOOR}` · `HARD_REGRESSION_FLOOR = "
|
||||
f"{HARD_REGRESSION_FLOOR}` (current `arborist/substrate/fork_score.py`). "
|
||||
f"The production parent is at ceiling, so these are exercised against "
|
||||
f"a synthetic *degraded parent* — every 5F sub-battery rate knocked "
|
||||
f"to {knock:.2f}, giving a child headroom to improve."
|
||||
)
|
||||
out.append("")
|
||||
out.append("| scenario | verdict |")
|
||||
out.append("|---|---|")
|
||||
out.append(f"| identity — `fork_score(HEAD, HEAD)` | {_fmt_verdict(identity)} |")
|
||||
out.append(f"| degraded parent → child recovers **all 5** 5F subs by +0.06 | {_fmt_verdict(accept_uniform)} |")
|
||||
out.append(f"| degraded parent → child recovers **only 1** 5F sub by +0.06 | {_fmt_verdict(marginal_single)} |")
|
||||
out.append(f"| degraded parent → child drops one 5F sub by −{HARD_REGRESSION_FLOOR} | {_fmt_verdict(reject_regress)} |")
|
||||
out.append("")
|
||||
out.append(
|
||||
"Row 2 vs row 3 is the **5× averaging dilution**: `_delta_5f` means "
|
||||
"over all 5 sub-batteries, so the same +0.06 gain scores `1.0·0.06 = "
|
||||
"0.06` (ACCEPT) when applied to all five subs but only `1.0·(0.06/5) "
|
||||
"= 0.012` (MARGINAL) when applied to one. Each battery contributes a "
|
||||
"separate Δ-term at weight 1.0, so a uniform +`SIGNAL_FLOOR` across "
|
||||
"*every* sub of *all three* batteries would score `~0.15`."
|
||||
)
|
||||
out.append("")
|
||||
out.append("## 4. Recommendation for #000012")
|
||||
out.append("")
|
||||
out.append(
|
||||
f"- **Keep `SIGNAL_FLOOR = {SIGNAL_FLOOR}` and "
|
||||
f"`HARD_REGRESSION_FLOOR = {HARD_REGRESSION_FLOOR}`.** They match "
|
||||
"`docs/bench-maxing.md`'s 5-pp signal floor and the sanity checks "
|
||||
"in §3 behave as documented."
|
||||
)
|
||||
out.append(
|
||||
f"- **Granularity caveat (load-bearing):** the foundational 5S "
|
||||
f"packs are *coarser* than the floors — `syntax` (n=10 → 10.0pp) "
|
||||
f"and `semantics` (n=8 → 12.5pp). On those packs a single fixture "
|
||||
f"flip is ≥ {HARD_REGRESSION_FLOOR*100:.0f}pp, so *any* regression "
|
||||
f"there trips the hard-reject. That is the intended zero-tolerance "
|
||||
f"behaviour on the syntax/semantics base — not a bug — but #000012 "
|
||||
f"should document it: HARD_REGRESSION_FLOOR is not a tunable knob "
|
||||
f"on the small packs, it's effectively 'one fixture'. The n=30 / "
|
||||
f"n=50 / n=62 packs have 3.3pp / 2.0pp / 1.6pp granularity, so "
|
||||
f"there {HARD_REGRESSION_FLOOR*100:.0f}pp ≈ 1.5–3 fixture flips."
|
||||
)
|
||||
out.append(
|
||||
"- **5× averaging dilution (document this in #000012):** a single "
|
||||
"sub-battery's rate gain is worth a fifth of its face value because "
|
||||
"`_delta_5{s,t,f}` means over 5 subs. So 'a fork must improve "
|
||||
f"by `SIGNAL_FLOOR`' really means *one of*: ~`{SIGNAL_FLOOR*5:.2f}` "
|
||||
f"on a single sub, ~`{SIGNAL_FLOOR:.2f}` uniform across one "
|
||||
f"battery's 5 subs, or ~`{SIGNAL_FLOOR/3:.3f}` uniform across the "
|
||||
"whole 15-sub suite. If #000012 wants single-sub improvements to "
|
||||
"weigh equally it should switch `_delta_*` from mean to "
|
||||
"max-or-sum — but that's a #000012-owned design call, not a "
|
||||
"calibration finding."
|
||||
)
|
||||
out.append(
|
||||
"- **Ceiling saturation:** until harder fixtures drop a pack's "
|
||||
"baseline below 1.0, the `α·Δ5s + β·Δ5t + γ·Δ5f` terms can only be "
|
||||
"≤ 0. ForkScore acceptance at the current pack difficulty is "
|
||||
"driven by the efficiency bonuses + non-bench terms. If #000012 "
|
||||
"wants the bench Δ-rate terms to carry real positive signal, the "
|
||||
"5S/5T/5F packs need a harder tier (or a deliberately-degraded "
|
||||
"parent baseline) — track that as a #000025 follow-up, not a "
|
||||
"#000012 blocker."
|
||||
)
|
||||
out.append(
|
||||
"- **No constant change shipped by this calibration.** This is a "
|
||||
"handoff document; if #000012 decides to move a floor it owns that "
|
||||
"edit (and the resulting `governance_policy_hash` is unaffected — "
|
||||
"ForkScore constants don't fold into it; they're estimator "
|
||||
"parameters pinned by `ESTIMATOR_VERSION`)."
|
||||
)
|
||||
out.append("")
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
p.add_argument(
|
||||
"--out",
|
||||
type=Path,
|
||||
default=None,
|
||||
help="Write the markdown report here (default: stdout).",
|
||||
)
|
||||
args = p.parse_args(argv)
|
||||
md = build_report()
|
||||
if args.out:
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out.write_text(md, encoding="utf-8")
|
||||
print(f"wrote {args.out}")
|
||||
else:
|
||||
print(md)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
171
bench/scripts/selfmodel_chain_snapshot.py
Normal file
171
bench/scripts/selfmodel_chain_snapshot.py
Normal file
|
|
@ -0,0 +1,171 @@
|
|||
"""Append one SelfModel snapshot to a persistent chain shard, with
|
||||
capability claims drawn from a fresh 5S/5T/5F bench run (#000025 §10.11).
|
||||
|
||||
Why this exists: Phase 1a's Finetuning sub-battery scored *synthetic*
|
||||
parent→child SelfModel pairs; Phase 1b.2's "live" path round-trips a
|
||||
parent + child through a *fresh temp* shard each run. §10.11 wants the
|
||||
real thing — a SelfModel lineage that *persists across runs*, so the
|
||||
Finetuning runner can measure improvement between two genuine,
|
||||
chained snapshots instead of a pair fabricated for the occasion.
|
||||
|
||||
Each invocation:
|
||||
|
||||
1. Runs the canonical 5S/5T/5F sub-batteries (embedded packs — the
|
||||
ones `fork_score._BATTERY_RATE_KEYS` reads). Deterministic, ~1s,
|
||||
no LLM call.
|
||||
2. Builds one :class:`arborist.selfmodel.CapabilityClaim` per
|
||||
sub-battery: ``metric = "5S-syntax"`` etc., ``measured_value`` =
|
||||
that pack's rate, ``eval_digest`` = that pack's fixture digest,
|
||||
``threshold`` = ``fork_score.SIGNAL_FLOOR`` (the rate floor below
|
||||
which a fork-score Δ on this capability is a hard regression).
|
||||
3. Calls :func:`arborist.selfmodel.snapshot` (which auto-parents to
|
||||
the latest root in the shard) → :func:`with_claims` →
|
||||
:func:`store_snapshot`. The new snapshot's distinct parent makes
|
||||
it a distinct root, so the chain grows by exactly one per run.
|
||||
|
||||
Output shard defaults to ``~/.arborist/shards/selfmodel-chain.db``
|
||||
(override with ``--shard`` or ``ARBORIST_SELFMODEL_CHAIN_DB``). The
|
||||
shard carries an ``audit_events`` table (every ``store_snapshot``
|
||||
write chains through it), so ``make chain-check-shards`` covers it.
|
||||
|
||||
Idempotent on content: re-running with no chain growth in between is
|
||||
a no-op only if the parent root is unchanged — which it isn't after
|
||||
the first append, so successive runs always extend the lineage. The
|
||||
bench rates are at ceiling (1.0) today, so the chained Δ is 0.0; the
|
||||
point is that the *mechanism* is real and the lineage outlives any
|
||||
single process.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(_REPO_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(_REPO_ROOT))
|
||||
|
||||
from bench.batteries import b_5f, b_5s, b_5t # noqa: E402
|
||||
from arborist.selfmodel import CapabilityClaim, snapshot, store_snapshot # noqa: E402
|
||||
from arborist.selfmodel.canonical import with_claims # noqa: E402
|
||||
from arborist.selfmodel.store import claims_for, latest # noqa: E402
|
||||
from arborist.store import connect, transaction # noqa: E402
|
||||
from arborist.substrate.fork_score import SIGNAL_FLOOR, _BATTERY_RATE_KEYS # noqa: E402
|
||||
|
||||
_FIX = _REPO_ROOT / "bench" / "fixtures"
|
||||
|
||||
# Canonical sub-battery → embedded fixture pack (the keys fork_score
|
||||
# reads). Mirrors bench.batteries.runner._DEFAULT_FIXTURES minus the
|
||||
# legacy 5t "transfer" the scorer ignores.
|
||||
_PACKS: list[tuple[str, str, Path]] = [
|
||||
("5s", "syntax", _FIX / "5s" / "syntax-v1.jsonl"),
|
||||
("5s", "semantics", _FIX / "5s" / "semantics-v1.jsonl"),
|
||||
("5s", "syllogism", _FIX / "5s" / "syllogism-v1.jsonl"),
|
||||
("5s", "synthesis", _FIX / "5s" / "synthesis-v1.jsonl"),
|
||||
("5s", "semiotics", _FIX / "5s" / "semiotics-v1.jsonl"),
|
||||
("5t", "transfer-learning", _FIX / "5t" / "transfer-learning-v2.jsonl"),
|
||||
("5t", "triangulation", _FIX / "5t" / "triangulation-v1.jsonl"),
|
||||
("5t", "truthtables", _FIX / "5t" / "truthtables-v1.jsonl"),
|
||||
("5t", "transitivity", _FIX / "5t" / "transitivity-v1.jsonl"),
|
||||
("5t", "time", _FIX / "5t" / "time-v1.jsonl"),
|
||||
("5f", "function", _FIX / "5f" / "function-v1.jsonl"),
|
||||
("5f", "finetuning", _FIX / "5f" / "finetuning-v1.jsonl"),
|
||||
("5f", "falsification", _FIX / "5f" / "falsification-v1.jsonl"),
|
||||
("5f", "formulate", _FIX / "5f" / "formulate-v1.jsonl"),
|
||||
("5f", "feedback-loop", _FIX / "5f" / "feedback-loop-v1.jsonl"),
|
||||
]
|
||||
|
||||
_SUB_RUNNERS = {"5s": b_5s.SUB_BATTERIES, "5t": b_5t.SUB_BATTERIES, "5f": b_5f.SUB_BATTERIES}
|
||||
|
||||
|
||||
def default_shard() -> Path:
|
||||
env = os.environ.get("ARBORIST_SELFMODEL_CHAIN_DB")
|
||||
if env:
|
||||
return Path(env).expanduser()
|
||||
return Path.home() / ".arborist" / "shards" / "selfmodel-chain.db"
|
||||
|
||||
|
||||
def _battery_claims(measured_at: int) -> list[CapabilityClaim]:
|
||||
claims: list[CapabilityClaim] = []
|
||||
for battery, sub, path in _PACKS:
|
||||
res = _SUB_RUNNERS[battery][sub](path)
|
||||
metric_key = _BATTERY_RATE_KEYS[battery][sub]
|
||||
rate = float(res.metrics.get(metric_key, 0.0))
|
||||
claims.append(
|
||||
CapabilityClaim(
|
||||
metric=f"{battery.upper()}-{sub}",
|
||||
threshold=float(SIGNAL_FLOOR),
|
||||
eval_digest=res.fixture_digest,
|
||||
measured_value=rate,
|
||||
measured_at=measured_at,
|
||||
validity_horizon="next-checkpoint",
|
||||
claim_text=f"{battery.upper()} {sub} {metric_key} on the embedded pack",
|
||||
)
|
||||
)
|
||||
return claims
|
||||
|
||||
|
||||
def append_snapshot(shard: Path, *, ts: int | None = None) -> dict:
|
||||
"""Run the suite, append one chained SelfModel snapshot, return a
|
||||
summary dict ``{root, parent_root, depth, claims: [...]}``."""
|
||||
if ts is None:
|
||||
ts = int(time.time())
|
||||
shard.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = connect(shard)
|
||||
try:
|
||||
prev = latest(conn)
|
||||
claims = _battery_claims(ts)
|
||||
with transaction(conn):
|
||||
sm = with_claims(snapshot(conn), claims)
|
||||
root = store_snapshot(conn, sm, claims=claims, ts=ts)
|
||||
# Walk the parent chain to report depth.
|
||||
depth = 1
|
||||
cur = load_parent(conn, root)
|
||||
while cur is not None:
|
||||
depth += 1
|
||||
cur = load_parent(conn, cur)
|
||||
rows = claims_for(conn, root)
|
||||
return {
|
||||
"shard": str(shard),
|
||||
"root": root,
|
||||
"parent_root": prev["selfmodel_root"] if prev else None,
|
||||
"depth": depth,
|
||||
"claims": {r["metric"]: r["measured_value"] for r in rows},
|
||||
}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def load_parent(conn, root: str) -> str | None:
|
||||
row = conn.execute(
|
||||
"SELECT parent_selfmodel_root FROM selfmodel_records WHERE selfmodel_root = ?",
|
||||
(root,),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
return row["parent_selfmodel_root"]
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
p.add_argument(
|
||||
"--shard",
|
||||
type=Path,
|
||||
default=None,
|
||||
help="Chain shard path (default: $ARBORIST_SELFMODEL_CHAIN_DB or ~/.arborist/shards/selfmodel-chain.db)",
|
||||
)
|
||||
args = p.parse_args(argv)
|
||||
shard = args.shard.expanduser() if args.shard else default_shard()
|
||||
summary = append_snapshot(shard)
|
||||
import json as _json
|
||||
|
||||
print(_json.dumps(summary, indent=2, default=str))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Loading…
Add table
Add a link
Reference in a new issue