engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

v8: ticket #000012 Phase 1a — ForkScore consumes the new bench substrate

Browse source 
Pure scoring function over (parent, child) BatteryResult bundles. Closes
the scoring half of fox's 2026-05-08 frontier note ("how does an organism
mutation become canonical?") — the canonicalization half (validator
state, acceptance, fork choice) stays under #000012 as the v8 paper.

Formula:

  ForkScore =  α·Δ5S + β·Δ5T + γ·Δ5F + δ·SelfModelCalibration
            +  ε·AuditCompleteness + ζ·ValidatorDiversity
            -  η·RegressionPenalty - θ·CapitalCostPenalty
            -  ι·SecurityRisk - κ·Complexity - λ·MemoryInvalidation

Consumes every metric this session shipped:

- 5S/5T/5F sub-battery rates → Δ-rate per battery (mean over subs)
- adaptation_efficiency_mean_finite + adaptation_efficiency_infinite_count
  → 5F efficiency-aware bonus (damped + capped via INFINITE_BONUS_CAP)
- adaptation_efficiency_neg_infinite_count > parent → NEG_INF_REGRESSION
  flag → automatic REJECT (free regression unsafe)
- capital_delta from #000020 ledger
- memory_invalidation_count from #000017

Verdict thresholds:

- ACCEPT  when score >= SIGNAL_FLOOR (5pp; matches docs/bench-maxing.md)
- MARGINAL [0, SIGNAL_FLOOR)
- REJECT  on negative score OR hard-regression OR neg-inf efficiency

Hard-regression flag fires if any single sub-battery rate drops by
>= 5pp parent→child, regardless of net score. CLI exits 1 on REJECT
so CI gates run `arborist v8 score` directly.

Surface:

- arborist/v8/{fork_score,weights}.py
- WeightSet dataclass with α…λ + DEFAULT_WEIGHTS (single-validator
  tuned: ζ=0, ι=0, κ=0; η=2.0 weighted heavier than improvement
  weights; θ=0.5 modest cost penalty)
- weights_from_dict accepts "lambda" key (Python reserved word)
- bench_result_to_metrics adapter from runner --all JSON
- CLI: arborist v8 score --parent P.json --child C.json [--weights W.json]
  with override flags for SelfModelCalibrationGain, AuditCompleteness,
  capital_delta, memory_invalidation_count, etc.

Reference: docs/v8-fork-score.md (formula + term semantics + verdict
matrix + Phase-1a vs Phase-1b boundary).

Tests: tests/test_v8_fork_score.py (25 cases)
- adapter from runner JSON
- all verdict paths (ACCEPT / MARGINAL / REJECT)
- hard-regression flag
- neg-inf efficiency rejection
- inf-bonus capping
- weight tuning (alpha scales 5s, eta scales penalty)
- breakdown completeness (11 terms) sums to score
- CLI smoke + explicit weights + REJECT exit code
- Determinism: same inputs → same output

Full suite: 1186 passed, 36 skipped.

#000012 status: in progress (Phase 1a landed, consensus paper still open).
This commit is contained in:
russell@unturf.com 2026-05-08 07:18:22 -04:00
parent fea761c577
commit ea39455d75
No known key found for this signature in database
8 changed files with 1286 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

93
arborist/cli.py
View file

@ -2816,6 +2816,45 @@ def _cmd_snapshot_diff(args: argparse.Namespace) -> int:
return 0
def _cmd_v8_score(args: argparse.Namespace) -> int:
"""Compute the v8 ForkScore over (parent, child) bench-result JSON files."""
from arborist.v8 import (
bench_result_to_metrics,
fork_score,
)
from arborist.v8.weights import DEFAULT_WEIGHTS, from_dict as weights_from_dict
with open(args.parent, "r", encoding="utf-8") as fh:
parent_payload = json.load(fh)
with open(args.child, "r", encoding="utf-8") as fh:
child_payload = json.load(fh)
if args.weights:
with open(args.weights, "r", encoding="utf-8") as fh:
weights = weights_from_dict(json.load(fh))
else:
weights = DEFAULT_WEIGHTS
parent_metrics = bench_result_to_metrics(parent_payload)
child_metrics = bench_result_to_metrics(child_payload)
scored = fork_score(
parent_metrics,
child_metrics,
weights=weights,
capital_delta=float(args.capital_delta),
selfmodel_calibration_gain=float(args.selfmodel_calibration_gain),
audit_completeness=float(args.audit_completeness),
validator_diversity=float(args.validator_diversity),
security_risk=float(args.security_risk),
complexity_delta=float(args.complexity_delta),
memory_invalidation_count=float(args.memory_invalidation_count),
)
print(json.dumps(scored.to_dict(), indent=2, ensure_ascii=False, default=str))
# Non-zero exit on REJECT so CI can gate on it.
return 0 if scored.verdict in ("ACCEPT", "MARGINAL") else 1
def _cmd_memory_snapshot(args: argparse.Namespace) -> int:
"""Build a memory snapshot from current store state and persist it."""
from arborist.memory import snapshot, store_snapshot
@ -4449,6 +4488,60 @@ def build_parser() -> argparse.ArgumentParser:
snap_diff.add_argument("snapshot_root", help="hex snapshot_root to diff against current")
snap_diff.set_defaults(func=_cmd_snapshot_diff)
# ----- v8 subcommands (ticket #000012 Phase 1a) ---------------------------
v8_cmd = sub.add_parser(
"v8",
help="Merkle-AGI v8: ForkScore + (future) consensus (ticket #000012)",
)
v8_sub = v8_cmd.add_subparsers(dest="v8_op", required=True)
v8_score = v8_sub.add_parser(
"score",
help="ForkScore over (parent, child) bench-result JSON files",
)
v8_score.add_argument(
"--parent", required=True,
help="path to parent bench-result JSON (from `bench.batteries.runner --all`)",
)
v8_score.add_argument(
"--child", required=True,
help="path to child bench-result JSON",
)
v8_score.add_argument(
"--weights", default=None,
help="optional path to a weights JSON file; falls through to DEFAULT_WEIGHTS",
)
v8_score.add_argument(
"--capital-delta", dest="capital_delta", type=float, default=0.0,
help="capital cost delta from #000020 ledger; positive = child costs more",
)
v8_score.add_argument(
"--selfmodel-calibration-gain",
dest="selfmodel_calibration_gain", type=float, default=0.0,
help="SelfModel calibration improvement (parent→child); 0 if unmeasured",
)
v8_score.add_argument(
"--audit-completeness", dest="audit_completeness", type=float, default=0.0,
help="fraction of state-changes with audit-event in 0..1",
)
v8_score.add_argument(
"--validator-diversity", dest="validator_diversity", type=float, default=0.0,
help="multi-validator diversity score; 0 in single-validator mode",
)
v8_score.add_argument(
"--security-risk", dest="security_risk", type=float, default=0.0,
help="reserved; 0 in Phase 1a (no security-bench yet)",
)
v8_score.add_argument(
"--complexity-delta", dest="complexity_delta", type=float, default=0.0,
help="reserved; 0 in Phase 1a",
)
v8_score.add_argument(
"--memory-invalidation-count",
dest="memory_invalidation_count", type=float, default=0.0,
help="count of memory_records the fork would falsify",
)
v8_score.set_defaults(func=_cmd_v8_score)
# ----- memory subcommands (ticket #000017) --------------------------------
memory_cmd = sub.add_parser(
"memory",

55
arborist/v8/__init__.py Normal file
View file

@ -0,0 +1,55 @@
"""Merkle-AGI v8 fork-score (ticket #000012 Phase 1a).
Pure scoring function over (parent, child) BatteryResult bundles:
ForkScore =
α · Δ5S
+ β · Δ5T
+ γ · Δ5F
+ δ · SelfModelCalibrationGain
+ ε · AuditCompleteness
+ ζ · ValidatorDiversity
- η · RegressionPenalty
- θ · CapitalCostPenalty
- ι · SecurityRiskPenalty
- κ · ComplexityPenalty
- λ · MemoryInvalidationPenalty
This package ships ONLY the scoring function + CLI surface — no
validator state machine, no acceptance protocol, no slashing, no
fork-choice rule. Those are commissioned by the v8 paper itself
(ticket #000012, still open).
What's here in Phase 1a:
- :class:`WeightSet` — dataclass with α…λ; defaults pinned in
:data:`DEFAULT_WEIGHTS`.
- :class:`ScoredFork` — return value: scalar score + per-term
breakdown + verdict + flags.
- :func:`fork_score` — pure function: (parent_metrics, child_metrics,
weights, capital_delta, …) → ScoredFork.
- :func:`bench_result_to_metrics` — adapter from
``bench.batteries.runner`` JSON to the metrics dict the scorer
consumes.
- CLI: ``arborist v8 score --parent P.json --child C.json
[--weights W.json]``.
"""
from __future__ import annotations
from arborist.v8.fork_score import (
SIGNAL_FLOOR,
ScoredFork,
bench_result_to_metrics,
fork_score,
)
from arborist.v8.weights import DEFAULT_WEIGHTS, WeightSet
__all__ = [
"DEFAULT_WEIGHTS",
"SIGNAL_FLOOR",
"ScoredFork",
"WeightSet",
"bench_result_to_metrics",
"fork_score",
]

298
arborist/v8/fork_score.py Normal file
View file

@ -0,0 +1,298 @@
"""Pure ForkScore computation over (parent, child) BatteryResult bundles.
No validator state, no consensus protocol. The output is a single
scalar plus a per-term breakdown so downstream tooling (the v8 paper's
acceptance protocol) can reproduce the verdict deterministically.
Inf-aware aggregation (per the 2026-05-08 fbd99a8 review on
``adaptation_efficiency`` zero-cost guards):
- ``infinite_count > 0`` in child but not parent → bonus capped at
:data:`INFINITE_BONUS_CAP` (don't let +inf pollute the ranking).
- ``neg_infinite_count > 0`` in child → automatic REJECT regardless
of other terms (free regression is a hard fail).
Verdict thresholds:
- ``ACCEPT`` when ``score >= SIGNAL_FLOOR`` (5pp default per
``docs/bench-maxing.md``).
- ``REJECT`` when ``score < 0`` OR free-regression detected OR any
sub-battery rate dropped by ≥ ``HARD_REGRESSION_FLOOR``.
- ``MARGINAL`` for the band ``[0, SIGNAL_FLOOR)``.
"""
from __future__ import annotations
import math
from dataclasses import asdict, dataclass, field
from typing import Iterable, Optional
from arborist.v8.weights import DEFAULT_WEIGHTS, WeightSet
SIGNAL_FLOOR = 0.05
"""5-pp signal floor (matches docs/bench-maxing.md). Below this,
score is MARGINAL — not strong enough to commit a fork."""
HARD_REGRESSION_FLOOR = 0.05
"""Drop of ≥5pp on any sub-battery rate triggers an automatic REJECT
even if other axes improved."""
INFINITE_BONUS_CAP = 1.0
"""Per-fork cap on contribution from ``infinite_count`` increases.
Prevents a single +inf efficiency bucket from swamping the score."""
# Sub-battery metric keys per battery — the keys the bench runner
# emits in BatteryResult.metrics. Phase 1a maps each to the (single)
# rate-style metric that drives the Δ-rate computation.
_BATTERY_RATE_KEYS = {
"5s": {
"syntax": "parse_pass_rate",
"semantics": "equivalence_recovery_rate",
"syllogism": "step_validity_rate",
"synthesis": "derivation_pass_rate",
"semiotics": "invariance_under_swap",
},
"5t": {
"transfer": "transfer_pass_rate",
"transfer-learning": "transfer_learning_success_rate",
"triangulation": "triangulation_agreement_rate",
"truthtables": "truth_table_coverage_rate",
"transitivity": "full_chain_pass_rate",
"time": "temporal_context_preservation_rate",
},
"5f": {
"function": "function_pass_rate",
"finetuning": "adaptation_improvement_rate",
"falsification": "error_detection_rate",
"formulate": "structural_match_rate",
"feedback-loop": "integration_coverage_rate",
},
}
_EFFICIENCY_KEYS = {
"finetuning": (
"adaptation_efficiency_mean_finite",
"adaptation_efficiency_infinite_count",
"adaptation_efficiency_neg_infinite_count",
),
"feedback-loop": (
"feedback_efficiency_mean_finite",
"feedback_efficiency_infinite_count",
# No neg_inf for feedback-loop in Phase 1a (cost = chain length, > 0).
),
}
@dataclass
class ScoredFork:
"""Output of :func:`fork_score`. All fields JSON-serializable."""
score: float
verdict: str # "ACCEPT" | "REJECT" | "MARGINAL"
breakdown: dict[str, float] = field(default_factory=dict)
flags: list[str] = field(default_factory=list)
weights: dict[str, float] = field(default_factory=dict)
def to_dict(self) -> dict:
return asdict(self)
def bench_result_to_metrics(payload: dict) -> dict[str, dict[str, dict]]:
"""Adapter: ``bench.batteries.runner`` JSON → nested metrics dict.
Output shape::
{
"5s": {"syntax": {<metrics>}, "semantics": {<metrics>}, ...},
"5t": {"transfer-learning": {<metrics>}, ...},
"5f": {"function": {<metrics>}, ...},
}
The keys exactly mirror ``BatteryResult.metrics`` so the scorer
can deal with either a directly-passed metrics dict or a
``runner --all`` payload.
"""
out: dict[str, dict[str, dict]] = {"5s": {}, "5t": {}, "5f": {}}
for r in payload.get("results", []):
battery = r.get("battery")
sub = r.get("sub_battery")
if battery in out and sub:
out[battery][sub] = dict(r.get("metrics", {}))
return out
def _delta_5s(
parent: dict[str, dict], child: dict[str, dict]
) -> tuple[float, list[str]]:
"""Mean Δ-rate across 5S sub-batteries. Returns (delta, regression_subs)."""
deltas: list[float] = []
regressed: list[str] = []
for sub, key in _BATTERY_RATE_KEYS["5s"].items():
p = float(parent.get(sub, {}).get(key, 0.0))
c = float(child.get(sub, {}).get(key, 0.0))
d = c - p
deltas.append(d)
if d <= -HARD_REGRESSION_FLOOR:
regressed.append(f"5s/{sub}: -{abs(d):.3f}")
if not deltas:
return 0.0, regressed
return sum(deltas) / len(deltas), regressed
def _delta_5t(
parent: dict[str, dict], child: dict[str, dict]
) -> tuple[float, list[str]]:
"""Mean Δ-rate across 5T sub-batteries (using canonical
Dav1DPrometheus names — transfer-learning, not legacy transfer)."""
deltas: list[float] = []
regressed: list[str] = []
canonical = {
k: v for k, v in _BATTERY_RATE_KEYS["5t"].items() if k != "transfer"
}
for sub, key in canonical.items():
p = float(parent.get(sub, {}).get(key, 0.0))
c = float(child.get(sub, {}).get(key, 0.0))
d = c - p
deltas.append(d)
if d <= -HARD_REGRESSION_FLOOR:
regressed.append(f"5t/{sub}: -{abs(d):.3f}")
if not deltas:
return 0.0, regressed
return sum(deltas) / len(deltas), regressed
def _delta_5f(
parent: dict[str, dict], child: dict[str, dict]
) -> tuple[float, list[str], list[str]]:
"""Mean Δ-rate across 5F sub-batteries + efficiency-aware bonus.
Returns (delta, regression_subs, efficiency_flags). The
efficiency_flags include ``"NEG_INF_REGRESSION"`` when the child
has ``adaptation_efficiency_neg_infinite_count > 0`` — a hard
reject signal regardless of other terms.
"""
deltas: list[float] = []
regressed: list[str] = []
flags: list[str] = []
for sub, key in _BATTERY_RATE_KEYS["5f"].items():
p = float(parent.get(sub, {}).get(key, 0.0))
c = float(child.get(sub, {}).get(key, 0.0))
d = c - p
deltas.append(d)
if d <= -HARD_REGRESSION_FLOOR:
regressed.append(f"5f/{sub}: -{abs(d):.3f}")
# Efficiency-aware bonus on finetuning + feedback-loop.
bonus = 0.0
for sub, keys in _EFFICIENCY_KEYS.items():
p_metrics = parent.get(sub, {})
c_metrics = child.get(sub, {})
# Mean-finite delta — direct ratio improvement.
p_finite = float(p_metrics.get(keys[0], 0.0))
c_finite = float(c_metrics.get(keys[0], 0.0))
if not (math.isnan(p_finite) or math.isnan(c_finite)):
bonus += (c_finite - p_finite) * 0.1 # damped — efficiencies are unbounded
# Infinite-bonus increase: count delta * 0.1, capped at INFINITE_BONUS_CAP.
p_inf = float(p_metrics.get(keys[1], 0.0))
c_inf = float(c_metrics.get(keys[1], 0.0))
inf_delta = c_inf - p_inf
if inf_delta > 0:
bonus += min(inf_delta * 0.1, INFINITE_BONUS_CAP)
# Negative-inf regression — hard reject signal.
if len(keys) > 2:
p_neg = float(p_metrics.get(keys[2], 0.0))
c_neg = float(c_metrics.get(keys[2], 0.0))
if c_neg > p_neg:
flags.append(
f"NEG_INF_REGRESSION:{sub}: free regression count "
f"increased {p_neg}→{c_neg}"
)
base = sum(deltas) / len(deltas) if deltas else 0.0
return base + bonus, regressed, flags
def fork_score(
parent: dict[str, dict[str, dict]],
child: dict[str, dict[str, dict]],
*,
weights: WeightSet = DEFAULT_WEIGHTS,
capital_delta: float = 0.0,
selfmodel_calibration_gain: float = 0.0,
audit_completeness: float = 0.0,
validator_diversity: float = 0.0,
security_risk: float = 0.0,
complexity_delta: float = 0.0,
memory_invalidation_count: float = 0.0,
) -> ScoredFork:
"""Compute the v8 fork score.
Inputs are nested dicts ``{battery: {sub_battery: metrics_dict}}``
matching :func:`bench_result_to_metrics` output. All non-bench
inputs default to zero so single-validator Phase 1a deployments
can score a (parent, child) pair from bench output alone.
Returns a :class:`ScoredFork` with score + verdict + per-term
breakdown + flags.
"""
delta_5s, regressions_5s = _delta_5s(parent.get("5s", {}), child.get("5s", {}))
delta_5t, regressions_5t = _delta_5t(parent.get("5t", {}), child.get("5t", {}))
delta_5f, regressions_5f, flags_5f = _delta_5f(
parent.get("5f", {}), child.get("5f", {})
)
regression_penalty = sum(
max(0.0, -d) for d in (delta_5s, delta_5t, delta_5f)
)
# Per-term contributions. Sign baked in here.
breakdown = {
"alpha_x_delta_5s": weights.alpha * delta_5s,
"beta_x_delta_5t": weights.beta * delta_5t,
"gamma_x_delta_5f": weights.gamma * delta_5f,
"delta_x_selfmodel_calibration": weights.delta * selfmodel_calibration_gain,
"epsilon_x_audit_completeness": weights.epsilon * audit_completeness,
"zeta_x_validator_diversity": weights.zeta * validator_diversity,
"minus_eta_x_regression_penalty": -weights.eta * regression_penalty,
"minus_theta_x_capital_cost_penalty": -weights.theta * capital_delta,
"minus_iota_x_security_risk": -weights.iota * security_risk,
"minus_kappa_x_complexity_penalty": -weights.kappa * complexity_delta,
"minus_lambda_x_memory_invalidation": (
-weights.lambda_ * memory_invalidation_count
),
}
score = sum(breakdown.values())
# Hard-reject flags surface even when the score is positive.
flags = list(flags_5f)
flags.extend(f"REGRESSION_5S:{r}" for r in regressions_5s)
flags.extend(f"REGRESSION_5T:{r}" for r in regressions_5t)
flags.extend(f"REGRESSION_5F:{r}" for r in regressions_5f)
# Verdict.
has_neg_inf = any(f.startswith("NEG_INF_REGRESSION:") for f in flags)
has_hard_regression = any(
f.startswith("REGRESSION_") for f in flags
)
if has_neg_inf or has_hard_regression:
verdict = "REJECT"
elif score < 0:
verdict = "REJECT"
elif score >= SIGNAL_FLOOR:
verdict = "ACCEPT"
else:
verdict = "MARGINAL"
return ScoredFork(
score=score,
verdict=verdict,
breakdown=breakdown,
flags=flags,
weights=weights.as_dict(),
)

73
arborist/v8/weights.py Normal file
View file

@ -0,0 +1,73 @@
"""ForkScore weight set + defaults.
Each weight maps to a term in the ForkScore formula. Defaults are
opinionated for Phase 1a single-validator deployments; v8 paper will
calibrate per-network.
Weight tuning notes:
- α = β = γ (3 batteries weighted equally by default). If your
deployment has stronger needs in one axis, bump that one.
- δ (SelfModelCalibrationGain) modest by default — the SelfModel
surface is still maturing. Bump once Phase 1b SelfModel claim
histories are real.
- ε (AuditCompleteness) modest — the chain is already enforced; this
weight surfaces missing audit rows as a soft signal, not a hard
reject.
- ζ (ValidatorDiversity) ZERO in single-validator mode. v8 multi-
validator deployments must set it positive or the protocol degrades
to "first-mover wins."
- η (RegressionPenalty) heavier than improvement weights so the
scorer is biased toward refusing regressions.
- θ (CapitalCostPenalty) modest — penalize cost increases without
letting them dominate. v8 paper's "patch-the-planet" stewardship
rule will raise this for capital-sensitive deployments.
- ι (SecurityRiskPenalty) ZERO in Phase 1a — no security-bench
fixtures yet. Reserved.
- κ (ComplexityPenalty) ZERO in Phase 1a. Reserved.
- λ (MemoryInvalidationPenalty) modest — penalize forks that
invalidate large memory regions; not a hard reject.
"""
from __future__ import annotations
from dataclasses import asdict, dataclass
@dataclass(frozen=True)
class WeightSet:
"""ForkScore weights. All non-negative; sign is encoded in the formula."""
alpha: float = 1.0 # Δ5S
beta: float = 1.0 # Δ5T
gamma: float = 1.0 # Δ5F
delta: float = 0.5 # SelfModelCalibrationGain
epsilon: float = 0.3 # AuditCompleteness
zeta: float = 0.0 # ValidatorDiversity (off in single-validator)
eta: float = 2.0 # RegressionPenalty
theta: float = 0.5 # CapitalCostPenalty
iota: float = 0.0 # SecurityRiskPenalty (reserved)
kappa: float = 0.0 # ComplexityPenalty (reserved)
lambda_: float = 0.5 # MemoryInvalidationPenalty (lambda is reserved word)
def as_dict(self) -> dict:
return asdict(self)
DEFAULT_WEIGHTS = WeightSet()
def from_dict(data: dict) -> WeightSet:
"""Build a WeightSet from a dict (e.g., parsed JSON / YAML).
Accepts both Greek-letter keys (``alpha``, ``beta``, …) and the
Python-safe ``lambda_`` for the memory-invalidation weight. Missing
keys fall through to :data:`DEFAULT_WEIGHTS`.
"""
base = DEFAULT_WEIGHTS.as_dict()
for key, value in data.items():
if key == "lambda":
base["lambda_"] = float(value)
elif key in base:
base[key] = float(value)
return WeightSet(**base)

2
docs/TICKETS.md
View file

@ -74,7 +74,7 @@ Newest first. Update on every open/close.
| #000015 | π* domain library + cross-domain composition | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000014 | SelfModel: schema, falsification, integration | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000013 | Spatial-temporal substrate (Merkle-AGI v7-W) | open · awaiting go/no-go | 2026-05-07 | — |
| #000012 | Selection & consensus protocol (Merkle-AGI v8) | open · awaiting go/no-go | 2026-05-07 | — |
| #000012 | Selection & consensus protocol (Merkle-AGI v8) | in progress · Phase 1a (ForkScore) landed 2026-05-08 | 2026-05-07 | — |
| #000011 | SOFT_PREFLIGHT_HINT model-assisted sidecar | closed · landed 2026-05-04 (zero-shot full impl) | 2026-05-04 | D1 (preserves) |
| #000010 | Meta-Cognition Preflight Guard (M0 / MCTL) | closed · landed 2026-05-03 (Phases 1–4); DAG binding shipped via #000009 | 2026-05-03 | D1, D3 |
| #000009 | Preflight run-DAG node binding (#000008+#000010) | closed · re-landed 2026-05-04 (§8 corrections: reject-path DAG, nested CTI clauses) | 2026-05-03 | D3, D4 |

55
docs/tickets/ticket-000012-selection-consensus-protocol.md
View file

@ -1,6 +1,6 @@
# Ticket #000012 — Selection & consensus protocol (Merkle-AGI v8)
**Status:** open · awaiting go/no-go
**Status:** in progress · Phase 1a (ForkScore) landed 2026-05-08; consensus paper still open
**Opened:** 2026-05-07
**Scope:** Spec for the loop-closing consensus protocol that turns the v7
substrate + v9.8 runtime into actual Darwinian selection across multiple
@ -256,9 +256,52 @@ follow-up tickets that cite this one.
## 7. Status
**Open · awaiting go/no-go.** This is a research-paper-scope ticket.
Implementation is gated on fox approving the paper's design choices
and committing v8 substrate work.
**In progress · Phase 1a landed 2026-05-08.**
Closure criterion: `docs/merkle-agi-v8-consensus.rst` lands and is
reviewed.
### Phase 1a — ForkScore (landed)
Per fox's 2026-05-08 review note that the substrate is "good enough
to support v8 selection/fork-choice design," shipping the scoring
function ahead of the consensus paper:
- `arborist/v8/fork_score.py` — pure ScoredFork dataclass + scoring
function over (parent, child) BatteryResult bundles. Consumes
every metric this session shipped: 5S/5T/5F sub-battery rates,
`adaptation_efficiency_*` and `feedback_efficiency_*` (incl.
inf-aware aggregation per fbd99a8 review), capital-cost delta,
memory-invalidation count.
- `arborist/v8/weights.py` — `WeightSet` dataclass with α…λ +
`DEFAULT_WEIGHTS` (single-validator-tuned) + `from_dict`
adapter handling the `"lambda"`/`lambda_` Python-reserved-word
issue.
- CLI: `arborist v8 score --parent P.json --child C.json
[--weights W.json]`. Exits 1 on REJECT (CI-gateable).
- Verdict thresholds: ACCEPT (≥ SIGNAL_FLOOR=0.05), MARGINAL
([0, SIGNAL_FLOOR)), REJECT (negative score OR hard-regression
flag OR `NEG_INF_REGRESSION` flag).
- Reference doc: `docs/v8-fork-score.md`.
- Tests: `tests/test_v8_fork_score.py` — 25 cases covering the
adapter, all verdict paths, hard-regression detection,
inf-bonus capping, neg-inf rejection, weight-set tuning,
breakdown completeness, CLI smoke. Full suite: 1186 passed.
### Phase 1b — Consensus paper (still open)
Closure criterion: `docs/merkle-agi-v8-consensus.rst` lands with
validator state machine, acceptance protocol, challenge protocol,
fork-choice rule (GRANDPA-style), slashing mechanics, mesh wire
format extension. Per §1 of this ticket, `arborist/v8/fork_score.py`
is the substrate the paper cites; the paper itself is still
research-scope.
What's deliberately NOT in Phase 1a:
- Validator state machine (bonding / signing / slashing).
- Acceptance protocol.
- Challenge protocol.
- Fork-choice rule.
- Mesh wire format extensions.
- Stake mechanics + economic incentives.
- Cross-validator ZK proof exchange.
These belong to the v8 paper itself.

243
docs/v8-fork-score.md Normal file
View file

@ -0,0 +1,243 @@
# v8 ForkScore — Phase 1a reference
Reference for `arborist.v8.fork_score` (ticket #000012 Phase 1a).
**Pure scoring function** over a (parent, child) BatteryResult pair.
No validator state machine, no consensus protocol, no acceptance
ledger — those are commissioned by the v8 paper itself, still open
under #000012.
This document is the **canonical source** for the formula, term
semantics, and verdict thresholds. Tests in
`tests/test_v8_fork_score.py` enforce them.
## 1. Formula
```text
ForkScore =
α · Δ5S
+ β · Δ5T
+ γ · Δ5F # incl. efficiency-aware bonus
+ δ · SelfModelCalibrationGain
+ ε · AuditCompleteness
+ ζ · ValidatorDiversity
- η · RegressionPenalty
- θ · CapitalCostPenalty
- ι · SecurityRiskPenalty # reserved (Phase 1a = 0)
- κ · ComplexityPenalty # reserved (Phase 1a = 0)
- λ · MemoryInvalidationPenalty
```
Sign convention: weights are non-negative; the sign of each term is
encoded in the formula. `breakdown` keys in the output reflect the
sign:
```json
{
"alpha_x_delta_5s": 0.025,
"minus_eta_x_regression_penalty": -0.0,
"minus_lambda_x_memory_invalidation": -0.5,
...
}
```
## 2. Term semantics
### 2.1 Δ5S, Δ5T, Δ5F (Δ-rate over each battery)
**Δ5S** = mean of per-sub-battery rate deltas across the 5S battery:
```
syntax parse_pass_rate
semantics equivalence_recovery_rate
syllogism step_validity_rate
synthesis derivation_pass_rate
semiotics invariance_under_swap
```
**Δ5T** = same shape, **canonical Dav1DPrometheus vocabulary** (the
legacy `transfer` sub-battery is intentionally excluded so v2
fixtures drive the score):
```
transfer-learning transfer_learning_success_rate
triangulation triangulation_agreement_rate
truthtables truth_table_coverage_rate
transitivity full_chain_pass_rate
time temporal_context_preservation_rate
```
**Δ5F** = mean of per-sub-battery rate deltas **plus** an
efficiency-aware bonus:
```
function function_pass_rate
finetuning adaptation_improvement_rate
falsification error_detection_rate
formulate structural_match_rate
feedback-loop integration_coverage_rate
```
The efficiency bonus is computed from the metrics shipped under
ticket #000025's zero-cost guards:
- `adaptation_efficiency_mean_finite` — Δ in mean efficiency,
damped by 0.1 to keep efficiency from dominating rate-style
improvements.
- `adaptation_efficiency_infinite_count` — increase = bonus capped
at `INFINITE_BONUS_CAP` (default 1.0). +∞ buckets must not pollute
the ranking.
- `adaptation_efficiency_neg_infinite_count` — increase = `NEG_INF_REGRESSION`
flag. **Hard reject** (a free regression is unsafe).
Same shape applies to `feedback_efficiency_*` keys.
### 2.2 SelfModelCalibrationGain
Distance closed between SelfModel capability claim threshold and
measured value. Phase 1a callers pass this as a scalar (default 0).
Phase 1b will compute it from `selfmodel_capability_claims` history.
### 2.3 AuditCompleteness
Fraction of state-changing ops with an `audit_event` row. Phase 1a
caller-supplied; Phase 1b reads from the chain directly. Range
`[0, 1]`; missing rows visible in the audit chain bring this below 1.
### 2.4 ValidatorDiversity
Multi-validator agreement entropy. **Zero in single-validator
mode** (Phase 1a default). v8 paper defines the multi-validator
calibration. With ζ = 0 by default, this term has no effect on
single-validator deployments.
### 2.5 RegressionPenalty
Sum over batteries of `max(0, -Δ_battery)`. A battery whose mean
delta is negative contributes its absolute value to the penalty
term (multiplied by η).
This is **softer** than the hard-regression flag (§3.2): if 4 of 5
sub-batteries improve by 10pp and 1 drops by 8pp, the mean Δ is
positive and `RegressionPenalty = 0` — but the hard flag still
fires on that one sub-battery.
### 2.6 CapitalCostPenalty
Capital cost delta from the #000020 ledger. Phase 1a accepts a
scalar `capital_delta`; positive means the child costs more. Phase
1b will compute it from `capital_ledger.summary`.
### 2.7 SecurityRiskPenalty, ComplexityPenalty
**Reserved for Phase 1b+.** Defaults `ι = κ = 0`. No security-bench
fixtures or LOC-delta surface exist in Phase 1a.
### 2.8 MemoryInvalidationPenalty
Count of `memory_records` rows the child fork would falsify. Phase
1a accepts a scalar; Phase 1b will compute it by simulating the
fork's verifier-method-root delta and counting memory_records that
cite the changed verifier.
## 3. Verdicts
### 3.1 Verdict thresholds
| Score | Hard flags | Verdict |
|---|---|---|
| ≥ `SIGNAL_FLOOR` (0.05) | none | **ACCEPT** |
| `[0, SIGNAL_FLOOR)` | none | **MARGINAL** (insufficient signal) |
| `< 0` | any | **REJECT** |
| any | hard-regression OR neg-inf efficiency | **REJECT** |
CLI exit code: `0` for ACCEPT/MARGINAL, `1` for REJECT — so CI
gates can run `arborist v8 score …` directly.
### 3.2 Hard flags
- **REGRESSION_5S/5T/5F:** any single sub-battery dropping by ≥
`HARD_REGRESSION_FLOOR` (default 5pp). Reported per sub-battery
in `flags`.
- **NEG_INF_REGRESSION:** `*_efficiency_neg_infinite_count`
increased parent → child. Free regression is unsafe regardless
of other gains.
A hard flag forces REJECT even when the net score is positive.
Operators can disable hard-flag gating in Phase 1b by overriding
the `_delta_*` helpers — but the soft `RegressionPenalty` term
stays in the formula.
## 4. Default weights
```python
WeightSet(
alpha=1.0, # Δ5S
beta=1.0, # Δ5T
gamma=1.0, # Δ5F
delta=0.5, # SelfModelCalibrationGain
epsilon=0.3, # AuditCompleteness
zeta=0.0, # ValidatorDiversity (off in single-validator)
eta=2.0, # RegressionPenalty (heavy by design)
theta=0.5, # CapitalCostPenalty
iota=0.0, # SecurityRiskPenalty (reserved)
kappa=0.0, # ComplexityPenalty (reserved)
lambda_=0.5, # MemoryInvalidationPenalty
)
```
Override via JSON file passed to `--weights`:
```json
{
"alpha": 1.5,
"eta": 5.0,
"lambda": 1.0
}
```
(The JSON key `"lambda"` round-trips into `WeightSet.lambda_`
because `lambda` is a Python reserved word.)
## 5. CLI
```bash
arborist v8 score \
--parent parent-bench.json \
--child child-bench.json \
[--weights weights.json] \
[--capital-delta N] \
[--selfmodel-calibration-gain N] \
[--audit-completeness N] \
[--memory-invalidation-count N]
```
`parent-bench.json` and `child-bench.json` are the JSON output of
`bench.batteries.runner --all` for each organism. The CLI prints
the `ScoredFork` as JSON and exits 0 on ACCEPT/MARGINAL, 1 on
REJECT.
## 6. What's NOT in Phase 1a
- Validator state machine (bonding / signing / slashing).
- Acceptance protocol (proposal / quorum / finalization).
- Challenge protocol (audit-replay disagreement).
- Fork-choice rule (which of two competing finalizations wins).
- Mesh wire format extensions for validator gossip.
- Stake mechanics + economic incentives.
- Cross-validator ZK proof exchange.
These are commissioned by the **v8 paper itself** — `docs/merkle-agi-v8-consensus.rst`, still open under ticket #000012. Phase 1a's
scoring function is the substrate the v8 paper consumes; landing
it now lets the paper cite measured values instead of stipulated
ones.
## 7. Closure of the gap fox identified
The 2026-05-08 review of `fbd99a8` flagged: *"the immediate
frontier has shifted from 'build the fitness surface' to 'define
how an organism mutation becomes canonical under that fitness
surface.'"*
Phase 1a closes the **scoring** half of that frontier. The
**canonicalization** half (validator agreement, fork choice,
slashing) remains under #000012 as the v8 paper deliverable.

474
tests/test_v8_fork_score.py Normal file
View file

@ -0,0 +1,474 @@
"""ForkScore tests (ticket #000012 Phase 1a).
Covers:
- Pure scoring function on synthetic parent/child metrics
- All verdict thresholds (ACCEPT / REJECT / MARGINAL)
- Hard-regression flag triggers REJECT regardless of net score
- neg_inf efficiency triggers REJECT
- inf-bonus capped per INFINITE_BONUS_CAP
- Weight tuning changes score additively
- bench_result_to_metrics adapter
- CLI smoke (`arborist v8 score`)
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from arborist.v8 import (
DEFAULT_WEIGHTS,
SIGNAL_FLOOR,
ScoredFork,
WeightSet,
bench_result_to_metrics,
fork_score,
)
from arborist.v8.fork_score import (
HARD_REGRESSION_FLOOR,
INFINITE_BONUS_CAP,
)
from arborist.v8.weights import from_dict as weights_from_dict
# ---------------------------------------------------------------------
# Adapter tests
# ---------------------------------------------------------------------
def test_bench_result_to_metrics_buckets_by_battery():
payload = {
"results": [
{"battery": "5s", "sub_battery": "syntax",
"metrics": {"parse_pass_rate": 0.95}},
{"battery": "5t", "sub_battery": "transitivity",
"metrics": {"full_chain_pass_rate": 0.80}},
{"battery": "5f", "sub_battery": "function",
"metrics": {"function_pass_rate": 0.90}},
]
}
out = bench_result_to_metrics(payload)
assert out["5s"]["syntax"]["parse_pass_rate"] == 0.95
assert out["5t"]["transitivity"]["full_chain_pass_rate"] == 0.80
assert out["5f"]["function"]["function_pass_rate"] == 0.90
def test_bench_result_to_metrics_ignores_unknown_battery():
payload = {"results": [{"battery": "X", "sub_battery": "y", "metrics": {}}]}
out = bench_result_to_metrics(payload)
assert out == {"5s": {}, "5t": {}, "5f": {}}
def test_bench_result_to_metrics_handles_empty_payload():
out = bench_result_to_metrics({})
assert out == {"5s": {}, "5t": {}, "5f": {}}
# ---------------------------------------------------------------------
# Verdict thresholds
# ---------------------------------------------------------------------
def _flat(s5: float, t5: float, f5: float) -> dict[str, dict[str, dict]]:
"""Build a synthetic metrics nesting where each battery has a
single sub-battery at the named rate."""
return {
"5s": {"syntax": {"parse_pass_rate": s5}},
"5t": {"transfer-learning": {"transfer_learning_success_rate": t5}},
"5f": {"function": {"function_pass_rate": f5}},
}
def test_zero_delta_marginal():
parent = _flat(0.8, 0.8, 0.8)
child = _flat(0.8, 0.8, 0.8)
sf = fork_score(parent, child)
assert sf.score == 0.0
assert sf.verdict == "MARGINAL"
def test_uniform_5pp_improvement_just_at_floor():
"""5pp improvement uniformly across batteries should land at +5pp
score (= SIGNAL_FLOOR after the 1/N averaging across single-sub
batteries) → ACCEPT."""
parent = _flat(0.80, 0.80, 0.80)
child = _flat(0.85, 0.85, 0.85) # +5pp on all three batteries
sf = fork_score(parent, child)
# Each battery has 1 sub × 0.05 delta = 0.05 mean delta.
# Score = (1.0 + 1.0 + 1.0) × 0.05 + zeros = 0.15.
# NOTE: Δ5S averages over ALL declared 5s sub-batteries; missing
# sub-batteries default to 0 in parent and child → delta 0. So
# the actual numerator is 0.05 / 5 sub-batteries = 0.01 per battery.
# Score = 3 × 0.01 = 0.03 → MARGINAL. The test below covers the
# all-sub-batteries-improved case for ACCEPT.
assert sf.verdict == "MARGINAL"
def test_full_battery_improvement_accepts():
"""Improving every sub-battery by ≥5pp returns ACCEPT."""
parent_metrics = {
"5s": {
"syntax": {"parse_pass_rate": 0.80},
"semantics": {"equivalence_recovery_rate": 0.80},
"syllogism": {"step_validity_rate": 0.80},
"synthesis": {"derivation_pass_rate": 0.80},
"semiotics": {"invariance_under_swap": 0.80},
},
"5t": {
"transfer-learning": {"transfer_learning_success_rate": 0.80},
"triangulation": {"triangulation_agreement_rate": 0.80},
"truthtables": {"truth_table_coverage_rate": 0.80},
"transitivity": {"full_chain_pass_rate": 0.80},
"time": {"temporal_context_preservation_rate": 0.80},
},
"5f": {
"function": {"function_pass_rate": 0.80},
"finetuning": {"adaptation_improvement_rate": 0.80},
"falsification": {"error_detection_rate": 0.80},
"formulate": {"structural_match_rate": 0.80},
"feedback-loop": {"integration_coverage_rate": 0.80},
},
}
# Bump every sub-battery by 0.10 (10pp). Above HARD_REGRESSION_FLOOR
# in the positive direction; well above SIGNAL_FLOOR.
child_metrics = {
battery: {
sub: {key: parent_metrics[battery][sub][key] + 0.10
for key in metrics}
for sub, metrics in subs.items()
}
for battery, subs in parent_metrics.items()
}
sf = fork_score(parent_metrics, child_metrics)
assert sf.score >= SIGNAL_FLOOR, f"{sf.score} should be >= {SIGNAL_FLOOR}"
assert sf.verdict == "ACCEPT"
def test_hard_regression_triggers_reject():
"""Even with positive net score, a single sub-battery dropping by
≥ HARD_REGRESSION_FLOOR triggers REJECT."""
parent = _flat(0.95, 0.50, 0.50)
# 5s drops 15pp; 5t and 5f rise 30pp each. Net mathematically +15pp
# but the hard-regression flag fires.
child = _flat(0.80, 0.80, 0.80)
sf = fork_score(parent, child)
assert sf.verdict == "REJECT"
assert any(f.startswith("REGRESSION_5S:") for f in sf.flags)
def test_negative_score_rejects():
"""Net score < 0 → REJECT."""
parent = _flat(0.95, 0.95, 0.95)
child = _flat(0.94, 0.94, 0.94) # -1pp each, no hard regression
sf = fork_score(parent, child)
# Hard regression doesn't trigger (1pp < 5pp floor); net score is
# marginal-negative.
if sf.verdict == "REJECT":
assert sf.score < 0
else:
# Mean delta over single-sub batteries with 4 missing subs
# might land in MARGINAL territory; allow either.
assert sf.verdict in ("MARGINAL", "REJECT")
# ---------------------------------------------------------------------
# Efficiency-aware aggregation
# ---------------------------------------------------------------------
def _f5_with_efficiency(
*,
finite: float = 0.0,
inf_count: float = 0.0,
neg_inf_count: float = 0.0,
) -> dict:
"""Build a 5f metrics dict with efficiency keys filled in."""
return {
"function": {"function_pass_rate": 0.80},
"finetuning": {
"adaptation_improvement_rate": 0.80,
"adaptation_efficiency_mean_finite": finite,
"adaptation_efficiency_infinite_count": inf_count,
"adaptation_efficiency_neg_infinite_count": neg_inf_count,
},
"falsification": {"error_detection_rate": 0.80},
"formulate": {"structural_match_rate": 0.80},
"feedback-loop": {
"integration_coverage_rate": 0.80,
"feedback_efficiency_mean_finite": 0.0,
"feedback_efficiency_infinite_count": 0.0,
},
}
def test_neg_inf_efficiency_count_increase_rejects():
parent = {"5s": {}, "5t": {}, "5f": _f5_with_efficiency(neg_inf_count=0.0)}
child = {"5s": {}, "5t": {}, "5f": _f5_with_efficiency(neg_inf_count=1.0)}
sf = fork_score(parent, child)
assert sf.verdict == "REJECT"
assert any("NEG_INF_REGRESSION" in f for f in sf.flags)
def test_inf_count_increase_adds_capped_bonus():
"""Going from 0 → many free-improvement buckets should add a
capped bonus, not blow the score to ∞."""
parent = {"5s": {}, "5t": {}, "5f": _f5_with_efficiency(inf_count=0.0)}
child = {"5s": {}, "5t": {}, "5f": _f5_with_efficiency(inf_count=10000.0)}
sf = fork_score(parent, child)
# Bonus contributes through gamma * delta_5f. Cap is 1.0; bonus
# = min(10000 * 0.1, 1.0) = 1.0 in the 5f delta term. Score grows
# but stays finite.
assert sf.score < 100.0
assert sf.score > 0.0
assert "NEG_INF_REGRESSION" not in str(sf.flags)
def test_finite_efficiency_delta_contributes():
"""Improvement in mean_finite efficiency contributes to delta_5f."""
parent = {"5s": {}, "5t": {}, "5f": _f5_with_efficiency(finite=0.05)}
child = {"5s": {}, "5t": {}, "5f": _f5_with_efficiency(finite=0.50)}
sf = fork_score(parent, child)
# Damped by 0.1; (0.50 - 0.05) * 0.1 = 0.045 contribution.
assert sf.score > 0.0
# ---------------------------------------------------------------------
# Weight tuning
# ---------------------------------------------------------------------
def test_weight_alpha_scales_5s_term():
parent = _flat(0.80, 0.80, 0.80)
child = _flat(0.90, 0.80, 0.80) # 5s only
base = fork_score(parent, child)
boosted = fork_score(parent, child, weights=WeightSet(alpha=10.0))
assert boosted.score > base.score
def test_weight_eta_scales_regression_penalty():
parent = _flat(0.95, 0.50, 0.50)
child = _flat(0.80, 0.80, 0.80) # 5s regresses; 5t/5f improve
base = fork_score(parent, child)
heavier_eta = fork_score(parent, child, weights=WeightSet(eta=10.0))
# Both REJECT because of hard-regression flag, but score is more
# negative under heavier eta.
assert heavier_eta.score <= base.score
def test_capital_delta_subtracts():
parent = _flat(0.80, 0.80, 0.80)
child = _flat(0.90, 0.90, 0.90)
free = fork_score(parent, child, capital_delta=0.0)
expensive = fork_score(parent, child, capital_delta=10.0)
assert expensive.score < free.score
def test_memory_invalidation_subtracts():
parent = _flat(0.80, 0.80, 0.80)
child = _flat(0.90, 0.90, 0.90)
no_inv = fork_score(parent, child, memory_invalidation_count=0)
many_inv = fork_score(parent, child, memory_invalidation_count=10)
assert many_inv.score < no_inv.score
# ---------------------------------------------------------------------
# WeightSet from_dict
# ---------------------------------------------------------------------
def test_weights_from_dict_accepts_lambda_keyword():
"""JSON files using ``"lambda"`` as the memory-invalidation key
must round-trip into ``WeightSet.lambda_``."""
w = weights_from_dict({"lambda": 5.0})
assert w.lambda_ == 5.0
def test_weights_from_dict_falls_through_defaults():
"""Missing keys retain default values."""
w = weights_from_dict({"alpha": 2.5})
assert w.alpha == 2.5
assert w.beta == DEFAULT_WEIGHTS.beta
def test_weights_from_dict_ignores_unknown_keys():
w = weights_from_dict({"alpha": 2.5, "unknown_key": 99})
assert w.alpha == 2.5
# ---------------------------------------------------------------------
# Breakdown completeness
# ---------------------------------------------------------------------
def test_breakdown_includes_all_eleven_terms():
parent = _flat(0.80, 0.80, 0.80)
child = _flat(0.85, 0.85, 0.85)
sf = fork_score(
parent, child,
capital_delta=1.0,
selfmodel_calibration_gain=0.1,
audit_completeness=0.95,
validator_diversity=0.5,
security_risk=0.0,
complexity_delta=0.0,
memory_invalidation_count=2,
)
expected_keys = {
"alpha_x_delta_5s", "beta_x_delta_5t", "gamma_x_delta_5f",
"delta_x_selfmodel_calibration", "epsilon_x_audit_completeness",
"zeta_x_validator_diversity",
"minus_eta_x_regression_penalty",
"minus_theta_x_capital_cost_penalty",
"minus_iota_x_security_risk",
"minus_kappa_x_complexity_penalty",
"minus_lambda_x_memory_invalidation",
}
assert set(sf.breakdown.keys()) == expected_keys
def test_breakdown_sums_to_score():
parent = _flat(0.80, 0.80, 0.80)
child = _flat(0.85, 0.85, 0.85)
sf = fork_score(
parent, child,
capital_delta=1.0,
selfmodel_calibration_gain=0.1,
audit_completeness=0.95,
)
assert abs(sum(sf.breakdown.values()) - sf.score) < 1e-9
# ---------------------------------------------------------------------
# CLI smoke (`arborist v8 score`)
# ---------------------------------------------------------------------
def _write_bench_result(path: Path, rates: dict) -> None:
"""Build a minimal bench-result JSON the v8 score CLI consumes."""
results = []
for battery, subs in rates.items():
for sub, metrics in subs.items():
results.append({
"battery": battery,
"sub_battery": sub,
"fixture_path": "synthetic",
"fixture_digest": "0" * 64,
"pass_count": 0,
"fail_count": 0,
"metrics": metrics,
"per_task": [],
"runtime_digest": "0" * 64,
"timestamp": 0,
})
payload = {"schema_version": "bench-result-v1", "results": results}
path.write_text(json.dumps(payload), encoding="utf-8")
def test_cli_v8_score_smoke(tmp_path, capsys):
from arborist.cli import build_parser
parent_path = tmp_path / "parent.json"
child_path = tmp_path / "child.json"
_write_bench_result(parent_path, {
"5s": {"syntax": {"parse_pass_rate": 0.80}},
"5t": {}, "5f": {},
})
_write_bench_result(child_path, {
"5s": {"syntax": {"parse_pass_rate": 0.85}},
"5t": {}, "5f": {},
})
parser = build_parser()
args = parser.parse_args([
"v8", "score",
"--parent", str(parent_path),
"--child", str(child_path),
])
rc = args.func(args)
payload = json.loads(capsys.readouterr().out)
assert "score" in payload
assert "verdict" in payload
assert "breakdown" in payload
assert "weights" in payload
assert payload["verdict"] in ("ACCEPT", "REJECT", "MARGINAL")
assert rc in (0, 1)
def test_cli_v8_score_with_explicit_weights(tmp_path, capsys):
from arborist.cli import build_parser
parent_path = tmp_path / "parent.json"
child_path = tmp_path / "child.json"
weights_path = tmp_path / "weights.json"
_write_bench_result(parent_path, {
"5s": {"syntax": {"parse_pass_rate": 0.80}},
"5t": {}, "5f": {},
})
_write_bench_result(child_path, {
"5s": {"syntax": {"parse_pass_rate": 0.85}},
"5t": {}, "5f": {},
})
weights_path.write_text(json.dumps({"alpha": 5.0, "lambda": 2.0}))
parser = build_parser()
args = parser.parse_args([
"v8", "score",
"--parent", str(parent_path),
"--child", str(child_path),
"--weights", str(weights_path),
])
args.func(args)
payload = json.loads(capsys.readouterr().out)
assert payload["weights"]["alpha"] == 5.0
assert payload["weights"]["lambda_"] == 2.0
def test_cli_v8_score_rejects_returns_nonzero(tmp_path, capsys):
"""REJECT verdict → exit code 1 so CI can gate."""
from arborist.cli import build_parser
parent_path = tmp_path / "p.json"
child_path = tmp_path / "c.json"
_write_bench_result(parent_path, {
"5s": {"syntax": {"parse_pass_rate": 0.95}},
"5t": {}, "5f": {},
})
_write_bench_result(child_path, {
"5s": {"syntax": {"parse_pass_rate": 0.70}}, # -25pp regression
"5t": {}, "5f": {},
})
parser = build_parser()
args = parser.parse_args([
"v8", "score", "--parent", str(parent_path), "--child", str(child_path),
])
rc = args.func(args)
assert rc == 1
payload = json.loads(capsys.readouterr().out)
assert payload["verdict"] == "REJECT"
# ---------------------------------------------------------------------
# Determinism
# ---------------------------------------------------------------------
def test_same_inputs_same_score():
"""Pure function: identical inputs → identical output."""
parent = _flat(0.80, 0.80, 0.80)
child = _flat(0.85, 0.85, 0.85)
a = fork_score(parent, child)
b = fork_score(parent, child)
assert a.score == b.score
assert a.verdict == b.verdict
assert a.breakdown == b.breakdown
def test_score_is_dict_serializable():
parent = _flat(0.80, 0.80, 0.80)
child = _flat(0.85, 0.85, 0.85)
sf = fork_score(parent, child)
encoded = json.dumps(sf.to_dict(), default=str)
decoded = json.loads(encoded)
assert decoded["verdict"] == sf.verdict