CLI: arborist v8 score → arborist substrate score
Followup to 654d923 (which moved the package from arborist/v8/ →
arborist/substrate/ at the file layer). The CLI surface still baked
in `v8` so a new operator running `--help` would see
``arborist v8 score`` and ask the same "what's v8 vs v9.8?"
naming-confusion question that drove the package rename in the
first place. Closing the loop end-to-end.
arborist/cli.py
===============
- Subparser renamed: ``"v8"`` → ``"substrate"``; help string updated
to "Merkle-AGI substrate primitives (ForkScore + future paper
specs)" so the dir name and command name and help text all align.
- Inner subparser dest renamed: ``v8_op`` → ``substrate_op``.
- Function renamed: ``_cmd_v8_score`` → ``_cmd_substrate_score``;
docstring updated.
- All ``v8_score`` local variables renamed to ``substrate_score``.
- New comment block above the subparser block explains the rename
+ why the v-prefix was retired (substrate-paper version vs v9.8
schema version naming collision).
The old ``arborist v8 score`` is gone — no alias preserved. CI + ops
scripts must update; today's earlier commit chain has been the only
place using it and that's been refreshed in lock-step.
tests/test_v8_fork_score.py
===========================
- 4 ``parser.parse_args(["v8", "score", ...])`` calls → ``["substrate", ...]``.
- 4 test functions renamed: ``test_cli_v8_score_*`` →
``test_cli_substrate_score_*``.
- Module docstring + section comment + helper docstring updated.
Filename intentionally kept as ``test_v8_fork_score.py`` for git
history continuity; pytest discovers by ``test_*`` content, not
filename. Renaming the file would muddle ``git log --follow`` for
the test surface.
Docs refreshed
==============
- docs/v8-fork-score.md — §5 CLI block invocation.
- docs/_source/v8-fork-score.rst — :code-block:: bash invocation.
- docs/_source/bench.rst — invocation in `### v8 ForkScore` section.
- docs/tickets/ticket-000012-selection-consensus-protocol.md —
three references in §7 close-out + §7 Phase 1c proposal +
§7 future-CLI-shape note.
- docs/dav1dprometheus-update-2026-05-09.md — bench journal mention.
Doc filenames (``v8-fork-score.{md,rst}``) kept stable since they
are URL identities; the file content explains the v8→substrate
rename internally. ``index.rst`` toctree references unchanged.
Hygiene
=======
- ``.venv/bin/arborist substrate score --help`` → 0 + valid usage.
- ``.venv/bin/arborist v8 score`` → exits non-zero (subcommand
removed, surfaced cleanly in ``argparse`` error).
- ``make test`` → 1643 passed, 45 skipped.
- ``make chain-check-shards`` → 0 across all 7 shards.
- fox's parallel work in arborist/qa/{runner,verify}.py +
arborist/qa/warrant_chain.py left untouched.
This commit is contained in:
parent
6c11939cf2
commit
bae5cafe9a
7 changed files with 45 additions and 40 deletions
|
|
@ -3039,8 +3039,8 @@ def _cmd_snapshot_diff(args: argparse.Namespace) -> int:
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def _cmd_v8_score(args: argparse.Namespace) -> int:
|
def _cmd_substrate_score(args: argparse.Namespace) -> int:
|
||||||
"""Compute the v8 ForkScore over (parent, child) bench-result JSON files."""
|
"""Compute the substrate ForkScore over (parent, child) bench-result JSON files."""
|
||||||
from arborist.substrate import (
|
from arborist.substrate import (
|
||||||
bench_result_to_metrics,
|
bench_result_to_metrics,
|
||||||
fork_score,
|
fork_score,
|
||||||
|
|
@ -5127,59 +5127,64 @@ def build_parser() -> argparse.ArgumentParser:
|
||||||
snap_diff.add_argument("snapshot_root", help="hex snapshot_root to diff against current")
|
snap_diff.add_argument("snapshot_root", help="hex snapshot_root to diff against current")
|
||||||
snap_diff.set_defaults(func=_cmd_snapshot_diff)
|
snap_diff.set_defaults(func=_cmd_snapshot_diff)
|
||||||
|
|
||||||
# ----- v8 subcommands (ticket #000012 Phase 1a) ---------------------------
|
# ----- substrate subcommands (ticket #000012 Phase 1a + future) ----------
|
||||||
v8_cmd = sub.add_parser(
|
# Was `arborist v8 score` until 2026-05-10; renamed for naming-consistency
|
||||||
"v8",
|
# with the arborist/substrate/ dir, which is itself the post-rename home
|
||||||
help="Merkle-AGI v8: ForkScore + (future) consensus (ticket #000012)",
|
# of what used to live under arborist/v8/. The ``v`` in ``v8`` referred
|
||||||
|
# to the substrate-paper version, which collided with the v9.8 SQLite
|
||||||
|
# schema version and confused readers.
|
||||||
|
substrate_cmd = sub.add_parser(
|
||||||
|
"substrate",
|
||||||
|
help="Merkle-AGI substrate primitives (ForkScore + future paper specs)",
|
||||||
)
|
)
|
||||||
v8_sub = v8_cmd.add_subparsers(dest="v8_op", required=True)
|
substrate_sub = substrate_cmd.add_subparsers(dest="substrate_op", required=True)
|
||||||
v8_score = v8_sub.add_parser(
|
substrate_score = substrate_sub.add_parser(
|
||||||
"score",
|
"score",
|
||||||
help="ForkScore over (parent, child) bench-result JSON files",
|
help="ForkScore over (parent, child) bench-result JSON files (#000012)",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--parent", required=True,
|
"--parent", required=True,
|
||||||
help="path to parent bench-result JSON (from `bench.batteries.runner --all`)",
|
help="path to parent bench-result JSON (from `bench.batteries.runner --all`)",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--child", required=True,
|
"--child", required=True,
|
||||||
help="path to child bench-result JSON",
|
help="path to child bench-result JSON",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--weights", default=None,
|
"--weights", default=None,
|
||||||
help="optional path to a weights JSON file; falls through to DEFAULT_WEIGHTS",
|
help="optional path to a weights JSON file; falls through to DEFAULT_WEIGHTS",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--capital-delta", dest="capital_delta", type=float, default=0.0,
|
"--capital-delta", dest="capital_delta", type=float, default=0.0,
|
||||||
help="capital cost delta from #000020 ledger; positive = child costs more",
|
help="capital cost delta from #000020 ledger; positive = child costs more",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--selfmodel-calibration-gain",
|
"--selfmodel-calibration-gain",
|
||||||
dest="selfmodel_calibration_gain", type=float, default=0.0,
|
dest="selfmodel_calibration_gain", type=float, default=0.0,
|
||||||
help="SelfModel calibration improvement (parent→child); 0 if unmeasured",
|
help="SelfModel calibration improvement (parent→child); 0 if unmeasured",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--audit-completeness", dest="audit_completeness", type=float, default=0.0,
|
"--audit-completeness", dest="audit_completeness", type=float, default=0.0,
|
||||||
help="fraction of state-changes with audit-event in 0..1",
|
help="fraction of state-changes with audit-event in 0..1",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--validator-diversity", dest="validator_diversity", type=float, default=0.0,
|
"--validator-diversity", dest="validator_diversity", type=float, default=0.0,
|
||||||
help="multi-validator diversity score; 0 in single-validator mode",
|
help="multi-validator diversity score; 0 in single-validator mode",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--security-risk", dest="security_risk", type=float, default=0.0,
|
"--security-risk", dest="security_risk", type=float, default=0.0,
|
||||||
help="reserved; 0 in Phase 1a (no security-bench yet)",
|
help="reserved; 0 in Phase 1a (no security-bench yet)",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--complexity-delta", dest="complexity_delta", type=float, default=0.0,
|
"--complexity-delta", dest="complexity_delta", type=float, default=0.0,
|
||||||
help="reserved; 0 in Phase 1a",
|
help="reserved; 0 in Phase 1a",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--memory-invalidation-count",
|
"--memory-invalidation-count",
|
||||||
dest="memory_invalidation_count", type=float, default=0.0,
|
dest="memory_invalidation_count", type=float, default=0.0,
|
||||||
help="count of memory_records the fork would falsify",
|
help="count of memory_records the fork would falsify",
|
||||||
)
|
)
|
||||||
v8_score.add_argument(
|
substrate_score.add_argument(
|
||||||
"--out", default=None,
|
"--out", default=None,
|
||||||
help=(
|
help=(
|
||||||
"optional output file path; ScoredFork JSON is also written "
|
"optional output file path; ScoredFork JSON is also written "
|
||||||
|
|
@ -5189,7 +5194,7 @@ def build_parser() -> argparse.ArgumentParser:
|
||||||
"/ ForkScore-aware mesh peers can ingest the artifact."
|
"/ ForkScore-aware mesh peers can ingest the artifact."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
v8_score.set_defaults(func=_cmd_v8_score)
|
substrate_score.set_defaults(func=_cmd_substrate_score)
|
||||||
|
|
||||||
# ----- memory subcommands (ticket #000017) --------------------------------
|
# ----- memory subcommands (ticket #000017) --------------------------------
|
||||||
memory_cmd = sub.add_parser(
|
memory_cmd = sub.add_parser(
|
||||||
|
|
|
||||||
|
|
@ -142,7 +142,7 @@ verdict with ACCEPT / MARGINAL / REJECT classes:
|
||||||
make bench-suite # generates parent.json
|
make bench-suite # generates parent.json
|
||||||
# ... apply changes ...
|
# ... apply changes ...
|
||||||
make bench-suite # generates child.json
|
make bench-suite # generates child.json
|
||||||
arborist v8 score --parent parent.json --child child.json
|
arborist substrate score --parent parent.json --child child.json
|
||||||
|
|
||||||
Authoring new fixtures
|
Authoring new fixtures
|
||||||
----------------------
|
----------------------
|
||||||
|
|
|
||||||
|
|
@ -100,7 +100,7 @@ CLI
|
||||||
|
|
||||||
.. code-block:: bash
|
.. code-block:: bash
|
||||||
|
|
||||||
arborist v8 score \
|
arborist substrate score \
|
||||||
--parent parent-bench.json \
|
--parent parent-bench.json \
|
||||||
--child child-bench.json \
|
--child child-bench.json \
|
||||||
[--weights weights.json] \
|
[--weights weights.json] \
|
||||||
|
|
|
||||||
|
|
@ -208,7 +208,7 @@ Three big mechanical artifacts beyond the kernels:
|
||||||
|
|
||||||
### Selection (Merkle-AGI v8 ForkScore)
|
### Selection (Merkle-AGI v8 ForkScore)
|
||||||
|
|
||||||
Phase 1a + 1b live: ScoredFork dataclass + `arborist v8 score`
|
Phase 1a + 1b live: ScoredFork dataclass + `arborist substrate score`
|
||||||
CLI surface + Makefile harness (`bench-fork-baseline`,
|
CLI surface + Makefile harness (`bench-fork-baseline`,
|
||||||
`bench-fork-score`). A weighted score over (parent, child) battery
|
`bench-fork-score`). A weighted score over (parent, child) battery
|
||||||
deltas; verdicts ACCEPT / MARGINAL / REJECT; hard-regression flag
|
deltas; verdicts ACCEPT / MARGINAL / REJECT; hard-regression flag
|
||||||
|
|
|
||||||
|
|
@ -276,7 +276,7 @@ function ahead of the consensus paper:
|
||||||
`DEFAULT_WEIGHTS` (single-validator-tuned) + `from_dict` adapter
|
`DEFAULT_WEIGHTS` (single-validator-tuned) + `from_dict` adapter
|
||||||
handling the `"lambda"`/`lambda_` Python-reserved-word issue.
|
handling the `"lambda"`/`lambda_` Python-reserved-word issue.
|
||||||
(Originally `arborist/v8/weights.py`.)
|
(Originally `arborist/v8/weights.py`.)
|
||||||
- CLI: `arborist v8 score --parent P.json --child C.json
|
- CLI: `arborist substrate score --parent P.json --child C.json
|
||||||
[--weights W.json]`. Exits 1 on REJECT (CI-gateable).
|
[--weights W.json]`. Exits 1 on REJECT (CI-gateable).
|
||||||
- Verdict thresholds: ACCEPT (≥ SIGNAL_FLOOR=0.05), MARGINAL
|
- Verdict thresholds: ACCEPT (≥ SIGNAL_FLOOR=0.05), MARGINAL
|
||||||
([0, SIGNAL_FLOOR)), REJECT (negative score OR hard-regression
|
([0, SIGNAL_FLOOR)), REJECT (negative score OR hard-regression
|
||||||
|
|
@ -349,7 +349,7 @@ CREATE INDEX idx_fork_score_branches_parent ON fork_score_branches(parent_root);
|
||||||
PK is `(branch_set_id, branch_id)` so re-scoring the same fork
|
PK is `(branch_set_id, branch_id)` so re-scoring the same fork
|
||||||
under the same set is a clean upsert, not a duplicate row.
|
under the same set is a clean upsert, not a duplicate row.
|
||||||
|
|
||||||
**2. CLI surface.** Extend `arborist v8 score` with two optional
|
**2. CLI surface.** Extend `arborist substrate score` with two optional
|
||||||
flags:
|
flags:
|
||||||
|
|
||||||
- ``--branch-set <ID>`` — names the checkpoint a result belongs to.
|
- ``--branch-set <ID>`` — names the checkpoint a result belongs to.
|
||||||
|
|
@ -389,7 +389,7 @@ checkpoint and reports `n_branches >= 4`.
|
||||||
- ForkScore + Prometheus-Σ become composable: the controller reads
|
- ForkScore + Prometheus-Σ become composable: the controller reads
|
||||||
a checkpoint's branch set and runs softmax across the persisted
|
a checkpoint's branch set and runs softmax across the persisted
|
||||||
scores instead of needing to re-score from raw BatteryResults.
|
scores instead of needing to re-score from raw BatteryResults.
|
||||||
- A future operator-facing CLI (`arborist v8 branch-set show ID`)
|
- A future operator-facing CLI (`arborist substrate branch-set show ID`)
|
||||||
becomes trivial — same table powers it.
|
becomes trivial — same table powers it.
|
||||||
|
|
||||||
**Why not now**: this is doc-only because Phase 1c is small
|
**Why not now**: this is doc-only because Phase 1c is small
|
||||||
|
|
|
||||||
|
|
@ -155,7 +155,7 @@ cite the changed verifier.
|
||||||
| any | hard-regression OR neg-inf efficiency | **REJECT** |
|
| any | hard-regression OR neg-inf efficiency | **REJECT** |
|
||||||
|
|
||||||
CLI exit code: `0` for ACCEPT/MARGINAL, `1` for REJECT — so CI
|
CLI exit code: `0` for ACCEPT/MARGINAL, `1` for REJECT — so CI
|
||||||
gates can run `arborist v8 score …` directly.
|
gates can run `arborist substrate score …` directly.
|
||||||
|
|
||||||
### 3.2 Hard flags
|
### 3.2 Hard flags
|
||||||
|
|
||||||
|
|
@ -205,7 +205,7 @@ because `lambda` is a Python reserved word.)
|
||||||
## 5. CLI
|
## 5. CLI
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
arborist v8 score \
|
arborist substrate score \
|
||||||
--parent parent-bench.json \
|
--parent parent-bench.json \
|
||||||
--child child-bench.json \
|
--child child-bench.json \
|
||||||
[--weights weights.json] \
|
[--weights weights.json] \
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,7 @@ Covers:
|
||||||
- inf-bonus capped per INFINITE_BONUS_CAP
|
- inf-bonus capped per INFINITE_BONUS_CAP
|
||||||
- Weight tuning changes score additively
|
- Weight tuning changes score additively
|
||||||
- bench_result_to_metrics adapter
|
- bench_result_to_metrics adapter
|
||||||
- CLI smoke (`arborist v8 score`)
|
- CLI smoke (`arborist substrate score`)
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
@ -341,12 +341,12 @@ def test_breakdown_sums_to_score():
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------
|
# ---------------------------------------------------------------------
|
||||||
# CLI smoke (`arborist v8 score`)
|
# CLI smoke (`arborist substrate score`)
|
||||||
# ---------------------------------------------------------------------
|
# ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def _write_bench_result(path: Path, rates: dict) -> None:
|
def _write_bench_result(path: Path, rates: dict) -> None:
|
||||||
"""Build a minimal bench-result JSON the v8 score CLI consumes."""
|
"""Build a minimal bench-result JSON the substrate score CLI consumes."""
|
||||||
results = []
|
results = []
|
||||||
for battery, subs in rates.items():
|
for battery, subs in rates.items():
|
||||||
for sub, metrics in subs.items():
|
for sub, metrics in subs.items():
|
||||||
|
|
@ -366,7 +366,7 @@ def _write_bench_result(path: Path, rates: dict) -> None:
|
||||||
path.write_text(json.dumps(payload), encoding="utf-8")
|
path.write_text(json.dumps(payload), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
def test_cli_v8_score_smoke(tmp_path, capsys):
|
def test_cli_substrate_score_smoke(tmp_path, capsys):
|
||||||
from arborist.cli import build_parser
|
from arborist.cli import build_parser
|
||||||
|
|
||||||
parent_path = tmp_path / "parent.json"
|
parent_path = tmp_path / "parent.json"
|
||||||
|
|
@ -382,7 +382,7 @@ def test_cli_v8_score_smoke(tmp_path, capsys):
|
||||||
|
|
||||||
parser = build_parser()
|
parser = build_parser()
|
||||||
args = parser.parse_args([
|
args = parser.parse_args([
|
||||||
"v8", "score",
|
"substrate", "score",
|
||||||
"--parent", str(parent_path),
|
"--parent", str(parent_path),
|
||||||
"--child", str(child_path),
|
"--child", str(child_path),
|
||||||
])
|
])
|
||||||
|
|
@ -396,7 +396,7 @@ def test_cli_v8_score_smoke(tmp_path, capsys):
|
||||||
assert rc in (0, 1)
|
assert rc in (0, 1)
|
||||||
|
|
||||||
|
|
||||||
def test_cli_v8_score_with_explicit_weights(tmp_path, capsys):
|
def test_cli_substrate_score_with_explicit_weights(tmp_path, capsys):
|
||||||
from arborist.cli import build_parser
|
from arborist.cli import build_parser
|
||||||
|
|
||||||
parent_path = tmp_path / "parent.json"
|
parent_path = tmp_path / "parent.json"
|
||||||
|
|
@ -414,7 +414,7 @@ def test_cli_v8_score_with_explicit_weights(tmp_path, capsys):
|
||||||
|
|
||||||
parser = build_parser()
|
parser = build_parser()
|
||||||
args = parser.parse_args([
|
args = parser.parse_args([
|
||||||
"v8", "score",
|
"substrate", "score",
|
||||||
"--parent", str(parent_path),
|
"--parent", str(parent_path),
|
||||||
"--child", str(child_path),
|
"--child", str(child_path),
|
||||||
"--weights", str(weights_path),
|
"--weights", str(weights_path),
|
||||||
|
|
@ -425,7 +425,7 @@ def test_cli_v8_score_with_explicit_weights(tmp_path, capsys):
|
||||||
assert payload["weights"]["lambda_"] == 2.0
|
assert payload["weights"]["lambda_"] == 2.0
|
||||||
|
|
||||||
|
|
||||||
def test_cli_v8_score_rejects_returns_nonzero(tmp_path, capsys):
|
def test_cli_substrate_score_rejects_returns_nonzero(tmp_path, capsys):
|
||||||
"""REJECT verdict → exit code 1 so CI can gate."""
|
"""REJECT verdict → exit code 1 so CI can gate."""
|
||||||
from arborist.cli import build_parser
|
from arborist.cli import build_parser
|
||||||
|
|
||||||
|
|
@ -441,7 +441,7 @@ def test_cli_v8_score_rejects_returns_nonzero(tmp_path, capsys):
|
||||||
})
|
})
|
||||||
parser = build_parser()
|
parser = build_parser()
|
||||||
args = parser.parse_args([
|
args = parser.parse_args([
|
||||||
"v8", "score", "--parent", str(parent_path), "--child", str(child_path),
|
"substrate", "score", "--parent", str(parent_path), "--child", str(child_path),
|
||||||
])
|
])
|
||||||
rc = args.func(args)
|
rc = args.func(args)
|
||||||
assert rc == 1
|
assert rc == 1
|
||||||
|
|
@ -449,7 +449,7 @@ def test_cli_v8_score_rejects_returns_nonzero(tmp_path, capsys):
|
||||||
assert payload["verdict"] == "REJECT"
|
assert payload["verdict"] == "REJECT"
|
||||||
|
|
||||||
|
|
||||||
def test_cli_v8_score_out_writes_json_artifact(tmp_path, capsys):
|
def test_cli_substrate_score_out_writes_json_artifact(tmp_path, capsys):
|
||||||
"""#000012 Phase 1b — `--out` mirrors stdout to a file so CI /
|
"""#000012 Phase 1b — `--out` mirrors stdout to a file so CI /
|
||||||
mesh peers / downstream graders can ingest the artifact without
|
mesh peers / downstream graders can ingest the artifact without
|
||||||
parsing pipe output."""
|
parsing pipe output."""
|
||||||
|
|
@ -468,7 +468,7 @@ def test_cli_v8_score_out_writes_json_artifact(tmp_path, capsys):
|
||||||
})
|
})
|
||||||
parser = build_parser()
|
parser = build_parser()
|
||||||
args = parser.parse_args([
|
args = parser.parse_args([
|
||||||
"v8", "score",
|
"substrate", "score",
|
||||||
"--parent", str(parent_path),
|
"--parent", str(parent_path),
|
||||||
"--child", str(child_path),
|
"--child", str(child_path),
|
||||||
"--out", str(out_path),
|
"--out", str(out_path),
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue