arborist/docs/tickets/ticket-000012-selection-consensus-protocol.md

Ticket #000012 — Selection & consensus protocol (Merkle-AGI v8)

Status: in progress · Phase 1a (ForkScore) landed 2026-05-08; consensus paper still open Opened: 2026-05-07 Scope: Spec for the loop-closing consensus protocol that turns the v7 substrate + v9.8 runtime into actual Darwinian selection across multiple validators. Covers fitness scoring, mutation acceptance, fork-choice on disagreement, Sybil resistance, validator economics. Doc-only — no code in this ticket; this is the design substrate paper that follow-up implementation tickets will reference. Audience: fox + future blackops shifts + downstream Merkle-AGI v8 authors. Hard constraint: no per-query consensus. Selection runs at checkpoint cadence (Proof-of-Upgrade scope), not at inference time. v9.8 cache_key invariant stays at 8 dims; consensus-state lives in a sibling table, not folded into the answer-cache.

---

1. Problem statement

The DNA ↔ Merkle-DAG analogy promises:

copy → vary → express → test → select → preserve → repeat

Merkle-AGI v7 ships everything except select. Section 13.4 ("Proof-of-Upgrade") sketches a procedure where a candidate model (M', C(M')) is admitted if it shows non-regression on a published eval set + non-regression of ε-coverage at sentinel frontiers. That procedure is single-validator regression testing with a Merkle receipt. It is not consensus.

Specifically v7 § 13.4 leaves these gaps:

Gap	Concrete failure under current spec
Validator set discovery	No protocol. Two labs reach different verdicts on (M'); no fork-choice.
Byzantine fault tolerance	One dishonest validator can sign acceptance for a poisoned (M').
Sybil resistance	A single actor spinning up 50 validators wins every quorum.
Fork-choice on disagreement	If validators V1, V2 publish conflicting acceptance receipts, downstream nodes have no rule to pick.
Liveness vs safety trade	No bound on how long acceptance can stall when validators are offline.
Validator incentives	Why would anyone run a validator? What's the slashing condition for cheating?
Stake / membership semantics	Permissionless? Permissioned? Hybrid? v7 silent.

Without a protocol that closes these, "digital evolution" is metaphor — a single lab signing its own upgrades, no different in trust model from current model-card releases.

1.1 Concrete failure scenario

Lab A trains M_t → M_{t+1} with a backdoor that triggers on a rare input pattern. Lab A signs Proof-of-Upgrade: ε-coverage non- regressed at every published frontier (because the backdoor lives at a non-published frontier). Lab A publishes C(M_{t+1}). Anyone pulling the registry sees an "accepted" upgrade. v7 has no way to surface that no independent validator audited the upgrade.

The selection protocol must make "accepted under v8" mean "accepted by N independent validators meeting policy P," verifiable to anyone, not "Lab A signed it."

---

2. Design choices

2.1 Validator set: permissionless vs permissioned

A. Permissionless (Bitcoin-style). Anyone with stake (or proof-of-work or proof-of-storage) can validate. Maximum censorship resistance. Costs: economic incentive design, possible centralization through mining/staking concentration, latency.

B. Permissioned (consortium). Validator set is curated by a governance body. Easier to bootstrap, easier to slash, lower latency. Costs: who curates? captures regulatory risk; "patch the planet" mission frowns on gatekeepers.

C. Hybrid (delegated proof-of-stake-like). Permissionless participation but stake required; misbehavior slashable. Common middle-ground (Tezos / Cosmos). Bootstrappable and Sybil-resistant.

Recommendation: Hybrid. Permissionless joining with stake is compatible with permacomputer values (no gatekeepers) and Sybil- resistant in practice. Bootstrap from a small honest set with explicit graduation criteria.

2.2 Fitness function: who defines it?

A. Lab-defined (per upgrade). Submitter declares the metric set and eval digests; validators check non-regression. Flexible, gameable.

B. Registry-defined (canonical bench suite). A single canonical suite gates every upgrade. Simple, brittle, hard to evolve.

C. Layered (canonical floor + lab-declared ceiling). Every upgrade must non-regress on the canonical floor; lab can additionally declare metrics they want validated. Default-on safety; allows specialization.

Recommendation: C (Layered). Canonical floor is the safety substrate every release passes through. Layered specialization keeps domain models from being stuck behind irrelevant gates.

2.3 Quorum rule

A. Simple majority (51%). Liveness-friendly. Vulnerable to slim majorities and hostile takeovers.

B. Supermajority (2/3 +). Standard BFT bound. Tolerates 1/3 Byzantine. Tighter than majority, slower under partition.

C. Threshold signature (k-of-n). Cryptographic accumulator; single signature represents quorum. Cheap verification downstream; needs ceremony to mint signing key.

Recommendation: B for safety floor, with threshold signature (C) as a downstream optimization once the protocol stabilizes. Tolerates the canonical 1/3 Byzantine fraction without giving up liveness on small disagreements.

2.4 Slashing condition

A validator that signs acceptance for a candidate that subsequently fails the canonical floor's audit replay loses stake. A validator that signs conflicting acceptances (forks) loses stake. A validator that double-signs (signs both accept and reject for same C(M')) loses stake.

This requires:

• An audit-replay protocol that re-runs the canonical floor and publishes a Merkle-bound result.
• A challenge window during which any party can submit a counter-receipt invalidating an earlier acceptance.
• Time-locked stake unbonding so a validator can't sign and exit before challenges land.

2.5 Fork choice

When two valid acceptance chains diverge, downstream nodes must pick one. Options:

A. Longest valid chain. Bitcoin-style. Vulnerable to deep reorgs.

B. Highest-stake-weighted acceptance. Eth-style finality. Resistant to short-range reorgs.

C. First-finalized wins. GRANDPA-style. Once 2/3+ stake signs, no reorg.

Recommendation: C. Once 2/3+ validators finalize an upgrade, it's permanent. Latency is acceptable for checkpoint-cadence selection (not inference).

---

3. Recommendation

Hybrid permissionless validator set with stake, layered fitness floor + per-upgrade ceiling, supermajority quorum (2/3+), slashing on audit-replay disagreement and equivocation, GRANDPA-style fork choice. Bootstrap from a small honest set with explicit slashing window before opening to permissionless joining.

---

4. Implementation sketch

This ticket commissions Merkle-AGI v8 as a sister paper to v7. v8 must specify:

1. Validator state machine.
   • States: bonding, active, challenged, slashed, unbonding.
   • Transitions: stake deposit, signature, challenge, slash, exit.
2. Acceptance protocol.
   • Proposer submits (C(M'), eval_digest, frontier_coverage_diff, metric_delta_signed).
   • Validators run audit-replay, sign accept/reject within window.
   • Aggregate signature minted at quorum.
3. Challenge protocol.
   • Anyone submits (C(M'), counter_evidence) within challenge window.
   • If counter-evidence verifies (re-runs canonical floor and finds regression), all signing validators are slashed.
4. Fork choice rule.
   • Validators only sign on candidates whose parent C(M_t) is finalized.
   • Once 2/3+ stake signs C(M_{t+1}), it's finalized.
5. Stake mechanics.
   • Bond / unbond windows.
   • Slashing fraction per offense class.
   • Reward distribution per honest signature.
6. Mesh wire format.
   • Extension to arborist mesh/wire.py for validator gossip.
   • Aggregate signature canonicalization (so verification is stake-weight-independent).

The paper itself is the ticket-#000012 deliverable. Code lives in follow-up tickets that cite this one.

4.1 Concrete artifacts this ticket produces

• docs/merkle-agi-v8-consensus.rst — sister to the v7 substrate paper. Sections: validator state machine, acceptance, challenge, fork choice, slashing, mesh wire format, BFT analysis.
• docs/v8-policy-fields.md — the policy fields v8 introduces and how they fold into governance_policy_hash (or whether they live in a sibling consensus_policy_hash).
• A worked-example Merkle-AGI v8 acceptance ledger reflecting one imaginary upgrade cycle (paper's appendix; does not require running validators).

4.2 What does not change

• v9.8 8-dim cache_key. Consensus state lives in consensus_events (sibling table), not in cache_key.
• arborist's per-shard audit chain. v8 is checkpoint-cadence, cross-validator; per-shard audit chain stays per-node.
• Existing falsification-state semantics (live, failed, stale, quarantined).

---

5. Out of scope

• Implementation of the v8 protocol in code. That is at minimum 3-4 follow-up tickets (validator state machine, mesh wire extension, audit-replay harness, slashing accountant).
• Economic parameter calibration (stake amounts, slashing fractions, reward rates). v8 paper specifies the form; calibration is a governance decision.
• Cross-chain anchoring (publishing v8 finalizations to Bitcoin / Ethereum / etc.). Optional bolt-on.
• Selection of frontier benchmark fixtures (covered by ticket #000021).

---

6. Risks & open questions

• Liveness vs censorship. A validator set that requires 2/3+ to finalize stalls under 1/3 hostile partition. Acceptable for checkpoint cadence; needs explicit liveness floor in the spec.
• Bootstrap honesty. The initial validator set must be honest for the protocol to converge. Solution: explicit bootstrap window with permissioned set + scheduled transition to permissionless.
• Stake captures. Large staker can dominate. Mitigation: cap on individual stake weight, or convex weighting (sqrt-stake).
• Re-staking attacks. Validators staking the same capital across multiple v8 instances. Out of scope here; addressed by cross-instance slashing accumulator if/when v8 multiplies.

---

7. Status

In progress · Phase 1a landed 2026-05-08.

Phase 1a — ForkScore (landed)

Per fox's 2026-05-08 review note that the substrate is "good enough to support v8 selection/fork-choice design," shipping the scoring function ahead of the consensus paper:

• arborist/v8/fork_score.py — pure ScoredFork dataclass + scoring function over (parent, child) BatteryResult bundles. Consumes every metric this session shipped: 5S/5T/5F sub-battery rates, adaptation_efficiency_* and feedback_efficiency_* (incl. inf-aware aggregation per fbd99a8 review), capital-cost delta, memory-invalidation count.
• arborist/v8/weights.py — WeightSet dataclass with α…λ + DEFAULT_WEIGHTS (single-validator-tuned) + from_dict adapter handling the "lambda"/lambda_ Python-reserved-word issue.
• CLI: arborist v8 score --parent P.json --child C.json [--weights W.json]. Exits 1 on REJECT (CI-gateable).
• Verdict thresholds: ACCEPT (≥ SIGNAL_FLOOR=0.05), MARGINAL ([0, SIGNAL_FLOOR)), REJECT (negative score OR hard-regression flag OR NEG_INF_REGRESSION flag).
• Reference doc: docs/v8-fork-score.md.
• Tests: tests/test_v8_fork_score.py — 25 cases covering the adapter, all verdict paths, hard-regression detection, inf-bonus capping, neg-inf rejection, weight-set tuning, breakdown completeness, CLI smoke. Full suite: 1186 passed.

Phase 1b — Consensus paper (still open)

Closure criterion: docs/merkle-agi-v8-consensus.rst lands with validator state machine, acceptance protocol, challenge protocol, fork-choice rule (GRANDPA-style), slashing mechanics, mesh wire format extension. Per §1 of this ticket, arborist/v8/fork_score.py is the substrate the paper cites; the paper itself is still research-scope.

What's deliberately NOT in Phase 1a:

• Validator state machine (bonding / signing / slashing).
• Acceptance protocol.
• Challenge protocol.
• Fork-choice rule.
• Mesh wire format extensions.
• Stake mechanics + economic incentives.
• Cross-validator ZK proof exchange.

These belong to the v8 paper itself.