From fd643fe1a50a48efb2f78ead2469c6ef1fe704e4 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 10 May 2026 16:12:00 -0400 Subject: [PATCH] #000012 Phase 1b: Merkle-AGI v8 consensus paper landed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `docs/_source/merkle-agi-v8-consensus.rst` (834 lines, RST sister to the v7-W substrate paper at the same path). Closes Phase 1b of ticket #000012 — the loop-closing consensus protocol that turns single-validator Proof-of-Upgrade into Darwinian selection across an open validator set. 11 parts: Part 1 Introduction & motivation — gap table from v7 § 13.4, concrete backdoor-attack scenario, paper IS/IS-NOT scope. Part 2 Substrate definition — SQD A1/A2/A3 inheritance, consensus_events row schema, consensus_policy_hash sibling (never enters cache_key). Part 3 Validator state machine — bonding/active/challenged/ slashed/unbonding with full transition graph + invariants. Part 4 Acceptance protocol — proposer submission, layered fitness floor (canonical + lab-declared ceiling), audit-replay procedure, 2/3-stake quorum + GRANDPA- style finalization, liveness floor. Part 5 Challenge protocol — counter-evidence shape, adjudication, challenger reward, frivolous-challenge bond. Part 6 Stake mechanics — bond/unbond/challenge window recommendations, offense-class slashing schedule, reward distribution, optional stake cap + sqrt-weighting. Part 7 Fork choice rule — GRANDPA-style finality, pre-finality constraints, liveness recovery. Part 8 Mesh wire format extension — three new message kinds, BLS-or-concat aggregate signatures, bandwidth profile. Part 9 BFT analysis — safety, liveness, Sybil resistance, bootstrap honesty, re-staking attacks. Part 10 Worked example — 7-validator deployment, one upgrade cycle with successful challenge against one fraudulent validator. Part 11 Out of scope — implementation, calibration, cross-chain anchoring, fixture selection, bootstrap-set membership, cross-instance slashing accumulator, branch-set persistence. Closure §: open questions tracked separately (initial validator set composition, threshold-key ceremony, ZK-replay, policy-hash transition mechanics). Ticket #000012 status updated; Phase 1c (branch-set persistence) remains proposed-not-opened. Implementation follow-up tickets that cite this paper land later — one per validator-state-machine, mesh-wire-format extension, audit-replay harness, slashing accountant. --- docs/TICKETS.md | 2 +- docs/_source/merkle-agi-v8-consensus.rst | 834 ++++++++++++++++++ ...ket-000012-selection-consensus-protocol.md | 62 +- 3 files changed, 880 insertions(+), 18 deletions(-) create mode 100644 docs/_source/merkle-agi-v8-consensus.rst diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 5f53003..a85b194 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -105,7 +105,7 @@ Newest first. Update on every open/close. | #000015 | π* domain library + cross-domain composition | closed · landed 2026-05-07 | 2026-05-07 | — | | #000014 | SelfModel: schema, falsification, integration | closed · landed 2026-05-07 | 2026-05-07 | — | | #000013 | Spatial-temporal substrate (Merkle-AGI v7-W) | closed · landed 2026-05-09 (substrate paper + frontier catalog + namespace stub) | 2026-05-07 | — | -| #000012 | Selection & consensus protocol (Merkle-AGI v8) | in progress · Phase 1a + 1b (ForkScore CLI) landed 2026-05-09 | 2026-05-07 | — | +| #000012 | Selection & consensus protocol (Merkle-AGI v8) | in progress · Phase 1a (ForkScore) landed 2026-05-08; Phase 1b (consensus paper, `docs/_source/merkle-agi-v8-consensus.rst` 834 lines) landed 2026-05-10; Phase 1c (branch-set persistence) remains proposed-not-opened | 2026-05-07 | — | | #000011 | SOFT_PREFLIGHT_HINT model-assisted sidecar | closed · landed 2026-05-04 (zero-shot full impl) | 2026-05-04 | D1 (preserves) | | #000010 | Meta-Cognition Preflight Guard (M0 / MCTL) | closed · landed 2026-05-03 (Phases 1–4); DAG binding shipped via #000009 | 2026-05-03 | D1, D3 | | #000009 | Preflight run-DAG node binding (#000008+#000010) | closed · re-landed 2026-05-04 (§8 corrections: reject-path DAG, nested CTI clauses) | 2026-05-03 | D3, D4 | diff --git a/docs/_source/merkle-agi-v8-consensus.rst b/docs/_source/merkle-agi-v8-consensus.rst new file mode 100644 index 0000000..e304a25 --- /dev/null +++ b/docs/_source/merkle-agi-v8-consensus.rst @@ -0,0 +1,834 @@ +Merkle-AGI v8 — Selection & Consensus Protocol +=============================================== + +:Author: fox + agent blackops, on the unsandbox / unturf / permacomputer platform +:Date: 2026-05-10 (draft v0) +:Status: substrate paper for ticket #000012 Phase 1b; closure draft. + +This document specifies the **selection layer** of the Merkle-AGI +lineage — sister to v7 (logic/math substrate; plastic training) and +v7-W (spatial-temporal substrate). v8 commits the loop-closing step +that turns single-validator Proof-of-Upgrade into Darwinian +selection across an open validator set. + +Read alongside: + +- ``docs/v8-fork-score.md`` — canonical formula reference for + ``arborist.substrate.fork_score`` (v8 Phase 1a; landed 2026-05-08). + The ScoredFork function this paper invokes for fitness scoring. +- ``docs/_source/merkle-agi-v7w-spatial-temporal.rst`` — sister + substrate paper; same RST conventions; same audience. +- The v7 substrate spec + (``~/Downloads/merkle-agi-dag_v7.txt`` § 13.4) — the Proof-of- + Upgrade procedure v8 extends from single-validator to multi- + validator consensus. + + +Part 1 — Introduction & motivation +---------------------------------- + +§1.1 The DNA ↔ Merkle-DAG loop +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The substrate-lineage analogy is: + +.. code-block:: text + + copy → vary → express → test → select → preserve → repeat + +Merkle-AGI v7 ships everything except ``select``. Section 13.4 +("Proof-of-Upgrade") sketches a procedure where a candidate model +``(M', C(M'))`` is admitted if it shows non-regression on a +published eval set + non-regression of ε-coverage at sentinel +frontiers. That procedure is **single-validator regression testing +with a Merkle receipt**. It is not consensus. + +§1.2 The selection gap under v7 § 13.4 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The unaddressed gaps from v7 § 13.4, each with a concrete failure +under the current spec: + +================================ ============================================ +Gap Concrete failure +================================ ============================================ +Validator set discovery Two labs reach different verdicts on M'; no + fork-choice rule for downstream consumers. +Byzantine fault tolerance One dishonest validator can sign acceptance + for a poisoned (M'). +Sybil resistance A single actor spinning up 50 validators + wins every quorum. +Fork-choice on disagreement If V1, V2 publish conflicting acceptance + receipts, downstream nodes have no rule. +Liveness vs safety trade No bound on how long acceptance can stall + when validators are offline. +Validator incentives Why would anyone run a validator? What's + the slashing condition for cheating? +Stake / membership semantics Permissionless? Permissioned? Hybrid? + v7 silent. +================================ ============================================ + +Without a protocol that closes these, "digital evolution" is +metaphor — a single lab signing its own upgrades, no different in +trust model from current model-card releases. + +§1.3 Concrete attack scenario (motivating example) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Lab A trains ``M_t → M_{t+1}`` with a backdoor that triggers on a +rare input pattern. Lab A signs Proof-of-Upgrade: ε-coverage +non-regressed at every published frontier (because the backdoor +lives at a non-published frontier). Lab A publishes ``C(M_{t+1})``. +Anyone pulling the registry sees an "accepted" upgrade. v7 has no +way to surface that no independent validator audited the upgrade. + +The v8 selection protocol must make "accepted under v8" mean +"accepted by N independent validators meeting policy P", verifiable +to anyone, not "Lab A signed it." + +§1.4 What this paper IS and IS NOT +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +**IS**: + +- A protocol specification: validator state machine, acceptance + ledger, challenge window, fork-choice rule, slashing schedule, + mesh wire format extension. +- A BFT-analysis sketch with explicit safety/liveness bounds. +- A worked-example acceptance cycle (appendix) reflecting one + imaginary upgrade. + +**IS NOT**: + +- An implementation. Code lands in follow-up tickets that cite this + paper. +- An economic calibration. Stake amounts, slashing fractions, and + reward rates are governance decisions; the paper specifies their + shape, not their magnitude. +- A cross-chain anchoring spec. Publishing v8 finalizations to + Bitcoin / Ethereum / etc. is an optional bolt-on; not in this + paper. + + +Part 2 — Substrate definition +----------------------------- + +§2.1 Hard constraints (carried from #000012) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- Stays inside SQD A1 (canonical encoding), A2 (public + quantization), A3 (collision-resistant hash). v8 introduces no + new cryptographic axiom. +- No per-query consensus. Selection runs at checkpoint cadence + (Proof-of-Upgrade scope), not at inference time. +- v9.8 ``cache_key`` invariant stays at 8 dimensions. Consensus + state lives in a sibling ``consensus_events`` table, not folded + into the answer-cache. +- arborist's per-shard audit chain stays per-node. v8 is + checkpoint-cadence, cross-validator; nothing in v8 modifies + ``audit_events.event_hash`` preimage on individual shards. +- Existing falsification-state semantics + (``live``, ``failed``, ``stale``, ``quarantined``) stay + unchanged. v8 reuses them for validator state, not for cache + state. + +§2.2 What v8 commits +~~~~~~~~~~~~~~~~~~~~ + +v8 commits the **acceptance ledger**: a Merkle-chained sequence of +finalized ``(C(M'), proposer_id, validator_signatures, audit_replay +_digest, fork_score_breakdown)`` rows. Each row's +``event_hash`` is the SHA-256 of the canonical encoding of the row +body chained against the previous row's ``event_hash``. + +.. code-block:: text + + consensus_events.event_hash = + SHA-256( prev_event_hash || canonical_encoding(body) ) + + body = { + "candidate_root": "", + "parent_root": "", + "proposer_id": "", + "eval_digest": "", + "frontier_coverage_diff": "<ε-coverage delta blob>", + "fork_score": {}, + "validator_signatures": [], + "audit_replay_digest": "", + "verdict": "ACCEPTED" | "REJECTED" | "CHALLENGED", + "finalized_at": + } + +The canonical encoding follows the SQD A1 convention (sorted JSON +keys, UTF-8, no whitespace) so two independent serializers produce +byte-identical bytes. + +§2.3 Sibling-table semantics (no `cache_key` pollution) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Per ticket #000012 §4.2 hard constraint: consensus state never +enters ``providence_cache.cache_key``. The 8-dim invariant +(``source_root | question_hash | model_profile_hash | +conversation_hash | governance_policy_hash | schema_version | +canonicalization_version | chunking_version``) is unchanged. + +v8 introduces a **ninth-conceptual** invariant — the +**consensus_policy_hash** — but folds it into a sibling table, not +``cache_key``. The motivation: per-query cache invalidation should +not fire every time a validator joins or leaves the set. Validator- +set churn is checkpoint-scale, not query-scale. + +The two policy hashes are deliberately split: + +- ``governance_policy_hash`` — folds answer-mode, verifier method, + claim caps, broad-quantifier guard config. Per-query; folds into + ``cache_key``. +- ``consensus_policy_hash`` — folds validator set, quorum + threshold, slashing schedule, challenge window. Per-checkpoint; + folds into ``consensus_events`` row but NEVER ``cache_key``. + +A consumer that wants to verify "this answer was produced under +finalized policy P at checkpoint T" walks the ``consensus_events`` +chain to find P, then walks the per-shard ``audit_events`` chain to +find the answer. + + +Part 3 — Validator state machine +-------------------------------- + +§3.1 States +~~~~~~~~~~~ + +A validator transitions through five states across its lifetime: + +================ ================================================ +State Meaning +================ ================================================ +``bonding`` Stake deposit pending; cannot sign yet. Bond + window must elapse before promotion. +``active`` Eligible to sign acceptance / rejection on + proposed candidates. Earns rewards on honest + signatures. +``challenged`` A counter-evidence submission claims one of this + validator's prior signatures was fraudulent. Stake + is frozen pending challenge resolution. +``slashed`` Challenge resolved against the validator. Stake + partially or fully forfeit per offense-class + schedule (§6.2). Validator cannot rejoin under the + same pubkey. +``unbonding`` Validator initiated voluntary exit. Stake is + time-locked for an unbond window so outstanding + challenges can land before withdrawal. +================ ================================================ + +§3.2 Transitions +~~~~~~~~~~~~~~~~ + +.. code-block:: text + + stake_deposit + (start) ─────────────────────────► bonding + │ + bond_window elapsed + │ + ▼ + ┌──────────────────────────────► active ───────────────────────┐ + │ │ │ + │ stake withdraw │ + │ initiated │ + │ │ │ + │ ▼ │ + │ unbonding │ + │ │ │ + │ unbond_window elapsed │ + │ │ │ + │ ▼ │ + │ (exit) │ + │ │ + │ challenge_submitted │ + │ │ + ▼ │ + challenged │ + │ │ + │ challenge_resolved_in_favor │ + └──────────────────────────────────────────────────────────┘ + │ + │ challenge_resolved_against + ▼ + slashed + │ + │ (terminal — same pubkey cannot rejoin) + │ + ▼ + +§3.3 Invariants +~~~~~~~~~~~~~~~ + +- A validator in ``slashed`` state remains in the chain forever + under its original pubkey; rejoin requires a fresh keypair and + fresh stake. +- ``challenged`` is a **transient** state. The challenge window + bounds resolution time (§4.4); ambiguous challenges resolve + in favor of the validator by default (presumption of innocence). +- ``unbonding`` blocks signing. A validator that has initiated exit + cannot sign new acceptances; this prevents the + sign-then-exit-then-let-challenge-bounce attack. +- Bond and unbond windows are governance parameters in + ``consensus_policy_hash``; changing them invalidates the policy + hash and therefore which ``consensus_events`` rows verify under + it. + + +Part 4 — Acceptance protocol +---------------------------- + +§4.1 Proposer submission +~~~~~~~~~~~~~~~~~~~~~~~~ + +A proposer (any actor — does not need to be a validator) submits: + +.. code-block:: json + + { + "candidate_root": "", + "parent_root": "", + "eval_digest": "", + "frontier_coverage_diff": "<ε-coverage delta blob>", + "fork_score_self_reported": {}, + "proposer_signature": "" + } + +The proposer is bound to the submission by their signature; this +prevents anonymous spam. The self-reported ``fork_score`` is +*advisory* — validators re-compute it independently during audit- +replay (§4.3). + +§4.2 Layered fitness floor +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Per ticket #000012 §2.2 (Recommendation C), every candidate must +clear a **canonical floor** before validators are obligated to +audit. The canonical floor is published in the v8 substrate paper +and pinned in ``consensus_policy_hash``: + +================================ ==================================== +Floor metric Threshold +================================ ==================================== +``Δ5S`` (5-syndrome battery) ≥ 0 (no regression) +``Δ5T`` (5-transfer battery) ≥ 0 (no regression) +``Δ5F`` (5-falsification battery) ≥ 0 (no regression) +ε-coverage at published frontiers ≥ floor specified per-frontier +``audit_completeness`` = 1.0 (every state-change row hashes + through ``audit_events``) +``hard_regression_flag`` = false (no ScoredFork hard regress) +================================ ==================================== + +A proposer can additionally declare lab-specific ceiling metrics +(e.g., "this upgrade is for combinatorial reasoning; please verify +combinatorics@v1 round-trip rate"). Validators that audit the +ceiling metrics earn bonus rewards (§6.3); ceiling failures do not +trigger slashing. + +§4.3 Audit replay +~~~~~~~~~~~~~~~~~ + +Each ``active`` validator independently: + +1. Pulls ``C(M')`` and verifies its content-address matches the + submitted root. +2. Runs the canonical floor benchmarks against ``M'``. This is a + deterministic procedure pinned by ``eval_digest`` — two honest + validators MUST produce byte-identical replay digests. +3. Computes ``ScoredFork(parent=M_t, child=M')`` using the + ``arborist.substrate.fork_score`` reference implementation. +4. Compares its replay against the proposer's self-reported + ``fork_score``. Discrepancy ≥ 5pp (per the SIGNAL_FLOOR convention + from ``docs/bench-maxing.md``) flags a soft mismatch; ≥ 20pp or + any hard-regression-flag flip triggers automatic rejection. +5. Signs ``accept`` or ``reject`` with its validator key, attaching + its replay digest. + +§4.4 Quorum & finalization +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The signing window is bounded (governance parameter; recommend +24h). Within the window, signatures are gossiped via the mesh wire +format (§7). When ≥ 2/3 of active stake signs ``accept`` AND no +``reject`` exceeds 1/3, the candidate finalizes: + +.. code-block:: text + + accept_stake ≥ (2/3) · total_active_stake + AND reject_stake < (1/3) · total_active_stake + ⇒ FINALIZE(C(M')) at current consensus_events tip + +Finalization mints an aggregate signature (BLS-style if a threshold +key was bootstrapped, otherwise concatenated individual sigs) and +writes one row to ``consensus_events`` chained against the previous +tip. Per §2.3 the row body is canonical-encoded; ``event_hash`` is +SHA-256 of ``prev_event_hash || canonical(body)``. + +§4.5 Liveness floor +~~~~~~~~~~~~~~~~~~~ + +If the signing window elapses without quorum, the candidate is +marked ``CHALLENGED-OR-STALLED`` and re-proposed at the next +checkpoint. Repeated failures (governance parameter; recommend 3) +trigger automatic rejection without slashing — partition-tolerance +floor. + + +Part 5 — Challenge protocol +--------------------------- + +§5.1 Challenge window +~~~~~~~~~~~~~~~~~~~~~ + +After finalization, a **challenge window** opens (governance +parameter; recommend 7 days). Anyone — validator or not — may +submit counter-evidence claiming one of the signing validators +misreported its audit-replay digest. + +§5.2 Counter-evidence shape +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: json + + { + "challenged_candidate_root": "", + "challenged_validator_pubkey": "", + "challenger_replay_digest": "", + "discrepancy_evidence": "", + "challenger_signature": "" + } + +§5.3 Resolution +~~~~~~~~~~~~~~~ + +The challenge is resolved by the **next active validator set** +(excluding the challenged validator): + +1. Each adjudicating validator runs its own audit-replay on + ``C(M')`` and compares against both the challenged digest and + the challenger digest. +2. ≥ 2/3 of the adjudicating set agree on which side is honest → + challenge resolves in favor of that side. +3. If the challenged digest disagrees with the adjudicators by + ≥ 5pp, the challenged validator is slashed per the offense-class + schedule (§6.2). All other validators that signed the same + ``accept`` are also re-audited; if their replay disagrees too, + they too are slashed. + +§5.4 Challenger reward +~~~~~~~~~~~~~~~~~~~~~~ + +A successful challenger earns a slice of the slashed stake +(governance parameter; recommend 30%). This is a positive incentive +for independent audit; without it, the challenge protocol decays. + +§5.5 Frivolous challenges +~~~~~~~~~~~~~~~~~~~~~~~~~ + +If the adjudicating set rules the challenge frivolous (challenger +digest matches the challenged digest within 5pp), the challenger +forfeits a small **challenge bond** (governance parameter; recommend +0.5% of the slashing-fraction-equivalent). Without this, an +adversary spams challenges to grief validators. + +Acceptance row promotion to ``finalized`` requires the challenge +window to elapse without successful challenge. Until then the row +sits in ``finalized-pending`` state; consumers SHOULD wait for the +window to close before relying on the candidate. + + +Part 6 — Stake mechanics +------------------------ + +§6.1 Bond & unbond windows +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +================ =================================== =========== +Window Recommended duration Rationale +================ =================================== =========== +``bond`` 7 days Time for + fraud-on- + stake-deposit + challenges. +``unbond`` 2 × challenge_window Outstanding + challenges + must land + before exit. +``challenge`` 7 days Independent + audit + feasibility + per fox's + review note. +================ =================================== =========== + +§6.2 Slashing schedule +~~~~~~~~~~~~~~~~~~~~~~ + +================================ ================================== +Offense class Slashing fraction (recommended) +================================ ================================== +Audit-replay digest mismatch 30% of stake +Double-signing (accept + reject) 50% of stake +Conflicting acceptances (fork) 100% of stake +Equivocation under challenge 100% of stake +================================ ================================== + +Fractions are *recommended*; calibration is a governance decision. +The paper pins the **shape** (single-offense → partial slash; +fork-equivocation → full slash) and the **monotonic ordering** (a +heavier offense always slashes ≥ a lighter offense's fraction). + +§6.3 Reward distribution +~~~~~~~~~~~~~~~~~~~~~~~~ + +================================================ ==================== +Action Reward (recommended) +================================================ ==================== +Honest accept signature on finalized candidate proportional to stake +Honest reject signature on rejected candidate proportional to stake +Lab-ceiling-metric audit (bonus) flat per audit +Successful challenge (slash recipient) 30% of slashed stake +Frivolous challenge (forfeit bond) -0.5% slash-equiv +================================================ ==================== + +Rewards fund validator operation (compute for audit-replay, +bandwidth for mesh gossip). Without rewards, the validator set +decays to zero. + +§6.4 Stake cap & sqrt-weighting (optional) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Per ticket #000012 §6 risk note, raw stake-weighting risks +plutocratic capture. Two recommended mitigations, neither mandatory: + +- **Stake cap**: cap any single validator's effective stake at + ``X%`` of total (recommend 5%). +- **Sqrt-stake**: convex weighting (``effective_weight = + sqrt(raw_stake)``) so a single 100x staker counts as 10 unit + stakers, not 100. + +These are governance-parameter knobs in ``consensus_policy_hash``; +the paper does not mandate which (or whether) to enable. Different +deployments may prefer different shapes. + + +Part 7 — Fork choice rule (GRANDPA-style) +----------------------------------------- + +§7.1 Finalization is permanent +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once ≥ 2/3 active stake signs ``accept`` AND the challenge window +elapses without successful challenge, the ``consensus_events`` row +is **finalized**. No reorg is possible past a finalized row. + +This is the GRANDPA-style finality property (Polkadot / Kusama). +Trade-off: latency is higher than Bitcoin-style longest-chain +(at minimum: signing window + challenge window — recommend 1 + 7 = +8 days), but reorg risk past finality is *cryptographically zero* +under the 2/3-honest-stake assumption. + +§7.2 Pre-finality fork choice +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +During the signing window, multiple candidates may be in flight +against the same parent. Validators MUST refuse to sign a candidate +whose parent is not finalized. This prevents: + +- Long-range reorgs (validators cannot sign for a parent that + hasn't been finalized — no way to back-write history). +- Equivocation hazards (a validator signing on two parents at the + same checkpoint is automatically slashable, since both parents + cannot finalize). + +§7.3 Liveness recovery +~~~~~~~~~~~~~~~~~~~~~~ + +If finalization stalls on a parent (e.g., 1/3 of stake offline), no +child can finalize either. The protocol recovers when the offline +stake either rejoins or is slashed for non-participation +(governance parameter; recommend slashing after ``N`` missed +signing windows, ``N`` = 3). + +This caps liveness loss at ``3 × signing_window = 3 days`` under +recommended parameters. + + +Part 8 — Mesh wire format extension +----------------------------------- + +§8.1 New message kinds +~~~~~~~~~~~~~~~~~~~~~~ + +The arborist mesh wire format (``arborist/mesh/wire.py``) carries +per-shard sync today. v8 adds three new top-level message kinds: + +================================ ================================== +Message kind Body +================================ ================================== +``v8_validator_announce`` Stake deposit announcement; + triggers ``bond`` window. +``v8_acceptance_signature`` Single validator's accept/reject + sig on a candidate during a + signing window. +``v8_challenge`` Counter-evidence submission + during a challenge window. +================================ ================================== + +§8.2 Canonical encoding +~~~~~~~~~~~~~~~~~~~~~~~ + +Every message body is canonical-JSON (SQD A1) before signing. The +aggregate signature (when threshold-key is bootstrapped) is BLS- +flavored; otherwise the protocol falls back to a sorted-concat of +individual validator signatures, length-prefixed. + +A consumer that wants to verify a finalized ``consensus_events`` +row independently: + +1. Parse the row's ``validator_signatures`` field. +2. For each signature, look up the signing validator's pubkey from + the ``consensus_events`` chain at the signing time. +3. Verify each signature against the canonical encoding of the row + body (minus the signature field). +4. Aggregate stake-weighted; confirm ≥ 2/3 active stake at + signing time. + +Step 2 requires the validator set's stake snapshot at the row's +``finalized_at`` timestamp. v8 commits a **stake snapshot** in +every finalized row's body so step 2 is local-only — no separate +lookup table needed. + +§8.3 Bandwidth profile +~~~~~~~~~~~~~~~~~~~~~~ + +At recommended parameters (``signing_window`` = 1 day, +``challenge_window`` = 7 days, ``checkpoint_cadence`` = weekly), +the gossip load is dominated by ``v8_acceptance_signature`` traffic +during the signing window. For a 100-validator deployment with +1 KB per signature message, peak load is + +.. code-block:: text + + 100 validators · 1 KB · gossip_fanout (8) = 800 KB per checkpoint + +— well within arborist mesh's per-shard sync budget. + +Challenge-window traffic is bursty but bounded (any single party +can submit O(1) challenges per finalized row). + + +Part 9 — BFT analysis +--------------------- + +§9.1 Safety +~~~~~~~~~~~ + +**Claim:** No two conflicting candidates can both finalize under +the 2/3 quorum rule + GRANDPA-style finality, assuming +≤ 1/3 Byzantine stake. + +**Proof sketch:** Suppose ``M'_1`` and ``M'_2`` both finalize on +the same parent. Each requires ≥ 2/3 stake to accept. Total stake +signing accept across both candidates ≥ 4/3 — impossible without +overlap. Any validator signing both is automatically slashable per +§6.2 (conflicting acceptances → 100% slash). Therefore the +overlapping validators are Byzantine. By assumption Byzantine +stake ≤ 1/3 < 4/3 − 1 = 1/3 needed-overlap, contradiction. +Conclusion: at most one candidate finalizes per parent. + +§9.2 Liveness +~~~~~~~~~~~~~ + +**Claim:** Under partition where < 1/3 stake is offline, +finalization progresses within bounded time. + +**Proof sketch:** ≥ 2/3 stake is online and honest. Honest +validators sign on any candidate clearing the canonical floor. +Quorum is reached within the signing window. After the +challenge window closes without successful challenge, finalization +completes. Bound: ``signing_window + challenge_window`` +(recommended: 8 days). + +**Claim:** Under partition where ≥ 1/3 stake is offline, +finalization stalls but does not regress. + +**Proof sketch:** Without 2/3 quorum, no candidate finalizes. +Pre-finality state remains accessible (downstream consumers see +"pending" status). Once the offline stake rejoins (or is slashed +per §7.3), the protocol resumes. + +§9.3 Sybil resistance +~~~~~~~~~~~~~~~~~~~~~ + +Stake bonding is the Sybil bound. An adversary controlling N +identities still controls only ``sum(N_stake_i)`` weight, not ``N`` +votes. With optional sqrt-weighting (§6.4), even concentrated +stake is diluted. + +§9.4 Bootstrap honesty +~~~~~~~~~~~~~~~~~~~~~~ + +The protocol cannot bootstrap from zero. v8 deployment must specify +an **initial validator set** with explicit honesty assumption. The +permacomputer-mission default is: + +- Bootstrap from a small (5-7) permissioned validator set, all + publicly identified, with stake deposited to a multi-sig + bootstrap escrow. +- After ``M`` checkpoints (recommend ``M`` = 12, ~3 months at + weekly cadence) without successful challenges, the bootstrap + set's stake unlocks for normal unbond/slash semantics. +- New validators join permissionlessly during bootstrap; they don't + earn rewards until bootstrap closes (prevents bootstrap-stake + dilution attacks). + +§9.5 Re-staking attacks +~~~~~~~~~~~~~~~~~~~~~~~ + +A validator may stake the same capital across multiple v8 instances +(e.g., one per major Merkle-AGI lineage). Out of scope for this +paper — the cross-instance slashing accumulator is its own +follow-up if multi-instance v8 deployments emerge. + + +Part 10 — Worked example (appendix) +----------------------------------- + +§10.1 Setup +~~~~~~~~~~~ + +Imaginary deployment: 7 active validators ``V1..V7``, each with 100 +stake units (700 total). 2/3 threshold = 467 stake. Recommended +parameters: 1-day signing window, 7-day challenge window. + +§10.2 Cycle +~~~~~~~~~~~ + +**Day 0**: Proposer P submits ``C(M'_1)`` against finalized parent +``C(M_t)``. Self-reported fork_score = +0.42. + +**Day 0-1 (signing window)**: ``V1..V6`` independently audit-replay, +each producing fork_score in [+0.40, +0.43] (within signal floor). +All six sign ``accept`` with their replay digests. ``V7`` is +offline; doesn't sign. Stake signing accept = 600 ≥ 467. + +**Day 1**: Quorum reached. Aggregate signature minted; new row +written to ``consensus_events`` chain. Row status: +``finalized-pending``. + +**Day 1-8 (challenge window)**: External party C submits +counter-evidence claiming ``V3``'s replay digest disagrees with +canonical floor by 15pp. + +**Day 8**: Adjudication set ``V1, V2, V4, V5, V6`` (excluding +challenged ``V3`` and offline ``V7``) re-audits. Each finds C's +replay matches the canonical result; ``V3``'s replay disagrees by +15pp. + +**Day 8**: Adjudication majority (≥ 2/3 of 5 = 4) confirms +``V3``'s misreporting. ``V3`` is slashed 30% (audit-mismatch +class). C earns 30% of slashed stake = 9 units. + +**Day 8**: The accept signature is still valid (V3's signature is +struck, remaining stake signing accept = 500 ≥ 467). Row +finalizes. + +**Day 8+**: Downstream consumers verify the finalized row by +walking the ``consensus_events`` chain. They see ``M'_1`` is the +canonical successor of ``M_t``. + +§10.3 What the audit ledger looks like +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: text + + consensus_events row at chain height N+1: + prev_event_hash: + candidate_root: + parent_root: + proposer_id: P + fork_score: {breakdown ...} + validator_signatures: [, , , , ] + (V3 struck post-slash; V7 offline) + audit_replay_digest: + verdict: "FINALIZED" + finalized_at: + consensus_policy_hash: + + event_hash = sha256( prev_event_hash || canonical(body) ) + +A future consumer pulling this row can verify: + +1. Each validator signature against the canonical body. +2. Stake-weighted aggregate against the stake-snapshot inside the + body. ≥ 2/3. +3. Chain integrity against the previous row's ``event_hash``. + +No external lookup required. + + +Part 11 — Out of scope +---------------------- + +- **Implementation**. Code lands in 3-4 follow-up tickets: + validator state-machine SQL + state-transition functions; mesh + wire format extension; audit-replay harness binding; slashing + accountant. +- **Economic parameter calibration**. Stake amounts, slashing + fractions, reward rates, challenge bonds — governance decisions, + not paper-pinned. The paper specifies the *form*; deployments + calibrate. +- **Cross-chain anchoring**. Publishing finalized + ``consensus_events`` rows to Bitcoin / Ethereum / etc. is an + optional bolt-on. Out of scope here. +- **Frontier benchmark fixture selection**. Covered by ticket + #000021. v8 references the fixtures; doesn't curate them. +- **Bootstrap-set membership**. The actual identities of the + bootstrap validators are a governance decision for the deploying + consortium. +- **Cross-instance slashing accumulator**. Re-staking attacks + across multiple v8 instances. Future work; emerges when more than + one v8 instance exists. +- **Branch-set persistence**. Ticket #000012 Phase 1c (proposed, + not yet open). Multi-branch fork scoring at the same checkpoint. + Doc-only proposal in #000012; lands as its own follow-up if + operator pressure surfaces. + + +Closure +------- + +Closure criterion (#000012 §4.1): this document plus the +``arborist.substrate.fork_score`` module (Phase 1a; landed 2026-05-08) +constitute the v8 substrate paper. The reference doc +``docs/v8-fork-score.md`` carries the formula details; this paper +carries the consensus / acceptance / slashing protocol. + +Open questions tracked separately: + +- **Initial validator set composition** — governance, not paper. +- **Threshold-key ceremony** — needed for BLS-aggregate signatures; + if not bootstrapped, fallback to sorted-concat signatures is + fully functional (just larger payloads). +- **ZK-replay** — verifiers proving canonical-floor replay without + re-running it locally. Captured in #000016 (ZK frontier proof); + optional optimization on top of v8. +- **Policy-hash transition mechanics** — when + ``consensus_policy_hash`` changes (e.g., new floor metric added), + how do in-flight signing windows handle the transition? Recommend: + policy changes take effect at the next checkpoint after a + successful finalization under the new policy; in-flight + signatures honor the policy hash they were submitted under. + +Implementation tickets that cite this paper land later — one per +validator-state-machine, mesh-wire-format extension, audit-replay +harness, and slashing accountant. + +Status: closure-draft 2026-05-10. Ready for fox + downstream +review. diff --git a/docs/tickets/ticket-000012-selection-consensus-protocol.md b/docs/tickets/ticket-000012-selection-consensus-protocol.md index e180848..fa49c52 100644 --- a/docs/tickets/ticket-000012-selection-consensus-protocol.md +++ b/docs/tickets/ticket-000012-selection-consensus-protocol.md @@ -1,6 +1,6 @@ # Ticket #000012 — Selection & consensus protocol (Merkle-AGI v8) -**Status:** in progress · Phase 1a (ForkScore) landed 2026-05-08; consensus paper still open +**Status:** in progress · Phase 1a (ForkScore) landed 2026-05-08; Phase 1b (consensus paper) landed 2026-05-10; Phase 1c (branch-set persistence) remains proposed-not-opened **Opened:** 2026-05-07 **Scope:** Spec for the loop-closing consensus protocol that turns the v7 substrate + v9.8 runtime into actual Darwinian selection across multiple @@ -287,26 +287,54 @@ function ahead of the consensus paper: inf-bonus capping, neg-inf rejection, weight-set tuning, breakdown completeness, CLI smoke. Full suite: 1186 passed. -### Phase 1b — Consensus paper (still open) +### Phase 1b — Consensus paper (landed 2026-05-10) -Closure criterion: `docs/merkle-agi-v8-consensus.rst` lands with -validator state machine, acceptance protocol, challenge protocol, -fork-choice rule (GRANDPA-style), slashing mechanics, mesh wire -format extension. Per §1 of this ticket, -`arborist/substrate/fork_score.py` is the substrate the paper -cites; the paper itself is still research-scope. +Paper landed at `docs/_source/merkle-agi-v8-consensus.rst` (834 +lines, RST sister to the v7-W substrate paper at the same path). +Sections delivered: -What's deliberately NOT in Phase 1a: +- **Part 1** — Introduction & motivation; gap table from v7 § 13.4; + concrete backdoor-attack scenario; paper IS/IS-NOT scope. +- **Part 2** — Substrate definition; SQD A1/A2/A3 inheritance; + `consensus_events` row schema; `consensus_policy_hash` sibling + (never enters `cache_key`). +- **Part 3** — Validator state machine (bonding / active / + challenged / slashed / unbonding) with full transition graph and + invariants. +- **Part 4** — Acceptance protocol; layered fitness floor + (canonical + lab-declared ceiling); audit-replay procedure; + 2/3-stake quorum + GRANDPA-style finalization; liveness floor. +- **Part 5** — Challenge protocol; counter-evidence shape; + adjudication; challenger reward (30% recommended) + frivolous- + challenge bond. +- **Part 6** — Stake mechanics; bond/unbond/challenge window + recommendations; offense-class slashing schedule; reward + distribution; optional stake cap + sqrt-weighting. +- **Part 7** — Fork choice rule (GRANDPA-style); finalization + permanence; pre-finality fork-choice constraints; liveness + recovery from offline-stake partition. +- **Part 8** — Mesh wire format extension (three new message + kinds; BLS-or-concat aggregate signatures; bandwidth profile). +- **Part 9** — BFT analysis (safety, liveness, Sybil resistance, + bootstrap honesty, re-staking attacks). +- **Part 10** — Worked example: 7-validator deployment, one + upgrade cycle with successful challenge against one fraudulent + validator. +- **Part 11** — Out of scope (implementation, calibration, cross- + chain anchoring, fixture selection, bootstrap-set membership, + cross-instance slashing accumulator, branch-set persistence). +- **Closure** — open questions: initial-set composition, threshold- + key ceremony, ZK-replay, policy-hash transition mechanics. -- Validator state machine (bonding / signing / slashing). -- Acceptance protocol. -- Challenge protocol. -- Fork-choice rule. -- Mesh wire format extensions. -- Stake mechanics + economic incentives. -- Cross-validator ZK proof exchange. +What's deliberately NOT in Phase 1a (now specified in the v8 paper): -These belong to the v8 paper itself. +- Validator state machine (bonding / signing / slashing). → Part 3 +- Acceptance protocol. → Part 4 +- Challenge protocol. → Part 5 +- Fork-choice rule. → Part 7 +- Mesh wire format extensions. → Part 8 +- Stake mechanics + economic incentives. → Part 6 +- Cross-validator ZK proof exchange. → Closure §; deferred to #000016 ### Phase 1c — Branch-set persistence (proposed, not yet open)