engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

#000012 Phase 1b: Merkle-AGI v8 consensus paper landed

Browse source 
`docs/_source/merkle-agi-v8-consensus.rst` (834 lines, RST sister
to the v7-W substrate paper at the same path). Closes Phase 1b of
ticket #000012 — the loop-closing consensus protocol that turns
single-validator Proof-of-Upgrade into Darwinian selection across
an open validator set.

11 parts:

  Part 1   Introduction & motivation — gap table from v7 § 13.4,
           concrete backdoor-attack scenario, paper IS/IS-NOT
           scope.
  Part 2   Substrate definition — SQD A1/A2/A3 inheritance,
           consensus_events row schema, consensus_policy_hash
           sibling (never enters cache_key).
  Part 3   Validator state machine — bonding/active/challenged/
           slashed/unbonding with full transition graph + invariants.
  Part 4   Acceptance protocol — proposer submission, layered
           fitness floor (canonical + lab-declared ceiling),
           audit-replay procedure, 2/3-stake quorum + GRANDPA-
           style finalization, liveness floor.
  Part 5   Challenge protocol — counter-evidence shape,
           adjudication, challenger reward, frivolous-challenge
           bond.
  Part 6   Stake mechanics — bond/unbond/challenge window
           recommendations, offense-class slashing schedule,
           reward distribution, optional stake cap + sqrt-weighting.
  Part 7   Fork choice rule — GRANDPA-style finality, pre-finality
           constraints, liveness recovery.
  Part 8   Mesh wire format extension — three new message kinds,
           BLS-or-concat aggregate signatures, bandwidth profile.
  Part 9   BFT analysis — safety, liveness, Sybil resistance,
           bootstrap honesty, re-staking attacks.
  Part 10  Worked example — 7-validator deployment, one upgrade
           cycle with successful challenge against one fraudulent
           validator.
  Part 11  Out of scope — implementation, calibration, cross-chain
           anchoring, fixture selection, bootstrap-set membership,
           cross-instance slashing accumulator, branch-set
           persistence.

Closure §: open questions tracked separately (initial validator
set composition, threshold-key ceremony, ZK-replay, policy-hash
transition mechanics).

Ticket #000012 status updated; Phase 1c (branch-set persistence)
remains proposed-not-opened. Implementation follow-up tickets that
cite this paper land later — one per validator-state-machine,
mesh-wire-format extension, audit-replay harness, slashing
accountant.
This commit is contained in:
russell@unturf.com 2026-05-10 16:12:00 -04:00
parent 883289d00e
commit fd643fe1a5
No known key found for this signature in database
3 changed files with 880 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/TICKETS.md
View file

@ -105,7 +105,7 @@ Newest first. Update on every open/close.
| #000015 | π* domain library + cross-domain composition | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000014 | SelfModel: schema, falsification, integration | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000013 | Spatial-temporal substrate (Merkle-AGI v7-W) | closed · landed 2026-05-09 (substrate paper + frontier catalog + namespace stub) | 2026-05-07 | — |
| #000012 | Selection & consensus protocol (Merkle-AGI v8) | in progress · Phase 1a + 1b (ForkScore CLI) landed 2026-05-09 | 2026-05-07 | — |
| #000012 | Selection & consensus protocol (Merkle-AGI v8) | in progress · Phase 1a (ForkScore) landed 2026-05-08; Phase 1b (consensus paper, `docs/_source/merkle-agi-v8-consensus.rst` 834 lines) landed 2026-05-10; Phase 1c (branch-set persistence) remains proposed-not-opened | 2026-05-07 | — |
| #000011 | SOFT_PREFLIGHT_HINT model-assisted sidecar | closed · landed 2026-05-04 (zero-shot full impl) | 2026-05-04 | D1 (preserves) |
| #000010 | Meta-Cognition Preflight Guard (M0 / MCTL) | closed · landed 2026-05-03 (Phases 1–4); DAG binding shipped via #000009 | 2026-05-03 | D1, D3 |
| #000009 | Preflight run-DAG node binding (#000008+#000010) | closed · re-landed 2026-05-04 (§8 corrections: reject-path DAG, nested CTI clauses) | 2026-05-03 | D3, D4 |

834
docs/_source/merkle-agi-v8-consensus.rst Normal file
View file

@ -0,0 +1,834 @@
Merkle-AGI v8 — Selection & Consensus Protocol
===============================================
:Author: fox + agent blackops, on the unsandbox / unturf / permacomputer platform
:Date: 2026-05-10 (draft v0)
:Status: substrate paper for ticket #000012 Phase 1b; closure draft.
This document specifies the **selection layer** of the Merkle-AGI
lineage — sister to v7 (logic/math substrate; plastic training) and
v7-W (spatial-temporal substrate). v8 commits the loop-closing step
that turns single-validator Proof-of-Upgrade into Darwinian
selection across an open validator set.
Read alongside:
- ``docs/v8-fork-score.md`` — canonical formula reference for
``arborist.substrate.fork_score`` (v8 Phase 1a; landed 2026-05-08).
The ScoredFork function this paper invokes for fitness scoring.
- ``docs/_source/merkle-agi-v7w-spatial-temporal.rst`` — sister
substrate paper; same RST conventions; same audience.
- The v7 substrate spec
(``~/Downloads/merkle-agi-dag_v7.txt`` § 13.4) — the Proof-of-
Upgrade procedure v8 extends from single-validator to multi-
validator consensus.
Part 1 — Introduction & motivation
----------------------------------
§1.1 The DNA ↔ Merkle-DAG loop
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The substrate-lineage analogy is:
.. code-block:: text
copy → vary → express → test → select → preserve → repeat
Merkle-AGI v7 ships everything except ``select``. Section 13.4
("Proof-of-Upgrade") sketches a procedure where a candidate model
``(M', C(M'))`` is admitted if it shows non-regression on a
published eval set + non-regression of ε-coverage at sentinel
frontiers. That procedure is **single-validator regression testing
with a Merkle receipt**. It is not consensus.
§1.2 The selection gap under v7 § 13.4
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The unaddressed gaps from v7 § 13.4, each with a concrete failure
under the current spec:
================================ ============================================
Gap Concrete failure
================================ ============================================
Validator set discovery Two labs reach different verdicts on M'; no
fork-choice rule for downstream consumers.
Byzantine fault tolerance One dishonest validator can sign acceptance
for a poisoned (M').
Sybil resistance A single actor spinning up 50 validators
wins every quorum.
Fork-choice on disagreement If V1, V2 publish conflicting acceptance
receipts, downstream nodes have no rule.
Liveness vs safety trade No bound on how long acceptance can stall
when validators are offline.
Validator incentives Why would anyone run a validator? What's
the slashing condition for cheating?
Stake / membership semantics Permissionless? Permissioned? Hybrid?
v7 silent.
================================ ============================================
Without a protocol that closes these, "digital evolution" is
metaphor — a single lab signing its own upgrades, no different in
trust model from current model-card releases.
§1.3 Concrete attack scenario (motivating example)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Lab A trains ``M_t → M_{t+1}`` with a backdoor that triggers on a
rare input pattern. Lab A signs Proof-of-Upgrade: ε-coverage
non-regressed at every published frontier (because the backdoor
lives at a non-published frontier). Lab A publishes ``C(M_{t+1})``.
Anyone pulling the registry sees an "accepted" upgrade. v7 has no
way to surface that no independent validator audited the upgrade.
The v8 selection protocol must make "accepted under v8" mean
"accepted by N independent validators meeting policy P", verifiable
to anyone, not "Lab A signed it."
§1.4 What this paper IS and IS NOT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**IS**:
- A protocol specification: validator state machine, acceptance
ledger, challenge window, fork-choice rule, slashing schedule,
mesh wire format extension.
- A BFT-analysis sketch with explicit safety/liveness bounds.
- A worked-example acceptance cycle (appendix) reflecting one
imaginary upgrade.
**IS NOT**:
- An implementation. Code lands in follow-up tickets that cite this
paper.
- An economic calibration. Stake amounts, slashing fractions, and
reward rates are governance decisions; the paper specifies their
shape, not their magnitude.
- A cross-chain anchoring spec. Publishing v8 finalizations to
Bitcoin / Ethereum / etc. is an optional bolt-on; not in this
paper.
Part 2 — Substrate definition
-----------------------------
§2.1 Hard constraints (carried from #000012)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- Stays inside SQD A1 (canonical encoding), A2 (public
quantization), A3 (collision-resistant hash). v8 introduces no
new cryptographic axiom.
- No per-query consensus. Selection runs at checkpoint cadence
(Proof-of-Upgrade scope), not at inference time.
- v9.8 ``cache_key`` invariant stays at 8 dimensions. Consensus
state lives in a sibling ``consensus_events`` table, not folded
into the answer-cache.
- arborist's per-shard audit chain stays per-node. v8 is
checkpoint-cadence, cross-validator; nothing in v8 modifies
``audit_events.event_hash`` preimage on individual shards.
- Existing falsification-state semantics
(``live``, ``failed``, ``stale``, ``quarantined``) stay
unchanged. v8 reuses them for validator state, not for cache
state.
§2.2 What v8 commits
~~~~~~~~~~~~~~~~~~~~
v8 commits the **acceptance ledger**: a Merkle-chained sequence of
finalized ``(C(M'), proposer_id, validator_signatures, audit_replay
_digest, fork_score_breakdown)`` rows. Each row's
``event_hash`` is the SHA-256 of the canonical encoding of the row
body chained against the previous row's ``event_hash``.
.. code-block:: text
consensus_events.event_hash =
SHA-256( prev_event_hash || canonical_encoding(body) )
body = {
"candidate_root": "<C(M') SHA-256>",
"parent_root": "<C(M_t) SHA-256>",
"proposer_id": "<validator pubkey>",
"eval_digest": "<canonical eval-suite digest>",
"frontier_coverage_diff": "<ε-coverage delta blob>",
"fork_score": {<ScoredFork breakdown>},
"validator_signatures": [<aggregate or individual sigs>],
"audit_replay_digest": "<replay result digest>",
"verdict": "ACCEPTED" | "REJECTED" | "CHALLENGED",
"finalized_at": <unix-ts>
}
The canonical encoding follows the SQD A1 convention (sorted JSON
keys, UTF-8, no whitespace) so two independent serializers produce
byte-identical bytes.
§2.3 Sibling-table semantics (no `cache_key` pollution)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Per ticket #000012 §4.2 hard constraint: consensus state never
enters ``providence_cache.cache_key``. The 8-dim invariant
(``source_root | question_hash | model_profile_hash |
conversation_hash | governance_policy_hash | schema_version |
canonicalization_version | chunking_version``) is unchanged.
v8 introduces a **ninth-conceptual** invariant — the
**consensus_policy_hash** — but folds it into a sibling table, not
``cache_key``. The motivation: per-query cache invalidation should
not fire every time a validator joins or leaves the set. Validator-
set churn is checkpoint-scale, not query-scale.
The two policy hashes are deliberately split:
- ``governance_policy_hash`` — folds answer-mode, verifier method,
claim caps, broad-quantifier guard config. Per-query; folds into
``cache_key``.
- ``consensus_policy_hash`` — folds validator set, quorum
threshold, slashing schedule, challenge window. Per-checkpoint;
folds into ``consensus_events`` row but NEVER ``cache_key``.
A consumer that wants to verify "this answer was produced under
finalized policy P at checkpoint T" walks the ``consensus_events``
chain to find P, then walks the per-shard ``audit_events`` chain to
find the answer.
Part 3 — Validator state machine
--------------------------------
§3.1 States
~~~~~~~~~~~
A validator transitions through five states across its lifetime:
================ ================================================
State Meaning
================ ================================================
``bonding`` Stake deposit pending; cannot sign yet. Bond
window must elapse before promotion.
``active`` Eligible to sign acceptance / rejection on
proposed candidates. Earns rewards on honest
signatures.
``challenged`` A counter-evidence submission claims one of this
validator's prior signatures was fraudulent. Stake
is frozen pending challenge resolution.
``slashed`` Challenge resolved against the validator. Stake
partially or fully forfeit per offense-class
schedule (§6.2). Validator cannot rejoin under the
same pubkey.
``unbonding`` Validator initiated voluntary exit. Stake is
time-locked for an unbond window so outstanding
challenges can land before withdrawal.
================ ================================================
§3.2 Transitions
~~~~~~~~~~~~~~~~
.. code-block:: text
stake_deposit
(start) ─────────────────────────► bonding
│
bond_window elapsed
│
▼
┌──────────────────────────────► active ───────────────────────┐
│ │ │
│ stake withdraw │
│ initiated │
│ │ │
│ ▼ │
│ unbonding │
│ │ │
│ unbond_window elapsed │
│ │ │
│ ▼ │
│ (exit) │
│ │
│ challenge_submitted │
│ │
▼ │
challenged │
│ │
│ challenge_resolved_in_favor │
└──────────────────────────────────────────────────────────┘
│
│ challenge_resolved_against
▼
slashed
│
│ (terminal — same pubkey cannot rejoin)
│
▼
§3.3 Invariants
~~~~~~~~~~~~~~~
- A validator in ``slashed`` state remains in the chain forever
under its original pubkey; rejoin requires a fresh keypair and
fresh stake.
- ``challenged`` is a **transient** state. The challenge window
bounds resolution time (§4.4); ambiguous challenges resolve
in favor of the validator by default (presumption of innocence).
- ``unbonding`` blocks signing. A validator that has initiated exit
cannot sign new acceptances; this prevents the
sign-then-exit-then-let-challenge-bounce attack.
- Bond and unbond windows are governance parameters in
``consensus_policy_hash``; changing them invalidates the policy
hash and therefore which ``consensus_events`` rows verify under
it.
Part 4 — Acceptance protocol
----------------------------
§4.1 Proposer submission
~~~~~~~~~~~~~~~~~~~~~~~~
A proposer (any actor — does not need to be a validator) submits:
.. code-block:: json
{
"candidate_root": "<C(M') SHA-256>",
"parent_root": "<C(M_t) SHA-256>",
"eval_digest": "<canonical eval-suite digest>",
"frontier_coverage_diff": "<ε-coverage delta blob>",
"fork_score_self_reported": {<ScoredFork breakdown>},
"proposer_signature": "<sig over above fields>"
}
The proposer is bound to the submission by their signature; this
prevents anonymous spam. The self-reported ``fork_score`` is
*advisory* — validators re-compute it independently during audit-
replay (§4.3).
§4.2 Layered fitness floor
~~~~~~~~~~~~~~~~~~~~~~~~~~
Per ticket #000012 §2.2 (Recommendation C), every candidate must
clear a **canonical floor** before validators are obligated to
audit. The canonical floor is published in the v8 substrate paper
and pinned in ``consensus_policy_hash``:
================================ ====================================
Floor metric Threshold
================================ ====================================
``Δ5S`` (5-syndrome battery) ≥ 0 (no regression)
``Δ5T`` (5-transfer battery) ≥ 0 (no regression)
``Δ5F`` (5-falsification battery) ≥ 0 (no regression)
ε-coverage at published frontiers ≥ floor specified per-frontier
``audit_completeness`` = 1.0 (every state-change row hashes
through ``audit_events``)
``hard_regression_flag`` = false (no ScoredFork hard regress)
================================ ====================================
A proposer can additionally declare lab-specific ceiling metrics
(e.g., "this upgrade is for combinatorial reasoning; please verify
combinatorics@v1 round-trip rate"). Validators that audit the
ceiling metrics earn bonus rewards (§6.3); ceiling failures do not
trigger slashing.
§4.3 Audit replay
~~~~~~~~~~~~~~~~~
Each ``active`` validator independently:
1. Pulls ``C(M')`` and verifies its content-address matches the
submitted root.
2. Runs the canonical floor benchmarks against ``M'``. This is a
deterministic procedure pinned by ``eval_digest`` — two honest
validators MUST produce byte-identical replay digests.
3. Computes ``ScoredFork(parent=M_t, child=M')`` using the
``arborist.substrate.fork_score`` reference implementation.
4. Compares its replay against the proposer's self-reported
``fork_score``. Discrepancy ≥ 5pp (per the SIGNAL_FLOOR convention
from ``docs/bench-maxing.md``) flags a soft mismatch; ≥ 20pp or
any hard-regression-flag flip triggers automatic rejection.
5. Signs ``accept`` or ``reject`` with its validator key, attaching
its replay digest.
§4.4 Quorum & finalization
~~~~~~~~~~~~~~~~~~~~~~~~~~
The signing window is bounded (governance parameter; recommend
24h). Within the window, signatures are gossiped via the mesh wire
format (§7). When ≥ 2/3 of active stake signs ``accept`` AND no
``reject`` exceeds 1/3, the candidate finalizes:
.. code-block:: text
accept_stake ≥ (2/3) · total_active_stake
AND reject_stake < (1/3) · total_active_stake
⇒ FINALIZE(C(M')) at current consensus_events tip
Finalization mints an aggregate signature (BLS-style if a threshold
key was bootstrapped, otherwise concatenated individual sigs) and
writes one row to ``consensus_events`` chained against the previous
tip. Per §2.3 the row body is canonical-encoded; ``event_hash`` is
SHA-256 of ``prev_event_hash || canonical(body)``.
§4.5 Liveness floor
~~~~~~~~~~~~~~~~~~~
If the signing window elapses without quorum, the candidate is
marked ``CHALLENGED-OR-STALLED`` and re-proposed at the next
checkpoint. Repeated failures (governance parameter; recommend 3)
trigger automatic rejection without slashing — partition-tolerance
floor.
Part 5 — Challenge protocol
---------------------------
§5.1 Challenge window
~~~~~~~~~~~~~~~~~~~~~
After finalization, a **challenge window** opens (governance
parameter; recommend 7 days). Anyone — validator or not — may
submit counter-evidence claiming one of the signing validators
misreported its audit-replay digest.
§5.2 Counter-evidence shape
~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: json
{
"challenged_candidate_root": "<C(M') SHA-256>",
"challenged_validator_pubkey": "<validator pubkey>",
"challenger_replay_digest": "<independent replay result>",
"discrepancy_evidence": "<diff between challenger and
challenged digests>",
"challenger_signature": "<sig over above fields>"
}
§5.3 Resolution
~~~~~~~~~~~~~~~
The challenge is resolved by the **next active validator set**
(excluding the challenged validator):
1. Each adjudicating validator runs its own audit-replay on
``C(M')`` and compares against both the challenged digest and
the challenger digest.
2. ≥ 2/3 of the adjudicating set agree on which side is honest →
challenge resolves in favor of that side.
3. If the challenged digest disagrees with the adjudicators by
≥ 5pp, the challenged validator is slashed per the offense-class
schedule (§6.2). All other validators that signed the same
``accept`` are also re-audited; if their replay disagrees too,
they too are slashed.
§5.4 Challenger reward
~~~~~~~~~~~~~~~~~~~~~~
A successful challenger earns a slice of the slashed stake
(governance parameter; recommend 30%). This is a positive incentive
for independent audit; without it, the challenge protocol decays.
§5.5 Frivolous challenges
~~~~~~~~~~~~~~~~~~~~~~~~~
If the adjudicating set rules the challenge frivolous (challenger
digest matches the challenged digest within 5pp), the challenger
forfeits a small **challenge bond** (governance parameter; recommend
0.5% of the slashing-fraction-equivalent). Without this, an
adversary spams challenges to grief validators.
Acceptance row promotion to ``finalized`` requires the challenge
window to elapse without successful challenge. Until then the row
sits in ``finalized-pending`` state; consumers SHOULD wait for the
window to close before relying on the candidate.
Part 6 — Stake mechanics
------------------------
§6.1 Bond & unbond windows
~~~~~~~~~~~~~~~~~~~~~~~~~~
================ =================================== ===========
Window Recommended duration Rationale
================ =================================== ===========
``bond`` 7 days Time for
fraud-on-
stake-deposit
challenges.
``unbond`` 2 × challenge_window Outstanding
challenges
must land
before exit.
``challenge`` 7 days Independent
audit
feasibility
per fox's
review note.
================ =================================== ===========
§6.2 Slashing schedule
~~~~~~~~~~~~~~~~~~~~~~
================================ ==================================
Offense class Slashing fraction (recommended)
================================ ==================================
Audit-replay digest mismatch 30% of stake
Double-signing (accept + reject) 50% of stake
Conflicting acceptances (fork) 100% of stake
Equivocation under challenge 100% of stake
================================ ==================================
Fractions are *recommended*; calibration is a governance decision.
The paper pins the **shape** (single-offense → partial slash;
fork-equivocation → full slash) and the **monotonic ordering** (a
heavier offense always slashes ≥ a lighter offense's fraction).
§6.3 Reward distribution
~~~~~~~~~~~~~~~~~~~~~~~~
================================================ ====================
Action Reward (recommended)
================================================ ====================
Honest accept signature on finalized candidate proportional to stake
Honest reject signature on rejected candidate proportional to stake
Lab-ceiling-metric audit (bonus) flat per audit
Successful challenge (slash recipient) 30% of slashed stake
Frivolous challenge (forfeit bond) -0.5% slash-equiv
================================================ ====================
Rewards fund validator operation (compute for audit-replay,
bandwidth for mesh gossip). Without rewards, the validator set
decays to zero.
§6.4 Stake cap & sqrt-weighting (optional)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Per ticket #000012 §6 risk note, raw stake-weighting risks
plutocratic capture. Two recommended mitigations, neither mandatory:
- **Stake cap**: cap any single validator's effective stake at
``X%`` of total (recommend 5%).
- **Sqrt-stake**: convex weighting (``effective_weight =
sqrt(raw_stake)``) so a single 100x staker counts as 10 unit
stakers, not 100.
These are governance-parameter knobs in ``consensus_policy_hash``;
the paper does not mandate which (or whether) to enable. Different
deployments may prefer different shapes.
Part 7 — Fork choice rule (GRANDPA-style)
-----------------------------------------
§7.1 Finalization is permanent
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Once ≥ 2/3 active stake signs ``accept`` AND the challenge window
elapses without successful challenge, the ``consensus_events`` row
is **finalized**. No reorg is possible past a finalized row.
This is the GRANDPA-style finality property (Polkadot / Kusama).
Trade-off: latency is higher than Bitcoin-style longest-chain
(at minimum: signing window + challenge window — recommend 1 + 7 =
8 days), but reorg risk past finality is *cryptographically zero*
under the 2/3-honest-stake assumption.
§7.2 Pre-finality fork choice
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
During the signing window, multiple candidates may be in flight
against the same parent. Validators MUST refuse to sign a candidate
whose parent is not finalized. This prevents:
- Long-range reorgs (validators cannot sign for a parent that
hasn't been finalized — no way to back-write history).
- Equivocation hazards (a validator signing on two parents at the
same checkpoint is automatically slashable, since both parents
cannot finalize).
§7.3 Liveness recovery
~~~~~~~~~~~~~~~~~~~~~~
If finalization stalls on a parent (e.g., 1/3 of stake offline), no
child can finalize either. The protocol recovers when the offline
stake either rejoins or is slashed for non-participation
(governance parameter; recommend slashing after ``N`` missed
signing windows, ``N`` = 3).
This caps liveness loss at ``3 × signing_window = 3 days`` under
recommended parameters.
Part 8 — Mesh wire format extension
-----------------------------------
§8.1 New message kinds
~~~~~~~~~~~~~~~~~~~~~~
The arborist mesh wire format (``arborist/mesh/wire.py``) carries
per-shard sync today. v8 adds three new top-level message kinds:
================================ ==================================
Message kind Body
================================ ==================================
``v8_validator_announce`` Stake deposit announcement;
triggers ``bond`` window.
``v8_acceptance_signature`` Single validator's accept/reject
sig on a candidate during a
signing window.
``v8_challenge`` Counter-evidence submission
during a challenge window.
================================ ==================================
§8.2 Canonical encoding
~~~~~~~~~~~~~~~~~~~~~~~
Every message body is canonical-JSON (SQD A1) before signing. The
aggregate signature (when threshold-key is bootstrapped) is BLS-
flavored; otherwise the protocol falls back to a sorted-concat of
individual validator signatures, length-prefixed.
A consumer that wants to verify a finalized ``consensus_events``
row independently:
1. Parse the row's ``validator_signatures`` field.
2. For each signature, look up the signing validator's pubkey from
the ``consensus_events`` chain at the signing time.
3. Verify each signature against the canonical encoding of the row
body (minus the signature field).
4. Aggregate stake-weighted; confirm ≥ 2/3 active stake at
signing time.
Step 2 requires the validator set's stake snapshot at the row's
``finalized_at`` timestamp. v8 commits a **stake snapshot** in
every finalized row's body so step 2 is local-only — no separate
lookup table needed.
§8.3 Bandwidth profile
~~~~~~~~~~~~~~~~~~~~~~
At recommended parameters (``signing_window`` = 1 day,
``challenge_window`` = 7 days, ``checkpoint_cadence`` = weekly),
the gossip load is dominated by ``v8_acceptance_signature`` traffic
during the signing window. For a 100-validator deployment with
1 KB per signature message, peak load is
.. code-block:: text
100 validators · 1 KB · gossip_fanout (8) = 800 KB per checkpoint
— well within arborist mesh's per-shard sync budget.
Challenge-window traffic is bursty but bounded (any single party
can submit O(1) challenges per finalized row).
Part 9 — BFT analysis
---------------------
§9.1 Safety
~~~~~~~~~~~
**Claim:** No two conflicting candidates can both finalize under
the 2/3 quorum rule + GRANDPA-style finality, assuming
≤ 1/3 Byzantine stake.
**Proof sketch:** Suppose ``M'_1`` and ``M'_2`` both finalize on
the same parent. Each requires ≥ 2/3 stake to accept. Total stake
signing accept across both candidates ≥ 4/3 — impossible without
overlap. Any validator signing both is automatically slashable per
§6.2 (conflicting acceptances → 100% slash). Therefore the
overlapping validators are Byzantine. By assumption Byzantine
stake ≤ 1/3 < 4/3 − 1 = 1/3 needed-overlap, contradiction.
Conclusion: at most one candidate finalizes per parent.
§9.2 Liveness
~~~~~~~~~~~~~
**Claim:** Under partition where < 1/3 stake is offline,
finalization progresses within bounded time.
**Proof sketch:** ≥ 2/3 stake is online and honest. Honest
validators sign on any candidate clearing the canonical floor.
Quorum is reached within the signing window. After the
challenge window closes without successful challenge, finalization
completes. Bound: ``signing_window + challenge_window``
(recommended: 8 days).
**Claim:** Under partition where ≥ 1/3 stake is offline,
finalization stalls but does not regress.
**Proof sketch:** Without 2/3 quorum, no candidate finalizes.
Pre-finality state remains accessible (downstream consumers see
"pending" status). Once the offline stake rejoins (or is slashed
per §7.3), the protocol resumes.
§9.3 Sybil resistance
~~~~~~~~~~~~~~~~~~~~~
Stake bonding is the Sybil bound. An adversary controlling N
identities still controls only ``sum(N_stake_i)`` weight, not ``N``
votes. With optional sqrt-weighting (§6.4), even concentrated
stake is diluted.
§9.4 Bootstrap honesty
~~~~~~~~~~~~~~~~~~~~~~
The protocol cannot bootstrap from zero. v8 deployment must specify
an **initial validator set** with explicit honesty assumption. The
permacomputer-mission default is:
- Bootstrap from a small (5-7) permissioned validator set, all
publicly identified, with stake deposited to a multi-sig
bootstrap escrow.
- After ``M`` checkpoints (recommend ``M`` = 12, ~3 months at
weekly cadence) without successful challenges, the bootstrap
set's stake unlocks for normal unbond/slash semantics.
- New validators join permissionlessly during bootstrap; they don't
earn rewards until bootstrap closes (prevents bootstrap-stake
dilution attacks).
§9.5 Re-staking attacks
~~~~~~~~~~~~~~~~~~~~~~~
A validator may stake the same capital across multiple v8 instances
(e.g., one per major Merkle-AGI lineage). Out of scope for this
paper — the cross-instance slashing accumulator is its own
follow-up if multi-instance v8 deployments emerge.
Part 10 — Worked example (appendix)
-----------------------------------
§10.1 Setup
~~~~~~~~~~~
Imaginary deployment: 7 active validators ``V1..V7``, each with 100
stake units (700 total). 2/3 threshold = 467 stake. Recommended
parameters: 1-day signing window, 7-day challenge window.
§10.2 Cycle
~~~~~~~~~~~
**Day 0**: Proposer P submits ``C(M'_1)`` against finalized parent
``C(M_t)``. Self-reported fork_score = +0.42.
**Day 0-1 (signing window)**: ``V1..V6`` independently audit-replay,
each producing fork_score in [+0.40, +0.43] (within signal floor).
All six sign ``accept`` with their replay digests. ``V7`` is
offline; doesn't sign. Stake signing accept = 600 ≥ 467.
**Day 1**: Quorum reached. Aggregate signature minted; new row
written to ``consensus_events`` chain. Row status:
``finalized-pending``.
**Day 1-8 (challenge window)**: External party C submits
counter-evidence claiming ``V3``'s replay digest disagrees with
canonical floor by 15pp.
**Day 8**: Adjudication set ``V1, V2, V4, V5, V6`` (excluding
challenged ``V3`` and offline ``V7``) re-audits. Each finds C's
replay matches the canonical result; ``V3``'s replay disagrees by
15pp.
**Day 8**: Adjudication majority (≥ 2/3 of 5 = 4) confirms
``V3``'s misreporting. ``V3`` is slashed 30% (audit-mismatch
class). C earns 30% of slashed stake = 9 units.
**Day 8**: The accept signature is still valid (V3's signature is
struck, remaining stake signing accept = 500 ≥ 467). Row
finalizes.
**Day 8+**: Downstream consumers verify the finalized row by
walking the ``consensus_events`` chain. They see ``M'_1`` is the
canonical successor of ``M_t``.
§10.3 What the audit ledger looks like
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: text
consensus_events row at chain height N+1:
prev_event_hash: <sha256 of row N>
candidate_root: <C(M'_1)>
parent_root: <C(M_t)>
proposer_id: P
fork_score: {breakdown ...}
validator_signatures: [<V1>, <V2>, <V4>, <V5>, <V6>]
(V3 struck post-slash; V7 offline)
audit_replay_digest: <canonical replay sha256>
verdict: "FINALIZED"
finalized_at: <Day 8 ts>
consensus_policy_hash: <hash of v8 policy params>
event_hash = sha256( prev_event_hash || canonical(body) )
A future consumer pulling this row can verify:
1. Each validator signature against the canonical body.
2. Stake-weighted aggregate against the stake-snapshot inside the
body. ≥ 2/3.
3. Chain integrity against the previous row's ``event_hash``.
No external lookup required.
Part 11 — Out of scope
----------------------
- **Implementation**. Code lands in 3-4 follow-up tickets:
validator state-machine SQL + state-transition functions; mesh
wire format extension; audit-replay harness binding; slashing
accountant.
- **Economic parameter calibration**. Stake amounts, slashing
fractions, reward rates, challenge bonds — governance decisions,
not paper-pinned. The paper specifies the *form*; deployments
calibrate.
- **Cross-chain anchoring**. Publishing finalized
``consensus_events`` rows to Bitcoin / Ethereum / etc. is an
optional bolt-on. Out of scope here.
- **Frontier benchmark fixture selection**. Covered by ticket
#000021. v8 references the fixtures; doesn't curate them.
- **Bootstrap-set membership**. The actual identities of the
bootstrap validators are a governance decision for the deploying
consortium.
- **Cross-instance slashing accumulator**. Re-staking attacks
across multiple v8 instances. Future work; emerges when more than
one v8 instance exists.
- **Branch-set persistence**. Ticket #000012 Phase 1c (proposed,
not yet open). Multi-branch fork scoring at the same checkpoint.
Doc-only proposal in #000012; lands as its own follow-up if
operator pressure surfaces.
Closure
-------
Closure criterion (#000012 §4.1): this document plus the
``arborist.substrate.fork_score`` module (Phase 1a; landed 2026-05-08)
constitute the v8 substrate paper. The reference doc
``docs/v8-fork-score.md`` carries the formula details; this paper
carries the consensus / acceptance / slashing protocol.
Open questions tracked separately:
- **Initial validator set composition** — governance, not paper.
- **Threshold-key ceremony** — needed for BLS-aggregate signatures;
if not bootstrapped, fallback to sorted-concat signatures is
fully functional (just larger payloads).
- **ZK-replay** — verifiers proving canonical-floor replay without
re-running it locally. Captured in #000016 (ZK frontier proof);
optional optimization on top of v8.
- **Policy-hash transition mechanics** — when
``consensus_policy_hash`` changes (e.g., new floor metric added),
how do in-flight signing windows handle the transition? Recommend:
policy changes take effect at the next checkpoint after a
successful finalization under the new policy; in-flight
signatures honor the policy hash they were submitted under.
Implementation tickets that cite this paper land later — one per
validator-state-machine, mesh-wire-format extension, audit-replay
harness, and slashing accountant.
Status: closure-draft 2026-05-10. Ready for fox + downstream
review.

62
docs/tickets/ticket-000012-selection-consensus-protocol.md
View file

@ -1,6 +1,6 @@
# Ticket #000012 — Selection & consensus protocol (Merkle-AGI v8)
**Status:** in progress · Phase 1a (ForkScore) landed 2026-05-08; consensus paper still open
**Status:** in progress · Phase 1a (ForkScore) landed 2026-05-08; Phase 1b (consensus paper) landed 2026-05-10; Phase 1c (branch-set persistence) remains proposed-not-opened
**Opened:** 2026-05-07
**Scope:** Spec for the loop-closing consensus protocol that turns the v7
substrate + v9.8 runtime into actual Darwinian selection across multiple
@ -287,26 +287,54 @@ function ahead of the consensus paper:
inf-bonus capping, neg-inf rejection, weight-set tuning,
breakdown completeness, CLI smoke. Full suite: 1186 passed.
### Phase 1b — Consensus paper (still open)
### Phase 1b — Consensus paper (landed 2026-05-10)
Closure criterion: `docs/merkle-agi-v8-consensus.rst` lands with
validator state machine, acceptance protocol, challenge protocol,
fork-choice rule (GRANDPA-style), slashing mechanics, mesh wire
format extension. Per §1 of this ticket,
`arborist/substrate/fork_score.py` is the substrate the paper
cites; the paper itself is still research-scope.
Paper landed at `docs/_source/merkle-agi-v8-consensus.rst` (834
lines, RST sister to the v7-W substrate paper at the same path).
Sections delivered:
What's deliberately NOT in Phase 1a:
- **Part 1** — Introduction & motivation; gap table from v7 § 13.4;
concrete backdoor-attack scenario; paper IS/IS-NOT scope.
- **Part 2** — Substrate definition; SQD A1/A2/A3 inheritance;
`consensus_events` row schema; `consensus_policy_hash` sibling
(never enters `cache_key`).
- **Part 3** — Validator state machine (bonding / active /
challenged / slashed / unbonding) with full transition graph and
invariants.
- **Part 4** — Acceptance protocol; layered fitness floor
(canonical + lab-declared ceiling); audit-replay procedure;
2/3-stake quorum + GRANDPA-style finalization; liveness floor.
- **Part 5** — Challenge protocol; counter-evidence shape;
adjudication; challenger reward (30% recommended) + frivolous-
challenge bond.
- **Part 6** — Stake mechanics; bond/unbond/challenge window
recommendations; offense-class slashing schedule; reward
distribution; optional stake cap + sqrt-weighting.
- **Part 7** — Fork choice rule (GRANDPA-style); finalization
permanence; pre-finality fork-choice constraints; liveness
recovery from offline-stake partition.
- **Part 8** — Mesh wire format extension (three new message
kinds; BLS-or-concat aggregate signatures; bandwidth profile).
- **Part 9** — BFT analysis (safety, liveness, Sybil resistance,
bootstrap honesty, re-staking attacks).
- **Part 10** — Worked example: 7-validator deployment, one
upgrade cycle with successful challenge against one fraudulent
validator.
- **Part 11** — Out of scope (implementation, calibration, cross-
chain anchoring, fixture selection, bootstrap-set membership,
cross-instance slashing accumulator, branch-set persistence).
- **Closure** — open questions: initial-set composition, threshold-
key ceremony, ZK-replay, policy-hash transition mechanics.
- Validator state machine (bonding / signing / slashing).
- Acceptance protocol.
- Challenge protocol.
- Fork-choice rule.
- Mesh wire format extensions.
- Stake mechanics + economic incentives.
- Cross-validator ZK proof exchange.
What's deliberately NOT in Phase 1a (now specified in the v8 paper):
These belong to the v8 paper itself.
- Validator state machine (bonding / signing / slashing). → Part 3
- Acceptance protocol. → Part 4
- Challenge protocol. → Part 5
- Fork-choice rule. → Part 7
- Mesh wire format extensions. → Part 8
- Stake mechanics + economic incentives. → Part 6
- Cross-validator ZK proof exchange. → Closure §; deferred to #000016
### Phase 1c — Branch-set persistence (proposed, not yet open)