arborist/docs/mesh-deploy.md

Mesh deploy — two-host runbook

This is the operator runbook for standing up an aborist mesh between two real hosts. The protocol is documented in docs/mesh.md; this file covers what to type, in what order, with what to verify at each step.

Every step assumes both peers have aborist installed and have already ingested some local corpus. If you only want to test the wire on localhost-loopback, see tests/test_mesh_wire_e2e.py instead.

Naming

Two hosts in this runbook:

• alice — admin, founder of group myteam. Runs mesh serve.
• bob — joining member. Runs mesh sync outbound to alice.

Substitute hostnames / IPs / ports per your environment.

0. Pre-flight on each host

# Confirm aborist + extras + tests:
make bootstrap
make test
.venv/bin/aborist --version

Both hosts should be on the same git commit. Mismatched schema_version / chunking_version / canonicalization_version will be rejected by the v9.8 admissibility check; aborist refuses silent corruption.

1. Initialize mesh on each peer

# alice — first peer becomes founder of epoch 0:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh init --group myteam --member-id alice
.venv/bin/aborist --db ~/.aborist/aborist.db mesh enable
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status

# bob — initializes a separate identity in the same group name:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh init --group myteam --member-id bob
.venv/bin/aborist --db ~/.aborist/aborist.db mesh enable
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status

At this point alice and bob both think they're the sole member of myteam. The next step adds bob to alice's roster — and vice versa — so both rosters carry both pubkeys at the same epoch.

2. Exchange pubkeys (out-of-band)

bob's public keys must reach alice via a trusted channel. Mesh has no built-in introduction protocol; signal, in-person hand-off, or a signed file all work.

# On bob:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status | jq '.identity'
# -> { "member_id": "bob", "sign_pub_hex": "...", "dh_pub_hex": "..." }

Send the two hex strings to alice. Verify the channel out of band.

Same direction in reverse so bob can enroll alice:

# On alice:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh status | jq '.identity'

3. Mutual enrollment

# On alice (admin) — enroll bob at alice's epoch:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh add \
    --member-id bob \
    --sign-pub <bob_sign_pub_hex> \
    --dh-pub   <bob_dh_pub_hex>

# On bob — enroll alice at bob's epoch (so bob's local roster knows
# alice's pubkey for signature verification when alice's announces
# arrive):
.venv/bin/aborist --db ~/.aborist/aborist.db mesh add \
    --member-id alice \
    --sign-pub <alice_sign_pub_hex> \
    --dh-pub   <alice_dh_pub_hex>

Each mesh add bumps the local epoch and writes a mesh_epoch_rotate audit event. Verify:

.venv/bin/aborist --db ~/.aborist/aborist.db mesh status
.venv/bin/aborist --db ~/.aborist/aborist.db mesh members

Both peers should now show 2 members at epoch 1.

4. Start alice's gossip server

# On alice — bind on a routable interface:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh serve \
    --host 0.0.0.0 --port 8400
# Stdout: {"status": "serving", "url": "http://0.0.0.0:8400", ...}

Leave this running. Open the firewall to allow bob to reach alice:8400. Production deployments should put a TLS terminator (Caddy / nginx) in front; the wire speaks plain HTTP — TLS is the operator's choice.

Verify reachability from bob:

# On bob:
curl http://alice.example.com:8400/mesh/info | jq
# -> { "v": 1, "member_id": "alice", "group_name": "myteam",
#      "current_epoch": 1, "sign_pub_hex": "..." }

If member_id matches what bob has in his roster for "alice", and the sign_pub_hex matches what bob enrolled, you're good. If they don't match, bob is talking to the wrong peer or alice's enrollment was wrong; do not push gossip until the discrepancy is resolved.

5. Bob announces his local roots to alice

# On bob:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh sync \
    --peer http://alice.example.com:8400 \
    --limit 100

Output: JSON status, count of acked vs errored announces.

Each accepted announce writes one mesh_received event into alice's audit chain. Verify on alice:

sqlite3 ~/.aborist/aborist.db \
    "SELECT COUNT(*) FROM audit_events WHERE event_type='mesh_received'"

Cross-check chain integrity:

make chain-check

Should return 0 chain breaks.

6. Make sync bidirectional

The current mesh sync only pushes outbound. To get alice's roots onto bob, run mesh sync from alice pointed at bob:

# On alice (in a new terminal — keep mesh serve running):
.venv/bin/aborist --db ~/.aborist/aborist.db mesh sync \
    --peer http://bob.example.com:8400 --limit 100

This requires bob to also be running mesh serve. Symmetric setup is the standard mode.

7. Pulling missing bodies (when shipped)

Today only ANNOUNCE_ROOT is exercised by mesh sync. The wire layer already implements REQUEST_BODY/DELIVER_BODY with Merkle verification on the client side; the CLI hookup (mesh pull --root <hex> --peer <url>) is on the queue. Once shipped, the pull-on-miss workflow is:

# bob discovers via announces that alice has document X he doesn't:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh pull \
    --root <document_root_hex> \
    --peer http://alice.example.com:8400
# bob's local store now has X (Merkle-verified at receive).

8. Eviction protocol

If bob leaves the team, alice (admin) kicks. This bumps alice's epoch and rewraps the next epoch secret to the post-bob roster:

# On alice:
.venv/bin/aborist --db ~/.aborist/aborist.db mesh kick \
    --member-id bob \
    --reason "left team 2026-04-28"

Bob's prior signatures stay verifiable forever (his roster row at older epochs is preserved on disk). Any AEAD-protected gossip from epoch+1 onward is opaque to him — that's the eviction guarantee.

Important: kicking only mutates alice's local state. If bob's host is still running mesh serve, alice should also stop sending to him (or he'll keep accepting alice's announces under the OLD epoch she's no longer broadcasting from). Operationally, kicks should pair with either: (a) dropping bob's URL from alice's sync targets, OR (b) announcing the kick to remaining members so they update their rosters.

9. Operational checks

Every chain-write op (mesh init, mesh add, mesh kick, mesh rotate, every received announce) extends the local audit chain. The fast probe is:

make chain-check         # single db
make chain-check-shards  # every db in $(SHARDS_DIR)

Both should always print 0. Non-zero = operator must investigate before further operations; data has diverged from the audit trail.

10. Reset / teardown

To take a peer fully out of the mesh (irreversible — fresh keys needed to rejoin):

.venv/bin/aborist --db ~/.aborist/aborist.db mesh disable
# Identity + roster history stay on disk for forensics. Re-enable
# requires another peer to re-add this member at a fresh epoch.

For full local nuke (developers only — destroys identity + chain):

sqlite3 ~/.aborist/aborist.db <<'SQL'
DELETE FROM mesh_identity;
DELETE FROM mesh_roster;
DELETE FROM mesh_epochs;
SQL

What's not yet shipped

These features are documented in the protocol but not wired into the CLI yet:

• mesh pull <root> — fetch a body on cache miss.
• Per-peer audit-chain merge — receiver tracks each sender's last known event_hash and rejects fork events. Today the receiver verifies signatures only; chain-of-claims tracking is deferred.
• AEAD body encryption — wire signs envelope bytes for integrity; body confidentiality (encrypting against the per-epoch shared secret) is optional in the contract and not yet wired.

When these land, this runbook will gain steps for them.