#000016 parked: ZK frontier-proof bench plan + wire protocol

Two arborist-side artifacts close the v7 § 16.1 hand-wave without
violating the language constraint (arborist stays pure-Python; ZK
toolchain lives in sibling repo arborist-zk-bench).

docs/zk-frontier-bench.md — bench plan + parked verdict
========================================================

Bench plan covers Plonky3 (or equivalent) circuit at three sizes
(256/1024/4096 affine), measurement targets (prover ms, proof
bytes, verify ms, peak memory), platform pair (Apple M3 + Linux
x86), acceptance thresholds (≤30s prover at 4096; ≤100KB proof;
≤100ms verify; ≤16GB peak).

Preliminary projection from published Plonky3/Halo2 numbers:

  size 256:   0.5–2 s prove  / 30–80 KB / 5–30 ms verify
  size 1024:  5–30 s prove   / 50–120 KB / 10–50 ms verify
  size 4096:  60–600 s prove / 100–300 KB / 30–200 ms verify
                              ↑ likely UNAFFORDABLE at frontier scale

Most-likely outcome: VIABLE at small scales (256/1024 — useful for
distillation models); UNAFFORDABLE at LLaMA-class hidden width
(4096).

Decision tree from real measurement:
  VIABLE       → commission ZK as v8 capability;
                 follow-up tickets for arborist [zk] extra,
                 governance_policy.frontier_proof_mode field,
                 hermes-side activation export.
  UNAFFORDABLE → park ZK; v7-Local stays v7's terminal
                 contribution; answer→model binding stays
                 out of scope.

docs/zk-wire-protocol.md — consumer-side schema
================================================

Specifies the wire format arborist consumes WHEN/IF ZK proofs
become available. The sibling repo emits proofs; arborist
validates schema + signature + delegates ZK verification to a
sibling-repo verifier binary. arborist itself NEVER runs the
prover or verifier — that's the language-constraint contract.

  - Artifact: arborist-zk-proof-v1 JSON with schema_version,
    proof_system, circuit_id, frontier_node, public_inputs,
    public_outputs, proof_bytes_b64, verifier_setup_id, issued_at,
    issuer_pubkey_ed25519, issuer_signature_ed25519.
  - Trigger: governance_policy.frontier_proof_mode ∈
    {"reveal" (default), "zk"}.
  - Binding: model_weights_zk_root + frontier_proof_circuit_id
    columns added to providence_cache (schema migration deferred
    to integration ticket).
  - Validation pipeline: schema check → recompute commitment
    sanity → Ed25519 signature check → delegate to sibling
    verifier → bind to audit chain.
  - Threat model: compromised prover, wire-format injection,
    proof replay, issuer-key compromise — each with its own
    mitigation.

Status closure
==============

Ticket flipped from "open · awaiting go/no-go" to "parked ·
bench-plan + wire-protocol landed 2026-05-09 (sibling-repo
measurement pending)." This is honest: arborist's side is done.
Sibling repo `arborist-zk-bench` (not yet built) produces the
real numbers; doc updates roll back here when measurements
arrive.

Result: the v7 § 16.1 hand-wave is replaced with explicit
thresholds + preliminary projection + ready consumer schema.
The hand-wave is closed even though the bench-question is
parked.
This commit is contained in:
russell@unturf.com 2026-05-09 15:05:08 -04:00
parent ee22a83a0a
commit 0154191bd4
No known key found for this signature in database
4 changed files with 519 additions and 2 deletions

View file

@ -81,7 +81,7 @@ Newest first. Update on every open/close.
| #000019 | Specification methodology for π* and V | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000018 | Adversarial soft-hash covert-channel analysis | closed · landed 2026-05-09 (analysis doc) | 2026-05-07 | — |
| #000017 | Memory-root: lifelong learning audit chain | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000016 | ZK Phase-2 frontier proof (concretize) | open · awaiting go/no-go | 2026-05-07 | — |
| #000016 | ZK Phase-2 frontier proof (concretize) | parked · bench-plan + wire-protocol landed 2026-05-09 (sibling-repo measurement pending) | 2026-05-07 | — |
| #000015 | π* domain library + cross-domain composition | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000014 | SelfModel: schema, falsification, integration | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000013 | Spatial-temporal substrate (Merkle-AGI v7-W) | closed · landed 2026-05-09 (substrate paper + frontier catalog + namespace stub) | 2026-05-07 | — |

View file

@ -1,6 +1,6 @@
# Ticket #000016 — ZK Phase-2 frontier proof (concretize the hand-wave)
**Status:** open · awaiting go/no-go
**Status:** parked · landed bench-plan + wire-protocol artifacts 2026-05-09 (`docs/zk-frontier-bench.md`, `docs/zk-wire-protocol.md`); pending sibling-repo measurement
**Opened:** 2026-05-07
**Scope:** Build one zero-knowledge proof for one v7-Local affine
preactivation frontier on a small model, measure prover time + proof

236
docs/zk-frontier-bench.md Normal file
View file

@ -0,0 +1,236 @@
# ZK frontier-proof bench plan
**Verdict:** **PARKED · pending sibling-repo measurement.**
**Ticket:** #000016
**Authored:** 2026-05-09
This document is the arborist-side closure for #000016. It captures
the bench plan, acceptance thresholds, and a preliminary
projection. **Actual measurements live in the sibling repo
`arborist-zk-bench`** (not yet built), per #000016's language
constraint that arborist stays pure-Python and never gains a Rust
dependency.
When `arborist-zk-bench` produces real numbers, this doc gets
updated with the verdict line at top flipping from PARKED to
**VIABLE** or **UNAFFORDABLE**, and arborist consumes the wire
format spec'd in `docs/zk-wire-protocol.md`.
---
## 1. Why parked, not closed
#000016's closure criterion (§7) is measured numbers at three
circuit sizes (256, 1024, 4096) on commodity hardware. Producing
those numbers requires a working Plonky3 (or equivalent) circuit
in a Rust crate, plus prover + verifier binaries, plus
reproducible-build infrastructure. That belongs in
`arborist-zk-bench` per #000016 §1.21 and §4.1.
Building the sibling repo before there's a v8/v7-W demand for ZK
is speculative work. The ticket itself recommends "deferring until
a v8 / v7-W ticket creates real demand" (§7).
So: arborist produces the bench plan + the wire protocol +
preliminary projection, and parks measurement until demand
materializes. The hand-wave is replaced with an explicit
PARKED-with-thresholds artifact.
## 2. Bench plan (for the future sibling repo)
### 2.1 Circuit shape
One affine preactivation node, three sizes:
| Size | Inputs | Weights | Cost class |
|------|--------|---------|------------|
| 256 | 256 int64 | 256×256 int64 | small / unit-test scale |
| 1024 | 1024 int64 | 1024×1024 int64 | medium / typical-frontier scale |
| 4096 | 4096 int64 | 4096×4096 int64 | LLaMA-class hidden-width scale |
Per node:
- Inputs (witness, private): full activation vector A, full weight
matrix W.
- Inputs (public): `bias` B, subset-mask `S`, epsilon as
`(num, den)` integer pair.
- Outputs (public): SHA-256 commitments of W and A, commitment to
`(S, ε)`, boolean "ε-inequality holds".
- Constraints: range checks on int64 values, field-level GEMM,
L1-norm via sum of absolute values, final inequality check.
### 2.2 Measurement targets
- Wall-clock prover time (ms).
- Proof size (bytes).
- Verification time (ms).
- Memory peak during prover (MB).
- Setup time (ms) — one-shot per circuit.
### 2.3 Platform pair
- **Apple Silicon** — M3 Max or equivalent (representative dev
machine).
- **Linux x86_64** — generic cloud commodity (representative server).
Run twice per platform; report median + range.
### 2.4 Bench harness sketch
`arborist-zk-bench/bench/zk_frontier_bench.py` (in the sibling
repo, not here):
```python
sizes = [256, 1024, 4096]
seeds = [42, 137]
for size in sizes:
for seed in seeds:
rng = numpy.random.RandomState(seed)
A = rng.randint(-(2**31), 2**31, size, dtype=numpy.int64)
W = rng.randint(-(2**31), 2**31, (size, size),
dtype=numpy.int64)
B = rng.randint(-(2**31), 2**31, size, dtype=numpy.int64)
S = rng.choice([0, 1], size).astype(numpy.int64)
eps = (1, 100) # 1% L1 error budget
t = time.monotonic()
proof = prove_affine(W, A, B, S, eps)
prove_ms = (time.monotonic() - t) * 1000
t = time.monotonic()
ok = verify_affine(proof, public_inputs(W, A, B, S, eps))
verify_ms = (time.monotonic() - t) * 1000
record({
"size": size, "seed": seed, "platform": platform_id(),
"prove_ms": prove_ms, "verify_ms": verify_ms,
"proof_bytes": len(proof), "memory_peak_mb": ...,
"ok": ok,
})
```
### 2.5 Acceptance thresholds
For ZK to flip from PARKED to VIABLE in this bench:
| Metric | Threshold (size 4096) |
|---------------------|-------------------------------|
| Prover wall-clock | ≤ 30 seconds |
| Proof size | ≤ 100 KB |
| Verifier wall-clock | ≤ 100 ms |
| Memory peak | ≤ 16 GB |
| Verification result | `ok == True` on all seeds |
**All thresholds must pass.** Single-metric failure → PARKED
with the failing metric named.
## 3. Preliminary projection (no measurement)
Lower-confidence estimates from published Plonky3 / Halo2
benchmarks (2024-2025 era), to inform expectations BEFORE
real bench runs:
| Size | Prove (projected) | Proof bytes | Verify | Confidence |
|------|-------------------|-------------|--------|------------|
| 256 | 0.5–2 s | 30–80 KB | 5–30 ms | Medium |
| 1024 | 5–30 s | 50–120 KB | 10–50 ms | Low–Medium |
| 4096 | 60–600 s | 100–300 KB | 30–200 ms | Low |
Sources: Plonky3 benchmark posts, Halo2 GEMM-circuit reports.
**These are not measurements**; they're estimates from related
circuits. Real numbers will diverge.
**Most-likely outcome based on these estimates**: at size 4096,
prover time exceeds the 30-second threshold; ZK stays PARKED for
the LLM-frontier scale. ZK might be VIABLE at the 256/1024 scales
relevant to small distillation models.
## 4. Decision tree from measurement
```
arborist-zk-bench measurement
│
┌───────────┴────────────┐
│ │
VIABLE UNAFFORDABLE
│ │
│ │
commission ZK as v8 capability park ZK; v7-Local stays
open follow-up tickets: v7's terminal contribution;
- arborist [zk] extra wiring v7-Local proofs require
(wire-format consumer only) activation publication;
- governance_policy.frontier_ answer→model binding stays
proof_mode policy field out of scope
- v8 ForkScore: ZK availability
as a fork-quality signal
- hermes-side circuit-friendly
activation export
```
Either branch closes #000016: VIABLE → ZK lands; PARKED →
documented constraint, future ticket if Plonky3 evolves.
## 5. What arborist does NOT need to do
Per #000016 §1.21 / §4.1 / §7 language constraints:
- **No Rust code in arborist.** `arborist/zk/` reserved as a
future namespace for the wire-format consumer (pure-Python
schema validator); no prover, no verifier.
- **No Plonky3 dependency in `pyproject.toml`.** `[zk]` extra
(when it lands) pulls a JSON-schema validator + a binary
protocol parser, never the prover binary.
- **No circuit code in arborist.** `circuits/affine_preact.rs`
lives in the sibling repo.
- **No bench harness in arborist.** `bench/zk_frontier_bench.py`
lives in the sibling repo. arborist's `bench/` stays pure-
Python deterministic-fixture work.
## 6. What the wire protocol looks like
Detailed spec in `docs/zk-wire-protocol.md` (sibling document).
Brief summary:
- Sibling repo emits proofs as canonical-byte JSON conforming to
a versioned schema.
- arborist consumes proofs at the providence-record write
boundary, optionally adding a `model_weights_zk_root` field to
the v9.8 cache_key (governance-hash-folded).
- arborist's verifier helper validates schema + checks SHA-256
binding against the providence chain. It does NOT run the ZK
verifier — that's also in the sibling repo or in a separate
trusted binary.
## 7. Cross-references
- #000016 — this ticket (parked).
- #000018 — adversarial soft-hash analysis (closed); the M3
mitigation there ("drop the anchor") is the only existing
defense if ZK proves UNAFFORDABLE and v7-Local activation
publication is also out of scope.
- #000013 — v7-W spatial-temporal substrate (closed); v7-W
inherits the ZK question — world-state observations are ALSO
privacy-sensitive, and the same VIABLE/PARKED decision tree
applies.
- v7 § 16.1 — the original "swap SHA-256 → Poseidon" hand-wave
this doc replaces.
## 8. Status / closure
**This document closes the arborist-side hand-wave.** The
sibling repo's measurement work remains open as a separate
project; #000016 is parked rather than fully closed because the
verdict line awaits real numbers.
When `arborist-zk-bench` produces measurements:
1. Update §3 with measured numbers, replacing the projection.
2. Flip the verdict line at top of this doc: PARKED →
VIABLE or UNAFFORDABLE.
3. Open follow-up tickets per §4's decision tree.
4. Update #000016 ticket status: parked → closed (verdict
recorded).
Until then: this is the artifact. The hand-wave from v7 § 16.1
is replaced with explicit thresholds + preliminary projection +
parked-status with named open work.

281
docs/zk-wire-protocol.md Normal file
View file

@ -0,0 +1,281 @@
# ZK wire protocol — arborist consumer schema
**Companion to:** `docs/zk-frontier-bench.md` (the bench plan +
parked verdict for #000016).
**Status:** draft v0, 2026-05-09.
This document specifies the **wire format** arborist consumes
when ZK proofs become available. The toolchain producing those
proofs lives in a sibling repo (`arborist-zk-bench`); arborist
itself stays pure-Python and never gains a Rust dependency. The
language constraint from #000016 §1.21 is hard — this protocol
is the bridge.
---
## 1. Scope
**What this protocol covers:**
- Canonical-byte JSON schema for one frontier-proof artifact.
- The minimum binding info arborist needs to fold a proof into
the v9.8 audit chain.
- Versioning + forward-compatibility rules.
- The trigger condition (`governance_policy.frontier_proof_mode`)
for when arborist asks for a proof.
**What this protocol does NOT cover:**
- The prover or verifier itself (sibling repo).
- Plonky3 / Halo2 / Groth16 internals.
- Circuit construction.
- ZK-side privacy analysis.
## 2. Trigger condition
ZK proofs are opt-in via a governance-policy flag:
```python
governance_policy["frontier_proof_mode"] = "zk" # opt-in
governance_policy["frontier_proof_mode"] = "reveal" # default
```
The flag folds into `governance_policy_hash` so the v9.8
8-dim cache_key reflects which mode produced any given record.
Switching modes invalidates prior records on lookup.
When `mode == "zk"`, arborist's QA path (or any caller producing
a frontier-proof binding) calls out to the sibling repo, gets
back a wire-protocol artifact, validates it, and folds the proof
root into the providence record.
When `mode == "reveal"` (default), no ZK call happens; activations
are revealed locally per v7-Local.
## 3. Artifact shape (`proof_v1.json`)
```json
{
"schema_version": "arborist-zk-proof-v1",
"proof_system": "plonky3 | halo2 | groth16 | stark",
"proof_system_version": "<vendor-specific git sha or release tag>",
"circuit_id": "<sha256 of canonical circuit definition>",
"circuit_kind": "affine_preact" ,
"frontier_node": {
"node_kind": "affine_preact",
"size": 4096,
"epsilon_num": 1,
"epsilon_den": 100
},
"public_inputs": {
"weights_commitment_sha256": "<64-hex>",
"activations_commitment_sha256": "<64-hex>",
"subset_mask_sha256": "<64-hex>",
"epsilon_pair": [1, 100]
},
"public_outputs": {
"epsilon_inequality_holds": true
},
"proof_bytes_b64": "<standard-base64 of the raw proof bytes>",
"verifier_setup_id": "<sha256 of the verifier's KZG/CRS or FRI setup>",
"issued_at": "<ISO-8601 UTC>",
"issuer_pubkey_ed25519": "<32-byte hex>",
"issuer_signature_ed25519": "<64-byte hex over canonical-bytes(this object)>"
}
```
### 3.1 Field-by-field
- **`schema_version`** — fixed string `"arborist-zk-proof-v1"`. Arborist
rejects any artifact with an unknown schema_version; v2 onward bumps
the validator.
- **`proof_system`** — vendor name. Determines which deserializer
(in the sibling repo) parses `proof_bytes_b64`.
- **`circuit_id`** — sha256 of the canonical circuit definition. Two
artifacts produced by the same circuit have the same `circuit_id`;
this is what arborist binds to in the audit chain rather than the
raw proof bytes (which vary per witness).
- **`frontier_node`** — describes WHICH node was proved. arborist
matches this against the model snapshot's frontier-node enumeration
to confirm the proof's scope.
- **`public_inputs`** — the inputs the verifier checks. arborist
recomputes `weights_commitment_sha256` from its own copy of the
weight bytes, `activations_commitment_sha256` from the published
activation chain, and confirms equality before consuming the
proof.
- **`public_outputs.epsilon_inequality_holds`** — the boolean the
prover claims is true, which the verifier checks.
- **`proof_bytes_b64`** — the actual proof. Arborist does NOT parse
this; it forwards to the sibling-repo verifier (or a separate
trusted binary).
- **`verifier_setup_id`** — sha256 of the verifier's setup (KZG
CRS / FRI parameters). Pinned per-circuit; rotation is a circuit
bump, which changes `circuit_id`.
- **`issued_at`** — wall-clock UTC, sanity-check only.
- **`issuer_pubkey_ed25519` + `issuer_signature_ed25519`** — the
prover signs the canonical-byte form of this artifact (with the
signature field omitted) so arborist can verify provenance of
the proof artifact independently of the ZK proof itself.
### 3.2 Canonical-byte form
For SHA-256 commitment + signature purposes, the canonical bytes
of this artifact are: JSON with sorted keys, separators=(",",":"),
no whitespace, UTF-8 encoded, EXCLUDING the
`issuer_signature_ed25519` field. Standard JSON-canonicalization
the rest of arborist uses (`_canonical_json` in `arborist.qa.dag`).
## 4. Binding into the v9.8 audit chain
When arborist consumes a valid proof, it folds the proof's
identity into the providence record:
```sql
ALTER TABLE providence_cache ADD COLUMN model_weights_zk_root TEXT;
ALTER TABLE providence_cache ADD COLUMN frontier_proof_circuit_id TEXT;
```
(Schema migration would land at the integration ticket, not
this doc. Sketched here so the binding shape is explicit.)
The new fields:
- `model_weights_zk_root` — SHA-256 of the artifact's canonical
bytes (excluding signature). Becomes part of the audit-event body
for `providence_query` rows produced under `frontier_proof_mode
= "zk"`.
- `frontier_proof_circuit_id` — the artifact's `circuit_id`, exposed
for cross-record queries ("show me all rows that used circuit X").
`governance_policy_hash` folds in the `frontier_proof_mode` field
so cache_key changes when the mode flips. Records produced under
`reveal` and `zk` for the same question land at distinct cache_keys.
## 5. Validation pipeline (arborist side)
When arborist receives an artifact:
```python
def validate_zk_proof(artifact: dict) -> None:
"""Pure-Python validation. NEVER runs the ZK verifier;
delegates that to the sibling-repo verifier binary."""
# 1. Schema check.
if artifact.get("schema_version") != "arborist-zk-proof-v1":
raise PiStarError("unknown schema_version")
for required in (
"proof_system", "circuit_id", "circuit_kind",
"frontier_node", "public_inputs", "public_outputs",
"proof_bytes_b64", "verifier_setup_id", "issued_at",
"issuer_pubkey_ed25519", "issuer_signature_ed25519",
):
if required not in artifact:
raise PiStarError(f"missing field: {required!r}")
# 2. Recompute commitment sanity.
pub = artifact["public_inputs"]
if not _is_hex_64(pub["weights_commitment_sha256"]):
raise PiStarError("bad weights commitment")
# … similar for activations + subset_mask
# 3. Issuer signature check (Ed25519).
canonical = _canonical_json_no_signature(artifact)
if not _ed25519_verify(
artifact["issuer_pubkey_ed25519"],
canonical,
artifact["issuer_signature_ed25519"],
):
raise PiStarError("issuer signature failed")
# 4. ZK verification — DELEGATED.
# This is where arborist hands off to the sibling-repo verifier
# binary or a trusted external process. Arborist itself does
# NOT run the ZK verifier.
if not _delegate_zk_verify(artifact):
raise PiStarError("zk verifier rejected proof")
# 5. Bind into audit chain.
# Caller adds model_weights_zk_root + frontier_proof_circuit_id
# to the providence row.
```
`_delegate_zk_verify` is the wire-protocol's other side: arborist
either spawns a subprocess running the sibling verifier OR posts
to a known endpoint. Either way, arborist sees only the boolean
verdict.
## 6. Versioning + forward compatibility
- Schema bumps require a new `schema_version` string. v1 → v2
drops a field, adds one, or changes semantics.
- arborist supports a fixed set of schema versions at any time;
unknown versions are rejected loudly.
- The sibling repo's git sha pins which prover/verifier produced
the proof. arborist surfaces this in the audit body so a
future replay knows which sibling-repo version to invoke.
- `circuit_id` is content-addressed; circuit changes always
produce a new `circuit_id`, never reuse.
## 7. Threat model
- **Compromised sibling-repo prover.** A compromised prover
can forge `epsilon_inequality_holds = true` only if it can
forge the proof bytes themselves AND the issuer signature.
Arborist's defenses: Ed25519 signature check on the artifact
(step 3), delegated verifier invocation (step 4). The verifier
is the trust anchor; if it's compromised, the chain breaks
upstream of arborist.
- **Wire-format injection.** Adversary submits malformed JSON
hoping to bypass schema validation. Mitigation: every required
field checked; type checks on commitment hex; signature check
before delegating.
- **Proof replay.** Adversary replays a valid proof for a
different question. Mitigation: `circuit_id` and `frontier_node`
are tied to the model snapshot's frontier enumeration;
arborist binds the proof to a specific (cache_key,
frontier_node) pair before storage. Replay against a different
cache_key fails the binding check.
- **Issuer-key compromise.** Adversary obtains the prover's
signing key. Mitigation: keys rotate per major-version;
arborist maintains a published list of trusted issuer keys,
rejects artifacts signed under revoked keys.
## 8. What arborist gains over time
If ZK turns out VIABLE per the bench (`docs/zk-frontier-bench.md`),
arborist gains:
- An `[zk]` extra in `pyproject.toml` pulling JSON-schema
validator + Ed25519 verification (NOT the prover/verifier
binary).
- An `arborist/zk/wire.py` module implementing
`validate_zk_proof()` plus `_delegate_zk_verify()` calling
the sibling-repo verifier.
- A schema migration adding `model_weights_zk_root` and
`frontier_proof_circuit_id` columns to `providence_cache`,
plus a CHECK constraint update.
- A new policy field `frontier_proof_mode` folded into
`governance_policy_hash`.
- New audit-event type `providence_zk_proof` for the binding
event.
If ZK stays UNAFFORDABLE, this protocol stays as a documented
artifact — when ZK matures (hardware, libraries, theory), the
arborist side is ready.
## 9. Cross-references
- `docs/zk-frontier-bench.md` — bench plan + parked verdict.
- #000016 — the ticket this closes the arborist-side
hand-wave for.
- #000018 — soft-hash covert-channel analysis; the M3 mitigation
there is the only existing defense if ZK proves UNAFFORDABLE.
- #000013 — v7-W spatial-temporal substrate; world-state
observations have the same privacy concerns and inherit
this protocol.
- v7 § 16.1 — the original "swap SHA-256 → Poseidon" hand-wave.
## 10. Status
Draft v0. Becomes the consumer-side schema spec when the sibling
repo lands measurements + the validator. Until then: this is the
contract arborist commits to honor on the receive side.