arborist/docs/tickets/ticket-000035-prg-choice-phi-prg.md

Ticket #000035 — PRG choice for φ_PRG

Status: closed · 2026-05-11 — Phase 1 + Phase 2 landed. Phase 1: arborist/substrate/anchor_prg.py (HMAC-SHA-512 counter-mode KDF, pure stdlib, PHI_PRG_VERSION = "phi-prg-v1-hmac-sha512-le", dim_h ≤ 16·2^32 guard + bool-reject), 31 tests, 10 KAT vectors. Amendment text dav1d-reviewed-final 2026-05-11 (manifest field anchor_prg_seed; float-map prose "negligible mean −2^−32" not "unbiased"; seed-independence / single-purpose; §9.10.1 M1-enablement-policy separation; endianness little-endian per v7 §A1, counter + uint32-word reads flipped, KATs regenerated; HMAC-SHA-512 / 32-byte seed / SHALL-replace all LOCKED). Phase 2: § 9.10 + § 9.10.1 folded into the in-repo v7 substrate spec — docs/_source/merkle-agi-dag-v7.rst (the v7 formal substrate was brought into the repo for this; supersedes the prior "edit upstream ~/Downloads/merkle-agi-dag_v7.txt" plan — the spec is now version-controlled and amendments land there). #000018 §9.2 ("which PRG?") resolves: HMAC-SHA-512 with a 32-byte committed seed. (A full upstream v7 spec revision remains exogenous; this lands the §9.10 amendment into the tracked in-repo copy.) Opened: 2026-05-09 Scope: Pin a specific cryptographic PRG construction for the M1 mitigation (PRG-based anchor map) proposed in #000018 §5.2 + §9.2. The §4 reduction in docs/soft-hash-channel-analysis.md treats φ_PRG as a random oracle; in deployment we need a deterministic PRG with explicit cryptographic strength assumption. Audience: maintainers of the v7 plastic-training spec; #000018 follow-up. Hard constraint: the PRG security parameter MUST match SHA-256 in the substrate (≥128-bit security level). Adding a weaker PRG would create a cryptographic-strength asymmetry — adversary breaks the PRG, channel re-opens.

---

1. Problem statement

#000018 §5.2 proposes φ_PRG as the M1 mitigation:

φ_PRG(digest) = PRG(seed = published, output_len = dim_h * 4 bytes)
                then bytes → floats in [-1, 1]

The §4 reduction proves channel-bound under the random-oracle model for φ_PRG. In deployment, "random oracle" becomes "PRG with public seed and computational indistinguishability from random."

The reduction holds modulo PRG security. Pick a weak PRG and the reduction's looseness is the PRG's distinguishing advantage. This ticket pins which PRG.

2. Design choices

2.1 Candidate PRG constructions

A. HMAC-SHA-512 expansion.

def phi_prg(digest_32bytes, dim_h):
    out = b""
    counter = 0
    while len(out) < dim_h * 4:
        out += hmac.new(seed, digest_32bytes + counter.to_bytes(4, 'big'),
                        hashlib.sha512).digest()
        counter += 1
    return bytes_to_floats(out[:dim_h * 4])

• Strength: HMAC-SHA-512 is a PRF under the standard SHA-512 Merkle-Damgård + HMAC assumption. Distinguishing advantage is bounded by the SHA-512 collision-resistance bound (~2^256).
• Dependency: keeps the substrate's hash family at SHA-{256,512} — already in the dependency surface.
• Speed: ~1 GB/s on a modern x86 CPU; negligible per-checkpoint.

B. AES-256-CTR with public IV.

def phi_prg(digest_32bytes, dim_h):
    aes = AES.new(seed[:32], AES.MODE_CTR, nonce=digest_32bytes[:8])
    out = aes.encrypt(b"\\0" * (dim_h * 4))
    return bytes_to_floats(out)

• Strength: AES-256 PRP under the standard AES key-recovery hardness assumption. Distinguishing advantage bounded by AES key search (~2^256) under known plaintext.
• Dependency: adds cryptography package's AES (already in arborist core deps as cryptography>=42).
• Speed: ~1.5 GB/s on x86 with AES-NI; faster than HMAC-SHA-512 but the difference doesn't matter at v7 checkpoint cadence.

C. ChaCha20 with public nonce.

• Strength: ChaCha20 PRF under the cryptographic standard.
• Dependency: cryptography provides; same as AES.
• Considered: redundant with AES — pick one.

D. SHA-256 as a hash chain.

def phi_prg(digest_32bytes, dim_h):
    out = b""
    h = digest_32bytes
    while len(out) < dim_h * 4:
        h = hashlib.sha256(seed + h).digest()
        out += h
    return bytes_to_floats(out[:dim_h * 4])

• Strength: Insufficient. SHA-256 hash chains are NOT guaranteed PRFs — known length-extension and predictable fixed-points make this unsuitable. Reject.

2.2 Recommendation

A — HMAC-SHA-512 expansion.

Reasons:

1. Tightest dependency: stays inside the SHA family already committed to by SHA-256 (the substrate's hard-hash). Adding AES would multiply the assumption surface without cryptographic gain.
2. Standard NIST-approved PRF construction (SP 800-108 KDF in counter mode using HMAC).
3. Speed parity with AES at v7 cadence (per-checkpoint cost negligible compared to gradient computation).
4. Provable security reduction: HMAC-SHA-512 PRF security reduces to SHA-512 collision-resistance, which is structurally stronger than SHA-256 (output space is 2× larger).

2.3 Float conversion

PRG outputs uniform-random bytes. The anchor space is [-1, 1]^dim_h. Convert via:

def bytes_to_floats(b):
    """uint32 → float in [-1, 1) via 2 * (u32/2^32) - 1."""
    n = len(b) // 4
    floats = []
    for i in range(n):
        u32 = int.from_bytes(b[4*i:4*(i+1)], 'big')
        floats.append(2.0 * (u32 / 2**32) - 1.0)
    return floats

Distribution: uniform on [-1, 1) modulo a 2^-32 quantization discontinuity; ignored at our precision.

2.4 Seed handling

The PRG seed is published (committed in v7 boot manifest). It's NOT a secret — secrecy isn't the security property. The property is computational indistinguishability of the OUTPUT from random, which holds even when the seed is public, because the adversary can't invert the PRF without solving the underlying SHA-512 problem.

Rotating the seed per-checkpoint becomes the M2 mitigation (#000018 §5.3) — orthogonal concern; M1 + M2 stack cleanly.

3. Implementation sketch

3.1 New module

arborist/v7/anchor_prg.py (new directory if v7 work hasn't landed yet; or wherever v7's plastic-training surface is implemented):

"""φ_PRG — HMAC-SHA-512 anchor map for v7 § 9.10."""
import hashlib
import hmac

PUBLISHED_SEED = b"..."  # 32-byte committed seed

def phi_prg(hard_hash_32: bytes, dim_h: int) -> list[float]:
    if len(hard_hash_32) != 32:
        raise ValueError("hard hash must be 32 bytes (SHA-256)")
    block_bytes = dim_h * 4
    out = bytearray()
    counter = 0
    while len(out) < block_bytes:
        msg = hard_hash_32 + counter.to_bytes(4, 'big')
        out += hmac.new(PUBLISHED_SEED, msg, hashlib.sha512).digest()
        counter += 1
    return _bytes_to_floats(bytes(out[:block_bytes]))

3.2 Tests

tests/test_anchor_prg.py:

• Determinism: same input + same seed → same output bytes.
• Distinguishing-from-random sanity: chi² test on a sample of outputs (loose; just catches gross bugs).
• Boundary: dim_h=1, dim_h=10⁶ both produce sensible outputs.
• Seed-change sensitivity: flipping one bit of the seed produces outputs uncorrelated with the original (Hamming distance ≈ output_size / 2).

3.3 Bench fixture

bench/fixtures/phi-prg/known-answer-tests.jsonl — fixed (seed, hard_hash, dim_h) → expected_output_sha256 triples (10 entries) for regression coverage. Regenerated by scripts/generate_phi_prg_kat.py (run it after any algorithm change, then bump PHI_PRG_VERSION so the fixture's version field changes too). tests/test_anchor_prg.py::test_phi_prg_known_answer_tests pins these values and asserts the version matches the module.

3.4 Spec amendment

v7 § 9.10 amendment text — dav1d-reviewed final (2026-05-11), little-endian variant. Decision set (RESPONSE 2026-05-11) + endianness resolution (2026-05-11): HMAC-SHA-512 — LOCK; manifest field anchor_prg_seed (purpose-scoped, not implementation-scoped — phi_prg_seed is an acceptable code-local alias only); seed size 32 bytes — LOCK; counter uint32 little-endian from 0, and the uint32-word interpretation is also little-endian — matching v7's canonical TLV integer encoding (verified against merkle-agi-dag_v7.txt §A1: TLV length prefixes to_bytes(4, "little"), enc_int is to_bytes(8, "little"), quantized tensors are <i8). Per dav1d's review rule ("if v7 TLV canonical integer encoding is little-endian, flip §3.4 to little-endian before KAT freeze"). Float map 2·(u32_le/2^32)−1; prose says "uniform over a 2^32-point grid in [-1, 1) with negligible finite-grid mean −2^−32", not "unbiased"; SHALL replace embed_hard_to_vec when M1 is enabled (whether M1 itself is mandatory is a separate mitigation-selection-policy question). Plus an explicit dim_h ≤ 16·2^32 exhaustion guard and a seed-independence requirement. PHI_PRG_VERSION carries the -le suffix (phi-prg-v1-hmac-sha512-le); the 10 KAT vectors were regenerated 2026-05-11 under the little-endian counter.

§ 9.10  Anchor PRG map φ_PRG

When M1 is enabled, the v7 anchor map φ_PRG(C(M), dim_h) is defined
as follows. All integers below are encoded little-endian, matching
v7's canonical TLV integer convention (§A1).

Let anchor_prg_seed be the v7 boot manifest's dedicated 32-byte PRG
seed. The seed MUST be generated independently of the model
checkpoint and training data, and MUST NOT be adversary-selected.
The seed MAY be public once committed, but it MUST be committed
before the corresponding anchor map is evaluated. anchor_prg_seed
is single-purpose — it MUST NOT be reused for any other PRG domain.

Let C(M) be the fixed-length committed model digest. Let counter
range over unsigned 4-byte little-endian integers starting at 0.

For counter = 0, 1, 2, … compute:

    block_counter = HMAC-SHA-512(anchor_prg_seed, C(M) ‖ counter_le32)

Concatenate successive blocks until at least dim_h · 4 bytes are
available; truncate to exactly dim_h · 4 bytes. A 4-byte counter
admits 2^32 HMAC-SHA-512 blocks of 64 bytes each, so dim_h MUST
satisfy dim_h ≤ 16 · 2^32; a request beyond that is an error.

Partition the byte stream into dim_h successive 4-byte words.
Interpret each word as an unsigned little-endian integer u32. Map
each u32 to a float by:

    x = 2 · (u32 / 2^32) − 1

The resulting dim_h-vector is φ_PRG(C(M), dim_h).

This maps uniformly onto a 2^32-point grid in [-1, 1). The value
-1.0 is reachable; +1.0 is not. The finite-grid mean is −2^−32,
which is negligible for the anchor-map use case. If exact
zero-mean sampling is ever required, the formula MUST be
version-bumped (e.g. to the midpoint map x = 2·((u32+0.5)/2^32)−1)
rather than silently changed, and new KAT vectors emitted.

When M1 is enabled, φ_PRG SHALL replace the v7 reference linear
projection embed_hard_to_vec for the hard-anchor-to-vector map.
An operator MUST NOT claim M1 while continuing to use
embed_hard_to_vec.

§ 9.10.1  M1 enablement policy (non-normative pointer)

Whether a given deployment MUST enable M1 is decided by the
mitigation-selection policy, not by this section. M1 enablement
MAY be skipped only under an explicit policy rule — for example
when a current #000034 φ/Hessian alignment probe returns
NO_ALIGNMENT and the deployment policy accepts the residual risk.
Once M1 is enabled, § 9.10 above applies in full.

Endianness — RESOLVED 2026-05-11. v7's canonical integer byte-order was confirmed little-endian (merkle-agi-dag_v7.txt §A1 — every to_bytes/astype in the TLV encoding is little- endian; no big-endian anywhere). The §9.10 text above uses little-endian throughout to match. The reference implementation (arborist/substrate/anchor_prg.py), the 10 regenerated KAT vectors, and test_anchor_prg.py are all pinned to little-endian; PHI_PRG_VERSION = "phi-prg-v1-hmac-sha512-le". (The earlier big-endian draft + its KATs are superseded — a re-flip would need another PHI_PRG_VERSION bump.)

4. Out of scope

• v7 spec full revision; this ticket lands the §9.10 amendment text, not the surrounding chapter.
• M2 (per-checkpoint nonce) implementation; orthogonal.
• Empirical Hessian-alignment measurement under φ_PRG (covered by #000034).
• Performance tuning (PRG eval cost is ~µs per checkpoint; irrelevant).

5. Acceptance criteria

1. arborist/v7/anchor_prg.py (or equivalent location) lands.
2. Tests pin determinism + seed-sensitivity + boundary cases.
3. Known-answer-test fixture committed under bench/fixtures.
4. v7 § 9.10 amendment text drafted (this ticket §3.4 is the draft; final landing waits on v7 spec maintainer review).
5. #000018 § 9.2 closes — the PRG choice question becomes "HMAC-SHA-512 with 32-byte committed seed."

6. Risks / Considerations

• Future SHA-512 weakness. If SHA-512 is broken in a way that affects HMAC-SHA-512 PRF security, the M1 mitigation's cryptographic foundation cracks. Mitigation: the v7 manifest's phi_prg_version field lets future deployments swap to a successor PRF without breaking historical replay.
• Seed compromise. If the published seed is somehow swapped by an adversary (e.g., supply-chain attack on the manifest), the PRG is still PRF-secure but the seed is now adversarial- controlled. The hard-hash chain (SHA-256 of the manifest) catches the swap as a chain-break, so this reduces to a manifest-integrity attack — out of scope for #000018, in scope for chain-check.

7. Status

Closed · 2026-05-11 — Phase 1 + Phase 2 landed. Reference implementation (Phase 1) shipped ahead of v7 plastic-training deployment so the cryptographic primitive is unit-tested + KAT-pinned the moment v7 needs it; the § 9.10 amendment (Phase 2) is folded into the in-repo v7 substrate spec (docs/_source/merkle-agi-dag-v7.rst).

Phase 1 — reference implementation (landed 2026-05-10)

• arborist/substrate/anchor_prg.py — phi_prg(hard_hash_32, dim_h, *, seed) per §3.1; HMAC-SHA-512 counter-mode KDF; pure stdlib (hashlib + hmac); no third-party dependency. Module also exports PHI_PRG_VERSION = "phi-prg-v1-hmac-sha512-le" so future algorithm rotations can be detected at the call site without string-comparing module paths.

  Path note (2026-05-10): §3.1's original sketch placed the module at arborist/v7/anchor_prg.py. The version-prefixed namespace pattern was retired the same day in favour of arborist/substrate/ — the v in v7 referred to the substrate-paper version, which collided with the v9.8 SQLite schema version and confused readers about whether the dir tracked paper version or schema version. arborist/substrate/ is now the topic dir for paper-spec'd Merkle-AGI primitives, decoupled from any version number. Acceptance-criteria §5 item 1 reads through to "or equivalent location" so the move doesn't invalidate the original criterion.

• tests/test_anchor_prg.py — 31 tests covering determinism, range invariants, chi² loose-uniformity sanity, dim_h boundary (1, 16384), seed-bit-flip avalanche, hash-bit-flip avalanche, hand-computed HMAC-SHA-512 first-block formula, prefix-extension closure, parametrized invalid-input cones, input validation (short/long/non-bytes hashes; non-positive / non-int dim_h), module export shape, and KAT regression. (Was 20; +7 from de997f7 2026-05-10 pattern backfill per docs/calculator-test-patterns.md.)

• bench/fixtures/phi-prg/known-answer-tests.jsonl — 10 pinned KAT vectors covering the placeholder seed (smoke), block-boundary cases (dim_h=16 = exactly one HMAC-SHA-512 block; dim_h=17 = two blocks with truncation), seed/hash one-bit-flip variants, and a 4096-element stress sample to catch counter-rollover bugs. Bytes-output SHA-256 is the durable contract; float layout changes do NOT invalidate the fixture.

Phase 2 — v7 §9.10 amendment landing (landed 2026-05-11)

§3.4 of this ticket held the draft amendment text. Phase 2 landed it by bringing the v7 formal substrate spec into the repo as docs/_source/merkle-agi-dag-v7.rst (previously referenced only as the un-version-controlled ~/Downloads/merkle-agi-dag_v7.txt) and folding § 9.10 + § 9.10.1 in at their numbered positions (after § 9.9, "Security review (plasticity)"), flagged with a .. note:: citing this ticket and the reference implementation. Added to the docs/_source/index.rst "Substrate" toctree.

The original Phase-2 gate ("v7 plastic-training spec gains a deployment target" + "spec maintainer reviews the §3.4 text") was reframed: review is done (dav1d-final 2026-05-11); a full upstream v7 spec revision remains exogenous, but the amendment no longer needs to wait on it — it lives in the version-controlled in-repo copy, where future amendments also land.

Closure criterion

CLOSED 2026-05-11: § 9.10 landed in docs/_source/merkle-agi-dag-v7.rst; #000018 §9.2 ("which PRG?") is resolved as "HMAC-SHA-512 with a 32-byte committed seed." (Original wording preserved below for the log:) Closes when Phase 2 lands the §9.10 amendment into the v7 spec and #000018 §9.2 (which pins the open question "which PRG?") is resolved as "HMAC-SHA-512 with 32-byte committed seed."