cuda-fanout plans: Day-2/3 progress + Form D pivot RESULTS docs

Captures empirical record from three agents that finished today: form-A-day2-progress.md (STOPPED, no merge) v2 Montgomery batch inversion regressed v1 by 0.63x-0.94x across n in {10k, 100k, 1M}. Byte-identity PASS at every N; math correct. Root cause: at this N, scalar_mul (256 Jacobian doubles x 5-6 ModMult each) dominates, NOT _ModInv. v2's Phase B/D used 1 thread/block leaving ~97% of SMs idle. v1 baseline re-measured at 7.88 Mkeys/s at n=1M (catalog upward correction from initial 6.51). Day-3 path: warp-level prefix scan OR windowed-G ladder. form-D-axis-flip-RESULTS.md (Form D opt 1 shipped foxhop 1f7ac9d) Per-candidate axis kernel measured 217 Mops/s at K=32 M=4 on 3090; 23.7x over per-shot N=4 at same M. Both axes saturate at the same ~220-250 Mops/s, refuting the bandwidth-bound diagnosis. Axis flip's win is occupancy-amortization at small M, not bandwidth redistribution. form-D-build-progress.md (Form D AG kernel parked) Aaronson-Gottesman tableau dead-end: our point-add circuit contains no H or S, so state never leaves the computational basis and AG buys nothing. Three pivot options proposed; fox picked options 1 + 3 in parallel. Day-1 binary at lumbda 7661788 stays canonical for cuda-secp256k1-batched-mul; v2 working-tree code lives uncommitted on the build host as Day-3 scratch.
2026-06-05 15:32:32 -04:00 · 2026-06-05 15:32:32 -04:00 · b2209ac337
commit b2209ac337
parent 95b3b1af21
3 changed files with 348 additions and 0 deletions
--- a/examples/cuda-fanout/plans/form-A-day2-progress.md
+++ b/examples/cuda-fanout/plans/form-A-day2-progress.md
@ -0,0 +1,80 @@
+# Form A Day-2 progress — Montgomery batch inversion landed correctly but regressed throughput
+
+Status: **STOPPED, no commit.** v2 kernel validates byte-identically against coincurve through n=1M but runs slower than Day-1 v1 across all tested batch sizes on 3090.
+
+## What shipped (uncommitted, working tree)
+
+`secp256k1-batch-mul.cu` now carries two kernel variants:
+
+- `secp_mul_batch_kernel_v1` (renamed from `secp_mul_batch_kernel`) — Day-1 path, per-thread `_ModInv`.
+- `v2_phase_a` / `v2_phase_b` / `v2_phase_c` / `v2_phase_d` — 4-launch Montgomery batch inversion pipeline.
+
+CLI gates v2 via `--batch-inv` (default ON) or `--no-batch-inv` (selects v1). Stderr labels each run `[v1-per-thread-inv]` or `[v2-batch-inv]`.
+
+## Algorithm landed correctly
+
+Block decomposition: K = ceil(n / B) where B = 256.
+
+1. **Phase A** (full-parallel, N threads): per-thread scalar_mul, writes (Jx, Jy, Jz) to global scratch `jac[n*12]`; writes effective z[i] = Jz (or 1 for infinity) to scratch `z[n*4]`.
+2. **Phase B** (1 thread/block, K blocks): per-block prefix product c[i] = z[base..i] mod p; writes block_prod[b] = c[end-1].
+3. **Phase C** (single thread total): walks left-to-right computing G_left[b] = product of block_prod[0..b-1]; one `_ModInv` on the global product G; walks right-to-left to produce u_entry[b] = (block_prod[0..b])^-1.
+4. **Phase D** (1 thread/block, K blocks): walks each block backward i = end-1..base, computing z_inv[i] = u * c_global[i-1] where c_global[i-1] = G_left[b] * c[i-1] for i > base, else G_left[b] (b > 0) or 1 (b == 0); updates u *= z[i]; finishes affine projection (Zi^2, Zi^3, Ax, Ay); emits BE bytes.
+
+## Validation — passes byte-identical
+
+Coincurve oracle on 3090-ai (after `python3 -m venv ~/venv-secp && ~/venv-secp/bin/pip install coincurve`):
+
+```
+n=8 known-small             PASS
+n=32 random                 PASS
+n=1000 random               PASS
+n=10000 random              PASS
+n=100000 random             PASS
+n=1000000 random            PASS
+```
+
+Math is correct. The "STOPPED" condition is throughput, not correctness.
+
+## Throughput — v2 regresses across all N
+
+Three runs each on 3090-ai, kernel ms reported by the worker:
+
+| n       | v1 Mkeys/s | v1 kernel ms | v2 Mkeys/s | v2 kernel ms | v2 / v1 |
+|---------|------------|--------------|------------|--------------|---------|
+| 10,000  | 4.37       | 2.29         | 2.76       | 3.62         | 0.63x   |
+| 100,000 | 7.43       | 13.47        | 5.97       | 16.74        | 0.80x   |
+| 1,000,000 | 7.88     | 126.85       | 7.39       | 139.42       | 0.94x   |
+
+(Note: today's measured Day-1 baseline at n=1M is 7.88 Mkeys/s — higher than the 6.51 Mkeys/s cited in the task prompt. v1 was re-benchmarked on the same box in the same session.)
+
+## Root cause
+
+`_ModInv` on a 3090 is not the bottleneck at this n. v1 already saturates at ~7.9 Mkeys/s at n=1M because **scalar_mul** dominates the kernel: 256 Jacobian doubles + ~128 adds per scalar, each calling 5-6 `_ModMult` / `_ModSqr`. That's ~2000 field mults per scalar in scalar_mul vs ~20-30 field mults amortized for the inversion. Replacing _ModInv with batch inversion saves a small fraction of total work.
+
+The v2 implementation as designed makes the situation worse because:
+
+- Phase B and Phase D use **1 thread per block** (block size 256). At n=1M that's 3906 active threads vs v1's 1M threads. 3090 has 82 SMs × 1536 max-resident-threads = ~125k concurrent threads; v2 leaves >97% of SM throughput idle in the prefix and suffix passes.
+- Per-block serial walk in Phase D performs ~256 ModMult4 calls sequentially per thread; v1 spreads ModInv (~20-30 ModMult-equivalent ops) across all N threads in parallel.
+- v2 adds ~120 MB of device scratch (`jac`, `z`, `c`, `block_prod`, `G_left`, `u_entry`) at n=1M plus extra kernel launches.
+
+## What v2 needs to become a win
+
+Two non-obvious follow-ups, neither in scope today:
+
+1. **Warp-level prefix scan inside Phase B + suffix walk in Phase D.** Use Kogge-Stone or Sklansky scan with 32 threads/warp on the prefix product. At n=1M with B=256, each warp scans 8 elements — turns the serial 256-mult chain into 8 mults of warp-coordinated work. Block_size could grow to 1024 (warp == 32, scan 32 elements/warp).
+2. **Move scalar_mul to a windowed ladder (precomputed G, 2G, ..., 256G).** Cuts scalar_mul cost ~4x and shifts the kernel's bottleneck onto inversion, where batching actually wins. Upstream `GPUGroup.h` carries the precomputed tables — vendoring that header is the Day-3+ move flagged in the original plan §2.
+
+Neither follow-up was in the Day-2 mandate. Logical halt point.
+
+## Files touched (uncommitted, in working tree)
+
+- `examples/cuda-fanout/secp256k1-batch-mul.cu` — split v1/v2 kernels, added `--batch-inv` flag, v2 default-ON in flag wiring (but never reached commit because throughput failed).
+
+## Rebuild + revert path
+
+```
+ssh 3090-ai.foxhop.net 'cd /home/fox/git/lumbda/examples/cuda-fanout && rm -f secp256k1-batch-mul && make secp256k1-batch-mul'
+git -C ~/git/lumbda checkout -- examples/cuda-fanout/secp256k1-batch-mul.cu   # if reverting
+```
+
+Day-1 binary @ commit 7661788 stays canonical until Day-3 lands either the warp-scan inner kernel or the windowed-ladder scalar_mul.
--- a/examples/cuda-fanout/plans/form-D-axis-flip-RESULTS.md
+++ b/examples/cuda-fanout/plans/form-D-axis-flip-RESULTS.md
@ -0,0 +1,141 @@
+# Form D Option 1 — axis-flip refactor RESULTS
+
+Companion to `form-D-build-progress.md` (which halted the AG-tableau
+Form D before kernel write & recommended an axis-flip refactor of
+`sim_gpu.cu`). This document records what landed.
+
+## What shipped
+
+| artifact | path | role |
+|----------|------|------|
+| `sim_gpu_axis.cu`     | `ecdsa/cuda/sim_gpu_axis.cu`     | per-candidate axis kernel, single launch covers K×M lanes |
+| `main_axis_demo.c`    | `ecdsa/cuda/main_axis_demo.c`    | host driver, CPU/GPU byte-identity per lane, portal output |
+| `Makefile` (+demo_axis target) | `ecdsa/cuda/Makefile`   | builds via `make demo_axis` |
+
+`sim_gpu.cu` (per-shot striped) is untouched. `ops_loader.c` & `sim_cpu.c`
+are untouched. Axis kernel reuses both as canonical references.
+
+## Hardware & workload
+
+- Host: `3090-ai.foxhop.net`, RTX 3090 (24 GB), CUDA 12.0, sm_86.
+- Workload: `/tmp/ops.bin` (716 MB, 12,788,119 ops, 1355 qubits, 1,300,679
+  bits, 2,100,266 RNG ops — point-add-toy Phase B emission).
+- RNG: deterministic LFSR per `(candidate, lane)` seed — shared by CPU &
+  GPU so byte-identity holds.
+
+## Byte-identity
+
+Every reported configuration passed `cpu phase==gpu phase &&
+cpu cliff==gpu cliff && cpu toff==gpu toff` for every lane.
+
+| config | total lanes | result |
+|--------|-------------|--------|
+| K=1,  M=4 | 4   | PASS |
+| K=4,  M=4 | 16  | PASS |
+| K=16, M=4 | 64  | PASS |
+| K=32, M=4 | 128 | PASS |
+
+K=64 M=4 OOMed at device alloc (bits arena alone = 2.7 GB × the rest
+exceeded 24 GB on 3090). No incorrect result — kernel never reached
+launch.
+
+## Throughput
+
+Apples-to-apples: same total quantum work in ops (`Σ_c n_ops_c × M_c`
+= `12.788M × total_lanes`).
+
+| kernel | K | M | total lanes | kernel ms | Mops/s | GPU speedup vs CPU |
+|--------|---|---|-------------|-----------|--------|-------------------|
+| per-shot striped (existing `sim_gpu.cu`) | 1 | 4   | 4   | 5,590 |   9.15 | 0.03× |
+| **per-cand axis (new `sim_gpu_axis.cu`)** | 1 | 4   | 4   | 5,780 |   8.85 | 0.03× |
+| **per-cand axis** | 4   | 4 | 16  | 6,938 |  29.49 | 0.08× |
+| **per-cand axis** | 16  | 4 | 64  | 7,402 | 110.58 | 0.32× |
+| **per-cand axis** | 32  | 4 | 128 | 7,546 | 216.91 | 0.61× |
+| per-shot striped  | 1   | 128 | 128 | 6,604 | 247.86 | 1.06× |
+
+CPU reference scales near-linearly (~350 Mops/s steady) — single-core
+LLVM-vectorized bit ops on host's Zen.
+
+## Reading the numbers
+
+1. **Both kernels reach the same throughput ceiling at full saturation.**
+   - Per-shot at 128 lanes: 247 Mops/s.
+   - Axis at 128 lanes (K=32, M=4): 217 Mops/s.
+   - Within 12% of each other. The flip did **not** unlock new headroom.
+2. **Small-M underutilization: axis wins by candidate-stacking.**
+   - At M=4 with per-shot kernel (one candidate), only 4 lanes / 4
+     threads run — kernel is launch-overhead-dominated, 9 Mops/s.
+   - Axis K=32 at M=4 fills the same 128-lane budget by stacking 32
+     candidates → **23.7× more throughput** than per-shot at the same
+     per-candidate M (5,590 ms → 7,546 ms while doing 32× more work).
+   - This is the only regime where the axis flip pays.
+3. **Per-candidate ms drops cleanly with K.**
+   - K=1:  5,780 ms/cand.
+   - K=4:  1,735 ms/cand.
+   - K=16:   463 ms/cand.
+   - K=32:   236 ms/cand. → 24.5× per-candidate amortization vs K=1.
+   - Useful when a lumbda search loop sweeps many candidates with
+     few shots each (typical early-screen pattern: M=1–4, K=many).
+
+## Did the bandwidth-bound diagnosis hold?
+
+**Partially — and the failure mode matters.**
+
+The form-D-build-progress.md diagnosis was: "1.07× ceiling traces to
+memory bandwidth on per-shot striped state." If that were the dominant
+constraint, swapping the striping axis should redistribute the
+bandwidth pressure & raise the ceiling.
+
+What the data shows:
+
+- At fully-saturated launch (128 lanes), both layouts land at ~220–250
+  Mops/s. The ceiling did **not** move materially. → bandwidth was not
+  uniquely bound to the striping axis; redistributing across candidates
+  buys nothing once the device is full.
+- At underutilized launch (M=4, 4 lanes), the axis flip's win is
+  100% occupancy-driven (more lanes per launch), not bandwidth-driven.
+  Same speedup would have come from any axis that filled SMs.
+
+So the diagnosis was wrong about the **mechanism** but the **refactor
+still pays** in the regime the lumbda search actually targets: many
+candidates × few shots each. The win is occupancy-amortization, not
+bandwidth-redistribution.
+
+## Where this leaves us
+
+- **Use axis kernel** when sweeping K ≥ 16 candidates with M ≤ 16
+  shots each — lumbda's early-screen pattern. Per-candidate ms drops
+  ~20–25× vs per-shot kernel run sequentially per candidate.
+- **Use per-shot kernel** when one candidate needs many shots (M ≥ 64)
+  — its at-saturation throughput is 14% higher & memory layout is
+  proven across thousands of test runs.
+- **Don't pursue further axis variants** chasing the 1.07× → Nx
+  speedup. The 1.07× ceiling is a saturation ceiling on this device
+  for this op mix, not an axis-choice artifact. Future wins live in:
+  - ops.bin packing (Op = 28 bytes → 16 bytes halves op-stream
+    bandwidth; per `sim.h` most fields are NO_SLOT for most kinds).
+  - Persistent kernel + ops streaming from device-resident pool.
+  - True multi-GPU dispatch via lumbda's TCP-portal primitive (cammy
+    + guile + 4090-ai/ai are sitting idle for this workload).
+
+## Reproducing
+
+On the GPU host (`3090-ai.foxhop.net` in this run):
+
+```bash
+cd /tmp/ecdsa-cuda
+make demo_axis NVCC=/usr/bin/nvcc ARCH=sm_86
+
+# K=1, M=4 — byte-identity check, baseline
+./demo_axis /tmp/ops.bin --shots 4
+
+# K=32, M=4 — fully-saturated axis run (needs --force-mem ~3.5 GB)
+./demo_axis $(printf '/tmp/ops.bin %.0s' {1..32}) \
+  --shots 4 --force-mem --portal /tmp/axis-K32-M4.portal
+
+# baseline per-shot comparison at same total lanes
+./demo_ops /tmp/ops.bin 128
+```
+
+Portal format `(cuda-axis-result ...)` — lumbda search loops ingest
+this directly. Not yet wired into `gpu-worker.lsp` per task scope.
--- a/examples/cuda-fanout/plans/form-D-build-progress.md
+++ b/examples/cuda-fanout/plans/form-D-build-progress.md
@ -0,0 +1,127 @@
+# Form D — build progress, halted 2026-06-05
+
+Plan: `form-D-clifford-stabilizer.md`.
+Status: **HALTED before kernel-write.** No code committed. Reason
+below.
+
+## What ran
+
+Step 0 of plan workflow only: Toffoli-fraction measurement on
+`/tmp/ops.bin` (716 MB, 12,788,119 ops, Phase B point-add-toy emission
+on this host).
+
+Per-kind histogram:
+
+| kind | name      | count        | %      |
+|------|-----------|--------------|--------|
+| 1    | REGISTER  | 4            | 0.00%  |
+| 2    | APPEND_REG| 1,024        | 0.01%  |
+| 6    | X         | 1,810,486    | 14.16% |
+| 8    | CX        | 5,336,861    | 41.73% |
+| 9    | CZ        | 1,319,391    | 10.32% |
+| 10   | SWAP      | 447,076      | 3.50%  |
+| 11   | R         | 800,099      | 6.26%  |
+| 12   | HMR       | 1,300,167    | 10.17% |
+| 13   | CCX       | 1,773,011    | 13.86% |
+
+- Toffoli fraction of all quantum ops: **13.87%** — well under our 40%
+  STOP threshold. Plan §3's "~half" suspicion did not hold.
+- Amdahl ceiling on overall speedup, given Toffoli stays on CPU:
+  `1 / (0.139 + 0.861 / S)` where `S` is the Clifford-only speedup. At
+  the plan's 186× target, overall hits ~6.9× — still meaningful, but
+  the headline number does not apply.
+
+That part of the answer is clean: **fraction is fine, proceed gated
+on the next finding.**
+
+## Why I stopped before writing the kernel
+
+Re-reading `ecdsa/cuda/sim.h` & `sim_gpu.cu` against the plan exposed
+a structural mismatch:
+
+1. Upstream Roetteler point-add (kinds 0–17 enumerated in `sim.h`)
+   **carries no Hadamard, no S, no Y, no measurement-onto-basis-rotation
+   op.** Search across `ecdsa/lumbda/`, `ecdsa/cuda/`, and
+   `ecdsafail-challenge/` for any H gate emission returned zero hits.
+   The op histogram above is the entire op alphabet that ever fires.
+2. With no H or S in the circuit, the quantum state never leaves the
+   computational basis. The "stabilizer simulation" reduces to
+   **classical reversible computation + a global phase ledger**.
+   That is exactly what `sim_gpu.cu`'s per-shot bit-packed simulator
+   already does — and proven exactly correct for this gateset, not an
+   approximation.
+3. Aaronson-Gottesman tableau over 2n × (2n+1) bits gains us nothing
+   we don't already have. Its parallelism win (row-XOR primitives,
+   warp-cooperative tableau updates) shows up only when **superposition
+   matters** — i.e., when H or S introduce mixed-basis states the
+   per-shot simulator cannot represent in one bit per shot. We have
+   none of that.
+4. The existing simulator's 1.07× ceiling on a 3090 (per plan
+   reference to `sim_gpu.cu`) traces to **memory bandwidth on
+   per-shot striped state**, not to algorithm choice. Switching the
+   algorithm to AG cannot help; switching the parallelism axis (per-
+   candidate over per-shot, as the plan correctly identifies) might —
+   but the axis flip belongs to the existing kernel, not a clean-room
+   AG re-implementation.
+
+## What this means for Form D
+
+The plan's headline target (186× from STABSim-class Clifford speedup)
+does not transfer because:
+
+- Our circuit is a strict subset of Clifford that the existing kernel
+  already handles exactly in one bit per shot.
+- AG's wins come from compressing exponentially-large stabilizer
+  state into 2n bits per shot; we already compress further (1 bit
+  per qubit per shot) because we never leave |0⟩/|1⟩.
+
+The **correct Form D**, if we still want a per-candidate axis flip,
+is a refactor of `ecdsa/cuda/sim_gpu.cu` that:
+
+- maps one CUDA block to one candidate circuit instead of one thread
+  to one batch of shots,
+- keeps the existing bit-packed phase/qubits/bits state,
+- moves cross-thread cooperation onto the candidate dimension (warp
+  shares work per gate),
+- continues to count CCX symbolically alongside the dynamic counter.
+
+That is **not** a clean-room Aaronson-Gottesman implementation. It is
+"sim_gpu.cu, axis-flipped." I should not write a binary that pretends
+to be the former when our real bottleneck wants the latter.
+
+## Recommendation to fox
+
+Three options, your call:
+
+1. **Re-scope Form D** to "axis-flip refactor of existing sim_gpu.cu
+   kernel, per-candidate parallelism." Drop AG framing entirely.
+   Drop BSTB/BSTR magic — reuse BSHK with a new `op_id`. Aim for the
+   kickmix speedup the plan actually wants. Effort ~3 days.
+
+2. **Build Form D as written** (clean-room AG tableau) for circuits
+   that **do** have H/S — e.g. error-correction codes, surface-code
+   patches, future research workloads. Keep the kernel in `cuda-fanout/`
+   as a general capability not coupled to our point-add metric. Effort
+   ~10 days, throughput gain on point-add is zero (no AG-eligible
+   structure to exploit).
+
+3. **Park Form D**, redirect engineering to options the kickmix
+   actually rewards: ops.bin packing (reduce per-op bytes from 56 →
+   16 to halve global-memory traffic), or per-candidate `sim_gpu.cu`
+   refactor (option 1 above), or revisit Form B/C numbers.
+
+Hard rule from session: I do not commit broken code & I do not write
+a kernel that misroutes around the actual bottleneck. Halting here.
+
+## Files touched
+
+None — no commit, no kernel write. This progress doc is the only
+artifact.
+
+## Numbers for CATALOG once a path forward is chosen
+
+- Toffoli fraction (point-add-toy): 13.87% of quantum ops
+- Op-mix dominated by CX (41.7%), X (14.2%), CCX (13.9%), CZ (10.3%),
+  HMR (10.2%), R (6.3%), SWAP (3.5%)
+- No H, no S, no Y in upstream point-add gateset — confirms
+  computational-basis simulator is exact, not approximate.