# Form D Option 1 — axis-flip refactor RESULTS

Companion to `form-D-build-progress.md` (which halted the AG-tableau
Form D before kernel write & recommended an axis-flip refactor of
`sim_gpu.cu`). This document records what landed.

## What shipped

| artifact | path | role |
|----------|------|------|
| `sim_gpu_axis.cu`     | `ecdsa/cuda/sim_gpu_axis.cu`     | per-candidate axis kernel, single launch covers K×M lanes |
| `main_axis_demo.c`    | `ecdsa/cuda/main_axis_demo.c`    | host driver, CPU/GPU byte-identity per lane, portal output |
| `Makefile` (+demo_axis target) | `ecdsa/cuda/Makefile`   | builds via `make demo_axis` |

`sim_gpu.cu` (per-shot striped) is untouched. `ops_loader.c` & `sim_cpu.c`
are untouched. Axis kernel reuses both as canonical references.

## Hardware & workload

- Host: `3090-ai.foxhop.net`, RTX 3090 (24 GB), CUDA 12.0, sm_86.
- Workload: `/tmp/ops.bin` (716 MB, 12,788,119 ops, 1355 qubits, 1,300,679
  bits, 2,100,266 RNG ops — point-add-toy Phase B emission).
- RNG: deterministic LFSR per `(candidate, lane)` seed — shared by CPU &
  GPU so byte-identity holds.

## Byte-identity

Every reported configuration passed `cpu phase==gpu phase &&
cpu cliff==gpu cliff && cpu toff==gpu toff` for every lane.

| config | total lanes | result |
|--------|-------------|--------|
| K=1,  M=4 | 4   | PASS |
| K=4,  M=4 | 16  | PASS |
| K=16, M=4 | 64  | PASS |
| K=32, M=4 | 128 | PASS |

K=64 M=4 OOMed at device alloc (bits arena alone = 2.7 GB × the rest
exceeded 24 GB on 3090). No incorrect result — kernel never reached
launch.

## Throughput

Apples-to-apples: same total quantum work in ops (`Σ_c n_ops_c × M_c`
= `12.788M × total_lanes`).

| kernel | K | M | total lanes | kernel ms | Mops/s | GPU speedup vs CPU |
|--------|---|---|-------------|-----------|--------|-------------------|
| per-shot striped (existing `sim_gpu.cu`) | 1 | 4   | 4   | 5,590 |   9.15 | 0.03× |
| **per-cand axis (new `sim_gpu_axis.cu`)** | 1 | 4   | 4   | 5,780 |   8.85 | 0.03× |
| **per-cand axis** | 4   | 4 | 16  | 6,938 |  29.49 | 0.08× |
| **per-cand axis** | 16  | 4 | 64  | 7,402 | 110.58 | 0.32× |
| **per-cand axis** | 32  | 4 | 128 | 7,546 | 216.91 | 0.61× |
| per-shot striped  | 1   | 128 | 128 | 6,604 | 247.86 | 1.06× |

CPU reference scales near-linearly (~350 Mops/s steady) — single-core
LLVM-vectorized bit ops on host's Zen.

## Reading the numbers

1. **Both kernels reach the same throughput ceiling at full saturation.**
   - Per-shot at 128 lanes: 247 Mops/s.
   - Axis at 128 lanes (K=32, M=4): 217 Mops/s.
   - Within 12% of each other. The flip did **not** unlock new headroom.
2. **Small-M underutilization: axis wins by candidate-stacking.**
   - At M=4 with per-shot kernel (one candidate), only 4 lanes / 4
     threads run — kernel is launch-overhead-dominated, 9 Mops/s.
   - Axis K=32 at M=4 fills the same 128-lane budget by stacking 32
     candidates → **23.7× more throughput** than per-shot at the same
     per-candidate M (5,590 ms → 7,546 ms while doing 32× more work).
   - This is the only regime where the axis flip pays.
3. **Per-candidate ms drops cleanly with K.**
   - K=1:  5,780 ms/cand.
   - K=4:  1,735 ms/cand.
   - K=16:   463 ms/cand.
   - K=32:   236 ms/cand. → 24.5× per-candidate amortization vs K=1.
   - Useful when a lumbda search loop sweeps many candidates with
     few shots each (typical early-screen pattern: M=1–4, K=many).

## Did the bandwidth-bound diagnosis hold?

**Partially — and the failure mode matters.**

The form-D-build-progress.md diagnosis was: "1.07× ceiling traces to
memory bandwidth on per-shot striped state." If that were the dominant
constraint, swapping the striping axis should redistribute the
bandwidth pressure & raise the ceiling.

What the data shows:

- At fully-saturated launch (128 lanes), both layouts land at ~220–250
  Mops/s. The ceiling did **not** move materially. → bandwidth was not
  uniquely bound to the striping axis; redistributing across candidates
  buys nothing once the device is full.
- At underutilized launch (M=4, 4 lanes), the axis flip's win is
  100% occupancy-driven (more lanes per launch), not bandwidth-driven.
  Same speedup would have come from any axis that filled SMs.

So the diagnosis was wrong about the **mechanism** but the **refactor
still pays** in the regime the lumbda search actually targets: many
candidates × few shots each. The win is occupancy-amortization, not
bandwidth-redistribution.

## Where this leaves us

- **Use axis kernel** when sweeping K ≥ 16 candidates with M ≤ 16
  shots each — lumbda's early-screen pattern. Per-candidate ms drops
  ~20–25× vs per-shot kernel run sequentially per candidate.
- **Use per-shot kernel** when one candidate needs many shots (M ≥ 64)
  — its at-saturation throughput is 14% higher & memory layout is
  proven across thousands of test runs.
- **Don't pursue further axis variants** chasing the 1.07× → Nx
  speedup. The 1.07× ceiling is a saturation ceiling on this device
  for this op mix, not an axis-choice artifact. Future wins live in:
  - ops.bin packing (Op = 28 bytes → 16 bytes halves op-stream
    bandwidth; per `sim.h` most fields are NO_SLOT for most kinds).
  - Persistent kernel + ops streaming from device-resident pool.
  - True multi-GPU dispatch via lumbda's TCP-portal primitive (cammy
    + guile + 4090-ai/ai are sitting idle for this workload).

## Reproducing

On the GPU host (`3090-ai.foxhop.net` in this run):

```bash
cd /tmp/ecdsa-cuda
make demo_axis NVCC=/usr/bin/nvcc ARCH=sm_86

# K=1, M=4 — byte-identity check, baseline
./demo_axis /tmp/ops.bin --shots 4

# K=32, M=4 — fully-saturated axis run (needs --force-mem ~3.5 GB)
./demo_axis $(printf '/tmp/ops.bin %.0s' {1..32}) \
  --shots 4 --force-mem --portal /tmp/axis-K32-M4.portal

# baseline per-shot comparison at same total lanes
./bend-cuda /tmp/ops.bin 128
```

Portal format `(cuda-axis-result ...)` — lumbda search loops ingest
this directly. Not yet wired into `gpu-worker.lsp` per task scope.
