# Form D — Option 3 (ops.bin packing) results, 2026-06-05

Plan referent: `form-D-build-progress.md` § Recommendation, option 3
("ops.bin packing — reduce per-op bytes from 56 → 16 to halve global-
memory traffic").

Status: **landed, measured, & byte-identical**. Speedup matches the
build-progress diagnosis (per-shot state, not per-op stream, owns our
ceiling); does NOT match the task-author's 3.5× hypothesis.

## Findings up front

| n_batches | unpacked kernel ms | packed kernel ms | packed speedup |
|-----------|-------------------:|-----------------:|---------------:|
| 16        | 5842.2             | 5427.9           | **1.076×**     |
| 64        | 6225.9             | 5830.2           | **1.068×**     |
| 128       | 6604.2             | 6201.0           | **1.065×**     |

- Byte-identity: **PASS** at every n_batches across CPU vs unpacked-GPU
  vs packed-GPU. Phase, clifford & toffoli counters, full qubit & bit
  state vectors — all match.
- Diagnosis from `form-D-build-progress.md` § 4 ("per-shot striped
  state owns the 1.07× ceiling") **stands**. Packing the op stream
  does not flip the bottleneck.

## Why 1.07× & not 3.5×

The task-author's number was anchored to "56 B → 16 B = 3.5×" on the
**on-disk** layout. Two corrections found in code review before the
kernel write:

1. `sim.h` Op struct is **already 28 B** in VRAM (narrowed u64 → u32
   in `ops_loader.c::load_ops_bin`). The 56 B figure is QECCOPS1 wire
   format only. Maximum-feasible VRAM reduction is therefore
   28 → 24 B = 1.17×, not 3.5×.
2. Op fetches `Op op = ops[i]` warp-broadcast: every thread in a warp
   reads the same address each iteration, so L1/L2 already absorbs
   per-op traffic. The diverging reads are per-shot state, i.e.
   `qubits[op.q_target * n_batches + b]`, where every thread hits a
   distinct address.

Result: packing op records reduces op-stream traffic by 14% but barely
moves the kernel because op fetches were never the bottleneck. The
measured 1.07× tracks the upper bound nearly exactly — packing is
working, the bottleneck is elsewhere.

## Confirmed slot-index ranges on /tmp/ops.bin

Walked all 12,788,119 ops via `convert_ops_bin`:

- `n_ops`         : 12,788,119
- `num_qubits`    : 1,355  (max index 1,354 — fits u11)
- `num_bits`      : 1,300,679  (max index 1,300,678 — fits u21)
- `max_qubit`     : 1,354 (well under u32 = 4.29 billion)
- `max_bit`       : 1,300,678 (also well under u32)

**Conclusion: u32 slot indices are safe for this circuit & for any
realistic point-add-toy descendant.** The packing scheme is sound on
index width.

The original task-spec's proposal of `u16 c_target_or_condition_*`
would **fail** — max bit 1,300,678 > 65,535.

## QECCOPS2 format spec (what landed)

```
magic   : "QECCOPS2"   (8 bytes)
n_ops   : u64 LE       (8 bytes)
ops     : n_ops × 24 bytes:
  [0]   u8  kind                       (0..=17 validated)
  [1]   u8  flags                      bit0=has_q_c2,  bit1=has_q_c1,
                                       bit2=has_q_t,   bit3=has_c_t,
                                       bit4=has_c_cond. bits 5-7 reserved
  [2-3] u16 reserved                   (zero, alignment)
  [4-7]  u32 q_control2                (0 if !has_q_c2)
  [8-11] u32 q_control1
  [12-15] u32 q_target
  [16-19] u32 c_target
  [20-23] u32 c_condition
```

- `r_target` is **omitted**. The kernel (`sim_gpu.cu`) never reads it,
  so we drop it from VRAM-resident state. Upstream QECCOPS1 still
  carries it for round-trip with `eval_circuit.rs`.
- File-size ratio at n_ops = 12.8M:
  - QECCOPS1 :  716,134,680 B = 716 MB
  - QECCOPS2 :  306,914,872 B = 307 MB
  - Ratio   : **2.33× smaller on disk**.

## VRAM resident bandwidth — packed vs unpacked

Per-shot state remains the dominant traffic. Concrete numbers from the
n_batches=128 run:

- Op stream traffic (warp-broadcast, mostly L2/L1): 12.8 M × 24 B =
  307 MB per shot (packed) vs 12.8 M × 28 B = 358 MB (unpacked) — a
  51 MB delta which the cache absorbs.
- State traffic (per-shot, all threads diverge): each op reads ~2 u64
  state lanes × n_batches threads = 16 × 128 = 2048 B per op =
  12.8 M × 2048 B = 26 GB streamed per shot at n_batches=128.

The per-shot state stream is ~85× larger than the op stream. Reducing
the op stream by 14% delivers ~14% × (op-fraction of total) speedup.
Op fraction ≈ 358 MB / (358 MB + 26 GB) ≈ 1.4%. So packing's
theoretical max impact at this n_batches is ~ 1.014×. Measured 1.065×
exceeds that, suggesting cache-line alignment & smaller register
spill (the kernel pulls a smaller struct into registers per op) also
contribute marginally.

## Did the bandwidth-bound diagnosis flip to compute-bound?

**No.** Per-shot state traffic still dominates. To flip the diagnosis
we would need to (a) packed-bit-state storage (8 shots per byte
instead of 64 shots per u64 — wrong direction, that's the same lane
width); or (b) change the parallelism axis from per-shot to per-
candidate, which is the **option 1** refactor in `form-D-build-
progress.md`. That refactor remains the open lever for moving the
1.07× ceiling. Op packing is a clean ~7% boost that stacks on top of
whichever axis-flip lands next.

## Memory safety

- Per-side state at n_batches=128: 3.5 GB device + 3.5 GB host.
- Two device contexts simultaneous (unpacked + packed) — peak ≈
  10.5 GB. Stayed under 16 GB cap.
- 3090 has 23561 MB free; we used ~10.5 GB; no OOM, no QEMU envelope
  needed for this CUDA workload (3090-ai host runs no concurrent
  search loop while we benchmark).

## Files landed under ecdsa/cuda/

- `sim_packed.h` — `PackedOp` (24 B) struct + QECCOPS2 magic / flags
- `ops_loader_v2.c` — `load_ops_bin_v2` (→ unpacked Op) +
  `load_packed_ops_bin_v2` (→ PackedOp)
- `ops_loader_v2_writer.c` — `convert_ops_bin <in> <out>` (QECCOPS1 →
  QECCOPS2)
- `sim_gpu_packed.cu` — `sim_kernel_packed` reading PackedOp directly
- `main_packed_demo.c` — drives unpacked & packed kernels on the same
  circuit + CPU ground-truth, reports byte-identity + timings
- `Makefile` — `demo_packed`, `convert_ops_bin`, `remote-packed`
  targets

`ops_loader.c`, `sim_cpu.c`, `sim_gpu.cu`, `sim.h`, & the lumbda
`emit-ops-bin.lsp` walker remain **untouched**. Upstream contract is
preserved; QECCOPS2 lives only under our cuda/ tree.

## Follow-on (deferred)

- `emit-ops-bin-v2.lsp` (lumbda walker emitting QECCOPS2 natively) —
  defer until we see a workload where the ~14% on-disk shrink matters
  more than maintaining a single emit path.
- Per-candidate axis-flip refactor of `sim_gpu.cu` (build-progress doc
  option 1) — that is the lever that can actually move 1.07× → useful.
  Packing stacks on top once the axis lands.
- Bend / GPU-worker `gpu-worker.lsp` integration with QECCOPS2 — only
  worth wiring once a search loop wants the 7% kernel boost
  consistently.

## Conclusion

Packing landed clean. Byte-identical. ~7% kernel speedup at every
batch size, ~2.33× on-disk shrink. The 3.5× headline hypothesis was
anchored to the on-disk format, not the VRAM-resident format the
kernel actually consumes; it does not apply. The build-progress
diagnosis (per-shot state owns the ceiling) survives unchallenged.
