bend: planning docs for forms A, B, D (CATALOG.md build runway)
Three implementation plans land per CATALOG.md "Build order —
recommended 3-form runway":
form-B-bignum-cgbn.md (3-5 dev-days, fewest unknowns):
direct NVlabs CGBN consumption. 9-op dispatcher
(mod-add/sub/mul/sqr/inv/exp/reduce + add-no-mod +
mul-no-mod-2x-width). BSHK/BSHR mirrors shake256-fanout.
15k crossover (wire overhead vs raw kernel). Risks:
CGBN maintenance freshness + sm_86/sm_89 verification.
form-A-secp256k1-batched-mul.md (6 dev-days):
FixedPaul/VanitySearch-Bitcrack picked (AGPL-3, active
2025-05-16) over JeanLucPons (GPL-3, stale 2022-04-04).
Vendor 3 headers ~2400 LoC + write secp_mul_batch_kernel.
BSPK/BSPR wire mirrors BSHK. Kernel extraction is the
dominant risk.
form-D-clifford-stabilizer.md (10 dev-days):
STABSim & Qimax not open-source; clean-room
Aaronson-Gottesman tableau recommended. Reuse QECCOPS1
ops.bin over BSHK op_id=0x02 — no .stim translation.
Toffoli option (b): score symbolically, simulate
Clifford-only — gated on first measuring Toffoli fraction
(point-add suspected ~50%, Amdahl eats gains if worse).
Plans live as actionable instructions in second-person voice;
each cites URLs for every claim. CATALOG.md & bend.html
untouched here — plans inform future surveys without inventing
numbers.
This commit is contained in:
parent
a33ae4331d
commit
4c5e04b07f
3 changed files with 768 additions and 0 deletions
272
examples/cuda-fanout/plans/form-A-secp256k1-batched-mul.md
Normal file
272
examples/cuda-fanout/plans/form-A-secp256k1-batched-mul.md
Normal file
|
|
@ -0,0 +1,272 @@
|
|||
# Form A — `cuda-secp256k1-batched-mul` implementation plan
|
||||
|
||||
Status: design only. No binary built. No remote GPU touched.
|
||||
|
||||
Reference catalog entry: `../CATALOG.md` § A. Target: a `secp256k1-batch-mul`
|
||||
daemon binary that speaks the BSHK binary wire protocol the worker already
|
||||
drives for `shake256-fanout`, sitting next to it in `examples/cuda-fanout/`.
|
||||
|
||||
## 1. Upstream selection — pick FixedPaul/VanitySearch-Bitcrack
|
||||
|
||||
You would fork FixedPaul/VanitySearch-Bitcrack, not JeanLucPons. Reasons:
|
||||
|
||||
- **License.** FixedPaul ships AGPL-3.0
|
||||
(https://github.com/FixedPaul/VanitySearch-Bitcrack), matching
|
||||
`shake256-fanout.cu`'s AGPLv3 header. JeanLucPons is GPL-3.0
|
||||
(https://github.com/JeanLucPons/VanitySearch). AGPL ↔ AGPL stays a
|
||||
clean derivative.
|
||||
- **Activity.** FixedPaul last touched master 2025-05-16; JeanLucPons
|
||||
last commit 2022-04-04 (commits pages on each repo). Three years
|
||||
stale vs actively maintained.
|
||||
- **Speed.** FixedPaul publishes 6.9 Gkeys/s on 4090, 8.8 Gkeys/s on
|
||||
5090 (repo README). Their "optimized CUDA modular math" patch over
|
||||
upstream is the win we want.
|
||||
|
||||
No 3090 number published; you would measure on `3090-ai.foxhop.net`
|
||||
once a binary exists. Expect ~3 Gkeys/s extrapolating from 3080's 2.65
|
||||
Gkeys/s in our catalog.
|
||||
|
||||
## 2. Kernel extraction — files you need
|
||||
|
||||
Upstream layout
|
||||
(https://github.com/JeanLucPons/VanitySearch/tree/master/GPU; fork
|
||||
preserves shape):
|
||||
|
||||
- **`GPU/GPUMath.h`** (~900 lines) — `__device__` modular arithmetic
|
||||
with inline PTX: `_ModMult`, `_ModSqr`, `ModSub256`, `_ModInv`, and
|
||||
crucially `_ModInvGrouped` (Montgomery batched inversion: one inverse
|
||||
per `GRP_SIZE/2+1` points). Keep verbatim.
|
||||
- **`GPU/GPUGroup.h`** (~1024 lines) — precomputed `Gx[][]`, `Gy[][]`
|
||||
for G, 2G, … 512G plus `_2Gnx`/`_2Gny`. Windowed-mul ladder. Keep
|
||||
verbatim.
|
||||
- **`GPU/GPUCompute.h`** (~450 lines) — `ComputeKeys()` Shamir-trick
|
||||
loop (P+iG and P-iG share one delta-x). Keep field arithmetic;
|
||||
**strip** SHA256 + RIPEMD160 + base58 + address-prefix branches.
|
||||
- **`GPU/GPUEngine.cu`** (~920 lines, 5 `__global__` kernels:
|
||||
`comp_keys`, `comp_keys_p2sh`, `comp_keys_comp`, `comp_keys_pattern`,
|
||||
`comp_keys_p2sh_pattern`) — **drop entirely.** Write one new
|
||||
`secp_mul_batch_kernel(scalars[N], base, points[N])` calling
|
||||
`_ModInvGrouped`.
|
||||
|
||||
Drop: `main.cpp`, `Vanity.cpp`, `Base58.cpp`, `Bech32.cpp`,
|
||||
`Wildcard.cpp`, `hash/*` (SSE host hashes), `GPUBase58.h`, `GPUHash.h`,
|
||||
`GPUWildcard.h`, `GPUGenerate.cpp` — 19 .cpp files per upstream
|
||||
Makefile, zero kept.
|
||||
|
||||
License: copy each file's header verbatim, add SPDX line, drop a
|
||||
`LICENSE-vanitysearch-bitcrack` beside our existing license note.
|
||||
|
||||
## 3. Wire protocol — BSHK request, BSHR response
|
||||
|
||||
Mirror the existing shake protocol (`shake256-fanout.cu` lines 286-292):
|
||||
|
||||
Request, little-endian throughout, no alignment padding:
|
||||
|
||||
```
|
||||
magic u8[4] = "BSHK"
|
||||
op_id u32 = 0x01 (secp_mul_batch)
|
||||
n u32 = number of scalars
|
||||
base_x u8[32] base point X (big-endian field element, secp256k1 native)
|
||||
base_y u8[32] base point Y
|
||||
scalars u8[n*32] big-endian 256-bit ints
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```
|
||||
magic u8[4] = "BSHR" on success, "BERR" on failure
|
||||
status u32 = 0 ok, nonzero error code (1 bad-magic, 2 bad-n,
|
||||
3 cuda-launch-fail, 4 truncated-input)
|
||||
n u32 = echoed scalar count
|
||||
points u8[n*64] N points, each (X32 || Y32) big-endian; affine,
|
||||
infinity encoded as all-zero
|
||||
```
|
||||
|
||||
Endianness: LE u32 headers (matches shake binary at
|
||||
`shake256-fanout.cu:286`), BE field elements (matches secp256k1 native;
|
||||
ships straight to openssl/python-ecdsa for validation). Worker's
|
||||
`wire-send-raw` treats payload as opaque (`gpu-worker.lsp:264`), no
|
||||
change there.
|
||||
|
||||
## 4. `gpu-worker.lsp` handler sketch
|
||||
|
||||
You add one handler analogous to `handle-binary-shake` at
|
||||
`gpu-worker.lsp:253`. The daemon stays warm for the whole worker
|
||||
lifetime — same pattern, same `daemon-process` helper:
|
||||
|
||||
```scheme
|
||||
(define *binary-secp-mul* "./secp256k1-batch-mul")
|
||||
|
||||
;; in main: register at boot, paying CUDA init once.
|
||||
(register-daemon! 'cuda-secp-mul-batch *binary-secp-mul*)
|
||||
|
||||
;; new binary dispatcher: payload[0..4] = "BSPK" magic to disambiguate
|
||||
;; from "BSHK" shake requests.
|
||||
(define (handle-binary-secp-mul client payload)
|
||||
(let* ((daemon (cdr (assoc 'cuda-secp-mul-batch *daemons*)))
|
||||
(in-path (gensym-path "/tmp/bend-secp-in" ".bin"))
|
||||
(out-path (gensym-path "/tmp/bend-secp-out" ".bin"))
|
||||
(blob (substring payload 4 (string-length payload))))
|
||||
(write-binary-file in-path blob)
|
||||
(let ((status (daemon-process daemon in-path out-path #t)))
|
||||
(cond
|
||||
((eq? status 'ok)
|
||||
(let ((result (read-binary-file out-path)))
|
||||
(delete-file in-path) (delete-file out-path)
|
||||
(wire-send-raw client (string-append "BSPR" result))))
|
||||
(else
|
||||
(delete-file in-path) (delete-file out-path)
|
||||
(wire-send-raw client (string-append "BERR" (cdr status))))))))
|
||||
```
|
||||
|
||||
Extend `handle-one` (`gpu-worker.lsp:271`) to branch on first 4 bytes:
|
||||
`BSHK` → shake, `BSPK` → secp-mul. S-expression path stays for `ping`
|
||||
and any text-mode ops. CUDA context init runs twice at startup, never
|
||||
per request.
|
||||
|
||||
## 5. Daemon protocol — match shake256-fanout exactly
|
||||
|
||||
`secp256k1-batch-mul --daemon` reads stdin one cmd per line, emits
|
||||
one response per line, identical to `shake256-fanout.cu:226-269`:
|
||||
|
||||
```
|
||||
ready ; once at startup after cudaFree(0)
|
||||
process-bin <in> <out> ; binary request
|
||||
done <out> ; success
|
||||
error <reason> ; failure
|
||||
quit / bye ; shutdown
|
||||
```
|
||||
|
||||
`daemon-process` in `gpu-worker.lsp:71` already writes
|
||||
`process-bin <in> <out>\n` & expects `done`/`error` — zero handler-side
|
||||
change. `start-daemon` (line 62) waits for `ready` before serving, so
|
||||
CUDA init blocks worker readiness deterministically.
|
||||
|
||||
CLI fallback: `secp256k1-batch-mul --binary <in> <out>` for one-shot
|
||||
test (matches `shake256-fanout.cu:272`).
|
||||
|
||||
## 6. Build recipe
|
||||
|
||||
Two options:
|
||||
|
||||
- **A — submodule.** `git submodule add
|
||||
https://github.com/FixedPaul/VanitySearch-Bitcrack
|
||||
third_party/vanitysearch-bitcrack`. Pro: trivial upstream-bump.
|
||||
Con: ~12 MB of unused .cpp / .vcxproj noise.
|
||||
- **B — vendor three headers.** Copy `GPUMath.h`, `GPUGroup.h`,
|
||||
`GPUCompute.h` into
|
||||
`examples/cuda-fanout/vendor/secp256k1-vanitysearch/` with
|
||||
AGPL-3.0 + SPDX preserved, add a `PROVENANCE.md` carrying source
|
||||
commit SHA. Pro: lean, audit-friendly. Con: manual sync.
|
||||
|
||||
Pick B — ~2400 lines, mirrors how `shake256-fanout.cu` carries its
|
||||
tiny-sha3 distillation inline.
|
||||
|
||||
Makefile additions:
|
||||
|
||||
```
|
||||
NVCCFLAGS_SECP ?= -O3 -arch=sm_86 -Xcompiler="-O3 -Wall" \
|
||||
-maxrregcount=0 --ptxas-options=-v
|
||||
|
||||
all: shake256-fanout secp256k1-batch-mul
|
||||
|
||||
secp256k1-batch-mul: secp256k1-batch-mul.cu \
|
||||
vendor/secp256k1-vanitysearch/GPUMath.h \
|
||||
vendor/secp256k1-vanitysearch/GPUGroup.h \
|
||||
vendor/secp256k1-vanitysearch/GPUCompute.h
|
||||
$(NVCC) $(NVCCFLAGS_SECP) -Ivendor/secp256k1-vanitysearch -o $@ $<
|
||||
|
||||
test-secp: secp256k1-batch-mul test_secp_roundtrip.py
|
||||
./secp256k1-batch-mul --binary /tmp/secp-in.bin /tmp/secp-out.bin
|
||||
python3 test_secp_roundtrip.py /tmp/secp-in.bin /tmp/secp-out.bin
|
||||
|
||||
bench-secp: secp256k1-batch-mul bench_secp.py
|
||||
python3 bench_secp.py ./secp256k1-batch-mul
|
||||
```
|
||||
|
||||
`sm_86` covers 3090; `sm_89` covers 4090. Build a fat binary with
|
||||
`-gencode arch=compute_86,code=sm_86 -gencode arch=compute_89,code=sm_89`
|
||||
for both fleet hosts.
|
||||
|
||||
Worker auto-registration: one line, `(register-daemon!
|
||||
'cuda-secp-mul-batch *binary-secp-mul*)` next to the shake
|
||||
registration at `gpu-worker.lsp:299`.
|
||||
|
||||
## 7. Validation plan
|
||||
|
||||
**Correctness.** `test_secp_roundtrip.py` runs N=100 known scalars
|
||||
(1, 2, 3, …, plus edge cases: curve order n → infinity, n-1, scalars
|
||||
near 2^128, 2^256-1). Computes `k*G` on CPU via `coincurve` (or
|
||||
`ecdsa.SECP256k1.generator * k`), byte-compares GPU output. Halt on
|
||||
first mismatch. Cross-check at N=10 against openssl `EC_POINT_mul` to
|
||||
catch a coincurve defect masking a kernel defect.
|
||||
|
||||
**Performance.** `bench_secp.py` sweeps N ∈ {1k, 10k, 100k, 1M},
|
||||
records `wire_ms`, `kernel_ms` (extend response to carry `u32
|
||||
kernel_us` after `n`), `gpu/cpu` ratio. CPU baseline: in-process
|
||||
`coincurve` loop. Median over 20 runs. Format mirrors
|
||||
`bench_tiers.py:174` so the summary table renders unchanged.
|
||||
|
||||
Expected: 3090 ~3 Gkeys/s, 4090 ~6.9 Gkeys/s. CPU baseline ~50
|
||||
kkeys/s single-thread coincurve → ~60,000× per core, ~5,000× vs
|
||||
12-thread. Below 1,000× signals a defect — most likely PCIe transfer
|
||||
saturating (1M × 64 B = 64 MB out, ~2 ms at PCIe 4.0).
|
||||
|
||||
## 8. Risks & open questions
|
||||
|
||||
- **License audit.** FixedPaul declares AGPL-3.0 at repo level but
|
||||
`GPU/*.h` headers may still carry GPLv3 inherited from JeanLucPons.
|
||||
Audit each vendored file before commit. If a file is GPLv3-only,
|
||||
our binary becomes GPLv3 (still compatible with our AGPLv3 worker
|
||||
via §7 of GPLv3).
|
||||
- **Endomorphism in fork.** FixedPaul claims "optimized CUDA modular
|
||||
math" but lists no specifics. If the speedup is a private
|
||||
`GPUMath.h` patch, vendoring upstream headers alone loses it.
|
||||
Diff `FixedPaul/GPU/GPUMath.h` vs `JeanLucPons/GPU/GPUMath.h` first;
|
||||
take FixedPaul's.
|
||||
- **Host RAM at N=1M.** 1M × 64 B = 64 MB out, 1M × 32 B = 32 MB in.
|
||||
Worker holds ~3 copies (in-portal + daemon + out-portal): peak ~300
|
||||
MB transient. Bounded on 24 GB / 80 GB hosts.
|
||||
- **GRP_SIZE preservation.** Upstream's `GRP_SIZE=1024` means each
|
||||
thread does 1024 keys sequentially sharing one `_ModInvGrouped` call.
|
||||
A naive one-thread-per-scalar kernel loses the amortisation — ~50×
|
||||
worse. New kernel launches `ceil(N/1024)` threads, each handling a
|
||||
1024-scalar group; N rounds up to multiple of 1024 with sentinel
|
||||
scalar = 1, pad outputs discarded.
|
||||
- **Device memory.** 1M-point request allocates ~96 MB device-side
|
||||
(in/out + scratch). `cudaFree` between requests to avoid fragmentation
|
||||
over 10k+ requests. Add explicit free in `process_one_bin`.
|
||||
|
||||
## 9. Effort estimate
|
||||
|
||||
Experienced CUDA dev, no fleet-side work counted:
|
||||
|
||||
| step | days |
|
||||
|---|---|
|
||||
| vendor headers, audit licenses, write PROVENANCE.md | 0.5 |
|
||||
| write `secp_mul_batch_kernel` reusing `_ModInvGrouped` group loop | 1.5 |
|
||||
| daemon shell (stdin loop, --binary, --daemon modes) — copy shake | 0.5 |
|
||||
| binary I/O, BSHK/BSHR parse + emit, host-side scalar layout | 0.5 |
|
||||
| Makefile, smoke test on local nvcc (host has no GPU — compile only) | 0.5 |
|
||||
| `test_secp_roundtrip.py` + coincurve oracle + openssl cross-check | 1.0 |
|
||||
| `bench_secp.py` + sweeps + first 3090/4090 measurement | 1.0 |
|
||||
| `gpu-worker.lsp` handler + dispatch branch + integration test | 0.5 |
|
||||
| catalog promotion (live row, measured speedup) | 0.0 (fox's call) |
|
||||
| **total** | **6 days** |
|
||||
|
||||
Concentration: ~40% kernel-extraction (steps 1+2), ~25% validation,
|
||||
~15% bench harness, ~10% daemon/wire, ~10% handler. Kernel-extraction
|
||||
risk dominates; everything else mirrors shake256-fanout.
|
||||
|
||||
## URLs cited
|
||||
|
||||
- Catalog § A: ../CATALOG.md
|
||||
- Existing shake daemon: ../shake256-fanout.cu
|
||||
- Worker handler pattern: ../gpu-worker.lsp lines 62, 71, 253, 264, 299
|
||||
- Wire bench reference: ../bench_tiers.py lines 150-158, 174
|
||||
- VanitySearch original: https://github.com/JeanLucPons/VanitySearch
|
||||
- VanitySearch GPU folder: https://github.com/JeanLucPons/VanitySearch/tree/master/GPU
|
||||
- VanitySearch last commit: https://github.com/JeanLucPons/VanitySearch/commits/master
|
||||
- Bitcrack fork: https://github.com/FixedPaul/VanitySearch-Bitcrack
|
||||
- Bitcrack last commit: https://github.com/FixedPaul/VanitySearch-Bitcrack/commits/master
|
||||
- gECC paper (speedup reference): https://arxiv.org/pdf/2501.03245
|
||||
240
examples/cuda-fanout/plans/form-B-bignum-cgbn.md
Normal file
240
examples/cuda-fanout/plans/form-B-bignum-cgbn.md
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
# Form B — `cuda-bignum-cgbn` implementation plan
|
||||
|
||||
You wrap NVlabs CGBN (Cooperative Groups Big Numbers) as a BSHK-speaking
|
||||
daemon under our existing bend-form pattern. Output: a single CUDA
|
||||
binary `cgbn-batch-worker`, a Scheme handler in `gpu-worker.lsp`, a
|
||||
Makefile target, & a validation harness. Cross-cite: catalog entry
|
||||
`cuda-bignum-cgbn` in `CATALOG.md` (form B, build order step 3), source
|
||||
NVlabs/CGBN at https://github.com/NVlabs/CGBN, midsize-int benchmark at
|
||||
https://arxiv.org/pdf/2405.14642.
|
||||
|
||||
## 1. CGBN model → wire model
|
||||
|
||||
CGBN groups 4/8/16/32 threads into one **instance**; each instance owns
|
||||
one bignum (32-bit to 32 K-bit). The library handles cooperative-group
|
||||
dispatch (https://github.com/NVlabs/CGBN/blob/master/samples/sample_01_add/add.cu);
|
||||
you write a kernel templated on `cgbn_env_t<context_t, BITS>` & call
|
||||
`cgbn_load / cgbn_mul / cgbn_modular_inverse / cgbn_store`.
|
||||
|
||||
For our 256-bit secp256k1 case: `TPI=8` (threads per instance) at 256 bits
|
||||
is the published sweet spot per CGBN README. One warp (32 threads) =
|
||||
4 instances. A 3090 SM holds 1024 threads = 128 instances per SM × 82 SMs
|
||||
= **10,496 in-flight 256-bit ops per kernel wave**. Map: one BSHK
|
||||
request = one launch = N instances. We never expose CGBN's cooperative-
|
||||
group primitive across our wire; we expose **op-coded batches**.
|
||||
|
||||
## 2. Operation dispatcher — `cgbn-batch-worker`
|
||||
|
||||
Single binary, op-coded. Op table (one byte each, room to grow):
|
||||
|
||||
| op_id | mnemonic | inputs | output | kernel |
|
||||
|------:|----------------|--------|--------|---------------------------------------|
|
||||
| 0x01 | mod-add | a, b | a+b mod m | `cgbn_add` + `cgbn_rem` |
|
||||
| 0x02 | mod-sub | a, b | a-b mod m | `cgbn_sub` + conditional `cgbn_add` |
|
||||
| 0x03 | mod-mul | a, b | a*b mod m | `cgbn_mont_mul` (precomputed `m`) |
|
||||
| 0x04 | mod-sqr | a | a^2 mod m | `cgbn_mont_sqr` |
|
||||
| 0x05 | mod-inv | a | a^-1 mod m | `cgbn_modular_inverse` |
|
||||
| 0x06 | mod-exp | a, b | a^b mod m | `cgbn_modular_power` (binary ladder) |
|
||||
| 0x07 | mod-reduce | a | a mod m | `cgbn_rem` standalone |
|
||||
| 0x08 | add-no-mod | a, b | a+b | `cgbn_add` straight |
|
||||
| 0x09 | mul-no-mod | a, b | a*b (lift to 2 × width) | `cgbn_mul_wide` |
|
||||
|
||||
Kernel dispatch via single `__global__` switch on op_id. CGBN environment
|
||||
template is bit-width parameterized; you compile one specialization per
|
||||
expected bit-width (256, 384, 521) & route via a runtime `switch
|
||||
(bitwidth)` over the templated launchers. Default bit-widths to ship:
|
||||
256, 384. Wider widths land on demand.
|
||||
|
||||
## 3. Wire protocol — BSHK / BSHR for CGBN
|
||||
|
||||
Request (little-endian throughout, matching `bench_tiers.py`
|
||||
`make_payload_binary` in this dir):
|
||||
|
||||
```
|
||||
"BSHK" (4 bytes magic)
|
||||
u32 op_id
|
||||
u32 bitwidth ; one of {256, 384}
|
||||
u32 n_instances
|
||||
u8[bitwidth/8] modulus m (omitted for op 0x08, 0x09)
|
||||
u8[n * bitwidth/8] operand_a
|
||||
u8[n * bitwidth/8] operand_b (omitted for 0x04, 0x05, 0x07)
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```
|
||||
"BSHR" (4 bytes magic — matches handle-binary-shake in gpu-worker.lsp)
|
||||
u32 status ; 0 = ok; nonzero = error code
|
||||
u32 n_instances
|
||||
u8[n * width_out/8] result ; width_out = bitwidth or 2*bitwidth
|
||||
```
|
||||
|
||||
For 256-bit secp256k1 case @ N=1M instances: input ~64 MB
|
||||
(modulus + 2 operand arrays), output ~32 MB. CGBN documents 32K-bit max
|
||||
per instance (see `cgbn.h` `BITS` template parameter
|
||||
https://github.com/NVlabs/CGBN/blob/master/include/cgbn/cgbn.h); 256-bit
|
||||
fits trivially. TPI=8 at 256 bits leaves no SM under-utilization on
|
||||
3090/4090 because each SM still hosts 128 in-flight instances.
|
||||
|
||||
## 4. `gpu-worker.lsp` handler
|
||||
|
||||
You add `handle-cuda-bignum-cgbn` modeled on `handle-cuda-shake-fanout`
|
||||
plus the binary path in `handle-binary-shake` already in this dir's
|
||||
`gpu-worker.lsp`. Two surfaces:
|
||||
|
||||
S-expression (small N, dev convenience):
|
||||
|
||||
```scheme
|
||||
(cgbn-batch
|
||||
(op . mod-mul)
|
||||
(bitwidth . 256)
|
||||
(modulus . "fffffffffffffffffffffffffffffffffffffffffffffffffffffffefffffc2f")
|
||||
(a . ("..." "..."))
|
||||
(b . ("..." "...")))
|
||||
```
|
||||
|
||||
Binary (production): client sends `BSHK | u32 op_id | …` blob exactly as
|
||||
above. Worker writes blob to `/tmp/bend-cgbn-in-<ts>.bin`, calls daemon
|
||||
`process-bin <in> <out>`, reads `/tmp/bend-cgbn-out-<ts>.bin`, prepends
|
||||
`"BSHR"`, sends back via `wire-send-raw`. Register the daemon at startup
|
||||
next to `register-daemon!` for `'cuda-shake-fanout`:
|
||||
|
||||
```scheme
|
||||
(define *binary-cgbn-batch*
|
||||
(or (get-environment-variable "CGBN_BATCH_WORKER")
|
||||
"./cgbn-batch-worker"))
|
||||
(register-daemon! 'cuda-bignum-cgbn *binary-cgbn-batch*)
|
||||
```
|
||||
|
||||
Dispatch in `handle-request`: `((eq? op 'cgbn-batch) (handle-cuda-bignum-cgbn args))`.
|
||||
|
||||
## 5. Daemon protocol
|
||||
|
||||
Inherited verbatim from `shake256-fanout.cu --daemon` (this dir): stdin
|
||||
line-protocol `process-bin <in> <out>` / `quit`, stdout `ready` once
|
||||
CUDA context warms, `done <out>` / `error <reason>` per command, `bye`
|
||||
on quit. CUDA context init pays once per worker startup, never per BSHK
|
||||
request — same architectural point the SHAKE form already proved.
|
||||
|
||||
## 6. Build recipe
|
||||
|
||||
```makefile
|
||||
CGBN_DIR := vendor/CGBN
|
||||
NVCC_ARCH_3090 := -arch=sm_86
|
||||
NVCC_ARCH_4090 := -arch=sm_89
|
||||
NVCC_FLAGS := -O3 -I$(CGBN_DIR)/include --extended-lambda
|
||||
|
||||
cgbn-batch-worker: cgbn-batch-worker.cu $(CGBN_DIR)/include/cgbn/cgbn.h
|
||||
nvcc $(NVCC_FLAGS) $(NVCC_ARCH_3090) -gencode=arch=compute_89,code=sm_89 \
|
||||
-o $@ cgbn-batch-worker.cu -lgmp
|
||||
|
||||
vendor/CGBN/include/cgbn/cgbn.h:
|
||||
git submodule add https://github.com/NVlabs/CGBN vendor/CGBN
|
||||
|
||||
cgbn-bench: cgbn-batch-worker
|
||||
python3 bench_cgbn.py --small 1000 --medium 100000 --huge 1000000
|
||||
|
||||
cgbn-test: cgbn-batch-worker
|
||||
python3 test_cgbn_known_answers.py
|
||||
```
|
||||
|
||||
CGBN ships as header-only inside `include/cgbn/` — no separate
|
||||
`libcgbn.so` build. Vendor the headers; do not pull the full repo into
|
||||
our build graph. License: BSD 3-Clause per https://github.com/NVlabs/CGBN/blob/master/LICENSE
|
||||
— compatible with our AGPLv3 binary distribution (CGBN headers ship
|
||||
permissively, our wrapper binary stays AGPLv3).
|
||||
|
||||
## 7. Validation plan
|
||||
|
||||
Three layers, cheapest first:
|
||||
|
||||
1. **openssl bignum CLI** for op 0x03 mod-mul on tiny operands. Drive
|
||||
from `test_cgbn_known_answers.py`: pass N=32 hand-picked triples
|
||||
(a, b, m, expected = a*b mod m). Sanity, not throughput.
|
||||
2. **python gmpy2** for batches at N ∈ {1k, 10k, 100k}. Generate
|
||||
random a, b ∈ [0, m); compute reference on host (`gmpy2.f_mod(a*b, m)`),
|
||||
compare byte-for-byte against GPU result. Cover ops 0x01, 0x03, 0x04,
|
||||
0x05, 0x06. gmpy2 docs: https://gmpy2.readthedocs.io.
|
||||
3. **secp256k1 round-trip**: m = secp256k1 field prime
|
||||
`0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffefffffc2f`,
|
||||
load N random 256-bit a, b; compute a*b mod m through CGBN; verify
|
||||
against Python `coincurve` reference (https://github.com/ofek/coincurve).
|
||||
Halt promotion if any byte mismatches.
|
||||
|
||||
Cross-tier portal validation: identical BSHK input bytes to Python,
|
||||
C tree-walker, asm-gc workers via our `bench_tiers.py` shape (already
|
||||
binary-wire-aware) must produce byte-identical BSHR responses. Mismatch
|
||||
on any tier halts promotion per `ecdsa/CLAUDE.md` cross-tier rule.
|
||||
|
||||
## 8. Performance projection
|
||||
|
||||
Sources: midsize-int benchmark https://arxiv.org/pdf/2405.14642 (Table 3
|
||||
reports ~100×+ on 256-bit dense mul vs Xeon-20c GMP+OpenMP on V100);
|
||||
NVlabs CGBN README claims 5-10× over GMP per-op on Volta. A 3090 is
|
||||
~3× V100 throughput on dense integer work (per
|
||||
https://www.nvidia.com/en-us/data-center/a100/ comparison & 3090
|
||||
whitepaper).
|
||||
|
||||
Rough numbers, to be confirmed on hardware:
|
||||
|
||||
- **mod-mul 256-bit**: ~3-5 Gops/s on a 3090 (TPI=8 at warp-saturated
|
||||
occupancy, kernel-only). 4090 ~1.6× that. Reference: midsize-int paper
|
||||
Fig 7, scaled by 3090/V100 fp32 ratio.
|
||||
- **mod-inv 256-bit** (CGBN's binary GCD path): ~50-100 Mops/s on 3090.
|
||||
Bernstein-Yang batched variant (form E in catalog) wins ~3-5× over
|
||||
this once we add it.
|
||||
- **mod-exp 256-bit** (256-bit ladder): ~10-30 Mops/s on 3090.
|
||||
- **wire-overhead crossover**: BSHK round-trip pays ~3 ms median per call
|
||||
per `bench_tiers.py` binary-mode at small N (this dir's existing
|
||||
measurements). At 3 Gops/s kernel rate, you break even with CPU GMP
|
||||
(~5 Mops/s single-thread) around **N ≈ 15,000 instances per BSHK
|
||||
call**. Stay above N=100k for clean 100×.
|
||||
|
||||
## 9. Risks & open questions
|
||||
|
||||
- **CGBN maintenance.** Last public commit on https://github.com/NVlabs/CGBN
|
||||
is 2021-era. NVlabs has not formally archived it. You must verify
|
||||
it compiles under CUDA 12.x on Ampere & Ada (sm_86, sm_89). If not,
|
||||
patch `cgbn.h` toolchain checks ourselves & vendor the patched copy.
|
||||
- **License.** BSD 3-Clause per https://github.com/NVlabs/CGBN/blob/master/LICENSE.
|
||||
No AGPL incompatibility for our binary distribution.
|
||||
- **sm_86 / sm_89 support.** CGBN documents Volta + Turing as primary
|
||||
targets. Ampere (sm_86) cooperative-groups API is a superset, so the
|
||||
cooperative-thread-array primitives CGBN calls remain valid. Open
|
||||
risk: warp-shuffle reductions in `cgbn_add` may need width
|
||||
adjustment for newer warp-scheduler behavior. Mitigation: ship
|
||||
validation suite first; any divergence shows up against gmpy2 ref.
|
||||
- **Under-utilization at 256-bit / TPI=8.** Each warp owns 4
|
||||
independent instances. At N=1M & 82 SMs × 128 in-flight, you get
|
||||
~95 kernel waves on a 3090. Each wave is ~µs; total kernel ~100 µs.
|
||||
Memory bandwidth (32 MB input, 32 MB output → 64 MB at 936 GB/s =
|
||||
~70 µs) becomes the floor. Kernel & PCIe transfer are co-dominant.
|
||||
- **PCIe transfer budget.** 64 MB host→device + 32 MB back via PCIe
|
||||
Gen4 x16 (32 GB/s) = ~3 ms. Already comparable to wire overhead;
|
||||
pinned-host memory & overlap with kernel via streams cuts this in
|
||||
half.
|
||||
|
||||
## 10. Effort estimate
|
||||
|
||||
You should land this in **3-5 days** end-to-end as an experienced CUDA
|
||||
developer. Breakdown:
|
||||
|
||||
- Day 1: vendor CGBN, build sample_01_add unmodified on 3090 host,
|
||||
confirm sm_86 compatibility, build baseline `cgbn-batch-worker.cu`
|
||||
with op 0x03 mod-mul only.
|
||||
- Day 2: daemon mode (clone shake256-fanout `--daemon` skeleton), add
|
||||
ops 0x01, 0x04, 0x05, wire validation against gmpy2 at N=1k, 10k.
|
||||
- Day 3: gpu-worker.lsp handler, BSHK dispatch, cross-tier validation
|
||||
(Python + C + asm via `bench_tiers.py`-style harness), portal test
|
||||
for secp256k1 mod-mul.
|
||||
- Day 4: remaining ops (0x02, 0x06, 0x07, 0x08, 0x09), bench_cgbn.py
|
||||
driver covering all ops at N ∈ {1k, 100k, 1M}, telemetry lines in
|
||||
Scheme handler (wall-ms / cpu-ms / gpu-ms / gpu/cpu ratio).
|
||||
- Day 5 (buffer): polish, README, CATALOG.md row promotion to `live`,
|
||||
Makefile install/test targets, runs/lumbda-cgbn-001/ baseline
|
||||
artifacts.
|
||||
|
||||
Lower than form A (`cuda-secp256k1-batched-mul`) since CGBN is a
|
||||
**direct API consumption** — no extraction effort from a research
|
||||
codebase, no curve-specific Montgomery tuning to port. Form A risks 7-10
|
||||
days; form B caps at 5.
|
||||
256
examples/cuda-fanout/plans/form-D-clifford-stabilizer.md
Normal file
256
examples/cuda-fanout/plans/form-D-clifford-stabilizer.md
Normal file
|
|
@ -0,0 +1,256 @@
|
|||
# Form D — `cuda-clifford-stabilizer` implementation plan
|
||||
|
||||
You will not build this binary yet. This document gives you (fox) a
|
||||
fully-specified runway you can hand to a CUDA + quantum-sim engineer or
|
||||
to a future blackops session. Catalog entry lives at
|
||||
`CATALOG.md` (Form D). Numbers cited here all carry URLs.
|
||||
|
||||
## 1. Reference selection — what you can fork vs what you must write
|
||||
|
||||
Three candidates surveyed 2026-06-05:
|
||||
|
||||
- **STABSim** (Garner, Liu, Wang, Stein, Li — UW lead, seangarn@uw.edu)
|
||||
arXiv 2507.03092 https://arxiv.org/abs/2507.03092 . Warp-level CUDA
|
||||
primitives, scales better than CPU on QEC workloads. **No public
|
||||
repo.** GitHub search for `stabsim cuda` returns zero results
|
||||
https://github.com/search?q=stabsim+cuda&type=repositories . Treat as
|
||||
reference design only. Email seangarn@uw.edu before sinking weeks
|
||||
into a clean-room port. Our CATALOG's 186× figure derives from the
|
||||
equivalence-checking benchmark — re-extract once you have full PDF
|
||||
in hand.
|
||||
|
||||
- **Qimax** (Vu Tuan Hai et al, NAIST) arXiv 2505.03307
|
||||
https://arxiv.org/abs/2505.03307 . Extended-stabilizer on multi-core
|
||||
/ GPU; compares vs Qiskit & PennyLane, **not vs Stim**. No GitHub
|
||||
on arXiv landing. A re-implementation matching Qimax does not yet
|
||||
beat our CPU floor.
|
||||
|
||||
- **Clean-room Aaronson-Gottesman tableau on CUDA**
|
||||
https://arxiv.org/abs/quant-ph/0406196 . ~500 LoC CUDA for a binary
|
||||
symplectic tableau (2n × 2n bits + sign), gates H/S/CNOT/X/Z + M —
|
||||
every Clifford gate boils down to row-XOR + sign twiddle. **Our
|
||||
recommended target** because (a) you own our wire format, (b) STABSim
|
||||
& Qimax did not release source, (c) Aaronson-Gottesman runs at known
|
||||
O(n²) per gate, maps cleanly onto a CUDA warp.
|
||||
|
||||
Recommendation: write our own, citing STABSim's per-warp parallelism
|
||||
as inspiration. Re-evaluate STABSim source if UW responds.
|
||||
|
||||
## 2. Circuit format — translate, do not adopt
|
||||
|
||||
Our circuit lives as Phase B lumbda Scheme & emits to QECCOPS1
|
||||
`ops.bin` (op kinds in `ecdsa/cuda/sim.h:21-38`). Stim's `.stim` text
|
||||
format is the de-facto wire for stabilizer circuits
|
||||
(https://github.com/quantumlib/Stim , gate names `H S CX CZ SWAP R M
|
||||
TICK`).
|
||||
|
||||
Pick **(b)** — `demo_stab_sim_ops.cu` accepts our existing QECCOPS1
|
||||
`ops.bin` natively, treats Clifford ops as primary. Reasons:
|
||||
|
||||
- `ops.bin` already exists, generated by every lumbda tier; a `.stim`
|
||||
translator adds a redundant emission path.
|
||||
- Op kinds `OP_X, OP_Z, OP_CX, OP_CZ, OP_SWAP, OP_R, OP_HMR` map 1:1
|
||||
onto Aaronson-Gottesman tableau ops. `OP_CCX, OP_CCZ` route to our
|
||||
Toffoli handler (§3).
|
||||
- Cross-validation against Stim CPU still happens via a one-shot
|
||||
`emit-stim-circuit.lsp` test utility under `ecdsa/tests/` — small
|
||||
circuits only, byte-equivalence spot checks, never our production
|
||||
wire.
|
||||
|
||||
## 3. Toffoli problem — pick option (b), score symbolically
|
||||
|
||||
Stabilizer formalism handles Clifford only. Our Roetteler 12-step
|
||||
point-add (`ecdsa/lumbda/point-add-real.lsp`) carries both Clifford &
|
||||
Toffoli. Our CUDA sim already tracks them separately:
|
||||
`ecdsa/cuda/sim_gpu.cu:155` counts `OP_CCZ / OP_CCX` as Toffoli, all
|
||||
others as Clifford.
|
||||
|
||||
Three options surfaced in our brief; you pick **(b)**:
|
||||
|
||||
- **(a) Decompose each CCX into 7T.** Inflates op count ~7× per
|
||||
Toffoli, requires magic-state tracking. Wrong for a scoring sim.
|
||||
- **(b) Run Clifford fragment only on GPU; fold Toffoli count from
|
||||
circuit structure.** Our score is a *count*, not a measurement
|
||||
outcome. Toffoli ops are symbolic — you tally them at circuit-emit
|
||||
time, skip them in our stabilizer kernel. GPU lane handles only
|
||||
Clifford ops; CPU emits Σ-Toffoli alongside as a static circuit
|
||||
property. **This is sufficient for our metric.** Speedup numbers in
|
||||
CATALOG (186× equivalence-checking) come from Clifford-only
|
||||
benchmarks, so this matches what published work measures.
|
||||
- **(c) Extended stabilizer / magic-state sim.** Qimax territory.
|
||||
Slower & more code; only revisit if our scoring metric changes to
|
||||
require sampled outcomes through Toffoli gates.
|
||||
|
||||
Phase B point-add Toffoli fraction (from existing kernel counters):
|
||||
roughly half of all gates per circuit are CCX / CCZ at current ancilla
|
||||
widths. Confirm exact ratio by running `demo_ops` against
|
||||
`point-add.bin` — GPU counters at `sim_gpu.cu:238-239` emit
|
||||
`cliff_out` & `toff_out` per batch. Record this ratio before you sink
|
||||
time into Form D: if Toffoli > 90% of ops, a Clifford-only sim
|
||||
accelerates only 10% of work & a 186× fragment win yields 1.1× overall
|
||||
(Amdahl).
|
||||
|
||||
## 4. Wire protocol — BSHK reuse
|
||||
|
||||
Reuse the BSHK / BSHR framing already wired through `gpu-worker.lsp`
|
||||
(handler `handle-binary-shake` at lines 253-267). Layout:
|
||||
|
||||
- Request: `BSHK | u32 op_id=0x02 | u32 n_qubits | u32 n_shots |
|
||||
u32 circuit_blob_size | circuit_blob[N]`
|
||||
- circuit_blob: our existing QECCOPS1 `ops.bin` bytes, no translation.
|
||||
- Response: `BSHR | u32 status | u32 n_shots |
|
||||
samples[n_shots * ceil(n_qubits/8)]` bit-packed measurement outcomes.
|
||||
|
||||
Confirm match: STABSim & Qimax did not publish wire schemas, so you
|
||||
own this protocol fully — design freedom, no compatibility tax.
|
||||
`op_id` discriminator lets the daemon serve both `cuda-sim-ops-bin`
|
||||
(kickmix, `op_id=0x01`) and `cuda-clifford-stabilizer`
|
||||
(`op_id=0x02`) without renegotiating magic bytes.
|
||||
|
||||
## 5. `gpu-worker.lsp` handler sketch
|
||||
|
||||
Mirror `handle-cuda-sim-ops-bin` (lines 187-233). Pseudocode:
|
||||
|
||||
```scheme
|
||||
(define *binary-stab-sim*
|
||||
(or (get-environment-variable "STAB_SIM")
|
||||
"/home/fox/git/www.foxhop.net/ecdsa/cuda/demo_stab_sim_ops"))
|
||||
|
||||
(define (handle-cuda-clifford-stabilizer args)
|
||||
(let* ((ops-path (car args))
|
||||
(n-qubits (car (cdr args)))
|
||||
(n-shots (car (cdr (cdr args))))
|
||||
(portal (gensym-path "/tmp/bend-stab" ".portal"))
|
||||
(t-start (current-time-ms))
|
||||
(argv (list ops-path
|
||||
(number->string n-qubits)
|
||||
(number->string n-shots)
|
||||
"--portal" portal))
|
||||
(daemon (cdr (assoc 'cuda-clifford-stabilizer *daemons*))))
|
||||
(daemon-process daemon ops-path portal #t)
|
||||
;; portal carries (cuda-stab-result (samples blob) (sigma-clifford k)
|
||||
;; (sigma-toffoli k) (timing-ms ...))
|
||||
;; same shape as cuda-sim-ops-bin, parse via existing portal-find-number.
|
||||
...))
|
||||
```
|
||||
|
||||
Dispatch line in `handle-request`:
|
||||
|
||||
```scheme
|
||||
((eq? op 'cuda-clifford-stabilizer)
|
||||
(handle-cuda-clifford-stabilizer args))
|
||||
```
|
||||
|
||||
Register daemon at startup:
|
||||
|
||||
```scheme
|
||||
(register-daemon! 'cuda-clifford-stabilizer *binary-stab-sim*)
|
||||
```
|
||||
|
||||
## 6. Daemon protocol
|
||||
|
||||
Mirror `shake256-fanout --daemon` (called via `start-daemon` line 62).
|
||||
On stdin one line per request, on stdout one line per response.
|
||||
Two modes — text (`process`) for dev, binary (`process-bin`) for hot
|
||||
path:
|
||||
|
||||
```
|
||||
> process-bin /tmp/in.bin /tmp/out.bin\n
|
||||
< done\n (or `error <message>\n`)
|
||||
```
|
||||
|
||||
On start, daemon prints `ready\n` after CUDA context init &
|
||||
tableau-row buffer allocation. CUDA context warm-up pays once per
|
||||
worker lifetime.
|
||||
|
||||
## 7. Build recipe
|
||||
|
||||
Add to `ecdsa/cuda/Makefile`:
|
||||
|
||||
```make
|
||||
demo_stab_sim_ops: stab_sim_gpu.o ops_loader.o main_stab_sim.o
|
||||
nvcc -O3 -arch=sm_80 -o $@ $^
|
||||
|
||||
stab_sim_gpu.o: stab_sim_gpu.cu
|
||||
nvcc -O3 -arch=sm_80 -c $< -o $@
|
||||
```
|
||||
|
||||
`arch=sm_80` covers RTX 3090 (sm_86) & RTX 4090 (sm_89) via JIT;
|
||||
hard-code at deploy time if you measure a perf loss.
|
||||
|
||||
Source files to write fresh:
|
||||
|
||||
- `stab_sim_gpu.cu` — Aaronson-Gottesman tableau kernel, one CUDA
|
||||
block per circuit (= per candidate), threads cooperate on the
|
||||
tableau rows. Reference: quant-ph/0406196 §3 update rules.
|
||||
https://arxiv.org/abs/quant-ph/0406196
|
||||
- `main_stab_sim.c` — daemon loop, ops.bin load, dispatch.
|
||||
- Reuse `ops_loader.c` from existing simulator (already parses
|
||||
QECCOPS1).
|
||||
|
||||
## 8. Validation plan
|
||||
|
||||
Two axes:
|
||||
|
||||
- **Σ-Clifford equivalence vs Stim CPU.** Generate a 16-qubit
|
||||
toy circuit covering every supported gate; run our sim on
|
||||
CPU+GPU; emit the same circuit through
|
||||
`emit-stim-circuit.lsp` (one-shot translator, test-only); run
|
||||
Stim CPU via `stim sample --in toy.stim` (Stim install on
|
||||
permacomputer fleet, not on host). Σ-Clifford & measurement
|
||||
outcomes must match byte-for-byte over 1024 shots seeded
|
||||
identically.
|
||||
|
||||
- **Per-candidate axis flip benchmark.** Run N independent candidate
|
||||
circuits as N GPU blocks (= N lanes). Compare:
|
||||
- existing `demo_ops` per-shot: N candidates × 8192 shots each = 8192N
|
||||
threads, one thread per shot of a single candidate;
|
||||
- new `demo_stab_sim_ops` per-candidate: N candidates × 1 lane each,
|
||||
each lane runs its candidate to completion serially in tableau.
|
||||
At N=256 candidates expect Stim CPU to take ~2 s/candidate (512 s
|
||||
total); Form D target on RTX 3090 ~2.7 s total based on a 186×
|
||||
scaling factor. Record actual numbers; if our ratio < 30× you
|
||||
surface our crossover N on CATALOG.
|
||||
|
||||
Bench target: a 16-qubit, 1000-Clifford-gate test circuit (small
|
||||
enough to avoid memory blowup on first run; large enough that
|
||||
kernel-launch overhead amortizes).
|
||||
|
||||
## 9. Risks & open questions
|
||||
|
||||
- **STABSim source.** No public repo. Either email seangarn@uw.edu &
|
||||
wait, or build clean-room from quant-ph/0406196. Clean-room is our
|
||||
default path. Risk: clean-room reaches Aaronson-Gottesman baseline,
|
||||
not STABSim's published wins (likely 10-30× over CPU, not 186×).
|
||||
- **Per-candidate axis vs per-shot axis.** STABSim hints at warp-level
|
||||
parallelism without specifying whether warps map to circuits or
|
||||
shots. Read PDF §III before committing — our axis-flip thesis
|
||||
depends on per-circuit being the right grain.
|
||||
- **Toffoli ratio.** If point-add ends up >90% Toffoli, Form D wins
|
||||
only 1.1× overall (Amdahl). Measure Toffoli fraction before build.
|
||||
- **Memory.** A 16-qubit tableau is ~1 KB per candidate; a 256-qubit
|
||||
point-add fragment is 128 KB per candidate. N=1024 × 128 KB = 128
|
||||
MB GPU state — fits. Cap N×state at 8 GB on 3090.
|
||||
- **Stim install footprint.** Validation needs Stim on fleet;
|
||||
`pip install stim`. Test-only dep, not a worker dep.
|
||||
|
||||
## 10. Effort estimate
|
||||
|
||||
For one experienced CUDA + quantum-sim engineer (assumes prior
|
||||
Aaronson-Gottesman familiarity):
|
||||
|
||||
- Day 1: read STABSim PDF, read Aaronson-Gottesman paper, run
|
||||
existing `demo_ops` to measure Toffoli ratio, decide go/no-go.
|
||||
- Days 2-4: `stab_sim_gpu.cu` kernel — tableau rep, X/Z/H/S/CX/M
|
||||
update rules, warp-level row-XOR primitive.
|
||||
- Day 5: `main_stab_sim.c` daemon loop, BSHK plumbing.
|
||||
- Day 6: `gpu-worker.lsp` handler, daemon registration, smoke run.
|
||||
- Days 7-8: validation vs Stim CPU on 16-qubit toy, fix divergence.
|
||||
- Days 9-10: per-candidate benchmark, record numbers, update
|
||||
CATALOG.
|
||||
|
||||
**Total: ~10 working days.** Halve to 5 if STABSim source surfaces;
|
||||
double to 20 if Toffoli folding turns out insufficient & you must add
|
||||
extended-stabilizer (option c) support.
|
||||
|
||||
You decide go/no-go after Day 1 measurement.
|
||||
Loading…
Add table
Add a link
Reference in a new issue