russell@unturf.com
f11e310b24
Establishes the integration pattern for lumbda's future cuda primitive
across Python / C / asm tiers without dragging the CUDA toolchain into
lumbda's core build.
Shape: leaf binary that every tier spawns via its existing process-
spawn primitive & talks to through S-expression input + output
portals. Asm tier inherits via fork + execve syscalls; no libcudart
linkage; no DKMS dependency at lumbda build time.
Files:
shake256-fanout.cu self-contained CUDA SHAKE256 fan-out, Keccak
permutation derived from FIPS 202 reference
(tiny-sha3 lineage, CC0 → re-licensed AGPLv3)
Makefile nvcc build + make test + make bench
test_roundtrip.py validates output byte-identical to
hashlib.shake_256
bench.py device vs host throughput at N = 1k / 10k / 100k
lumbda-call.lsp reference Scheme wrapper showing the
(cuda-shake-fanout inputs out-bytes) API shape
lumbda's core would dispatch to per-tier
README.md full integration story, wire contract, the
three changes each tier needs (~20 LoC each),
generalization path for other CUDA primitives
Tested on 3090-ai (RTX 3090):
make test → PASS — 4 / 4 hashes byte-identical to hashlib.shake_256
Honest bench (32-byte inputs):
N host (Python hashlib) device (kernel launch dominated)
1,000 0.6 ms 188.1 ms
10,000 5.9 ms 195.7 ms
100,000 58.7 ms 313.8 ms
Useful primitive when inputs are larger (KB+) or N reaches millions;
honest about the launch-overhead break-even point. This is the
reference, not the win — the win is locking the API shape so each
tier registers under one stable name.
Provenance: extracted as the generic pattern from
~/git/www.foxhop.net/ecdsa/cuda/sim_gpu.cu where on-device SHAKE
delivered 2.6× memory compression for batched reversible-circuit
simulation. Re-shipping the primitive back to the lumbda repo so the
ecosystem inherits the work.
84 lines
3.8 KiB
Text
84 lines
3.8 KiB
Text
;;; lumbda-call.lsp — calling shake256-fanout from any lumbda tier.
|
|
;;;
|
|
;;; This is the API SHAPE lumbda's core would expose as a primitive.
|
|
;;; Each tier already has spawn-process + portal I/O primitives; the
|
|
;;; cuda fan-out primitive is a thin Scheme wrapper that composes them.
|
|
;;;
|
|
;;; Once locked, each tier (Python / C / asm) registers the same name
|
|
;;; in its primitive dispatch table. Asm tier uses fork + execve +
|
|
;;; portal-write syscalls — no libcudart linkage needed. Python &
|
|
;;; C tiers can use the same shape via subprocess / fork respectively,
|
|
;;; or graduate to a true FFI later.
|
|
;;;
|
|
;;; Usage:
|
|
;;;
|
|
;;; (cuda-shake-fanout
|
|
;;; '("00" "01" "deadbeef") ; list of hex-string inputs
|
|
;;; 32) ; output bytes per input
|
|
;;; ⇒ '("0xhash0" "0xhash1" "0xhash2")
|
|
|
|
(define *cuda-shake-fanout-binary*
|
|
;; Override per host; reference points at the example dir.
|
|
"/home/fox/git/lumbda/examples/cuda-fanout/shake256-fanout")
|
|
|
|
(define (with-temp-files names body)
|
|
"Allocate gensym'd temp file paths, invoke body on them, cleanup
|
|
regardless of exit path. Each tier should implement gensym +
|
|
delete-file in its stdlib."
|
|
(let* ((paths (map (lambda (n) (string-append "/tmp/cf-" n "-"
|
|
(number->string (current-time-ms))))
|
|
names))
|
|
(result (apply body paths)))
|
|
(for-each delete-file paths)
|
|
result))
|
|
|
|
(define (write-input-portal! path inputs output-bytes)
|
|
(let ((port (open-output-file path)))
|
|
(display "(cuda-shake-fanout\n" port)
|
|
(display " (output-bytes " port) (display output-bytes port) (display ")\n" port)
|
|
(display " (inputs\n" port)
|
|
(for-each (lambda (hex)
|
|
(display " \"" port) (display hex port) (display "\"\n" port))
|
|
inputs)
|
|
(display "))\n" port)
|
|
(close-port port)))
|
|
|
|
(define (read-hashes-portal path)
|
|
;; Use sim.lsp-style portal reader if available; fall back to text scan.
|
|
(let* ((src (file->string path))
|
|
;; Crude hex-string extractor — production code should use a real
|
|
;; parser. The portal format is small & predictable so this works
|
|
;; for the reference example.
|
|
(after-hashes (substring src
|
|
(+ (string-search "(hashes" src) 7)
|
|
(string-length src))))
|
|
(let loop ((rest after-hashes) (acc '()))
|
|
(let ((open (string-search "\"" rest)))
|
|
(if (not open) (reverse acc)
|
|
(let* ((tail (substring rest (+ open 1) (string-length rest)))
|
|
(close (string-search "\"" tail)))
|
|
(if (not close) (reverse acc)
|
|
(loop (substring tail (+ close 1) (string-length tail))
|
|
(cons (substring tail 0 close) acc)))))))))
|
|
|
|
(define (cuda-shake-fanout inputs output-bytes)
|
|
"Hash each input on its own CUDA thread via the reference fan-out
|
|
binary. Inputs are hex strings; output is a list of hex hash strings."
|
|
(with-temp-files '("in.portal" "out.portal")
|
|
(lambda (in-path out-path)
|
|
(write-input-portal! in-path inputs output-bytes)
|
|
;; spawn-process is each tier's process-spawn primitive name —
|
|
;; substitute when lumbda's core dispatches under a stable name.
|
|
(spawn-process *cuda-shake-fanout-binary* (list in-path out-path))
|
|
(read-hashes-portal out-path))))
|
|
|
|
;;; ── demo ──────────────────────────────────────────────────────
|
|
|
|
(define (demo)
|
|
(let ((hashes (cuda-shake-fanout '("00" "01" "deadbeef") 32)))
|
|
(display "got ") (display (length hashes)) (display " hashes:\n")
|
|
(for-each (lambda (h)
|
|
(display " ") (display h) (newline))
|
|
hashes)))
|
|
|
|
;; (demo) ;; uncomment to run; needs spawn-process + file primitives
|