whitepaper: expand §7.5 with 7.5.1 on kernel-entropy-on-one-side flow
Previous revision mentioned (random-seed-from-os!) in one sentence; this treatment does not match the weight fox assigned — 'a random seed from os on at least one side is very important.' New sub-subsection 7.5.1 'Kernel Entropy on One Side' spells out: - why hand-rolled /dev/urandom seeding is a reproducibility-defect farm - why we picked /dev/urandom over /dev/random and getrandom(2) - the real-world Machine A -> portal -> Machine B flow as concrete Scheme - the audit property: a captured run-id replays the trajectory exactly - kernel entropy enters the system once, under user control, then never again — every resume reads the same stream off the portal Ticket: docs/tickets/0002-os-entropy-seed.md
This commit is contained in:
parent
4960381c67
commit
fc5bc7fb79
3 changed files with 1256 additions and 921 deletions
|
|
@ -1833,9 +1833,29 @@ bi_portal_resume:
|
|||
<li><p><strong>S-expression portal:</strong> one line — <span class="docutils literal"><span class="pre">(random-state!</span> '(w0_lo w0_hi …))</span> — which every tier already evaluates. No format change. The language really is the wire format.</p></li>
|
||||
</ul>
|
||||
<p>Builtins shared across all three: <span class="docutils literal"><span class="pre">(random-seed!</span> k)</span>, <span class="docutils literal"><span class="pre">(random-int</span> n)</span>, <span class="docutils literal"><span class="pre">(random-state)</span></span>, <span class="docutils literal"><span class="pre">(random-state!</span> lst)</span>. Python and C also export <span class="docutils literal">(random)</span> returning a float in <span class="docutils literal">[0, 1)</span>; the asm tier has no floats, so this one builtin is omitted by construction. All three tiers seed with <span class="docutils literal">k = 0</span> at process startup, so a program that never calls <span class="docutils literal"><span class="pre">random-seed!</span></span> still gets a non-zero, deterministic stream.</p>
|
||||
<p>A fifth shared builtin — <span class="docutils literal"><span class="pre">(random-seed-from-os!)</span></span> — reads 8 bytes from <span class="docutils literal">/dev/urandom</span> and seeds xoshiro256** with them, so the typical real-world flow <em>(Machine A seeds from kernel entropy → runs simulation → portal-saves → Machine B resumes the same stream bit-for-bit)</em> is one call away. Determinism stays the default; kernel entropy is opt-in and fails loud if <span class="docutils literal">/dev/urandom</span> is unavailable. Ticket: <span class="docutils literal"><span class="pre">docs/tickets/0002-os-entropy-seed.md</span></span>.</p>
|
||||
<p>Verification: <span class="docutils literal"><span class="pre">tests/portal-cross-test.sh</span></span> runs a 3×3 producer × consumer matrix where each tier writes a portal file capturing the RNG state, every other tier reads it, draws the next 5 values, and the values must match bit-for-bit against a baseline the producer also wrote into the file. All 9 cells green. The baseline for seed=42 is <span class="docutils literal">558742 543102 559009 124193 317476 …</span> — identical in Python, C, and asm, identical after any portal round-trip through any format.</p>
|
||||
<p><strong>Ticket:</strong> <span class="docutils literal"><span class="pre">docs/tickets/0001-portal-rng.md</span></span>. Contributed by <strong>Zoë Trout</strong>, whose question — <em>does the portal tech keep the random list seed which will allow transferring random entropy between processes when continuing a simulation?</em> — surfaced a gap we had not noticed. The earlier claim <em>portal = portable machine state</em> carried a silent asterisk: <em>deterministic code only</em>. That asterisk is now gone.</p>
|
||||
<section id="kernel-entropy-on-one-side">
|
||||
<h4>7.5.1 Kernel Entropy on One Side</h4>
|
||||
<p>Pure determinism is the feature, not the whole story. A program that <em>wants</em> a different stream every run — a simulation, a secret-generation step, a shuffler — has to seed itself from outside. Without a helper, every user hand-rolls one, and hand-rolled entropy is where half the reproducibility defects in science live: wrong endianness, short-read-not-handled, FD leaked, nondeterministic seed baked into logs you can't re-derive.</p>
|
||||
<p>A fifth shared builtin closes this gap: <span class="docutils literal"><span class="pre">(random-seed-from-os!)</span></span> reads exactly 8 bytes from <span class="docutils literal">/dev/urandom</span> as a little-endian <span class="docutils literal">u64</span>, passes the value to the same <span class="docutils literal">rng_seed</span> path as <span class="docutils literal"><span class="pre">(random-seed!)</span></span>, and returns void. Failure (<span class="docutils literal">/dev/urandom</span> missing or short read) raises loud — <span class="docutils literal">LispErr</span> in Python and C, stderr + <span class="docutils literal">exit(1)</span> in asm — with no silent fallback to a weak seed. The default startup seed stays <span class="docutils literal">k = 0</span>; kernel entropy is opt-in, so ticket 0001's portal-reproducibility contract is unchanged.</p>
|
||||
<p><span class="docutils literal">/dev/urandom</span> is the intentional pick over <span class="docutils literal">/dev/random</span>: <span class="docutils literal">man 4 urandom</span> confirms it is cryptographically strong on any Linux kernel after boot, and unlike <span class="docutils literal">/dev/random</span> it never blocks waiting for "enough" entropy. We also do not use <span class="docutils literal">getrandom(2)</span> — portability to older kernels matters more than saving one <span class="docutils literal">open</span> + <span class="docutils literal">read</span> + <span class="docutils literal">close</span>.</p>
|
||||
<p><strong>The typical real-world flow</strong> — the one fox called out as <em>"very important"</em> — uses kernel entropy on <strong>one</strong> side only, then lets the portal carry a bit-identical continuation to every other process that reloads it:</p>
|
||||
<pre class="code scheme literal-block"><code><span class="comment single">;; Machine A — the fresh-run side.</span><span class="whitespace">
|
||||
</span><span class="punctuation">(</span><span class="name function">random-seed-from-os!</span><span class="punctuation">)</span><span class="whitespace"> </span><span class="comment single">; pull 8 bytes of kernel entropy</span><span class="whitespace">
|
||||
</span><span class="punctuation">(</span><span class="keyword">define</span><span class="whitespace"> </span><span class="name variable">run-id</span><span class="whitespace"> </span><span class="punctuation">(</span><span class="name function">random-state</span><span class="punctuation">))</span><span class="whitespace"> </span><span class="comment single">; snapshot so we can replay later</span><span class="whitespace">
|
||||
</span><span class="punctuation">(</span><span class="name function">run-simulation</span><span class="whitespace"> </span><span class="literal number integer">1000000</span><span class="punctuation">)</span><span class="whitespace"> </span><span class="comment single">; draws millions of randoms</span><span class="whitespace">
|
||||
</span><span class="punctuation">(</span><span class="name function">portal-save</span><span class="whitespace"> </span><span class="literal string">"state.portal"</span><span class="punctuation">)</span><span class="whitespace"> </span><span class="comment single">; captures xoshiro256** state + heap</span><span class="whitespace">
|
||||
|
||||
</span><span class="comment single">;; Machine B — the resume side, possibly days later, different box.</span><span class="whitespace">
|
||||
</span><span class="punctuation">(</span><span class="name function">portal-resume</span><span class="whitespace"> </span><span class="literal string">"state.portal"</span><span class="punctuation">)</span><span class="whitespace"> </span><span class="comment single">; xoshiro256** state restored byte-for-byte</span><span class="whitespace">
|
||||
</span><span class="punctuation">(</span><span class="name function">run-simulation</span><span class="whitespace"> </span><span class="literal number integer">1000000</span><span class="punctuation">)</span><span class="whitespace"> </span><span class="comment single">; same stream from here, bit-for-bit</span><span class="whitespace">
|
||||
|
||||
</span><span class="comment single">;; Audit (any machine, any tier) — replay the original from run-id.</span><span class="whitespace">
|
||||
</span><span class="punctuation">(</span><span class="name function">random-state!</span><span class="whitespace"> </span><span class="name variable">run-id</span><span class="punctuation">)</span><span class="whitespace">
|
||||
</span><span class="punctuation">(</span><span class="name function">run-simulation</span><span class="whitespace"> </span><span class="literal number integer">2000000</span><span class="punctuation">)</span><span class="whitespace"> </span><span class="comment single">; reproduces A's entire trajectory</span></code></pre>
|
||||
<p>One call on one side. Kernel entropy enters the system exactly once, under the user's control, at the moment a new experiment begins. Every subsequent process reads the same stream off the portal without touching <span class="docutils literal">/dev/urandom</span> again — that is what makes the run <em>auditable</em>. An adversary who captures <span class="docutils literal">state.portal</span> can replay the simulation; a scientist who captures <span class="docutils literal"><span class="pre">run-id</span></span> up front can prove the run was not altered mid-flight. Ticket: <span class="docutils literal"><span class="pre">docs/tickets/0002-os-entropy-seed.md</span></span>.</p>
|
||||
</section>
|
||||
</section>
|
||||
<section id="cross-process-benchmarks">
|
||||
<h3>7.6 Cross-Process Benchmarks</h3>
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -805,12 +805,39 @@ Lumbda addresses this across all three tiers with one shared PRNG: **xoshiro256\
|
|||
|
||||
Builtins shared across all three: ``(random-seed! k)``, ``(random-int n)``, ``(random-state)``, ``(random-state! lst)``. Python and C also export ``(random)`` returning a float in ``[0, 1)``; the asm tier has no floats, so this one builtin is omitted by construction. All three tiers seed with ``k = 0`` at process startup, so a program that never calls ``random-seed!`` still gets a non-zero, deterministic stream.
|
||||
|
||||
A fifth shared builtin — ``(random-seed-from-os!)`` — reads 8 bytes from ``/dev/urandom`` and seeds xoshiro256\*\* with them, so the typical real-world flow *(Machine A seeds from kernel entropy → runs simulation → portal-saves → Machine B resumes the same stream bit-for-bit)* is one call away. Determinism stays the default; kernel entropy is opt-in and fails loud if ``/dev/urandom`` is unavailable. Ticket: ``docs/tickets/0002-os-entropy-seed.md``.
|
||||
|
||||
Verification: ``tests/portal-cross-test.sh`` runs a 3×3 producer × consumer matrix where each tier writes a portal file capturing the RNG state, every other tier reads it, draws the next 5 values, and the values must match bit-for-bit against a baseline the producer also wrote into the file. All 9 cells green. The baseline for seed=42 is ``558742 543102 559009 124193 317476 …`` — identical in Python, C, and asm, identical after any portal round-trip through any format.
|
||||
|
||||
**Ticket:** ``docs/tickets/0001-portal-rng.md``. Contributed by **Zoë Trout**, whose question — *does the portal tech keep the random list seed which will allow transferring random entropy between processes when continuing a simulation?* — surfaced a gap we had not noticed. The earlier claim *portal = portable machine state* carried a silent asterisk: *deterministic code only*. That asterisk is now gone.
|
||||
|
||||
7.5.1 Kernel Entropy on One Side
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Pure determinism is the feature, not the whole story. A program that *wants* a different stream every run — a simulation, a secret-generation step, a shuffler — has to seed itself from outside. Without a helper, every user hand-rolls one, and hand-rolled entropy is where half the reproducibility defects in science live: wrong endianness, short-read-not-handled, FD leaked, nondeterministic seed baked into logs you can't re-derive.
|
||||
|
||||
A fifth shared builtin closes this gap: ``(random-seed-from-os!)`` reads exactly 8 bytes from ``/dev/urandom`` as a little-endian ``u64``, passes the value to the same ``rng_seed`` path as ``(random-seed!)``, and returns void. Failure (``/dev/urandom`` missing or short read) raises loud — ``LispErr`` in Python and C, stderr + ``exit(1)`` in asm — with no silent fallback to a weak seed. The default startup seed stays ``k = 0``; kernel entropy is opt-in, so ticket 0001's portal-reproducibility contract is unchanged.
|
||||
|
||||
``/dev/urandom`` is the intentional pick over ``/dev/random``: ``man 4 urandom`` confirms it is cryptographically strong on any Linux kernel after boot, and unlike ``/dev/random`` it never blocks waiting for "enough" entropy. We also do not use ``getrandom(2)`` — portability to older kernels matters more than saving one ``open`` + ``read`` + ``close``.
|
||||
|
||||
**The typical real-world flow** — the one fox called out as *"very important"* — uses kernel entropy on **one** side only, then lets the portal carry a bit-identical continuation to every other process that reloads it:
|
||||
|
||||
.. code-block:: scheme
|
||||
|
||||
;; Machine A — the fresh-run side.
|
||||
(random-seed-from-os!) ; pull 8 bytes of kernel entropy
|
||||
(define run-id (random-state)) ; snapshot so we can replay later
|
||||
(run-simulation 1000000) ; draws millions of randoms
|
||||
(portal-save "state.portal") ; captures xoshiro256** state + heap
|
||||
|
||||
;; Machine B — the resume side, possibly days later, different box.
|
||||
(portal-resume "state.portal") ; xoshiro256** state restored byte-for-byte
|
||||
(run-simulation 1000000) ; same stream from here, bit-for-bit
|
||||
|
||||
;; Audit (any machine, any tier) — replay the original from run-id.
|
||||
(random-state! run-id)
|
||||
(run-simulation 2000000) ; reproduces A's entire trajectory
|
||||
|
||||
One call on one side. Kernel entropy enters the system exactly once, under the user's control, at the moment a new experiment begins. Every subsequent process reads the same stream off the portal without touching ``/dev/urandom`` again — that is what makes the run *auditable*. An adversary who captures ``state.portal`` can replay the simulation; a scientist who captures ``run-id`` up front can prove the run was not altered mid-flight. Ticket: ``docs/tickets/0002-os-entropy-seed.md``.
|
||||
|
||||
7.6 Cross-Process Benchmarks
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue