Closes the loop on docker-based repro: builds a real Elixir Mix release
(with :ssl included) containing one GenServer registered as
Elixir.Wallet.Bridge, runs it under TLS dist with the production
inet_tls.conf shape, hits it from erldistpy.Node.call.
Passes cleanly with erldistpy 0.1.7. So the production portal failure
('peer closed after 0/4 bytes' against portal@unsandbox.com) isn't
reproducible in isolation, even with every dimension matched (Mix
release, OTP 25, TLS dist with permissive verify_fun + tls1.2/1.3 +
secure_renegotiate, Elixir GenServer, long FQDN node names).
The bug must be in interaction with portal's other dist connections or
its larger supervisor tree — beyond what we can repro without standing
up the full portal app locally.
Iterates on the docker-based repro infrastructure while diagnosing the
portal@unsandbox.com 'peer closed after 0/4 bytes' failure:
- _write_ssl_config_portal_match() emits an ssl_dist.config with the
exact options portal runs: permissive verify_fun (accepts bad_cert),
versions [tlsv1.3, tlsv1.2], secure_renegotiate, server_name_indication
disabled. Same shape as /opt/unsandbox/certs/inet_tls.conf.
- Elixir docker peer now boots with --name (long FQDN, like prod)
instead of --sname (short). Production portal is portal@unsandbox.com
so the dist driver's routing path is different from short-name peers.
- Elixir GenTarget now logs init + handle_call + handle_info. Fixture
redirects container stdout to a file; test prints it on
pass-or-fail so we can see whether the peer received our gen_call.
- elixir image bumped to 1.16-otp-25 (portal runs OTP 25.3.2.5 / erts
13.2.2.5, NOT OTP 26 — discovered via the portal release's bundled
erts version).
Despite matching every dimension I can find (OTP version, TLS dist
config, Elixir GenServer wrapping, long node names), the test still
passes locally — so the production failure is something specific to
the live portal beam state, not a general protocol or version issue.
Reproduces the production failure locally so we can iterate in seconds
instead of waiting on deploy cycles across 3 repos and PyPI.
Existing test_node.py / test_tls.py use the system Erlang which is
OTP 24 on most dev boxes (Ubuntu 22.04 default) — silently masks
flag-negotiation and protocol bugs that only surface against OTP 25+.
Adds four docker-backed tests:
- plain dist + hand-coded receive (works on all OTP versions)
- TLS dist + hand-coded receive (works on all OTP versions)
- TLS dist + Elixir GenServer (FAILS on OTP 25, passes on 26)
The last one is the minimum repro of the portal@unsandbox.com failure
mode. Once a fix lands, that test goes green and we know the
production smoke test will too.
Uses --network host to share the host's epmd (Linux-only). Docker mounts
from $HOME/.erldistpy-test/ because snap-confined docker can't see /tmp.
Tests skip cleanly when docker isn't installed.
Against real-world OTP 26 peers the v6 handshake "succeeded" but the
first REG_SEND silently dropped on the peer side — peer accepted the
connection then closed the link with no bytes when we tried to call a
registered process. Hit during MPS↔portal wallet RPC smoke test.
Root cause: OTP 25+ requires DFLAG_MANDATORY_25_DIGEST to be present
in our advertised flag set. The digest is the hash of the OTP-25
mandatory flag set; without it the peer's dist driver loses confidence
in the negotiation and drops messages from us without surfacing an
error.
Adds DFLAG_MANDATORY_25_DIGEST to DEFAULT_FLAGS. Also extends
_decode_message to accept both legacy pass-through (0x70 ...) and
dist-header framing (0x83 0x44 0x00 ...) on receive — modern OTP may
send dist-headed messages even when we didn't negotiate
DFLAG_DIST_HDR_ATOM_CACHE. Fragments (0x83 0x45 / 0x83 0x46) still
TODO; we surface a clear ChannelError instead of silent corruption.
Send side still uses pass-through framing — we don't yet implement
the atom-cache encode/decode that DFLAG_DIST_HDR_ATOM_CACHE would
require. Peer routes our pass-through sends without issue.
122/122 tests pass including live integration against a local Erlang
node and the TLS dist suite.
make_dist_tls_context() builds an ssl.SSLContext tuned for OTP defaults
(verify_peer, mTLS, TLSv1.2 minimum). Node accepts tls_context= and
wraps the TCP socket in TLS before the v6 handshake runs.
Critical quirk found by experimentation: inet_tls_dist uses {packet, 4}
on the SSL socket during the handshake. Plain inet_tcp_dist uses
{packet, 2} for handshake then switches to {packet, 4} post-nodeup.
handshake() now takes a frame_size= kwarg (2 or 4); Node auto-selects 4
whenever tls_context is supplied.
Cert requirements (found by experimentation against Erlang E2E):
- CA cert with basicConstraints CA:TRUE
- Leaf certs with SAN including the dist hostname (and localhost)
- extendedKeyUsage covering both serverAuth and clientAuth
Tests:
- make_dist_tls_context unit tests
- Live: spawn erl -proto_dist inet_tls with SAN-bearing certs,
Node.call(gen_target, {ping, 99}) round-trips through the tunnel
- Live negative: plaintext connection to TLS-only peer must fail
- Live negative: client cert from a different CA must fail
115 tests green across 5 consecutive runs, lint clean.
Node wraps EPMD lookup + handshake + Channel into a single client
object. Constructor eagerly opens the dist connection; call() runs the
synchronous $gen_call protocol against a registered name on the peer:
caller -> {'$gen_call', {FromPid, Ref}, Request} (REG_SEND)
server -> {Ref, Reply} (SEND)
Synthesized FromPid and a Node-lifetime Ref counter route replies back
to us; mismatched Ref or unexpected control op raises CallProtocolError.
Reply timeout raises CallTimeout (also covers Erlang's silent-drop case
when the registered name doesn't exist).
Tests against an erl peer running a $gen_call-aware loop:
- {ping, X} -> {pong, X}
- {add, A, B} -> {ok, A + B}
- five sequential calls with monotonically increasing Refs
- server error response surfaces as Python tuple
- slow responder triggers CallTimeout
- unknown registered name surfaces as CallTimeout
- ref uniqueness across 100 synthesized refs
111 tests green across 10 consecutive runs, lint clean.
Channel wraps the post-handshake socket and carries 4-byte length-
prefixed distribution messages: pass-through byte ('p') + ETF control
tuple + optional payload term.
API surface:
send_raw / recv_raw -- raw 4-byte framed bytes, empty == net_tick
send_tick -- send keepalive frame
send_control / recv_message -- structured control + payload
send_reg_send -- helper for the REG_SEND case (FromPid,
registered name, payload)
recv_message() transparently skips inbound ticks; callers wanting tick
awareness use recv_raw().
etf.decode_term(data, offset) exposed as a streaming decoder so the
channel can read control + payload back-to-back from one frame body.
Tests:
- pure encode/decode round-trips
- socketpair tests for framing, ticks, helper signatures
- live end-to-end against an erl node with a registered echo process:
EPMD -> handshake -> REG_SEND -> recv reply, payload matches
- boot script writes a /tmp ready-flag after registering `echo`;
fixture waits for both EPMD registration AND the flag to dodge
the race where EPMD registers the node before -eval runs
101 tests green, lint clean.
handshake(sock, our_name=..., cookie=...) drives the OTP 23+ dance over
a 2-byte length-prefixed frame stream:
send_name (N) client -> server
recv_status (s) server -> client
recv_challenge (N) server -> client
challenge_reply (r) client -> server
challenge_ack (a) server -> client
Cookie digest formula md5(cookie ++ integer_to_list(challenge)) was
cross-checked against erlang:md5/1 output as a test reference.
Distribution flags in erldistpy/flags.py advertise the minimum useful
set: extended refs/pids, new fun tags, utf8 atoms, maps, big creation,
v6 handshake, unlink id, v4 node containers.
86 tests green: frame builders + parsers as pure functions, digest
reference, full live handshake against `erl -sname -setcookie`, and a
wrong-cookie rejection test.
Newer SHA-256 digest (DFLAG_MANDATORY_25_DIGEST) deferred until a peer
requires it.
Synchronous TCP client for Erlang Port Mapper Daemon. One request type
(PORT_PLEASE2_REQ, tag 122), one response type (PORT2_RESP, tag 119).
Returns EpmdInfo dataclass or None if the node is not registered.
Tests run two layers:
- Unit tests against recorded byte streams captured from a real EPMD
answering for `erl -sname testnode` and for an unregistered name.
- Integration tests spawn `erl -sname erldistpy_itest` in a fixture
and verify lookup() returns the live port; skipped if erl or EPMD
are absent.
10 new tests, 68 total green, lint clean.
