Closes the loop on docker-based repro: builds a real Elixir Mix release
(with :ssl included) containing one GenServer registered as
Elixir.Wallet.Bridge, runs it under TLS dist with the production
inet_tls.conf shape, hits it from erldistpy.Node.call.
Passes cleanly with erldistpy 0.1.7. So the production portal failure
('peer closed after 0/4 bytes' against portal@unsandbox.com) isn't
reproducible in isolation, even with every dimension matched (Mix
release, OTP 25, TLS dist with permissive verify_fun + tls1.2/1.3 +
secure_renegotiate, Elixir GenServer, long FQDN node names).
The bug must be in interaction with portal's other dist connections or
its larger supervisor tree — beyond what we can repro without standing
up the full portal app locally.
Iterates on the docker-based repro infrastructure while diagnosing the
portal@unsandbox.com 'peer closed after 0/4 bytes' failure:
- _write_ssl_config_portal_match() emits an ssl_dist.config with the
exact options portal runs: permissive verify_fun (accepts bad_cert),
versions [tlsv1.3, tlsv1.2], secure_renegotiate, server_name_indication
disabled. Same shape as /opt/unsandbox/certs/inet_tls.conf.
- Elixir docker peer now boots with --name (long FQDN, like prod)
instead of --sname (short). Production portal is portal@unsandbox.com
so the dist driver's routing path is different from short-name peers.
- Elixir GenTarget now logs init + handle_call + handle_info. Fixture
redirects container stdout to a file; test prints it on
pass-or-fail so we can see whether the peer received our gen_call.
- elixir image bumped to 1.16-otp-25 (portal runs OTP 25.3.2.5 / erts
13.2.2.5, NOT OTP 26 — discovered via the portal release's bundled
erts version).
Despite matching every dimension I can find (OTP version, TLS dist
config, Elixir GenServer wrapping, long node names), the test still
passes locally — so the production failure is something specific to
the live portal beam state, not a general protocol or version issue.
Reproduces the production failure locally so we can iterate in seconds
instead of waiting on deploy cycles across 3 repos and PyPI.
Existing test_node.py / test_tls.py use the system Erlang which is
OTP 24 on most dev boxes (Ubuntu 22.04 default) — silently masks
flag-negotiation and protocol bugs that only surface against OTP 25+.
Adds four docker-backed tests:
- plain dist + hand-coded receive (works on all OTP versions)
- TLS dist + hand-coded receive (works on all OTP versions)
- TLS dist + Elixir GenServer (FAILS on OTP 25, passes on 26)
The last one is the minimum repro of the portal@unsandbox.com failure
mode. Once a fix lands, that test goes green and we know the
production smoke test will too.
Uses --network host to share the host's epmd (Linux-only). Docker mounts
from $HOME/.erldistpy-test/ because snap-confined docker can't see /tmp.
Tests skip cleanly when docker isn't installed.
Against real-world OTP 26 peers the v6 handshake "succeeded" but the
first REG_SEND silently dropped on the peer side — peer accepted the
connection then closed the link with no bytes when we tried to call a
registered process. Hit during MPS↔portal wallet RPC smoke test.
Root cause: OTP 25+ requires DFLAG_MANDATORY_25_DIGEST to be present
in our advertised flag set. The digest is the hash of the OTP-25
mandatory flag set; without it the peer's dist driver loses confidence
in the negotiation and drops messages from us without surfacing an
error.
Adds DFLAG_MANDATORY_25_DIGEST to DEFAULT_FLAGS. Also extends
_decode_message to accept both legacy pass-through (0x70 ...) and
dist-header framing (0x83 0x44 0x00 ...) on receive — modern OTP may
send dist-headed messages even when we didn't negotiate
DFLAG_DIST_HDR_ATOM_CACHE. Fragments (0x83 0x45 / 0x83 0x46) still
TODO; we surface a clear ChannelError instead of silent corruption.
Send side still uses pass-through framing — we don't yet implement
the atom-cache encode/decode that DFLAG_DIST_HDR_ATOM_CACHE would
require. Peer routes our pass-through sends without issue.
122/122 tests pass including live integration against a local Erlang
node and the TLS dist suite.
After protecting the * tag pattern on the python/ group, the group-
scoped Protected TWINE_USERNAME/TWINE_PASSWORD vars inject into tag
pipelines. Twine 6+ sees them set and uses classic auth directly,
skipping Trusted Publishing (which can't work for self-hosted
git.unturf.com — see docs/PYPI-TRUSTED-PUBLISHING.md).
What this commit changes:
- .gitlab-ci.yml drops the --trusted-publishing flag (unrecognized
by the runner's twine anyway) — twine sees env vars and is happy
- pyproject.toml stays on the latest setuptools (no <77 cap needed
since we're not pinning twine<6 anymore)
- Diagnostic for TWINE_USERNAME/TWINE_PASSWORD presence stays so
future failures surface fast
Pipeline #40358 (v0.1.4) failed with the same OIDC error as 0.1.0:
TrustedPublishingFailure: Unable to retrieve an OIDC token from
the CI platform for trusted publishing GitLab: Environment
variable PYPI_ID_TOKEN not found
Twine 6's default is --trusted-publishing automatic, which ALWAYS
tries OIDC first when it detects GitLab CI env vars, regardless of
whether TWINE_USERNAME/TWINE_PASSWORD are set. Adding
'--trusted-publishing never' forces classic API token auth via env
vars (or .pypirc) and skips the OIDC dance entirely.
Plus a sanity-check that the group-scoped TWINE_USERNAME and
TWINE_PASSWORD vars actually landed in the env on this pipeline
('set' or 'MISSING' — never echoes the value). If they're MISSING
despite being configured at the python/ group level, the most likely
cause is the vars being marked Protected while the tag isn't a
protected ref (Settings -> Repository -> Protected tags).
Trusted Publishing OIDC is blocked for git.unturf.com — PyPI's GitLab
provider hardcodes the issuer to gitlab.com (no self-hosted instance
field in the Add publisher form). Until PyPI lights up self-hosted
GitLab support, classic API token auth is our path.
Auth source: TWINE_USERNAME=__token__ + TWINE_PASSWORD=<pypi-...> set
as group-level CI variables on python/. Both python/ago and
python/erldistpy inherit them automatically; engineering/ group gets
the same vars for make-post-sell + remarkbox.
What this commit changes:
- .gitlab-ci.yml pypi-twine stage drops the id_tokens block we
briefly tried (OIDC), uses latest twine (no <6 pin) — twine 6
reads TWINE_USERNAME/TWINE_PASSWORD env vars and prefers them
over the OIDC attempt
- pyproject.toml build-system requires drops the setuptools<77
cap (latest setuptools emits Metadata-Version 2.4 which twine 6
reads fine; locally verified)
- docs/PYPI-TRUSTED-PUBLISHING.md updated to note the issuer-
hardcoded blocker and the recipe to migrate later when it
unblocks (or if we mirror to gitlab.com)
Pending publisher registered on PyPI for python/erldistpy with
workflow filepath .gitlab-ci.yml — GitLab now mints a short-lived
OIDC ID token (audience=pypi) per pipeline, twine 6+ exchanges it
for a scoped upload token. No long-lived secret on the runner.
What this commit changes:
- .gitlab-ci.yml pypi-twine stage gains id_tokens: PYPI_ID_TOKEN
- pip install twine (no <6 pin) — twine 6 is OIDC-native
- pyproject.toml build-system requires drops the setuptools<77 cap
(latest setuptools emits Metadata-Version 2.4 which twine 6 reads
fine; locally verified)
After this tag publishes successfully, the same pattern goes to:
- ago (python/ago)
- make-post-sell (engineering/make-post-sell)
- remarkbox (engineering/remarkbox)
each gets a Trusted Publisher entry added on the PyPI project page,
then the id_tokens block lands in their CI.
Recipe lives at docs/PYPI-TRUSTED-PUBLISHING.md (updated to reflect
the new auth model is now live for erldistpy).
Pipeline #40349 (v0.1.2) failed because the test stage ran on the tag
push and failed (env mismatch on the runner), which skipped the
pypi-twine stage downstream. MPS uses 'except: - tags' on test so
only pypi-twine runs for tag pipelines; mirror that.
Test stage still runs on every branch push, just not on tag pushes.
When we're ready to drop the twine<6 + setuptools<77 pins on the
build runner, this doc is the recipe. Covers:
- PyPI side: registering a pending publisher per project
- GitLab side: id_tokens: PYPI_ID_TOKEN: aud: pypi
- What pins to drop after migration
- Why coordinated across all four python/* repos in one go
Today we ship via classic ~/.pypirc on the build runner. Trusted
Publishing replaces that with short-lived OIDC tokens minted per
pipeline. Per-project scope, per-pipeline expiry, no long-lived
secret on the runner.
Pipeline #40344 (v0.1.1) failed at twine check with 'Metadata is
missing required fields: Name, Version' — twine 5.x doesn't recognize
Metadata-Version 2.4 wheels (it caps at 2.3). setuptools 77.0 (March
2025) bumped wheels to 2.4 for PEP 639 license-expression support.
Capping the build env's setuptools at <77 keeps wheels on Metadata-
Version 2.2 (verified locally — twine 5 PASSED).
Long-term fix is Trusted Publishing OIDC, which lets us drop the
twine<6 pin AND the setuptools cap. See the migration plan in our
next commit / pipeline discussion.
twine 6 (Sep 2025) auto-detects GitLab CI envvars and refuses to
fall back to ~/.pypirc on the runner, requiring PYPI_ID_TOKEN
(Trusted Publishing OIDC) instead. Pipeline #40338 (v0.1.0 tag) hit
this and failed with TrustedPublishingFailure.
Pinning twine<6 restores classic ~/.pypirc auth that ago /
make_post_sell / remarkbox already use on the same runner. When we
migrate to Trusted Publishing as a coordinated change across all four
python/* repos, we'll drop this pin and add an id_tokens block + PyPI
trusted-publisher config.
7 phases shipped: ETF codec, EPMD client, v6 handshake (MD5 cookie),
data channel (4-byte framing), gen_call protocol, TLS dist
(inet_tls_dist), unfeed integration. Maps + cross-stack validation
through wallet.unsandbox.com's test pyramid.
122 tests across unit / handshake / channel / node / tls / etf-maps.
Tag this commit with vX.Y.Z and push the tag — CI's pypi-twine stage
takes over from there.
pyproject.toml gains the metadata PyPI expects:
- Real author name + email
- Keywords (erlang, elixir, distribution, gen_server, ...)
- Classifiers (status, license, Python versions, topic)
- Project URLs (Homepage / Repository / Bug Tracker)
- readme content-type = text/markdown so PyPI renders our README
- build + twine added to the [dev] extra
.gitlab-ci.yml mirrors ago's two-stage pattern:
- test stage on every push (venv + pip install -e .[dev] + pytest)
- pypi-twine stage only on tags: python -m build, twine check,
twine upload. Credentials come from GitLab CI variables
TWINE_USERNAME (typically __token__) and TWINE_PASSWORD.
Makefile gains build / dist-check / publish-test / publish targets so
the release flow is also runnable locally if a dry run is needed.
Release flow:
1. Bump version in pyproject.toml
2. Commit + push
3. git tag -a vX.Y.Z -m 'release vX.Y.Z' && git push --tags
4. CI's pypi-twine stage picks up the tag and uploads
README.md adds an Install section + Quick start (plain dist + TLS dist)
so PyPI's project page shows usable copy on first visit. dist/ artifacts
build cleanly and both pass twine check.
ErlangDistTransport itself lives in unfeed/unfeed/wallet.py to keep
erldistpy app-agnostic. Tests + docs there. Pending the Elixir-side
wallet_rpc gen_server, then staging soak, then prod switch.
make_dist_tls_context() builds an ssl.SSLContext tuned for OTP defaults
(verify_peer, mTLS, TLSv1.2 minimum). Node accepts tls_context= and
wraps the TCP socket in TLS before the v6 handshake runs.
Critical quirk found by experimentation: inet_tls_dist uses {packet, 4}
on the SSL socket during the handshake. Plain inet_tcp_dist uses
{packet, 2} for handshake then switches to {packet, 4} post-nodeup.
handshake() now takes a frame_size= kwarg (2 or 4); Node auto-selects 4
whenever tls_context is supplied.
Cert requirements (found by experimentation against Erlang E2E):
- CA cert with basicConstraints CA:TRUE
- Leaf certs with SAN including the dist hostname (and localhost)
- extendedKeyUsage covering both serverAuth and clientAuth
Tests:
- make_dist_tls_context unit tests
- Live: spawn erl -proto_dist inet_tls with SAN-bearing certs,
Node.call(gen_target, {ping, 99}) round-trips through the tunnel
- Live negative: plaintext connection to TLS-only peer must fail
- Live negative: client cert from a different CA must fail
115 tests green across 5 consecutive runs, lint clean.
Node wraps EPMD lookup + handshake + Channel into a single client
object. Constructor eagerly opens the dist connection; call() runs the
synchronous $gen_call protocol against a registered name on the peer:
caller -> {'$gen_call', {FromPid, Ref}, Request} (REG_SEND)
server -> {Ref, Reply} (SEND)
Synthesized FromPid and a Node-lifetime Ref counter route replies back
to us; mismatched Ref or unexpected control op raises CallProtocolError.
Reply timeout raises CallTimeout (also covers Erlang's silent-drop case
when the registered name doesn't exist).
Tests against an erl peer running a $gen_call-aware loop:
- {ping, X} -> {pong, X}
- {add, A, B} -> {ok, A + B}
- five sequential calls with monotonically increasing Refs
- server error response surfaces as Python tuple
- slow responder triggers CallTimeout
- unknown registered name surfaces as CallTimeout
- ref uniqueness across 100 synthesized refs
111 tests green across 10 consecutive runs, lint clean.
Channel wraps the post-handshake socket and carries 4-byte length-
prefixed distribution messages: pass-through byte ('p') + ETF control
tuple + optional payload term.
API surface:
send_raw / recv_raw -- raw 4-byte framed bytes, empty == net_tick
send_tick -- send keepalive frame
send_control / recv_message -- structured control + payload
send_reg_send -- helper for the REG_SEND case (FromPid,
registered name, payload)
recv_message() transparently skips inbound ticks; callers wanting tick
awareness use recv_raw().
etf.decode_term(data, offset) exposed as a streaming decoder so the
channel can read control + payload back-to-back from one frame body.
Tests:
- pure encode/decode round-trips
- socketpair tests for framing, ticks, helper signatures
- live end-to-end against an erl node with a registered echo process:
EPMD -> handshake -> REG_SEND -> recv reply, payload matches
- boot script writes a /tmp ready-flag after registering `echo`;
fixture waits for both EPMD registration AND the flag to dodge
the race where EPMD registers the node before -eval runs
101 tests green, lint clean.
handshake(sock, our_name=..., cookie=...) drives the OTP 23+ dance over
a 2-byte length-prefixed frame stream:
send_name (N) client -> server
recv_status (s) server -> client
recv_challenge (N) server -> client
challenge_reply (r) client -> server
challenge_ack (a) server -> client
Cookie digest formula md5(cookie ++ integer_to_list(challenge)) was
cross-checked against erlang:md5/1 output as a test reference.
Distribution flags in erldistpy/flags.py advertise the minimum useful
set: extended refs/pids, new fun tags, utf8 atoms, maps, big creation,
v6 handshake, unlink id, v4 node containers.
86 tests green: frame builders + parsers as pure functions, digest
reference, full live handshake against `erl -sname -setcookie`, and a
wrong-cookie rejection test.
Newer SHA-256 digest (DFLAG_MANDATORY_25_DIGEST) deferred until a peer
requires it.
Synchronous TCP client for Erlang Port Mapper Daemon. One request type
(PORT_PLEASE2_REQ, tag 122), one response type (PORT2_RESP, tag 119).
Returns EpmdInfo dataclass or None if the node is not registered.
Tests run two layers:
- Unit tests against recorded byte streams captured from a real EPMD
answering for `erl -sname testnode` and for an unregistered name.
- Integration tests spawn `erl -sname erldistpy_itest` in a fixture
and verify lookup() returns the live port; skipped if erl or EPMD
are absent.
10 new tests, 68 total green, lint clean.
