phase 0 + 1: repo bones and ETF codec

Repo scaffolding (LICENSE, Makefile, pyproject.toml, README) matching
unfeed conventions. Flat package layout, ruff config, Unlicense.

ETF codec covers the subset needed for gen_call against an Elixir node:
small/int/big_int, atom_utf8 (legacy atom_ext on decode), binary, nil,
list, small/large tuple, new_pid, newer_reference. Booleans round-trip
as atoms true/false; Python None as atom nil; str encodes to utf-8
binary to match Elixir convention.

Golden vectors were generated from real Erlang term_to_binary/1 output
(generator script at docs/etf_vectors.erl). Decode tests verify wire
compatibility; round-trip tests verify encoder consistency.

make all green: 58 passed, lint clean.

Next phases tracked in docs/ROADMAP.md (EPMD, handshake, channel,
gen_call, TLS, unfeed integration).
This commit is contained in:
russell@unturf.com 2026-06-16 10:39:01 -04:00
commit b9fd28ca3b
No known key found for this signature in database
11 changed files with 680 additions and 0 deletions

12
.gitignore vendored Normal file
View file

@ -0,0 +1,12 @@
.venv/
__pycache__/
*.pyc
*.pyo
.pytest_cache/
.ruff_cache/
.mypy_cache/
build/
dist/
*.egg-info/
.coverage
htmlcov/

21
LICENSE Normal file
View file

@ -0,0 +1,21 @@
This is free and unencumbered software released into the public domain.
Anyone is free to copy, modify, publish, use, compile, sell, or distribute this
software, either in source code form or as a compiled binary, for any purpose,
commercial or non-commercial, and by any means.
In jurisdictions that recognize copyright laws, the author or authors of this
software dedicate any and all copyright interest in the software to the public
domain. We make this dedication for the benefit of the public at large and to
the detriment of our heirs and successors. We intend this dedication to be an
overt act of relinquishment in perpetuity of all present and future rights to
this software under copyright law.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
For more information, please refer to <https://unlicense.org>

42
Makefile Normal file
View file

@ -0,0 +1,42 @@
VENV_DIR = $(shell pwd)/.venv
PYTHON = $(VENV_DIR)/bin/python
PIP = $(VENV_DIR)/bin/pip
PYTEST = $(VENV_DIR)/bin/pytest
RUFF = $(VENV_DIR)/bin/ruff
.DEFAULT_GOAL := all
.PHONY: all bootstrap venv install test lint format clean help
all: bootstrap test lint
help:
@echo "erldistpy targets"
@echo " make bootstrap - create venv, install editable + dev deps"
@echo " make test - run pytest"
@echo " make lint - ruff check"
@echo " make format - ruff format"
@echo " make clean - drop venv + caches"
venv:
test -d $(VENV_DIR) || python3 -m venv $(VENV_DIR)
$(PIP) install --upgrade pip wheel setuptools
bootstrap: venv
$(PIP) install -e ".[dev]"
install: bootstrap
test:
$(PYTEST) -v
lint:
$(RUFF) check erldistpy tests
format:
$(RUFF) format erldistpy tests
clean:
rm -rf $(VENV_DIR)
rm -rf .pytest_cache .ruff_cache
rm -rf build dist *.egg-info
find . -type d -name __pycache__ -exec rm -rf {} +

41
README.md Normal file
View file

@ -0,0 +1,41 @@
# erldistpy
Native Python client for our Erlang distribution protocol. Talk Erlang/Elixir
nodes from CPython without HTTP shim layers.
Built to swap into `unfeed`'s `WalletTransport` so a Python web app can call
`Unsandbox.WalletRPC` over native dist instead of HTTPS. Same gen_call
semantics, lower latency, fewer moving parts.
## Status
Phase 0 — repo bones. See `docs/ROADMAP.md` for the phase plan.
## Quick start
```bash
make bootstrap # create venv, install editable + dev deps
make test # run pytest
make lint # ruff check
```
## Scope
- ETF (External Term Format) codec — encode/decode Erlang terms
- EPMD client — node name → port lookup
- Distribution handshake — cookie auth, version negotiation
- gen_call to registered processes on a remote node
- TLS dist support (Erlang `inet_tls_dist`)
Out of scope: full Erlang node impersonation, link/monitor lifecycles,
distributed Mnesia. We are a *client*, not a peer node.
## Why not Pyrlang?
Pyrlang implements a full asyncio Erlang node. Heavy, asyncio-first, complex.
We need a small synchronous client that fits behind the same `WalletTransport`
Protocol as `httpx`. Different shape.
## License
Unlicense (public domain).

67
docs/ROADMAP.md Normal file
View file

@ -0,0 +1,67 @@
# erldistpy roadmap
Phases below are sized to land as discrete commits. Each phase ends with
`make all` green and a real-VM interop test where applicable.
## Phase 0 — Repo bones ✅
- LICENSE, README, .gitignore, Makefile, pyproject.toml
- Package skeleton, ruff config matches unfeed
- `make bootstrap test lint` works on a fresh checkout
## Phase 1 — ETF codec ✅
- Subset of External Term Format we need for gen_call to Elixir:
small_int, int, big_int, atom_utf8 (legacy atom_ext on decode), binary,
nil, list, tuple (small + large), pid (new_pid), ref (newer_reference)
- Booleans and `None` ride as atoms `true` / `false` / `nil`
- Golden vectors decoded from real `term_to_binary/1` output
- Round-trip tests for the encoder
- Generator script committed at `docs/etf_vectors.erl`
## Phase 2 — EPMD client
EPMD (Erlang Port Mapper Daemon) maps node names to TCP ports.
Tiny synchronous TCP client.
- Connect `epmd_host:4369` (configurable)
- `PORT_PLEASE2_REQ` (113) → port + dist version range
- Return `EpmdNodeInfo(port, lo_ver, hi_ver, node_type, proto)`
- Tests: live EPMD on localhost (skip if not running), unit tests
against a recorded byte stream
## Phase 3 — Distribution handshake
- TCP connect to the resolved port
- `send_name` / `recv_status` / `recv_challenge` / `send_challenge_reply`
/ `recv_challenge_ack`
- Cookie digest via `erlang:phash2`-equivalent (md5-based per spec)
- Version 6 ("v6") handshake, the modern one Elixir 1.15+ uses
- Tests: handshake against a live Erlang node started in conftest
## Phase 4 — Distribution channel
- After handshake, the socket carries control + payload messages framed
by a 4-byte length prefix
- Send `SEND_TT` / `REG_SEND` for outgoing messages
- Receive replies, route by ref
- Tick loop for keepalive (60s default per OTP)
## Phase 5 — gen_call convenience layer
- `Node.call(name_or_pid, request, timeout=5.0)` → reply term
- Wraps the `$gen_call` protocol used by `:gen_server`
- Idempotency-key handling lives in the caller; we just pass the term
## Phase 6 — TLS dist
- Wrap the post-EPMD socket in TLS
- Match `inet_tls_dist` config on the Erlang side (cert + key + ca paths)
- Same handshake, just runs inside the TLS tunnel
## Phase 7 — unfeed integration
- `ErlangDistTransport` implementing unfeed's `WalletTransport` Protocol
- Maps `/v1/health`, `/v1/address`, ... to gen_call requests
- `unfeed` deploys with a config switch: `wallet_transport = http | erldist`
- Cutover after a soak period on staging

47
docs/etf_vectors.erl Normal file
View file

@ -0,0 +1,47 @@
%% Generator for the golden ETF vectors used in tests/test_etf.py.
%% Re-run when adding new test cases:
%%
%% erlc -o /tmp docs/etf_vectors.erl
%% erl -noshell -pa /tmp -eval 'etf_vectors:main([]), init:stop().'
%% cat /tmp/etf_vectors.txt
%%
-module(etf_vectors).
-export([main/1]).
main(_) ->
Terms = [
{small_int_0, 0},
{small_int_42, 42},
{small_int_255, 255},
{int_256, 256},
{int_neg_1, -1},
{int_max32, 2147483647},
{int_min32, -2147483648},
{big_pos, 9999999999999999},
{big_neg, -9999999999999999},
{atom_hello, hello},
{atom_true, true},
{atom_false, false},
{atom_nil, nil},
{atom_unicode, 'привет'},
{binary_empty, <<>>},
{binary_hi, <<"hi">>},
{binary_utf8, <<"héllo"/utf8>>},
{tuple_empty, {}},
{tuple_pair, {ok, <<"value">>}},
{tuple_triple, {1, 2, 3}},
{list_empty, []},
{list_ints, [1, 2, 3]},
{list_mixed, [hello, <<"world">>, 42]},
{nested, {gen_call, [{node, 'wallet@cammy'}], {get_balance, xmr}}}
],
Out = [
io_lib:format("~s|~s~n", [atom_to_list(N), bin_to_hex(term_to_binary(T))])
|| {N, T} <- Terms
],
ok = file:write_file("/tmp/etf_vectors.txt", iolist_to_binary(Out)).
bin_to_hex(B) ->
<< <<(hex(N div 16)), (hex(N rem 16))>> || <<N>> <= B >>.
hex(N) when N < 10 -> $0 + N;
hex(N) -> $a + N - 10.

13
erldistpy/__init__.py Normal file
View file

@ -0,0 +1,13 @@
"""erldistpy — native Python client for our Erlang distribution protocol."""
__version__ = "0.0.1"
from erldistpy.etf import (
Atom,
Pid,
Reference,
decode,
encode,
)
__all__ = ["Atom", "Pid", "Reference", "decode", "encode", "__version__"]

276
erldistpy/etf.py Normal file
View file

@ -0,0 +1,276 @@
"""ETF — External Term Format codec.
Encodes/decodes the subset of Erlang terms we need to drive ``gen_call``
against an Elixir node: atoms, integers, binaries, lists, tuples, pids,
references. Booleans round-trip as atoms ``true`` / ``false``; Python
``None`` rides as atom ``nil``.
Spec: https://www.erlang.org/doc/apps/erts/erl_ext_dist.html
"""
from __future__ import annotations
import struct
from dataclasses import dataclass
MAGIC = 131
# Tag numbers from the ETF spec.
SMALL_INTEGER_EXT = 97
INTEGER_EXT = 98
ATOM_EXT = 100 # legacy latin1; we decode but never emit
SMALL_TUPLE_EXT = 104
LARGE_TUPLE_EXT = 105
NIL_EXT = 106
STRING_EXT = 107
LIST_EXT = 108
BINARY_EXT = 109
SMALL_BIG_EXT = 110
LARGE_BIG_EXT = 111
NEW_PID_EXT = 88
NEWER_REFERENCE_EXT = 90
ATOM_UTF8_EXT = 118
SMALL_ATOM_UTF8_EXT = 119
class ETFError(ValueError):
"""Decode failed or term cannot be encoded."""
@dataclass(frozen=True)
class Atom:
name: str
def __post_init__(self):
if len(self.name.encode("utf-8")) > 255:
# Erlang atoms cap at 255 bytes in their UTF-8 form.
raise ETFError(f"atom too long: {self.name!r}")
@dataclass(frozen=True)
class Pid:
node: Atom
id: int
serial: int
creation: int
@dataclass(frozen=True)
class Reference:
node: Atom
creation: int
ids: tuple[int, ...]
# ---------------------------------------------------------------------------
# encode
# ---------------------------------------------------------------------------
def encode(term: object) -> bytes:
"""Encode a Python value to ETF bytes including the magic version byte."""
return bytes([MAGIC]) + _encode(term)
def _encode(term: object) -> bytes:
if isinstance(term, bool):
# bool is an int subclass — check before int
return _encode_atom(Atom("true" if term else "false"))
if term is None:
return _encode_atom(Atom("nil"))
if isinstance(term, Atom):
return _encode_atom(term)
if isinstance(term, int):
return _encode_int(term)
if isinstance(term, bytes):
return _encode_binary(term)
if isinstance(term, str):
return _encode_binary(term.encode("utf-8"))
if isinstance(term, tuple):
return _encode_tuple(term)
if isinstance(term, list):
return _encode_list(term)
if isinstance(term, Pid):
return _encode_pid(term)
if isinstance(term, Reference):
return _encode_ref(term)
raise ETFError(f"cannot encode {type(term).__name__}")
def _encode_atom(a: Atom) -> bytes:
raw = a.name.encode("utf-8")
if len(raw) < 256:
return bytes([SMALL_ATOM_UTF8_EXT, len(raw)]) + raw
return bytes([ATOM_UTF8_EXT]) + struct.pack(">H", len(raw)) + raw
def _encode_int(n: int) -> bytes:
if 0 <= n <= 255:
return bytes([SMALL_INTEGER_EXT, n])
if -(2**31) <= n <= (2**31) - 1:
return bytes([INTEGER_EXT]) + struct.pack(">i", n)
# bigint: pack magnitude little-endian, sign as 0/1
sign = 0 if n >= 0 else 1
mag = abs(n)
body = bytearray()
while mag:
body.append(mag & 0xFF)
mag >>= 8
if len(body) < 256:
return bytes([SMALL_BIG_EXT, len(body), sign]) + bytes(body)
return bytes([LARGE_BIG_EXT]) + struct.pack(">I", len(body)) + bytes([sign]) + bytes(body)
def _encode_binary(b: bytes) -> bytes:
return bytes([BINARY_EXT]) + struct.pack(">I", len(b)) + b
def _encode_tuple(t: tuple) -> bytes:
n = len(t)
if n < 256:
out = bytes([SMALL_TUPLE_EXT, n])
else:
out = bytes([LARGE_TUPLE_EXT]) + struct.pack(">I", n)
return out + b"".join(_encode(x) for x in t)
def _encode_list(lst: list) -> bytes:
if not lst:
return bytes([NIL_EXT])
out = bytes([LIST_EXT]) + struct.pack(">I", len(lst))
out += b"".join(_encode(x) for x in lst)
out += bytes([NIL_EXT]) # tail
return out
def _encode_pid(p: Pid) -> bytes:
return (
bytes([NEW_PID_EXT])
+ _encode_atom(p.node)
+ struct.pack(">II I", p.id, p.serial, p.creation)
)
def _encode_ref(r: Reference) -> bytes:
body = (
struct.pack(">H", len(r.ids))
+ _encode_atom(r.node)
+ struct.pack(">I", r.creation)
+ b"".join(struct.pack(">I", i) for i in r.ids)
)
return bytes([NEWER_REFERENCE_EXT]) + body
# ---------------------------------------------------------------------------
# decode
# ---------------------------------------------------------------------------
def decode(data: bytes) -> object:
"""Decode ETF bytes (with magic version byte) into a Python value."""
if not data or data[0] != MAGIC:
raise ETFError(f"bad magic: {data[:1]!r}")
term, off = _decode(data, 1)
if off != len(data):
raise ETFError(f"trailing bytes after term: {len(data) - off} unread")
return term
def _decode(data: bytes, off: int) -> tuple[object, int]:
if off >= len(data):
raise ETFError("unexpected end of input")
tag = data[off]
off += 1
if tag == SMALL_INTEGER_EXT:
return data[off], off + 1
if tag == INTEGER_EXT:
return struct.unpack(">i", data[off:off + 4])[0], off + 4
if tag == SMALL_BIG_EXT:
n = data[off]
sign = data[off + 1]
off += 2
val = int.from_bytes(data[off:off + n], "little")
return (-val if sign else val), off + n
if tag == LARGE_BIG_EXT:
n = struct.unpack(">I", data[off:off + 4])[0]
off += 4
sign = data[off]
off += 1
val = int.from_bytes(data[off:off + n], "little")
return (-val if sign else val), off + n
if tag == SMALL_ATOM_UTF8_EXT:
n = data[off]
off += 1
return _atom_or_alias(data[off:off + n].decode("utf-8")), off + n
if tag == ATOM_UTF8_EXT:
n = struct.unpack(">H", data[off:off + 2])[0]
off += 2
return _atom_or_alias(data[off:off + n].decode("utf-8")), off + n
if tag == ATOM_EXT:
n = struct.unpack(">H", data[off:off + 2])[0]
off += 2
return _atom_or_alias(data[off:off + n].decode("latin-1")), off + n
if tag == BINARY_EXT:
n = struct.unpack(">I", data[off:off + 4])[0]
off += 4
return data[off:off + n], off + n
if tag == STRING_EXT:
n = struct.unpack(">H", data[off:off + 2])[0]
off += 2
return list(data[off:off + n]), off + n
if tag == NIL_EXT:
return [], off
if tag == SMALL_TUPLE_EXT:
n = data[off]
off += 1
return _decode_tuple(data, off, n)
if tag == LARGE_TUPLE_EXT:
n = struct.unpack(">I", data[off:off + 4])[0]
return _decode_tuple(data, off + 4, n)
if tag == LIST_EXT:
n = struct.unpack(">I", data[off:off + 4])[0]
off += 4
items = []
for _ in range(n):
item, off = _decode(data, off)
items.append(item)
tail, off = _decode(data, off)
if tail != []:
# Improper list — uncommon in our protocol; surface it
raise ETFError(f"improper list tail: {tail!r}")
return items, off
if tag == NEW_PID_EXT:
node, off = _decode(data, off)
if not isinstance(node, Atom):
raise ETFError("pid node is not an atom")
id_, serial, creation = struct.unpack(">III", data[off:off + 12])
return Pid(node=node, id=id_, serial=serial, creation=creation), off + 12
if tag == NEWER_REFERENCE_EXT:
n = struct.unpack(">H", data[off:off + 2])[0]
off += 2
node, off = _decode(data, off)
if not isinstance(node, Atom):
raise ETFError("ref node is not an atom")
creation = struct.unpack(">I", data[off:off + 4])[0]
off += 4
ids = struct.unpack(f">{n}I", data[off:off + 4 * n])
return Reference(node=node, creation=creation, ids=ids), off + 4 * n
raise ETFError(f"unknown tag: {tag}")
def _decode_tuple(data: bytes, off: int, arity: int) -> tuple[tuple, int]:
items = []
for _ in range(arity):
item, off = _decode(data, off)
items.append(item)
return tuple(items), off
def _atom_or_alias(name: str) -> object:
if name == "true":
return True
if name == "false":
return False
if name == "nil":
return None
return Atom(name)

37
pyproject.toml Normal file
View file

@ -0,0 +1,37 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "erldistpy"
version = "0.0.1"
description = "Native Python client for Erlang distribution protocol"
readme = "README.md"
requires-python = ">=3.10"
license = { text = "Unlicense" }
authors = [
{ name = "fox", email = "russell@unturf.com" },
]
dependencies = []
[project.optional-dependencies]
dev = [
"pytest >=8.0,<9",
"pytest-cov >=5.0,<7",
"ruff >=0.6,<1",
]
[tool.setuptools.packages.find]
include = ["erldistpy*"]
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-ra --strict-markers"
[tool.ruff]
line-length = 100
target-version = "py310"
[tool.ruff.lint]
select = ["E", "F", "I", "B", "UP", "SIM"]
ignore = ["E501"]

0
tests/__init__.py Normal file
View file

124
tests/test_etf.py Normal file
View file

@ -0,0 +1,124 @@
"""ETF codec tests.
Golden vectors below were generated by a real Erlang VM via
``term_to_binary/1``. See ``docs/etf_vectors.erl`` for the generator script.
Decode tests prove we read the wire spec correctly; round-trip tests prove
the encoder is internally consistent.
"""
from __future__ import annotations
import pytest
from erldistpy.etf import (
Atom,
ETFError,
Pid,
Reference,
decode,
encode,
)
# Golden vectors: (name, hex_from_real_vm, expected_python_value)
GOLDEN = [
("small_int_0", "836100", 0),
("small_int_42", "83612a", 42),
("small_int_255", "8361ff", 255),
("int_256", "836200000100", 256),
("int_neg_1", "8362ffffffff", -1),
("int_max32", "83627fffffff", 2147483647),
("int_min32", "836280000000", -2147483648),
("big_pos", "836e0700ffffc06ff28623", 9999999999999999),
("big_neg", "836e0701ffffc06ff28623", -9999999999999999),
("atom_hello", "8364000568656c6c6f", Atom("hello")),
("atom_true", "8364000474727565", True),
("atom_false", "8364000566616c7365", False),
("atom_nil", "836400036e696c", None),
("atom_unicode", "83770cd0bfd180d0b8d0b2d0b5d182", Atom("привет")),
("binary_empty", "836d00000000", b""),
("binary_hi", "836d000000026869", b"hi"),
("binary_utf8", "836d0000000668c3a96c6c6f", "héllo".encode()),
("tuple_empty", "836800", ()),
("tuple_pair", "8368026400026f6b6d0000000576616c7565", (Atom("ok"), b"value")),
("tuple_triple", "836803610161026103", (1, 2, 3)),
("list_empty", "836a", []),
# NOTE: small ints all < 256 → Erlang emits STRING_EXT. Decodes to list of ints.
("list_ints", "836b0003010203", [1, 2, 3]),
("list_mixed", "836c0000000364000568656c6c6f6d00000005776f726c64612a6a",
[Atom("hello"), b"world", 42]),
("nested",
"83680364000867656e5f63616c6c6c0000000168026400046e6f646564000c77616c6c65744063616d6d796a680264000b6765745f62616c616e6365640003786d72",
(Atom("gen_call"), [(Atom("node"), Atom("wallet@cammy"))], (Atom("get_balance"), Atom("xmr")))),
]
@pytest.mark.parametrize("name,hex_in,expected", GOLDEN, ids=[g[0] for g in GOLDEN])
def test_decode_golden(name, hex_in, expected):
assert decode(bytes.fromhex(hex_in)) == expected
# Round-trip — our encoder may pick different tags (e.g. SMALL_ATOM_UTF8_EXT
# instead of legacy ATOM_EXT) but the term must survive a round-trip intact.
@pytest.mark.parametrize("term", [
0, 1, 255, 256, -1, 2**31 - 1, -(2**31),
10**20, -(10**20),
Atom("hello"), Atom("ok"), True, False, None,
b"", b"hi", b"\x00\x01\x02",
(), (1,), (Atom("ok"), b"value"),
[], [1, 2, 3], [Atom("a"), b"b", 3],
(Atom("gen_call"), [], (Atom("ping"),)),
])
def test_round_trip(term):
assert decode(encode(term)) == term
def test_str_encodes_as_binary():
# Python str → utf-8 binary (matches Elixir convention)
assert decode(encode("hello")) == b"hello"
def test_pid_round_trip():
p = Pid(node=Atom("wallet@cammy"), id=42, serial=1, creation=7)
assert decode(encode(p)) == p
def test_reference_round_trip():
r = Reference(node=Atom("wallet@cammy"), creation=3, ids=(100, 200, 300))
assert decode(encode(r)) == r
def test_bool_before_int():
# bool is a subclass of int; must encode as atom, not as integer
out = encode(True)
assert decode(out) is True
assert decode(encode(False)) is False
def test_atom_too_long():
with pytest.raises(ETFError):
Atom("x" * 256)
def test_bad_magic():
with pytest.raises(ETFError, match="bad magic"):
decode(b"\x00\x00")
def test_unknown_tag():
with pytest.raises(ETFError, match="unknown tag"):
decode(bytes([131, 0xFE]))
def test_trailing_bytes():
with pytest.raises(ETFError, match="trailing bytes"):
decode(bytes.fromhex("836100") + b"\x00")
def test_unencodable_type():
with pytest.raises(ETFError, match="cannot encode"):
encode(object())
def test_empty_input():
with pytest.raises(ETFError, match="bad magic"):
decode(b"")