Six bench runs aborted with:
UnicodeEncodeError: 'utf-8' codec can't encode characters
in position N-M: surrogates not allowed
The error fires inside httpx's json-encode path: when the
request body's JSON contains lone surrogates (from Wikipedia
chunks ingested with invalid-UTF-8 source bytes), httpx's
.encode('utf-8') raises before the request even leaves the
client.
Earlier surrogate fixes (3b91223) hardened the OUTPUT side —
sha256 hashers now use errors='surrogatepass' so the run-DAG
roots survive surrogate-bearing model output. But the INPUT
side (corpus text injected into the prompt) was still
vulnerable: the LLM never sees the surrogate but the HTTP
client tries to send it.
Fix: scrub message content via WTF-8 → UTF-8-with-replace
roundtrip in OpenAICompatibleClient.chat_completion. Lone
surrogates become U+FFFD (REPLACEMENT CHARACTER); the prompt
serializes cleanly. Verified: 'tell me about the roman empire'
under claim_lattice mode now classifies HYBRID 5/7 instead of
erroring out (this question was 6/6 lattice runs failing on
the 2026-05-02 c=4 bench).
The scrub lives in the client because the hot path needs to
guarantee the outbound HTTP body is valid UTF-8, regardless of
what upstream code injected. Defense-in-depth: ingest-time
sanitization would be cleaner but the existing corpus already
has surrogates baked in, and re-ingest would invalidate every
document_root in 6 GB of shards.
Tests: 751/34 still pass clean in 11s with pytest -n auto.
russell@unturf.com
41d1d9b71e