qa(client): scrub lone UTF-16 surrogates before httpx encode

Six bench runs aborted with:

  UnicodeEncodeError: 'utf-8' codec can't encode characters
  in position N-M: surrogates not allowed

The error fires inside httpx's json-encode path: when the
request body's JSON contains lone surrogates (from Wikipedia
chunks ingested with invalid-UTF-8 source bytes), httpx's
.encode('utf-8') raises before the request even leaves the
client.

Earlier surrogate fixes (3b91223) hardened the OUTPUT side —
sha256 hashers now use errors='surrogatepass' so the run-DAG
roots survive surrogate-bearing model output. But the INPUT
side (corpus text injected into the prompt) was still
vulnerable: the LLM never sees the surrogate but the HTTP
client tries to send it.

Fix: scrub message content via WTF-8 → UTF-8-with-replace
roundtrip in OpenAICompatibleClient.chat_completion. Lone
surrogates become U+FFFD (REPLACEMENT CHARACTER); the prompt
serializes cleanly. Verified: 'tell me about the roman empire'
under claim_lattice mode now classifies HYBRID 5/7 instead of
erroring out (this question was 6/6 lattice runs failing on
the 2026-05-02 c=4 bench).

The scrub lives in the client because the hot path needs to
guarantee the outbound HTTP body is valid UTF-8, regardless of
what upstream code injected. Defense-in-depth: ingest-time
sanitization would be cleaner but the existing corpus already
has surrogates baked in, and re-ingest would invalidate every
document_root in 6 GB of shards.

Tests: 751/34 still pass clean in 11s with pytest -n auto.
This commit is contained in:
russell@unturf.com 2026-05-02 11:58:25 -04:00
parent 0177c2d278
commit 41d1d9b71e
No known key found for this signature in database

View file

@ -114,6 +114,22 @@ class OpenAICompatibleClient:
self.close()
return False
@staticmethod
def _scrub_surrogates(s: str) -> str:
"""Replace lone UTF-16 surrogates with U+FFFD.
Wikipedia chunks (and other ingested text) occasionally
contain unpaired surrogates from the ingest of invalid-UTF-8
source. httpx's json= path does ``.encode('utf-8')`` on the
serialized request body, which raises UnicodeEncodeError on
any lone surrogate. Sanitize incoming message content here
so the outbound HTTP request always serializes cleanly. We
round-trip through WTF-8 (surrogatepass) bytes, then decode
as standard UTF-8 with replacement invalid sequences
become U+FFFD (REPLACEMENT CHARACTER).
"""
return s.encode("utf-8", errors="surrogatepass").decode("utf-8", errors="replace")
def chat_completion(
self,
messages: list[dict],
@ -130,6 +146,18 @@ class OpenAICompatibleClient:
client = self._http # persistent connection pool from __init__
headers = {"Content-Type": "application/json"}
# Sanitize message content for httpx's json-encode path.
messages = [
{
**m,
"content": (
self._scrub_surrogates(m["content"])
if isinstance(m.get("content"), str)
else m.get("content")
),
}
for m in messages
]
if self.api_key:
headers["Authorization"] = f"Bearer {self.api_key}"
payload = {