qa(client): scrub lone UTF-16 surrogates before httpx encode
Six bench runs aborted with:
UnicodeEncodeError: 'utf-8' codec can't encode characters
in position N-M: surrogates not allowed
The error fires inside httpx's json-encode path: when the
request body's JSON contains lone surrogates (from Wikipedia
chunks ingested with invalid-UTF-8 source bytes), httpx's
.encode('utf-8') raises before the request even leaves the
client.
Earlier surrogate fixes (3b91223) hardened the OUTPUT side —
sha256 hashers now use errors='surrogatepass' so the run-DAG
roots survive surrogate-bearing model output. But the INPUT
side (corpus text injected into the prompt) was still
vulnerable: the LLM never sees the surrogate but the HTTP
client tries to send it.
Fix: scrub message content via WTF-8 → UTF-8-with-replace
roundtrip in OpenAICompatibleClient.chat_completion. Lone
surrogates become U+FFFD (REPLACEMENT CHARACTER); the prompt
serializes cleanly. Verified: 'tell me about the roman empire'
under claim_lattice mode now classifies HYBRID 5/7 instead of
erroring out (this question was 6/6 lattice runs failing on
the 2026-05-02 c=4 bench).
The scrub lives in the client because the hot path needs to
guarantee the outbound HTTP body is valid UTF-8, regardless of
what upstream code injected. Defense-in-depth: ingest-time
sanitization would be cleaner but the existing corpus already
has surrogates baked in, and re-ingest would invalidate every
document_root in 6 GB of shards.
Tests: 751/34 still pass clean in 11s with pytest -n auto.
This commit is contained in:
parent
0177c2d278
commit
41d1d9b71e
1 changed files with 28 additions and 0 deletions
|
|
@ -114,6 +114,22 @@ class OpenAICompatibleClient:
|
|||
self.close()
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _scrub_surrogates(s: str) -> str:
|
||||
"""Replace lone UTF-16 surrogates with U+FFFD.
|
||||
|
||||
Wikipedia chunks (and other ingested text) occasionally
|
||||
contain unpaired surrogates from the ingest of invalid-UTF-8
|
||||
source. httpx's json= path does ``.encode('utf-8')`` on the
|
||||
serialized request body, which raises UnicodeEncodeError on
|
||||
any lone surrogate. Sanitize incoming message content here
|
||||
so the outbound HTTP request always serializes cleanly. We
|
||||
round-trip through WTF-8 (surrogatepass) bytes, then decode
|
||||
as standard UTF-8 with replacement — invalid sequences
|
||||
become U+FFFD (REPLACEMENT CHARACTER).
|
||||
"""
|
||||
return s.encode("utf-8", errors="surrogatepass").decode("utf-8", errors="replace")
|
||||
|
||||
def chat_completion(
|
||||
self,
|
||||
messages: list[dict],
|
||||
|
|
@ -130,6 +146,18 @@ class OpenAICompatibleClient:
|
|||
|
||||
client = self._http # persistent connection pool from __init__
|
||||
headers = {"Content-Type": "application/json"}
|
||||
# Sanitize message content for httpx's json-encode path.
|
||||
messages = [
|
||||
{
|
||||
**m,
|
||||
"content": (
|
||||
self._scrub_surrogates(m["content"])
|
||||
if isinstance(m.get("content"), str)
|
||||
else m.get("content")
|
||||
),
|
||||
}
|
||||
for m in messages
|
||||
]
|
||||
if self.api_key:
|
||||
headers["Authorization"] = f"Bearer {self.api_key}"
|
||||
payload = {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue