From 41d1d9b71eb073a0ad898014d2115265ad34b2cd Mon Sep 17 00:00:00 2001
From: "russell@unturf.com" <russell@unturf.com>
Date: Sat, 2 May 2026 11:58:25 -0400
Subject: [PATCH] qa(client): scrub lone UTF-16 surrogates before httpx encode
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Six bench runs aborted with:

  UnicodeEncodeError: 'utf-8' codec can't encode characters
  in position N-M: surrogates not allowed

The error fires inside httpx's json-encode path: when the
request body's JSON contains lone surrogates (from Wikipedia
chunks ingested with invalid-UTF-8 source bytes), httpx's
.encode('utf-8') raises before the request even leaves the
client.

Earlier surrogate fixes (3b91223) hardened the OUTPUT side —
sha256 hashers now use errors='surrogatepass' so the run-DAG
roots survive surrogate-bearing model output. But the INPUT
side (corpus text injected into the prompt) was still
vulnerable: the LLM never sees the surrogate but the HTTP
client tries to send it.

Fix: scrub message content via WTF-8 → UTF-8-with-replace
roundtrip in OpenAICompatibleClient.chat_completion. Lone
surrogates become U+FFFD (REPLACEMENT CHARACTER); the prompt
serializes cleanly. Verified: 'tell me about the roman empire'
under claim_lattice mode now classifies HYBRID 5/7 instead of
erroring out (this question was 6/6 lattice runs failing on
the 2026-05-02 c=4 bench).

The scrub lives in the client because the hot path needs to
guarantee the outbound HTTP body is valid UTF-8, regardless of
what upstream code injected. Defense-in-depth: ingest-time
sanitization would be cleaner but the existing corpus already
has surrogates baked in, and re-ingest would invalidate every
document_root in 6 GB of shards.

Tests: 751/34 still pass clean in 11s with pytest -n auto.
---
 aborist/qa/client.py | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/aborist/qa/client.py b/aborist/qa/client.py
index 26d0a97..dc7b4b9 100644
--- a/aborist/qa/client.py
+++ b/aborist/qa/client.py
@@ -114,6 +114,22 @@ class OpenAICompatibleClient:
         self.close()
         return False
 
+    @staticmethod
+    def _scrub_surrogates(s: str) -> str:
+        """Replace lone UTF-16 surrogates with U+FFFD.
+
+        Wikipedia chunks (and other ingested text) occasionally
+        contain unpaired surrogates from the ingest of invalid-UTF-8
+        source. httpx's json= path does ``.encode('utf-8')`` on the
+        serialized request body, which raises UnicodeEncodeError on
+        any lone surrogate. Sanitize incoming message content here
+        so the outbound HTTP request always serializes cleanly. We
+        round-trip through WTF-8 (surrogatepass) bytes, then decode
+        as standard UTF-8 with replacement — invalid sequences
+        become U+FFFD (REPLACEMENT CHARACTER).
+        """
+        return s.encode("utf-8", errors="surrogatepass").decode("utf-8", errors="replace")
+
     def chat_completion(
         self,
         messages: list[dict],
@@ -130,6 +146,18 @@ class OpenAICompatibleClient:
 
         client = self._http  # persistent connection pool from __init__
         headers = {"Content-Type": "application/json"}
+        # Sanitize message content for httpx's json-encode path.
+        messages = [
+            {
+                **m,
+                "content": (
+                    self._scrub_surrogates(m["content"])
+                    if isinstance(m.get("content"), str)
+                    else m.get("content")
+                ),
+            }
+            for m in messages
+        ]
         if self.api_key:
             headers["Authorization"] = f"Bearer {self.api_key}"
         payload = {