speed: pytest-xdist, bench smoke, concurrency default; UTF surrogate fix

Bench-max sprint 1a + sprint 3 + speed audit. Five wins, none of
them traded calibration.

UTF-16 surrogate fix (sprint 1a)
================================
Hermes occasionally emits text with lone UTF-16 surrogates. Bare
.encode('utf-8') raises UnicodeEncodeError on those, which aborted
the run with no Merkle root. Two errors per lattice mode in the
2026-05-02 bench were this exact path on the 'tell me about the
roman empire' question.

Fix: errors='surrogatepass' on the four sha256 helpers that hash
model-derived text, plus the two audit-chain encode sites in
store.py for defense-in-depth (audit body could carry user text in
some flows). The hash stays deterministic because WTF-8 bytes are
reversible & unique per input.

Touched:
  aborist/qa/dag.py:_sha256_hex            (the loud one)
  aborist/qa/keys.py:_sha256
  aborist/qa/evidence.py:_sha256_hex
  aborist/store.py: chain_audit_events + append_audit

Predicted Δ on next bench: +1pp on lattice modes (the 2 errors
become valid runs).

Smoke fixture (sprint 3)
========================
bench/qa_questions_smoke.txt — 5 questions, all anchor classes,
each currently failing pointer mode 100% while JSON aces 100% per
the 2026-05-02 bench. Wired as 'make bench-qa-smoke', --n 1
--concurrency 4, ~30-90s wall-clock depending on vLLM warmth. The
inner loop for prompt iteration; the full 71-question sweep stays
the scoreboard.

Smoke verified: pointer=0/5 STRICT, JSON=5/5, quote=2/5. Confirms
the gap pattern from the journal.

Concurrency default
===================
Makefile bench-qa now defaults to BENCH_QA_CONCURRENCY=4 (was
sequential). Override via BENCH_QA_CONCURRENCY=N. Combined with
the --concurrency landing in 0870af6, full sweep drops from ~107
min projected to ~51 min actual.

pytest-xdist (test-speed)
=========================
Added pytest-xdist>=3.5 to dev extras. 'make test' now uses
-n auto (= one worker per logical CPU). Measured: 36s → 10s on
the 641-test suite. 3.6× speedup, no test changes required.

Bench-max scoreboard (predicted lift from this commit alone):
  +1pp lattice modes (UTF fix)
  +cycle-time enabler (smoke fixture, xdist)
  no calibration cost — none of the verifier checks moved.

Makefile

@@ -183,7 +183,8 @@ BENCH_QA_OUT       ?= bench/qa_results
 BENCH_QA_MODES     ?= quote,claim_lattice_pointer,claim_lattice
 BENCH_QA_LIMIT     ?= 0
 BENCH_QA_N         ?= 3
-bench-qa: bootstrap ## QA-quality sweep: questions × modes × N samples [BENCH_QA_N=3 BENCH_QA_LIMIT=N BENCH_QA_MODES=...]
+BENCH_QA_CONCURRENCY ?= 4
+bench-qa: bootstrap ## QA-quality sweep: questions × modes × N samples [BENCH_QA_N=3 BENCH_QA_LIMIT=N BENCH_QA_MODES=... BENCH_QA_CONCURRENCY=4]
 	PYTHONUNBUFFERED=1 $(PY) bench/qa_sweep.py \
 	    --questions $(BENCH_QA_QUESTIONS) \
 	    --shards-dir $(SHARDS_DIR) \
@@ -191,7 +192,21 @@ bench-qa: bootstrap ## QA-quality sweep: questions × modes × N samples [BENCH_
 	    --top-k $(QUERY_TOP_K) \
 	    --modes $(BENCH_QA_MODES) \
 	    --limit $(BENCH_QA_LIMIT) \
-	    --n $(BENCH_QA_N)
+	    --n $(BENCH_QA_N) \
+	    --concurrency $(BENCH_QA_CONCURRENCY)
+
+# Smoke fixture: 5 questions, all anchor classes, all currently failing
+# pointer mode 100% while JSON aces 100%. Inner loop for prompt iteration.
+# ~30s wall-clock at concurrency=4. Use this between full sweeps.
+bench-qa-smoke: bootstrap ## quick 5-question smoke (all anchor classes; ~30s)
+	PYTHONUNBUFFERED=1 $(PY) bench/qa_sweep.py \
+	    --questions bench/qa_questions_smoke.txt \
+	    --shards-dir $(SHARDS_DIR) \
+	    --out-dir $(BENCH_QA_OUT) \
+	    --top-k $(QUERY_TOP_K) \
+	    --modes $(BENCH_QA_MODES) \
+	    --n 1 \
+	    --concurrency $(BENCH_QA_CONCURRENCY)
 
 test-live: bootstrap ## live QA quality tests against Hermes (gated; ~1 min)
 	ABORIST_LIVE_TESTS=1 ABORIST_LIVE_SHARDS_DIR=$(SHARDS_DIR) \
@@ -371,7 +386,7 @@ stats: bootstrap ## counts: documents, chunks, edges, audit chain
 	$(ABORIST) --db $(DB) stats
 
 test: bootstrap ## run pytest suite (excludes opt-in crawler tests)
-	$(VENV)/bin/pytest -q --ignore=tests/crawler
+	$(VENV)/bin/pytest -q --ignore=tests/crawler -n auto
 
 # Crawler tests are off-by-default — they hit the network in many cases
 # and require the heavy [crawler] extras (aiohttp, bs4, lxml, etc.).

aborist/qa/dag.py

@@ -45,7 +45,13 @@ from aborist.merkle import MerkleTree
 
 
 def _sha256_hex(s: str) -> str:
-    return hashlib.sha256(s.encode("utf-8")).hexdigest()
+    # ``errors='surrogatepass'`` lets lone UTF-16 surrogates through as
+    # their WTF-8 form. Hermes occasionally emits text with unpaired
+    # surrogates inside multi-byte sequences; bare ``.encode('utf-8')``
+    # raises UnicodeEncodeError on those, which previously aborted the
+    # run with no Merkle root. The hash stays deterministic because the
+    # WTF-8 byte sequence is reversible & unique per input.
+    return hashlib.sha256(s.encode("utf-8", errors="surrogatepass")).hexdigest()
 
 
 def _canonical_json(obj) -> str:

aborist/qa/evidence.py

@@ -51,7 +51,9 @@ from aborist.merkle import MerkleTree
 
 
 def _sha256_hex(s: str) -> str:
-    return hashlib.sha256(s.encode("utf-8")).hexdigest()
+    # ``errors='surrogatepass'`` for model-output text containing lone
+    # UTF-16 surrogates; same rationale as ``aborist.qa.dag._sha256_hex``.
+    return hashlib.sha256(s.encode("utf-8", errors="surrogatepass")).hexdigest()
 
 
 @dataclass(frozen=True)

aborist/qa/keys.py

@@ -28,7 +28,9 @@ from aborist.document import canonicalize
 
 
 def _sha256(s: str) -> str:
-    return hashlib.sha256(s.encode("utf-8")).hexdigest()
+    # ``errors='surrogatepass'`` survives lone UTF-16 surrogates from
+    # model output; same rationale as ``aborist.qa.dag._sha256_hex``.
+    return hashlib.sha256(s.encode("utf-8", errors="surrogatepass")).hexdigest()
 
 
 def _canonical_json(obj) -> str:

aborist/store.py

@@ -936,7 +936,7 @@ def chain_audit_events(
         h = hashlib.sha256()
         if prev is not None:
             h.update(bytes.fromhex(prev))
-        h.update(body_json.encode("utf-8"))
+        h.update(body_json.encode("utf-8", errors="surrogatepass"))
         event_hash = h.hexdigest()
         rows.append(
             (
@@ -973,7 +973,7 @@ def append_audit(
     h = hashlib.sha256()
     if prev is not None:
         h.update(bytes.fromhex(prev))
-    h.update(body_json.encode("utf-8"))
+    h.update(body_json.encode("utf-8", errors="surrogatepass"))
     event_hash = h.hexdigest()
     conn.execute(
         "INSERT INTO audit_events (event_hash, prev_event_hash, event_type, subject_root, body, ts) "

bench/qa_questions_smoke.txt

@@ -0,0 +1,12 @@
+# Smoke fixture — 5 questions covering all anchor classes (date, relation,
+# entity-list, count/place, why-cause), each chosen because the 2026-05-02
+# bench shows pointer mode failing 100% (HH) while JSON mode aces 100% (SS).
+# These are the load-bearing test cases for prompt-iteration loops aimed at
+# closing the pointer-vs-JSON gap. Run with --concurrency 4 --n 1 → ~30s
+# wall-clock. The full bench (71 questions) stays the scoreboard; this is
+# the inner loop.
+when did the soviet union dissolve?
+where is mount kilimanjaro located?
+who painted the mona lisa?
+who were the original seven mercury astronauts?
+why did the dinosaurs go extinct?

pyproject.toml

@@ -51,6 +51,7 @@ crawler = [
 dev = [
   "pytest>=8",
   "pytest-asyncio>=0.23",
+  "pytest-xdist>=3.5",
   "aborist[html]",
   "aborist[wikitext]",
   "aborist[mesh]",