engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(bench): consolidate 2026-05-02 journal into single narrative

Browse source 
The journal had two stapled sections — the morning 11:31Z bench
(n=2 cell-grouped) followed by an 'Update' divider and the
afternoon 15:07Z bench (n=3 sample-shuffled). Two parts that
made readers walk past a horizontal rule to compare numbers, and
duplicated the directive-coverage / outputs / per-mode-rec
sections.

Rewrote as one coherent narrative:

  - Frontmatter mentions both stamps in one table (when/what/wall)
  - 'Hardening' section walks the chronology in two beats:
    * pre-11:31Z (Rule 8, warrant gen, ladder, etc.)
    * 11:31Z → 15:07Z (Sprint 1b, Sprint 2, DRY, keep-alive,
      sample-shuffle, --resume, concurrency-sweep, surrogate v2)
  - 'Aggregate' = authoritative 15:07Z table
  - 'Δ across the day' = 3-column comparison
    (2026-04-30 → 11:31Z → 15:07Z) with net deltas
  - 'Per-bucket strict-rate' = 15:07Z bucket data
  - 'Recommended context budget' = final
  - 'Pointer-mode signal' kept (lazy-anchor analysis still valid)
  - 'Wall-clock & throughput evolution' = both benches in one
    table + concurrency sweep table
  - 'Errors' = surrogate story across both benches
  - 'Verdict' = final recommendation
  - 'Outputs' = both jsonl files

212 lines → 175 lines (-36). One source of truth, one read.
This commit is contained in:
russell@unturf.com 2026-05-02 12:13:27 -04:00
parent a0e916ee1b
commit 238f5b0470
No known key found for this signature in database
1 changed files with 112 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

260
docs/qa-modes-bench-2026-05-02.md
View file

@ -1,90 +1,111 @@
# QA-modes bench — 2026-05-02 (post-Rule-8, post-frame, post-warrant-generalization)
> **Update — 2026-05-02T15:07Z full bench at c=4 with sample-shuffle.**
> A second sweep landed after Sprint 1b (per-mode `max_context_chars`),
> Sprint 2 (pointer chunk-specificity rule), the DRY collapse of lattice
> prompts, the httpx keep-alive client refactor, and the bench harness's
> new sample-level shuffled scheduling. See bottom section for the
> updated headlines & wall-clock.
# QA-modes bench — 2026-05-02
**Date:** 2026-05-02
**Bench:** `bench/qa_sweep.py`, 71 questions × 3 modes × 2 samples = 426 LLM calls (213 cells)
**Endpoint:** `https://hermes.ai.unturf.com/v1` (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx)
**Corpus:** Wikipedia 2003-05-16 cur snapshot, sharded under `~/.aborist/shards`
**Wall-clock:** ~51 min with `--concurrency 4`
**Verifier hardening since 2026-04-30:** Rule 8 title-relevance promoted to hard check, anchor-class warrant generalized to entity-list / count / why-cause shapes (Ticket #000003), retrieval-plan hash bound into the run-DAG (Ticket #000001), reference-frame polarity contract (Ticket #000002), four-rung ladder display (Ticket #000005), bench harness extended with directive coverage + log-scale buckets to 1M (Ticket #000004).
## Aggregate
Two sweeps landed today, each 71 questions × 3 modes:
| stamp | samples | concurrency | scheduling | runs | wall-clock |
|-------|---------|-------------|------------|------|------------|
| `11:31Z` | n=2 | c=4 | cell-grouped | 426 | 51 min |
| `15:07Z` | n=3 | c=4 | sample-shuffled, `--seed 0` | 639 | **48 min** |
The 15:07Z sweep is the **authoritative state of the substrate** at end-of-day; the 11:31Z sweep is the intermediate witnessed before the second wave of hardening landed. Headlines below show both.
## Hardening since the prior bench (2026-04-30 post-retry)
Substrate-side (between `2026-04-30T18:46Z` and `2026-05-02T11:31Z`):
- Rule 8 — title-relevance promoted to hard verifier check (`TITLE_MISMATCH` violation demotes `STRICT → HYBRID` when no cited evidence's source title shares a stemmed token with the claim).
- Anchor-class warrant generalized to entity-list / count / why-cause shapes (Ticket #000003).
- Retrieval-plan hash bound into the run-DAG retrieval stage (Ticket #000001).
- Reference-frame polarity contract (Ticket #000002).
- Four-rung ladder display (Ticket #000005) — `POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED reserved)` mapped from the v9.8 audit_mode trichotomy at render time.
- Bench harness extended with directive coverage + log-scale buckets to 1M (Ticket #000004).
Substrate-side (between `2026-05-02T11:31Z` and `2026-05-02T15:07Z`):
- **Sprint 1b** — per-mode `max_context_chars`. Bench's recommended-context-budget table flows back into `DEFAULT_QUERY_POLICY["max_context_chars_by_mode"]`: quote 24 KB, pointer 24 KB, JSON 48 KB. Folds into `governance_policy_hash`.
- **Sprint 2** — pointer Rule 9: chunk-specificity instruction added to the lattice-pointer system prompt.
- **DRY collapse** — the four lattice prompts (system + grounding × pointer + JSON) lifted to `aborist/qa/prompts.py` as a single source of truth, imported by both `runner.DEFAULT_POLICY` and `query.DEFAULT_QUERY_POLICY`.
- **httpx persistent client** — the chat-completion path used to construct a fresh `httpx.Client` per call, paying a TLS handshake every request. Move to `__init__`; HTTP/1.1 keep-alive across calls. Save 1-2 min on a 426-call bench.
- **Sample-level shuffled bench scheduling** — every `(question, mode, sample_idx)` is a task, shuffled with `--seed`, dispatched concurrently. Per-cell `Lock` dict serializes burn-then-write on the shared cache_key. True i.i.d. n=3 variance; vLLM batcher fed a diverse request stream.
- **Bench `--resume`** — read existing JSONL, skip done tasks, append fresh rows. Stop/start-able.
- **Concurrency sweep on the smoke fixture.** vLLM peaks at c=3-4; saturates badly past c=4 (see Wall-clock & throughput below).
- **UTF-16 surrogate fix v2** (`41d1d9b`) — corpus chunks with lone surrogates broke httpx's outbound JSON encode. Scrub at the client boundary by WTF-8 → UTF-8-with-replace roundtrip. (The earlier v1 fix `3b91223` hardened the OUTPUT side — SHA-256 hashers — but missed the INPUT side. Both paths now safe.)
## Aggregate (15:07Z, authoritative)
| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | mean ratio | mean latency |
|------|------|--------|--------|------------|-----|-------------|------------|--------------|
| `quote` | 142 | 71 | 45 | 26 | 0 | **0.50** | 0.705 | 29.1s |
| `claim_lattice_pointer` | 142 | 32 | 83 | 25 | 2 | **0.23** | 0.564 | 26.7s |
| `claim_lattice` (JSON) | 142 | 62 | 51 | 27 | 2 | **0.44** | 0.698 | 28.2s |
| `quote` | 213 | 116 | 51 | 46 | 0 | **0.54** | 0.699 | **16.9 s** |
| `claim_lattice_pointer` | 213 | 43 | 146 | 21 | 3 | **0.20** | 0.642 | 18.5 s |
| `claim_lattice` (JSON) | 213 | 89 | 82 | 39 | 3 | **0.42** | 0.698 | 18.3 s |
**Δ vs 2026-04-30** (post-retry rerun reference):
## Δ across the day
| mode | 2026-04-30 | 2026-05-02 | Δ |
|------|-----------|-----------|----|
| `quote` | 0.47 | **0.50** | +3pp |
| `claim_lattice_pointer` | 0.24 | **0.23** | −1pp |
| `claim_lattice` (JSON) | 0.50 | **0.44** | −6pp |
| mode | 2026-04-30 (post-retry) | 2026-05-02T11:31Z | 2026-05-02T15:07Z | net Δ |
|------|------------------------|-------------------|-------------------|-------|
| `quote` | 0.47 | 0.50 (+3pp) | **0.54** (+4pp) | **+7pp** |
| `claim_lattice_pointer` | 0.24 | 0.23 (−1pp) | 0.20 (−3pp) | −4pp |
| `claim_lattice` (JSON) | 0.50 | 0.44 (−6pp) | 0.42 (−2pp) | −8pp |
(Earlier message used the pre-retry 2026-04-30 numbers (0.36 / 0.49) — the post-retry rerun in the same journal is the correct reference; pointer mode is essentially flat, JSON mode took a 6pp honesty cost.)
**Quote climbed the most across the day**, ending at 0.54. The Sprint 1b 24 KB cap surfaces tighter retrievals that quote can ground verbatim (the bucket data confirms — quote's peak migrated to 8-16 KB at 0.58 strict-rate).
**Pointer & JSON each took an honesty cost.** Rule 8 promotion + warrant-class generalization demote more cases that *would* have classified STRICT under the looser pre-2026-05-02 verifier. The architectural win — 99% directive coverage on lattice modes — is the price for those drops; false-positive STRICT was corruption, and we converted it to honest HYBRID.
## Per-bucket strict-rate (15:07Z)
| mode | bucket | runs | strict-rate | note |
|------|--------|------|-------------|------|
| `quote` | 8-16 KB | 105 | **0.58** | peak |
| `quote` | 16-32 KB | 105 | 0.52 | |
| `claim_lattice_pointer` | 16-32 KB | 210 | **0.20** | peak (only bucket) |
| `claim_lattice` (JSON) | 32-64 KB | 99 | **0.48** | peak |
| `claim_lattice` (JSON) | 16-32 KB | 108 | 0.38 | |
**Sprint 1b's intent confirmed.** JSON's peak at 32-64 KB justifies the 48 KB default. Quote's peak migrated to 8-16 KB after the 24 KB cap; the cap may be tighter than optimal — quote could plausibly be cut to 16 KB (mid of 8-16 KB bucket) for another small lift in a follow-up sprint.
`claim_lattice` (JSON) **peaks at LARGER context** than the other modes. The structured per-claim evidence linkage that JSON enforces benefits from more evidence per claim. Pointer & quote degrade past 16-32 KB.
## Recommended context budget (per-mode)
5-run minimum to reduce noise; from the 15:07Z bucket data:
| mode | peak bucket | strict-rate | n |
|------|-------------|-------------|---|
| `quote` | 8-16 KB | 0.58 | 105 |
| `claim_lattice_pointer` | 16-32 KB | 0.20 | 210 |
| `claim_lattice` (JSON) | **32-64 KB** | **0.48** | 99 |
These flow into `DEFAULT_QUERY_POLICY["max_context_chars_by_mode"]` (currently 24 / 24 / 48 KB — mid of each peak bucket). When the bench shifts those peaks, retune the policy and let `governance_policy_hash` partition the cache.
## Directive coverage (seven-point program)
| mode | D2 pointer | D3 cti-ready | D4 ev-map bound | D6 warrant | D7 honest label |
|------|-----------|--------------|-----------------|-----------|----------------|
| `quote` | 0/142 (0%) | 0/142 (0%) | 142/142 (100%) | 0/142 (0%) | 142/142 (100%) |
| `claim_lattice_pointer` | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) |
| `claim_lattice` | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) |
| `quote` | 0/213 (0%) | 0/213 (0%) | 213/213 (100%) | 0/213 (0%) | 213/213 (100%) |
| `claim_lattice_pointer` | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) |
| `claim_lattice` | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) |
Lattice modes hit **99% on every observable directive**; the 1% gap is the 2 errors per mode (UTF-16 surrogate from the Roman Empire question — model output had lone surrogates that broke UTF-8 encoding). Quote mode shows 0% on D2/D3/D6 by construction (those are claim-lattice-only directives) and 100% on D4/D7.
## Strict-rate by prompt size
Buckets capacity in `messages_total` chars. Tells you whether the model strict-rate degrades with input size.
| mode | bucket | runs | STRICT | strict-rate | mean evidence chars | mean answer chars |
|------|--------|------|--------|-------------|---------------------|-------------------|
| `quote` | 8-16KB | 4 | 2 | 0.50 | 8460 | 267 |
| `quote` | 16-32KB | 26 | 14 | 0.54 | 26946 | 1095 |
| `quote` | 32-64KB | 112 | 55 | 0.49 | 37038 | 783 |
| `claim_lattice_pointer` | 8-16KB | 2 | 0 | 0.00 | 13066 | 2335 |
| `claim_lattice_pointer` | 16-32KB | 62 | 18 | 0.29 | 22846 | 1522 |
| `claim_lattice_pointer` | 32-64KB | 76 | 14 | 0.18 | 36205 | 931 |
| `claim_lattice` | 8-16KB | 2 | 0 | 0.00 | 13066 | 140 |
| `claim_lattice` | 16-32KB | 66 | 25 | 0.38 | 23274 | 1044 |
| `claim_lattice` | 32-64KB | 72 | 37 | 0.51 | 36555 | 991 |
## Recommended context budget
Per-mode peak strict-rate bucket (5-run minimum to reduce noise):
| mode | peak bucket | strict-rate | n |
|------|-------------|-------------|---|
| `quote` | 16-32KB | 0.54 | 26 |
| `claim_lattice_pointer` | 16-32KB | 0.29 | 62 |
| `claim_lattice` | **32-64KB** | **0.51** | **72** |
Notable: `claim_lattice` (JSON) **peaks at LARGER context** than the other modes. The structured per-claim evidence linkage that JSON enforces appears to *benefit* from more evidence per claim. Pointer & quote degrade past 16-32KB.
Lattice modes hit **99% on every observable directive**; the 1% gap is the 6 surrogate errors (already fixed in `41d1d9b`, next bench will hit 100%). Quote shows 0% on D2 / D3 / D6 by construction (those are lattice-only directives) and 100% on D4 / D7.
## Pointer-mode signal — where the gap lives
Pointer mode's strict-rate is unchanged from 2026-04-30 in absolute terms (0.24 → 0.23) but the failure mode has shifted. The **lazy-anchor ratio histogram** is the load-bearing signal:
Pointer mode's strict-rate did not lift on Sprint 2's chunk-specificity prompt nudge. The load-bearing failure pattern is **lazy-anchoring**: the model cites a topic-overview chunk for every claim instead of the specific chunk that supports each individual claim. Lazy-anchor ratio histogram (11:31Z bench, 142 pointer rows):
| `lazy_anchor_ratio` bucket | pointer-mode rows |
|----------------------------|-------------------|
| `lazy_anchor_ratio` bucket | rows |
|----------------------------|------|
| `0.00` | 25 |
| `<0.25` | 9 |
| `<0.5` | 11 |
| `<0.75` | 46 |
| `>=0.75` | **49** |
| `>=0.75` | **49** (35%) |
49 of 142 pointer rows (35%) anchored ≥75% of claims to a topic-level chunk rather than the specific claim-supporting chunk. **This is the failure pattern Rule 8 was promoted to catch** — when the lazy-anchor target's source title shares zero stemmed content tokens with the claim, the run demotes STRICT → HYBRID via `TITLE_MISMATCH`.
35% of pointer rows anchor ≥75% of claims to a topic chunk. **This is the failure pattern Rule 8 was promoted to catch** — when the lazy-anchor target's source title shares zero stemmed content tokens with the claim, the run demotes `STRICT → HYBRID` via `TITLE_MISMATCH`. The honesty surfaces; the structural lift requires a stronger intervention (evidence-map ranking change, model fine-tune, or a different prompt frame).
19 of 71 questions show JSON > pointer by 50pp+:
19 of 71 questions show JSON ≥ 50pp above pointer, including:
```
pointer=HH json=SS did napoleon really die on saint helena?
@ -107,105 +128,48 @@ pointer=HH json=SS why did the dinosaurs go extinct?
`subject_in_answer` is high across all rungs (STRICT 100%, HYBRID 92%, UNGROUNDED 84%) — the subject is in the answer; the answer just doesn't anchor cleanly to a single chunk. This is **structural-honesty signal**, not deflection.
## Verdict
## Wall-clock & throughput evolution
**Pointer-mode strict-rate is essentially flat from the prior bench.** The honesty cost of Rule 8 hits both lattice modes; JSON's structured-per-claim linkage absorbs it better than pointer's prose-with-tags.
| stamp | concurrency | scheduling | n | tasks | wall-clock | tasks/min | mean latency (mode) |
|-------|-------------|-----------|---|-------|------------|-----------|---------------------|
| 11:31Z | 4 | cell-grouped | 2 | 426 | 51 min | 8.4 | 26-29 s |
| 15:07Z | 4 | sample-shuffled | 3 | 639 | **48 min** | **13.3** | **17-19 s** |
**JSON's 6pp drop is the real news.** Likely causes:
1. Rule 8 demoting more cases (same as pointer, less severe).
2. Frame detector / polarity preamble changing prompt budget on reference-frame questions.
3. Anchor-class warrant generalization (entity-list / count / why-cause) catching cases the prior bench missed entirely.
Sample-level shuffled scheduling at the same concurrency delivers **+58% throughput**. n=3 (50% more total work) ran in 6% LESS wall-clock than n=2 cell-grouped. Per-call mean latency dropped 35-42% across all modes (quote 29.1 s → 16.9 s, pointer 26.7 s → 18.5 s, JSON 28.2 s → 18.3 s) because vLLM's continuous batcher fills better when fed a diverse, uncorrelated request stream.
The directive-coverage table (99% across every observable D2-D7 on lattice modes) is the architectural win. The strict-rate cost is the price of that calibration.
### Concurrency sweep (smoke fixture, 15-task)
## Per-mode recommendation
| c | wall-clock | tasks/min | normalized |
|---|------------|-----------|------------|
| 3 | 102 s | 8.8 | **1.00** (peak) |
| 4 | 106 s | 8.5 | 0.96 (chosen — within 4% of peak, more forgiving on single-call hiccups) |
| 5 | 119 s | 7.6 | 0.86 |
| 6 | 185 s | 4.9 | 0.55 (vLLM batching ceiling, brutal) |
- **`claim_lattice` (JSON)** stays the recommended default for cache-grade provenance. Best strict-rate among lattice modes (0.44), JSON-mode wins on structured per-claim discipline, peaks at larger context (32-64KB) so it scales with evidence growth.
- **`quote` mode** is best for raw lexical-grounding signal (0.50 strict-rate) but doesn't surface anchor-class warrants — use when the four-rung ladder isn't needed.
- **`claim_lattice_pointer`** stays useful for low-context-budget scenarios (16-32KB peak) and for prose-distribution models that struggle with JSON grammar; the lazy-anchor honesty cost is the trade.
## Concurrency landed alongside this bench
The bench harness gained a `--concurrency N` flag (commit `0870af6`) for parallel (question, mode) cells. First attempt at concurrency=4 surfaced a thread-safety defect in `aborist/compress.py`: the module-level `ZstdCompressor` / `ZstdDecompressor` singletons were not thread-safe — concurrent decompress calls corrupted the libzstd context and raised `ZstdError: Data corruption detected` on ~19% of retrievals. Fixed via `threading.local()` caches; same commit.
Wall-clock for this 426-run sweep dropped from ~107 min projected sequential to ~51 min actual at concurrency=4. Speedup is below theoretical 4× because vLLM's continuous batcher queues at 4 concurrent requests, stretching individual call latency. Concurrency=2 may give better per-call latency at lower throughput; concurrency=8 may give higher throughput at the cost of per-call latency. Not yet tuned.
## Outputs
- `bench/qa_results/2026-05-02T11-31-55Z.jsonl` — 426 rows, durable artifact
- `bench/qa_results/2026-05-02T11-31-55Z.md` — auto-generated summary
(Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.)
---
# Update — 15:07Z full bench (post-Sprint-1b/Sprint-2/DRY/keep-alive/shuffle)
**Stamp:** `2026-05-02T15-07-24Z`
**Bench:** 71 questions × 3 modes × 3 samples = 639 LLM calls (213 cells)
**Wall-clock:** **48 min** at `--concurrency 4 --seed 0` (sample-level shuffled scheduling)
**Hardening since the 11:31Z bench above:**
- Sprint 1b: per-mode `max_context_chars` (quote/pointer 24KB, JSON 48KB) folded into `governance_policy_hash`
- Sprint 2: pointer Rule 9 — chunk-specificity instruction
- DRY: lattice prompts collapsed to `aborist/qa/prompts.py` (single source of truth)
- httpx persistent client (TLS handshake amortization across calls)
- Bench harness sample-level shuffled scheduling with per-cell locks (true i.i.d. n=3 variance estimation)
- Concurrency sweep on smoke fixture: c=3 peak, c=4 within 4% (chosen for full bench), c=5 12% slower, c=6 45% slower
## Aggregate
| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | mean ratio | mean latency |
|------|------|--------|--------|------------|-----|-------------|------------|--------------|
| `quote` | 213 | 116 | 51 | 46 | 0 | **0.54** | 0.699 | **16.9s** |
| `claim_lattice_pointer` | 213 | 43 | 146 | 21 | 3 | **0.20** | 0.642 | 18.5s |
| `claim_lattice` (JSON) | 213 | 89 | 82 | 39 | 3 | **0.42** | 0.698 | 18.3s |
## Δ vs 11:31Z bench (same date, 4h earlier)
| mode | 11:31Z | 15:07Z | Δ |
|------|--------|--------|----|
| `quote` | 0.50 | **0.54** | +4pp |
| `claim_lattice_pointer` | 0.23 | **0.20** | −3pp |
| `claim_lattice` (JSON) | 0.44 | **0.42** | −2pp |
Quote moved sharply UP. Pointer & JSON dropped slightly — within bench variance for small samples but worth noting. Quote's lift is the cleanest signal: the per-mode 24KB cap is exactly the bucket where quote peaks (8-16KB at 0.58 in this bench), so smaller-context-default surfaces tighter retrievals that quote can ground verbatim.
Pointer's slight drop is consistent with Sprint 2's smoke result — the chunk-specificity rule didn't materially shift Hermes-3-8B's lazy-anchoring at n=3 either. The structural lift will need a stronger intervention (LLM-side fine-tune, evidence-map ranking change, or a different prompt frame).
JSON's slight drop is within bench noise; structurally JSON should benefit most from the 32-64KB peak budget Sprint 1b targeted, and the per-bucket data confirms (JSON 32-64KB strict-rate 0.48 vs 16-32KB at 0.38).
## Per-bucket strict-rate
| mode | bucket | runs | strict-rate |
|------|--------|------|-------------|
| `quote` | 8-16KB | 105 | **0.58** ← peak |
| `quote` | 16-32KB | 105 | 0.52 |
| `claim_lattice_pointer` | 16-32KB | 210 | **0.20** ← peak (only bucket) |
| `claim_lattice` (JSON) | 32-64KB | 99 | **0.48** ← peak |
| `claim_lattice` (JSON) | 16-32KB | 108 | 0.38 |
**Sprint 1b's intent confirmed.** JSON's peak at 32-64KB justifies the 48KB default. Quote's peak migrated to 8-16KB after the 24KB cap; the cap may be tighter than optimal — quote could plausibly be cut to 16KB (mid of 8-16KB bucket) for another small lift in a follow-up sprint.
## Directive coverage
Lattice modes hit **99% on every observable D2-D7**; the 1% gap is the 6 surrogate errors. Quote shows 0% on D2/D3/D6 by construction (those are lattice-only) and 100% on D4/D7.
vLLM saturates at c=3-4 on this endpoint. More concurrent requests fill the batch better, but only up to the point where per-call latency growth outpaces parallelism gain. Re-sweep when the endpoint is upgraded, when corpus shape changes the average context size, or when other tenants change the queue.
## Errors
6 errors (3 per lattice mode). All on the same question — `tell me about the roman empire`. Root cause traced & fixed in `41d1d9b`: lone UTF-16 surrogates in the Wikipedia chunk content broke httpx's outbound JSON encode (different path from the earlier 3b91223 fix which hardened SHA-256 hashers on the OUTPUT side). Next bench will land 0 errors.
Across both benches: **2 errors at 11:31Z, 6 errors at 15:07Z** — all on the same question (`tell me about the roman empire`). Lone UTF-16 surrogates in Wikipedia chunk content, two distinct paths:
## Wall-clock & throughput
- v1 (`3b91223`) — hardened SHA-256 hashers on the OUTPUT side (`aborist/qa/dag.py`, `keys.py`, `evidence.py`, `store.py` audit chain) with `errors='surrogatepass'` so the run-DAG roots survive surrogate-bearing model output.
- v2 (`41d1d9b`) — scrubs message content INSIDE `OpenAICompatibleClient.chat_completion` before httpx's outbound JSON encode. The corpus chunk text was the path; httpx's `.encode('utf-8')` on the request body raised before the call left the client.
| run | concurrency | scheduling | tasks | wall | tasks/min |
|-----|-------------|-----------|-------|------|-----------|
| 11:31Z | 4 | cell-grouped | 426 (n=2) | 51 min | 8.4 |
| 15:07Z | 4 | sample-shuffled | 639 (n=3) | 48 min | **13.3** |
Verified: `tell me about the roman empire` under `claim_lattice` now classifies HYBRID 5/7 instead of erroring. Next bench will land 0 errors.
Sample-level shuffled scheduling at the same concurrency delivers **+58% throughput** by feeding vLLM's continuous batcher a properly diverse request stream. The cell-grouped predecessor correlated batches by cache_key; vLLM couldn't fill efficiently. n=3 instead of n=2 is **50% more total work**, executed in **6% less wall-clock**. Per-call mean latency dropped 35-42% across all modes (quote 29.1s→16.9s, pointer 26.7s→18.5s, JSON 28.2s→18.3s).
## Verdict & per-mode recommendation
**Quote leads on raw lexical grounding (0.54 strict-rate).** Best when the four-rung ladder isn't needed and the answer is short-and-cite-able. Doesn't surface anchor-class warrants.
**JSON leads among lattice modes (0.42).** Recommended default for cache-grade provenance. Structured per-claim evidence linkage; peaks at larger context (32-64 KB) so it scales with evidence growth.
**Pointer (0.20) stays useful for low-context-budget scenarios** (16-32 KB peak) and prose-distribution models that struggle with JSON grammar; the lazy-anchor honesty cost is the trade.
**The directive-coverage table (99% across every observable D2-D7 on lattice modes) is the architectural win.** False-positive STRICT was corruption; converting it to honest HYBRID was the price of calibration. Strict-rate slipped on lattice modes; the substrate is more honest.
## Outputs
- `bench/qa_results/2026-05-02T15-07-24Z.jsonl` — 639 rows
- `bench/qa_results/2026-05-02T15-07-24Z.md` — auto-generated summary
- `bench/qa_results/2026-05-02T11-31-55Z.{jsonl,md}` — 426 rows
- `bench/qa_results/2026-05-02T15-07-24Z.{jsonl,md}` — 639 rows
(Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.)