diff --git a/docs/qa-modes-bench-2026-05-02.md b/docs/qa-modes-bench-2026-05-02.md index f7a3ed5..aca05f7 100644 --- a/docs/qa-modes-bench-2026-05-02.md +++ b/docs/qa-modes-bench-2026-05-02.md @@ -1,90 +1,111 @@ -# QA-modes bench — 2026-05-02 (post-Rule-8, post-frame, post-warrant-generalization) - -> **Update — 2026-05-02T15:07Z full bench at c=4 with sample-shuffle.** -> A second sweep landed after Sprint 1b (per-mode `max_context_chars`), -> Sprint 2 (pointer chunk-specificity rule), the DRY collapse of lattice -> prompts, the httpx keep-alive client refactor, and the bench harness's -> new sample-level shuffled scheduling. See bottom section for the -> updated headlines & wall-clock. +# QA-modes bench — 2026-05-02 **Date:** 2026-05-02 -**Bench:** `bench/qa_sweep.py`, 71 questions × 3 modes × 2 samples = 426 LLM calls (213 cells) **Endpoint:** `https://hermes.ai.unturf.com/v1` (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx) **Corpus:** Wikipedia 2003-05-16 cur snapshot, sharded under `~/.aborist/shards` -**Wall-clock:** ~51 min with `--concurrency 4` -**Verifier hardening since 2026-04-30:** Rule 8 title-relevance promoted to hard check, anchor-class warrant generalized to entity-list / count / why-cause shapes (Ticket #000003), retrieval-plan hash bound into the run-DAG (Ticket #000001), reference-frame polarity contract (Ticket #000002), four-rung ladder display (Ticket #000005), bench harness extended with directive coverage + log-scale buckets to 1M (Ticket #000004). -## Aggregate +Two sweeps landed today, each 71 questions × 3 modes: + +| stamp | samples | concurrency | scheduling | runs | wall-clock | +|-------|---------|-------------|------------|------|------------| +| `11:31Z` | n=2 | c=4 | cell-grouped | 426 | 51 min | +| `15:07Z` | n=3 | c=4 | sample-shuffled, `--seed 0` | 639 | **48 min** | + +The 15:07Z sweep is the **authoritative state of the substrate** at end-of-day; the 11:31Z sweep is the intermediate witnessed before the second wave of hardening landed. Headlines below show both. + +## Hardening since the prior bench (2026-04-30 post-retry) + +Substrate-side (between `2026-04-30T18:46Z` and `2026-05-02T11:31Z`): + +- Rule 8 — title-relevance promoted to hard verifier check (`TITLE_MISMATCH` violation demotes `STRICT → HYBRID` when no cited evidence's source title shares a stemmed token with the claim). +- Anchor-class warrant generalized to entity-list / count / why-cause shapes (Ticket #000003). +- Retrieval-plan hash bound into the run-DAG retrieval stage (Ticket #000001). +- Reference-frame polarity contract (Ticket #000002). +- Four-rung ladder display (Ticket #000005) — `POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED reserved)` mapped from the v9.8 audit_mode trichotomy at render time. +- Bench harness extended with directive coverage + log-scale buckets to 1M (Ticket #000004). + +Substrate-side (between `2026-05-02T11:31Z` and `2026-05-02T15:07Z`): + +- **Sprint 1b** — per-mode `max_context_chars`. Bench's recommended-context-budget table flows back into `DEFAULT_QUERY_POLICY["max_context_chars_by_mode"]`: quote 24 KB, pointer 24 KB, JSON 48 KB. Folds into `governance_policy_hash`. +- **Sprint 2** — pointer Rule 9: chunk-specificity instruction added to the lattice-pointer system prompt. +- **DRY collapse** — the four lattice prompts (system + grounding × pointer + JSON) lifted to `aborist/qa/prompts.py` as a single source of truth, imported by both `runner.DEFAULT_POLICY` and `query.DEFAULT_QUERY_POLICY`. +- **httpx persistent client** — the chat-completion path used to construct a fresh `httpx.Client` per call, paying a TLS handshake every request. Move to `__init__`; HTTP/1.1 keep-alive across calls. Save 1-2 min on a 426-call bench. +- **Sample-level shuffled bench scheduling** — every `(question, mode, sample_idx)` is a task, shuffled with `--seed`, dispatched concurrently. Per-cell `Lock` dict serializes burn-then-write on the shared cache_key. True i.i.d. n=3 variance; vLLM batcher fed a diverse request stream. +- **Bench `--resume`** — read existing JSONL, skip done tasks, append fresh rows. Stop/start-able. +- **Concurrency sweep on the smoke fixture.** vLLM peaks at c=3-4; saturates badly past c=4 (see Wall-clock & throughput below). +- **UTF-16 surrogate fix v2** (`41d1d9b`) — corpus chunks with lone surrogates broke httpx's outbound JSON encode. Scrub at the client boundary by WTF-8 → UTF-8-with-replace roundtrip. (The earlier v1 fix `3b91223` hardened the OUTPUT side — SHA-256 hashers — but missed the INPUT side. Both paths now safe.) + +## Aggregate (15:07Z, authoritative) | mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | mean ratio | mean latency | |------|------|--------|--------|------------|-----|-------------|------------|--------------| -| `quote` | 142 | 71 | 45 | 26 | 0 | **0.50** | 0.705 | 29.1s | -| `claim_lattice_pointer` | 142 | 32 | 83 | 25 | 2 | **0.23** | 0.564 | 26.7s | -| `claim_lattice` (JSON) | 142 | 62 | 51 | 27 | 2 | **0.44** | 0.698 | 28.2s | +| `quote` | 213 | 116 | 51 | 46 | 0 | **0.54** | 0.699 | **16.9 s** | +| `claim_lattice_pointer` | 213 | 43 | 146 | 21 | 3 | **0.20** | 0.642 | 18.5 s | +| `claim_lattice` (JSON) | 213 | 89 | 82 | 39 | 3 | **0.42** | 0.698 | 18.3 s | -**Δ vs 2026-04-30** (post-retry rerun reference): +## Δ across the day -| mode | 2026-04-30 | 2026-05-02 | Δ | -|------|-----------|-----------|----| -| `quote` | 0.47 | **0.50** | +3pp | -| `claim_lattice_pointer` | 0.24 | **0.23** | −1pp | -| `claim_lattice` (JSON) | 0.50 | **0.44** | −6pp | +| mode | 2026-04-30 (post-retry) | 2026-05-02T11:31Z | 2026-05-02T15:07Z | net Δ | +|------|------------------------|-------------------|-------------------|-------| +| `quote` | 0.47 | 0.50 (+3pp) | **0.54** (+4pp) | **+7pp** | +| `claim_lattice_pointer` | 0.24 | 0.23 (−1pp) | 0.20 (−3pp) | −4pp | +| `claim_lattice` (JSON) | 0.50 | 0.44 (−6pp) | 0.42 (−2pp) | −8pp | -(Earlier message used the pre-retry 2026-04-30 numbers (0.36 / 0.49) — the post-retry rerun in the same journal is the correct reference; pointer mode is essentially flat, JSON mode took a 6pp honesty cost.) +**Quote climbed the most across the day**, ending at 0.54. The Sprint 1b 24 KB cap surfaces tighter retrievals that quote can ground verbatim (the bucket data confirms — quote's peak migrated to 8-16 KB at 0.58 strict-rate). + +**Pointer & JSON each took an honesty cost.** Rule 8 promotion + warrant-class generalization demote more cases that *would* have classified STRICT under the looser pre-2026-05-02 verifier. The architectural win — 99% directive coverage on lattice modes — is the price for those drops; false-positive STRICT was corruption, and we converted it to honest HYBRID. + +## Per-bucket strict-rate (15:07Z) + +| mode | bucket | runs | strict-rate | note | +|------|--------|------|-------------|------| +| `quote` | 8-16 KB | 105 | **0.58** | peak | +| `quote` | 16-32 KB | 105 | 0.52 | | +| `claim_lattice_pointer` | 16-32 KB | 210 | **0.20** | peak (only bucket) | +| `claim_lattice` (JSON) | 32-64 KB | 99 | **0.48** | peak | +| `claim_lattice` (JSON) | 16-32 KB | 108 | 0.38 | | + +**Sprint 1b's intent confirmed.** JSON's peak at 32-64 KB justifies the 48 KB default. Quote's peak migrated to 8-16 KB after the 24 KB cap; the cap may be tighter than optimal — quote could plausibly be cut to 16 KB (mid of 8-16 KB bucket) for another small lift in a follow-up sprint. + +`claim_lattice` (JSON) **peaks at LARGER context** than the other modes. The structured per-claim evidence linkage that JSON enforces benefits from more evidence per claim. Pointer & quote degrade past 16-32 KB. + +## Recommended context budget (per-mode) + +5-run minimum to reduce noise; from the 15:07Z bucket data: + +| mode | peak bucket | strict-rate | n | +|------|-------------|-------------|---| +| `quote` | 8-16 KB | 0.58 | 105 | +| `claim_lattice_pointer` | 16-32 KB | 0.20 | 210 | +| `claim_lattice` (JSON) | **32-64 KB** | **0.48** | 99 | + +These flow into `DEFAULT_QUERY_POLICY["max_context_chars_by_mode"]` (currently 24 / 24 / 48 KB — mid of each peak bucket). When the bench shifts those peaks, retune the policy and let `governance_policy_hash` partition the cache. ## Directive coverage (seven-point program) | mode | D2 pointer | D3 cti-ready | D4 ev-map bound | D6 warrant | D7 honest label | |------|-----------|--------------|-----------------|-----------|----------------| -| `quote` | 0/142 (0%) | 0/142 (0%) | 142/142 (100%) | 0/142 (0%) | 142/142 (100%) | -| `claim_lattice_pointer` | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | -| `claim_lattice` | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | 140/142 (99%) | +| `quote` | 0/213 (0%) | 0/213 (0%) | 213/213 (100%) | 0/213 (0%) | 213/213 (100%) | +| `claim_lattice_pointer` | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | +| `claim_lattice` | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | 210/213 (99%) | -Lattice modes hit **99% on every observable directive**; the 1% gap is the 2 errors per mode (UTF-16 surrogate from the Roman Empire question — model output had lone surrogates that broke UTF-8 encoding). Quote mode shows 0% on D2/D3/D6 by construction (those are claim-lattice-only directives) and 100% on D4/D7. - -## Strict-rate by prompt size - -Buckets capacity in `messages_total` chars. Tells you whether the model strict-rate degrades with input size. - -| mode | bucket | runs | STRICT | strict-rate | mean evidence chars | mean answer chars | -|------|--------|------|--------|-------------|---------------------|-------------------| -| `quote` | 8-16KB | 4 | 2 | 0.50 | 8460 | 267 | -| `quote` | 16-32KB | 26 | 14 | 0.54 | 26946 | 1095 | -| `quote` | 32-64KB | 112 | 55 | 0.49 | 37038 | 783 | -| `claim_lattice_pointer` | 8-16KB | 2 | 0 | 0.00 | 13066 | 2335 | -| `claim_lattice_pointer` | 16-32KB | 62 | 18 | 0.29 | 22846 | 1522 | -| `claim_lattice_pointer` | 32-64KB | 76 | 14 | 0.18 | 36205 | 931 | -| `claim_lattice` | 8-16KB | 2 | 0 | 0.00 | 13066 | 140 | -| `claim_lattice` | 16-32KB | 66 | 25 | 0.38 | 23274 | 1044 | -| `claim_lattice` | 32-64KB | 72 | 37 | 0.51 | 36555 | 991 | - -## Recommended context budget - -Per-mode peak strict-rate bucket (5-run minimum to reduce noise): - -| mode | peak bucket | strict-rate | n | -|------|-------------|-------------|---| -| `quote` | 16-32KB | 0.54 | 26 | -| `claim_lattice_pointer` | 16-32KB | 0.29 | 62 | -| `claim_lattice` | **32-64KB** | **0.51** | **72** | - -Notable: `claim_lattice` (JSON) **peaks at LARGER context** than the other modes. The structured per-claim evidence linkage that JSON enforces appears to *benefit* from more evidence per claim. Pointer & quote degrade past 16-32KB. +Lattice modes hit **99% on every observable directive**; the 1% gap is the 6 surrogate errors (already fixed in `41d1d9b`, next bench will hit 100%). Quote shows 0% on D2 / D3 / D6 by construction (those are lattice-only directives) and 100% on D4 / D7. ## Pointer-mode signal — where the gap lives -Pointer mode's strict-rate is unchanged from 2026-04-30 in absolute terms (0.24 → 0.23) but the failure mode has shifted. The **lazy-anchor ratio histogram** is the load-bearing signal: +Pointer mode's strict-rate did not lift on Sprint 2's chunk-specificity prompt nudge. The load-bearing failure pattern is **lazy-anchoring**: the model cites a topic-overview chunk for every claim instead of the specific chunk that supports each individual claim. Lazy-anchor ratio histogram (11:31Z bench, 142 pointer rows): -| `lazy_anchor_ratio` bucket | pointer-mode rows | -|----------------------------|-------------------| +| `lazy_anchor_ratio` bucket | rows | +|----------------------------|------| | `0.00` | 25 | | `<0.25` | 9 | | `<0.5` | 11 | | `<0.75` | 46 | -| `>=0.75` | **49** | +| `>=0.75` | **49** (35%) | -49 of 142 pointer rows (35%) anchored ≥75% of claims to a topic-level chunk rather than the specific claim-supporting chunk. **This is the failure pattern Rule 8 was promoted to catch** — when the lazy-anchor target's source title shares zero stemmed content tokens with the claim, the run demotes STRICT → HYBRID via `TITLE_MISMATCH`. +35% of pointer rows anchor ≥75% of claims to a topic chunk. **This is the failure pattern Rule 8 was promoted to catch** — when the lazy-anchor target's source title shares zero stemmed content tokens with the claim, the run demotes `STRICT → HYBRID` via `TITLE_MISMATCH`. The honesty surfaces; the structural lift requires a stronger intervention (evidence-map ranking change, model fine-tune, or a different prompt frame). -19 of 71 questions show JSON > pointer by 50pp+: +19 of 71 questions show JSON ≥ 50pp above pointer, including: ``` pointer=HH json=SS did napoleon really die on saint helena? @@ -107,105 +128,48 @@ pointer=HH json=SS why did the dinosaurs go extinct? `subject_in_answer` is high across all rungs (STRICT 100%, HYBRID 92%, UNGROUNDED 84%) — the subject is in the answer; the answer just doesn't anchor cleanly to a single chunk. This is **structural-honesty signal**, not deflection. -## Verdict +## Wall-clock & throughput evolution -**Pointer-mode strict-rate is essentially flat from the prior bench.** The honesty cost of Rule 8 hits both lattice modes; JSON's structured-per-claim linkage absorbs it better than pointer's prose-with-tags. +| stamp | concurrency | scheduling | n | tasks | wall-clock | tasks/min | mean latency (mode) | +|-------|-------------|-----------|---|-------|------------|-----------|---------------------| +| 11:31Z | 4 | cell-grouped | 2 | 426 | 51 min | 8.4 | 26-29 s | +| 15:07Z | 4 | sample-shuffled | 3 | 639 | **48 min** | **13.3** | **17-19 s** | -**JSON's 6pp drop is the real news.** Likely causes: -1. Rule 8 demoting more cases (same as pointer, less severe). -2. Frame detector / polarity preamble changing prompt budget on reference-frame questions. -3. Anchor-class warrant generalization (entity-list / count / why-cause) catching cases the prior bench missed entirely. +Sample-level shuffled scheduling at the same concurrency delivers **+58% throughput**. n=3 (50% more total work) ran in 6% LESS wall-clock than n=2 cell-grouped. Per-call mean latency dropped 35-42% across all modes (quote 29.1 s → 16.9 s, pointer 26.7 s → 18.5 s, JSON 28.2 s → 18.3 s) because vLLM's continuous batcher fills better when fed a diverse, uncorrelated request stream. -The directive-coverage table (99% across every observable D2-D7 on lattice modes) is the architectural win. The strict-rate cost is the price of that calibration. +### Concurrency sweep (smoke fixture, 15-task) -## Per-mode recommendation +| c | wall-clock | tasks/min | normalized | +|---|------------|-----------|------------| +| 3 | 102 s | 8.8 | **1.00** (peak) | +| 4 | 106 s | 8.5 | 0.96 (chosen — within 4% of peak, more forgiving on single-call hiccups) | +| 5 | 119 s | 7.6 | 0.86 | +| 6 | 185 s | 4.9 | 0.55 (vLLM batching ceiling, brutal) | -- **`claim_lattice` (JSON)** stays the recommended default for cache-grade provenance. Best strict-rate among lattice modes (0.44), JSON-mode wins on structured per-claim discipline, peaks at larger context (32-64KB) so it scales with evidence growth. -- **`quote` mode** is best for raw lexical-grounding signal (0.50 strict-rate) but doesn't surface anchor-class warrants — use when the four-rung ladder isn't needed. -- **`claim_lattice_pointer`** stays useful for low-context-budget scenarios (16-32KB peak) and for prose-distribution models that struggle with JSON grammar; the lazy-anchor honesty cost is the trade. - -## Concurrency landed alongside this bench - -The bench harness gained a `--concurrency N` flag (commit `0870af6`) for parallel (question, mode) cells. First attempt at concurrency=4 surfaced a thread-safety defect in `aborist/compress.py`: the module-level `ZstdCompressor` / `ZstdDecompressor` singletons were not thread-safe — concurrent decompress calls corrupted the libzstd context and raised `ZstdError: Data corruption detected` on ~19% of retrievals. Fixed via `threading.local()` caches; same commit. - -Wall-clock for this 426-run sweep dropped from ~107 min projected sequential to ~51 min actual at concurrency=4. Speedup is below theoretical 4× because vLLM's continuous batcher queues at 4 concurrent requests, stretching individual call latency. Concurrency=2 may give better per-call latency at lower throughput; concurrency=8 may give higher throughput at the cost of per-call latency. Not yet tuned. - -## Outputs - -- `bench/qa_results/2026-05-02T11-31-55Z.jsonl` — 426 rows, durable artifact -- `bench/qa_results/2026-05-02T11-31-55Z.md` — auto-generated summary - -(Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.) - - ---- - - -# Update — 15:07Z full bench (post-Sprint-1b/Sprint-2/DRY/keep-alive/shuffle) - -**Stamp:** `2026-05-02T15-07-24Z` -**Bench:** 71 questions × 3 modes × 3 samples = 639 LLM calls (213 cells) -**Wall-clock:** **48 min** at `--concurrency 4 --seed 0` (sample-level shuffled scheduling) -**Hardening since the 11:31Z bench above:** -- Sprint 1b: per-mode `max_context_chars` (quote/pointer 24KB, JSON 48KB) folded into `governance_policy_hash` -- Sprint 2: pointer Rule 9 — chunk-specificity instruction -- DRY: lattice prompts collapsed to `aborist/qa/prompts.py` (single source of truth) -- httpx persistent client (TLS handshake amortization across calls) -- Bench harness sample-level shuffled scheduling with per-cell locks (true i.i.d. n=3 variance estimation) -- Concurrency sweep on smoke fixture: c=3 peak, c=4 within 4% (chosen for full bench), c=5 12% slower, c=6 45% slower - -## Aggregate - -| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | mean ratio | mean latency | -|------|------|--------|--------|------------|-----|-------------|------------|--------------| -| `quote` | 213 | 116 | 51 | 46 | 0 | **0.54** | 0.699 | **16.9s** | -| `claim_lattice_pointer` | 213 | 43 | 146 | 21 | 3 | **0.20** | 0.642 | 18.5s | -| `claim_lattice` (JSON) | 213 | 89 | 82 | 39 | 3 | **0.42** | 0.698 | 18.3s | - -## Δ vs 11:31Z bench (same date, 4h earlier) - -| mode | 11:31Z | 15:07Z | Δ | -|------|--------|--------|----| -| `quote` | 0.50 | **0.54** | +4pp | -| `claim_lattice_pointer` | 0.23 | **0.20** | −3pp | -| `claim_lattice` (JSON) | 0.44 | **0.42** | −2pp | - -Quote moved sharply UP. Pointer & JSON dropped slightly — within bench variance for small samples but worth noting. Quote's lift is the cleanest signal: the per-mode 24KB cap is exactly the bucket where quote peaks (8-16KB at 0.58 in this bench), so smaller-context-default surfaces tighter retrievals that quote can ground verbatim. - -Pointer's slight drop is consistent with Sprint 2's smoke result — the chunk-specificity rule didn't materially shift Hermes-3-8B's lazy-anchoring at n=3 either. The structural lift will need a stronger intervention (LLM-side fine-tune, evidence-map ranking change, or a different prompt frame). - -JSON's slight drop is within bench noise; structurally JSON should benefit most from the 32-64KB peak budget Sprint 1b targeted, and the per-bucket data confirms (JSON 32-64KB strict-rate 0.48 vs 16-32KB at 0.38). - -## Per-bucket strict-rate - -| mode | bucket | runs | strict-rate | -|------|--------|------|-------------| -| `quote` | 8-16KB | 105 | **0.58** ← peak | -| `quote` | 16-32KB | 105 | 0.52 | -| `claim_lattice_pointer` | 16-32KB | 210 | **0.20** ← peak (only bucket) | -| `claim_lattice` (JSON) | 32-64KB | 99 | **0.48** ← peak | -| `claim_lattice` (JSON) | 16-32KB | 108 | 0.38 | - -**Sprint 1b's intent confirmed.** JSON's peak at 32-64KB justifies the 48KB default. Quote's peak migrated to 8-16KB after the 24KB cap; the cap may be tighter than optimal — quote could plausibly be cut to 16KB (mid of 8-16KB bucket) for another small lift in a follow-up sprint. - -## Directive coverage - -Lattice modes hit **99% on every observable D2-D7**; the 1% gap is the 6 surrogate errors. Quote shows 0% on D2/D3/D6 by construction (those are lattice-only) and 100% on D4/D7. +vLLM saturates at c=3-4 on this endpoint. More concurrent requests fill the batch better, but only up to the point where per-call latency growth outpaces parallelism gain. Re-sweep when the endpoint is upgraded, when corpus shape changes the average context size, or when other tenants change the queue. ## Errors -6 errors (3 per lattice mode). All on the same question — `tell me about the roman empire`. Root cause traced & fixed in `41d1d9b`: lone UTF-16 surrogates in the Wikipedia chunk content broke httpx's outbound JSON encode (different path from the earlier 3b91223 fix which hardened SHA-256 hashers on the OUTPUT side). Next bench will land 0 errors. +Across both benches: **2 errors at 11:31Z, 6 errors at 15:07Z** — all on the same question (`tell me about the roman empire`). Lone UTF-16 surrogates in Wikipedia chunk content, two distinct paths: -## Wall-clock & throughput +- v1 (`3b91223`) — hardened SHA-256 hashers on the OUTPUT side (`aborist/qa/dag.py`, `keys.py`, `evidence.py`, `store.py` audit chain) with `errors='surrogatepass'` so the run-DAG roots survive surrogate-bearing model output. +- v2 (`41d1d9b`) — scrubs message content INSIDE `OpenAICompatibleClient.chat_completion` before httpx's outbound JSON encode. The corpus chunk text was the path; httpx's `.encode('utf-8')` on the request body raised before the call left the client. -| run | concurrency | scheduling | tasks | wall | tasks/min | -|-----|-------------|-----------|-------|------|-----------| -| 11:31Z | 4 | cell-grouped | 426 (n=2) | 51 min | 8.4 | -| 15:07Z | 4 | sample-shuffled | 639 (n=3) | 48 min | **13.3** | +Verified: `tell me about the roman empire` under `claim_lattice` now classifies HYBRID 5/7 instead of erroring. Next bench will land 0 errors. -Sample-level shuffled scheduling at the same concurrency delivers **+58% throughput** by feeding vLLM's continuous batcher a properly diverse request stream. The cell-grouped predecessor correlated batches by cache_key; vLLM couldn't fill efficiently. n=3 instead of n=2 is **50% more total work**, executed in **6% less wall-clock**. Per-call mean latency dropped 35-42% across all modes (quote 29.1s→16.9s, pointer 26.7s→18.5s, JSON 28.2s→18.3s). +## Verdict & per-mode recommendation + +**Quote leads on raw lexical grounding (0.54 strict-rate).** Best when the four-rung ladder isn't needed and the answer is short-and-cite-able. Doesn't surface anchor-class warrants. + +**JSON leads among lattice modes (0.42).** Recommended default for cache-grade provenance. Structured per-claim evidence linkage; peaks at larger context (32-64 KB) so it scales with evidence growth. + +**Pointer (0.20) stays useful for low-context-budget scenarios** (16-32 KB peak) and prose-distribution models that struggle with JSON grammar; the lazy-anchor honesty cost is the trade. + +**The directive-coverage table (99% across every observable D2-D7 on lattice modes) is the architectural win.** False-positive STRICT was corruption; converting it to honest HYBRID was the price of calibration. Strict-rate slipped on lattice modes; the substrate is more honest. ## Outputs -- `bench/qa_results/2026-05-02T15-07-24Z.jsonl` — 639 rows -- `bench/qa_results/2026-05-02T15-07-24Z.md` — auto-generated summary +- `bench/qa_results/2026-05-02T11-31-55Z.{jsonl,md}` — 426 rows +- `bench/qa_results/2026-05-02T15-07-24Z.{jsonl,md}` — 639 rows + +(Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.)