arborist/docs/qa-modes-bench.md

QA-modes bench — 2026-05-02

Date: 2026-05-02 Endpoint: https://hermes.ai.unturf.com/v1 (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx) Corpus: Wikipedia 2003-05-16 cur snapshot, sharded under ~/.aborist/shards

Two sweeps landed today, each 71 questions × 3 modes:

stamp	samples	concurrency	scheduling	runs	wall-clock
11:31Z	n=2	c=4	cell-grouped	426	51 min
15:07Z	n=3	c=4	sample-shuffled, --seed 0	639	48 min

The 15:07Z sweep is the authoritative state of the substrate at end-of-day; the 11:31Z sweep is the intermediate witnessed before the second wave of hardening landed. Headlines below show both.

Hardening since the prior bench (2026-04-30 post-retry)

Substrate-side (between 2026-04-30T18:46Z and 2026-05-02T11:31Z):

• Rule 8 — title-relevance promoted to hard verifier check (TITLE_MISMATCH violation demotes STRICT → HYBRID when no cited evidence's source title shares a stemmed token with the claim).
• Anchor-class warrant generalized to entity-list / count / why-cause shapes (Ticket #000003).
• Retrieval-plan hash bound into the run-DAG retrieval stage (Ticket #000001).
• Reference-frame polarity contract (Ticket #000002).
• Four-rung ladder display (Ticket #000005) — POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED reserved) mapped from the v9.8 audit_mode trichotomy at render time.
• Bench harness extended with directive coverage + log-scale buckets to 1M (Ticket #000004).

Substrate-side (between 2026-05-02T11:31Z and 2026-05-02T15:07Z):

• Sprint 1b — per-mode max_context_chars. Bench's recommended-context-budget table flows back into DEFAULT_QUERY_POLICY["max_context_chars_by_mode"]: quote 24 KB, pointer 24 KB, JSON 48 KB. Folds into governance_policy_hash.
• Sprint 2 — pointer Rule 9: chunk-specificity instruction added to the lattice-pointer system prompt.
• DRY collapse — the four lattice prompts (system + grounding × pointer + JSON) lifted to aborist/qa/prompts.py as a single source of truth, imported by both runner.DEFAULT_POLICY and query.DEFAULT_QUERY_POLICY.
• httpx persistent client — the chat-completion path used to construct a fresh httpx.Client per call, paying a TLS handshake every request. Move to __init__; HTTP/1.1 keep-alive across calls. Save 1-2 min on a 426-call bench.
• Sample-level shuffled bench scheduling — every (question, mode, sample_idx) is a task, shuffled with --seed, dispatched concurrently. Per-cell Lock dict serializes burn-then-write on the shared cache_key. True i.i.d. n=3 variance; vLLM batcher fed a diverse request stream.
• Bench --resume — read existing JSONL, skip done tasks, append fresh rows. Stop/start-able.
• Concurrency sweep on the smoke fixture. vLLM peaks at c=3-4; saturates badly past c=4 (see Wall-clock & throughput below).
• UTF-16 surrogate fix v2 (41d1d9b) — corpus chunks with lone surrogates broke httpx's outbound JSON encode. Scrub at the client boundary by WTF-8 → UTF-8-with-replace roundtrip. (The earlier v1 fix 3b91223 hardened the OUTPUT side — SHA-256 hashers — but missed the INPUT side. Both paths now safe.)

Aggregate (15:07Z, authoritative)

mode	runs	STRICT	HYBRID	UNGROUNDED	err	strict-rate	mean ratio	mean latency
quote	213	116	51	46	0	0.54	0.699	16.9 s
claim_lattice_pointer	213	43	146	21	3	0.20	0.642	18.5 s
claim_lattice (JSON)	213	89	82	39	3	0.42	0.698	18.3 s

Δ across the day

mode	2026-04-30 (post-retry)	2026-05-02T11:31Z	2026-05-02T15:07Z	net Δ
quote	0.47	0.50 (+3pp)	0.54 (+4pp)	+7pp
claim_lattice_pointer	0.24	0.23 (−1pp)	0.20 (−3pp)	−4pp
claim_lattice (JSON)	0.50	0.44 (−6pp)	0.42 (−2pp)	−8pp

Quote climbed the most across the day, ending at 0.54. The Sprint 1b 24 KB cap surfaces tighter retrievals that quote can ground verbatim (the bucket data confirms — quote's peak migrated to 8-16 KB at 0.58 strict-rate).

Pointer & JSON each took an honesty cost. Rule 8 promotion + warrant-class generalization demote more cases that would have classified STRICT under the looser pre-2026-05-02 verifier. The architectural win — 99% directive coverage on lattice modes — is the price for those drops; false-positive STRICT was corruption, and we converted it to honest HYBRID.

Per-bucket strict-rate (15:07Z)

mode	bucket	runs	strict-rate	note
quote	8-16 KB	105	0.58	peak
quote	16-32 KB	105	0.52
claim_lattice_pointer	16-32 KB	210	0.20	peak (only bucket)
claim_lattice (JSON)	32-64 KB	99	0.48	peak
claim_lattice (JSON)	16-32 KB	108	0.38

Sprint 1b's intent confirmed. JSON's peak at 32-64 KB justifies the 48 KB default. Quote's peak migrated to 8-16 KB after the 24 KB cap; the cap may be tighter than optimal — quote could plausibly be cut to 16 KB (mid of 8-16 KB bucket) for another small lift in a follow-up sprint.

claim_lattice (JSON) peaks at LARGER context than the other modes. The structured per-claim evidence linkage that JSON enforces benefits from more evidence per claim. Pointer & quote degrade past 16-32 KB.

Recommended context budget (per-mode)

5-run minimum to reduce noise; from the 15:07Z bucket data:

mode	peak bucket	strict-rate	n
quote	8-16 KB	0.58	105
claim_lattice_pointer	16-32 KB	0.20	210
claim_lattice (JSON)	32-64 KB	0.48	99

These flow into DEFAULT_QUERY_POLICY["max_context_chars_by_mode"] (currently 24 / 24 / 48 KB — mid of each peak bucket). When the bench shifts those peaks, retune the policy and let governance_policy_hash partition the cache.

Directive coverage (seven-point program)

mode	D2 pointer	D3 cti-ready	D4 ev-map bound	D6 warrant	D7 honest label
quote	0/213 (0%)	0/213 (0%)	213/213 (100%)	0/213 (0%)	213/213 (100%)
claim_lattice_pointer	210/213 (99%)	210/213 (99%)	210/213 (99%)	210/213 (99%)	210/213 (99%)
claim_lattice	210/213 (99%)	210/213 (99%)	210/213 (99%)	210/213 (99%)	210/213 (99%)

Lattice modes hit 99% on every observable directive; the 1% gap is the 6 surrogate errors (already fixed in 41d1d9b, next bench will hit 100%). Quote shows 0% on D2 / D3 / D6 by construction (those are lattice-only directives) and 100% on D4 / D7.

Pointer-mode signal — where the gap lives

Pointer mode's strict-rate did not lift on Sprint 2's chunk-specificity prompt nudge. The load-bearing failure pattern is lazy-anchoring: the model cites a topic-overview chunk for every claim instead of the specific chunk that supports each individual claim. Lazy-anchor ratio histogram (11:31Z bench, 142 pointer rows):

lazy_anchor_ratio bucket	rows
0.00	25
<0.25	9
<0.5	11
<0.75	46
>=0.75	49 (35%)

35% of pointer rows anchor ≥75% of claims to a topic chunk. This is the failure pattern Rule 8 was promoted to catch — when the lazy-anchor target's source title shares zero stemmed content tokens with the claim, the run demotes STRICT → HYBRID via TITLE_MISMATCH. The honesty surfaces; the structural lift requires a stronger intervention (evidence-map ranking change, model fine-tune, or a different prompt frame).

19 of 71 questions show JSON ≥ 50pp above pointer, including:

pointer=HH  json=SS   did napoleon really die on saint helena?
pointer=HH  json=SS   name simpsons family members including pets?
pointer=HH  json=SS   what are the planets of our solar system?
pointer=HH  json=SS   what country is the city of prague in?
pointer=HH  json=SS   what is the boltzmann constant?
pointer=UU  json=SS   what is the difference between http and ftp?
pointer=HH  json=SS   when did the soviet union dissolve?
pointer=HH  json=SS   where does the nile river begin?
pointer=HH  json=SS   where is mount kilimanjaro located?
pointer=HH  json=SS   who is supermans girlfriend?
pointer=HH  json=SS   who painted the mona lisa?
pointer=HH  json=SS   who said may the force be with you?
pointer=HH  json=SS   who were the original seven mercury astronauts?
pointer=HH  json=SS   why did the dinosaurs go extinct?

(plus 5 more.) These are factual single-fact questions where JSON's structured per-claim evidence linkage forces the model to pick the specific chunk; pointer's prose-with-tags lets the model lazy-anchor to the topic article and Rule 8 catches it.

subject_in_answer is high across all rungs (STRICT 100%, HYBRID 92%, UNGROUNDED 84%) — the subject is in the answer; the answer just doesn't anchor cleanly to a single chunk. This is structural-honesty signal, not deflection.

Wall-clock & throughput evolution

stamp	concurrency	scheduling	n	tasks	wall-clock	tasks/min	mean latency (mode)
11:31Z	4	cell-grouped	2	426	51 min	8.4	26-29 s
15:07Z	4	sample-shuffled	3	639	48 min	13.3	17-19 s

Sample-level shuffled scheduling at the same concurrency delivers +58% throughput. n=3 (50% more total work) ran in 6% LESS wall-clock than n=2 cell-grouped. Per-call mean latency dropped 35-42% across all modes (quote 29.1 s → 16.9 s, pointer 26.7 s → 18.5 s, JSON 28.2 s → 18.3 s) because vLLM's continuous batcher fills better when fed a diverse, uncorrelated request stream.

Concurrency sweep (smoke fixture, 15-task)

c	wall-clock	tasks/min	normalized
3	102 s	8.8	1.00 (peak)
4	106 s	8.5	0.96 (chosen — within 4% of peak, more forgiving on single-call hiccups)
5	119 s	7.6	0.86
6	185 s	4.9	0.55 (vLLM batching ceiling, brutal)

vLLM saturates at c=3-4 on this endpoint. More concurrent requests fill the batch better, but only up to the point where per-call latency growth outpaces parallelism gain. Re-sweep when the endpoint is upgraded, when corpus shape changes the average context size, or when other tenants change the queue.

Errors

Across both benches: 2 errors at 11:31Z, 6 errors at 15:07Z — all on the same question (tell me about the roman empire). Lone UTF-16 surrogates in Wikipedia chunk content, two distinct paths:

• v1 (3b91223) — hardened SHA-256 hashers on the OUTPUT side (aborist/qa/dag.py, keys.py, evidence.py, store.py audit chain) with errors='surrogatepass' so the run-DAG roots survive surrogate-bearing model output.
• v2 (41d1d9b) — scrubs message content INSIDE OpenAICompatibleClient.chat_completion before httpx's outbound JSON encode. The corpus chunk text was the path; httpx's .encode('utf-8') on the request body raised before the call left the client.

Verified: tell me about the roman empire under claim_lattice now classifies HYBRID 5/7 instead of erroring. Next bench will land 0 errors.

Verdict & per-mode recommendation

Quote leads on raw lexical grounding (0.54 strict-rate). Best when the four-rung ladder isn't needed and the answer is short-and-cite-able. Doesn't surface anchor-class warrants.

JSON leads among lattice modes (0.42). Recommended default for cache-grade provenance. Structured per-claim evidence linkage; peaks at larger context (32-64 KB) so it scales with evidence growth.

Pointer (0.20) stays useful for low-context-budget scenarios (16-32 KB peak) and prose-distribution models that struggle with JSON grammar; the lazy-anchor honesty cost is the trade.

The directive-coverage table (99% across every observable D2-D7 on lattice modes) is the architectural win. False-positive STRICT was corruption; converting it to honest HYBRID was the price of calibration. Strict-rate slipped on lattice modes; the substrate is more honest.

Outputs

• bench/qa_results/2026-05-02T11-31-55Z.{jsonl,md} — 426 rows
• bench/qa_results/2026-05-02T15-07-24Z.{jsonl,md} — 639 rows

(Both gitignored under bench/qa_results/. Headlines & analysis live in this journal.)

---

Addendum — 2026-05-03 broad-quantifier A/B (Ticket #000008)

This journal froze the 2026-05-02 substrate baseline. On 2026-05-03 Ticket #000008 (broad-quantifier preflight guard) landed Phases 0–4 and ran a four-cell A/B on a 9-question broad subset (bench/qa_questions_quantifier_subset.txt, 81 runs per cell).

Findings relevant to this journal's per-mode recommendations:

• JSON-mode strict-rate moves under cap-only: 0.19 baseline → 0.33 with quantifier_guard_apply_caps=true (+14pp on the broad subset). The recommendation here that "JSON leads among lattice modes" still holds, but the headroom above 0.42 globally is partly dependent on broad-vs-narrow question mix.
• Pointer-mode stays at 0/27 STRICT on broad questions across all four A/B cells. The 0.20 strict-rate above is across the full 71-question set; the broad subset alone is structurally unfavorable to pointer mode regardless of cap or reminder.
• Reminder eliminates FORMAT_COLLAPSED: 2/27 → 0/27 on pointer mode under quantifier_reminder_enabled=true.
• Cap and reminder help DIFFERENT failure modes: reminder rescues UNGROUNDED → HYBRID (restates citation rule); cap rescues HYBRID → STRICT (forces fewer-but-better claims). Compound effect on pointer mean-ratio is best at 0.684 (vs 0.473 baseline).

See docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md §12 for the four-cell data and the §10.8 decision-tree verdict. Bench artifacts: bench/qa_results/2026-05-03T12-{29-53,38-53,47-23,54-11}Z.{jsonl,md}.