arborist/docs/qa-modes-bench.md

QA-modes bench — 2026-05-02

Date: 2026-05-02 Endpoint: https://hermes.ai.unturf.com/v1 (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx) Corpus: Wikipedia 2003-05-16 cur snapshot, sharded under ~/.arborist/shards

Two sweeps landed today, each 71 questions × 3 modes:

stamp	samples	concurrency	scheduling	runs	wall-clock
11:31Z	n=2	c=4	cell-grouped	426	51 min
15:07Z	n=3	c=4	sample-shuffled, --seed 0	639	48 min

The 15:07Z sweep is the authoritative state of the substrate at end-of-day; the 11:31Z sweep is the intermediate witnessed before the second wave of hardening landed. Headlines below show both.

Hardening since the prior bench (2026-04-30 post-retry)

Substrate-side (between 2026-04-30T18:46Z and 2026-05-02T11:31Z):

• Rule 8 — title-relevance promoted to hard verifier check (TITLE_MISMATCH violation demotes STRICT → HYBRID when no cited evidence's source title shares a stemmed token with the claim).
• Anchor-class warrant generalized to entity-list / count / why-cause shapes (Ticket #000003).
• Retrieval-plan hash bound into the run-DAG retrieval stage (Ticket #000001).
• Reference-frame polarity contract (Ticket #000002).
• Four-rung ladder display (Ticket #000005) — POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → (ENTAILMENT-VERIFIED reserved) mapped from the v9.8 audit_mode trichotomy at render time.
• Bench harness extended with directive coverage + log-scale buckets to 1M (Ticket #000004).

Substrate-side (between 2026-05-02T11:31Z and 2026-05-02T15:07Z):

• Sprint 1b — per-mode max_context_chars. Bench's recommended-context-budget table flows back into DEFAULT_QUERY_POLICY["max_context_chars_by_mode"]: quote 24 KB, pointer 24 KB, JSON 48 KB. Folds into governance_policy_hash.
• Sprint 2 — pointer Rule 9: chunk-specificity instruction added to the lattice-pointer system prompt.
• DRY collapse — the four lattice prompts (system + grounding × pointer + JSON) lifted to arborist/qa/prompts.py as a single source of truth, imported by both runner.DEFAULT_POLICY and query.DEFAULT_QUERY_POLICY.
• httpx persistent client — the chat-completion path used to construct a fresh httpx.Client per call, paying a TLS handshake every request. Move to __init__; HTTP/1.1 keep-alive across calls. Save 1-2 min on a 426-call bench.
• Sample-level shuffled bench scheduling — every (question, mode, sample_idx) is a task, shuffled with --seed, dispatched concurrently. Per-cell Lock dict serializes burn-then-write on the shared cache_key. True i.i.d. n=3 variance; vLLM batcher fed a diverse request stream.
• Bench --resume — read existing JSONL, skip done tasks, append fresh rows. Stop/start-able.
• Concurrency sweep on the smoke fixture. vLLM peaks at c=3-4; saturates badly past c=4 (see Wall-clock & throughput below).
• UTF-16 surrogate fix v2 (41d1d9b) — corpus chunks with lone surrogates broke httpx's outbound JSON encode. Scrub at the client boundary by WTF-8 → UTF-8-with-replace roundtrip. (The earlier v1 fix 3b91223 hardened the OUTPUT side — SHA-256 hashers — but missed the INPUT side. Both paths now safe.)

Aggregate (15:07Z, authoritative)

mode	runs	STRICT	HYBRID	UNGROUNDED	err	strict-rate	mean ratio	mean latency
quote	213	116	51	46	0	0.54	0.699	16.9 s
claim_lattice_pointer	213	43	146	21	3	0.20	0.642	18.5 s
claim_lattice (JSON)	213	89	82	39	3	0.42	0.698	18.3 s

Δ across the day

mode	2026-04-30 (post-retry)	2026-05-02T11:31Z	2026-05-02T15:07Z	net Δ
quote	0.47	0.50 (+3pp)	0.54 (+4pp)	+7pp
claim_lattice_pointer	0.24	0.23 (−1pp)	0.20 (−3pp)	−4pp
claim_lattice (JSON)	0.50	0.44 (−6pp)	0.42 (−2pp)	−8pp

Quote climbed the most across the day, ending at 0.54. The Sprint 1b 24 KB cap surfaces tighter retrievals that quote can ground verbatim (the bucket data confirms — quote's peak migrated to 8-16 KB at 0.58 strict-rate).

Pointer & JSON each took an honesty cost. Rule 8 promotion + warrant-class generalization demote more cases that would have classified STRICT under the looser pre-2026-05-02 verifier. The architectural win — 99% directive coverage on lattice modes — is the price for those drops; false-positive STRICT was corruption, and we converted it to honest HYBRID.

Per-bucket strict-rate (15:07Z)

mode	bucket	runs	strict-rate	note
quote	8-16 KB	105	0.58	peak
quote	16-32 KB	105	0.52
claim_lattice_pointer	16-32 KB	210	0.20	peak (only bucket)
claim_lattice (JSON)	32-64 KB	99	0.48	peak
claim_lattice (JSON)	16-32 KB	108	0.38

Sprint 1b's intent confirmed. JSON's peak at 32-64 KB justifies the 48 KB default. Quote's peak migrated to 8-16 KB after the 24 KB cap; the cap may be tighter than optimal — quote could plausibly be cut to 16 KB (mid of 8-16 KB bucket) for another small lift in a follow-up sprint.

claim_lattice (JSON) peaks at LARGER context than the other modes. The structured per-claim evidence linkage that JSON enforces benefits from more evidence per claim. Pointer & quote degrade past 16-32 KB.

Recommended context budget (per-mode)

5-run minimum to reduce noise; from the 15:07Z bucket data:

mode	peak bucket	strict-rate	n
quote	8-16 KB	0.58	105
claim_lattice_pointer	16-32 KB	0.20	210
claim_lattice (JSON)	32-64 KB	0.48	99

These flow into DEFAULT_QUERY_POLICY["max_context_chars_by_mode"] (currently 24 / 24 / 48 KB — mid of each peak bucket). When the bench shifts those peaks, retune the policy and let governance_policy_hash partition the cache.

Directive coverage (seven-point program)

mode	D2 pointer	D3 cti-ready	D4 ev-map bound	D6 warrant	D7 honest label
quote	0/213 (0%)	0/213 (0%)	213/213 (100%)	0/213 (0%)	213/213 (100%)
claim_lattice_pointer	210/213 (99%)	210/213 (99%)	210/213 (99%)	210/213 (99%)	210/213 (99%)
claim_lattice	210/213 (99%)	210/213 (99%)	210/213 (99%)	210/213 (99%)	210/213 (99%)

Lattice modes hit 99% on every observable directive; the 1% gap is the 6 surrogate errors (already fixed in 41d1d9b, next bench will hit 100%). Quote shows 0% on D2 / D3 / D6 by construction (those are lattice-only directives) and 100% on D4 / D7.

Pointer-mode signal — where the gap lives

Pointer mode's strict-rate did not lift on Sprint 2's chunk-specificity prompt nudge. The load-bearing failure pattern is lazy-anchoring: the model cites a topic-overview chunk for every claim instead of the specific chunk that supports each individual claim. Lazy-anchor ratio histogram (11:31Z bench, 142 pointer rows):

lazy_anchor_ratio bucket	rows
0.00	25
<0.25	9
<0.5	11
<0.75	46
>=0.75	49 (35%)

35% of pointer rows anchor ≥75% of claims to a topic chunk. This is the failure pattern Rule 8 was promoted to catch — when the lazy-anchor target's source title shares zero stemmed content tokens with the claim, the run demotes STRICT → HYBRID via TITLE_MISMATCH. The honesty surfaces; the structural lift requires a stronger intervention (evidence-map ranking change, model fine-tune, or a different prompt frame).

19 of 71 questions show JSON ≥ 50pp above pointer, including:

pointer=HH  json=SS   did napoleon really die on saint helena?
pointer=HH  json=SS   name simpsons family members including pets?
pointer=HH  json=SS   what are the planets of our solar system?
pointer=HH  json=SS   what country is the city of prague in?
pointer=HH  json=SS   what is the boltzmann constant?
pointer=UU  json=SS   what is the difference between http and ftp?
pointer=HH  json=SS   when did the soviet union dissolve?
pointer=HH  json=SS   where does the nile river begin?
pointer=HH  json=SS   where is mount kilimanjaro located?
pointer=HH  json=SS   who is supermans girlfriend?
pointer=HH  json=SS   who painted the mona lisa?
pointer=HH  json=SS   who said may the force be with you?
pointer=HH  json=SS   who were the original seven mercury astronauts?
pointer=HH  json=SS   why did the dinosaurs go extinct?

(plus 5 more.) These are factual single-fact questions where JSON's structured per-claim evidence linkage forces the model to pick the specific chunk; pointer's prose-with-tags lets the model lazy-anchor to the topic article and Rule 8 catches it.

subject_in_answer is high across all rungs (STRICT 100%, HYBRID 92%, UNGROUNDED 84%) — the subject is in the answer; the answer just doesn't anchor cleanly to a single chunk. This is structural-honesty signal, not deflection.

Wall-clock & throughput evolution

stamp	concurrency	scheduling	n	tasks	wall-clock	tasks/min	mean latency (mode)
11:31Z	4	cell-grouped	2	426	51 min	8.4	26-29 s
15:07Z	4	sample-shuffled	3	639	48 min	13.3	17-19 s

Sample-level shuffled scheduling at the same concurrency delivers +58% throughput. n=3 (50% more total work) ran in 6% LESS wall-clock than n=2 cell-grouped. Per-call mean latency dropped 35-42% across all modes (quote 29.1 s → 16.9 s, pointer 26.7 s → 18.5 s, JSON 28.2 s → 18.3 s) because vLLM's continuous batcher fills better when fed a diverse, uncorrelated request stream.

Concurrency sweep (smoke fixture, 15-task)

c	wall-clock	tasks/min	normalized
3	102 s	8.8	1.00 (peak)
4	106 s	8.5	0.96 (chosen — within 4% of peak, more forgiving on single-call hiccups)
5	119 s	7.6	0.86
6	185 s	4.9	0.55 (vLLM batching ceiling, brutal)

vLLM saturates at c=3-4 on this endpoint. More concurrent requests fill the batch better, but only up to the point where per-call latency growth outpaces parallelism gain. Re-sweep when the endpoint is upgraded, when corpus shape changes the average context size, or when other tenants change the queue.

Errors

Across both benches: 2 errors at 11:31Z, 6 errors at 15:07Z — all on the same question (tell me about the roman empire). Lone UTF-16 surrogates in Wikipedia chunk content, two distinct paths:

• v1 (3b91223) — hardened SHA-256 hashers on the OUTPUT side (arborist/qa/dag.py, keys.py, evidence.py, store.py audit chain) with errors='surrogatepass' so the run-DAG roots survive surrogate-bearing model output.
• v2 (41d1d9b) — scrubs message content INSIDE OpenAICompatibleClient.chat_completion before httpx's outbound JSON encode. The corpus chunk text was the path; httpx's .encode('utf-8') on the request body raised before the call left the client.

Verified: tell me about the roman empire under claim_lattice now classifies HYBRID 5/7 instead of erroring. Next bench will land 0 errors.

Verdict & per-mode recommendation

Quote leads on raw lexical grounding (0.54 strict-rate). Best when the four-rung ladder isn't needed and the answer is short-and-cite-able. Doesn't surface anchor-class warrants.

JSON leads among lattice modes (0.42). Recommended default for cache-grade provenance. Structured per-claim evidence linkage; peaks at larger context (32-64 KB) so it scales with evidence growth.

Pointer (0.20) stays useful for low-context-budget scenarios (16-32 KB peak) and prose-distribution models that struggle with JSON grammar; the lazy-anchor honesty cost is the trade.

The directive-coverage table (99% across every observable D2-D7 on lattice modes) is the architectural win. False-positive STRICT was corruption; converting it to honest HYBRID was the price of calibration. Strict-rate slipped on lattice modes; the substrate is more honest.

Outputs

• bench/qa_results/2026-05-02T11-31-55Z.{jsonl,md} — 426 rows
• bench/qa_results/2026-05-02T15-07-24Z.{jsonl,md} — 639 rows

(Both gitignored under bench/qa_results/. Headlines & analysis live in this journal.)

---

Addendum — 2026-05-03 broad-quantifier A/B (Ticket #000008)

This journal froze the 2026-05-02 substrate baseline. On 2026-05-03 Ticket #000008 (broad-quantifier preflight guard) landed Phases 0–4 and ran a four-cell A/B on a 9-question broad subset (bench/qa_questions_quantifier_subset.txt, 81 runs per cell).

Findings relevant to this journal's per-mode recommendations:

• JSON-mode strict-rate moves under cap-only: 0.19 baseline → 0.33 with quantifier_guard_apply_caps=true (+14pp on the broad subset). The recommendation here that "JSON leads among lattice modes" still holds, but the headroom above 0.42 globally is partly dependent on broad-vs-narrow question mix.
• Pointer-mode stays at 0/27 STRICT on broad questions across all four A/B cells. The 0.20 strict-rate above is across the full 71-question set; the broad subset alone is structurally unfavorable to pointer mode regardless of cap or reminder.
• Reminder eliminates FORMAT_COLLAPSED: 2/27 → 0/27 on pointer mode under quantifier_reminder_enabled=true.
• Cap and reminder help DIFFERENT failure modes: reminder rescues UNGROUNDED → HYBRID (restates citation rule); cap rescues HYBRID → STRICT (forces fewer-but-better claims). Compound effect on pointer mean-ratio is best at 0.684 (vs 0.473 baseline).

See docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md §12 for the four-cell data and the §10.8 decision-tree verdict. Bench artifacts: bench/qa_results/2026-05-03T12-{29-53,38-53,47-23,54-11}Z.{jsonl,md}.

Addendum 2 — preflight ON vs OFF validation (2026-05-03T23-06-21Z)

After ticket #000010 flipped quantifier_reminder_enabled=True and shipped metacognition_enabled=True defaults, ran a preflight-OFF cell to validate the flip didn't regress baseline behavior. Same 9-question broad subset (bench/qa_questions_quantifier_subset.txt), n=3, policy override --policy metacognition_enabled=false --policy quantifier_reminder_enabled=false.

Compared against §12.6 reminder-only baseline (cleanest single-knob on-cell). On this 9-question subset, none of the metacognition detectors fire (no temporal / contradiction / false-premise / out- of-corpus shapes), so the comparison effectively isolates the reminder contribution.

Metric	OFF	ON	Δ
quote strict-rate	0.63	0.52	−11pp (noise band on 27 samples)
pointer strict-rate	0.00	0.00	0
JSON strict-rate	0.26	0.22	−4pp
pointer mean ratio	0.483	0.643	+16pp
JSON mean ratio	0.570	0.735	+17pp
JSON UNGROUNDED rate	7/27 (26%)	1/27 (4%)	−22pp
pointer FORMAT_COLLAPSED	2/27	0/27	−100%
pointer NO_EVIDENCE_POINTER	8/27	6/27	−7pp

Interpretation:

• Mean-ratio improvement is robust (+16-17pp on lattice modes). Grounded rows are MORE thoroughly grounded under preflight ON.
• JSON UNGROUNDED collapse is dramatic (−22pp). The "didn't ground" pool reclassifies into "partially grounded" — operator- visible win.
• FORMAT_COLLAPSED elimination on pointer mode (2 → 0). The reminder restates the [E\d+] citation rule and Hermes follows it.
• STRICT-rate moves are within noise. Quote-mode dipped 11pp, but quote is mode-gated off the guard so this is pure Hermes nondeterminism — the §10.8 5pp floor exists exactly to filter this. JSON SR moved −4pp (within floor).

Verdict: the #000010 default-on flip is doing what was claimed. Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all improve by ≥5pp on lattice modes; STRICT-rate is within noise. Defaults stay on.

Bench artifact: bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}.

Addendum 3 — full-bench regression check (2026-05-03T23-30-12Z)

Validates that the #000010 default flip (reminder default-on for lattice modes) doesn't regress narrow-question performance. The prior validations in Addendum 1 + Addendum 2 used the 9-question broad subset only; this run sweeps the full 75-question bench/qa_questions.txt (~10% broad, ~89% narrow factoid / descriptive / list shapes).

Same harness, n=3 × 3 modes × 75 questions = 225 runs per mode. Compared against the frozen 15:07Z baseline (the authoritative pre-#000008/9/10 state of the substrate, 71 questions × n=3 = 213 runs per mode):

Mode	Pre-flip (15:07Z)	Post-flip (23:30Z)	Δ SR	Δ mean ratio
quote	116S/51H/46U · SR 0.54 · ratio 0.699	116S/60H/49U · SR 0.52 · ratio 0.692	−2pp	−1pp
claim_lattice_pointer	43S/146H/21U · SR 0.20 · ratio 0.642	48S/145H/32U · SR 0.21 · ratio 0.663	+1pp	+2pp
claim_lattice (JSON)	89S/82H/39U · SR 0.42 · ratio 0.698	99S/80H/46U · SR 0.44 · ratio 0.730	+2pp	+3pp

All STRICT-rate deltas within the 5pp signal floor. Same question set ±2 (3 added: winners of all major sports?, name all members of the beatles, list all planets in the solar system).

Substrate-level findings:

• Pointer-mode FORMAT_COLLAPSED: 0/225 across the full sweep. The default-on reminder eliminates the collapse mode globally, not just on broad questions where it was originally measured.
• Pointer-mode NO_EVIDENCE_POINTER: 29/225 (13%) — lower than the 33% we saw on broad-only (9/27) when reminder was off. Reminder discipline propagates to non-broad questions even though the reminder text only fires on broad shapes (the broader signal is that the reminder text reinforces the citation rule across the model's session attention).
• Quote-mode is essentially unchanged (-2pp SR, -1pp ratio). Quote opts out of the guard (quantifier_guard_modes default excludes quote), so this is pure Hermes nondeterminism on a 225-sample cell.
• JSON-mode mean ratio +3pp — small but positive, consistent with the broad-subset finding that reminder boosts per-claim grounding density.

Latency note. Wall-clock per call jumped from ~17-19s (pre-flip baseline) to ~33-35s on this run. This is endpoint contention on Hermes-3 at the time of measurement, not a feature regression (the new substrate adds zero LLM calls — preflight is pure regex). The throughput numbers in §6 of the journal pre-date this evening's network conditions; future bench cycles should normalize.

Verdict — no regression. The #000010 default flip ships clean across the full corpus, not just the broad-quantifier subset where it was originally validated. Defaults stay on. The substrate is strictly more honest under preflight (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without sacrificing throughput on non-broad questions.

Bench artifact: bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}.

Addendum 4 — metacog-trigger detector validation (2026-05-04T02-18-42Z)

The validations above (Addendum 1-3) covered the broad-quantifier subset and the full corpus, neither of which contains questions that fire the #000010 metacog detectors (temporal sensitivity, contradiction, false-premise-lite, out-of-corpus). This addendum closes that empirical gap.

Fixture: bench/qa_questions_metacog_subset.txt — 6 questions, one per detector kind plus a control:

who is the current CEO of OpenAI?               # stale_risk
which character is alive and dead?              # contradictory
when did George Washington stop being president  # false_premise +
  of France?                                    #   stale_risk (multi)
what does my uploaded contract say about        # out_of_corpus
  clause 9?
when did the current CEO of Twitter stop        # stale_risk +
  being CEO of Tesla?                           #   false_premise
who painted the mona lisa?                      # control (well_formed)

Run: n=3 × 6 questions × 3 modes = 54 rows.

Per-question verdict matrix:

question (detector)	quote	pointer	JSON
Current CEO (stale_risk)	0/0/3 U	0/0/3 U	0/0/3 U
Alive+dead (contradictory)	0/0/3 U	3/0/0 S	3/0/0 S
Wash. stop France (FP)	0/0/3 U	0/3/0 H	3/0/0 S
Uploaded contract (OOC)	0/3/0 H	0/1/2 U	3/0/0 S
Twitter→Tesla (multi)	0/0/3 U	3/0/0 S	0/0/3 U
Mona Lisa (control)	3/0/0 S	1/2/0 S	3/0/0 S

(S/H/U = STRICT/HYBRID/UNGROUNDED; n=3 each cell.)

Findings:

1. Detector accuracy is 6/6. All trigger questions fire the expected logical_statuses value during preflight (verified programmatically before the bench: stale_risk, contradictory_question, false_premise_suspected, out_of_corpus_risk — matched 1:1 with fixture intent). The detectors are doing what their unit tests claim.

2. Quote mode is the most honest fallback. 4 of 5 trigger questions land all-UNGROUNDED on quote mode. The paraphrase verifier won't substring-match across the corpus when the question's premise has no anchor. Quote mode's mode-gated-off guard works in our favor here.

3. Lattice modes accidentally ground 2 trigger questions to STRICT. Schrödinger's cat (alive+dead) JSON STRICT is defensible — the corpus contains quantum-mechanics articles that legitimately discuss the state. But:

   • JSON STRICT on "when did George Washington stop being president of France?" is NOT defensible. False premise; the model invented an answer that lexically grounded against some chunk. The metacog detector correctly flagged false_premise_suspected; the audit-line tail surfaced · false premise; but the verdict still lands STRICT.
   • JSON STRICT on "what does my uploaded contract say about clause 9?" is the same shape: out-of-corpus reference, model fabricates a grounding.

4. Audit-line tails are doing operator-warning duty correctly. The bench rows persist preflight_logical_statuses and the render layer tails (· stale risk, · false premise, etc.) even on STRICT verdicts — operator sees the warning. But the substrate doesn't refuse execution by default for these shapes.

Implication for #000011 (SOFT_PREFLIGHT_HINT). This bench validates the design rationale for the soft-sidecar ticket: the deterministic metacog detectors flag these shapes correctly, but the corpus accidentally grounds 2/5 of them to STRICT. A model- assisted soft preflight could add independent semantic skepticism ("does George Washington being president of France match historical reality?") that the lexical detectors can't supply. Soft sidecar output would surface as · soft: false_premise_* on the audit-line, distinct from the hard · false premise tail, giving operators a stronger warning when both signals fire.

Implication for default policy. Keep metacognition_block_on_contradiction=False as the default. Schrödinger's cat (alive+dead) would have been rejected unnecessarily under a hard-block, and that's a real-world question with a legitimate answer. The label-only default is correct; operators wanting strictness opt in via --block-on-contradiction.

Bench artifact: bench/qa_results/2026-05-04T02-18-42Z.{jsonl,md}.