arborist/docs/qa-modes-bench-2026-04-30.md

Pointer / Quote / JSON answer modes — bench, failure analysis, roadmap

Date: 2026-04-30 Bench: bench/qa_sweep.py, 22 questions × 3 samples × 3 modes = 198 LLM calls Endpoint: https://hermes.ai.unturf.com/v1 (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx) Corpus: Wikipedia 2003-05-16 cur snapshot, sharded under ~/.aborist/shards Verifier hardening at the time of bench: chunk cap=2, pointer cap=2, coverage threshold=0.30, manual_quote rule removed, partial-grounding split, bare-name guard (≥2 content tokens), lazy-anchor demote, noisy-marker tie-ins.

Aggregate

mode	runs	STRICT	HYBRID	UNGROUNDED	err	strict-rate	grounded (S+H)	mean ratio	mean latency
quote	66	31	20	15	0	47%	51	0.70	7.7s
claim_lattice_pointer	66	14	34	18	0	21%	48	0.54	4.4s
claim_lattice (JSON, guided_json)	66	26	12	9	19	39%*	38	0.51	4.4s

* JSON strict-rate is 26/66 over all runs but 26/47 (55%) over error-free runs. The 19 errors are operationally visible to the user.

Post-retry / post-trim-and-verify rerun (same day)

After landing two improvements derived from the analysis below — HTTP retry on transient 5xx in OpenAICompatibleClient and pointer-cap trim-and-verify in verify_claim_lattice — the bench was rerun on the same 22-question set:

mode	runs	STRICT	HYBRID	UNGROUNDED	err	strict-rate	grounded (S+H)	mean ratio	mean latency
quote	66	31	18	17	0	47%	49	0.70	5.9s
claim_lattice_pointer	66	16	40	10	0	24%	56	0.59	4.6s
claim_lattice (JSON, guided_json + retry)	66	33	21	12	0	50%	54	0.70	5.8s

Deltas vs the pre-improvement bench above:

• JSON errors: 19 → 0 — retry cleared the 502 cluster entirely. The errors were upstream vLLM 502s, not Hermes-can't-produce-JSON failures (the pre-improvement diagnosis above was wrong; HTTP-status inspection proved it).
• JSON strict-rate: 39% → 50% — now leads all three modes (was lowest).
• JSON grounded count: 38 → 54 (+16) — directly from the recovered runs.
• Pointer STRICT: 14 → 16 (+2) — trim-and-verify rescued correct over-cited claims (Mona Lisa case).
• Pointer grounded: 48 → 56 (+8) — UNGROUNDED 18 → 10.
• Quote: largely unchanged within sampling noise, latency dropped 7.7s → 5.9s.

The picture flipped. Pre-improvement, the recommendation was "don't switch default to JSON, the 29% error rate is unacceptable." Post-improvement, JSON has the highest strict-rate, highest grounded count tied with pointer, zero errors, and similar latency. JSON is now the strongest default candidate.

The make query target's default ANSWER_MODE was flipped from claim_lattice_pointer to claim_lattice on 2026-04-30 to reflect this. The library-level DEFAULT_ANSWER_MODE stays "quote" so unit tests using StubClient aren't disrupted; pointer mode is still available via ANSWER_MODE=claim_lattice_pointer.

Architectural fix — JSON mode uses pointer IDs (E1, E2, …) instead of content-addressed evidence_ids

A separate failure mode surfaced after the retry/trim work: cross-document relationship questions consistently landed UNGROUNDED 0/1 in JSON mode despite the model writing the correct answer text. Diagnosis of who is homer simpson's boss? in JSON mode showed:

{"claims":[{"text":"Homer Simpson's boss is Mr. Burns.","evidence_ids":["E1b6e396"]}]}

The runtime had Eed1b6e396 for that chunk; Hermes-3-8B emitted E1b6e396 — a plausible-looking near-miss the verifier rightly flagged as UNKNOWN_EVIDENCE_ID. The model was fabricating content-addressed evidence_ids when the real ones felt awkwardly long.

Fix landed in commit bb8450d: the JSON-mode prompt-facing surface switched from content-addressed evidence_ids (Eed1b6e396) to pointer IDs (E1, E2, …) — same as claim_lattice_pointer mode. The runtime still resolves each pointer_id to its content-addressed evidence_id internally and stores that in evidence_id_pairs for cache & run-DAG continuity. Only the prompt-facing string changes. After the fix, who is homer simpson's boss? lands STRICT 1/1 in JSON mode.

Why pointer IDs work where content-addressed didn't:

• short, enumerable, fabrication-obvious — if only E1-E10 are shown, an emitted E27 reads as a schema violation at a glance
• distribution-natural for small models (citation-style is heavily represented in training)
• the proof path stays content-addressed (the verifier's audit chain still hashes content-addressed ids), so the human/model surface change doesn't weaken the v9.8 admissibility ledger

Token-runaway guard — JSON-mode stop-sequence

Post-pointer-ID-switch bench (2026-04-30T19-55-11Z) found a residual JSON-mode failure on broad-descriptive questions: ~4 of 66 runs landed UNGROUNDED 0/0 at 12-15s instead of the normal 2-5s. Inspection: Hermes emitted a valid claim object then kept generating whitespace / blank lines until max_tokens=512 exhausted. The truncated payload didn't parse and the lenient pre-parser returned no claims.

Concrete instances:

• tell me about the apollo program — 3/3 samples runaway
• tell me about the python programming language — 1/3 runaway

Fix landed in commit f23d3a3: pass stop=["\n\n"] to vLLM in JSON mode. Well-formed JSON-mode output never contains a blank line — the model emits one object on a single line (or with simple internal newlines), never \n\n. The stop sequence is the runaway signature itself; legitimate output is never truncated. Folds into governance_policy_hash via claim_lattice_json_stop_sequences policy field so changing the list invalidates prior cached records.

Bench progression summary

run	bench	quote STRICT	pointer STRICT	JSON STRICT	JSON err	JSON grounded
baseline (no improvements)	morning n=3	31	14	26	19	38
post-retry + trim-and-verify	midday n=3	31	16	33	0	54
post-pointer-ID switch	evening n=3	27	15	31	0	56
post-stop-sequence	late-evening n=3	29	14	37	0	60

JSON mode net: 26 → 37 STRICT (+11), 19 → 0 errors, 38 → 60 grounded (+22) over the day. Strict-rate climbed 39% → 56% (+17pp). Each of the four improvements addressed a named failure mode surfaced by the prior bench.

Each step in the journey was a fix to a specific failure mode named by the prior step's bench. The methodology delivered: name the failure → fix in code → re-bench → confirm or surface the next failure.

Headline: the three modes occupy distinct points on a strict-vs-honest-vs-stable trade-off:

• quote — highest strict-rate but rests on the older verifier path (substring quote-pair extraction). Includes false-STRICT cases the pointer-mode hardening discovered (e.g. claims that pass token-coincidence but cite the wrong source). Slowest of the three.
• claim_lattice_pointer — hardened path; lowest strict-rate because previously-bogus STRICTs were honestly demoted to HYBRID. "Most truthful" mode but visibly fewer perfect-confidence ratings.
• claim_lattice (JSON) — fastest strong-form mode on narrow factoids (5-for-5 perfect strict on the simplest questions) but brittle: 29% raw error rate driven by Hermes-3-8B failing to satisfy guided_json constraints on certain question shapes.

Per-question shape × mode breakdown

Verdict notation: S=STRICT H=HYBRID U=UNGROUNDED e=error. Three samples per cell; majority shown.

Narrow factoids — 5 questions

question	quote	pointer	json
who founded apple computer?	0/3 STRICT (always UNGROUNDED)	3/3 STRICT	3/3 STRICT
what is the capital of france?	2 STRICT, 1 UNGROUNDED	3/3 STRICT	3/3 STRICT
who wrote the linux kernel?	2 STRICT, 1 HYBRID	3/3 HYBRID	3/3 STRICT
when was the python programming language created?	3/3 STRICT	0/3 STRICT (3 UNGROUNDED)	3/3 STRICT
who painted the mona lisa?	0/3 STRICT (1H, 2U)	0/3 STRICT (3 UNGROUNDED)	3/3 STRICT

JSON dominates narrow factoids. Pointer mode is unstable here — Mona Lisa, Apollo Python date, both 0/3 STRICT in pointer because Hermes paraphrases the source span enough that the coverage check trips. quote is similarly variable. JSON's grammar-constrained output makes it cleanest because the model emits short structured {"text":"X","evidence_ids":[...]} records that don't need to negotiate prose-style coverage thresholds.

Broad descriptive — 4 questions

question	quote	pointer	json
tell me about connecticut	3/3 STRICT	3/3 HYBRID	3/3 UNGROUNDED
tell me about python language	3/3 STRICT	3/3 HYBRID	2 HYBRID, 1 UNGROUNDED
tell me about charles babbage	3/3 STRICT	3/3 HYBRID	1 HYBRID, 2 UNGROUNDED
tell me about the apollo program	3/3 STRICT	3/3 HYBRID	3/3 UNGROUNDED

quote mode wins these on aggregate — but the win is partly because the verifier's looser quote-pair check passes claims the pointer / JSON paths reject. Pointer mode produces honest HYBRID (some claims grounded, some not) which is closer to the truth of what Hermes is actually doing on encyclopedic prompts. JSON mode collapses heavily on descriptives: the model often emits no claims it considers solidly grounded → empty-or-near-empty claims array → UNGROUNDED.

Entity list — 3 questions

question	quote	pointer	json
list the founders of microsoft	2 STRICT, 1 UNGROUNDED	3/3 HYBRID	3/3 STRICT
what dinosaurs were in the first jurassic park film?	1 STRICT, 2 HYBRID	0/3 STRICT (1H, 2U)	3/3 HYBRID
who are the members of the beatles?	3/3 STRICT	3/3 HYBRID	3/3 HYBRID

JSON handles list shapes well — structured output is a natural fit. Pointer mode's bare-name guard correctly catches the JP-dinosaurs case (2 UNGROUNDED) where the model emits one-token names; JSON mode sidesteps the bare-name guard because each claim text has the surrounding context required by the schema.

Relationship / multi-fact — 3 questions

question	quote	pointer	json
who is supermans girlfriend?	0 STRICT, 2 HYBRID, 1 UNGROUNDED	0 STRICT, 2 HYBRID, 1 UNGROUNDED	2 STRICT, 1 ERROR
who is bilbo baggins's nephew?	0/3 STRICT (3 UNGROUNDED)	3/3 STRICT	0/3 (3 ERRORS)
what is the relationship between linux and unix?	1 STRICT, 2 HYBRID	3/3 HYBRID	0/3 (3 ERRORS)

Pointer mode shines on bilbo's nephew (3/3 STRICT). Quote and JSON both fail. Mixed picture overall.

Comparison — 2 questions

question	quote	pointer	json
what's the difference between linux and bsd?	2 STRICT, 1 HYBRID	0/3 STRICT (3 UNGROUNDED)	0/3 (3 ERRORS)
how does intel compare to amd?	0/3 STRICT (3 UNGROUNDED)	0/3 STRICT (3 UNGROUNDED)	0/3 (3 ERRORS)

Comparison questions break JSON mode catastrophically (6 errors / 6 runs). Quote handles linux-vs-bsd; pointer fails it. Intel-vs-AMD is hard for everyone — corpus likely thin or rivalry-exclusion is too aggressive.

Niche / partial — 2 questions

question	quote	pointer	json
what is the boltzmann constant?	1 STRICT, 2 HYBRID	2 STRICT, 1 HYBRID	0/3 (3 ERRORS)
who invented the doppler effect?	2 STRICT, 1 HYBRID	3/3 STRICT	0/3 (3 ERRORS)

Pointer wins niche-factoid where corpus content is solid. JSON fails completely — these queries return very fast (0.8–1.5s) with errors, suggesting Hermes is producing malformed JSON rather than schema-valid output.

Adversarial / honest-refusal — 3 questions

question	quote	pointer	json
isn't it true that the great wall of china is visible from space?	3/3 HYBRID	3/3 HYBRID	3/3 STRICT
who is the prime minister of mars?	0 STRICT, 2 HYBRID, 1 UNGROUNDED	3/3 UNGROUNDED	3/3 STRICT ← honest "no PM exists" + cited a definitional chunk
what year did the cold fusion breakthrough happen?	3/3 HYBRID	3/3 HYBRID	3/3 HYBRID

JSON's strict-form on Mars is a correct refutation: model wrote "There is no prime minister of Mars, as Mars is not a sovereign nation" and cited a chunk explaining what "prime minister" means (Canadian PM definition span). Coverage passed because the term prime minister overlapped. This is the right behavior for a refute-the-premise question — the verifier passed honest definitional grounding.

Failure modes, by class

F1 — JSON-mode parse-failure errors (29% of JSON runs)

Six question shapes consistently 3/3 ERROR in JSON mode at 0.8–1.5s latency:

• comparison (linux vs unix, linux vs bsd, intel vs amd)
• technical-term niche (boltzmann constant, doppler effect)
• relationship (bilbo's nephew)
• partial (supermans girlfriend, 1 of 3)

Pattern: very fast latency = Hermes returns text that fails JSON parsing or violates the guided_json schema. Same questions in pointer mode often succeed (bilbo: pointer 3/3 STRICT vs JSON 3/3 ERROR; doppler: pointer 3/3 STRICT vs JSON 3/3 ERROR).

Root cause hypothesis: vLLM's guided_json is forcing a particular output structure, but Hermes-3-8B's training distribution for these question shapes wants prose. The grammar mask collides with the model's natural completion → degenerate output → schema violation → _lenient_json_parse produces json_fixups warnings or fails outright.

F2 — Pointer-mode mode-collapse on broad descriptives

Confirmed: connecticut, python, charles babbage, apollo program — all 3/3 HYBRID with claim ratios in 0.33–0.78 range. Hermes writes encyclopedic paragraphs that paraphrase source but each claim text has more tokens than the cited span supports → coverage threshold passes some claims, rejects others.

This is correct honest behavior given the chunk cap (2 chunks per source). Quote mode "wins" these only because its verifier has more lenient acceptance.

F3 — Pointer-mode strict-rate softness on narrow factoids

Surprising: pointer mode hits 0/3 STRICT on mona lisa, python date, on the same questions JSON nails 3/3 STRICT. The model writes narrow correct claims but the coverage threshold or claim-shape check fires.

Inspection of failed mona lisa pointer answers needed — likely the model wrote a multi-clause claim like "The Mona Lisa was painted by Leonardo da Vinci, an Italian Renaissance artist." (~6 content tokens) and the cited span has only "Leonardo da Vinci" verbatim — coverage 2/6 = 33% → just over threshold but the claim contains "Italian", "Renaissance", "artist" not in the span. Need to verify.

F4 — Quote-mode false-STRICT on bogus citations

Quote-mode's strict-rate (47%) is the highest but includes false-positives we discovered while hardening pointer mode. Specifically: claims wrapped in "..." whose quoted content contains tokens that exist somewhere in context but not in a span that supports the claim. Pre-pointer-hardening, a similar test set would show pointer at 50% STRICT too — the pointer hardening exposed and rejected the false-STRICTs that quote still credits.

This means quote-mode's strict-rate over-reports honest grounding. The correct comparison metric is strict-rate over an audited subset which is laborious but is the only honest scoreboard.

F5 — Lazy-anchor smell still active

JP-dinosaurs in pointer mode: 0/3 STRICT (1 HYBRID, 2 UNGROUNDED) — the bare-name guard correctly rejected single-word claims. JSON mode: 3/3 HYBRID — JSON's structured text field carries enough context to clear the bare-name guard, but the citations are still anchoring on game tie-in chunks (Jurassic Park (NES game) made it into top-K despite the noisy-marker addition for "operation genesis" + "the game" + "video games" — "(NES game)" parens form passes the substring check but not in the way I'd hoped — the marker "the game" matches "the NES game" only if the title is exactly that, which it isn't here. Need to extend markers further or generalize via regex).

Conclusions

1. Don't switch the default to JSON. The 29% raw error rate on diverse questions is operationally unacceptable — 6 of 22 question shapes catastrophically fail. JSON wins narrow factoids cleanly but Hermes-3-8B can't sustain valid JSON output on comparison, technical, and relationship questions.

2. Pointer mode is the most honest default. Lower strict-rate, but every "lost" STRICT vs quote mode was a false-positive caught by the hardening (chunk cap, coverage threshold, bare-name guard, lazy-anchor demote). The honest narrative beats the optimistic one in a verifier substrate.

3. Quote mode's strict-rate is misleading. The 47% includes false-STRICTs the pointer hardening exposed; those same patterns would demote to HYBRID under the pointer verifier's checks.

4. JSON mode is the right answer for narrow-factoid hot paths. If a deployment can detect "narrow factoid" question shape ahead of time, routing to JSON gives the cleanest perfect-grounding rate at 4.4s mean latency. For the rest, pointer is more robust.

5. A fallback strategy makes JSON viable as default. Try JSON first; on parse error, fall back to pointer mode with the same retrieval. Captures JSON's 5/5 narrow-factoid wins while bounding the worst case at pointer mode's 21% strict-rate / 48 grounded.

6. Mode-collapse on broad descriptives is structural to Hermes-3-8B. No prompt or policy tweak fully cures it. Larger models with better instruction-following (Hermes 70B class, Qwen 3.6 reasoner) would probably help. The chunk cap and bare-name guard make the failure honest rather than hidden.

Roadmap — making solutions better

Highest leverage

1. JSON-fallback dispatch in the runtime. When answer_mode="claim_lattice" returns a parse error / schema violation, retry once with answer_mode="claim_lattice_pointer" against the same retrieved context (no second LLM round trip needed if JSON failure is detected pre-call; otherwise one extra call). Result records both attempts in the run-DAG so the audit chain documents what happened. Estimated effort: small (one new dispatch path in query.py).
2. Stronger JSON prompt — or: relax guided_json for diverse shapes. Investigate whether guided_json: false + lenient JSON parsing (which the codebase already supports) reduces the error rate below 29%. May trade some narrow-factoid wins for fewer errors.
3. Pointer narrow-factoid recovery. Mona Lisa / python-date pointer flunks need root-cause diagnosis. Likely the bare-name guard is too aggressive on legitimate narrow claims that have only 2 content tokens after the spotlight stopword filter, or the coverage threshold is biting on multi-clause claims. Specific fix candidates:
   • Add a "narrow-factoid" detection (≤3 source chunks visible, question has a single clear interrogative) → relax coverage to 0.20 for these.
   • OR: move the bare-name guard threshold down to 1 (only catches truly empty claims like "Triceratops" with 0 tokens), and rely on coverage threshold alone for the JP-dinosaurs failure mode.

Medium leverage

4. Extend noisy markers to handle parenthetical disambiguation. "(NES game)", "(arcade game)", "(Sega adaptation)" are common Wikipedia spinoff disambiguations. A regex \((.*\bgame\b.*)\) would catch all (X game) parens variants. Folds into governance_policy_hash.
5. Per-source chunk cap responsive to source role. Primary answer source gets 4 chunks; background/noisy sources get 0–1. Already partly there via SOURCE_ROLE_BUDGET_WEIGHTS but the count cap is uniform. Combining char budget + chunk count + role weighting.
6. Comparison-question repair tier. Detect "X vs Y" / "compare X and Y" / "difference between X and Y" patterns and retrieve sources for both X and Y before assembling context. Currently retrieval treats the full query as one keyword bag and rivalry-exclusion fights against this.

Long-tail / experimental

7. Larger model class for descriptives. Route broad-descriptive questions to a 70B-class model (Hermes 70B, Qwen 3.6 reasoner) with claim_lattice JSON mode + grammar guidance. Honesty gains on connecticut/apollo would be measurable.
8. Question-shape classifier. Detect narrow-factoid / broad-descriptive / list / comparison / niche before retrieval; route per-shape to optimal mode + chunk cap + repair tier. Expensive in code but each individual rule is cheap.
9. Verifier semantic check (soft signal). A "did the claim's predicate match the cited span's frame?" signal that runs alongside the lexical checks but never enters the proof path. Could catch the JP-dinos "Triceratops in JP1 cited to Operation Genesis" failure (predicate mismatch). NER + relation extraction territory; substantial.

Persistence

• Bench: bench/qa_results/2026-04-30T17-05-28Z.{jsonl,md} (3-mode merged not yet — quote+pointer in 17-05-28, JSON-only in a separate timestamp)
• Code state at bench time: commit 873cfa8 (bare-name + smell-demote + game tie-in markers) on top of c3da725 (JSON-mode runner wiring + partial-grounding split).
• Endpoint: live hermes.ai.unturf.com/v1 — Hermes-3 Llama-3.1-8B-FP8-Dynamic, no auth.
• Reproduce: make bench-qa (default modes now = quote + pointer + JSON) or make bench-qa BENCH_QA_MODES=claim_lattice for JSON-only.