# Pointer / Quote / JSON answer modes — bench, failure analysis, roadmap

**Date:** 2026-04-30
**Bench:** `bench/qa_sweep.py`, 22 questions × 3 samples × 3 modes = 198 LLM calls
**Endpoint:** `https://hermes.ai.unturf.com/v1` (Hermes-3-Llama-3.1-8B-FP8-Dynamic, vLLM, 82K ctx)
**Corpus:** Wikipedia 2003-05-16 cur snapshot, sharded under `~/.aborist/shards`
**Verifier hardening at the time of bench:** chunk cap=2, pointer cap=2, coverage threshold=0.30, manual_quote rule removed, partial-grounding split, bare-name guard (≥2 content tokens), lazy-anchor demote, noisy-marker tie-ins.

## Aggregate

| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | grounded (S+H) | mean ratio | mean latency |
|------|------|--------|--------|------------|-----|-------------|----------------|------------|--------------|
| `quote` | 66 | 31 | 20 | 15 | 0 | **47%** | 51 | 0.70 | 7.7s |
| `claim_lattice_pointer` | 66 | 14 | 34 | 18 | 0 | 21% | 48 | 0.54 | **4.4s** |
| `claim_lattice` (JSON, guided_json) | 66 | 26 | 12 | 9 | **19** | 39%* | 38 | 0.51 | 4.4s |

\* JSON strict-rate is 26/66 over all runs but 26/47 (55%) over error-free runs. The 19 errors are operationally visible to the user.

### Post-retry / post-trim-and-verify rerun (same day)

After landing two improvements derived from the analysis below — HTTP retry on transient 5xx in `OpenAICompatibleClient` and pointer-cap trim-and-verify in `verify_claim_lattice` — the bench was rerun on the same 22-question set:

| mode | runs | STRICT | HYBRID | UNGROUNDED | err | strict-rate | grounded (S+H) | mean ratio | mean latency |
|------|------|--------|--------|------------|-----|-------------|----------------|------------|--------------|
| `quote` | 66 | 31 | 18 | 17 | 0 | 47% | 49 | 0.70 | 5.9s |
| `claim_lattice_pointer` | 66 | 16 | 40 | 10 | 0 | 24% | **56** | 0.59 | **4.6s** |
| `claim_lattice` (JSON, guided_json + retry) | 66 | **33** | 21 | 12 | **0** | **50%** | 54 | 0.70 | 5.8s |

**Deltas vs the pre-improvement bench above:**
- JSON errors: **19 → 0** — retry cleared the 502 cluster entirely. The errors *were* upstream vLLM 502s, not Hermes-can't-produce-JSON failures (the pre-improvement diagnosis above was wrong; HTTP-status inspection proved it).
- JSON strict-rate: 39% → **50%** — now leads all three modes (was lowest).
- JSON grounded count: 38 → 54 (+16) — directly from the recovered runs.
- Pointer STRICT: 14 → 16 (+2) — trim-and-verify rescued correct over-cited claims (Mona Lisa case).
- Pointer grounded: 48 → 56 (+8) — UNGROUNDED 18 → 10.
- Quote: largely unchanged within sampling noise, latency dropped 7.7s → 5.9s.

**The picture flipped.** Pre-improvement, the recommendation was "don't switch default to JSON, the 29% error rate is unacceptable." Post-improvement, JSON has the **highest strict-rate**, **highest grounded count tied with pointer**, **zero errors**, and similar latency. JSON is now the strongest default candidate.

The `make query` target's default `ANSWER_MODE` was flipped from `claim_lattice_pointer` to `claim_lattice` on 2026-04-30 to reflect this. The library-level `DEFAULT_ANSWER_MODE` stays `"quote"` so unit tests using `StubClient` aren't disrupted; pointer mode is still available via `ANSWER_MODE=claim_lattice_pointer`.

### Architectural fix — JSON mode uses pointer IDs (E1, E2, …) instead of content-addressed evidence_ids

A separate failure mode surfaced after the retry/trim work: cross-document relationship questions consistently landed `UNGROUNDED 0/1` in JSON mode despite the model writing the correct answer text. Diagnosis of `who is homer simpson's boss?` in JSON mode showed:

```
{"claims":[{"text":"Homer Simpson's boss is Mr. Burns.","evidence_ids":["E1b6e396"]}]}
```

The runtime had `Eed1b6e396` for that chunk; Hermes-3-8B emitted `E1b6e396` — a plausible-looking near-miss the verifier rightly flagged as `UNKNOWN_EVIDENCE_ID`. The model was *fabricating* content-addressed evidence_ids when the real ones felt awkwardly long.

Fix landed in commit `bb8450d`: the JSON-mode prompt-facing surface switched from content-addressed evidence_ids (`Eed1b6e396`) to pointer IDs (`E1`, `E2`, …) — same as `claim_lattice_pointer` mode. The runtime still resolves each pointer_id to its content-addressed evidence_id internally and stores **that** in `evidence_id_pairs` for cache & run-DAG continuity. Only the prompt-facing string changes. After the fix, `who is homer simpson's boss?` lands `STRICT 1/1` in JSON mode.

Why pointer IDs work where content-addressed didn't:
- short, enumerable, fabrication-obvious — if only `E1`-`E10` are shown, an emitted `E27` reads as a schema violation at a glance
- distribution-natural for small models (citation-style is heavily represented in training)
- the proof path stays content-addressed (the verifier's audit chain still hashes content-addressed ids), so the human/model surface change doesn't weaken the v9.8 admissibility ledger

### Token-runaway guard — JSON-mode stop-sequence

Post-pointer-ID-switch bench (2026-04-30T19-55-11Z) found a residual JSON-mode failure on broad-descriptive questions: ~4 of 66 runs landed `UNGROUNDED 0/0` at 12-15s instead of the normal 2-5s. Inspection: Hermes emitted a valid claim object then kept generating whitespace / blank lines until `max_tokens=512` exhausted. The truncated payload didn't parse and the lenient pre-parser returned no claims.

Concrete instances:
- `tell me about the apollo program` — 3/3 samples runaway
- `tell me about the python programming language` — 1/3 runaway

Fix landed in commit `f23d3a3`: pass `stop=["\n\n"]` to vLLM in JSON mode. Well-formed JSON-mode output never contains a blank line — the model emits one object on a single line (or with simple internal newlines), never `\n\n`. The stop sequence is the runaway signature itself; legitimate output is never truncated. Folds into `governance_policy_hash` via `claim_lattice_json_stop_sequences` policy field so changing the list invalidates prior cached records.

### Bench progression summary

| run | bench | quote STRICT | pointer STRICT | JSON STRICT | JSON err | JSON grounded |
|-----|-------|--------------|----------------|-------------|----------|----------------|
| baseline (no improvements) | morning n=3 | 31 | 14 | 26 | **19** | 38 |
| post-retry + trim-and-verify | midday n=3 | 31 | 16 | **33** | 0 | 54 |
| post-pointer-ID switch | evening n=3 | 27 | 15 | 31 | 0 | 56 |
| post-stop-sequence | late-evening n=3 | 29 | 14 | **37** | 0 | **60** |

JSON mode net: 26 → 37 STRICT (+11), 19 → 0 errors, 38 → 60 grounded (+22) over the day. Strict-rate climbed 39% → 56% (+17pp). Each of the four improvements addressed a named failure mode surfaced by the prior bench.

Each step in the journey was a fix to a specific failure mode named by the prior step's bench. The methodology delivered: name the failure → fix in code → re-bench → confirm or surface the next failure.

**Headline:** the three modes occupy distinct points on a strict-vs-honest-vs-stable trade-off:

- `quote` — highest strict-rate but rests on the older verifier path (substring quote-pair extraction). Includes false-STRICT cases the pointer-mode hardening discovered (e.g. claims that pass token-coincidence but cite the wrong source). Slowest of the three.
- `claim_lattice_pointer` — hardened path; lowest strict-rate because previously-bogus STRICTs were honestly demoted to HYBRID. "Most truthful" mode but visibly fewer perfect-confidence ratings.
- `claim_lattice` (JSON) — fastest strong-form mode on narrow factoids (5-for-5 perfect strict on the simplest questions) but **brittle**: 29% raw error rate driven by Hermes-3-8B failing to satisfy `guided_json` constraints on certain question shapes.

## Per-question shape × mode breakdown

Verdict notation: `S=STRICT  H=HYBRID  U=UNGROUNDED  e=error`. Three samples per cell; majority shown.

### Narrow factoids — 5 questions

| question | quote | pointer | json |
|----------|-------|---------|------|
| who founded apple computer? | 0/3 STRICT (always UNGROUNDED) | 3/3 STRICT | **3/3 STRICT** |
| what is the capital of france? | 2 STRICT, 1 UNGROUNDED | 3/3 STRICT | **3/3 STRICT** |
| who wrote the linux kernel? | 2 STRICT, 1 HYBRID | 3/3 HYBRID | **3/3 STRICT** |
| when was the python programming language created? | 3/3 STRICT | 0/3 STRICT (3 UNGROUNDED) | **3/3 STRICT** |
| who painted the mona lisa? | 0/3 STRICT (1H, 2U) | 0/3 STRICT (3 UNGROUNDED) | **3/3 STRICT** |

JSON dominates narrow factoids. Pointer mode is unstable here — Mona Lisa, Apollo Python date, both 0/3 STRICT in pointer because Hermes paraphrases the source span enough that the coverage check trips. `quote` is similarly variable. JSON's grammar-constrained output makes it cleanest because the model emits short structured `{"text":"X","evidence_ids":[...]}` records that don't need to negotiate prose-style coverage thresholds.

### Broad descriptive — 4 questions

| question | quote | pointer | json |
|----------|-------|---------|------|
| tell me about connecticut | 3/3 STRICT | 3/3 HYBRID | 3/3 UNGROUNDED |
| tell me about python language | 3/3 STRICT | 3/3 HYBRID | 2 HYBRID, 1 UNGROUNDED |
| tell me about charles babbage | 3/3 STRICT | 3/3 HYBRID | 1 HYBRID, 2 UNGROUNDED |
| tell me about the apollo program | 3/3 STRICT | 3/3 HYBRID | 3/3 UNGROUNDED |

`quote` mode wins these on aggregate — but the win is partly because the verifier's looser quote-pair check passes claims the pointer / JSON paths reject. Pointer mode produces honest HYBRID (some claims grounded, some not) which is closer to the truth of what Hermes is actually doing on encyclopedic prompts. JSON mode collapses heavily on descriptives: the model often emits no claims it considers solidly grounded → empty-or-near-empty `claims` array → UNGROUNDED.

### Entity list — 3 questions

| question | quote | pointer | json |
|----------|-------|---------|------|
| list the founders of microsoft | 2 STRICT, 1 UNGROUNDED | 3/3 HYBRID | **3/3 STRICT** |
| what dinosaurs were in the first jurassic park film? | 1 STRICT, 2 HYBRID | 0/3 STRICT (1H, 2U) | **3/3 HYBRID** |
| who are the members of the beatles? | 3/3 STRICT | 3/3 HYBRID | 3/3 HYBRID |

JSON handles list shapes well — structured output is a natural fit. Pointer mode's bare-name guard correctly catches the JP-dinosaurs case (2 UNGROUNDED) where the model emits one-token names; JSON mode sidesteps the bare-name guard because each claim text has the surrounding context required by the schema.

### Relationship / multi-fact — 3 questions

| question | quote | pointer | json |
|----------|-------|---------|------|
| who is supermans girlfriend? | 0 STRICT, 2 HYBRID, 1 UNGROUNDED | 0 STRICT, 2 HYBRID, 1 UNGROUNDED | 2 STRICT, 1 ERROR |
| who is bilbo baggins's nephew? | 0/3 STRICT (3 UNGROUNDED) | **3/3 STRICT** | 0/3 (3 ERRORS) |
| what is the relationship between linux and unix? | 1 STRICT, 2 HYBRID | 3/3 HYBRID | 0/3 (3 ERRORS) |

Pointer mode shines on `bilbo's nephew` (3/3 STRICT). Quote and JSON both fail. Mixed picture overall.

### Comparison — 2 questions

| question | quote | pointer | json |
|----------|-------|---------|------|
| what's the difference between linux and bsd? | 2 STRICT, 1 HYBRID | 0/3 STRICT (3 UNGROUNDED) | 0/3 (3 ERRORS) |
| how does intel compare to amd? | 0/3 STRICT (3 UNGROUNDED) | 0/3 STRICT (3 UNGROUNDED) | 0/3 (3 ERRORS) |

Comparison questions break JSON mode catastrophically (6 errors / 6 runs). Quote handles linux-vs-bsd; pointer fails it. Intel-vs-AMD is hard for everyone — corpus likely thin or rivalry-exclusion is too aggressive.

### Niche / partial — 2 questions

| question | quote | pointer | json |
|----------|-------|---------|------|
| what is the boltzmann constant? | 1 STRICT, 2 HYBRID | 2 STRICT, 1 HYBRID | 0/3 (3 ERRORS) |
| who invented the doppler effect? | 2 STRICT, 1 HYBRID | **3/3 STRICT** | 0/3 (3 ERRORS) |

Pointer wins niche-factoid where corpus content is solid. JSON fails completely — these queries return very fast (0.8–1.5s) with errors, suggesting Hermes is producing malformed JSON rather than schema-valid output.

### Adversarial / honest-refusal — 3 questions

| question | quote | pointer | json |
|----------|-------|---------|------|
| isn't it true that the great wall of china is visible from space? | 3/3 HYBRID | 3/3 HYBRID | 3/3 STRICT |
| who is the prime minister of mars? | 0 STRICT, 2 HYBRID, 1 UNGROUNDED | 3/3 UNGROUNDED | **3/3 STRICT** ← honest "no PM exists" + cited a definitional chunk |
| what year did the cold fusion breakthrough happen? | 3/3 HYBRID | 3/3 HYBRID | 3/3 HYBRID |

JSON's strict-form on Mars is a *correct refutation*: model wrote "There is no prime minister of Mars, as Mars is not a sovereign nation" and cited a chunk explaining what "prime minister" means (Canadian PM definition span). Coverage passed because the term `prime minister` overlapped. This is the **right** behavior for a refute-the-premise question — the verifier passed honest definitional grounding.

## Failure modes, by class

### F1 — JSON-mode parse-failure errors (29% of JSON runs)

Six question shapes consistently 3/3 ERROR in JSON mode at 0.8–1.5s latency:

- comparison (linux vs unix, linux vs bsd, intel vs amd)
- technical-term niche (boltzmann constant, doppler effect)
- relationship (bilbo's nephew)
- partial (supermans girlfriend, 1 of 3)

Pattern: very fast latency = Hermes returns text that fails JSON parsing or violates the `guided_json` schema. Same questions in pointer mode often succeed (bilbo: pointer 3/3 STRICT vs JSON 3/3 ERROR; doppler: pointer 3/3 STRICT vs JSON 3/3 ERROR).

**Root cause hypothesis:** vLLM's `guided_json` is forcing a particular output structure, but Hermes-3-8B's training distribution for these question shapes wants prose. The grammar mask collides with the model's natural completion → degenerate output → schema violation → `_lenient_json_parse` produces `json_fixups` warnings or fails outright.

### F2 — Pointer-mode mode-collapse on broad descriptives

Confirmed: connecticut, python, charles babbage, apollo program — all 3/3 HYBRID with claim ratios in 0.33–0.78 range. Hermes writes encyclopedic paragraphs that paraphrase source but each claim text has more tokens than the cited span supports → coverage threshold passes some claims, rejects others.

This is *correct* honest behavior given the chunk cap (2 chunks per source). Quote mode "wins" these only because its verifier has more lenient acceptance.

### F3 — Pointer-mode strict-rate softness on narrow factoids

Surprising: pointer mode hits 0/3 STRICT on `mona lisa`, `python date`, on the same questions JSON nails 3/3 STRICT. The model writes narrow correct claims but the coverage threshold or claim-shape check fires.

Inspection of failed mona lisa pointer answers needed — likely the model wrote a multi-clause claim like "The Mona Lisa was painted by Leonardo da Vinci, an Italian Renaissance artist." (~6 content tokens) and the cited span has only "Leonardo da Vinci" verbatim — coverage 2/6 = 33% → just over threshold but the claim contains "Italian", "Renaissance", "artist" not in the span. Need to verify.

### F4 — Quote-mode false-STRICT on bogus citations

Quote-mode's strict-rate (47%) is the highest but includes false-positives we discovered while hardening pointer mode. Specifically: claims wrapped in `"..."` whose quoted content contains tokens that exist somewhere in context but not in a span that supports the claim. Pre-pointer-hardening, a similar test set would show pointer at 50% STRICT too — the pointer hardening exposed and rejected the false-STRICTs that quote still credits.

This means quote-mode's strict-rate over-reports honest grounding. The correct comparison metric is **strict-rate over an audited subset** which is laborious but is the only honest scoreboard.

### F5 — Lazy-anchor smell still active

JP-dinosaurs in pointer mode: 0/3 STRICT (1 HYBRID, 2 UNGROUNDED) — the bare-name guard correctly rejected single-word claims. JSON mode: 3/3 HYBRID — JSON's structured `text` field carries enough context to clear the bare-name guard, but the citations are still anchoring on game tie-in chunks (`Jurassic Park (NES game)` made it into top-K despite the noisy-marker addition for "operation genesis" + "the game" + "video games" — "(NES game)" parens form passes the substring check but not in the way I'd hoped — the marker `"the game"` matches `"the NES game"` only if the title is exactly that, which it isn't here. Need to extend markers further or generalize via regex).

## Conclusions

1. **Don't switch the default to JSON.** The 29% raw error rate on diverse questions is operationally unacceptable — 6 of 22 question shapes catastrophically fail. JSON wins narrow factoids cleanly but Hermes-3-8B can't sustain valid JSON output on comparison, technical, and relationship questions.

2. **Pointer mode is the most honest default.** Lower strict-rate, but every "lost" STRICT vs quote mode was a false-positive caught by the hardening (chunk cap, coverage threshold, bare-name guard, lazy-anchor demote). The honest narrative beats the optimistic one in a verifier substrate.

3. **Quote mode's strict-rate is misleading.** The 47% includes false-STRICTs the pointer hardening exposed; those same patterns would demote to HYBRID under the pointer verifier's checks.

4. **JSON mode is the right answer for narrow-factoid hot paths.** If a deployment can detect "narrow factoid" question shape ahead of time, routing to JSON gives the cleanest perfect-grounding rate at 4.4s mean latency. For the rest, pointer is more robust.

5. **A fallback strategy makes JSON viable as default.** Try JSON first; on parse error, fall back to pointer mode with the same retrieval. Captures JSON's 5/5 narrow-factoid wins while bounding the worst case at pointer mode's 21% strict-rate / 48 grounded.

6. **Mode-collapse on broad descriptives is structural to Hermes-3-8B.** No prompt or policy tweak fully cures it. Larger models with better instruction-following (Hermes 70B class, Qwen 3.6 reasoner) would probably help. The chunk cap and bare-name guard make the failure honest rather than hidden.

## Roadmap — making solutions better

### Highest leverage

1. **JSON-fallback dispatch in the runtime.** When `answer_mode="claim_lattice"` returns a parse error / schema violation, retry once with `answer_mode="claim_lattice_pointer"` against the same retrieved context (no second LLM round trip needed if JSON failure is detected pre-call; otherwise one extra call). Result records both attempts in the run-DAG so the audit chain documents what happened. Estimated effort: small (one new dispatch path in `query.py`).
2. **Stronger JSON prompt — or: relax `guided_json` for diverse shapes.** Investigate whether `guided_json: false` + lenient JSON parsing (which the codebase already supports) reduces the error rate below 29%. May trade some narrow-factoid wins for fewer errors.
3. **Pointer narrow-factoid recovery.** Mona Lisa / python-date pointer flunks need root-cause diagnosis. Likely the bare-name guard is too aggressive on legitimate narrow claims that have only 2 content tokens after the spotlight stopword filter, or the coverage threshold is biting on multi-clause claims. Specific fix candidates:
   - Add a "narrow-factoid" detection (≤3 source chunks visible, question has a single clear interrogative) → relax coverage to 0.20 for these.
   - OR: move the bare-name guard threshold down to 1 (only catches truly empty claims like "Triceratops" with 0 tokens), and rely on coverage threshold alone for the JP-dinosaurs failure mode.

### Medium leverage

4. **Extend noisy markers to handle parenthetical disambiguation.** `"(NES game)"`, `"(arcade game)"`, `"(Sega adaptation)"` are common Wikipedia spinoff disambiguations. A regex `\((.*\bgame\b.*)\)` would catch all `(X game)` parens variants. Folds into `governance_policy_hash`.
5. **Per-source chunk cap responsive to source role.** Primary answer source gets 4 chunks; background/noisy sources get 0–1. Already partly there via `SOURCE_ROLE_BUDGET_WEIGHTS` but the count cap is uniform. Combining char budget + chunk count + role weighting.
6. **Comparison-question repair tier.** Detect "X vs Y" / "compare X and Y" / "difference between X and Y" patterns and retrieve sources for both X and Y before assembling context. Currently retrieval treats the full query as one keyword bag and rivalry-exclusion fights against this.

### Long-tail / experimental

7. **Larger model class for descriptives.** Route broad-descriptive questions to a 70B-class model (Hermes 70B, Qwen 3.6 reasoner) with `claim_lattice` JSON mode + grammar guidance. Honesty gains on connecticut/apollo would be measurable.
8. **Question-shape classifier.** Detect narrow-factoid / broad-descriptive / list / comparison / niche before retrieval; route per-shape to optimal mode + chunk cap + repair tier. Expensive in code but each individual rule is cheap.
9. **Verifier semantic check (soft signal).** A "did the claim's predicate match the cited span's frame?" signal that runs alongside the lexical checks but never enters the proof path. Could catch the JP-dinos "Triceratops in JP1 cited to Operation Genesis" failure (predicate mismatch). NER + relation extraction territory; substantial.

## Persistence

- Bench: `bench/qa_results/2026-04-30T17-05-28Z.{jsonl,md}` (3-mode merged not yet — quote+pointer in 17-05-28, JSON-only in a separate timestamp)
- Code state at bench time: commit `873cfa8` (bare-name + smell-demote + game tie-in markers) on top of `c3da725` (JSON-mode runner wiring + partial-grounding split).
- Endpoint: live `hermes.ai.unturf.com/v1` — Hermes-3 Llama-3.1-8B-FP8-Dynamic, no auth.
- Reproduce: `make bench-qa` (default modes now = quote + pointer + JSON) or `make bench-qa BENCH_QA_MODES=claim_lattice` for JSON-only.
