diff --git a/docs/qa-modes-bench-2026-04-30.md b/docs/qa-modes-bench-2026-04-30.md index ad85e0c..8e41304 100644 --- a/docs/qa-modes-bench-2026-04-30.md +++ b/docs/qa-modes-bench-2026-04-30.md @@ -38,6 +38,44 @@ After landing two improvements derived from the analysis below — HTTP retry on The `make query` target's default `ANSWER_MODE` was flipped from `claim_lattice_pointer` to `claim_lattice` on 2026-04-30 to reflect this. The library-level `DEFAULT_ANSWER_MODE` stays `"quote"` so unit tests using `StubClient` aren't disrupted; pointer mode is still available via `ANSWER_MODE=claim_lattice_pointer`. +### Architectural fix — JSON mode uses pointer IDs (E1, E2, …) instead of content-addressed evidence_ids + +A separate failure mode surfaced after the retry/trim work: cross-document relationship questions consistently landed `UNGROUNDED 0/1` in JSON mode despite the model writing the correct answer text. Diagnosis of `who is homer simpson's boss?` in JSON mode showed: + +``` +{"claims":[{"text":"Homer Simpson's boss is Mr. Burns.","evidence_ids":["E1b6e396"]}]} +``` + +The runtime had `Eed1b6e396` for that chunk; Hermes-3-8B emitted `E1b6e396` — a plausible-looking near-miss the verifier rightly flagged as `UNKNOWN_EVIDENCE_ID`. The model was *fabricating* content-addressed evidence_ids when the real ones felt awkwardly long. + +Fix landed in commit `bb8450d`: the JSON-mode prompt-facing surface switched from content-addressed evidence_ids (`Eed1b6e396`) to pointer IDs (`E1`, `E2`, …) — same as `claim_lattice_pointer` mode. The runtime still resolves each pointer_id to its content-addressed evidence_id internally and stores **that** in `evidence_id_pairs` for cache & run-DAG continuity. Only the prompt-facing string changes. After the fix, `who is homer simpson's boss?` lands `STRICT 1/1` in JSON mode. + +Why pointer IDs work where content-addressed didn't: +- short, enumerable, fabrication-obvious — if only `E1`-`E10` are shown, an emitted `E27` reads as a schema violation at a glance +- distribution-natural for small models (citation-style is heavily represented in training) +- the proof path stays content-addressed (the verifier's audit chain still hashes content-addressed ids), so the human/model surface change doesn't weaken the v9.8 admissibility ledger + +### Token-runaway guard — JSON-mode stop-sequence + +Post-pointer-ID-switch bench (2026-04-30T19-55-11Z) found a residual JSON-mode failure on broad-descriptive questions: ~4 of 66 runs landed `UNGROUNDED 0/0` at 12-15s instead of the normal 2-5s. Inspection: Hermes emitted a valid claim object then kept generating whitespace / blank lines until `max_tokens=512` exhausted. The truncated payload didn't parse and the lenient pre-parser returned no claims. + +Concrete instances: +- `tell me about the apollo program` — 3/3 samples runaway +- `tell me about the python programming language` — 1/3 runaway + +Fix landed in commit `f23d3a3`: pass `stop=["\n\n"]` to vLLM in JSON mode. Well-formed JSON-mode output never contains a blank line — the model emits one object on a single line (or with simple internal newlines), never `\n\n`. The stop sequence is the runaway signature itself; legitimate output is never truncated. Folds into `governance_policy_hash` via `claim_lattice_json_stop_sequences` policy field so changing the list invalidates prior cached records. + +### Bench progression summary + +| run | bench | quote STRICT | pointer STRICT | JSON STRICT | JSON err | JSON grounded | +|-----|-------|--------------|----------------|-------------|----------|----------------| +| baseline (no improvements) | morning n=3 | 31 | 14 | 26 | **19** | 38 | +| post-retry + trim-and-verify | midday n=3 | 31 | 16 | **33** | 0 | 54 | +| post-pointer-ID switch | evening n=3 | 27 | 15 | 31 | 0 | 56 | +| post-stop-sequence | (in progress) | — | — | — | — | — | + +Each step in the journey was a fix to a specific failure mode named by the prior step's bench. The methodology delivered: name the failure → fix in code → re-bench → confirm or surface the next failure. + **Headline:** the three modes occupy distinct points on a strict-vs-honest-vs-stable trade-off: - `quote` — highest strict-rate but rests on the older verifier path (substring quote-pair extraction). Includes false-STRICT cases the pointer-mode hardening discovered (e.g. claims that pass token-coincidence but cite the wrong source). Slowest of the three.