From 4aaa253dfd58ab9bb80d8bee02895b4c48a2b2f2 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Thu, 30 Apr 2026 16:21:16 -0400 Subject: [PATCH] docs(qa): pointer-ID switch + stop-sequence journey, bench progression MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds two architectural-fix sections to the QA-modes bench doc: 1. Pointer-IDs in JSON mode (commit bb8450d) — closes the content-addressed evidence_id hallucination loop where Hermes-3-8B emitted near-miss IDs (E1b6e396 vs the runtime's Eed1b6e396) on cross-document relationships, landing UNGROUNDED on factually correct answers. Switching to short pointer IDs (E1, E2, …) made fabrication obvious and the Homer Simpson fixture went UNGROUNDED 0/1 → STRICT 1/1. 2. JSON-mode stop-sequence (commit f23d3a3) — guards against post-brace token runaway where Hermes spammed whitespace until max_tokens exhausted on broad-descriptive shapes (apollo program 3/3 runaway in the post-pointer-ID bench). stop=["\n\n"] cuts the runaway since well-formed JSON-mode output never contains a blank line. Also adds a bench progression table showing the journey from the morning baseline (JSON 19 errors, 26 STRICT) to the post- pointer-ID evening run (0 errors, 31 STRICT, 56 grounded). Each row was a named-failure → fix → re-bench cycle. --- docs/qa-modes-bench-2026-04-30.md | 38 +++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/docs/qa-modes-bench-2026-04-30.md b/docs/qa-modes-bench-2026-04-30.md index ad85e0c..8e41304 100644 --- a/docs/qa-modes-bench-2026-04-30.md +++ b/docs/qa-modes-bench-2026-04-30.md @@ -38,6 +38,44 @@ After landing two improvements derived from the analysis below — HTTP retry on The `make query` target's default `ANSWER_MODE` was flipped from `claim_lattice_pointer` to `claim_lattice` on 2026-04-30 to reflect this. The library-level `DEFAULT_ANSWER_MODE` stays `"quote"` so unit tests using `StubClient` aren't disrupted; pointer mode is still available via `ANSWER_MODE=claim_lattice_pointer`. +### Architectural fix — JSON mode uses pointer IDs (E1, E2, …) instead of content-addressed evidence_ids + +A separate failure mode surfaced after the retry/trim work: cross-document relationship questions consistently landed `UNGROUNDED 0/1` in JSON mode despite the model writing the correct answer text. Diagnosis of `who is homer simpson's boss?` in JSON mode showed: + +``` +{"claims":[{"text":"Homer Simpson's boss is Mr. Burns.","evidence_ids":["E1b6e396"]}]} +``` + +The runtime had `Eed1b6e396` for that chunk; Hermes-3-8B emitted `E1b6e396` — a plausible-looking near-miss the verifier rightly flagged as `UNKNOWN_EVIDENCE_ID`. The model was *fabricating* content-addressed evidence_ids when the real ones felt awkwardly long. + +Fix landed in commit `bb8450d`: the JSON-mode prompt-facing surface switched from content-addressed evidence_ids (`Eed1b6e396`) to pointer IDs (`E1`, `E2`, …) — same as `claim_lattice_pointer` mode. The runtime still resolves each pointer_id to its content-addressed evidence_id internally and stores **that** in `evidence_id_pairs` for cache & run-DAG continuity. Only the prompt-facing string changes. After the fix, `who is homer simpson's boss?` lands `STRICT 1/1` in JSON mode. + +Why pointer IDs work where content-addressed didn't: +- short, enumerable, fabrication-obvious — if only `E1`-`E10` are shown, an emitted `E27` reads as a schema violation at a glance +- distribution-natural for small models (citation-style is heavily represented in training) +- the proof path stays content-addressed (the verifier's audit chain still hashes content-addressed ids), so the human/model surface change doesn't weaken the v9.8 admissibility ledger + +### Token-runaway guard — JSON-mode stop-sequence + +Post-pointer-ID-switch bench (2026-04-30T19-55-11Z) found a residual JSON-mode failure on broad-descriptive questions: ~4 of 66 runs landed `UNGROUNDED 0/0` at 12-15s instead of the normal 2-5s. Inspection: Hermes emitted a valid claim object then kept generating whitespace / blank lines until `max_tokens=512` exhausted. The truncated payload didn't parse and the lenient pre-parser returned no claims. + +Concrete instances: +- `tell me about the apollo program` — 3/3 samples runaway +- `tell me about the python programming language` — 1/3 runaway + +Fix landed in commit `f23d3a3`: pass `stop=["\n\n"]` to vLLM in JSON mode. Well-formed JSON-mode output never contains a blank line — the model emits one object on a single line (or with simple internal newlines), never `\n\n`. The stop sequence is the runaway signature itself; legitimate output is never truncated. Folds into `governance_policy_hash` via `claim_lattice_json_stop_sequences` policy field so changing the list invalidates prior cached records. + +### Bench progression summary + +| run | bench | quote STRICT | pointer STRICT | JSON STRICT | JSON err | JSON grounded | +|-----|-------|--------------|----------------|-------------|----------|----------------| +| baseline (no improvements) | morning n=3 | 31 | 14 | 26 | **19** | 38 | +| post-retry + trim-and-verify | midday n=3 | 31 | 16 | **33** | 0 | 54 | +| post-pointer-ID switch | evening n=3 | 27 | 15 | 31 | 0 | 56 | +| post-stop-sequence | (in progress) | — | — | — | — | — | + +Each step in the journey was a fix to a specific failure mode named by the prior step's bench. The methodology delivered: name the failure → fix in code → re-bench → confirm or surface the next failure. + **Headline:** the three modes occupy distinct points on a strict-vs-honest-vs-stable trade-off: - `quote` — highest strict-rate but rests on the older verifier path (substring quote-pair extraction). Includes false-STRICT cases the pointer-mode hardening discovered (e.g. claims that pass token-coincidence but cite the wrong source). Slowest of the three.