docs(qa): pointer-ID switch + stop-sequence journey, bench progression
Adds two architectural-fix sections to the QA-modes bench doc: 1. Pointer-IDs in JSON mode (commitbb8450d) — closes the content-addressed evidence_id hallucination loop where Hermes-3-8B emitted near-miss IDs (E1b6e396 vs the runtime's Eed1b6e396) on cross-document relationships, landing UNGROUNDED on factually correct answers. Switching to short pointer IDs (E1, E2, …) made fabrication obvious and the Homer Simpson fixture went UNGROUNDED 0/1 → STRICT 1/1. 2. JSON-mode stop-sequence (commitf23d3a3) — guards against post-brace token runaway where Hermes spammed whitespace until max_tokens exhausted on broad-descriptive shapes (apollo program 3/3 runaway in the post-pointer-ID bench). stop=["\n\n"] cuts the runaway since well-formed JSON-mode output never contains a blank line. Also adds a bench progression table showing the journey from the morning baseline (JSON 19 errors, 26 STRICT) to the post- pointer-ID evening run (0 errors, 31 STRICT, 56 grounded). Each row was a named-failure → fix → re-bench cycle.
This commit is contained in:
parent
f23d3a3067
commit
4aaa253dfd
1 changed files with 38 additions and 0 deletions
|
|
@ -38,6 +38,44 @@ After landing two improvements derived from the analysis below — HTTP retry on
|
|||
|
||||
The `make query` target's default `ANSWER_MODE` was flipped from `claim_lattice_pointer` to `claim_lattice` on 2026-04-30 to reflect this. The library-level `DEFAULT_ANSWER_MODE` stays `"quote"` so unit tests using `StubClient` aren't disrupted; pointer mode is still available via `ANSWER_MODE=claim_lattice_pointer`.
|
||||
|
||||
### Architectural fix — JSON mode uses pointer IDs (E1, E2, …) instead of content-addressed evidence_ids
|
||||
|
||||
A separate failure mode surfaced after the retry/trim work: cross-document relationship questions consistently landed `UNGROUNDED 0/1` in JSON mode despite the model writing the correct answer text. Diagnosis of `who is homer simpson's boss?` in JSON mode showed:
|
||||
|
||||
```
|
||||
{"claims":[{"text":"Homer Simpson's boss is Mr. Burns.","evidence_ids":["E1b6e396"]}]}
|
||||
```
|
||||
|
||||
The runtime had `Eed1b6e396` for that chunk; Hermes-3-8B emitted `E1b6e396` — a plausible-looking near-miss the verifier rightly flagged as `UNKNOWN_EVIDENCE_ID`. The model was *fabricating* content-addressed evidence_ids when the real ones felt awkwardly long.
|
||||
|
||||
Fix landed in commit `bb8450d`: the JSON-mode prompt-facing surface switched from content-addressed evidence_ids (`Eed1b6e396`) to pointer IDs (`E1`, `E2`, …) — same as `claim_lattice_pointer` mode. The runtime still resolves each pointer_id to its content-addressed evidence_id internally and stores **that** in `evidence_id_pairs` for cache & run-DAG continuity. Only the prompt-facing string changes. After the fix, `who is homer simpson's boss?` lands `STRICT 1/1` in JSON mode.
|
||||
|
||||
Why pointer IDs work where content-addressed didn't:
|
||||
- short, enumerable, fabrication-obvious — if only `E1`-`E10` are shown, an emitted `E27` reads as a schema violation at a glance
|
||||
- distribution-natural for small models (citation-style is heavily represented in training)
|
||||
- the proof path stays content-addressed (the verifier's audit chain still hashes content-addressed ids), so the human/model surface change doesn't weaken the v9.8 admissibility ledger
|
||||
|
||||
### Token-runaway guard — JSON-mode stop-sequence
|
||||
|
||||
Post-pointer-ID-switch bench (2026-04-30T19-55-11Z) found a residual JSON-mode failure on broad-descriptive questions: ~4 of 66 runs landed `UNGROUNDED 0/0` at 12-15s instead of the normal 2-5s. Inspection: Hermes emitted a valid claim object then kept generating whitespace / blank lines until `max_tokens=512` exhausted. The truncated payload didn't parse and the lenient pre-parser returned no claims.
|
||||
|
||||
Concrete instances:
|
||||
- `tell me about the apollo program` — 3/3 samples runaway
|
||||
- `tell me about the python programming language` — 1/3 runaway
|
||||
|
||||
Fix landed in commit `f23d3a3`: pass `stop=["\n\n"]` to vLLM in JSON mode. Well-formed JSON-mode output never contains a blank line — the model emits one object on a single line (or with simple internal newlines), never `\n\n`. The stop sequence is the runaway signature itself; legitimate output is never truncated. Folds into `governance_policy_hash` via `claim_lattice_json_stop_sequences` policy field so changing the list invalidates prior cached records.
|
||||
|
||||
### Bench progression summary
|
||||
|
||||
| run | bench | quote STRICT | pointer STRICT | JSON STRICT | JSON err | JSON grounded |
|
||||
|-----|-------|--------------|----------------|-------------|----------|----------------|
|
||||
| baseline (no improvements) | morning n=3 | 31 | 14 | 26 | **19** | 38 |
|
||||
| post-retry + trim-and-verify | midday n=3 | 31 | 16 | **33** | 0 | 54 |
|
||||
| post-pointer-ID switch | evening n=3 | 27 | 15 | 31 | 0 | 56 |
|
||||
| post-stop-sequence | (in progress) | — | — | — | — | — |
|
||||
|
||||
Each step in the journey was a fix to a specific failure mode named by the prior step's bench. The methodology delivered: name the failure → fix in code → re-bench → confirm or surface the next failure.
|
||||
|
||||
**Headline:** the three modes occupy distinct points on a strict-vs-honest-vs-stable trade-off:
|
||||
|
||||
- `quote` — highest strict-rate but rests on the older verifier path (substring quote-pair extraction). Includes false-STRICT cases the pointer-mode hardening discovered (e.g. claims that pass token-coincidence but cite the wrong source). Slowest of the three.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue