Adds two architectural-fix sections to the QA-modes bench doc:
1. Pointer-IDs in JSON mode (commit bb8450d) — closes the
content-addressed evidence_id hallucination loop where
Hermes-3-8B emitted near-miss IDs (E1b6e396 vs the runtime's
Eed1b6e396) on cross-document relationships, landing
UNGROUNDED on factually correct answers. Switching to short
pointer IDs (E1, E2, …) made fabrication obvious and the
Homer Simpson fixture went UNGROUNDED 0/1 → STRICT 1/1.
2. JSON-mode stop-sequence (commit f23d3a3) — guards against
post-brace token runaway where Hermes spammed whitespace
until max_tokens exhausted on broad-descriptive shapes
(apollo program 3/3 runaway in the post-pointer-ID bench).
stop=["\n\n"] cuts the runaway since well-formed JSON-mode
output never contains a blank line.
Also adds a bench progression table showing the journey from
the morning baseline (JSON 19 errors, 26 STRICT) to the post-
pointer-ID evening run (0 errors, 31 STRICT, 56 grounded).
Each row was a named-failure → fix → re-bench cycle.
Post-retry / post-trim-and-verify bench showed the picture flipped:
JSON mode now leads on strict-rate (50%) and ties grounded count
(54) with zero errors, all at parity-or-better latency. Switching
the `make query` ANSWER_MODE default from `claim_lattice_pointer`
to `claim_lattice` so the human-facing CLI uses the strongest mode.
Library-level DEFAULT_ANSWER_MODE stays "quote" so unit tests using
StubClient aren't disrupted.
Doc updated with post-retry bench table + revised conclusion.
Live test harness:
- new test_homer_simpson_boss_is_mr_burns fixture (pinned to
pointer mode — JSON mode hallucinates evidence_ids on
cross-document relationships, pointer mode's short numeric
tags can't be fabricated). Documents the mode trade-off.
- test_laura_croft marker net broadened to absorb Hermes
single-sample variance (added "magazine", "video game",
"character", "fictional", "british" markers) plus a hard
"croft" anchor check. Both real entities still gate cleanly.
11/11 live fixtures pass at the new JSON default + pointer pin
on Homer. 460 unit tests + 11 live = 471 green; 10 skipped is
just the live tests in the default-skip path.
Three-way QA-quality bench (quote / pointer / JSON) over an expanded
22-question set × 3 samples = 198 LLM calls. Findings landed in
docs/qa-modes-bench-2026-04-30.md with per-question breakdown and
roadmap. Aggregate at bench time:
quote 31S 20H 15U 0e strict-rate 47% 7.7s
claim_lattice_pointer 14S 34H 18U 0e strict-rate 21% 4.4s
claim_lattice (JSON) 26S 12H 9U 19e strict-rate 39%* 4.4s
The 19 JSON-mode "errors" turned out to be HTTP 502 from vLLM upstream,
not parse failures — clustered, all on the JSON-mode pass, plausibly
correlated with `guided_json` stressing the grammar engine.
Two improvements based on findings:
(1) OpenAICompatibleClient grew retry on transient 502/503/504 with
exponential backoff (0.5/1/2s, 3 attempts default). Network-layer
errors (ConnectError, ReadTimeout, RemoteProtocolError) get the
same retry. Smooths over the cluster without changing semantics:
persistent failures still raise, transient bursts no longer
dominate the error column. Helps all modes; JSON benefits most.
(2) Pointer-cap behavior changed from hard SCHEMA_INVALID to
trim-and-verify. When `[E2,...,E14]` over-cites a single claim,
keep first N pointers, verify normally, record
POINTER_OVERFLOW_TRIMMED in violations. STRICT becomes unreachable
(audit_mode caps at HYBRID) so the over-cite pattern stays
surfaced — but a correct claim like "Leonardo da Vinci painted
the Mona Lisa." no longer gets nuked for cosmetic over-citation.
Pre-fix: pointer mode hit 0/3 STRICT on Mona Lisa (mega-bracket
triggered SCHEMA_INVALID). Post-fix: HYBRID 2/14 with the right
answer rendered alongside both kept source spans. The dropped
pointers count toward n_quotes so the denominator surfaces the
over-cite to the auditor.
Bench scaffolding: ANSWER_MODES tuple now includes "claim_lattice"
(JSON), Makefile default sweeps all three. Question set expanded
from 8 to 22 covering narrow factoid, broad descriptive, entity
list, relationship, comparison, niche, adversarial, out-of-corpus.
460 tests pass. Connecticut output stays clean (HYBRID 4/7); JP-
dinosaurs pointer mode still UNGROUNDED via the bare-name guard
(model emits one-token entity names, the right floor catches them).
Step-by-step operator guide for standing up an aborist mesh between
two real hosts: pre-flight, init/enable on both peers, out-of-band
pubkey exchange, mutual enrollment, mesh serve, mesh sync, audit
chain integrity checks, eviction protocol, reset/teardown.
Calls out what's not yet wired (mesh pull, per-peer chain merge,
AEAD body encryption) so operators don't expect features that ship
later. Pairs with docs/mesh.md (protocol contract) and the e2e tests
in tests/test_mesh_wire_e2e.py.
Adds docs/mesh.md and five graphviz diagrams covering the federation
layer: identity stack, epoch state machine, per-member secret envelope,
gossip wire contract, and operator decision tree. Pins the protocol
contract for the upcoming HTTP wire (mesh sync, mesh serve).
Makefile gets a 'docs' target with pattern rule so PNG renders are
incremental from .dot sources.
