New docs/verifier-semantic-gap-design.md captures the deep
roadmap item from docs/qa-modes-bench-2026-04-30.md ("Verifier
semantic check (soft signal) — did the claim's predicate match
the cited span's frame?").
Problem: the claim-lattice verifier's lexical coverage check
(≥30% claim-token overlap with cited span) passes any (claim,
span) pair that shares enough surface tokens, even when the
span never asserts the claim's predicate. Three concrete cases
captured from live bench data:
- Great Wall elevation: STRICT 1/1 cited to a chunk that
discusses "Outer China beyond the Great Wall" but contains
no elevation information at all.
- JP-dinos Triceratops/Operation-Genesis: model cites the
2003 video-game article for a claim about the 1993 film.
- Boltzmann constant value: model cites a chunk that names
the constant and its unit but never states the numerical
value.
Three candidate designs evaluated against §1.1 cases + the
hard architectural constraints (proof-path purity, determinism,
no external endpoints beyond Hermes, latency budget):
3a NLI cross-encoder sidecar (recommended)
3b TF-IDF predicate matching (cheap but high false-negative)
3c Per-claim re-prompt to Hermes (latency + self-eval bias)
Recommendation: 3a with cross-encoder/nli-MiniLM2-L6-H768
(~80M params, ~100MB weights, CPU-runnable, deterministic at
fp32). Default off, opt-in via `claim_lattice_semantic_check`
policy field. Demote-only sidecar — moves STRICT → HYBRID
when claim entailment fails, never invents grounding. Soft
signal stays out of the proof path, mirrors the existing
`lazy_anchor_demoted` pattern.
Estimated impact: ~5-8 STRICT demotes per 66-run bench
(false-positives removed from the ledger), +0.4s typical
latency. Six open questions for fox at end of doc.
Doc-only commit. No code changes. Implementation deferred.
russell@unturf.com
49a85b97d6