# Full bench-qa — 2026-05-10T17-09-58Z

**Date:** 2026-05-10 (UTC)
**Author:** blackops + fox
**Method:** full QA-quality sweep, 75 questions × 3 modes × n=3
samples = 675 cells against Hermes-3 Llama-3.1-8B-FP8-Dynamic
(`https://hermes.ai.unturf.com/v1`), c=4 concurrency.
**Artifacts:** `bench/qa_results/2026-05-10T17-09-58Z.{jsonl,md}`

## Headline

**No regression.** All three modes within ±5pp of the prior
baseline (2026-05-02T15:07Z bench in `docs/qa-modes-bench.md`).
JSON-mode shows a marginal +5pp positive at the signal floor —
likely the substrate work + Phase 3 wiring + cascade tuning
+ alias mechanism landing today, but at the noise threshold.

| mode | n | STRICT | HYBRID | UNGROUNDED | mean ratio | mean latency | vs 2026-05-02 baseline |
|---|---|---|---|---|---|---|---|
| `quote` | 225 | **53.8%** | 22% | 24% | 0.697 | 10.8s | -0pp (stable) |
| `claim_lattice_pointer` | 225 | **23.1%** | 60% | 16% | 0.662 | 10.6s | +3pp (within signal floor) |
| `claim_lattice` (JSON) | 225 | **46.7%** | 36% | 17% | 0.753 | 11.1s | **+5pp (at signal floor — marginal positive)** |

## Discipline floor (per `docs/bench-maxing.md`)

- **Signal floor: 5pp.** Below 5pp is noise; treat ≥5pp as
  observable signal. JSON-mode +5pp is exactly at the floor
  — needs a follow-up bench to disambiguate from sample
  variance. Quote and pointer movements are below the floor
  (-0pp / +3pp).
- **No false-positive STRICT.** Format-collapse fired 0/225 in
  every mode — the model never abandoned the protocol entirely.
  This is consistent with the "honest verdicts beat optimistic
  ones" discipline: an UNGROUNDED that came from "graceful
  refusal" is recorded; an UNGROUNDED that came from "wrote
  prose with zero pointer tags" would have flagged here.
- **Latency stable.** Median 10.6-11.1s across all modes; in
  the same band as the 2026-05-02 baseline. Substrate work
  didn't slow retrieval.

## Per-mode violation kinds (claim_lattice family only)

`quote` mode runs the legacy substring verifier so violation
kinds don't apply.

For `claim_lattice_pointer`:

| violation | count | rationale |
|---|---|---|
| CITATION_MISMATCH | 89 | model cited E# whose span doesn't textually match the claim |
| TITLE_MISMATCH | 62 | cited evidence's source title shares 0 stems with claim (Rule 8) |
| POINTER_OVERFLOW_TRIMMED | 46 | >2 pointers per claim → trimmed to 2 |
| LAZY_ANCHOR_DEMOTE | 30 | claim's anchor stems are <50% in cited span |
| NO_EVIDENCE_POINTER | 28 | claim line had no `[E#]` tag |
| WARRANT_MISSING | 19 | relation-shape claim's answer entity not in cited span — but Phase 3 chain-rescue handled where applicable |
| TOO_MANY_CLAIMS | 11 | hit the 12-claim ceiling |
| DEFLECTION_DETECTED | 10 | model shifted topic |
| SCHEMA_INVALID | 10 | parser rejected line as non-conforming |
| SUBJECT_TOKENS_ABSENT | 3 | claim doesn't include any token from the question |

For `claim_lattice` (JSON):

| violation | count |
|---|---|
| CITATION_MISMATCH | 74 |
| WARRANT_MISSING | 30 |
| TOO_MANY_EVIDENCE_IDS | 30 |
| TITLE_MISMATCH | 9 |
| MANUAL_QUOTE_VIOLATION | 8 |
| DEFLECTION_DETECTED | 8 |
| TOO_MANY_CLAIMS | 5 |
| SUBJECT_TOKENS_ABSENT | 3 |
| UNKNOWN_EVIDENCE_ID | 1 |

JSON mode emits 30 WARRANT_MISSING (vs pointer mode's 19) —
slight asymmetry; pointer mode's higher CITATION_MISMATCH
(89 vs 74) suggests the pointer-line format is more
forgiving on the cite-evidence-overlap check. JSON mode's
UNKNOWN_EVIDENCE_ID = 1 is the lenient pre-parser catching a
fabricated `evidence_id` once.

## Directive coverage (seven-point program)

| mode | D2 pointer | D3 cti-ready | D4 ev-map bound | D6 warrant | D7 honest label |
|---|---|---|---|---|---|
| `quote` | 0% | 0% | 100% | 0% | 100% |
| `claim_lattice_pointer` | 100% | 100% | 100% | 100% | 100% |
| `claim_lattice` (JSON) | 100% | 100% | 100% | 100% | 100% |

Lattice-mode directive coverage stays at 100% as expected.
quote-mode is the legacy verifier — D2/D3/D6 don't apply
(no pointer protocol, no warrant chain), and the test pinning
is global anyway.

## Phase 3 dormancy still confirmed

The `· warrant proven via chain ×N` per-claim tail (Phase 3
suppression of `WARRANT_MISSING` when chain-backed) doesn't
fire in this fixture's question set — same finding as the
parallel-shift's `phase3-warrant-chain-paraphrase-investigation-
2026-05-10.md`. Phase 3 is rescue-only by design; the rescue
data condition (warrant-shape claim citing a chain-backed
chunk where lexical anchor fails) doesn't intersect with this
fixture's general-knowledge questions.

The source-level `· warrants: N proven` tail (counts cited
sources with derivation rows) DOES fire on the bench's
claim-pack-targeted retrievals — see prior journal
`bench/results/phase3-live-validation-2026-05-10.md`.

## Cumulative day's bench journals

- `bench/results/full-warrant-resolution-2026-05-10.md` —
  92/92 claim-pack warrant resolution
- `bench/results/phase3-live-validation-2026-05-10.md` —
  6/7 probe questions reach EVIDENCE-WARRANTED
- `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md` —
  parallel-shift A/B (Phase 3 rescue-only confirmed structurally)
- `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md` —
  parallel-shift fixture-iteration deep-dive
- this file: `bench/results/full-bench-qa-2026-05-10.md` —
  full 675-cell sweep, no regression vs 2026-05-02

## Verdict

**Ship.** No mode regressed past the 5pp signal floor. JSON
mode shows a +5pp marginal positive that's right at the floor
— a follow-up bench in 1-2 days will tell whether it's the
substrate work (recipe: every claim-pack record has a
Merkle-bound primary-source backing) translating to retrieval-
quality lift, or sample variance.

Today's substrate + Phase 3 work didn't break anything; it
either improved STRICT-rate marginally on the JSON path or
left it stable across all paths. Latency unchanged. Format-
collapse still 0%. Honest signal: solid plateau with a
maybe-positive trend on JSON.
