bench/full-bench-qa: no regression after substrate+Phase 3 sprint (675 cells)

Full QA-quality sweep, 75 questions × 3 modes × n=3 = 675 cells
against Hermes-3 8B at concurrency=4. No mode regressed past
the 5pp signal floor:

  quote                 53.8% STRICT  vs 54% baseline = -0pp (stable)
  claim_lattice_pointer 23.1% STRICT  vs 20% baseline = +3pp (within floor)
  claim_lattice (JSON)  46.7% STRICT  vs 42% baseline = +5pp (at floor — marginal positive)

JSON-mode +5pp is right at the noise threshold per
docs/bench-maxing.md — could be the 2026-05-10 substrate work
(92/92 warrant chains + 18 textbook ingests + cascade tuning +
Phase 3 wiring) translating to retrieval-quality lift, OR
sample variance. Follow-up bench in 1-2 days disambiguates.

Format-collapse 0/225 in every mode. Latency stable
(10.6-11.1s median). Lattice-mode directive coverage 100%.

Phase 3 per-claim warrant-chain tail still dormant on this
fixture (rescue-only by design; no warrant-shape question
retrieves a chain-backed chunk while failing lexical anchor —
parallel shift's same finding). Source-level warrants tail
fires on claim-pack-targeted retrievals as documented in
phase3-live-validation-2026-05-10.md.

Substrate + Phase 3 sprint shipped clean.
This commit is contained in:
russell@unturf.com 2026-05-10 13:42:12 -04:00
parent a4058a43bc
commit e1ac7c2ea2
No known key found for this signature in database

View file

@ -0,0 +1,137 @@
# Full bench-qa — 2026-05-10T17-09-58Z
**Date:** 2026-05-10 (UTC)
**Author:** blackops + fox
**Method:** full QA-quality sweep, 75 questions × 3 modes × n=3
samples = 675 cells against Hermes-3 Llama-3.1-8B-FP8-Dynamic
(`https://hermes.ai.unturf.com/v1`), c=4 concurrency.
**Artifacts:** `bench/qa_results/2026-05-10T17-09-58Z.{jsonl,md}`
## Headline
**No regression.** All three modes within ±5pp of the prior
baseline (2026-05-02T15:07Z bench in `docs/qa-modes-bench.md`).
JSON-mode shows a marginal +5pp positive at the signal floor —
likely the substrate work + Phase 3 wiring + cascade tuning
+ alias mechanism landing today, but at the noise threshold.
| mode | n | STRICT | HYBRID | UNGROUNDED | mean ratio | mean latency | vs 2026-05-02 baseline |
|---|---|---|---|---|---|---|---|
| `quote` | 225 | **53.8%** | 22% | 24% | 0.697 | 10.8s | -0pp (stable) |
| `claim_lattice_pointer` | 225 | **23.1%** | 60% | 16% | 0.662 | 10.6s | +3pp (within signal floor) |
| `claim_lattice` (JSON) | 225 | **46.7%** | 36% | 17% | 0.753 | 11.1s | **+5pp (at signal floor — marginal positive)** |
## Discipline floor (per `docs/bench-maxing.md`)
- **Signal floor: 5pp.** Below 5pp is noise; treat ≥5pp as
observable signal. JSON-mode +5pp is exactly at the floor
— needs a follow-up bench to disambiguate from sample
variance. Quote and pointer movements are below the floor
(-0pp / +3pp).
- **No false-positive STRICT.** Format-collapse fired 0/225 in
every mode — the model never abandoned the protocol entirely.
This is consistent with the "honest verdicts beat optimistic
ones" discipline: an UNGROUNDED that came from "graceful
refusal" is recorded; an UNGROUNDED that came from "wrote
prose with zero pointer tags" would have flagged here.
- **Latency stable.** Median 10.6-11.1s across all modes; in
the same band as the 2026-05-02 baseline. Substrate work
didn't slow retrieval.
## Per-mode violation kinds (claim_lattice family only)
`quote` mode runs the legacy substring verifier so violation
kinds don't apply.
For `claim_lattice_pointer`:
| violation | count | rationale |
|---|---|---|
| CITATION_MISMATCH | 89 | model cited E# whose span doesn't textually match the claim |
| TITLE_MISMATCH | 62 | cited evidence's source title shares 0 stems with claim (Rule 8) |
| POINTER_OVERFLOW_TRIMMED | 46 | >2 pointers per claim → trimmed to 2 |
| LAZY_ANCHOR_DEMOTE | 30 | claim's anchor stems are <50% in cited span |
| NO_EVIDENCE_POINTER | 28 | claim line had no `[E#]` tag |
| WARRANT_MISSING | 19 | relation-shape claim's answer entity not in cited span — but Phase 3 chain-rescue handled where applicable |
| TOO_MANY_CLAIMS | 11 | hit the 12-claim ceiling |
| DEFLECTION_DETECTED | 10 | model shifted topic |
| SCHEMA_INVALID | 10 | parser rejected line as non-conforming |
| SUBJECT_TOKENS_ABSENT | 3 | claim doesn't include any token from the question |
For `claim_lattice` (JSON):
| violation | count |
|---|---|
| CITATION_MISMATCH | 74 |
| WARRANT_MISSING | 30 |
| TOO_MANY_EVIDENCE_IDS | 30 |
| TITLE_MISMATCH | 9 |
| MANUAL_QUOTE_VIOLATION | 8 |
| DEFLECTION_DETECTED | 8 |
| TOO_MANY_CLAIMS | 5 |
| SUBJECT_TOKENS_ABSENT | 3 |
| UNKNOWN_EVIDENCE_ID | 1 |
JSON mode emits 30 WARRANT_MISSING (vs pointer mode's 19) —
slight asymmetry; pointer mode's higher CITATION_MISMATCH
(89 vs 74) suggests the pointer-line format is more
forgiving on the cite-evidence-overlap check. JSON mode's
UNKNOWN_EVIDENCE_ID = 1 is the lenient pre-parser catching a
fabricated `evidence_id` once.
## Directive coverage (seven-point program)
| mode | D2 pointer | D3 cti-ready | D4 ev-map bound | D6 warrant | D7 honest label |
|---|---|---|---|---|---|
| `quote` | 0% | 0% | 100% | 0% | 100% |
| `claim_lattice_pointer` | 100% | 100% | 100% | 100% | 100% |
| `claim_lattice` (JSON) | 100% | 100% | 100% | 100% | 100% |
Lattice-mode directive coverage stays at 100% as expected.
quote-mode is the legacy verifier — D2/D3/D6 don't apply
(no pointer protocol, no warrant chain), and the test pinning
is global anyway.
## Phase 3 dormancy still confirmed
The `· warrant proven via chain ×N` per-claim tail (Phase 3
suppression of `WARRANT_MISSING` when chain-backed) doesn't
fire in this fixture's question set — same finding as the
parallel-shift's `phase3-warrant-chain-paraphrase-investigation-
2026-05-10.md`. Phase 3 is rescue-only by design; the rescue
data condition (warrant-shape claim citing a chain-backed
chunk where lexical anchor fails) doesn't intersect with this
fixture's general-knowledge questions.
The source-level `· warrants: N proven` tail (counts cited
sources with derivation rows) DOES fire on the bench's
claim-pack-targeted retrievals — see prior journal
`bench/results/phase3-live-validation-2026-05-10.md`.
## Cumulative day's bench journals
- `bench/results/full-warrant-resolution-2026-05-10.md` —
92/92 claim-pack warrant resolution
- `bench/results/phase3-live-validation-2026-05-10.md` —
6/7 probe questions reach EVIDENCE-WARRANTED
- `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md` —
parallel-shift A/B (Phase 3 rescue-only confirmed structurally)
- `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md` —
parallel-shift fixture-iteration deep-dive
- this file: `bench/results/full-bench-qa-2026-05-10.md` —
full 675-cell sweep, no regression vs 2026-05-02
## Verdict
**Ship.** No mode regressed past the 5pp signal floor. JSON
mode shows a +5pp marginal positive that's right at the floor
— a follow-up bench in 1-2 days will tell whether it's the
substrate work (recipe: every claim-pack record has a
Merkle-bound primary-source backing) translating to retrieval-
quality lift, or sample variance.
Today's substrate + Phase 3 work didn't break anything; it
either improved STRICT-rate marginally on the JSON path or
left it stable across all paths. Latency unchanged. Format-
collapse still 0%. Honest signal: solid plateau with a
maybe-positive trend on JSON.