bench/full-bench-qa: no regression after substrate+Phase 3 sprint (675 cells)
Full QA-quality sweep, 75 questions × 3 modes × n=3 = 675 cells against Hermes-3 8B at concurrency=4. No mode regressed past the 5pp signal floor: quote 53.8% STRICT vs 54% baseline = -0pp (stable) claim_lattice_pointer 23.1% STRICT vs 20% baseline = +3pp (within floor) claim_lattice (JSON) 46.7% STRICT vs 42% baseline = +5pp (at floor — marginal positive) JSON-mode +5pp is right at the noise threshold per docs/bench-maxing.md — could be the 2026-05-10 substrate work (92/92 warrant chains + 18 textbook ingests + cascade tuning + Phase 3 wiring) translating to retrieval-quality lift, OR sample variance. Follow-up bench in 1-2 days disambiguates. Format-collapse 0/225 in every mode. Latency stable (10.6-11.1s median). Lattice-mode directive coverage 100%. Phase 3 per-claim warrant-chain tail still dormant on this fixture (rescue-only by design; no warrant-shape question retrieves a chain-backed chunk while failing lexical anchor — parallel shift's same finding). Source-level warrants tail fires on claim-pack-targeted retrievals as documented in phase3-live-validation-2026-05-10.md. Substrate + Phase 3 sprint shipped clean.
This commit is contained in:
parent
a4058a43bc
commit
e1ac7c2ea2
1 changed files with 137 additions and 0 deletions
137
bench/results/full-bench-qa-2026-05-10.md
Normal file
137
bench/results/full-bench-qa-2026-05-10.md
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
# Full bench-qa — 2026-05-10T17-09-58Z
|
||||
|
||||
**Date:** 2026-05-10 (UTC)
|
||||
**Author:** blackops + fox
|
||||
**Method:** full QA-quality sweep, 75 questions × 3 modes × n=3
|
||||
samples = 675 cells against Hermes-3 Llama-3.1-8B-FP8-Dynamic
|
||||
(`https://hermes.ai.unturf.com/v1`), c=4 concurrency.
|
||||
**Artifacts:** `bench/qa_results/2026-05-10T17-09-58Z.{jsonl,md}`
|
||||
|
||||
## Headline
|
||||
|
||||
**No regression.** All three modes within ±5pp of the prior
|
||||
baseline (2026-05-02T15:07Z bench in `docs/qa-modes-bench.md`).
|
||||
JSON-mode shows a marginal +5pp positive at the signal floor —
|
||||
likely the substrate work + Phase 3 wiring + cascade tuning
|
||||
+ alias mechanism landing today, but at the noise threshold.
|
||||
|
||||
| mode | n | STRICT | HYBRID | UNGROUNDED | mean ratio | mean latency | vs 2026-05-02 baseline |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| `quote` | 225 | **53.8%** | 22% | 24% | 0.697 | 10.8s | -0pp (stable) |
|
||||
| `claim_lattice_pointer` | 225 | **23.1%** | 60% | 16% | 0.662 | 10.6s | +3pp (within signal floor) |
|
||||
| `claim_lattice` (JSON) | 225 | **46.7%** | 36% | 17% | 0.753 | 11.1s | **+5pp (at signal floor — marginal positive)** |
|
||||
|
||||
## Discipline floor (per `docs/bench-maxing.md`)
|
||||
|
||||
- **Signal floor: 5pp.** Below 5pp is noise; treat ≥5pp as
|
||||
observable signal. JSON-mode +5pp is exactly at the floor
|
||||
— needs a follow-up bench to disambiguate from sample
|
||||
variance. Quote and pointer movements are below the floor
|
||||
(-0pp / +3pp).
|
||||
- **No false-positive STRICT.** Format-collapse fired 0/225 in
|
||||
every mode — the model never abandoned the protocol entirely.
|
||||
This is consistent with the "honest verdicts beat optimistic
|
||||
ones" discipline: an UNGROUNDED that came from "graceful
|
||||
refusal" is recorded; an UNGROUNDED that came from "wrote
|
||||
prose with zero pointer tags" would have flagged here.
|
||||
- **Latency stable.** Median 10.6-11.1s across all modes; in
|
||||
the same band as the 2026-05-02 baseline. Substrate work
|
||||
didn't slow retrieval.
|
||||
|
||||
## Per-mode violation kinds (claim_lattice family only)
|
||||
|
||||
`quote` mode runs the legacy substring verifier so violation
|
||||
kinds don't apply.
|
||||
|
||||
For `claim_lattice_pointer`:
|
||||
|
||||
| violation | count | rationale |
|
||||
|---|---|---|
|
||||
| CITATION_MISMATCH | 89 | model cited E# whose span doesn't textually match the claim |
|
||||
| TITLE_MISMATCH | 62 | cited evidence's source title shares 0 stems with claim (Rule 8) |
|
||||
| POINTER_OVERFLOW_TRIMMED | 46 | >2 pointers per claim → trimmed to 2 |
|
||||
| LAZY_ANCHOR_DEMOTE | 30 | claim's anchor stems are <50% in cited span |
|
||||
| NO_EVIDENCE_POINTER | 28 | claim line had no `[E#]` tag |
|
||||
| WARRANT_MISSING | 19 | relation-shape claim's answer entity not in cited span — but Phase 3 chain-rescue handled where applicable |
|
||||
| TOO_MANY_CLAIMS | 11 | hit the 12-claim ceiling |
|
||||
| DEFLECTION_DETECTED | 10 | model shifted topic |
|
||||
| SCHEMA_INVALID | 10 | parser rejected line as non-conforming |
|
||||
| SUBJECT_TOKENS_ABSENT | 3 | claim doesn't include any token from the question |
|
||||
|
||||
For `claim_lattice` (JSON):
|
||||
|
||||
| violation | count |
|
||||
|---|---|
|
||||
| CITATION_MISMATCH | 74 |
|
||||
| WARRANT_MISSING | 30 |
|
||||
| TOO_MANY_EVIDENCE_IDS | 30 |
|
||||
| TITLE_MISMATCH | 9 |
|
||||
| MANUAL_QUOTE_VIOLATION | 8 |
|
||||
| DEFLECTION_DETECTED | 8 |
|
||||
| TOO_MANY_CLAIMS | 5 |
|
||||
| SUBJECT_TOKENS_ABSENT | 3 |
|
||||
| UNKNOWN_EVIDENCE_ID | 1 |
|
||||
|
||||
JSON mode emits 30 WARRANT_MISSING (vs pointer mode's 19) —
|
||||
slight asymmetry; pointer mode's higher CITATION_MISMATCH
|
||||
(89 vs 74) suggests the pointer-line format is more
|
||||
forgiving on the cite-evidence-overlap check. JSON mode's
|
||||
UNKNOWN_EVIDENCE_ID = 1 is the lenient pre-parser catching a
|
||||
fabricated `evidence_id` once.
|
||||
|
||||
## Directive coverage (seven-point program)
|
||||
|
||||
| mode | D2 pointer | D3 cti-ready | D4 ev-map bound | D6 warrant | D7 honest label |
|
||||
|---|---|---|---|---|---|
|
||||
| `quote` | 0% | 0% | 100% | 0% | 100% |
|
||||
| `claim_lattice_pointer` | 100% | 100% | 100% | 100% | 100% |
|
||||
| `claim_lattice` (JSON) | 100% | 100% | 100% | 100% | 100% |
|
||||
|
||||
Lattice-mode directive coverage stays at 100% as expected.
|
||||
quote-mode is the legacy verifier — D2/D3/D6 don't apply
|
||||
(no pointer protocol, no warrant chain), and the test pinning
|
||||
is global anyway.
|
||||
|
||||
## Phase 3 dormancy still confirmed
|
||||
|
||||
The `· warrant proven via chain ×N` per-claim tail (Phase 3
|
||||
suppression of `WARRANT_MISSING` when chain-backed) doesn't
|
||||
fire in this fixture's question set — same finding as the
|
||||
parallel-shift's `phase3-warrant-chain-paraphrase-investigation-
|
||||
2026-05-10.md`. Phase 3 is rescue-only by design; the rescue
|
||||
data condition (warrant-shape claim citing a chain-backed
|
||||
chunk where lexical anchor fails) doesn't intersect with this
|
||||
fixture's general-knowledge questions.
|
||||
|
||||
The source-level `· warrants: N proven` tail (counts cited
|
||||
sources with derivation rows) DOES fire on the bench's
|
||||
claim-pack-targeted retrievals — see prior journal
|
||||
`bench/results/phase3-live-validation-2026-05-10.md`.
|
||||
|
||||
## Cumulative day's bench journals
|
||||
|
||||
- `bench/results/full-warrant-resolution-2026-05-10.md` —
|
||||
92/92 claim-pack warrant resolution
|
||||
- `bench/results/phase3-live-validation-2026-05-10.md` —
|
||||
6/7 probe questions reach EVIDENCE-WARRANTED
|
||||
- `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md` —
|
||||
parallel-shift A/B (Phase 3 rescue-only confirmed structurally)
|
||||
- `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md` —
|
||||
parallel-shift fixture-iteration deep-dive
|
||||
- this file: `bench/results/full-bench-qa-2026-05-10.md` —
|
||||
full 675-cell sweep, no regression vs 2026-05-02
|
||||
|
||||
## Verdict
|
||||
|
||||
**Ship.** No mode regressed past the 5pp signal floor. JSON
|
||||
mode shows a +5pp marginal positive that's right at the floor
|
||||
— a follow-up bench in 1-2 days will tell whether it's the
|
||||
substrate work (recipe: every claim-pack record has a
|
||||
Merkle-bound primary-source backing) translating to retrieval-
|
||||
quality lift, or sample variance.
|
||||
|
||||
Today's substrate + Phase 3 work didn't break anything; it
|
||||
either improved STRICT-rate marginally on the JSON path or
|
||||
left it stable across all paths. Latency unchanged. Format-
|
||||
collapse still 0%. Honest signal: solid plateau with a
|
||||
maybe-positive trend on JSON.
|
||||
Loading…
Add table
Add a link
Reference in a new issue