diff --git a/bench/results/full-bench-qa-2026-05-10.md b/bench/results/full-bench-qa-2026-05-10.md new file mode 100644 index 0000000..d0fbef9 --- /dev/null +++ b/bench/results/full-bench-qa-2026-05-10.md @@ -0,0 +1,137 @@ +# Full bench-qa — 2026-05-10T17-09-58Z + +**Date:** 2026-05-10 (UTC) +**Author:** blackops + fox +**Method:** full QA-quality sweep, 75 questions × 3 modes × n=3 +samples = 675 cells against Hermes-3 Llama-3.1-8B-FP8-Dynamic +(`https://hermes.ai.unturf.com/v1`), c=4 concurrency. +**Artifacts:** `bench/qa_results/2026-05-10T17-09-58Z.{jsonl,md}` + +## Headline + +**No regression.** All three modes within ±5pp of the prior +baseline (2026-05-02T15:07Z bench in `docs/qa-modes-bench.md`). +JSON-mode shows a marginal +5pp positive at the signal floor — +likely the substrate work + Phase 3 wiring + cascade tuning ++ alias mechanism landing today, but at the noise threshold. + +| mode | n | STRICT | HYBRID | UNGROUNDED | mean ratio | mean latency | vs 2026-05-02 baseline | +|---|---|---|---|---|---|---|---| +| `quote` | 225 | **53.8%** | 22% | 24% | 0.697 | 10.8s | -0pp (stable) | +| `claim_lattice_pointer` | 225 | **23.1%** | 60% | 16% | 0.662 | 10.6s | +3pp (within signal floor) | +| `claim_lattice` (JSON) | 225 | **46.7%** | 36% | 17% | 0.753 | 11.1s | **+5pp (at signal floor — marginal positive)** | + +## Discipline floor (per `docs/bench-maxing.md`) + +- **Signal floor: 5pp.** Below 5pp is noise; treat ≥5pp as + observable signal. JSON-mode +5pp is exactly at the floor + — needs a follow-up bench to disambiguate from sample + variance. Quote and pointer movements are below the floor + (-0pp / +3pp). +- **No false-positive STRICT.** Format-collapse fired 0/225 in + every mode — the model never abandoned the protocol entirely. + This is consistent with the "honest verdicts beat optimistic + ones" discipline: an UNGROUNDED that came from "graceful + refusal" is recorded; an UNGROUNDED that came from "wrote + prose with zero pointer tags" would have flagged here. +- **Latency stable.** Median 10.6-11.1s across all modes; in + the same band as the 2026-05-02 baseline. Substrate work + didn't slow retrieval. + +## Per-mode violation kinds (claim_lattice family only) + +`quote` mode runs the legacy substring verifier so violation +kinds don't apply. + +For `claim_lattice_pointer`: + +| violation | count | rationale | +|---|---|---| +| CITATION_MISMATCH | 89 | model cited E# whose span doesn't textually match the claim | +| TITLE_MISMATCH | 62 | cited evidence's source title shares 0 stems with claim (Rule 8) | +| POINTER_OVERFLOW_TRIMMED | 46 | >2 pointers per claim → trimmed to 2 | +| LAZY_ANCHOR_DEMOTE | 30 | claim's anchor stems are <50% in cited span | +| NO_EVIDENCE_POINTER | 28 | claim line had no `[E#]` tag | +| WARRANT_MISSING | 19 | relation-shape claim's answer entity not in cited span — but Phase 3 chain-rescue handled where applicable | +| TOO_MANY_CLAIMS | 11 | hit the 12-claim ceiling | +| DEFLECTION_DETECTED | 10 | model shifted topic | +| SCHEMA_INVALID | 10 | parser rejected line as non-conforming | +| SUBJECT_TOKENS_ABSENT | 3 | claim doesn't include any token from the question | + +For `claim_lattice` (JSON): + +| violation | count | +|---|---| +| CITATION_MISMATCH | 74 | +| WARRANT_MISSING | 30 | +| TOO_MANY_EVIDENCE_IDS | 30 | +| TITLE_MISMATCH | 9 | +| MANUAL_QUOTE_VIOLATION | 8 | +| DEFLECTION_DETECTED | 8 | +| TOO_MANY_CLAIMS | 5 | +| SUBJECT_TOKENS_ABSENT | 3 | +| UNKNOWN_EVIDENCE_ID | 1 | + +JSON mode emits 30 WARRANT_MISSING (vs pointer mode's 19) — +slight asymmetry; pointer mode's higher CITATION_MISMATCH +(89 vs 74) suggests the pointer-line format is more +forgiving on the cite-evidence-overlap check. JSON mode's +UNKNOWN_EVIDENCE_ID = 1 is the lenient pre-parser catching a +fabricated `evidence_id` once. + +## Directive coverage (seven-point program) + +| mode | D2 pointer | D3 cti-ready | D4 ev-map bound | D6 warrant | D7 honest label | +|---|---|---|---|---|---| +| `quote` | 0% | 0% | 100% | 0% | 100% | +| `claim_lattice_pointer` | 100% | 100% | 100% | 100% | 100% | +| `claim_lattice` (JSON) | 100% | 100% | 100% | 100% | 100% | + +Lattice-mode directive coverage stays at 100% as expected. +quote-mode is the legacy verifier — D2/D3/D6 don't apply +(no pointer protocol, no warrant chain), and the test pinning +is global anyway. + +## Phase 3 dormancy still confirmed + +The `· warrant proven via chain ×N` per-claim tail (Phase 3 +suppression of `WARRANT_MISSING` when chain-backed) doesn't +fire in this fixture's question set — same finding as the +parallel-shift's `phase3-warrant-chain-paraphrase-investigation- +2026-05-10.md`. Phase 3 is rescue-only by design; the rescue +data condition (warrant-shape claim citing a chain-backed +chunk where lexical anchor fails) doesn't intersect with this +fixture's general-knowledge questions. + +The source-level `· warrants: N proven` tail (counts cited +sources with derivation rows) DOES fire on the bench's +claim-pack-targeted retrievals — see prior journal +`bench/results/phase3-live-validation-2026-05-10.md`. + +## Cumulative day's bench journals + +- `bench/results/full-warrant-resolution-2026-05-10.md` — + 92/92 claim-pack warrant resolution +- `bench/results/phase3-live-validation-2026-05-10.md` — + 6/7 probe questions reach EVIDENCE-WARRANTED +- `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md` — + parallel-shift A/B (Phase 3 rescue-only confirmed structurally) +- `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md` — + parallel-shift fixture-iteration deep-dive +- this file: `bench/results/full-bench-qa-2026-05-10.md` — + full 675-cell sweep, no regression vs 2026-05-02 + +## Verdict + +**Ship.** No mode regressed past the 5pp signal floor. JSON +mode shows a +5pp marginal positive that's right at the floor +— a follow-up bench in 1-2 days will tell whether it's the +substrate work (recipe: every claim-pack record has a +Merkle-bound primary-source backing) translating to retrieval- +quality lift, or sample variance. + +Today's substrate + Phase 3 work didn't break anything; it +either improved STRICT-rate marginally on the JSON path or +left it stable across all paths. Latency unchanged. Format- +collapse still 0%. Honest signal: solid plateau with a +maybe-positive trend on JSON.