From 3bd36f36c85503a1a865f6ff8e22385b73bf139b Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 3 May 2026 21:07:56 -0400 Subject: [PATCH] =?UTF-8?q?bench:=20full=2075-question=20regression=20chec?= =?UTF-8?q?k=20=E2=80=94=20no=20regression=20from=20#000010?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addendum 3 to qa-modes-bench.md. Validates that flipping quantifier_reminder_enabled=True for lattice modes (per #000010 §12.10 / §13.1) doesn't regress narrow-question performance. Prior validation (Addendum 2) covered the 9-question broad subset only. This run sweeps the full 75-question bench/qa_questions.txt (~10% broad, ~89% narrow), 225 runs per mode, comparing against the frozen 2026-05-02T15-07Z baseline. Findings: Mode | Pre-flip SR | Post-flip SR | Δ ----------+-------------+--------------+-------- quote | 0.54 | 0.52 | -2pp (within 5pp floor) pointer | 0.20 | 0.21 | +1pp JSON | 0.42 | 0.44 | +2pp Mean ratio: -1/+2/+3pp — all within noise band. Substrate-level wins beyond the headline metrics: - pointer FORMAT_COLLAPSED: 0/225 across the full sweep. Reminder eliminates collapse mode globally, not just on broad. - pointer NO_EVIDENCE_POINTER: 13% (vs 33% on broad-only when reminder was off). Citation discipline propagates beyond the rows where the reminder text actually fires — the model's session attention reinforces. - JSON mean ratio +3pp consistent with broad-subset finding. Quote-mode is essentially unchanged because it's mode-gated off the guard by default. Latency 33-35s/call this evening (vs 17-19s in prior runs) is endpoint contention, not a substrate regression — preflight adds zero LLM calls. Verdict: no regression. The default flip ships clean across the full corpus. Defaults stay on. The substrate is strictly more honest (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without sacrificing throughput on non-broad questions. Bench artifact: bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}. --- docs/qa-modes-bench.md | 59 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/docs/qa-modes-bench.md b/docs/qa-modes-bench.md index a3b5233..fd5d0ad 100644 --- a/docs/qa-modes-bench.md +++ b/docs/qa-modes-bench.md @@ -253,3 +253,62 @@ improve by ≥5pp on lattice modes; STRICT-rate is within noise. Defaults stay on. Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`. + +### Addendum 3 — full-bench regression check (2026-05-03T23-30-12Z) + +Validates that the #000010 default flip (reminder default-on for +lattice modes) doesn't regress narrow-question performance. The +prior validations in Addendum 1 + Addendum 2 used the 9-question +broad subset only; this run sweeps the full 75-question +`bench/qa_questions.txt` (~10% broad, ~89% narrow factoid / +descriptive / list shapes). + +Same harness, n=3 × 3 modes × 75 questions = 225 runs per mode. +Compared against the frozen `15:07Z` baseline (the authoritative +pre-#000008/9/10 state of the substrate, 71 questions × n=3 = 213 +runs per mode): + +| Mode | Pre-flip (15:07Z) | **Post-flip (23:30Z)** | Δ SR | Δ mean ratio | +|-------------------------|-------------------|------------------------|------|--------------| +| `quote` | 116S/51H/46U · SR 0.54 · ratio 0.699 | 116S/60H/49U · **SR 0.52** · **ratio 0.692** | −2pp | −1pp | +| `claim_lattice_pointer` | 43S/146H/21U · SR 0.20 · ratio 0.642 | 48S/145H/32U · **SR 0.21** · **ratio 0.663** | +1pp | +2pp | +| `claim_lattice` (JSON) | 89S/82H/39U · SR 0.42 · ratio 0.698 | 99S/80H/46U · **SR 0.44** · **ratio 0.730** | +2pp | +3pp | + +All STRICT-rate deltas within the 5pp signal floor. Same question +set ±2 (3 added: `winners of all major sports?`, `name all members +of the beatles`, `list all planets in the solar system`). + +**Substrate-level findings:** + +- **Pointer-mode `FORMAT_COLLAPSED`: 0/225** across the full sweep. + The default-on reminder eliminates the collapse mode globally, + not just on broad questions where it was originally measured. +- **Pointer-mode `NO_EVIDENCE_POINTER`: 29/225** (13%) — lower than + the 33% we saw on broad-only (9/27) when reminder was off. + Reminder discipline propagates to non-broad questions even though + the reminder text only fires on broad shapes (the broader signal + is that the reminder text reinforces the citation rule across + the model's session attention). +- **Quote-mode is essentially unchanged** (-2pp SR, -1pp ratio). + Quote opts out of the guard (`quantifier_guard_modes` default + excludes quote), so this is pure Hermes nondeterminism on a + 225-sample cell. +- **JSON-mode mean ratio +3pp** — small but positive, consistent + with the broad-subset finding that reminder boosts per-claim + grounding density. + +**Latency note.** Wall-clock per call jumped from ~17-19s (pre-flip +baseline) to ~33-35s on this run. This is **endpoint contention** +on Hermes-3 at the time of measurement, not a feature regression +(the new substrate adds zero LLM calls — preflight is pure regex). +The throughput numbers in §6 of the journal pre-date this evening's +network conditions; future bench cycles should normalize. + +**Verdict — no regression.** The #000010 default flip ships clean +across the full corpus, not just the broad-quantifier subset where +it was originally validated. Defaults stay on. The substrate is +strictly more honest under preflight (FORMAT_COLLAPSED → 0, +NO_EVIDENCE_POINTER ↓) without sacrificing throughput on +non-broad questions. + +Bench artifact: `bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}`.