bench: full 75-question regression check — no regression from #000010
Addendum 3 to qa-modes-bench.md. Validates that flipping quantifier_reminder_enabled=True for lattice modes (per #000010 §12.10 / §13.1) doesn't regress narrow-question performance. Prior validation (Addendum 2) covered the 9-question broad subset only. This run sweeps the full 75-question bench/qa_questions.txt (~10% broad, ~89% narrow), 225 runs per mode, comparing against the frozen 2026-05-02T15-07Z baseline. Findings: Mode | Pre-flip SR | Post-flip SR | Δ ----------+-------------+--------------+-------- quote | 0.54 | 0.52 | -2pp (within 5pp floor) pointer | 0.20 | 0.21 | +1pp JSON | 0.42 | 0.44 | +2pp Mean ratio: -1/+2/+3pp — all within noise band. Substrate-level wins beyond the headline metrics: - pointer FORMAT_COLLAPSED: 0/225 across the full sweep. Reminder eliminates collapse mode globally, not just on broad. - pointer NO_EVIDENCE_POINTER: 13% (vs 33% on broad-only when reminder was off). Citation discipline propagates beyond the rows where the reminder text actually fires — the model's session attention reinforces. - JSON mean ratio +3pp consistent with broad-subset finding. Quote-mode is essentially unchanged because it's mode-gated off the guard by default. Latency 33-35s/call this evening (vs 17-19s in prior runs) is endpoint contention, not a substrate regression — preflight adds zero LLM calls. Verdict: no regression. The default flip ships clean across the full corpus. Defaults stay on. The substrate is strictly more honest (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without sacrificing throughput on non-broad questions. Bench artifact: bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}.
This commit is contained in:
parent
cd0e8ef64f
commit
3bd36f36c8
1 changed files with 59 additions and 0 deletions
|
|
@ -253,3 +253,62 @@ improve by ≥5pp on lattice modes; STRICT-rate is within noise.
|
|||
Defaults stay on.
|
||||
|
||||
Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`.
|
||||
|
||||
### Addendum 3 — full-bench regression check (2026-05-03T23-30-12Z)
|
||||
|
||||
Validates that the #000010 default flip (reminder default-on for
|
||||
lattice modes) doesn't regress narrow-question performance. The
|
||||
prior validations in Addendum 1 + Addendum 2 used the 9-question
|
||||
broad subset only; this run sweeps the full 75-question
|
||||
`bench/qa_questions.txt` (~10% broad, ~89% narrow factoid /
|
||||
descriptive / list shapes).
|
||||
|
||||
Same harness, n=3 × 3 modes × 75 questions = 225 runs per mode.
|
||||
Compared against the frozen `15:07Z` baseline (the authoritative
|
||||
pre-#000008/9/10 state of the substrate, 71 questions × n=3 = 213
|
||||
runs per mode):
|
||||
|
||||
| Mode | Pre-flip (15:07Z) | **Post-flip (23:30Z)** | Δ SR | Δ mean ratio |
|
||||
|-------------------------|-------------------|------------------------|------|--------------|
|
||||
| `quote` | 116S/51H/46U · SR 0.54 · ratio 0.699 | 116S/60H/49U · **SR 0.52** · **ratio 0.692** | −2pp | −1pp |
|
||||
| `claim_lattice_pointer` | 43S/146H/21U · SR 0.20 · ratio 0.642 | 48S/145H/32U · **SR 0.21** · **ratio 0.663** | +1pp | +2pp |
|
||||
| `claim_lattice` (JSON) | 89S/82H/39U · SR 0.42 · ratio 0.698 | 99S/80H/46U · **SR 0.44** · **ratio 0.730** | +2pp | +3pp |
|
||||
|
||||
All STRICT-rate deltas within the 5pp signal floor. Same question
|
||||
set ±2 (3 added: `winners of all major sports?`, `name all members
|
||||
of the beatles`, `list all planets in the solar system`).
|
||||
|
||||
**Substrate-level findings:**
|
||||
|
||||
- **Pointer-mode `FORMAT_COLLAPSED`: 0/225** across the full sweep.
|
||||
The default-on reminder eliminates the collapse mode globally,
|
||||
not just on broad questions where it was originally measured.
|
||||
- **Pointer-mode `NO_EVIDENCE_POINTER`: 29/225** (13%) — lower than
|
||||
the 33% we saw on broad-only (9/27) when reminder was off.
|
||||
Reminder discipline propagates to non-broad questions even though
|
||||
the reminder text only fires on broad shapes (the broader signal
|
||||
is that the reminder text reinforces the citation rule across
|
||||
the model's session attention).
|
||||
- **Quote-mode is essentially unchanged** (-2pp SR, -1pp ratio).
|
||||
Quote opts out of the guard (`quantifier_guard_modes` default
|
||||
excludes quote), so this is pure Hermes nondeterminism on a
|
||||
225-sample cell.
|
||||
- **JSON-mode mean ratio +3pp** — small but positive, consistent
|
||||
with the broad-subset finding that reminder boosts per-claim
|
||||
grounding density.
|
||||
|
||||
**Latency note.** Wall-clock per call jumped from ~17-19s (pre-flip
|
||||
baseline) to ~33-35s on this run. This is **endpoint contention**
|
||||
on Hermes-3 at the time of measurement, not a feature regression
|
||||
(the new substrate adds zero LLM calls — preflight is pure regex).
|
||||
The throughput numbers in §6 of the journal pre-date this evening's
|
||||
network conditions; future bench cycles should normalize.
|
||||
|
||||
**Verdict — no regression.** The #000010 default flip ships clean
|
||||
across the full corpus, not just the broad-quantifier subset where
|
||||
it was originally validated. Defaults stay on. The substrate is
|
||||
strictly more honest under preflight (FORMAT_COLLAPSED → 0,
|
||||
NO_EVIDENCE_POINTER ↓) without sacrificing throughput on
|
||||
non-broad questions.
|
||||
|
||||
Bench artifact: `bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue