bench: full 75-question regression check — no regression from #000010

Addendum 3 to qa-modes-bench.md. Validates that flipping
quantifier_reminder_enabled=True for lattice modes (per #000010
§12.10 / §13.1) doesn't regress narrow-question performance.

Prior validation (Addendum 2) covered the 9-question broad subset
only. This run sweeps the full 75-question bench/qa_questions.txt
(~10% broad, ~89% narrow), 225 runs per mode, comparing against
the frozen 2026-05-02T15-07Z baseline.

Findings:

  Mode      | Pre-flip SR | Post-flip SR | Δ
  ----------+-------------+--------------+--------
  quote     | 0.54        | 0.52         | -2pp (within 5pp floor)
  pointer   | 0.20        | 0.21         | +1pp
  JSON      | 0.42        | 0.44         | +2pp

  Mean ratio: -1/+2/+3pp — all within noise band.

Substrate-level wins beyond the headline metrics:
  - pointer FORMAT_COLLAPSED: 0/225 across the full sweep.
    Reminder eliminates collapse mode globally, not just on broad.
  - pointer NO_EVIDENCE_POINTER: 13% (vs 33% on broad-only when
    reminder was off). Citation discipline propagates beyond the
    rows where the reminder text actually fires — the model's
    session attention reinforces.
  - JSON mean ratio +3pp consistent with broad-subset finding.

Quote-mode is essentially unchanged because it's mode-gated off
the guard by default.

Latency 33-35s/call this evening (vs 17-19s in prior runs) is
endpoint contention, not a substrate regression — preflight adds
zero LLM calls.

Verdict: no regression. The default flip ships clean across the
full corpus. Defaults stay on. The substrate is strictly more
honest (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without
sacrificing throughput on non-broad questions.

Bench artifact: bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}.
This commit is contained in:
russell@unturf.com 2026-05-03 21:07:56 -04:00
parent cd0e8ef64f
commit 3bd36f36c8
No known key found for this signature in database

View file

@ -253,3 +253,62 @@ improve by ≥5pp on lattice modes; STRICT-rate is within noise.
Defaults stay on.
Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`.
### Addendum 3 — full-bench regression check (2026-05-03T23-30-12Z)
Validates that the #000010 default flip (reminder default-on for
lattice modes) doesn't regress narrow-question performance. The
prior validations in Addendum 1 + Addendum 2 used the 9-question
broad subset only; this run sweeps the full 75-question
`bench/qa_questions.txt` (~10% broad, ~89% narrow factoid /
descriptive / list shapes).
Same harness, n=3 × 3 modes × 75 questions = 225 runs per mode.
Compared against the frozen `15:07Z` baseline (the authoritative
pre-#000008/9/10 state of the substrate, 71 questions × n=3 = 213
runs per mode):
| Mode | Pre-flip (15:07Z) | **Post-flip (23:30Z)** | Δ SR | Δ mean ratio |
|-------------------------|-------------------|------------------------|------|--------------|
| `quote` | 116S/51H/46U · SR 0.54 · ratio 0.699 | 116S/60H/49U · **SR 0.52** · **ratio 0.692** | −2pp | −1pp |
| `claim_lattice_pointer` | 43S/146H/21U · SR 0.20 · ratio 0.642 | 48S/145H/32U · **SR 0.21** · **ratio 0.663** | +1pp | +2pp |
| `claim_lattice` (JSON) | 89S/82H/39U · SR 0.42 · ratio 0.698 | 99S/80H/46U · **SR 0.44** · **ratio 0.730** | +2pp | +3pp |
All STRICT-rate deltas within the 5pp signal floor. Same question
set ±2 (3 added: `winners of all major sports?`, `name all members
of the beatles`, `list all planets in the solar system`).
**Substrate-level findings:**
- **Pointer-mode `FORMAT_COLLAPSED`: 0/225** across the full sweep.
The default-on reminder eliminates the collapse mode globally,
not just on broad questions where it was originally measured.
- **Pointer-mode `NO_EVIDENCE_POINTER`: 29/225** (13%) — lower than
the 33% we saw on broad-only (9/27) when reminder was off.
Reminder discipline propagates to non-broad questions even though
the reminder text only fires on broad shapes (the broader signal
is that the reminder text reinforces the citation rule across
the model's session attention).
- **Quote-mode is essentially unchanged** (-2pp SR, -1pp ratio).
Quote opts out of the guard (`quantifier_guard_modes` default
excludes quote), so this is pure Hermes nondeterminism on a
225-sample cell.
- **JSON-mode mean ratio +3pp** — small but positive, consistent
with the broad-subset finding that reminder boosts per-claim
grounding density.
**Latency note.** Wall-clock per call jumped from ~17-19s (pre-flip
baseline) to ~33-35s on this run. This is **endpoint contention**
on Hermes-3 at the time of measurement, not a feature regression
(the new substrate adds zero LLM calls — preflight is pure regex).
The throughput numbers in §6 of the journal pre-date this evening's
network conditions; future bench cycles should normalize.
**Verdict — no regression.** The #000010 default flip ships clean
across the full corpus, not just the broad-quantifier subset where
it was originally validated. Defaults stay on. The substrate is
strictly more honest under preflight (FORMAT_COLLAPSED → 0,
NO_EVIDENCE_POINTER ↓) without sacrificing throughput on
non-broad questions.
Bench artifact: `bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}`.