docs: full-bench validation in #000010 §13.2 + seven-point-program
Cross-references the 2026-05-03T23-30-12Z full 75-question regression bench (Addendum 3 in qa-modes-bench.md) into: - #000010 §13.2 — full-bench validation alongside the §13.1 broad-subset validation. Same verdict: defaults stay on. Substrate-level wins beyond noise-bounded SR deltas: pointer FORMAT_COLLAPSED: 0/225 (eliminated globally) pointer NO_EVIDENCE_POINTER: 13% (down from 33% broad-only) JSON mean ratio: +3pp - seven-point-program.md addendum — bench-validation paragraph after the test-count line. Names the SR deltas + the substrate-level wins; concrete evidence the post-landing directive coverage claim survives full-corpus measurement. No code changes; 993 tests still passing.
This commit is contained in:
parent
3bd36f36c8
commit
4c38bdebc1
2 changed files with 50 additions and 0 deletions
|
|
@ -284,6 +284,22 @@ Three additional tickets landed extending D1, D3, and D4:
|
|||
Test count after these landings: **993 passing** (up from 734;
|
||||
259 new across the three tickets). All seven directives remain ✓.
|
||||
|
||||
**Bench-validated post-landing** (full 75-question sweep
|
||||
2026-05-03T23-30-12Z vs frozen 2026-05-02T15-07Z baseline,
|
||||
n=3 × 3 modes = 675 runs):
|
||||
|
||||
- quote SR 0.54 → 0.52 (−2pp, within noise)
|
||||
- pointer SR 0.20 → 0.21 (+1pp, within noise)
|
||||
- JSON SR 0.42 → 0.44 (+2pp, within noise)
|
||||
|
||||
All STRICT-rate deltas within the 5pp signal floor. Substrate-
|
||||
level wins: pointer-mode FORMAT_COLLAPSED eliminated globally
|
||||
(0/225), pointer-mode NO_EVIDENCE_POINTER drops from 33%
|
||||
(broad-only when reminder was off) to 13% (full sweep with
|
||||
reminder on). The default flip is strictly more honest without
|
||||
regressing throughput on narrow questions. See
|
||||
`docs/qa-modes-bench.md` Addendum 3 for the full comparison.
|
||||
|
||||
## Anti-regression test layer
|
||||
|
||||
Each directive has a structural pin in `tests/test_directives.py`.
|
||||
|
|
|
|||
|
|
@ -491,3 +491,37 @@ contribution; deferred.
|
|||
|
||||
Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`.
|
||||
Documented in `docs/qa-modes-bench.md` Addendum 2.
|
||||
|
||||
### 13.2 Full-bench regression check (2026-05-03T23-30-12Z)
|
||||
|
||||
The §13.1 validation covered the 9-question broad subset where the
|
||||
preflight contract activates. To confirm the default flip doesn't
|
||||
regress narrow-question performance (89% of the bench), ran the full
|
||||
75-question `bench/qa_questions.txt` × n=3 × 3 modes = 675 runs.
|
||||
Compared against the frozen `2026-05-02T15-07Z` baseline (the
|
||||
authoritative pre-#000008/9/10 state, 213 runs per mode):
|
||||
|
||||
| Mode | Pre-flip | **Post-flip** | Δ SR | Δ ratio |
|
||||
|-------------------------|------------------------|------------------------|------|---------|
|
||||
| `quote` | 116S/51H/46U · 0.54 | 116S/60H/49U · **0.52** | −2pp | −1pp |
|
||||
| `claim_lattice_pointer` | 43S/146H/21U · 0.20 | 48S/145H/32U · **0.21** | +1pp | +2pp |
|
||||
| `claim_lattice` (JSON) | 89S/82H/39U · 0.42 | 99S/80H/46U · **0.44** | +2pp | +3pp |
|
||||
|
||||
All deltas within the 5pp signal floor. Same question set ±2.
|
||||
|
||||
**Substrate-level wins beyond headline numbers:**
|
||||
|
||||
- pointer `FORMAT_COLLAPSED`: **0/225** across the full sweep
|
||||
(default-on reminder eliminates it globally, not just on broad)
|
||||
- pointer `NO_EVIDENCE_POINTER`: 13% (vs 33% on broad-only when
|
||||
reminder was off) — citation discipline propagates to non-broad
|
||||
questions even though the reminder text only fires on broad
|
||||
- JSON mean ratio +3pp consistent with broad-subset finding
|
||||
|
||||
**Verdict:** the default flip ships clean across the full corpus.
|
||||
Defaults stay on. The substrate is strictly more honest under
|
||||
preflight (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without
|
||||
sacrificing throughput on non-broad questions.
|
||||
|
||||
Bench artifact: `bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}`.
|
||||
Documented in `docs/qa-modes-bench.md` Addendum 3.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue