docs: full-bench validation in #000010 §13.2 + seven-point-program

Cross-references the 2026-05-03T23-30-12Z full 75-question
regression bench (Addendum 3 in qa-modes-bench.md) into:

  - #000010 §13.2 — full-bench validation alongside the §13.1
    broad-subset validation. Same verdict: defaults stay on.
    Substrate-level wins beyond noise-bounded SR deltas:
      pointer FORMAT_COLLAPSED: 0/225 (eliminated globally)
      pointer NO_EVIDENCE_POINTER: 13% (down from 33% broad-only)
      JSON mean ratio: +3pp

  - seven-point-program.md addendum — bench-validation paragraph
    after the test-count line. Names the SR deltas + the
    substrate-level wins; concrete evidence the post-landing
    directive coverage claim survives full-corpus measurement.

No code changes; 993 tests still passing.
This commit is contained in:
russell@unturf.com 2026-05-03 21:50:33 -04:00
parent 3bd36f36c8
commit 4c38bdebc1
No known key found for this signature in database
2 changed files with 50 additions and 0 deletions

View file

@ -284,6 +284,22 @@ Three additional tickets landed extending D1, D3, and D4:
Test count after these landings: **993 passing** (up from 734;
259 new across the three tickets). All seven directives remain ✓.
**Bench-validated post-landing** (full 75-question sweep
2026-05-03T23-30-12Z vs frozen 2026-05-02T15-07Z baseline,
n=3 × 3 modes = 675 runs):
- quote SR 0.54 → 0.52 (−2pp, within noise)
- pointer SR 0.20 → 0.21 (+1pp, within noise)
- JSON SR 0.42 → 0.44 (+2pp, within noise)
All STRICT-rate deltas within the 5pp signal floor. Substrate-
level wins: pointer-mode FORMAT_COLLAPSED eliminated globally
(0/225), pointer-mode NO_EVIDENCE_POINTER drops from 33%
(broad-only when reminder was off) to 13% (full sweep with
reminder on). The default flip is strictly more honest without
regressing throughput on narrow questions. See
`docs/qa-modes-bench.md` Addendum 3 for the full comparison.
## Anti-regression test layer
Each directive has a structural pin in `tests/test_directives.py`.

View file

@ -491,3 +491,37 @@ contribution; deferred.
Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`.
Documented in `docs/qa-modes-bench.md` Addendum 2.
### 13.2 Full-bench regression check (2026-05-03T23-30-12Z)
The §13.1 validation covered the 9-question broad subset where the
preflight contract activates. To confirm the default flip doesn't
regress narrow-question performance (89% of the bench), ran the full
75-question `bench/qa_questions.txt` × n=3 × 3 modes = 675 runs.
Compared against the frozen `2026-05-02T15-07Z` baseline (the
authoritative pre-#000008/9/10 state, 213 runs per mode):
| Mode | Pre-flip | **Post-flip** | Δ SR | Δ ratio |
|-------------------------|------------------------|------------------------|------|---------|
| `quote` | 116S/51H/46U · 0.54 | 116S/60H/49U · **0.52** | −2pp | −1pp |
| `claim_lattice_pointer` | 43S/146H/21U · 0.20 | 48S/145H/32U · **0.21** | +1pp | +2pp |
| `claim_lattice` (JSON) | 89S/82H/39U · 0.42 | 99S/80H/46U · **0.44** | +2pp | +3pp |
All deltas within the 5pp signal floor. Same question set ±2.
**Substrate-level wins beyond headline numbers:**
- pointer `FORMAT_COLLAPSED`: **0/225** across the full sweep
(default-on reminder eliminates it globally, not just on broad)
- pointer `NO_EVIDENCE_POINTER`: 13% (vs 33% on broad-only when
reminder was off) — citation discipline propagates to non-broad
questions even though the reminder text only fires on broad
- JSON mean ratio +3pp consistent with broad-subset finding
**Verdict:** the default flip ships clean across the full corpus.
Defaults stay on. The substrate is strictly more honest under
preflight (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without
sacrificing throughput on non-broad questions.
Bench artifact: `bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}`.
Documented in `docs/qa-modes-bench.md` Addendum 3.