From 4c38bdebc1de70cc125952d4fb1538adc5cedd35 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 3 May 2026 21:50:33 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20full-bench=20validation=20in=20#000010?= =?UTF-8?q?=20=C2=A713.2=20+=20seven-point-program?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-references the 2026-05-03T23-30-12Z full 75-question regression bench (Addendum 3 in qa-modes-bench.md) into: - #000010 §13.2 — full-bench validation alongside the §13.1 broad-subset validation. Same verdict: defaults stay on. Substrate-level wins beyond noise-bounded SR deltas: pointer FORMAT_COLLAPSED: 0/225 (eliminated globally) pointer NO_EVIDENCE_POINTER: 13% (down from 33% broad-only) JSON mean ratio: +3pp - seven-point-program.md addendum — bench-validation paragraph after the test-count line. Names the SR deltas + the substrate-level wins; concrete evidence the post-landing directive coverage claim survives full-corpus measurement. No code changes; 993 tests still passing. --- docs/seven-point-program.md | 16 +++++++++ ...et-000010-metacognition-preflight-guard.md | 34 +++++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/docs/seven-point-program.md b/docs/seven-point-program.md index 608f600..1a89ba9 100644 --- a/docs/seven-point-program.md +++ b/docs/seven-point-program.md @@ -284,6 +284,22 @@ Three additional tickets landed extending D1, D3, and D4: Test count after these landings: **993 passing** (up from 734; 259 new across the three tickets). All seven directives remain ✓. +**Bench-validated post-landing** (full 75-question sweep +2026-05-03T23-30-12Z vs frozen 2026-05-02T15-07Z baseline, +n=3 × 3 modes = 675 runs): + + - quote SR 0.54 → 0.52 (−2pp, within noise) + - pointer SR 0.20 → 0.21 (+1pp, within noise) + - JSON SR 0.42 → 0.44 (+2pp, within noise) + +All STRICT-rate deltas within the 5pp signal floor. Substrate- +level wins: pointer-mode FORMAT_COLLAPSED eliminated globally +(0/225), pointer-mode NO_EVIDENCE_POINTER drops from 33% +(broad-only when reminder was off) to 13% (full sweep with +reminder on). The default flip is strictly more honest without +regressing throughput on narrow questions. See +`docs/qa-modes-bench.md` Addendum 3 for the full comparison. + ## Anti-regression test layer Each directive has a structural pin in `tests/test_directives.py`. diff --git a/docs/tickets/ticket-000010-metacognition-preflight-guard.md b/docs/tickets/ticket-000010-metacognition-preflight-guard.md index 33d4fed..71af778 100644 --- a/docs/tickets/ticket-000010-metacognition-preflight-guard.md +++ b/docs/tickets/ticket-000010-metacognition-preflight-guard.md @@ -491,3 +491,37 @@ contribution; deferred. Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`. Documented in `docs/qa-modes-bench.md` Addendum 2. + +### 13.2 Full-bench regression check (2026-05-03T23-30-12Z) + +The §13.1 validation covered the 9-question broad subset where the +preflight contract activates. To confirm the default flip doesn't +regress narrow-question performance (89% of the bench), ran the full +75-question `bench/qa_questions.txt` × n=3 × 3 modes = 675 runs. +Compared against the frozen `2026-05-02T15-07Z` baseline (the +authoritative pre-#000008/9/10 state, 213 runs per mode): + +| Mode | Pre-flip | **Post-flip** | Δ SR | Δ ratio | +|-------------------------|------------------------|------------------------|------|---------| +| `quote` | 116S/51H/46U · 0.54 | 116S/60H/49U · **0.52** | −2pp | −1pp | +| `claim_lattice_pointer` | 43S/146H/21U · 0.20 | 48S/145H/32U · **0.21** | +1pp | +2pp | +| `claim_lattice` (JSON) | 89S/82H/39U · 0.42 | 99S/80H/46U · **0.44** | +2pp | +3pp | + +All deltas within the 5pp signal floor. Same question set ±2. + +**Substrate-level wins beyond headline numbers:** + +- pointer `FORMAT_COLLAPSED`: **0/225** across the full sweep + (default-on reminder eliminates it globally, not just on broad) +- pointer `NO_EVIDENCE_POINTER`: 13% (vs 33% on broad-only when + reminder was off) — citation discipline propagates to non-broad + questions even though the reminder text only fires on broad +- JSON mean ratio +3pp consistent with broad-subset finding + +**Verdict:** the default flip ships clean across the full corpus. +Defaults stay on. The substrate is strictly more honest under +preflight (FORMAT_COLLAPSED → 0, NO_EVIDENCE_POINTER ↓) without +sacrificing throughput on non-broad questions. + +Bench artifact: `bench/qa_results/2026-05-03T23-30-12Z.{jsonl,md}`. +Documented in `docs/qa-modes-bench.md` Addendum 3.