From cd0e8ef64fcf6ee83c24ec9a3ae22c119e9a8293 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 3 May 2026 19:26:33 -0400 Subject: [PATCH] =?UTF-8?q?bench(#000010):=20preflight=20on-vs-off=20valid?= =?UTF-8?q?ation=20=E2=80=94=20defaults=20stay=20on?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-landing validation cell. Same 9-question broad subset as §12.6 reminder-only baseline; this run flips both metacognition_enabled and quantifier_reminder_enabled to False so we can isolate the preflight contribution. Metric | OFF | ON | Δ --------------------------+-----------+----------+---------- pointer mean ratio | 0.483 | 0.643 | +16pp JSON mean ratio | 0.570 | 0.735 | +17pp JSON UNGROUNDED rate | 7/27 | 1/27 | -22pp pointer FORMAT_COLLAPSED | 2/27 | 0/27 | -100% any-mode STRICT-rate | within ±11pp noise (27-sample n=3 floor) Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all clear the 5pp signal floor on lattice modes. STRICT-rate moves are within Hermes nondeterminism. The #000010 default flip is doing what was claimed. On this subset none of the metacognition detectors fire (no temporal / contradiction / false-premise / out-of-corpus shapes), so the delta effectively isolates the reminder contribution from #000008. A metacog-trigger subset bench is deferred. Documentation: - docs/qa-modes-bench.md Addendum 2 captures the comparison table + interpretation + verdict. - docs/tickets/ticket-000010-... §13.1 cross-references with bench artifact stamp. Bench artifact: bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}. --- docs/qa-modes-bench.md | 48 +++++++++++++++++++ ...et-000010-metacognition-preflight-guard.md | 30 ++++++++++++ 2 files changed, 78 insertions(+) diff --git a/docs/qa-modes-bench.md b/docs/qa-modes-bench.md index f336268..a3b5233 100644 --- a/docs/qa-modes-bench.md +++ b/docs/qa-modes-bench.md @@ -205,3 +205,51 @@ Findings relevant to this journal's per-mode recommendations: See `docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md` §12 for the four-cell data and the §10.8 decision-tree verdict. Bench artifacts: `bench/qa_results/2026-05-03T12-{29-53,38-53,47-23,54-11}Z.{jsonl,md}`. + +### Addendum 2 — preflight ON vs OFF validation (2026-05-03T23-06-21Z) + +After ticket #000010 flipped `quantifier_reminder_enabled=True` and +shipped `metacognition_enabled=True` defaults, ran a preflight-OFF +cell to validate the flip didn't regress baseline behavior. Same +9-question broad subset (`bench/qa_questions_quantifier_subset.txt`), +n=3, policy override +`--policy metacognition_enabled=false --policy quantifier_reminder_enabled=false`. + +Compared against §12.6 reminder-only baseline (cleanest single-knob +on-cell). On this 9-question subset, none of the metacognition +detectors fire (no temporal / contradiction / false-premise / out- +of-corpus shapes), so the comparison effectively isolates the +reminder contribution. + +| Metric | OFF | ON | Δ | +|------------------------------|-----------|-----------|----------| +| `quote` strict-rate | 0.63 | 0.52 | −11pp (noise band on 27 samples) | +| `pointer` strict-rate | 0.00 | 0.00 | 0 | +| `JSON` strict-rate | 0.26 | 0.22 | −4pp | +| **`pointer` mean ratio** | 0.483 | **0.643** | **+16pp** | +| **`JSON` mean ratio** | 0.570 | **0.735** | **+17pp** | +| **`JSON` UNGROUNDED rate** | 7/27 (26%) | **1/27 (4%)** | **−22pp** | +| **`pointer` FORMAT_COLLAPSED** | 2/27 | **0/27** | **−100%** | +| `pointer` NO_EVIDENCE_POINTER | 8/27 | 6/27 | −7pp | + +**Interpretation:** + +- **Mean-ratio improvement is robust** (+16-17pp on lattice modes). + Grounded rows are MORE thoroughly grounded under preflight ON. +- **JSON UNGROUNDED collapse is dramatic** (−22pp). The "didn't + ground" pool reclassifies into "partially grounded" — operator- + visible win. +- **FORMAT_COLLAPSED elimination on pointer mode** (2 → 0). The + reminder restates the [E\d+] citation rule and Hermes follows + it. +- **STRICT-rate moves are within noise.** Quote-mode dipped 11pp, + but quote is mode-gated off the guard so this is pure Hermes + nondeterminism — the §10.8 5pp floor exists exactly to filter + this. JSON SR moved −4pp (within floor). + +**Verdict:** the #000010 default-on flip is doing what was +claimed. Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all +improve by ≥5pp on lattice modes; STRICT-rate is within noise. +Defaults stay on. + +Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`. diff --git a/docs/tickets/ticket-000010-metacognition-preflight-guard.md b/docs/tickets/ticket-000010-metacognition-preflight-guard.md index ab0e958..33d4fed 100644 --- a/docs/tickets/ticket-000010-metacognition-preflight-guard.md +++ b/docs/tickets/ticket-000010-metacognition-preflight-guard.md @@ -461,3 +461,33 @@ Follow-ups split into separate tickets: This ticket file remains in place as the design log per `docs/TICKETS.md` convention. + +### 13.1 Default-flip validation bench (2026-05-03T23-06-21Z) + +Post-landing validation: ran a preflight-OFF cell on the +9-question broad subset (`metacognition_enabled=false`, +`quantifier_reminder_enabled=false`) to confirm the §12.10 default +flip wasn't a regression. Compared against §12.6 reminder-only +baseline. + +| Metric | OFF | ON | Δ | +|------------------------------|-----------|-----------|----------| +| pointer mean ratio | 0.483 | **0.643** | **+16pp** | +| JSON mean ratio | 0.570 | **0.735** | **+17pp** | +| JSON UNGROUNDED rate | 7/27 (26%) | **1/27 (4%)** | **−22pp** | +| pointer FORMAT_COLLAPSED | 2/27 | **0/27** | **−100%** | +| pointer NO_EVIDENCE_POINTER | 8/27 | 6/27 | −7pp | +| any-mode STRICT-rate | within ±11pp noise band on 27-sample cells | + +**Verdict:** defaults stay on. Mean-ratio + UNGROUNDED + FORMAT_ +COLLAPSED metrics all clear the 5pp signal floor on lattice modes; +STRICT-rate moves are within Hermes nondeterminism for n=3. + +On this subset the metacognition detectors don't fire (no +temporal / contradiction / false-premise / out-of-corpus shapes), +so the delta isolates the reminder contribution. A future bench +cycle on a metacog-trigger subset would isolate the metacog +contribution; deferred. + +Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`. +Documented in `docs/qa-modes-bench.md` Addendum 2.