diff --git a/docs/qa-modes-bench.md b/docs/qa-modes-bench.md index f336268..a3b5233 100644 --- a/docs/qa-modes-bench.md +++ b/docs/qa-modes-bench.md @@ -205,3 +205,51 @@ Findings relevant to this journal's per-mode recommendations: See `docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md` §12 for the four-cell data and the §10.8 decision-tree verdict. Bench artifacts: `bench/qa_results/2026-05-03T12-{29-53,38-53,47-23,54-11}Z.{jsonl,md}`. + +### Addendum 2 — preflight ON vs OFF validation (2026-05-03T23-06-21Z) + +After ticket #000010 flipped `quantifier_reminder_enabled=True` and +shipped `metacognition_enabled=True` defaults, ran a preflight-OFF +cell to validate the flip didn't regress baseline behavior. Same +9-question broad subset (`bench/qa_questions_quantifier_subset.txt`), +n=3, policy override +`--policy metacognition_enabled=false --policy quantifier_reminder_enabled=false`. + +Compared against §12.6 reminder-only baseline (cleanest single-knob +on-cell). On this 9-question subset, none of the metacognition +detectors fire (no temporal / contradiction / false-premise / out- +of-corpus shapes), so the comparison effectively isolates the +reminder contribution. + +| Metric | OFF | ON | Δ | +|------------------------------|-----------|-----------|----------| +| `quote` strict-rate | 0.63 | 0.52 | −11pp (noise band on 27 samples) | +| `pointer` strict-rate | 0.00 | 0.00 | 0 | +| `JSON` strict-rate | 0.26 | 0.22 | −4pp | +| **`pointer` mean ratio** | 0.483 | **0.643** | **+16pp** | +| **`JSON` mean ratio** | 0.570 | **0.735** | **+17pp** | +| **`JSON` UNGROUNDED rate** | 7/27 (26%) | **1/27 (4%)** | **−22pp** | +| **`pointer` FORMAT_COLLAPSED** | 2/27 | **0/27** | **−100%** | +| `pointer` NO_EVIDENCE_POINTER | 8/27 | 6/27 | −7pp | + +**Interpretation:** + +- **Mean-ratio improvement is robust** (+16-17pp on lattice modes). + Grounded rows are MORE thoroughly grounded under preflight ON. +- **JSON UNGROUNDED collapse is dramatic** (−22pp). The "didn't + ground" pool reclassifies into "partially grounded" — operator- + visible win. +- **FORMAT_COLLAPSED elimination on pointer mode** (2 → 0). The + reminder restates the [E\d+] citation rule and Hermes follows + it. +- **STRICT-rate moves are within noise.** Quote-mode dipped 11pp, + but quote is mode-gated off the guard so this is pure Hermes + nondeterminism — the §10.8 5pp floor exists exactly to filter + this. JSON SR moved −4pp (within floor). + +**Verdict:** the #000010 default-on flip is doing what was +claimed. Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all +improve by ≥5pp on lattice modes; STRICT-rate is within noise. +Defaults stay on. + +Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`. diff --git a/docs/tickets/ticket-000010-metacognition-preflight-guard.md b/docs/tickets/ticket-000010-metacognition-preflight-guard.md index ab0e958..33d4fed 100644 --- a/docs/tickets/ticket-000010-metacognition-preflight-guard.md +++ b/docs/tickets/ticket-000010-metacognition-preflight-guard.md @@ -461,3 +461,33 @@ Follow-ups split into separate tickets: This ticket file remains in place as the design log per `docs/TICKETS.md` convention. + +### 13.1 Default-flip validation bench (2026-05-03T23-06-21Z) + +Post-landing validation: ran a preflight-OFF cell on the +9-question broad subset (`metacognition_enabled=false`, +`quantifier_reminder_enabled=false`) to confirm the §12.10 default +flip wasn't a regression. Compared against §12.6 reminder-only +baseline. + +| Metric | OFF | ON | Δ | +|------------------------------|-----------|-----------|----------| +| pointer mean ratio | 0.483 | **0.643** | **+16pp** | +| JSON mean ratio | 0.570 | **0.735** | **+17pp** | +| JSON UNGROUNDED rate | 7/27 (26%) | **1/27 (4%)** | **−22pp** | +| pointer FORMAT_COLLAPSED | 2/27 | **0/27** | **−100%** | +| pointer NO_EVIDENCE_POINTER | 8/27 | 6/27 | −7pp | +| any-mode STRICT-rate | within ±11pp noise band on 27-sample cells | + +**Verdict:** defaults stay on. Mean-ratio + UNGROUNDED + FORMAT_ +COLLAPSED metrics all clear the 5pp signal floor on lattice modes; +STRICT-rate moves are within Hermes nondeterminism for n=3. + +On this subset the metacognition detectors don't fire (no +temporal / contradiction / false-premise / out-of-corpus shapes), +so the delta isolates the reminder contribution. A future bench +cycle on a metacog-trigger subset would isolate the metacog +contribution; deferred. + +Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`. +Documented in `docs/qa-modes-bench.md` Addendum 2.