engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

bench(#000010): preflight on-vs-off validation — defaults stay on

Browse source 
Post-landing validation cell. Same 9-question broad subset as
§12.6 reminder-only baseline; this run flips both
metacognition_enabled and quantifier_reminder_enabled to False so
we can isolate the preflight contribution.

  Metric                    | OFF       | ON       | Δ
  --------------------------+-----------+----------+----------
  pointer mean ratio        | 0.483     | 0.643    | +16pp
  JSON mean ratio           | 0.570     | 0.735    | +17pp
  JSON UNGROUNDED rate      | 7/27      | 1/27     | -22pp
  pointer FORMAT_COLLAPSED  | 2/27      | 0/27     | -100%
  any-mode STRICT-rate      | within ±11pp noise (27-sample n=3 floor)

Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all clear the
5pp signal floor on lattice modes. STRICT-rate moves are within
Hermes nondeterminism. The #000010 default flip is doing what
was claimed.

On this subset none of the metacognition detectors fire (no
temporal / contradiction / false-premise / out-of-corpus shapes),
so the delta effectively isolates the reminder contribution from
#000008. A metacog-trigger subset bench is deferred.

Documentation:
  - docs/qa-modes-bench.md Addendum 2 captures the comparison
    table + interpretation + verdict.
  - docs/tickets/ticket-000010-... §13.1 cross-references with
    bench artifact stamp.

Bench artifact: bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}.
This commit is contained in:
russell@unturf.com 2026-05-03 19:26:33 -04:00
parent de07ad9392
commit cd0e8ef64f
No known key found for this signature in database
2 changed files with 78 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

48
docs/qa-modes-bench.md
View file

@ -205,3 +205,51 @@ Findings relevant to this journal's per-mode recommendations:
See `docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md`
§12 for the four-cell data and the §10.8 decision-tree verdict.
Bench artifacts: `bench/qa_results/2026-05-03T12-{29-53,38-53,47-23,54-11}Z.{jsonl,md}`.
### Addendum 2 — preflight ON vs OFF validation (2026-05-03T23-06-21Z)
After ticket #000010 flipped `quantifier_reminder_enabled=True` and
shipped `metacognition_enabled=True` defaults, ran a preflight-OFF
cell to validate the flip didn't regress baseline behavior. Same
9-question broad subset (`bench/qa_questions_quantifier_subset.txt`),
n=3, policy override
`--policy metacognition_enabled=false --policy quantifier_reminder_enabled=false`.
Compared against §12.6 reminder-only baseline (cleanest single-knob
on-cell). On this 9-question subset, none of the metacognition
detectors fire (no temporal / contradiction / false-premise / out-
of-corpus shapes), so the comparison effectively isolates the
reminder contribution.
| Metric | OFF | ON | Δ |
|------------------------------|-----------|-----------|----------|
| `quote` strict-rate | 0.63 | 0.52 | −11pp (noise band on 27 samples) |
| `pointer` strict-rate | 0.00 | 0.00 | 0 |
| `JSON` strict-rate | 0.26 | 0.22 | −4pp |
| **`pointer` mean ratio** | 0.483 | **0.643** | **+16pp** |
| **`JSON` mean ratio** | 0.570 | **0.735** | **+17pp** |
| **`JSON` UNGROUNDED rate** | 7/27 (26%) | **1/27 (4%)** | **−22pp** |
| **`pointer` FORMAT_COLLAPSED** | 2/27 | **0/27** | **−100%** |
| `pointer` NO_EVIDENCE_POINTER | 8/27 | 6/27 | −7pp |
**Interpretation:**
- **Mean-ratio improvement is robust** (+16-17pp on lattice modes).
Grounded rows are MORE thoroughly grounded under preflight ON.
- **JSON UNGROUNDED collapse is dramatic** (−22pp). The "didn't
ground" pool reclassifies into "partially grounded" — operator-
visible win.
- **FORMAT_COLLAPSED elimination on pointer mode** (2 → 0). The
reminder restates the [E\d+] citation rule and Hermes follows
it.
- **STRICT-rate moves are within noise.** Quote-mode dipped 11pp,
but quote is mode-gated off the guard so this is pure Hermes
nondeterminism — the §10.8 5pp floor exists exactly to filter
this. JSON SR moved −4pp (within floor).
**Verdict:** the #000010 default-on flip is doing what was
claimed. Mean-ratio + UNGROUNDED + FORMAT_COLLAPSED metrics all
improve by ≥5pp on lattice modes; STRICT-rate is within noise.
Defaults stay on.
Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`.

30
docs/tickets/ticket-000010-metacognition-preflight-guard.md
View file

@ -461,3 +461,33 @@ Follow-ups split into separate tickets:
This ticket file remains in place as the design log per
`docs/TICKETS.md` convention.
### 13.1 Default-flip validation bench (2026-05-03T23-06-21Z)
Post-landing validation: ran a preflight-OFF cell on the
9-question broad subset (`metacognition_enabled=false`,
`quantifier_reminder_enabled=false`) to confirm the §12.10 default
flip wasn't a regression. Compared against §12.6 reminder-only
baseline.
| Metric | OFF | ON | Δ |
|------------------------------|-----------|-----------|----------|
| pointer mean ratio | 0.483 | **0.643** | **+16pp** |
| JSON mean ratio | 0.570 | **0.735** | **+17pp** |
| JSON UNGROUNDED rate | 7/27 (26%) | **1/27 (4%)** | **−22pp** |
| pointer FORMAT_COLLAPSED | 2/27 | **0/27** | **−100%** |
| pointer NO_EVIDENCE_POINTER | 8/27 | 6/27 | −7pp |
| any-mode STRICT-rate | within ±11pp noise band on 27-sample cells |
**Verdict:** defaults stay on. Mean-ratio + UNGROUNDED + FORMAT_
COLLAPSED metrics all clear the 5pp signal floor on lattice modes;
STRICT-rate moves are within Hermes nondeterminism for n=3.
On this subset the metacognition detectors don't fire (no
temporal / contradiction / false-premise / out-of-corpus shapes),
so the delta isolates the reminder contribution. A future bench
cycle on a metacog-trigger subset would isolate the metacog
contribution; deferred.
Bench artifact: `bench/qa_results/2026-05-03T23-06-21Z.{jsonl,md}`.
Documented in `docs/qa-modes-bench.md` Addendum 2.