ticket(#000008,#000006,index): §12.6 reminder A/B verdict + cross-refs

§12.6 in #000008 captures the 2026-05-03T12-38-53Z reminder-only
A/B (apply_caps=False, reminder=True) on the same 9-question
broad subset:

  Mode      | Strict-rate    | Mean ratio        | UNGROUNDED
  ----------+----------------+-------------------+-----------
  quote     | 0.56 → 0.52    | 0.900 → 0.845     |  0 → 0
  pointer   | 0.00 → 0.00    | 0.473 → 0.643     |  9 → 6
  JSON      | 0.19 → 0.22    | 0.524 → 0.735     |  7 → 1

Pointer-mode violation deltas:
  FORMAT_COLLAPSED      2 →  0  (-100%)
  NO_EVIDENCE_POINTER   9 →  6  ( -33%)
  TITLE_MISMATCH       10 → 15  ( +50%)  ← side effect
  TOO_MANY_CLAIMS       7 →  8  ( +14%)

§10.8 gate verdict: MET. Both FORMAT_COLLAPSED and NO_EVIDENCE_
POINTER cleared the 5pp floor (−7pp absolute / −11pp absolute
respectively). Strongest signals are mean-ratio improvements
(+17pp pointer, +21pp JSON) — grounded rows ground BETTER under
reminder. JSON-mode UNGROUNDED dropped 7 → 1, a 22pp redistribution
from "didn't ground" to "partially grounded".

Caveat: TITLE_MISMATCH increased (+50%). Reminder may pressure
Hermes to cite *something* rather than say "no evidence", picking
up wrong-source citations as a side effect.

Recommendation: §10.8 gate met but hold default flip until §12.7
(cap-only) and §12.8 (cap+reminder) cells run, per §10.8 "if A+B
together outperform either alone by ≥5pp: land Option D".

Cross-references:

- TICKETS.md index: #000008 status flipped to "open · phases 0–4
  landed; bench A/B in progress".
- #000006 rolling log: cross-reference to #000008's bench cycles
  + the d24291b classifier-defect fix surfaced from the
  distribution scan.
This commit is contained in:
russell@unturf.com 2026-05-03 08:48:19 -04:00
parent 002f84c5a4
commit fa8d93c8ef
No known key found for this signature in database
3 changed files with 99 additions and 10 deletions

View file

@ -57,7 +57,7 @@ Newest first. Update on every open/close.
| ID | Title | Status | Opened | Directive |
|----------|------------------------------------------------|-----------------------|------------|-----------|
| #000008 | Broad-quantifier preflight guard | open · awaiting go/no-go | 2026-05-02 | — |
| #000008 | Broad-quantifier preflight guard | open · phases 04 landed; bench A/B in progress | 2026-05-02 | — |
| #000007 | Query-layer hyphen folding | closed · 2026-05-02 | 2026-05-02 | — |
| #000006 | Bench-emergent findings (rolling research log) | open · rolling | 2026-05-02 | — |
| #000005 | Label ladder migration (POINTER-LINKED → …) | closed · 2026-05-02 | 2026-05-01 | D7 |

View file

@ -381,3 +381,24 @@ Open · rolling research log. Three original tuning candidates:
sidecar tightened); (2) `xxviii` STRICT acceptance — kept as
designed; (3) metaphor sidecar calibration — deferred until
sample size grows. Re-amend as findings accumulate.
## Cross-reference: ticket #000008 bench cycles (2026-05-03)
Ticket #000008 (broad-quantifier preflight guard) landed Phases
04 on 2026-05-03 and now runs its own bench A/B cycles per its
§12 with the `--policy KEY=VALUE` harness flag (commit `002f84c`).
Findings relevant to the rolling-emergent stream:
- The `bench/qa_questions.txt` 73-question set carries 7 broad
questions (~10%) classified ALL/COMPREHENSIVE/OPEN_REQUEST by
the §10 classifier. Two bounded-universal fixtures added 2026-
05-03 (`name all members of the beatles`, `list all planets in
the solar system`).
- Classifier defect caught + fixed in `d24291b`: `how many X?`
was mis-classifying as MANY (4 false positives across the
bench). Fixed via leading-anchor count-question short-circuit.
- `metaphor_deflection` calibration (item 3 above) can leverage
#000008's larger bench cycles — the broad-subset A/B at n=3 ×
3 modes accumulates 81 rows per cell, useful sample size growth
for metaphor-sidecar tuning if the metaphor cue happens to fire
on broad-quantifier shapes.

View file

@ -1890,15 +1890,83 @@ Empirically:
Per §10.8 decision tree, A/B sequence with this same broad subset:
- [ ] **Reminder only** (apply_caps=False, reminder=True). Tests
whether reminder injection moves NO_EVIDENCE_POINTER rate.
Strongest single-knob candidate per §12.4.
- [ ] **Cap only** (apply_caps=True, reminder=False). Tests whether
cap-application alone shifts violation distribution. Expected
to move TOO_MANY_CLAIMS rate but not STRICT-rate.
- [ ] **Cap + reminder** (apply_caps=True, reminder=True). Tests
combined effect. Defaults flip on if and only if this beats
reminder-only by ≥5pp.
- [x] **Reminder only** (apply_caps=False, reminder=True) — §12.6.
- [ ] **Cap only** (apply_caps=True, reminder=False).
- [ ] **Cap + reminder** (apply_caps=True, reminder=True).
5pp signal floor per `docs/bench-maxing.md` for default-flip
decisions. Each cycle adds 81 runs at ~12-17min on Hermes.
### 12.6 Reminder-only A/B (2026-05-03T12-38-53Z)
Same 9-question broad subset (`bench/qa_questions_quantifier_subset.txt`),
3 modes × n=3 = 81 runs. Policy override:
`--policy quantifier_reminder_enabled=true`. Everything else
defaulted (apply_caps=False, reject_broad=False).
Comparison against §12.2 baseline (reminder=False):
| Mode | Strict-rate | Mean ratio | UNGROUNDED count |
|-------------------------|----------------|-------------------|------------------|
| `quote` | 0.56 → 0.52 | 0.900 → 0.845 | 0 → 0 |
| `claim_lattice_pointer` | 0.00 → 0.00 | 0.473 → **0.643** (+17pp) | 9 → 6 (3) |
| `claim_lattice` (JSON) | 0.19 → 0.22 | 0.524 → **0.735** (+21pp) | 7 → 1 (**22pp**) |
**Pointer-mode violation kind shifts:**
| Kind | OFF | ON | Δ |
|-----------------------|-----|-----|----------|
| **FORMAT_COLLAPSED** | 2 | 0 | 100% |
| **NO_EVIDENCE_POINTER** | 9 | 6 | 33% |
| TITLE_MISMATCH | 10 | 15 | +50% |
| TOO_MANY_CLAIMS | 7 | 8 | +14% |
| CITATION_MISMATCH | 14 | 14 | 0 |
| WARRANT_MISSING | 4 | 3 | 25% |
| LAZY_ANCHOR_DEMOTE | 3 | 3 | 0 |
| POINTER_OVERFLOW_TRIMMED | 3 | 3 | 0 |
| SCHEMA_INVALID | 4 | 3 | 25% |
**§10.8 gate verdict — MET.**
The §10.8 rule is *"If reminder injection reduces FORMAT_COLLAPSED
OR pointer-loss rate by ≥5pp: enable Option B (reminder default-on)."*
Both criteria cleared:
- **FORMAT_COLLAPSED**: 2/27 → 0/27 (Δ ≈ 7pp absolute, 100% relative).
- **NO_EVIDENCE_POINTER (pointer-loss)**: 9/27 → 6/27
(Δ ≈ 11pp absolute, 33% relative).
Strongest signals are NOT in the headline strict-rate column.
They're in:
- **Mean-ratio jumped on both lattice modes**: pointer +17pp,
JSON +21pp. Grounded rows are MORE thoroughly grounded under
reminder.
- **JSON mode UNGROUNDED collapsed 7 → 1**: 22pp reduction. The
HYBRID pool grew (15 → 20) and one row escaped to STRICT.
Operator-visible "didn't ground" reclassified as "partially
grounded".
- **FORMAT_COLLAPSED elimination**: the reminder explicitly
restated the [E\d+] citation rule, and Hermes followed it.
**Caveats:**
- n=3 × 9 questions = 27 per mode is small. Variance is real;
Hermes nondeterminism dominates at this sample size.
- TITLE_MISMATCH went UP (10 → 15). The reminder may pressure
Hermes to cite *something* rather than say "no evidence",
picking up wrong-source citations as a side effect. Trade-off
visible: NO_EVIDENCE_POINTER ↓ but TITLE_MISMATCH ↑.
- Quote-mode strict-rate dipped 4pp (0.56 → 0.52). The reminder
shouldn't fire on quote (mode-gated via
`quantifier_guard_modes`), so this is likely Hermes
nondeterminism — but worth verifying with tighter n.
**Recommendation:**
The §10.8 gate is met. Reminder default-on for lattice modes is
supported by the data. **But hold the default flip until §12.7
cap-only and §12.8 cap+reminder cells run** — per §10.8 we need
to know whether cap+reminder beats reminder-only by ≥5pp before
defaulting to D vs B.