ticket(#000008,#000006,index): §12.6 reminder A/B verdict + cross-refs
§12.6 in #000008 captures the 2026-05-03T12-38-53Z reminder-only
A/B (apply_caps=False, reminder=True) on the same 9-question
broad subset:
Mode | Strict-rate | Mean ratio | UNGROUNDED
----------+----------------+-------------------+-----------
quote | 0.56 → 0.52 | 0.900 → 0.845 | 0 → 0
pointer | 0.00 → 0.00 | 0.473 → 0.643 | 9 → 6
JSON | 0.19 → 0.22 | 0.524 → 0.735 | 7 → 1
Pointer-mode violation deltas:
FORMAT_COLLAPSED 2 → 0 (-100%)
NO_EVIDENCE_POINTER 9 → 6 ( -33%)
TITLE_MISMATCH 10 → 15 ( +50%) ← side effect
TOO_MANY_CLAIMS 7 → 8 ( +14%)
§10.8 gate verdict: MET. Both FORMAT_COLLAPSED and NO_EVIDENCE_
POINTER cleared the 5pp floor (−7pp absolute / −11pp absolute
respectively). Strongest signals are mean-ratio improvements
(+17pp pointer, +21pp JSON) — grounded rows ground BETTER under
reminder. JSON-mode UNGROUNDED dropped 7 → 1, a 22pp redistribution
from "didn't ground" to "partially grounded".
Caveat: TITLE_MISMATCH increased (+50%). Reminder may pressure
Hermes to cite *something* rather than say "no evidence", picking
up wrong-source citations as a side effect.
Recommendation: §10.8 gate met but hold default flip until §12.7
(cap-only) and §12.8 (cap+reminder) cells run, per §10.8 "if A+B
together outperform either alone by ≥5pp: land Option D".
Cross-references:
- TICKETS.md index: #000008 status flipped to "open · phases 0–4
landed; bench A/B in progress".
- #000006 rolling log: cross-reference to #000008's bench cycles
+ the d24291b classifier-defect fix surfaced from the
distribution scan.
This commit is contained in:
parent
002f84c5a4
commit
fa8d93c8ef
3 changed files with 99 additions and 10 deletions
|
|
@ -57,7 +57,7 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000008 | Broad-quantifier preflight guard | open · awaiting go/no-go | 2026-05-02 | — |
|
||||
| #000008 | Broad-quantifier preflight guard | open · phases 0–4 landed; bench A/B in progress | 2026-05-02 | — |
|
||||
| #000007 | Query-layer hyphen folding | closed · 2026-05-02 | 2026-05-02 | — |
|
||||
| #000006 | Bench-emergent findings (rolling research log) | open · rolling | 2026-05-02 | — |
|
||||
| #000005 | Label ladder migration (POINTER-LINKED → …) | closed · 2026-05-02 | 2026-05-01 | D7 |
|
||||
|
|
|
|||
|
|
@ -381,3 +381,24 @@ Open · rolling research log. Three original tuning candidates:
|
|||
sidecar tightened); (2) `xxviii` STRICT acceptance — kept as
|
||||
designed; (3) metaphor sidecar calibration — deferred until
|
||||
sample size grows. Re-amend as findings accumulate.
|
||||
|
||||
## Cross-reference: ticket #000008 bench cycles (2026-05-03)
|
||||
|
||||
Ticket #000008 (broad-quantifier preflight guard) landed Phases
|
||||
0–4 on 2026-05-03 and now runs its own bench A/B cycles per its
|
||||
§12 with the `--policy KEY=VALUE` harness flag (commit `002f84c`).
|
||||
Findings relevant to the rolling-emergent stream:
|
||||
|
||||
- The `bench/qa_questions.txt` 73-question set carries 7 broad
|
||||
questions (~10%) classified ALL/COMPREHENSIVE/OPEN_REQUEST by
|
||||
the §10 classifier. Two bounded-universal fixtures added 2026-
|
||||
05-03 (`name all members of the beatles`, `list all planets in
|
||||
the solar system`).
|
||||
- Classifier defect caught + fixed in `d24291b`: `how many X?`
|
||||
was mis-classifying as MANY (4 false positives across the
|
||||
bench). Fixed via leading-anchor count-question short-circuit.
|
||||
- `metaphor_deflection` calibration (item 3 above) can leverage
|
||||
#000008's larger bench cycles — the broad-subset A/B at n=3 ×
|
||||
3 modes accumulates 81 rows per cell, useful sample size growth
|
||||
for metaphor-sidecar tuning if the metaphor cue happens to fire
|
||||
on broad-quantifier shapes.
|
||||
|
|
|
|||
|
|
@ -1890,15 +1890,83 @@ Empirically:
|
|||
|
||||
Per §10.8 decision tree, A/B sequence with this same broad subset:
|
||||
|
||||
- [ ] **Reminder only** (apply_caps=False, reminder=True). Tests
|
||||
whether reminder injection moves NO_EVIDENCE_POINTER rate.
|
||||
Strongest single-knob candidate per §12.4.
|
||||
- [ ] **Cap only** (apply_caps=True, reminder=False). Tests whether
|
||||
cap-application alone shifts violation distribution. Expected
|
||||
to move TOO_MANY_CLAIMS rate but not STRICT-rate.
|
||||
- [ ] **Cap + reminder** (apply_caps=True, reminder=True). Tests
|
||||
combined effect. Defaults flip on if and only if this beats
|
||||
reminder-only by ≥5pp.
|
||||
- [x] **Reminder only** (apply_caps=False, reminder=True) — §12.6.
|
||||
- [ ] **Cap only** (apply_caps=True, reminder=False).
|
||||
- [ ] **Cap + reminder** (apply_caps=True, reminder=True).
|
||||
|
||||
5pp signal floor per `docs/bench-maxing.md` for default-flip
|
||||
decisions. Each cycle adds 81 runs at ~12-17min on Hermes.
|
||||
|
||||
### 12.6 Reminder-only A/B (2026-05-03T12-38-53Z)
|
||||
|
||||
Same 9-question broad subset (`bench/qa_questions_quantifier_subset.txt`),
|
||||
3 modes × n=3 = 81 runs. Policy override:
|
||||
`--policy quantifier_reminder_enabled=true`. Everything else
|
||||
defaulted (apply_caps=False, reject_broad=False).
|
||||
|
||||
Comparison against §12.2 baseline (reminder=False):
|
||||
|
||||
| Mode | Strict-rate | Mean ratio | UNGROUNDED count |
|
||||
|-------------------------|----------------|-------------------|------------------|
|
||||
| `quote` | 0.56 → 0.52 | 0.900 → 0.845 | 0 → 0 |
|
||||
| `claim_lattice_pointer` | 0.00 → 0.00 | 0.473 → **0.643** (+17pp) | 9 → 6 (−3) |
|
||||
| `claim_lattice` (JSON) | 0.19 → 0.22 | 0.524 → **0.735** (+21pp) | 7 → 1 (**−22pp**) |
|
||||
|
||||
**Pointer-mode violation kind shifts:**
|
||||
|
||||
| Kind | OFF | ON | Δ |
|
||||
|-----------------------|-----|-----|----------|
|
||||
| **FORMAT_COLLAPSED** | 2 | 0 | −100% |
|
||||
| **NO_EVIDENCE_POINTER** | 9 | 6 | −33% |
|
||||
| TITLE_MISMATCH | 10 | 15 | +50% |
|
||||
| TOO_MANY_CLAIMS | 7 | 8 | +14% |
|
||||
| CITATION_MISMATCH | 14 | 14 | 0 |
|
||||
| WARRANT_MISSING | 4 | 3 | −25% |
|
||||
| LAZY_ANCHOR_DEMOTE | 3 | 3 | 0 |
|
||||
| POINTER_OVERFLOW_TRIMMED | 3 | 3 | 0 |
|
||||
| SCHEMA_INVALID | 4 | 3 | −25% |
|
||||
|
||||
**§10.8 gate verdict — MET.**
|
||||
|
||||
The §10.8 rule is *"If reminder injection reduces FORMAT_COLLAPSED
|
||||
OR pointer-loss rate by ≥5pp: enable Option B (reminder default-on)."*
|
||||
|
||||
Both criteria cleared:
|
||||
|
||||
- **FORMAT_COLLAPSED**: 2/27 → 0/27 (Δ ≈ −7pp absolute, 100% relative).
|
||||
- **NO_EVIDENCE_POINTER (pointer-loss)**: 9/27 → 6/27
|
||||
(Δ ≈ −11pp absolute, 33% relative).
|
||||
|
||||
Strongest signals are NOT in the headline strict-rate column.
|
||||
They're in:
|
||||
|
||||
- **Mean-ratio jumped on both lattice modes**: pointer +17pp,
|
||||
JSON +21pp. Grounded rows are MORE thoroughly grounded under
|
||||
reminder.
|
||||
- **JSON mode UNGROUNDED collapsed 7 → 1**: 22pp reduction. The
|
||||
HYBRID pool grew (15 → 20) and one row escaped to STRICT.
|
||||
Operator-visible "didn't ground" reclassified as "partially
|
||||
grounded".
|
||||
- **FORMAT_COLLAPSED elimination**: the reminder explicitly
|
||||
restated the [E\d+] citation rule, and Hermes followed it.
|
||||
|
||||
**Caveats:**
|
||||
|
||||
- n=3 × 9 questions = 27 per mode is small. Variance is real;
|
||||
Hermes nondeterminism dominates at this sample size.
|
||||
- TITLE_MISMATCH went UP (10 → 15). The reminder may pressure
|
||||
Hermes to cite *something* rather than say "no evidence",
|
||||
picking up wrong-source citations as a side effect. Trade-off
|
||||
visible: NO_EVIDENCE_POINTER ↓ but TITLE_MISMATCH ↑.
|
||||
- Quote-mode strict-rate dipped 4pp (0.56 → 0.52). The reminder
|
||||
shouldn't fire on quote (mode-gated via
|
||||
`quantifier_guard_modes`), so this is likely Hermes
|
||||
nondeterminism — but worth verifying with tighter n.
|
||||
|
||||
**Recommendation:**
|
||||
|
||||
The §10.8 gate is met. Reminder default-on for lattice modes is
|
||||
supported by the data. **But hold the default flip until §12.7
|
||||
cap-only and §12.8 cap+reminder cells run** — per §10.8 we need
|
||||
to know whether cap+reminder beats reminder-only by ≥5pp before
|
||||
defaulting to D vs B.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue