ticket(#000008,#000006,index): §12.6 reminder A/B verdict + cross-refs
§12.6 in #000008 captures the 2026-05-03T12-38-53Z reminder-only
A/B (apply_caps=False, reminder=True) on the same 9-question
broad subset:
Mode | Strict-rate | Mean ratio | UNGROUNDED
----------+----------------+-------------------+-----------
quote | 0.56 → 0.52 | 0.900 → 0.845 | 0 → 0
pointer | 0.00 → 0.00 | 0.473 → 0.643 | 9 → 6
JSON | 0.19 → 0.22 | 0.524 → 0.735 | 7 → 1
Pointer-mode violation deltas:
FORMAT_COLLAPSED 2 → 0 (-100%)
NO_EVIDENCE_POINTER 9 → 6 ( -33%)
TITLE_MISMATCH 10 → 15 ( +50%) ← side effect
TOO_MANY_CLAIMS 7 → 8 ( +14%)
§10.8 gate verdict: MET. Both FORMAT_COLLAPSED and NO_EVIDENCE_
POINTER cleared the 5pp floor (−7pp absolute / −11pp absolute
respectively). Strongest signals are mean-ratio improvements
(+17pp pointer, +21pp JSON) — grounded rows ground BETTER under
reminder. JSON-mode UNGROUNDED dropped 7 → 1, a 22pp redistribution
from "didn't ground" to "partially grounded".
Caveat: TITLE_MISMATCH increased (+50%). Reminder may pressure
Hermes to cite *something* rather than say "no evidence", picking
up wrong-source citations as a side effect.
Recommendation: §10.8 gate met but hold default flip until §12.7
(cap-only) and §12.8 (cap+reminder) cells run, per §10.8 "if A+B
together outperform either alone by ≥5pp: land Option D".
Cross-references:
- TICKETS.md index: #000008 status flipped to "open · phases 0–4
landed; bench A/B in progress".
- #000006 rolling log: cross-reference to #000008's bench cycles
+ the d24291b classifier-defect fix surfaced from the
distribution scan.
This commit is contained in:
parent
002f84c5a4
commit
fa8d93c8ef
3 changed files with 99 additions and 10 deletions
|
|
@ -57,7 +57,7 @@ Newest first. Update on every open/close.
|
||||||
|
|
||||||
| ID | Title | Status | Opened | Directive |
|
| ID | Title | Status | Opened | Directive |
|
||||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||||
| #000008 | Broad-quantifier preflight guard | open · awaiting go/no-go | 2026-05-02 | — |
|
| #000008 | Broad-quantifier preflight guard | open · phases 0–4 landed; bench A/B in progress | 2026-05-02 | — |
|
||||||
| #000007 | Query-layer hyphen folding | closed · 2026-05-02 | 2026-05-02 | — |
|
| #000007 | Query-layer hyphen folding | closed · 2026-05-02 | 2026-05-02 | — |
|
||||||
| #000006 | Bench-emergent findings (rolling research log) | open · rolling | 2026-05-02 | — |
|
| #000006 | Bench-emergent findings (rolling research log) | open · rolling | 2026-05-02 | — |
|
||||||
| #000005 | Label ladder migration (POINTER-LINKED → …) | closed · 2026-05-02 | 2026-05-01 | D7 |
|
| #000005 | Label ladder migration (POINTER-LINKED → …) | closed · 2026-05-02 | 2026-05-01 | D7 |
|
||||||
|
|
|
||||||
|
|
@ -381,3 +381,24 @@ Open · rolling research log. Three original tuning candidates:
|
||||||
sidecar tightened); (2) `xxviii` STRICT acceptance — kept as
|
sidecar tightened); (2) `xxviii` STRICT acceptance — kept as
|
||||||
designed; (3) metaphor sidecar calibration — deferred until
|
designed; (3) metaphor sidecar calibration — deferred until
|
||||||
sample size grows. Re-amend as findings accumulate.
|
sample size grows. Re-amend as findings accumulate.
|
||||||
|
|
||||||
|
## Cross-reference: ticket #000008 bench cycles (2026-05-03)
|
||||||
|
|
||||||
|
Ticket #000008 (broad-quantifier preflight guard) landed Phases
|
||||||
|
0–4 on 2026-05-03 and now runs its own bench A/B cycles per its
|
||||||
|
§12 with the `--policy KEY=VALUE` harness flag (commit `002f84c`).
|
||||||
|
Findings relevant to the rolling-emergent stream:
|
||||||
|
|
||||||
|
- The `bench/qa_questions.txt` 73-question set carries 7 broad
|
||||||
|
questions (~10%) classified ALL/COMPREHENSIVE/OPEN_REQUEST by
|
||||||
|
the §10 classifier. Two bounded-universal fixtures added 2026-
|
||||||
|
05-03 (`name all members of the beatles`, `list all planets in
|
||||||
|
the solar system`).
|
||||||
|
- Classifier defect caught + fixed in `d24291b`: `how many X?`
|
||||||
|
was mis-classifying as MANY (4 false positives across the
|
||||||
|
bench). Fixed via leading-anchor count-question short-circuit.
|
||||||
|
- `metaphor_deflection` calibration (item 3 above) can leverage
|
||||||
|
#000008's larger bench cycles — the broad-subset A/B at n=3 ×
|
||||||
|
3 modes accumulates 81 rows per cell, useful sample size growth
|
||||||
|
for metaphor-sidecar tuning if the metaphor cue happens to fire
|
||||||
|
on broad-quantifier shapes.
|
||||||
|
|
|
||||||
|
|
@ -1890,15 +1890,83 @@ Empirically:
|
||||||
|
|
||||||
Per §10.8 decision tree, A/B sequence with this same broad subset:
|
Per §10.8 decision tree, A/B sequence with this same broad subset:
|
||||||
|
|
||||||
- [ ] **Reminder only** (apply_caps=False, reminder=True). Tests
|
- [x] **Reminder only** (apply_caps=False, reminder=True) — §12.6.
|
||||||
whether reminder injection moves NO_EVIDENCE_POINTER rate.
|
- [ ] **Cap only** (apply_caps=True, reminder=False).
|
||||||
Strongest single-knob candidate per §12.4.
|
- [ ] **Cap + reminder** (apply_caps=True, reminder=True).
|
||||||
- [ ] **Cap only** (apply_caps=True, reminder=False). Tests whether
|
|
||||||
cap-application alone shifts violation distribution. Expected
|
|
||||||
to move TOO_MANY_CLAIMS rate but not STRICT-rate.
|
|
||||||
- [ ] **Cap + reminder** (apply_caps=True, reminder=True). Tests
|
|
||||||
combined effect. Defaults flip on if and only if this beats
|
|
||||||
reminder-only by ≥5pp.
|
|
||||||
|
|
||||||
5pp signal floor per `docs/bench-maxing.md` for default-flip
|
5pp signal floor per `docs/bench-maxing.md` for default-flip
|
||||||
decisions. Each cycle adds 81 runs at ~12-17min on Hermes.
|
decisions. Each cycle adds 81 runs at ~12-17min on Hermes.
|
||||||
|
|
||||||
|
### 12.6 Reminder-only A/B (2026-05-03T12-38-53Z)
|
||||||
|
|
||||||
|
Same 9-question broad subset (`bench/qa_questions_quantifier_subset.txt`),
|
||||||
|
3 modes × n=3 = 81 runs. Policy override:
|
||||||
|
`--policy quantifier_reminder_enabled=true`. Everything else
|
||||||
|
defaulted (apply_caps=False, reject_broad=False).
|
||||||
|
|
||||||
|
Comparison against §12.2 baseline (reminder=False):
|
||||||
|
|
||||||
|
| Mode | Strict-rate | Mean ratio | UNGROUNDED count |
|
||||||
|
|-------------------------|----------------|-------------------|------------------|
|
||||||
|
| `quote` | 0.56 → 0.52 | 0.900 → 0.845 | 0 → 0 |
|
||||||
|
| `claim_lattice_pointer` | 0.00 → 0.00 | 0.473 → **0.643** (+17pp) | 9 → 6 (−3) |
|
||||||
|
| `claim_lattice` (JSON) | 0.19 → 0.22 | 0.524 → **0.735** (+21pp) | 7 → 1 (**−22pp**) |
|
||||||
|
|
||||||
|
**Pointer-mode violation kind shifts:**
|
||||||
|
|
||||||
|
| Kind | OFF | ON | Δ |
|
||||||
|
|-----------------------|-----|-----|----------|
|
||||||
|
| **FORMAT_COLLAPSED** | 2 | 0 | −100% |
|
||||||
|
| **NO_EVIDENCE_POINTER** | 9 | 6 | −33% |
|
||||||
|
| TITLE_MISMATCH | 10 | 15 | +50% |
|
||||||
|
| TOO_MANY_CLAIMS | 7 | 8 | +14% |
|
||||||
|
| CITATION_MISMATCH | 14 | 14 | 0 |
|
||||||
|
| WARRANT_MISSING | 4 | 3 | −25% |
|
||||||
|
| LAZY_ANCHOR_DEMOTE | 3 | 3 | 0 |
|
||||||
|
| POINTER_OVERFLOW_TRIMMED | 3 | 3 | 0 |
|
||||||
|
| SCHEMA_INVALID | 4 | 3 | −25% |
|
||||||
|
|
||||||
|
**§10.8 gate verdict — MET.**
|
||||||
|
|
||||||
|
The §10.8 rule is *"If reminder injection reduces FORMAT_COLLAPSED
|
||||||
|
OR pointer-loss rate by ≥5pp: enable Option B (reminder default-on)."*
|
||||||
|
|
||||||
|
Both criteria cleared:
|
||||||
|
|
||||||
|
- **FORMAT_COLLAPSED**: 2/27 → 0/27 (Δ ≈ −7pp absolute, 100% relative).
|
||||||
|
- **NO_EVIDENCE_POINTER (pointer-loss)**: 9/27 → 6/27
|
||||||
|
(Δ ≈ −11pp absolute, 33% relative).
|
||||||
|
|
||||||
|
Strongest signals are NOT in the headline strict-rate column.
|
||||||
|
They're in:
|
||||||
|
|
||||||
|
- **Mean-ratio jumped on both lattice modes**: pointer +17pp,
|
||||||
|
JSON +21pp. Grounded rows are MORE thoroughly grounded under
|
||||||
|
reminder.
|
||||||
|
- **JSON mode UNGROUNDED collapsed 7 → 1**: 22pp reduction. The
|
||||||
|
HYBRID pool grew (15 → 20) and one row escaped to STRICT.
|
||||||
|
Operator-visible "didn't ground" reclassified as "partially
|
||||||
|
grounded".
|
||||||
|
- **FORMAT_COLLAPSED elimination**: the reminder explicitly
|
||||||
|
restated the [E\d+] citation rule, and Hermes followed it.
|
||||||
|
|
||||||
|
**Caveats:**
|
||||||
|
|
||||||
|
- n=3 × 9 questions = 27 per mode is small. Variance is real;
|
||||||
|
Hermes nondeterminism dominates at this sample size.
|
||||||
|
- TITLE_MISMATCH went UP (10 → 15). The reminder may pressure
|
||||||
|
Hermes to cite *something* rather than say "no evidence",
|
||||||
|
picking up wrong-source citations as a side effect. Trade-off
|
||||||
|
visible: NO_EVIDENCE_POINTER ↓ but TITLE_MISMATCH ↑.
|
||||||
|
- Quote-mode strict-rate dipped 4pp (0.56 → 0.52). The reminder
|
||||||
|
shouldn't fire on quote (mode-gated via
|
||||||
|
`quantifier_guard_modes`), so this is likely Hermes
|
||||||
|
nondeterminism — but worth verifying with tighter n.
|
||||||
|
|
||||||
|
**Recommendation:**
|
||||||
|
|
||||||
|
The §10.8 gate is met. Reminder default-on for lattice modes is
|
||||||
|
supported by the data. **But hold the default flip until §12.7
|
||||||
|
cap-only and §12.8 cap+reminder cells run** — per §10.8 we need
|
||||||
|
to know whether cap+reminder beats reminder-only by ≥5pp before
|
||||||
|
defaulting to D vs B.
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue