diff --git a/docs/TICKETS.md b/docs/TICKETS.md index 451f768..0c4cfbb 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -57,7 +57,7 @@ Newest first. Update on every open/close. | ID | Title | Status | Opened | Directive | |----------|------------------------------------------------|-----------------------|------------|-----------| -| #000008 | Broad-quantifier preflight guard | open · awaiting go/no-go | 2026-05-02 | — | +| #000008 | Broad-quantifier preflight guard | open · phases 0–4 landed; bench A/B in progress | 2026-05-02 | — | | #000007 | Query-layer hyphen folding | closed · 2026-05-02 | 2026-05-02 | — | | #000006 | Bench-emergent findings (rolling research log) | open · rolling | 2026-05-02 | — | | #000005 | Label ladder migration (POINTER-LINKED → …) | closed · 2026-05-02 | 2026-05-01 | D7 | diff --git a/docs/tickets/ticket-000006-bench-emergent-findings.md b/docs/tickets/ticket-000006-bench-emergent-findings.md index 40a7c32..b2e88b3 100644 --- a/docs/tickets/ticket-000006-bench-emergent-findings.md +++ b/docs/tickets/ticket-000006-bench-emergent-findings.md @@ -381,3 +381,24 @@ Open · rolling research log. Three original tuning candidates: sidecar tightened); (2) `xxviii` STRICT acceptance — kept as designed; (3) metaphor sidecar calibration — deferred until sample size grows. Re-amend as findings accumulate. + +## Cross-reference: ticket #000008 bench cycles (2026-05-03) + +Ticket #000008 (broad-quantifier preflight guard) landed Phases +0–4 on 2026-05-03 and now runs its own bench A/B cycles per its +§12 with the `--policy KEY=VALUE` harness flag (commit `002f84c`). +Findings relevant to the rolling-emergent stream: + +- The `bench/qa_questions.txt` 73-question set carries 7 broad + questions (~10%) classified ALL/COMPREHENSIVE/OPEN_REQUEST by + the §10 classifier. Two bounded-universal fixtures added 2026- + 05-03 (`name all members of the beatles`, `list all planets in + the solar system`). +- Classifier defect caught + fixed in `d24291b`: `how many X?` + was mis-classifying as MANY (4 false positives across the + bench). Fixed via leading-anchor count-question short-circuit. +- `metaphor_deflection` calibration (item 3 above) can leverage + #000008's larger bench cycles — the broad-subset A/B at n=3 × + 3 modes accumulates 81 rows per cell, useful sample size growth + for metaphor-sidecar tuning if the metaphor cue happens to fire + on broad-quantifier shapes. diff --git a/docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md b/docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md index b344f39..b8cb29d 100644 --- a/docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md +++ b/docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md @@ -1890,15 +1890,83 @@ Empirically: Per §10.8 decision tree, A/B sequence with this same broad subset: -- [ ] **Reminder only** (apply_caps=False, reminder=True). Tests - whether reminder injection moves NO_EVIDENCE_POINTER rate. - Strongest single-knob candidate per §12.4. -- [ ] **Cap only** (apply_caps=True, reminder=False). Tests whether - cap-application alone shifts violation distribution. Expected - to move TOO_MANY_CLAIMS rate but not STRICT-rate. -- [ ] **Cap + reminder** (apply_caps=True, reminder=True). Tests - combined effect. Defaults flip on if and only if this beats - reminder-only by ≥5pp. +- [x] **Reminder only** (apply_caps=False, reminder=True) — §12.6. +- [ ] **Cap only** (apply_caps=True, reminder=False). +- [ ] **Cap + reminder** (apply_caps=True, reminder=True). 5pp signal floor per `docs/bench-maxing.md` for default-flip decisions. Each cycle adds 81 runs at ~12-17min on Hermes. + +### 12.6 Reminder-only A/B (2026-05-03T12-38-53Z) + +Same 9-question broad subset (`bench/qa_questions_quantifier_subset.txt`), +3 modes × n=3 = 81 runs. Policy override: +`--policy quantifier_reminder_enabled=true`. Everything else +defaulted (apply_caps=False, reject_broad=False). + +Comparison against §12.2 baseline (reminder=False): + +| Mode | Strict-rate | Mean ratio | UNGROUNDED count | +|-------------------------|----------------|-------------------|------------------| +| `quote` | 0.56 → 0.52 | 0.900 → 0.845 | 0 → 0 | +| `claim_lattice_pointer` | 0.00 → 0.00 | 0.473 → **0.643** (+17pp) | 9 → 6 (−3) | +| `claim_lattice` (JSON) | 0.19 → 0.22 | 0.524 → **0.735** (+21pp) | 7 → 1 (**−22pp**) | + +**Pointer-mode violation kind shifts:** + +| Kind | OFF | ON | Δ | +|-----------------------|-----|-----|----------| +| **FORMAT_COLLAPSED** | 2 | 0 | −100% | +| **NO_EVIDENCE_POINTER** | 9 | 6 | −33% | +| TITLE_MISMATCH | 10 | 15 | +50% | +| TOO_MANY_CLAIMS | 7 | 8 | +14% | +| CITATION_MISMATCH | 14 | 14 | 0 | +| WARRANT_MISSING | 4 | 3 | −25% | +| LAZY_ANCHOR_DEMOTE | 3 | 3 | 0 | +| POINTER_OVERFLOW_TRIMMED | 3 | 3 | 0 | +| SCHEMA_INVALID | 4 | 3 | −25% | + +**§10.8 gate verdict — MET.** + +The §10.8 rule is *"If reminder injection reduces FORMAT_COLLAPSED +OR pointer-loss rate by ≥5pp: enable Option B (reminder default-on)."* + +Both criteria cleared: + +- **FORMAT_COLLAPSED**: 2/27 → 0/27 (Δ ≈ −7pp absolute, 100% relative). +- **NO_EVIDENCE_POINTER (pointer-loss)**: 9/27 → 6/27 + (Δ ≈ −11pp absolute, 33% relative). + +Strongest signals are NOT in the headline strict-rate column. +They're in: + +- **Mean-ratio jumped on both lattice modes**: pointer +17pp, + JSON +21pp. Grounded rows are MORE thoroughly grounded under + reminder. +- **JSON mode UNGROUNDED collapsed 7 → 1**: 22pp reduction. The + HYBRID pool grew (15 → 20) and one row escaped to STRICT. + Operator-visible "didn't ground" reclassified as "partially + grounded". +- **FORMAT_COLLAPSED elimination**: the reminder explicitly + restated the [E\d+] citation rule, and Hermes followed it. + +**Caveats:** + +- n=3 × 9 questions = 27 per mode is small. Variance is real; + Hermes nondeterminism dominates at this sample size. +- TITLE_MISMATCH went UP (10 → 15). The reminder may pressure + Hermes to cite *something* rather than say "no evidence", + picking up wrong-source citations as a side effect. Trade-off + visible: NO_EVIDENCE_POINTER ↓ but TITLE_MISMATCH ↑. +- Quote-mode strict-rate dipped 4pp (0.56 → 0.52). The reminder + shouldn't fire on quote (mode-gated via + `quantifier_guard_modes`), so this is likely Hermes + nondeterminism — but worth verifying with tighter n. + +**Recommendation:** + +The §10.8 gate is met. Reminder default-on for lattice modes is +supported by the data. **But hold the default flip until §12.7 +cap-only and §12.8 cap+reminder cells run** — per §10.8 we need +to know whether cap+reminder beats reminder-only by ≥5pp before +defaulting to D vs B.