ticket(#000008): §12.7 cap-only A/B — cap and reminder help differently
§12.7 captures the 2026-05-03T12-47-23Z cap-only A/B (apply_caps= True, reminder=False) on the 9-question broad subset. Mode | Baseline | Reminder | Cap-only ----------+----------+----------+---------- JSON SR | 0.19 | 0.22 | 0.33 ← +14pp JSON U | 7 | 1 | 6 ← reminder dominates JSON S | 5 | 6 | 9 ← cap dominates ptr ratio | 0.473 | 0.643 | 0.516 JSON ratio| 0.524 | 0.735 | 0.643 §10.8 gate MET on JSON mode (+14pp STRICT-rate). Headline insight: cap and reminder help in DIFFERENT ways. - Reminder rescues UNGROUNDED → HYBRID (restates citation rule). - Cap rescues HYBRID → STRICT (forces fewer-but-better claims). The two are complementary, not redundant. If §12.8 (cap+reminder) confirms the compound effect, that's the §10.8 trigger to land Option D as default. Predicted JSON SR ~0.40 if effects compound. Pointer mode still 0/27 STRICT under cap-only — TOO_MANY_CLAIMS fires 20× (vs 7× baseline) but pointer-tag failures upstream of the cap still gate the verdict.
This commit is contained in:
parent
fa8d93c8ef
commit
17c1cde16d
1 changed files with 75 additions and 1 deletions
|
|
@ -1891,7 +1891,7 @@ Empirically:
|
|||
Per §10.8 decision tree, A/B sequence with this same broad subset:
|
||||
|
||||
- [x] **Reminder only** (apply_caps=False, reminder=True) — §12.6.
|
||||
- [ ] **Cap only** (apply_caps=True, reminder=False).
|
||||
- [x] **Cap only** (apply_caps=True, reminder=False) — §12.7.
|
||||
- [ ] **Cap + reminder** (apply_caps=True, reminder=True).
|
||||
|
||||
5pp signal floor per `docs/bench-maxing.md` for default-flip
|
||||
|
|
@ -1970,3 +1970,77 @@ supported by the data. **But hold the default flip until §12.7
|
|||
cap-only and §12.8 cap+reminder cells run** — per §10.8 we need
|
||||
to know whether cap+reminder beats reminder-only by ≥5pp before
|
||||
defaulting to D vs B.
|
||||
|
||||
### 12.7 Cap-only A/B (2026-05-03T12-47-23Z)
|
||||
|
||||
Same 9-question broad subset. Policy override:
|
||||
`--policy quantifier_guard_apply_caps=true`. Reminder=False.
|
||||
|
||||
Three-way comparison against §12.2 baseline and §12.6 reminder-only:
|
||||
|
||||
| Mode | Baseline | Reminder-only | Cap-only |
|
||||
|-------------------------|----------------|----------------|---------------|
|
||||
| `quote` strict-rate | 0.56 | 0.52 | 0.59 (noise) |
|
||||
| `pointer` strict-rate | 0.00 | 0.00 | 0.00 |
|
||||
| **`JSON` strict-rate** | **0.19** | **0.22** | **0.33** |
|
||||
| `pointer` mean ratio | 0.473 | **0.643** | 0.516 |
|
||||
| `JSON` mean ratio | 0.524 | **0.735** | 0.643 |
|
||||
|
||||
**JSON-mode verdict distribution:**
|
||||
|
||||
| Cell | STRICT | HYBRID | UNGROUNDED |
|
||||
|----------------|--------|--------|------------|
|
||||
| Baseline | 5 | 15 | 7 |
|
||||
| Reminder-only | 6 | 20 | 1 |
|
||||
| **Cap-only** | **9** | 12 | 6 |
|
||||
|
||||
**Headline: cap and reminder help in DIFFERENT ways.**
|
||||
|
||||
- **Reminder rescues UNGROUNDED → HYBRID.** JSON UNGROUNDED
|
||||
collapsed 7 → 1 (−22pp). Restating the citation rule keeps the
|
||||
model from giving up.
|
||||
- **Cap rescues HYBRID → STRICT.** JSON STRICT jumped 5 → 9
|
||||
(+14pp). Forcing fewer-but-better claims raises the per-claim
|
||||
grounding bar — the model can fully ground 5 claims it couldn't
|
||||
fully ground at 12.
|
||||
|
||||
**§10.8 gate verdict — MET on JSON mode.**
|
||||
|
||||
The §10.8 rule for cap-only is implicit ("Option A"): if cap
|
||||
reduction shifts STRICT/HYBRID/UNGROUNDED distribution by ≥5pp,
|
||||
enable Option A.
|
||||
|
||||
JSON mode +14pp on STRICT-rate alone clears the floor. Pointer
|
||||
mode unchanged at 0/27 (the cap forces TOO_MANY_CLAIMS to fire 20
|
||||
times instead of 7, but pointer-tag failures upstream of the cap
|
||||
still gate the verdict).
|
||||
|
||||
**Pointer-mode violation deltas (cap-only vs baseline):**
|
||||
|
||||
| Kind | Baseline | Cap-only | Δ |
|
||||
|-----------------------|----------|----------|----------|
|
||||
| **TOO_MANY_CLAIMS** | 7 | **20** | +186% |
|
||||
| TITLE_MISMATCH | 10 | 13 | +30% |
|
||||
| NO_EVIDENCE_POINTER | 9 | 7 | −22% |
|
||||
| CITATION_MISMATCH | 14 | 15 | +7% |
|
||||
| FORMAT_COLLAPSED | 2 | 2 | 0 |
|
||||
| WARRANT_MISSING | 4 | 4 | 0 |
|
||||
|
||||
Cap firing 20/27 in pointer mode is exactly what we'd expect when
|
||||
the cap drops from 12 to 5/8 per intensity. The cap demotes STRICT
|
||||
→ HYBRID via TOO_MANY_CLAIMS but doesn't fix pointer-tag discipline.
|
||||
|
||||
**Implication for cap+reminder (§12.8):**
|
||||
|
||||
The two interventions are **complementary, not redundant**. The
|
||||
reminder reduces the UNGROUNDED pool by improving format
|
||||
discipline; the cap promotes HYBRID to STRICT by tightening
|
||||
per-claim grounding requirements. If §12.8 confirms the compound
|
||||
effect, that's the §10.8 trigger to land Option D as default.
|
||||
|
||||
Predicted §12.8 outcome (n=3, 9 questions):
|
||||
|
||||
- JSON STRICT-rate: 9 + N more → 0.33 + ~5pp = ~0.40 (if effects compound)
|
||||
- JSON UNGROUNDED: 1 (already low; reminder dominates this metric)
|
||||
- Pointer STRICT-rate: still 0/27 (neither knob fixes pointer-tag
|
||||
discipline at this sample size)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue