ticket(#000008): §12.7 cap-only A/B — cap and reminder help differently

§12.7 captures the 2026-05-03T12-47-23Z cap-only A/B (apply_caps=
True, reminder=False) on the 9-question broad subset.

  Mode      | Baseline | Reminder | Cap-only
  ----------+----------+----------+----------
  JSON SR   |  0.19    |  0.22    |  0.33    ← +14pp
  JSON U    |   7      |   1      |   6      ← reminder dominates
  JSON S    |   5      |   6      |   9      ← cap dominates
  ptr ratio |  0.473   |  0.643   |  0.516
  JSON ratio|  0.524   |  0.735   |  0.643

§10.8 gate MET on JSON mode (+14pp STRICT-rate).

Headline insight: cap and reminder help in DIFFERENT ways.
- Reminder rescues UNGROUNDED → HYBRID (restates citation rule).
- Cap rescues HYBRID → STRICT (forces fewer-but-better claims).

The two are complementary, not redundant. If §12.8 (cap+reminder)
confirms the compound effect, that's the §10.8 trigger to land
Option D as default. Predicted JSON SR ~0.40 if effects compound.

Pointer mode still 0/27 STRICT under cap-only — TOO_MANY_CLAIMS
fires 20× (vs 7× baseline) but pointer-tag failures upstream of
the cap still gate the verdict.
This commit is contained in:
russell@unturf.com 2026-05-03 08:54:58 -04:00
parent fa8d93c8ef
commit 17c1cde16d
No known key found for this signature in database

View file

@ -1891,7 +1891,7 @@ Empirically:
Per §10.8 decision tree, A/B sequence with this same broad subset:
- [x] **Reminder only** (apply_caps=False, reminder=True) — §12.6.
- [ ] **Cap only** (apply_caps=True, reminder=False).
- [x] **Cap only** (apply_caps=True, reminder=False) — §12.7.
- [ ] **Cap + reminder** (apply_caps=True, reminder=True).
5pp signal floor per `docs/bench-maxing.md` for default-flip
@ -1970,3 +1970,77 @@ supported by the data. **But hold the default flip until §12.7
cap-only and §12.8 cap+reminder cells run** — per §10.8 we need
to know whether cap+reminder beats reminder-only by ≥5pp before
defaulting to D vs B.
### 12.7 Cap-only A/B (2026-05-03T12-47-23Z)
Same 9-question broad subset. Policy override:
`--policy quantifier_guard_apply_caps=true`. Reminder=False.
Three-way comparison against §12.2 baseline and §12.6 reminder-only:
| Mode | Baseline | Reminder-only | Cap-only |
|-------------------------|----------------|----------------|---------------|
| `quote` strict-rate | 0.56 | 0.52 | 0.59 (noise) |
| `pointer` strict-rate | 0.00 | 0.00 | 0.00 |
| **`JSON` strict-rate** | **0.19** | **0.22** | **0.33** |
| `pointer` mean ratio | 0.473 | **0.643** | 0.516 |
| `JSON` mean ratio | 0.524 | **0.735** | 0.643 |
**JSON-mode verdict distribution:**
| Cell | STRICT | HYBRID | UNGROUNDED |
|----------------|--------|--------|------------|
| Baseline | 5 | 15 | 7 |
| Reminder-only | 6 | 20 | 1 |
| **Cap-only** | **9** | 12 | 6 |
**Headline: cap and reminder help in DIFFERENT ways.**
- **Reminder rescues UNGROUNDED → HYBRID.** JSON UNGROUNDED
collapsed 7 → 1 (22pp). Restating the citation rule keeps the
model from giving up.
- **Cap rescues HYBRID → STRICT.** JSON STRICT jumped 5 → 9
(+14pp). Forcing fewer-but-better claims raises the per-claim
grounding bar — the model can fully ground 5 claims it couldn't
fully ground at 12.
**§10.8 gate verdict — MET on JSON mode.**
The §10.8 rule for cap-only is implicit ("Option A"): if cap
reduction shifts STRICT/HYBRID/UNGROUNDED distribution by ≥5pp,
enable Option A.
JSON mode +14pp on STRICT-rate alone clears the floor. Pointer
mode unchanged at 0/27 (the cap forces TOO_MANY_CLAIMS to fire 20
times instead of 7, but pointer-tag failures upstream of the cap
still gate the verdict).
**Pointer-mode violation deltas (cap-only vs baseline):**
| Kind | Baseline | Cap-only | Δ |
|-----------------------|----------|----------|----------|
| **TOO_MANY_CLAIMS** | 7 | **20** | +186% |
| TITLE_MISMATCH | 10 | 13 | +30% |
| NO_EVIDENCE_POINTER | 9 | 7 | 22% |
| CITATION_MISMATCH | 14 | 15 | +7% |
| FORMAT_COLLAPSED | 2 | 2 | 0 |
| WARRANT_MISSING | 4 | 4 | 0 |
Cap firing 20/27 in pointer mode is exactly what we'd expect when
the cap drops from 12 to 5/8 per intensity. The cap demotes STRICT
→ HYBRID via TOO_MANY_CLAIMS but doesn't fix pointer-tag discipline.
**Implication for cap+reminder (§12.8):**
The two interventions are **complementary, not redundant**. The
reminder reduces the UNGROUNDED pool by improving format
discipline; the cap promotes HYBRID to STRICT by tightening
per-claim grounding requirements. If §12.8 confirms the compound
effect, that's the §10.8 trigger to land Option D as default.
Predicted §12.8 outcome (n=3, 9 questions):
- JSON STRICT-rate: 9 + N more → 0.33 + ~5pp = ~0.40 (if effects compound)
- JSON UNGROUNDED: 1 (already low; reminder dominates this metric)
- Pointer STRICT-rate: still 0/27 (neither knob fixes pointer-tag
discipline at this sample size)