docs: §12.8/§12.9 cap+reminder verdict + cross-doc updates

#000008 §12.8 — Cap+reminder A/B (2026-05-03T12-54-11Z, 81 runs): Metric | Base | Rem | Cap | Cap+Rem --------------------+------+------+------+-------- JSON SR | 0.19 | 0.22 | 0.33 | 0.30 ← cap-only wins SR JSON UNGROUNDED | 7 | 1 | 6 | 1 ← rem dominates U-rescue pointer mean ratio | 0.473| 0.643| 0.516| 0.684 ← cap+rem best pointer FORMAT_COLL | 2 | 0 | 2 | 0 ← rem-driven §10.8 strict gate "compound beats either alone by ≥5pp on every metric" NOT cleanly met. Cap+reminder beats reminder-only by +8pp on JSON SR (clears floor) and beats cap-only by +17pp on pointer mean ratio (clears floor), but is -3pp vs cap-only on JSON SR (regression, within noise). #000008 §12.9 — Final verdict + recommendation: Mechanism asymmetry (clean signal): Reminder rescues UNGROUNDED → HYBRID Cap rescues HYBRID → STRICT Recommendation: Option A — single-knob defaults, per-mode tailored: claim_lattice (JSON): apply_caps=True + reminder=True claim_lattice_pointer: apply_caps=False + reminder=True (cap can't rescue pointer-tag discipline upstream of cap; cap fires 20× without verdict gain) quote: guard mode-gated off (already default) Caveat: n=3 × 9 = 27/cell variance is ~3-4pp; recommend n=5 verification on cap+reminder before flipping defaults. Cross-doc updates: - CLAUDE.md: architecture diagram now lists the three new Phase 1-3 modules (quantifier.py, model_profiles.py, quantifier_reminder.py). New "Broad-quantifier preflight guard" conventions entry covers the 7 policy fields, six-level disable hierarchy, dry-run defaults, CLI flags, and §12 bench summary. - docs/qa-modes-bench.md: addendum at end pointing at #000008 §12 four-cell A/B + 4 bench artifact stamps. Original 2026-05-02 journal frozen; 2026-05-03 broad-subset findings flagged as question-mix-dependent narrative on top of the global per-mode recommendation.
2026-05-03 15:45:06 -04:00 · 2026-05-03 15:45:06 -04:00 · 9780cca4d3
commit 9780cca4d3
parent 17c1cde16d
3 changed files with 168 additions and 1 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -58,6 +58,9 @@ aborist/
 │   ├── evidence.py  #   EvidenceObject + spotlight excerpt
 │   ├── inspect.py   #   read-only sidecars (deflection, title-relevance)
 │   ├── dag.py       #   per-run Merkle-DAG (7-stage quote / 9-stage CTI)
+│   ├── quantifier.py        # broad-quantifier classifier (#000008 P1)
+│   ├── model_profiles.py    # per-model claim-cap profiles (#000008 P2)
+│   ├── quantifier_reminder.py # broad-query reminder text (#000008 P3)
 │   └── runner.py    #   ask(): cache → infer → verify → write
 ├── wikitext.py      # to_base(): wikitext → plain prose (BASE_VERSION-pinned)
 └── cli.py           # ingest / search / verify / stats / distill /
@ -176,6 +179,28 @@ revert without reading why. When in doubt, walk the
  Catches "tell me all there is to know about X" runaway. Demotes
  STRICT → HYBRID without truncating. Folds into
  `governance_policy_hash`. See `aborist/qa/verify.py`.
+- **Broad-quantifier preflight guard (Ticket #000008)**: pure lexical
+  classifier (`aborist/qa/quantifier.py`) maps a question onto a
+  10-rung intensity ladder (ABSENT → SINGULAR → … → ALL →
+  COMPREHENSIVE). Per-model `aborist/qa/model_profiles.py` PROFILES
+  dict picks a per-call claim-cap from the (intensity, model) pair;
+  `aborist/qa/quantifier_reminder.py` synthesizes a one-line user-turn
+  reminder for broad questions. Six-level disable hierarchy
+  (per-test, per-call CLI, per-phase policy, per-mode, per-model,
+  master-via-governance-hash). 7 policy fields fold into
+  `governance_policy_hash` so flipping any of them invalidates prior
+  records. Defaults preserve dry-run discipline:
+  `quantifier_guard_apply_caps=False`,
+  `quantifier_reminder_enabled=False`,
+  `quantifier_reject_broad=False`. CLI flags on `aborist query`:
+  `--no-quantifier-guard`, `--allow-broad`, `--reject-broad`,
+  `--apply-quantifier-caps`. Bench A/B (2026-05-03, n=3 × 9 broad
+  questions × 3 modes): reminder default-on supported (FORMAT_COLLAPSED
+  −100%, NO_EVIDENCE_POINTER −33%, JSON UNGROUNDED −22pp); cap default-on
+  for JSON only (+14pp STRICT-rate, no gain on pointer); cap+reminder
+  best on pointer mean ratio (0.684) but not strictly best on JSON
+  STRICT-rate. See `docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md`
+  §12 for the four-cell A/B data.
 - **Wikitext base prose**: `aborist/wikitext.py:to_base()` runs
  before the LLM call AND inside `verify_quotes` so model and verifier
  see the same prose. Optional dep — graceful fallback when
--- a/docs/qa-modes-bench.md
+++ b/docs/qa-modes-bench.md
@ -173,3 +173,35 @@ Verified: `tell me about the roman empire` under `claim_lattice` now classifies
 - `bench/qa_results/2026-05-02T15-07-24Z.{jsonl,md}` — 639 rows

 (Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.)
+
+---
+
+## Addendum — 2026-05-03 broad-quantifier A/B (Ticket #000008)
+
+This journal froze the 2026-05-02 substrate baseline. On 2026-05-03
+Ticket #000008 (broad-quantifier preflight guard) landed Phases 0–4
+and ran a four-cell A/B on a 9-question broad subset
+(`bench/qa_questions_quantifier_subset.txt`, 81 runs per cell).
+
+Findings relevant to this journal's per-mode recommendations:
+
+- **JSON-mode strict-rate moves under cap-only**: 0.19 baseline →
+  0.33 with `quantifier_guard_apply_caps=true` (+14pp on the broad
+  subset). The recommendation here that "JSON leads among lattice
+  modes" still holds, but the *headroom above 0.42 globally* is
+  partly dependent on broad-vs-narrow question mix.
+- **Pointer-mode stays at 0/27 STRICT** on broad questions across
+  all four A/B cells. The 0.20 strict-rate above is *across the
+  full 71-question set*; the broad subset alone is structurally
+  unfavorable to pointer mode regardless of cap or reminder.
+- **Reminder eliminates FORMAT_COLLAPSED**: 2/27 → 0/27 on pointer
+  mode under `quantifier_reminder_enabled=true`.
+- **Cap and reminder help DIFFERENT failure modes**: reminder
+  rescues UNGROUNDED → HYBRID (restates citation rule); cap
+  rescues HYBRID → STRICT (forces fewer-but-better claims).
+  Compound effect on pointer mean-ratio is best at 0.684 (vs
+  0.473 baseline).
+
+See `docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md`
+§12 for the four-cell data and the §10.8 decision-tree verdict.
+Bench artifacts: `bench/qa_results/2026-05-03T12-{29-53,38-53,47-23,54-11}Z.{jsonl,md}`.
--- a/docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md
+++ b/docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md
@ -1892,7 +1892,8 @@ Per §10.8 decision tree, A/B sequence with this same broad subset:

 - [x] **Reminder only** (apply_caps=False, reminder=True) — §12.6.
 - [x] **Cap only** (apply_caps=True, reminder=False) — §12.7.
- [ ] **Cap + reminder** (apply_caps=True, reminder=True).
+- [x] **Cap + reminder** (apply_caps=True, reminder=True) — §12.8.
+- [ ] **n≥5 verification** before flipping defaults (§12.9 caveat).

 5pp signal floor per `docs/bench-maxing.md` for default-flip
 decisions. Each cycle adds 81 runs at ~12-17min on Hermes.
@ -2044,3 +2045,112 @@ Predicted §12.8 outcome (n=3, 9 questions):
 - JSON UNGROUNDED: 1 (already low; reminder dominates this metric)
 - Pointer STRICT-rate: still 0/27 (neither knob fixes pointer-tag
  discipline at this sample size)
+
+### 12.8 Cap + reminder A/B (2026-05-03T12-54-11Z)
+
+Same 9-question broad subset. Policy overrides:
+`--policy quantifier_guard_apply_caps=true --policy quantifier_reminder_enabled=true`.
+
+**Four-cell summary table:**
+
+| Metric                | Baseline | Reminder    | Cap-only    | Cap+Reminder |
+|-----------------------|----------|-------------|-------------|--------------|
+| `quote` strict-rate   | 0.56     | 0.52        | 0.59        | 0.59         |
+| `quote` mean ratio    | 0.900    | 0.845       | 0.870       | 0.861        |
+| `pointer` strict-rate | 0.00     | 0.00        | 0.00        | 0.00         |
+| `pointer` mean ratio  | 0.473    | 0.643       | 0.516       | **0.684**    |
+| `pointer` UNGROUNDED  | 9        | 6           | 9           | **6**        |
+| `JSON` strict-rate    | 0.19     | 0.22        | **0.33**    | 0.30         |
+| `JSON` mean ratio     | 0.524    | **0.735**   | 0.643       | 0.672        |
+| `JSON` UNGROUNDED     | 7        | **1**       | 6           | **1**        |
+| `JSON` STRICT count   | 5        | 6           | **9**       | 8            |
+| `pointer` FORMAT_COLLAPSED | 2  | **0**       | 2           | **0**        |
+| `pointer` NO_EVIDENCE_POINTER | 9 | **6**     | 7           | **7**        |
+
+**§10.8 strict-gate verdict for cap+reminder:**
+
+The §10.8 rule is *"If A+B together outperform either alone by ≥5pp:
+land Option D."*
+
+- vs cap-only (best STRICT-rate cell): cap+reminder JSON SR is **−3pp**
+  (regression, within noise).
+- vs reminder-only on JSON SR: cap+reminder is **+8pp** (clears floor).
+- vs cap-only on pointer mean ratio: cap+reminder is **+17pp** (clears floor).
+- vs reminder-only on pointer mean ratio: cap+reminder is **+4pp** (just
+  below floor).
+
+Mixed. Cap+reminder is **strictly better than reminder-only on STRICT-rate
+and strictly better than cap-only on mean ratio + UNGROUNDED-reduction**,
+but it is *not strictly better than both alone on every metric*.
+
+### 12.9 Final A/B verdict + recommendation
+
+The four cells reveal a clean **mechanism asymmetry**:
+
+```
+Reminder rescues UNGROUNDED → HYBRID
+    (restates [E\d+] citation rule; reduces format-discipline
+     failures upstream of the verifier)
+
+Cap rescues HYBRID → STRICT
+    (forces fewer-but-better claims; raises per-claim grounding
+     bar so the model can fully ground 5-8 claims it couldn't
+     fully ground at 12)
+```
+
+These are complementary, not redundant. But the §10.8 strict gate
+("compound beats either alone by ≥5pp on every metric") is **not
+cleanly met**. Cap+reminder is the best CELL on pointer mean ratio
+(0.684) but not the best on JSON STRICT-rate (cap-only wins 0.33 vs
+0.30).
+
+**Recommendation: Option A — single-knob defaults, per-mode tailored.**
+
+Lattice modes (`claim_lattice_pointer`, `claim_lattice`):
+```
+quantifier_guard_enabled       True   (already default)
+quantifier_guard_apply_caps    True   ← FLIP for claim_lattice (JSON) ONLY
+quantifier_reminder_enabled    True   ← FLIP for both lattice modes
+quantifier_reject_broad        False  (operator opt-in only)
+```
+
+Pointer mode keeps `apply_caps=False` because:
+
+- Pointer-mode STRICT-rate is 0/27 across all four cells; cap can't
+  rescue what's structurally upstream of cap (pointer-tag discipline).
+- Cap firing 20× under apply_caps=True adds TOO_MANY_CLAIMS noise
+  without verdict gain.
+- Reminder alone moves the right metric (NO_EVIDENCE_POINTER −33%,
+  FORMAT_COLLAPSED −100%, mean ratio +17pp).
+
+Quote mode keeps the guard mode-gated off (already default) because:
+
+- Quote mode handles broad questions well already (0.56-0.59 strict-rate
+  across all cells, paraphrase verifier).
+- Quote-mode strict-rate jitter (±3pp across cells) is Hermes
+  nondeterminism, not a guard effect.
+
+**Caveats blocking immediate flip:**
+
+- n=3 × 9 questions = 27 per cell. Variance band is ~3-4pp from
+  Hermes nondeterminism. Some of the cell-to-cell deltas (cap+reminder
+  0.30 vs cap-only 0.33 on JSON SR) are within noise.
+- Cap-only's +14pp gain on JSON STRICT-rate is clearly above the
+  floor; reminder's −22pp UNGROUNDED reduction is clearly above the
+  floor. Those two findings are robust.
+- Compound-effect verification at higher n (n=5 or n=10) would
+  tighten the cap+reminder vs cap-only comparison and confirm
+  whether the 3pp gap is real or noise.
+
+**Recommended next step:**
+
+Run **n=5 verification of the cap+reminder cell** (same 9 questions,
+135 runs at concurrency 4 ≈ ~6-8 min). If cap+reminder JSON STRICT-rate
+holds at ≥0.30 with n=5, the Option D gate is met (compound effect ≈
+cap-only on STRICT, with the reminder benefits preserved). If it
+drops below 0.27, Option A is the right landing.
+
+Either way, the disable hierarchy (§10.11.2) means flipping defaults
+is reversible per-call, per-mode, per-test. No commitment is
+permanent — flip the default, watch one bench cycle, revert if
+regression.