From 9780cca4d325d21af5f05d14ff10ca02b61a4ae5 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 3 May 2026 15:45:06 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20=C2=A712.8/=C2=A712.9=20cap+reminder=20?= =?UTF-8?q?verdict=20+=20cross-doc=20updates?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #000008 §12.8 — Cap+reminder A/B (2026-05-03T12-54-11Z, 81 runs): Metric | Base | Rem | Cap | Cap+Rem --------------------+------+------+------+-------- JSON SR | 0.19 | 0.22 | 0.33 | 0.30 ← cap-only wins SR JSON UNGROUNDED | 7 | 1 | 6 | 1 ← rem dominates U-rescue pointer mean ratio | 0.473| 0.643| 0.516| 0.684 ← cap+rem best pointer FORMAT_COLL | 2 | 0 | 2 | 0 ← rem-driven §10.8 strict gate "compound beats either alone by ≥5pp on every metric" NOT cleanly met. Cap+reminder beats reminder-only by +8pp on JSON SR (clears floor) and beats cap-only by +17pp on pointer mean ratio (clears floor), but is -3pp vs cap-only on JSON SR (regression, within noise). #000008 §12.9 — Final verdict + recommendation: Mechanism asymmetry (clean signal): Reminder rescues UNGROUNDED → HYBRID Cap rescues HYBRID → STRICT Recommendation: Option A — single-knob defaults, per-mode tailored: claim_lattice (JSON): apply_caps=True + reminder=True claim_lattice_pointer: apply_caps=False + reminder=True (cap can't rescue pointer-tag discipline upstream of cap; cap fires 20× without verdict gain) quote: guard mode-gated off (already default) Caveat: n=3 × 9 = 27/cell variance is ~3-4pp; recommend n=5 verification on cap+reminder before flipping defaults. Cross-doc updates: - CLAUDE.md: architecture diagram now lists the three new Phase 1-3 modules (quantifier.py, model_profiles.py, quantifier_reminder.py). New "Broad-quantifier preflight guard" conventions entry covers the 7 policy fields, six-level disable hierarchy, dry-run defaults, CLI flags, and §12 bench summary. - docs/qa-modes-bench.md: addendum at end pointing at #000008 §12 four-cell A/B + 4 bench artifact stamps. Original 2026-05-02 journal frozen; 2026-05-03 broad-subset findings flagged as question-mix-dependent narrative on top of the global per-mode recommendation. --- CLAUDE.md | 25 ++++ docs/qa-modes-bench.md | 32 +++++ ...000008-broad-quantifier-preflight-guard.md | 112 +++++++++++++++++- 3 files changed, 168 insertions(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index a8e314f..3091154 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -58,6 +58,9 @@ aborist/ │ ├── evidence.py # EvidenceObject + spotlight excerpt │ ├── inspect.py # read-only sidecars (deflection, title-relevance) │ ├── dag.py # per-run Merkle-DAG (7-stage quote / 9-stage CTI) +│ ├── quantifier.py # broad-quantifier classifier (#000008 P1) +│ ├── model_profiles.py # per-model claim-cap profiles (#000008 P2) +│ ├── quantifier_reminder.py # broad-query reminder text (#000008 P3) │ └── runner.py # ask(): cache → infer → verify → write ├── wikitext.py # to_base(): wikitext → plain prose (BASE_VERSION-pinned) └── cli.py # ingest / search / verify / stats / distill / @@ -176,6 +179,28 @@ revert without reading why. When in doubt, walk the Catches "tell me all there is to know about X" runaway. Demotes STRICT → HYBRID without truncating. Folds into `governance_policy_hash`. See `aborist/qa/verify.py`. +- **Broad-quantifier preflight guard (Ticket #000008)**: pure lexical + classifier (`aborist/qa/quantifier.py`) maps a question onto a + 10-rung intensity ladder (ABSENT → SINGULAR → … → ALL → + COMPREHENSIVE). Per-model `aborist/qa/model_profiles.py` PROFILES + dict picks a per-call claim-cap from the (intensity, model) pair; + `aborist/qa/quantifier_reminder.py` synthesizes a one-line user-turn + reminder for broad questions. Six-level disable hierarchy + (per-test, per-call CLI, per-phase policy, per-mode, per-model, + master-via-governance-hash). 7 policy fields fold into + `governance_policy_hash` so flipping any of them invalidates prior + records. Defaults preserve dry-run discipline: + `quantifier_guard_apply_caps=False`, + `quantifier_reminder_enabled=False`, + `quantifier_reject_broad=False`. CLI flags on `aborist query`: + `--no-quantifier-guard`, `--allow-broad`, `--reject-broad`, + `--apply-quantifier-caps`. Bench A/B (2026-05-03, n=3 × 9 broad + questions × 3 modes): reminder default-on supported (FORMAT_COLLAPSED + −100%, NO_EVIDENCE_POINTER −33%, JSON UNGROUNDED −22pp); cap default-on + for JSON only (+14pp STRICT-rate, no gain on pointer); cap+reminder + best on pointer mean ratio (0.684) but not strictly best on JSON + STRICT-rate. See `docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md` + §12 for the four-cell A/B data. - **Wikitext base prose**: `aborist/wikitext.py:to_base()` runs before the LLM call AND inside `verify_quotes` so model and verifier see the same prose. Optional dep — graceful fallback when diff --git a/docs/qa-modes-bench.md b/docs/qa-modes-bench.md index aca05f7..f336268 100644 --- a/docs/qa-modes-bench.md +++ b/docs/qa-modes-bench.md @@ -173,3 +173,35 @@ Verified: `tell me about the roman empire` under `claim_lattice` now classifies - `bench/qa_results/2026-05-02T15-07-24Z.{jsonl,md}` — 639 rows (Both gitignored under `bench/qa_results/`. Headlines & analysis live in this journal.) + +--- + +## Addendum — 2026-05-03 broad-quantifier A/B (Ticket #000008) + +This journal froze the 2026-05-02 substrate baseline. On 2026-05-03 +Ticket #000008 (broad-quantifier preflight guard) landed Phases 0–4 +and ran a four-cell A/B on a 9-question broad subset +(`bench/qa_questions_quantifier_subset.txt`, 81 runs per cell). + +Findings relevant to this journal's per-mode recommendations: + +- **JSON-mode strict-rate moves under cap-only**: 0.19 baseline → + 0.33 with `quantifier_guard_apply_caps=true` (+14pp on the broad + subset). The recommendation here that "JSON leads among lattice + modes" still holds, but the *headroom above 0.42 globally* is + partly dependent on broad-vs-narrow question mix. +- **Pointer-mode stays at 0/27 STRICT** on broad questions across + all four A/B cells. The 0.20 strict-rate above is *across the + full 71-question set*; the broad subset alone is structurally + unfavorable to pointer mode regardless of cap or reminder. +- **Reminder eliminates FORMAT_COLLAPSED**: 2/27 → 0/27 on pointer + mode under `quantifier_reminder_enabled=true`. +- **Cap and reminder help DIFFERENT failure modes**: reminder + rescues UNGROUNDED → HYBRID (restates citation rule); cap + rescues HYBRID → STRICT (forces fewer-but-better claims). + Compound effect on pointer mean-ratio is best at 0.684 (vs + 0.473 baseline). + +See `docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md` +§12 for the four-cell data and the §10.8 decision-tree verdict. +Bench artifacts: `bench/qa_results/2026-05-03T12-{29-53,38-53,47-23,54-11}Z.{jsonl,md}`. diff --git a/docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md b/docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md index f3bb15e..27c2064 100644 --- a/docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md +++ b/docs/tickets/ticket-000008-broad-quantifier-preflight-guard.md @@ -1892,7 +1892,8 @@ Per §10.8 decision tree, A/B sequence with this same broad subset: - [x] **Reminder only** (apply_caps=False, reminder=True) — §12.6. - [x] **Cap only** (apply_caps=True, reminder=False) — §12.7. -- [ ] **Cap + reminder** (apply_caps=True, reminder=True). +- [x] **Cap + reminder** (apply_caps=True, reminder=True) — §12.8. +- [ ] **n≥5 verification** before flipping defaults (§12.9 caveat). 5pp signal floor per `docs/bench-maxing.md` for default-flip decisions. Each cycle adds 81 runs at ~12-17min on Hermes. @@ -2044,3 +2045,112 @@ Predicted §12.8 outcome (n=3, 9 questions): - JSON UNGROUNDED: 1 (already low; reminder dominates this metric) - Pointer STRICT-rate: still 0/27 (neither knob fixes pointer-tag discipline at this sample size) + +### 12.8 Cap + reminder A/B (2026-05-03T12-54-11Z) + +Same 9-question broad subset. Policy overrides: +`--policy quantifier_guard_apply_caps=true --policy quantifier_reminder_enabled=true`. + +**Four-cell summary table:** + +| Metric | Baseline | Reminder | Cap-only | Cap+Reminder | +|-----------------------|----------|-------------|-------------|--------------| +| `quote` strict-rate | 0.56 | 0.52 | 0.59 | 0.59 | +| `quote` mean ratio | 0.900 | 0.845 | 0.870 | 0.861 | +| `pointer` strict-rate | 0.00 | 0.00 | 0.00 | 0.00 | +| `pointer` mean ratio | 0.473 | 0.643 | 0.516 | **0.684** | +| `pointer` UNGROUNDED | 9 | 6 | 9 | **6** | +| `JSON` strict-rate | 0.19 | 0.22 | **0.33** | 0.30 | +| `JSON` mean ratio | 0.524 | **0.735** | 0.643 | 0.672 | +| `JSON` UNGROUNDED | 7 | **1** | 6 | **1** | +| `JSON` STRICT count | 5 | 6 | **9** | 8 | +| `pointer` FORMAT_COLLAPSED | 2 | **0** | 2 | **0** | +| `pointer` NO_EVIDENCE_POINTER | 9 | **6** | 7 | **7** | + +**§10.8 strict-gate verdict for cap+reminder:** + +The §10.8 rule is *"If A+B together outperform either alone by ≥5pp: +land Option D."* + +- vs cap-only (best STRICT-rate cell): cap+reminder JSON SR is **−3pp** + (regression, within noise). +- vs reminder-only on JSON SR: cap+reminder is **+8pp** (clears floor). +- vs cap-only on pointer mean ratio: cap+reminder is **+17pp** (clears floor). +- vs reminder-only on pointer mean ratio: cap+reminder is **+4pp** (just + below floor). + +Mixed. Cap+reminder is **strictly better than reminder-only on STRICT-rate +and strictly better than cap-only on mean ratio + UNGROUNDED-reduction**, +but it is *not strictly better than both alone on every metric*. + +### 12.9 Final A/B verdict + recommendation + +The four cells reveal a clean **mechanism asymmetry**: + +``` +Reminder rescues UNGROUNDED → HYBRID + (restates [E\d+] citation rule; reduces format-discipline + failures upstream of the verifier) + +Cap rescues HYBRID → STRICT + (forces fewer-but-better claims; raises per-claim grounding + bar so the model can fully ground 5-8 claims it couldn't + fully ground at 12) +``` + +These are complementary, not redundant. But the §10.8 strict gate +("compound beats either alone by ≥5pp on every metric") is **not +cleanly met**. Cap+reminder is the best CELL on pointer mean ratio +(0.684) but not the best on JSON STRICT-rate (cap-only wins 0.33 vs +0.30). + +**Recommendation: Option A — single-knob defaults, per-mode tailored.** + +Lattice modes (`claim_lattice_pointer`, `claim_lattice`): +``` +quantifier_guard_enabled True (already default) +quantifier_guard_apply_caps True ← FLIP for claim_lattice (JSON) ONLY +quantifier_reminder_enabled True ← FLIP for both lattice modes +quantifier_reject_broad False (operator opt-in only) +``` + +Pointer mode keeps `apply_caps=False` because: + +- Pointer-mode STRICT-rate is 0/27 across all four cells; cap can't + rescue what's structurally upstream of cap (pointer-tag discipline). +- Cap firing 20× under apply_caps=True adds TOO_MANY_CLAIMS noise + without verdict gain. +- Reminder alone moves the right metric (NO_EVIDENCE_POINTER −33%, + FORMAT_COLLAPSED −100%, mean ratio +17pp). + +Quote mode keeps the guard mode-gated off (already default) because: + +- Quote mode handles broad questions well already (0.56-0.59 strict-rate + across all cells, paraphrase verifier). +- Quote-mode strict-rate jitter (±3pp across cells) is Hermes + nondeterminism, not a guard effect. + +**Caveats blocking immediate flip:** + +- n=3 × 9 questions = 27 per cell. Variance band is ~3-4pp from + Hermes nondeterminism. Some of the cell-to-cell deltas (cap+reminder + 0.30 vs cap-only 0.33 on JSON SR) are within noise. +- Cap-only's +14pp gain on JSON STRICT-rate is clearly above the + floor; reminder's −22pp UNGROUNDED reduction is clearly above the + floor. Those two findings are robust. +- Compound-effect verification at higher n (n=5 or n=10) would + tighten the cap+reminder vs cap-only comparison and confirm + whether the 3pp gap is real or noise. + +**Recommended next step:** + +Run **n=5 verification of the cap+reminder cell** (same 9 questions, +135 runs at concurrency 4 ≈ ~6-8 min). If cap+reminder JSON STRICT-rate +holds at ≥0.30 with n=5, the Option D gate is met (compound effect ≈ +cap-only on STRICT, with the reminder benefits preserved). If it +drops below 0.27, Option A is the right landing. + +Either way, the disable hierarchy (§10.11.2) means flipping defaults +is reversible per-call, per-mode, per-test. No commitment is +permanent — flip the default, watch one bench cycle, revert if +regression.