russell@unturf.com
dee6623e46
Bench follow-up to a57b194 ("ticket #000031 Phase 3: warrant-chain-
aware verifier suppresses WARRANT_MISSING"): land the warrant-
targeted probe fixture + capture an A/B comparing pre-Phase 3
(a57b194~1) vs HEAD on 7 questions × 3 modes × n=3.
bench/qa_questions_warrant_chain_probe.txt (new fixture)
========================================================
7 questions targeting claim-pack-resolved topics: Hilbert geometry
(2), Russell classes (1), Cantor diagonal (1), De Morgan (1),
Boolean absorption (1), pigeonhole in Bogart (1). Picked because
their topical match against the 18 textbook substrates +
warrant-resolver derivations should — in principle — drive
retrieval to chain-root chunks where Phase 3's suppression logic
fires.
bench/results/phase3-warrant-chain-A-B-2026-05-10.md (A/B report)
=================================================================
Findings:
- **21/21 cache_keys match BEFORE/AFTER.** Phase 3 doesn't fold
into governance_policy_hash (warrant_chain_roots is a runtime
parameter, not a policy field), so retrieval + prompt + LLM
inputs are byte-identical between conditions. Any STRICT-rate
movement is real verifier signal, not LLM dice.
- **`warrant_proven_claim_idxs` is non-empty in 0 / 63 AFTER cells.**
Phase 3's suppression path never fires on this fixture.
- **STRICT-rate: 42/63 → 43/63 (+1 cell, +1.6pp).** Four cells
move (2 up, 2 down); within n=3 LLM noise.
Why dormant despite warrant-targeted topic selection:
1. Pre-bench probe via `--dry-run --json` showed only 2/7 questions
retrieve any chain-root sources — the claim-pack textbooks have
far fewer chunks than Wikipedia, so retrieval defaults to the
bulkier corpus on general queries.
2. Even on the 2 questions that DO retrieve chain roots, the LLM
cites the larger Wikipedia chunks more often.
3. On the rare cell where the LLM cites a chain-root chunk, lexical
`warrant_check` already passes — Phase 3 has nothing to suppress.
Calibration finding, not a defect:
- Phase 3 mechanism is correct (no regression on cells where it
doesn't fire; cache_keys identical → can't move retrieval).
- The data condition for Phase 3 to actually fire requires a
fixture where (a) retrieval surfaces chain-root chunks AND
(b) the LLM cites them AND (c) the cited claim's lexical
warrant_check fails.
- Future fixture would need very specific terminology that
biases retrieval AWAY from Wikipedia (exact theorem labels,
axiom names, etc.).
Same calibration shape as yesterday's progressive-AND smoke A/B
finding: a targeted change can be dormant on the fixture you're
benching against. The bench-fixture-design discipline that ticket
#000037 §12 codifies — "measured trigger, not calendar date" —
applies here too: until a fixture lands that exercises Phase 3,
the suppression mechanism's value is theoretical.
23 lines
1.1 KiB
Text
23 lines
1.1 KiB
Text
# Warrant-chain probe fixture — questions targeting claim-pack-resolved
|
|
# textbooks (Hilbert geometry, Cantor set theory, Russell logic, De Morgan,
|
|
# Boole, Peano arithmetic, Dedekind reals, Judson abstract algebra).
|
|
#
|
|
# Purpose: exercise the post-#000031-Phase 3 verifier path that suppresses
|
|
# WARRANT_MISSING when a cited chunk's document_root has a warrant-resolver
|
|
# derivation row. Smoke fixture retrieval lands zero chain hits; this
|
|
# fixture's questions explicitly target the 92 chain-resolved sources.
|
|
|
|
# Pillar IV — Hilbert geometry (18 records)
|
|
what does Hilbert's first axiom of incidence state?
|
|
what does Hilbert mean by the axiom of order between three points?
|
|
|
|
# Pillar I — set theory / classes (13 records, Russell + Cantor)
|
|
what is Russell's definition of a class?
|
|
what does Cantor's diagonal argument prove about real numbers?
|
|
|
|
# Pillar II — logic (10 records, De Morgan + Boole)
|
|
what are De Morgan's laws?
|
|
in Boolean algebra, what is the absorption law?
|
|
|
|
# Pillar VII — combinatorics (14 records, Bogart + Levin)
|
|
what is the pigeonhole principle in Bogart?
|