diff --git a/bench/qa_questions_warrant_chain_probe.txt b/bench/qa_questions_warrant_chain_probe.txt new file mode 100644 index 0000000..6ff6255 --- /dev/null +++ b/bench/qa_questions_warrant_chain_probe.txt @@ -0,0 +1,23 @@ +# Warrant-chain probe fixture — questions targeting claim-pack-resolved +# textbooks (Hilbert geometry, Cantor set theory, Russell logic, De Morgan, +# Boole, Peano arithmetic, Dedekind reals, Judson abstract algebra). +# +# Purpose: exercise the post-#000031-Phase 3 verifier path that suppresses +# WARRANT_MISSING when a cited chunk's document_root has a warrant-resolver +# derivation row. Smoke fixture retrieval lands zero chain hits; this +# fixture's questions explicitly target the 92 chain-resolved sources. + +# Pillar IV — Hilbert geometry (18 records) +what does Hilbert's first axiom of incidence state? +what does Hilbert mean by the axiom of order between three points? + +# Pillar I — set theory / classes (13 records, Russell + Cantor) +what is Russell's definition of a class? +what does Cantor's diagonal argument prove about real numbers? + +# Pillar II — logic (10 records, De Morgan + Boole) +what are De Morgan's laws? +in Boolean algebra, what is the absorption law? + +# Pillar VII — combinatorics (14 records, Bogart + Levin) +what is the pigeonhole principle in Bogart? diff --git a/bench/results/phase3-warrant-chain-A-B-2026-05-10.md b/bench/results/phase3-warrant-chain-A-B-2026-05-10.md new file mode 100644 index 0000000..e55dccc --- /dev/null +++ b/bench/results/phase3-warrant-chain-A-B-2026-05-10.md @@ -0,0 +1,110 @@ +# #000031 Phase 3 verifier — A/B (2026-05-10) + +**Change**: commit `a57b194` ("ticket #000031 Phase 3: +warrant-chain-aware verifier suppresses WARRANT_MISSING") landed +2026-05-10 09:17 EDT. + +**Mechanism**: when a claim's lexical `warrant_check` fails AND any +cited evidence's `source_root` is in `warrant_chain_lookup()` (the +92 `document_root` values across all shards that have a derivation +row with `process_id LIKE 'warrant-resolver-v1%'`), the verifier +suppresses the `WARRANT_MISSING` violation and tracks the claim_idx +on a new `warrant_proven_claim_idxs` field. Render layer surfaces +this as `· warrant proven via chain ×N` in the audit-line tail. + +**Bench question**: does Phase 3 actually move STRICT-rate on real +questions? Or is it structurally dormant? + +## Method + +- Fixture: `bench/qa_questions_warrant_chain_probe.txt` — 7 questions + targeting claim-pack-resolved topics (Hilbert geometry, Russell + classes, Cantor diagonal, De Morgan, Boolean absorption, pigeonhole + in Bogart). +- Pre-bench probe via `arborist query --dry-run --json`: only **2/7 + questions retrieve any chain-root sources** (De Morgan: 1 source; + Bogart pigeonhole: 2 sources). The other 5 questions retrieve + Wikipedia chunks for their topics — the claim-pack textbooks have + vastly fewer chunks than Wikipedia, so retrieval defaults to the + bulkier corpus. +- Conditions: BEFORE = `a57b194~1` (verify.py / runner.py / query.py + reverted via `git show`); AFTER = HEAD. +- Modes: quote, claim_lattice_pointer, claim_lattice. n=3 with + `--burn` between samples. + +## Cache-key parity + +**21/21 cells match BEFORE/AFTER.** Phase 3 doesn't fold into +`governance_policy_hash` (the new `warrant_chain_roots` parameter is +runtime-passed, not policy-folded), so retrieval + prompt are +byte-identical between conditions. Any STRICT-rate movement is real +verifier-side signal, not LLM dice. + +## Phase 3 fire rate + +**`warrant_proven_claim_idxs` is non-empty in 0 / 63 AFTER cells.** +The Phase 3 suppression path never triggers on this fixture. + +## STRICT-rate + +| | BEFORE | AFTER | Δ | +|---|---|---|---| +| overall | 42/63 (66.7%) | 43/63 (68.3%) | +1 cell (+1.6pp) | + +Four cells move at the per-cell level — 2 up (Russell quote +2; De +Morgan pointer +1; Bogart pointer +1) and 2 down (Cantor quote −2; +Hilbert-incidence pointer −1). Net +1, within LLM noise on n=3. + +Per-cell parity reads as: cache_keys identical, Phase 3 mechanism +not firing, so the deltas are LLM sampling variance — exactly the +same shape as yesterday's smoke A/B finding. + +## Verdict + +Phase 3 is **mechanistically landed but currently dormant on this +fixture**. The data condition (claim cites chain-root chunk AND +lexical `warrant_check` fails) is not met by: + +1. Questions whose retrieval doesn't surface any chain-root chunks + (5/7 in this fixture). +2. Questions that DO retrieve chain-root chunks but where the LLM + cites the bulkier Wikipedia chunks instead (the remaining 2/7). +3. Questions where the LLM does cite a chain-root chunk but the + lexical `warrant_check` already passes — Phase 3 has nothing to + suppress. + +This is not a defect — it's a calibration finding. The mechanism is +correct + non-regressing on cells where it doesn't fire. To actually +exercise Phase 3, a future fixture needs questions where: + +- Specific terminology biases retrieval AWAY from Wikipedia toward + claim-pack chunks (e.g. exact theorem labels: "Hilbert pillar IV + axiom of incidence"). +- Claims about those topics phrase themselves in ways that don't + pass lexical anchors but DO have warrant-chain backing. + +## Reproducing + +```bash +# AFTER (HEAD) +.venv/bin/python bench/qa_sweep.py \ + --questions bench/qa_questions_warrant_chain_probe.txt \ + --shards-dir ~/.arborist/shards \ + --out-dir bench/qa_results_phase3_AB/after \ + --top-k 8 \ + --modes quote,claim_lattice_pointer,claim_lattice \ + --n 3 --concurrency 4 + +# BEFORE (Phase 3 reverted) +git show a57b194~1:arborist/qa/verify.py > arborist/qa/verify.py +git show a57b194~1:arborist/qa/runner.py > arborist/qa/runner.py +git show a57b194~1:arborist/qa/query.py > arborist/qa/query.py +# (re-run the same sweep with --out-dir bench/qa_results_phase3_AB/before) +git checkout HEAD -- arborist/qa/verify.py arborist/qa/runner.py arborist/qa/query.py +``` + +## Source files + +- Fixture: `bench/qa_questions_warrant_chain_probe.txt` +- BEFORE results: `bench/qa_results_phase3_AB/before/` +- AFTER results: `bench/qa_results_phase3_AB/after/`