From c5bc53f733f8a14d09dec8a9fad1bd6367e26ffc Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 10 May 2026 09:53:21 -0400 Subject: [PATCH] bench: aggressive warrant fixture confirms Phase 3 is rescue-only, not default-path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Followup to dee6623 ("bench: #000031 Phase 3 A/B finds mechanism dormant on warrant-targeted fixture") — the initial A/B reported Phase 3 dormant on a 7-question fixture where only 2/7 questions retrieved any chain-root sources. Open question: was the dormancy a fixture inadequacy or a structural property of Phase 3? This commit answers it. Aggressive fixture (13 questions using exact claim-pack title phrasing — "What is the Pigeonhole Principle?", "Define the Axiom of Extensionality", etc.; titles sampled directly from the 92-document chain-root set on shard 000.db) drives retrieval to chain-root chunks on **12/13 questions** (24 chain sources retrieved total). A/B vs a57b194~1, n=3, 117 cells per condition: cache_key parity: 39/39 cells match Phase 3 fire rate (warrant_proven_idxs): 0 / 117 cells STRICT-rate: 85/117 → 85/117 (Δ +0) per-cell movement (LLM dice): 4 ↑, 4 ↓ Phase 3 fires zero times even though retrieval does land on chain roots on 92% of questions. The structural reason now reads as: Phase 3 is a rescue mechanism, not a default path. It fires only when (a) a cited chunk has warrant-resolver chain backing AND (b) the lexical warrant_check would otherwise fail. On well-formed axiom questions, the LLM's answer contains the technical terminology that the lexical anchor heuristic looks for — so warrant_check passes via the cheap lexical path and Phase 3 has nothing to suppress. Two readings of this finding: 1. **Positive for the lexical layer.** It's strong enough on clean axiom questions that the chain-fallback isn't needed. The mechanism that lands warrants on cheap lexical evidence continues to do most of the work. 2. **Phase 3 calibration needs a different fixture shape.** To exercise the rescue path empirically, a future fixture would need to elicit lexical-fail-but-chain-pass claims — paraphrase-heavy questions, disputed-terminology topics, questions where the LLM naturally drifts from chunk vocabulary. Mechanism is correct + non-regressing (cache_keys identical → no retrieval movement; 0/117 fire rate confirms defensive-only behavior). No further changes warranted to the implementation. C — separate finding (refactor sweep verification): a57b194 has zero v7/v8 patterns. Final exhaustive sweep across arborist/ + tests/ shows only the intentional historical-note comments in arborist/cli.py (Was \`arborist v8 score\`...) and arborist/substrate/__init__.py (explaining why v-dirs were retired). Refactor cleanup is complete. B — separate finding (Makefile bench-fork-score smoke): make bench-fork-baseline + make bench-fork-score runs end-to-end under the renamed CLI. Verdict MARGINAL (score 0.0 for self- comparison), exit 0. Yesterday's Makefile:430 fix verified live. --- .../qa_questions_warrant_chain_aggressive.txt | 40 +++++++ ...warrant-chain-A-B-aggressive-2026-05-10.md | 110 ++++++++++++++++++ 2 files changed, 150 insertions(+) create mode 100644 bench/qa_questions_warrant_chain_aggressive.txt create mode 100644 bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md diff --git a/bench/qa_questions_warrant_chain_aggressive.txt b/bench/qa_questions_warrant_chain_aggressive.txt new file mode 100644 index 0000000..089559d --- /dev/null +++ b/bench/qa_questions_warrant_chain_aggressive.txt @@ -0,0 +1,40 @@ +# Aggressive warrant-chain fixture (2026-05-10). +# +# Phase 3 verifier (a57b194) suppresses WARRANT_MISSING when a cited +# chunk has a warrant-resolver derivation row. The first probe fixture +# (qa_questions_warrant_chain_probe.txt) found Phase 3 dormant — +# retrieval landed on Wikipedia chunks instead of claim-pack chunks +# even on warrant-targeted topics. +# +# This fixture uses **exact claim-pack title phrasing** to maximize +# title-token boost in the retrieval pipeline. Each question's noun +# phrase matches a specific claim-pack record's title verbatim, which +# should drive retrieval to surface that claim-pack chunk in the top +# slots. +# +# Source: titles sampled from documents where document_root ∈ +# warrant_chain_lookup() (the 92 chain-root set) on shard 000.db +# 2026-05-10. + +# Pillar VII — Combinatorics (titles match claim-pack records) +What is the Pigeonhole Principle? +What is Pascal's Rule? +What is the Multiplication Principle in combinatorics? + +# Pillar I — First-order logic (axiom titles) +What is the Axiom of Implication Introduction? +What is the Axiom of Reflexivity of Equality? +What is the Axiom of Universal Specification? + +# Pillar II — Set theory (ZFC axioms) +Define the Axiom of Extensionality. +What is the Axiom of Empty Set? +What is the Axiom of Power Set? +Define the Axiom of Foundation. + +# Pillar III — Peano arithmetic +What is the Axiom of Successor? +Define the Axiom of Addition (Zero). + +# Pillar IV — Hilbert geometry +State the Archimedean Axiom. diff --git a/bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md b/bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md new file mode 100644 index 0000000..7749594 --- /dev/null +++ b/bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md @@ -0,0 +1,110 @@ +# #000031 Phase 3 verifier — aggressive A/B (2026-05-10) + +**Followup to**: `phase3-warrant-chain-A-B-2026-05-10.md` (initial +A/B which found Phase 3 dormant on a 7-question warrant fixture). +The initial probe showed only 2/7 questions retrieved any chain- +root sources; this followup uses **exact claim-pack title phrasing** +(13 questions; titles sampled directly from the 92-document +chain-root set on shard 000.db) to maximize title-token boost +in retrieval. Goal: confirm whether dormancy was a fixture +inadequacy or a structural property of Phase 3. + +## Pre-bench probe + +``` +12/13 questions hit chain ✓ 24 chain sources retrieved total +``` + +Aggressive fixture works as designed: title-token-aligned questions +drive retrieval to claim-pack chunks. The single miss (Archimedean +Axiom) retrieves 8 sources but none are chain-root — likely the +Wikipedia article on the Archimedean property dominates. + +## A/B method + +- BEFORE: `a57b194~1` (verify.py / runner.py / query.py reverted + via `git show`). +- AFTER: HEAD (Phase 3 active). +- 13 questions × 3 modes (quote / claim_lattice_pointer / + claim_lattice) × n=3 = 117 cells per condition. +- `--burn` between samples. + +## Cache-key parity + +**39/39 cells match BEFORE/AFTER.** Phase 3 doesn't fold into +`governance_policy_hash` — retrieval + prompt + LLM input are +byte-identical between conditions. Same clean A/B shape as the +first probe. + +## Phase 3 fire rate + +**`warrant_proven_claim_idxs` is non-empty in 0 / 117 AFTER cells.** + +Phase 3's suppression path **never fires** even though the +aggressive fixture drives retrieval to chain-root chunks on 92% of +questions. + +## STRICT-rate + +| | BEFORE | AFTER | Δ | +|---|---|---|---| +| overall | 85/117 (72.6%) | 85/117 (72.6%) | **+0** (exactly identical aggregate) | + +8 cells move at the per-cell level — 4 up, 4 down. All within LLM +n=3 noise; cache_keys match so retrieval+prompt are byte-identical +and the variance is purely in LLM sampling. + +## Refined verdict + +The first A/B's "Phase 3 is structurally dormant" reading was +correct in fact but incomplete in *why*. Now, with retrieval driving +chain-root chunks into the source set and Phase 3 still firing zero +times, the structural reason becomes clear: + +**Phase 3 is a rescue mechanism, not a default path.** It fires only +when: + +1. A cited chunk has warrant-resolver chain backing, AND +2. The lexical `warrant_check` would otherwise fail on the + claim citing that chunk. + +On well-formed axiom questions, the LLM's answer typically contains +the technical terminology that the lexical anchor heuristic looks +for — so `warrant_check` passes via the cheap lexical path and +Phase 3 has nothing to suppress. + +This is a **positive** finding for the lexical layer: it's strong +enough on clean axiom questions that the chain-fallback isn't +needed. Phase 3's value will surface on questions where: + +- The LLM paraphrases its claim away from the chunk's exact + terminology (lexical anchor heuristic fails). +- BUT the chunk has chain backing (Phase 3 rescue available). + +That intersection is empirically rare on well-named axiom +questions where the model naturally produces well-anchored output. +Future Phase 3 calibration could target it explicitly with prompts +that encourage paraphrase / questions about disputed-terminology +topics where the LLM is likely to drift. + +## Concrete recommendation + +Phase 3 mechanism is **correct + non-regressing**; cache_keys +identical → cannot move retrieval; 0/117 fire rate confirms +defensive-only behavior. No further changes warranted to the +mechanism itself. + +For ticket #000031 closure: the warrant-chain rescue path is +landed and unit-tested (see `tests/test_warrant_chain.py` from +a57b194). Live-bench measurement of the rescue's effect awaits +a fixture designed to elicit lexical-fail-but-chain-pass claims — +which is itself a research-design problem orthogonal to whether +the mechanism works. + +## Source files + +- Aggressive fixture: `bench/qa_questions_warrant_chain_aggressive.txt` +- Probe fixture (initial): `bench/qa_questions_warrant_chain_probe.txt` +- Initial A/B report: `bench/results/phase3-warrant-chain-A-B-2026-05-10.md` +- BEFORE results: `bench/qa_results_phase3_aggressive/before/` +- AFTER results: `bench/qa_results_phase3_aggressive/after/`