russell@unturf.com
c5bc53f733
Followup todee6623("bench: #000031 Phase 3 A/B finds mechanism dormant on warrant-targeted fixture") — the initial A/B reported Phase 3 dormant on a 7-question fixture where only 2/7 questions retrieved any chain-root sources. Open question: was the dormancy a fixture inadequacy or a structural property of Phase 3? This commit answers it. Aggressive fixture (13 questions using exact claim-pack title phrasing — "What is the Pigeonhole Principle?", "Define the Axiom of Extensionality", etc.; titles sampled directly from the 92-document chain-root set on shard 000.db) drives retrieval to chain-root chunks on **12/13 questions** (24 chain sources retrieved total). A/B vs a57b194~1, n=3, 117 cells per condition: cache_key parity: 39/39 cells match Phase 3 fire rate (warrant_proven_idxs): 0 / 117 cells STRICT-rate: 85/117 → 85/117 (Δ +0) per-cell movement (LLM dice): 4 ↑, 4 ↓ Phase 3 fires zero times even though retrieval does land on chain roots on 92% of questions. The structural reason now reads as: Phase 3 is a rescue mechanism, not a default path. It fires only when (a) a cited chunk has warrant-resolver chain backing AND (b) the lexical warrant_check would otherwise fail. On well-formed axiom questions, the LLM's answer contains the technical terminology that the lexical anchor heuristic looks for — so warrant_check passes via the cheap lexical path and Phase 3 has nothing to suppress. Two readings of this finding: 1. **Positive for the lexical layer.** It's strong enough on clean axiom questions that the chain-fallback isn't needed. The mechanism that lands warrants on cheap lexical evidence continues to do most of the work. 2. **Phase 3 calibration needs a different fixture shape.** To exercise the rescue path empirically, a future fixture would need to elicit lexical-fail-but-chain-pass claims — paraphrase-heavy questions, disputed-terminology topics, questions where the LLM naturally drifts from chunk vocabulary. Mechanism is correct + non-regressing (cache_keys identical → no retrieval movement; 0/117 fire rate confirms defensive-only behavior). No further changes warranted to the implementation. C — separate finding (refactor sweep verification):a57b194has zero v7/v8 patterns. Final exhaustive sweep across arborist/ + tests/ shows only the intentional historical-note comments in arborist/cli.py (Was \`arborist v8 score\`...) and arborist/substrate/__init__.py (explaining why v-dirs were retired). Refactor cleanup is complete. B — separate finding (Makefile bench-fork-score smoke): make bench-fork-baseline + make bench-fork-score runs end-to-end under the renamed CLI. Verdict MARGINAL (score 0.0 for self- comparison), exit 0. Yesterday's Makefile:430 fix verified live.
40 lines
1.5 KiB
Text
40 lines
1.5 KiB
Text
# Aggressive warrant-chain fixture (2026-05-10).
|
|
#
|
|
# Phase 3 verifier (a57b194) suppresses WARRANT_MISSING when a cited
|
|
# chunk has a warrant-resolver derivation row. The first probe fixture
|
|
# (qa_questions_warrant_chain_probe.txt) found Phase 3 dormant —
|
|
# retrieval landed on Wikipedia chunks instead of claim-pack chunks
|
|
# even on warrant-targeted topics.
|
|
#
|
|
# This fixture uses **exact claim-pack title phrasing** to maximize
|
|
# title-token boost in the retrieval pipeline. Each question's noun
|
|
# phrase matches a specific claim-pack record's title verbatim, which
|
|
# should drive retrieval to surface that claim-pack chunk in the top
|
|
# slots.
|
|
#
|
|
# Source: titles sampled from documents where document_root ∈
|
|
# warrant_chain_lookup() (the 92 chain-root set) on shard 000.db
|
|
# 2026-05-10.
|
|
|
|
# Pillar VII — Combinatorics (titles match claim-pack records)
|
|
What is the Pigeonhole Principle?
|
|
What is Pascal's Rule?
|
|
What is the Multiplication Principle in combinatorics?
|
|
|
|
# Pillar I — First-order logic (axiom titles)
|
|
What is the Axiom of Implication Introduction?
|
|
What is the Axiom of Reflexivity of Equality?
|
|
What is the Axiom of Universal Specification?
|
|
|
|
# Pillar II — Set theory (ZFC axioms)
|
|
Define the Axiom of Extensionality.
|
|
What is the Axiom of Empty Set?
|
|
What is the Axiom of Power Set?
|
|
Define the Axiom of Foundation.
|
|
|
|
# Pillar III — Peano arithmetic
|
|
What is the Axiom of Successor?
|
|
Define the Axiom of Addition (Zero).
|
|
|
|
# Pillar IV — Hilbert geometry
|
|
State the Archimedean Axiom.
|