Followup to dee6623 ("bench: #000031 Phase 3 A/B finds mechanism
dormant on warrant-targeted fixture") — the initial A/B reported
Phase 3 dormant on a 7-question fixture where only 2/7 questions
retrieved any chain-root sources. Open question: was the dormancy
a fixture inadequacy or a structural property of Phase 3?
This commit answers it. Aggressive fixture (13 questions using
exact claim-pack title phrasing — "What is the Pigeonhole
Principle?", "Define the Axiom of Extensionality", etc.; titles
sampled directly from the 92-document chain-root set on shard
000.db) drives retrieval to chain-root chunks on **12/13
questions** (24 chain sources retrieved total).
A/B vs a57b194~1, n=3, 117 cells per condition:
cache_key parity: 39/39 cells match
Phase 3 fire rate (warrant_proven_idxs): 0 / 117 cells
STRICT-rate: 85/117 → 85/117 (Δ +0)
per-cell movement (LLM dice): 4 ↑, 4 ↓
Phase 3 fires zero times even though retrieval does land on chain
roots on 92% of questions. The structural reason now reads as:
Phase 3 is a rescue mechanism, not a default path. It fires only
when (a) a cited chunk has warrant-resolver chain backing AND
(b) the lexical warrant_check would otherwise fail. On
well-formed axiom questions, the LLM's answer contains the
technical terminology that the lexical anchor heuristic looks
for — so warrant_check passes via the cheap lexical path and
Phase 3 has nothing to suppress.
Two readings of this finding:
1. **Positive for the lexical layer.** It's strong enough on
clean axiom questions that the chain-fallback isn't needed.
The mechanism that lands warrants on cheap lexical evidence
continues to do most of the work.
2. **Phase 3 calibration needs a different fixture shape.** To
exercise the rescue path empirically, a future fixture would
need to elicit lexical-fail-but-chain-pass claims —
paraphrase-heavy questions, disputed-terminology topics,
questions where the LLM naturally drifts from chunk vocabulary.
Mechanism is correct + non-regressing (cache_keys identical → no
retrieval movement; 0/117 fire rate confirms defensive-only
behavior). No further changes warranted to the implementation.
C — separate finding (refactor sweep verification): a57b194 has
zero v7/v8 patterns. Final exhaustive sweep across arborist/ +
tests/ shows only the intentional historical-note comments in
arborist/cli.py (Was \`arborist v8 score\`...) and
arborist/substrate/__init__.py (explaining why v-dirs were
retired). Refactor cleanup is complete.
B — separate finding (Makefile bench-fork-score smoke):
make bench-fork-baseline + make bench-fork-score runs end-to-end
under the renamed CLI. Verdict MARGINAL (score 0.0 for self-
comparison), exit 0. Yesterday's Makefile:430 fix verified live.
russell@unturf.com
c5bc53f733