# #000031 Phase 3 verifier — aggressive A/B (2026-05-10)

**Followup to**: `phase3-warrant-chain-A-B-2026-05-10.md` (initial
A/B which found Phase 3 dormant on a 7-question warrant fixture).
The initial probe showed only 2/7 questions retrieved any chain-
root sources; this followup uses **exact claim-pack title phrasing**
(13 questions; titles sampled directly from the 92-document
chain-root set on shard 000.db) to maximize title-token boost
in retrieval. Goal: confirm whether dormancy was a fixture
inadequacy or a structural property of Phase 3.

## Pre-bench probe

```
12/13 questions hit chain ✓     24 chain sources retrieved total
```

Aggressive fixture works as designed: title-token-aligned questions
drive retrieval to claim-pack chunks. The single miss (Archimedean
Axiom) retrieves 8 sources but none are chain-root — likely the
Wikipedia article on the Archimedean property dominates.

## A/B method

- BEFORE: `a57b194~1` (verify.py / runner.py / query.py reverted
  via `git show`).
- AFTER: HEAD (Phase 3 active).
- 13 questions × 3 modes (quote / claim_lattice_pointer /
  claim_lattice) × n=3 = 117 cells per condition.
- `--burn` between samples.

## Cache-key parity

**39/39 cells match BEFORE/AFTER.** Phase 3 doesn't fold into
`governance_policy_hash` — retrieval + prompt + LLM input are
byte-identical between conditions. Same clean A/B shape as the
first probe.

## Phase 3 fire rate

**`warrant_proven_claim_idxs` is non-empty in 0 / 117 AFTER cells.**

Phase 3's suppression path **never fires** even though the
aggressive fixture drives retrieval to chain-root chunks on 92% of
questions.

## STRICT-rate

| | BEFORE | AFTER | Δ |
|---|---|---|---|
| overall | 85/117 (72.6%) | 85/117 (72.6%) | **+0** (exactly identical aggregate) |

8 cells move at the per-cell level — 4 up, 4 down. All within LLM
n=3 noise; cache_keys match so retrieval+prompt are byte-identical
and the variance is purely in LLM sampling.

## Refined verdict

The first A/B's "Phase 3 is structurally dormant" reading was
correct in fact but incomplete in *why*. Now, with retrieval driving
chain-root chunks into the source set and Phase 3 still firing zero
times, the structural reason becomes clear:

**Phase 3 is a rescue mechanism, not a default path.** It fires only
when:

1. A cited chunk has warrant-resolver chain backing, AND
2. The lexical `warrant_check` would otherwise fail on the
   claim citing that chunk.

On well-formed axiom questions, the LLM's answer typically contains
the technical terminology that the lexical anchor heuristic looks
for — so `warrant_check` passes via the cheap lexical path and
Phase 3 has nothing to suppress.

This is a **positive** finding for the lexical layer: it's strong
enough on clean axiom questions that the chain-fallback isn't
needed. Phase 3's value will surface on questions where:

- The LLM paraphrases its claim away from the chunk's exact
  terminology (lexical anchor heuristic fails).
- BUT the chunk has chain backing (Phase 3 rescue available).

That intersection is empirically rare on well-named axiom
questions where the model naturally produces well-anchored output.
Future Phase 3 calibration could target it explicitly with prompts
that encourage paraphrase / questions about disputed-terminology
topics where the LLM is likely to drift.

## Concrete recommendation

Phase 3 mechanism is **correct + non-regressing**; cache_keys
identical → cannot move retrieval; 0/117 fire rate confirms
defensive-only behavior. No further changes warranted to the
mechanism itself.

For ticket #000031 closure: the warrant-chain rescue path is
landed and unit-tested (see `tests/test_warrant_chain.py` from
a57b194). Live-bench measurement of the rescue's effect awaits
a fixture designed to elicit lexical-fail-but-chain-pass claims —
which is itself a research-design problem orthogonal to whether
the mechanism works.

## Source files

- Aggressive fixture: `bench/qa_questions_warrant_chain_aggressive.txt`
- Probe fixture (initial): `bench/qa_questions_warrant_chain_probe.txt`
- Initial A/B report: `bench/results/phase3-warrant-chain-A-B-2026-05-10.md`
- BEFORE results: `bench/qa_results_phase3_aggressive/before/`
- AFTER results: `bench/qa_results_phase3_aggressive/after/`
