bench: Phase 3 paraphrase fixture investigation — empirically dormant on current corpus

Three-iteration investigation following c5bc53f's "Phase 3 is a rescue
mechanism, not a default path" finding. Question: can a paraphrase-
eliciting fixture actually drive Phase 3 to fire on real questions?

Key reading from arborist/qa/warrant.py:507 ("warrant_check"):
the function vacuous-passes (returns (True, []) without any anchor
extraction) unless the question shape is one of:
- relation (proper-noun anchors)
- date (4-digit year in claim)
- entity-list
- count-shape
- why-cause-shape

If none of those shapes apply, Phase 3 has nothing to suppress.

Three fixture iterations, all probed before benching:

| iter | strategy                          | chain hits |
|------|-----------------------------------|------------|
| v1   | "What is X?" / "Define X"         | 12/13      |
| v2   | "How would you use X to ..."      |  1/10      |
| v2'  | "X: explain in plain English"     |  2/13      |
| v3   | "Who proposed X?" / "Why does X"  |  1/8       |
| v3'  | "X: who first stated it?"         |  1/8       |

Pattern: title-leading retrieval works (high title-token boost on
chain chunks) but those questions are definitional — no warrant
shape, no Phase 3 trigger. Warrant-shape questions ("who" / "when"
/ "why") trigger warrant_check but retrieval pulls toward
Wikipedia's prose articles instead of the terse claim-pack chunks.

Structural reason: claim-pack chunks (the 92 chain-root documents)
are deliberately terse — they state axioms / theorems precisely
with mathematical formalism but contain no biographical, historical,
or explanatory prose. So warrant-shape questions retrieve from
Wikipedia, where the prose context matches the question shape.
Phase 3's data condition (chain-backed chunk cited for a
warrant-shape claim) is empirically unreachable through normal
retrieval pathways on this corpus shape.

Verdict (`bench/results/phase3-warrant-chain-paraphrase-
investigation-2026-05-10.md`): Phase 3's runtime fire rate is
structurally bounded near zero on the current corpus. Mechanism
is correct, non-regressing, and unit-tested
(tests/test_warrant_chain.py). The lexical layer is strong enough
that the rescue isn't needed at measurable rates.

Recommendation: don't engineer Phase 3 fires via fixture design.
Track fire rate as a corpus-evolution signal. If claim-pack
content gains prose context or the corpus mix changes, Phase 3
will start firing organically and the audit-line tail
(`· warrant proven via chain ×N`) will surface it.

This closes the three-thread investigation cleanly. v1 + v2 + v3
fixtures + reports all retained as the empirical trail.
This commit is contained in:
russell@unturf.com 2026-05-10 10:06:23 -04:00
parent c5bc53f733
commit a9fcb4251d
No known key found for this signature in database
2 changed files with 146 additions and 0 deletions

View file

@ -0,0 +1,37 @@
# Paraphrase-eliciting warrant-chain fixture (2026-05-10, v2).
#
# v1 of this fixture used natural-language wrappers ("How would you
# use the Pigeonhole Principle to ..."). Pre-bench probe found only
# 1/10 questions hit chain — the wrapper words ("how would you use",
# "in your own words") match Wikipedia / general-knowledge articles
# better than concise claim-pack chunks, pushing retrieval AWAY
# from chain.
#
# v2 strategy: **title-leading + paraphrase suffix**. Each question
# starts with the exact claim-pack title (preserving title-token
# boost) but ends with a paraphrase-eliciting clause that should
# steer the LLM toward reformulation. Title leads retrieval; suffix
# leads generation.
# Pillar VII
Pigeonhole Principle: explain in plain English with a 13-people example.
Pascal's Rule: rephrase as a counting story.
Multiplication Principle: describe informally with a real-world example.
# Pillar I
Axiom of Implication Introduction: explain in your own words.
Axiom of Universal Specification: rephrase using simpler vocabulary.
Axiom of Reflexivity of Equality: describe informally.
# Pillar II
Axiom of Extensionality: rephrase using only common words.
Axiom of Empty Set: describe in plain English.
Axiom of Power Set: explain informally with no jargon.
Axiom of Foundation: rephrase as a rule about sets.
# Pillar III
Axiom of Successor: describe using only counting words.
Axiom of Addition (Zero): explain in plain language.
# Pillar IV
Archimedean Axiom: rephrase using simpler measurement vocabulary.

View file

@ -0,0 +1,109 @@
# #000031 Phase 3 — paraphrase fixture investigation (2026-05-10)
**Sequel to** the aggressive A/B in
`phase3-warrant-chain-A-B-aggressive-2026-05-10.md`, which found
Phase 3 firing 0/117 cells even with 12/13 questions retrieving
chain chunks. Refined hypothesis: Phase 3 is a **rescue mechanism
that fires only when lexical `warrant_check` fails**, and on
well-formed axiom questions the LLM produces lexically-anchored
output, so warrant_check passes via the cheap path.
This investigation tested whether paraphrase-eliciting fixture
shapes can drive Phase 3 to fire empirically.
## Key finding from `arborist/qa/warrant.py:507`
`warrant_check` **vacuous-passes** unless the question shape is one
of:
- relation (proper-noun anchors via `is_relation_question`)
- date (4-digit year in claim)
- entity-list (multi-entity question shape)
- count (count-shape question)
- why-cause (cause anchors gated on why-shape question)
If none of these shapes apply, `warrant_check` returns
`(True, [])` trivially — no `WARRANT_MISSING` to suppress, Phase 3
dormant by design.
## Fixture iteration probe results
| iteration | strategy | chain hits | Phase 3 fires |
|---|---|---|---|
| v1 (axiom titles) | "What is X?" / "Define X" | 12/13 | 0/117 |
| v2 (paraphrase wrappers) | "How would you use X to ..." / "In your own words ..." | 1/10 | n/a (didn't bench) |
| v2 (title + paraphrase suffix) | "X: explain in plain English ..." | 2/13 | n/a (didn't bench) |
| v3 (warrant-shape wrappers) | "Who proposed X?" / "Why does X imply Y?" | 1/8 | n/a |
| v3 (title-leading + warrant suffix) | "X: who first stated it?" | 1/8 | n/a |
**Pattern observed across all iterations:**
- Title-leading retrieval works (high title-token boost on chain
chunks) **but** the question is definitional — no warrant shape,
no Phase 3 trigger.
- Warrant-shape questions ("who", "why", "when") trigger
`warrant_check` **but** retrieval defaults to Wikipedia prose
articles which have biographical/historical context the
terse claim-pack chunks lack.
## Structural reason Phase 3 is empirically dormant
Claim-pack chunks (the 92 chain-root documents) are **terse and
title-driven**. They state the axiom or theorem precisely with
mathematical formalism but contain no biographical / historical /
explanatory prose (no "first proven by Dirichlet in 1834", no
"motivated by Cantor's earlier work on", no "this rules out
because of...").
This means warrant-shape questions ("who", "when", "why") will
preferentially retrieve from Wikipedia, where the prose context
matches the question shape. Phase 3's data condition (chain-backed
chunk cited for a warrant-shape claim) is empirically unreachable
through normal retrieval pathways on this corpus shape.
## What would make Phase 3 fire
Three paths, none easy:
1. **Fortify claim-pack chunks** with biographical / historical
prose so warrant-shape questions retrieve chain chunks. This
bloats the chunks and dilutes their canonical-form value.
2. **Force-cite via prompt control** — include a directive in the
system prompt that tells the LLM "if a chain-root chunk is in
the evidence set, cite it for at least one claim." Forces the
data condition. Bench-only / artificial.
3. **Cross-source compound claims** — questions where one claim
should cite a chain-root chunk (definitional anchor) AND another
claim has warrant-shape elements (who/when/why). The LLM
typically produces multi-claim answers in claim_lattice modes,
so this could happen organically. Hard to engineer reliably in
a fixture.
## Verdict
Phase 3's mechanism is correct, non-regressing, and well-tested in
unit tests (`tests/test_warrant_chain.py`). On the current corpus,
its **runtime fire rate is structurally bounded near zero** because
the data condition (warrant-shape claim citing chain-backed chunk)
intersects rarely.
This is consistent with Phase 3's design intent: a defensive rescue
path for the rare claim that survives generation but fails lexical
anchoring. The lexical layer is strong enough on the current corpus
shape that the rescue isn't needed at measurable rates.
**Recommendation**: don't try to engineer Phase 3 fires via fixture
design. Track fire rate as a corpus-evolution signal — if claim-pack
content gains prose context (or the corpus mix changes), Phase 3
will start firing organically and the audit-line tail
(`· warrant proven via chain ×N`) will surface it. Until then,
the mechanism is dormant-but-correct.
## Files
- v1 fixture: `bench/qa_questions_warrant_chain_probe.txt`
- v1 A/B: `bench/results/phase3-warrant-chain-A-B-2026-05-10.md`
- v2 fixture: `bench/qa_questions_warrant_chain_aggressive.txt`
- v2 A/B: `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md`
- v3 fixture (this investigation): `bench/qa_questions_warrant_chain_paraphrase.txt`
- This report: `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md`