bench: Phase 3 paraphrase fixture investigation — empirically dormant on current corpus
Three-iteration investigation following c5bc53f's "Phase 3 is a rescue
mechanism, not a default path" finding. Question: can a paraphrase-
eliciting fixture actually drive Phase 3 to fire on real questions?
Key reading from arborist/qa/warrant.py:507 ("warrant_check"):
the function vacuous-passes (returns (True, []) without any anchor
extraction) unless the question shape is one of:
- relation (proper-noun anchors)
- date (4-digit year in claim)
- entity-list
- count-shape
- why-cause-shape
If none of those shapes apply, Phase 3 has nothing to suppress.
Three fixture iterations, all probed before benching:
| iter | strategy | chain hits |
|------|-----------------------------------|------------|
| v1 | "What is X?" / "Define X" | 12/13 |
| v2 | "How would you use X to ..." | 1/10 |
| v2' | "X: explain in plain English" | 2/13 |
| v3 | "Who proposed X?" / "Why does X" | 1/8 |
| v3' | "X: who first stated it?" | 1/8 |
Pattern: title-leading retrieval works (high title-token boost on
chain chunks) but those questions are definitional — no warrant
shape, no Phase 3 trigger. Warrant-shape questions ("who" / "when"
/ "why") trigger warrant_check but retrieval pulls toward
Wikipedia's prose articles instead of the terse claim-pack chunks.
Structural reason: claim-pack chunks (the 92 chain-root documents)
are deliberately terse — they state axioms / theorems precisely
with mathematical formalism but contain no biographical, historical,
or explanatory prose. So warrant-shape questions retrieve from
Wikipedia, where the prose context matches the question shape.
Phase 3's data condition (chain-backed chunk cited for a
warrant-shape claim) is empirically unreachable through normal
retrieval pathways on this corpus shape.
Verdict (`bench/results/phase3-warrant-chain-paraphrase-
investigation-2026-05-10.md`): Phase 3's runtime fire rate is
structurally bounded near zero on the current corpus. Mechanism
is correct, non-regressing, and unit-tested
(tests/test_warrant_chain.py). The lexical layer is strong enough
that the rescue isn't needed at measurable rates.
Recommendation: don't engineer Phase 3 fires via fixture design.
Track fire rate as a corpus-evolution signal. If claim-pack
content gains prose context or the corpus mix changes, Phase 3
will start firing organically and the audit-line tail
(`· warrant proven via chain ×N`) will surface it.
This closes the three-thread investigation cleanly. v1 + v2 + v3
fixtures + reports all retained as the empirical trail.
This commit is contained in:
parent
c5bc53f733
commit
a9fcb4251d
2 changed files with 146 additions and 0 deletions
37
bench/qa_questions_warrant_chain_paraphrase.txt
Normal file
37
bench/qa_questions_warrant_chain_paraphrase.txt
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# Paraphrase-eliciting warrant-chain fixture (2026-05-10, v2).
|
||||
#
|
||||
# v1 of this fixture used natural-language wrappers ("How would you
|
||||
# use the Pigeonhole Principle to ..."). Pre-bench probe found only
|
||||
# 1/10 questions hit chain — the wrapper words ("how would you use",
|
||||
# "in your own words") match Wikipedia / general-knowledge articles
|
||||
# better than concise claim-pack chunks, pushing retrieval AWAY
|
||||
# from chain.
|
||||
#
|
||||
# v2 strategy: **title-leading + paraphrase suffix**. Each question
|
||||
# starts with the exact claim-pack title (preserving title-token
|
||||
# boost) but ends with a paraphrase-eliciting clause that should
|
||||
# steer the LLM toward reformulation. Title leads retrieval; suffix
|
||||
# leads generation.
|
||||
|
||||
# Pillar VII
|
||||
Pigeonhole Principle: explain in plain English with a 13-people example.
|
||||
Pascal's Rule: rephrase as a counting story.
|
||||
Multiplication Principle: describe informally with a real-world example.
|
||||
|
||||
# Pillar I
|
||||
Axiom of Implication Introduction: explain in your own words.
|
||||
Axiom of Universal Specification: rephrase using simpler vocabulary.
|
||||
Axiom of Reflexivity of Equality: describe informally.
|
||||
|
||||
# Pillar II
|
||||
Axiom of Extensionality: rephrase using only common words.
|
||||
Axiom of Empty Set: describe in plain English.
|
||||
Axiom of Power Set: explain informally with no jargon.
|
||||
Axiom of Foundation: rephrase as a rule about sets.
|
||||
|
||||
# Pillar III
|
||||
Axiom of Successor: describe using only counting words.
|
||||
Axiom of Addition (Zero): explain in plain language.
|
||||
|
||||
# Pillar IV
|
||||
Archimedean Axiom: rephrase using simpler measurement vocabulary.
|
||||
|
|
@ -0,0 +1,109 @@
|
|||
# #000031 Phase 3 — paraphrase fixture investigation (2026-05-10)
|
||||
|
||||
**Sequel to** the aggressive A/B in
|
||||
`phase3-warrant-chain-A-B-aggressive-2026-05-10.md`, which found
|
||||
Phase 3 firing 0/117 cells even with 12/13 questions retrieving
|
||||
chain chunks. Refined hypothesis: Phase 3 is a **rescue mechanism
|
||||
that fires only when lexical `warrant_check` fails**, and on
|
||||
well-formed axiom questions the LLM produces lexically-anchored
|
||||
output, so warrant_check passes via the cheap path.
|
||||
|
||||
This investigation tested whether paraphrase-eliciting fixture
|
||||
shapes can drive Phase 3 to fire empirically.
|
||||
|
||||
## Key finding from `arborist/qa/warrant.py:507`
|
||||
|
||||
`warrant_check` **vacuous-passes** unless the question shape is one
|
||||
of:
|
||||
|
||||
- relation (proper-noun anchors via `is_relation_question`)
|
||||
- date (4-digit year in claim)
|
||||
- entity-list (multi-entity question shape)
|
||||
- count (count-shape question)
|
||||
- why-cause (cause anchors gated on why-shape question)
|
||||
|
||||
If none of these shapes apply, `warrant_check` returns
|
||||
`(True, [])` trivially — no `WARRANT_MISSING` to suppress, Phase 3
|
||||
dormant by design.
|
||||
|
||||
## Fixture iteration probe results
|
||||
|
||||
| iteration | strategy | chain hits | Phase 3 fires |
|
||||
|---|---|---|---|
|
||||
| v1 (axiom titles) | "What is X?" / "Define X" | 12/13 | 0/117 |
|
||||
| v2 (paraphrase wrappers) | "How would you use X to ..." / "In your own words ..." | 1/10 | n/a (didn't bench) |
|
||||
| v2 (title + paraphrase suffix) | "X: explain in plain English ..." | 2/13 | n/a (didn't bench) |
|
||||
| v3 (warrant-shape wrappers) | "Who proposed X?" / "Why does X imply Y?" | 1/8 | n/a |
|
||||
| v3 (title-leading + warrant suffix) | "X: who first stated it?" | 1/8 | n/a |
|
||||
|
||||
**Pattern observed across all iterations:**
|
||||
|
||||
- Title-leading retrieval works (high title-token boost on chain
|
||||
chunks) **but** the question is definitional — no warrant shape,
|
||||
no Phase 3 trigger.
|
||||
- Warrant-shape questions ("who", "why", "when") trigger
|
||||
`warrant_check` **but** retrieval defaults to Wikipedia prose
|
||||
articles which have biographical/historical context the
|
||||
terse claim-pack chunks lack.
|
||||
|
||||
## Structural reason Phase 3 is empirically dormant
|
||||
|
||||
Claim-pack chunks (the 92 chain-root documents) are **terse and
|
||||
title-driven**. They state the axiom or theorem precisely with
|
||||
mathematical formalism but contain no biographical / historical /
|
||||
explanatory prose (no "first proven by Dirichlet in 1834", no
|
||||
"motivated by Cantor's earlier work on", no "this rules out
|
||||
because of...").
|
||||
|
||||
This means warrant-shape questions ("who", "when", "why") will
|
||||
preferentially retrieve from Wikipedia, where the prose context
|
||||
matches the question shape. Phase 3's data condition (chain-backed
|
||||
chunk cited for a warrant-shape claim) is empirically unreachable
|
||||
through normal retrieval pathways on this corpus shape.
|
||||
|
||||
## What would make Phase 3 fire
|
||||
|
||||
Three paths, none easy:
|
||||
|
||||
1. **Fortify claim-pack chunks** with biographical / historical
|
||||
prose so warrant-shape questions retrieve chain chunks. This
|
||||
bloats the chunks and dilutes their canonical-form value.
|
||||
2. **Force-cite via prompt control** — include a directive in the
|
||||
system prompt that tells the LLM "if a chain-root chunk is in
|
||||
the evidence set, cite it for at least one claim." Forces the
|
||||
data condition. Bench-only / artificial.
|
||||
3. **Cross-source compound claims** — questions where one claim
|
||||
should cite a chain-root chunk (definitional anchor) AND another
|
||||
claim has warrant-shape elements (who/when/why). The LLM
|
||||
typically produces multi-claim answers in claim_lattice modes,
|
||||
so this could happen organically. Hard to engineer reliably in
|
||||
a fixture.
|
||||
|
||||
## Verdict
|
||||
|
||||
Phase 3's mechanism is correct, non-regressing, and well-tested in
|
||||
unit tests (`tests/test_warrant_chain.py`). On the current corpus,
|
||||
its **runtime fire rate is structurally bounded near zero** because
|
||||
the data condition (warrant-shape claim citing chain-backed chunk)
|
||||
intersects rarely.
|
||||
|
||||
This is consistent with Phase 3's design intent: a defensive rescue
|
||||
path for the rare claim that survives generation but fails lexical
|
||||
anchoring. The lexical layer is strong enough on the current corpus
|
||||
shape that the rescue isn't needed at measurable rates.
|
||||
|
||||
**Recommendation**: don't try to engineer Phase 3 fires via fixture
|
||||
design. Track fire rate as a corpus-evolution signal — if claim-pack
|
||||
content gains prose context (or the corpus mix changes), Phase 3
|
||||
will start firing organically and the audit-line tail
|
||||
(`· warrant proven via chain ×N`) will surface it. Until then,
|
||||
the mechanism is dormant-but-correct.
|
||||
|
||||
## Files
|
||||
|
||||
- v1 fixture: `bench/qa_questions_warrant_chain_probe.txt`
|
||||
- v1 A/B: `bench/results/phase3-warrant-chain-A-B-2026-05-10.md`
|
||||
- v2 fixture: `bench/qa_questions_warrant_chain_aggressive.txt`
|
||||
- v2 A/B: `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md`
|
||||
- v3 fixture (this investigation): `bench/qa_questions_warrant_chain_paraphrase.txt`
|
||||
- This report: `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md`
|
||||
Loading…
Add table
Add a link
Reference in a new issue