russell@unturf.com
a9fcb4251d
Three-iteration investigation following c5bc53f's "Phase 3 is a rescue
mechanism, not a default path" finding. Question: can a paraphrase-
eliciting fixture actually drive Phase 3 to fire on real questions?
Key reading from arborist/qa/warrant.py:507 ("warrant_check"):
the function vacuous-passes (returns (True, []) without any anchor
extraction) unless the question shape is one of:
- relation (proper-noun anchors)
- date (4-digit year in claim)
- entity-list
- count-shape
- why-cause-shape
If none of those shapes apply, Phase 3 has nothing to suppress.
Three fixture iterations, all probed before benching:
| iter | strategy | chain hits |
|------|-----------------------------------|------------|
| v1 | "What is X?" / "Define X" | 12/13 |
| v2 | "How would you use X to ..." | 1/10 |
| v2' | "X: explain in plain English" | 2/13 |
| v3 | "Who proposed X?" / "Why does X" | 1/8 |
| v3' | "X: who first stated it?" | 1/8 |
Pattern: title-leading retrieval works (high title-token boost on
chain chunks) but those questions are definitional — no warrant
shape, no Phase 3 trigger. Warrant-shape questions ("who" / "when"
/ "why") trigger warrant_check but retrieval pulls toward
Wikipedia's prose articles instead of the terse claim-pack chunks.
Structural reason: claim-pack chunks (the 92 chain-root documents)
are deliberately terse — they state axioms / theorems precisely
with mathematical formalism but contain no biographical, historical,
or explanatory prose. So warrant-shape questions retrieve from
Wikipedia, where the prose context matches the question shape.
Phase 3's data condition (chain-backed chunk cited for a
warrant-shape claim) is empirically unreachable through normal
retrieval pathways on this corpus shape.
Verdict (`bench/results/phase3-warrant-chain-paraphrase-
investigation-2026-05-10.md`): Phase 3's runtime fire rate is
structurally bounded near zero on the current corpus. Mechanism
is correct, non-regressing, and unit-tested
(tests/test_warrant_chain.py). The lexical layer is strong enough
that the rescue isn't needed at measurable rates.
Recommendation: don't engineer Phase 3 fires via fixture design.
Track fire rate as a corpus-evolution signal. If claim-pack
content gains prose context or the corpus mix changes, Phase 3
will start firing organically and the audit-line tail
(`· warrant proven via chain ×N`) will surface it.
This closes the three-thread investigation cleanly. v1 + v2 + v3
fixtures + reports all retained as the empirical trail.
37 lines
1.5 KiB
Text
37 lines
1.5 KiB
Text
# Paraphrase-eliciting warrant-chain fixture (2026-05-10, v2).
|
|
#
|
|
# v1 of this fixture used natural-language wrappers ("How would you
|
|
# use the Pigeonhole Principle to ..."). Pre-bench probe found only
|
|
# 1/10 questions hit chain — the wrapper words ("how would you use",
|
|
# "in your own words") match Wikipedia / general-knowledge articles
|
|
# better than concise claim-pack chunks, pushing retrieval AWAY
|
|
# from chain.
|
|
#
|
|
# v2 strategy: **title-leading + paraphrase suffix**. Each question
|
|
# starts with the exact claim-pack title (preserving title-token
|
|
# boost) but ends with a paraphrase-eliciting clause that should
|
|
# steer the LLM toward reformulation. Title leads retrieval; suffix
|
|
# leads generation.
|
|
|
|
# Pillar VII
|
|
Pigeonhole Principle: explain in plain English with a 13-people example.
|
|
Pascal's Rule: rephrase as a counting story.
|
|
Multiplication Principle: describe informally with a real-world example.
|
|
|
|
# Pillar I
|
|
Axiom of Implication Introduction: explain in your own words.
|
|
Axiom of Universal Specification: rephrase using simpler vocabulary.
|
|
Axiom of Reflexivity of Equality: describe informally.
|
|
|
|
# Pillar II
|
|
Axiom of Extensionality: rephrase using only common words.
|
|
Axiom of Empty Set: describe in plain English.
|
|
Axiom of Power Set: explain informally with no jargon.
|
|
Axiom of Foundation: rephrase as a rule about sets.
|
|
|
|
# Pillar III
|
|
Axiom of Successor: describe using only counting words.
|
|
Axiom of Addition (Zero): explain in plain language.
|
|
|
|
# Pillar IV
|
|
Archimedean Axiom: rephrase using simpler measurement vocabulary.
|