diff --git a/bench/qa_questions_warrant_chain_paraphrase.txt b/bench/qa_questions_warrant_chain_paraphrase.txt new file mode 100644 index 0000000..e028132 --- /dev/null +++ b/bench/qa_questions_warrant_chain_paraphrase.txt @@ -0,0 +1,37 @@ +# Paraphrase-eliciting warrant-chain fixture (2026-05-10, v2). +# +# v1 of this fixture used natural-language wrappers ("How would you +# use the Pigeonhole Principle to ..."). Pre-bench probe found only +# 1/10 questions hit chain — the wrapper words ("how would you use", +# "in your own words") match Wikipedia / general-knowledge articles +# better than concise claim-pack chunks, pushing retrieval AWAY +# from chain. +# +# v2 strategy: **title-leading + paraphrase suffix**. Each question +# starts with the exact claim-pack title (preserving title-token +# boost) but ends with a paraphrase-eliciting clause that should +# steer the LLM toward reformulation. Title leads retrieval; suffix +# leads generation. + +# Pillar VII +Pigeonhole Principle: explain in plain English with a 13-people example. +Pascal's Rule: rephrase as a counting story. +Multiplication Principle: describe informally with a real-world example. + +# Pillar I +Axiom of Implication Introduction: explain in your own words. +Axiom of Universal Specification: rephrase using simpler vocabulary. +Axiom of Reflexivity of Equality: describe informally. + +# Pillar II +Axiom of Extensionality: rephrase using only common words. +Axiom of Empty Set: describe in plain English. +Axiom of Power Set: explain informally with no jargon. +Axiom of Foundation: rephrase as a rule about sets. + +# Pillar III +Axiom of Successor: describe using only counting words. +Axiom of Addition (Zero): explain in plain language. + +# Pillar IV +Archimedean Axiom: rephrase using simpler measurement vocabulary. diff --git a/bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md b/bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md new file mode 100644 index 0000000..968fbc8 --- /dev/null +++ b/bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md @@ -0,0 +1,109 @@ +# #000031 Phase 3 — paraphrase fixture investigation (2026-05-10) + +**Sequel to** the aggressive A/B in +`phase3-warrant-chain-A-B-aggressive-2026-05-10.md`, which found +Phase 3 firing 0/117 cells even with 12/13 questions retrieving +chain chunks. Refined hypothesis: Phase 3 is a **rescue mechanism +that fires only when lexical `warrant_check` fails**, and on +well-formed axiom questions the LLM produces lexically-anchored +output, so warrant_check passes via the cheap path. + +This investigation tested whether paraphrase-eliciting fixture +shapes can drive Phase 3 to fire empirically. + +## Key finding from `arborist/qa/warrant.py:507` + +`warrant_check` **vacuous-passes** unless the question shape is one +of: + +- relation (proper-noun anchors via `is_relation_question`) +- date (4-digit year in claim) +- entity-list (multi-entity question shape) +- count (count-shape question) +- why-cause (cause anchors gated on why-shape question) + +If none of these shapes apply, `warrant_check` returns +`(True, [])` trivially — no `WARRANT_MISSING` to suppress, Phase 3 +dormant by design. + +## Fixture iteration probe results + +| iteration | strategy | chain hits | Phase 3 fires | +|---|---|---|---| +| v1 (axiom titles) | "What is X?" / "Define X" | 12/13 | 0/117 | +| v2 (paraphrase wrappers) | "How would you use X to ..." / "In your own words ..." | 1/10 | n/a (didn't bench) | +| v2 (title + paraphrase suffix) | "X: explain in plain English ..." | 2/13 | n/a (didn't bench) | +| v3 (warrant-shape wrappers) | "Who proposed X?" / "Why does X imply Y?" | 1/8 | n/a | +| v3 (title-leading + warrant suffix) | "X: who first stated it?" | 1/8 | n/a | + +**Pattern observed across all iterations:** + +- Title-leading retrieval works (high title-token boost on chain + chunks) **but** the question is definitional — no warrant shape, + no Phase 3 trigger. +- Warrant-shape questions ("who", "why", "when") trigger + `warrant_check` **but** retrieval defaults to Wikipedia prose + articles which have biographical/historical context the + terse claim-pack chunks lack. + +## Structural reason Phase 3 is empirically dormant + +Claim-pack chunks (the 92 chain-root documents) are **terse and +title-driven**. They state the axiom or theorem precisely with +mathematical formalism but contain no biographical / historical / +explanatory prose (no "first proven by Dirichlet in 1834", no +"motivated by Cantor's earlier work on", no "this rules out +because of..."). + +This means warrant-shape questions ("who", "when", "why") will +preferentially retrieve from Wikipedia, where the prose context +matches the question shape. Phase 3's data condition (chain-backed +chunk cited for a warrant-shape claim) is empirically unreachable +through normal retrieval pathways on this corpus shape. + +## What would make Phase 3 fire + +Three paths, none easy: + +1. **Fortify claim-pack chunks** with biographical / historical + prose so warrant-shape questions retrieve chain chunks. This + bloats the chunks and dilutes their canonical-form value. +2. **Force-cite via prompt control** — include a directive in the + system prompt that tells the LLM "if a chain-root chunk is in + the evidence set, cite it for at least one claim." Forces the + data condition. Bench-only / artificial. +3. **Cross-source compound claims** — questions where one claim + should cite a chain-root chunk (definitional anchor) AND another + claim has warrant-shape elements (who/when/why). The LLM + typically produces multi-claim answers in claim_lattice modes, + so this could happen organically. Hard to engineer reliably in + a fixture. + +## Verdict + +Phase 3's mechanism is correct, non-regressing, and well-tested in +unit tests (`tests/test_warrant_chain.py`). On the current corpus, +its **runtime fire rate is structurally bounded near zero** because +the data condition (warrant-shape claim citing chain-backed chunk) +intersects rarely. + +This is consistent with Phase 3's design intent: a defensive rescue +path for the rare claim that survives generation but fails lexical +anchoring. The lexical layer is strong enough on the current corpus +shape that the rescue isn't needed at measurable rates. + +**Recommendation**: don't try to engineer Phase 3 fires via fixture +design. Track fire rate as a corpus-evolution signal — if claim-pack +content gains prose context (or the corpus mix changes), Phase 3 +will start firing organically and the audit-line tail +(`· warrant proven via chain ×N`) will surface it. Until then, +the mechanism is dormant-but-correct. + +## Files + +- v1 fixture: `bench/qa_questions_warrant_chain_probe.txt` +- v1 A/B: `bench/results/phase3-warrant-chain-A-B-2026-05-10.md` +- v2 fixture: `bench/qa_questions_warrant_chain_aggressive.txt` +- v2 A/B: `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md` +- v3 fixture (this investigation): `bench/qa_questions_warrant_chain_paraphrase.txt` +- This report: `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md`