From a9fcb4251d91d2353cd5524184e3cbabc35f57e1 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 10 May 2026 10:06:23 -0400 Subject: [PATCH] =?UTF-8?q?bench:=20Phase=203=20paraphrase=20fixture=20inv?= =?UTF-8?q?estigation=20=E2=80=94=20empirically=20dormant=20on=20current?= =?UTF-8?q?=20corpus?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three-iteration investigation following c5bc53f's "Phase 3 is a rescue mechanism, not a default path" finding. Question: can a paraphrase- eliciting fixture actually drive Phase 3 to fire on real questions? Key reading from arborist/qa/warrant.py:507 ("warrant_check"): the function vacuous-passes (returns (True, []) without any anchor extraction) unless the question shape is one of: - relation (proper-noun anchors) - date (4-digit year in claim) - entity-list - count-shape - why-cause-shape If none of those shapes apply, Phase 3 has nothing to suppress. Three fixture iterations, all probed before benching: | iter | strategy | chain hits | |------|-----------------------------------|------------| | v1 | "What is X?" / "Define X" | 12/13 | | v2 | "How would you use X to ..." | 1/10 | | v2' | "X: explain in plain English" | 2/13 | | v3 | "Who proposed X?" / "Why does X" | 1/8 | | v3' | "X: who first stated it?" | 1/8 | Pattern: title-leading retrieval works (high title-token boost on chain chunks) but those questions are definitional — no warrant shape, no Phase 3 trigger. Warrant-shape questions ("who" / "when" / "why") trigger warrant_check but retrieval pulls toward Wikipedia's prose articles instead of the terse claim-pack chunks. Structural reason: claim-pack chunks (the 92 chain-root documents) are deliberately terse — they state axioms / theorems precisely with mathematical formalism but contain no biographical, historical, or explanatory prose. So warrant-shape questions retrieve from Wikipedia, where the prose context matches the question shape. Phase 3's data condition (chain-backed chunk cited for a warrant-shape claim) is empirically unreachable through normal retrieval pathways on this corpus shape. Verdict (`bench/results/phase3-warrant-chain-paraphrase- investigation-2026-05-10.md`): Phase 3's runtime fire rate is structurally bounded near zero on the current corpus. Mechanism is correct, non-regressing, and unit-tested (tests/test_warrant_chain.py). The lexical layer is strong enough that the rescue isn't needed at measurable rates. Recommendation: don't engineer Phase 3 fires via fixture design. Track fire rate as a corpus-evolution signal. If claim-pack content gains prose context or the corpus mix changes, Phase 3 will start firing organically and the audit-line tail (`· warrant proven via chain ×N`) will surface it. This closes the three-thread investigation cleanly. v1 + v2 + v3 fixtures + reports all retained as the empirical trail. --- .../qa_questions_warrant_chain_paraphrase.txt | 37 ++++++ ...ain-paraphrase-investigation-2026-05-10.md | 109 ++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 bench/qa_questions_warrant_chain_paraphrase.txt create mode 100644 bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md diff --git a/bench/qa_questions_warrant_chain_paraphrase.txt b/bench/qa_questions_warrant_chain_paraphrase.txt new file mode 100644 index 0000000..e028132 --- /dev/null +++ b/bench/qa_questions_warrant_chain_paraphrase.txt @@ -0,0 +1,37 @@ +# Paraphrase-eliciting warrant-chain fixture (2026-05-10, v2). +# +# v1 of this fixture used natural-language wrappers ("How would you +# use the Pigeonhole Principle to ..."). Pre-bench probe found only +# 1/10 questions hit chain — the wrapper words ("how would you use", +# "in your own words") match Wikipedia / general-knowledge articles +# better than concise claim-pack chunks, pushing retrieval AWAY +# from chain. +# +# v2 strategy: **title-leading + paraphrase suffix**. Each question +# starts with the exact claim-pack title (preserving title-token +# boost) but ends with a paraphrase-eliciting clause that should +# steer the LLM toward reformulation. Title leads retrieval; suffix +# leads generation. + +# Pillar VII +Pigeonhole Principle: explain in plain English with a 13-people example. +Pascal's Rule: rephrase as a counting story. +Multiplication Principle: describe informally with a real-world example. + +# Pillar I +Axiom of Implication Introduction: explain in your own words. +Axiom of Universal Specification: rephrase using simpler vocabulary. +Axiom of Reflexivity of Equality: describe informally. + +# Pillar II +Axiom of Extensionality: rephrase using only common words. +Axiom of Empty Set: describe in plain English. +Axiom of Power Set: explain informally with no jargon. +Axiom of Foundation: rephrase as a rule about sets. + +# Pillar III +Axiom of Successor: describe using only counting words. +Axiom of Addition (Zero): explain in plain language. + +# Pillar IV +Archimedean Axiom: rephrase using simpler measurement vocabulary. diff --git a/bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md b/bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md new file mode 100644 index 0000000..968fbc8 --- /dev/null +++ b/bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md @@ -0,0 +1,109 @@ +# #000031 Phase 3 — paraphrase fixture investigation (2026-05-10) + +**Sequel to** the aggressive A/B in +`phase3-warrant-chain-A-B-aggressive-2026-05-10.md`, which found +Phase 3 firing 0/117 cells even with 12/13 questions retrieving +chain chunks. Refined hypothesis: Phase 3 is a **rescue mechanism +that fires only when lexical `warrant_check` fails**, and on +well-formed axiom questions the LLM produces lexically-anchored +output, so warrant_check passes via the cheap path. + +This investigation tested whether paraphrase-eliciting fixture +shapes can drive Phase 3 to fire empirically. + +## Key finding from `arborist/qa/warrant.py:507` + +`warrant_check` **vacuous-passes** unless the question shape is one +of: + +- relation (proper-noun anchors via `is_relation_question`) +- date (4-digit year in claim) +- entity-list (multi-entity question shape) +- count (count-shape question) +- why-cause (cause anchors gated on why-shape question) + +If none of these shapes apply, `warrant_check` returns +`(True, [])` trivially — no `WARRANT_MISSING` to suppress, Phase 3 +dormant by design. + +## Fixture iteration probe results + +| iteration | strategy | chain hits | Phase 3 fires | +|---|---|---|---| +| v1 (axiom titles) | "What is X?" / "Define X" | 12/13 | 0/117 | +| v2 (paraphrase wrappers) | "How would you use X to ..." / "In your own words ..." | 1/10 | n/a (didn't bench) | +| v2 (title + paraphrase suffix) | "X: explain in plain English ..." | 2/13 | n/a (didn't bench) | +| v3 (warrant-shape wrappers) | "Who proposed X?" / "Why does X imply Y?" | 1/8 | n/a | +| v3 (title-leading + warrant suffix) | "X: who first stated it?" | 1/8 | n/a | + +**Pattern observed across all iterations:** + +- Title-leading retrieval works (high title-token boost on chain + chunks) **but** the question is definitional — no warrant shape, + no Phase 3 trigger. +- Warrant-shape questions ("who", "why", "when") trigger + `warrant_check` **but** retrieval defaults to Wikipedia prose + articles which have biographical/historical context the + terse claim-pack chunks lack. + +## Structural reason Phase 3 is empirically dormant + +Claim-pack chunks (the 92 chain-root documents) are **terse and +title-driven**. They state the axiom or theorem precisely with +mathematical formalism but contain no biographical / historical / +explanatory prose (no "first proven by Dirichlet in 1834", no +"motivated by Cantor's earlier work on", no "this rules out +because of..."). + +This means warrant-shape questions ("who", "when", "why") will +preferentially retrieve from Wikipedia, where the prose context +matches the question shape. Phase 3's data condition (chain-backed +chunk cited for a warrant-shape claim) is empirically unreachable +through normal retrieval pathways on this corpus shape. + +## What would make Phase 3 fire + +Three paths, none easy: + +1. **Fortify claim-pack chunks** with biographical / historical + prose so warrant-shape questions retrieve chain chunks. This + bloats the chunks and dilutes their canonical-form value. +2. **Force-cite via prompt control** — include a directive in the + system prompt that tells the LLM "if a chain-root chunk is in + the evidence set, cite it for at least one claim." Forces the + data condition. Bench-only / artificial. +3. **Cross-source compound claims** — questions where one claim + should cite a chain-root chunk (definitional anchor) AND another + claim has warrant-shape elements (who/when/why). The LLM + typically produces multi-claim answers in claim_lattice modes, + so this could happen organically. Hard to engineer reliably in + a fixture. + +## Verdict + +Phase 3's mechanism is correct, non-regressing, and well-tested in +unit tests (`tests/test_warrant_chain.py`). On the current corpus, +its **runtime fire rate is structurally bounded near zero** because +the data condition (warrant-shape claim citing chain-backed chunk) +intersects rarely. + +This is consistent with Phase 3's design intent: a defensive rescue +path for the rare claim that survives generation but fails lexical +anchoring. The lexical layer is strong enough on the current corpus +shape that the rescue isn't needed at measurable rates. + +**Recommendation**: don't try to engineer Phase 3 fires via fixture +design. Track fire rate as a corpus-evolution signal — if claim-pack +content gains prose context (or the corpus mix changes), Phase 3 +will start firing organically and the audit-line tail +(`· warrant proven via chain ×N`) will surface it. Until then, +the mechanism is dormant-but-correct. + +## Files + +- v1 fixture: `bench/qa_questions_warrant_chain_probe.txt` +- v1 A/B: `bench/results/phase3-warrant-chain-A-B-2026-05-10.md` +- v2 fixture: `bench/qa_questions_warrant_chain_aggressive.txt` +- v2 A/B: `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md` +- v3 fixture (this investigation): `bench/qa_questions_warrant_chain_paraphrase.txt` +- This report: `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md`