diff --git a/docs/tickets/ticket-000006-bench-emergent-findings.md b/docs/tickets/ticket-000006-bench-emergent-findings.md index 75d4cbf..06193b5 100644 --- a/docs/tickets/ticket-000006-bench-emergent-findings.md +++ b/docs/tickets/ticket-000006-bench-emergent-findings.md @@ -618,3 +618,138 @@ Re-runnable via `make bench-witness-sweep`. `bench/fixtures/5f/falsification-witness-v1.jsonl` — extracted divergence fixtures (10 rows after two extraction passes today). + +## Amend — 2026-05-10 (Phase 3 warrant-chain dormancy: structurally rescue-only) + +A third qualitatively different experimental shape, parallel to +the 2026-05-09 witness-sweep widening. The earlier amends measured +honesty under no ground truth (random-word) and capability under +canonical ground truth (witness-sweep). This amend measures +**warrant-chain rescue activation under chain-backed ground truth** +— the #000031 Phase 3 verifier (`a57b194`) running against fixtures +explicitly targeting the 92 claim-pack-warrant-resolved records. + +### Setup + +#000031 Phase 3 (warrant-chain-aware verifier) landed 2026-05-10. +Mechanism: when the LLM cites a chunk that has chain-root warrant +backing AND the lexical `warrant_check` would otherwise emit +`WARRANT_MISSING`, Phase 3 suppresses that violation by promoting +via the chain. + +Three sequential fixture iterations probed whether the mechanism +fires empirically on the live corpus: + +1. **Live validation** (`1d2a558`) — 7-question parallel-shift + probe targeting chain-root documents. +2. **Aggressive A/B** (`c5bc53f`) — 13 questions using exact + claim-pack title phrasing; n=3, 117 cells per condition. +3. **Paraphrase investigation** (`a9fcb42`) — 5 fixture iterations + each probed-then-benched, varying question shape (definitional, + how-use, plain-English, who-proposed, who-first-stated). + +### Aggregate + +| Investigation | Chain-root retrieval | Phase 3 fires | STRICT-rate Δ | +|---|---|---|---| +| Live validation | 6/7 → EVIDENCE-WARRANTED | n/a (smoke) | n/a (no A/B) | +| Aggressive A/B | 12/13 questions hit chain roots | **0 / 117 cells** | +0 (85/117 → 85/117) | +| Paraphrase v1-v3' | 12/13 → 1/8 (varies) | **0** across all iterations | not measured | + +`cache_key parity 39/39` on aggressive A/B — Phase 3 is +mechanically observable but produces zero fires. + +### Root cause — `arborist/qa/warrant.py:507` + +`warrant_check` **vacuous-passes** (returns `(True, [])` without +anchor extraction) unless the question shape is one of: + +```text +relation — proper-noun anchors via is_relation_question +date — 4-digit year in claim +entity-list — multi-entity question shape +count — count-shape question +why-cause — cause anchors gated on why-shape question +``` + +Definitional questions ("what is X", "define X") don't have +warrant shape → no `WARRANT_MISSING` to suppress → Phase 3 has +nothing to do. + +### Structural reason for the dormancy + +Claim-pack chunks (the 92 chain-root documents) are deliberately +**terse** — they state axioms / theorems precisely with +mathematical formalism but contain no biographical, historical, +or explanatory prose. So warrant-shape questions ("who" / "when" +/ "why") retrieve from Wikipedia (which has matching prose +context) instead of from claim-pack chunks. + +The data condition Phase 3 needs — chain-backed chunk cited for a +warrant-shape claim — is **empirically unreachable through normal +retrieval pathways on this corpus shape**. + +### Verdict + +Phase 3's runtime fire rate is structurally bounded near zero on +the current corpus. **Mechanism is correct, non-regressing, +unit-tested** (`tests/test_warrant_chain.py`). + +Two readings, both true: + +1. **The lexical layer is strong enough** on clean axiom questions + that the chain-fallback isn't needed. The mechanism that lands + warrants on cheap lexical evidence does most of the work. +2. **Phase 3 is a latent capability.** It will activate when corpus + shape shifts — claim-pack content gaining prose context, or the + corpus mix changing toward warrant-shape questions. The audit- + line tail `· warrant proven via chain ×N` will surface fires + when they occur. + +### Recommendation (per the investigation's own conclusion) + +Don't engineer Phase 3 fires via fixture design. Track fire rate +as a **corpus-evolution signal**. If claim-pack content gains +prose context or the corpus mix changes, Phase 3 will start firing +organically and the audit-line tail will surface it. + +This closes the three-thread investigation cleanly. The fixtures ++ reports remain as the empirical trail for future shifts. + +### Adjacent landings (same-day infrastructure) + +Two cryptographic-primitive Phase 1 deliverables landed +2026-05-10 ahead of v7 plastic-training deployment: + +- **#000034 Phase 1a** (`1dfb8b9`): Hessian-alignment probe at + `bench/scripts/phi_alignment_probe.py` + 14 tests + + 10-vector KAT fixture at + `bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl`. +- **#000035 Phase 1** (earlier today): φ_PRG reference impl at + `arborist/substrate/anchor_prg.py` (HMAC-SHA-512 counter-mode + KDF) + 20 tests + 10-vector KAT fixture at + `bench/fixtures/phi-prg/known-answer-tests.jsonl`. + +Both are KAT-pinned + bench-pinned regression artifacts that +will surface signal the moment a real v7 checkpoint gives the +probes something to measure. Soft-hash analysis §9.1 + §9.2 +were updated to point at these landings (`28b57da`, `0b038f1`). + +### Distinct signal vs prior amends + +| Source | Catches | Repair path | +|---|---|---| +| Random-word emergent | honesty failures (false-STRICTs) | verifier ladder hardening | +| Witness-sweep | capability failures (LLM gives wrong answer) | prompt eng or fine-tune | +| Phase 3 warrant-chain | rescue-mechanism activation conditions | corpus shape evolution (NOT fixture engineering) | + +### Bench artifacts + +- `bench/results/phase3-live-validation-2026-05-10.md` +- `bench/results/phase3-warrant-chain-A-B-2026-05-10.md` (initial) +- `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md` +- `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md` + +Re-runnable via the bench-emergent harness when corpus shape +shifts and a measurable fire rate is expected. +