ticket #000006: 2026-05-10 amend — Phase 3 warrant-chain dormancy

Third qualitatively different experimental shape in the rolling
research log, parallel to the 2026-05-09 witness-sweep widening:

- random-word emergent (honesty under no ground truth)
- witness-sweep (capability under canonical ground truth)
- warrant-chain (rescue-mechanism activation under chain-backed
  ground truth)  ← new

Captures the structurally-grounded dormancy of #000031 Phase 3
verifier — 0/117 fires across an aggressive A/B with 12/13
chain-root retrieval, plus 5 paraphrase fixture iterations
none of which drive Phase 3 to fire empirically.

Root cause pinned at arborist/qa/warrant.py:507: warrant_check
vacuous-passes unless question shape is relation/date/entity-list/
count/why-cause. Definitional questions don't have warrant shape;
claim-pack chunks are terse axioms without prose context that
warrant-shape questions would retrieve. Phase 3 is structurally
rescue-only on the current corpus.

Mechanism correct + non-regressing + unit-tested. Recommendation:
track fire rate as a corpus-evolution signal, not a fixture-
engineering target. Audit-line tail will surface organic fires.

Adjacent same-day landings noted: #000034 Phase 1a (Hessian-
alignment probe) + #000035 Phase 1 (φ_PRG reference) — both
KAT-pinned cryptographic-primitive infrastructure ahead of v7
plastic-training deployment.

Doc-only update; no code change.
This commit is contained in:
russell@unturf.com 2026-05-10 13:22:30 -04:00
parent 7ea0fdf2b1
commit 2c683de6d8
No known key found for this signature in database

View file

@ -618,3 +618,138 @@ Re-runnable via `make bench-witness-sweep`.
`bench/fixtures/5f/falsification-witness-v1.jsonl` — extracted
divergence fixtures (10 rows after two extraction passes today).
## Amend — 2026-05-10 (Phase 3 warrant-chain dormancy: structurally rescue-only)
A third qualitatively different experimental shape, parallel to
the 2026-05-09 witness-sweep widening. The earlier amends measured
honesty under no ground truth (random-word) and capability under
canonical ground truth (witness-sweep). This amend measures
**warrant-chain rescue activation under chain-backed ground truth**
— the #000031 Phase 3 verifier (`a57b194`) running against fixtures
explicitly targeting the 92 claim-pack-warrant-resolved records.
### Setup
#000031 Phase 3 (warrant-chain-aware verifier) landed 2026-05-10.
Mechanism: when the LLM cites a chunk that has chain-root warrant
backing AND the lexical `warrant_check` would otherwise emit
`WARRANT_MISSING`, Phase 3 suppresses that violation by promoting
via the chain.
Three sequential fixture iterations probed whether the mechanism
fires empirically on the live corpus:
1. **Live validation** (`1d2a558`) — 7-question parallel-shift
probe targeting chain-root documents.
2. **Aggressive A/B** (`c5bc53f`) — 13 questions using exact
claim-pack title phrasing; n=3, 117 cells per condition.
3. **Paraphrase investigation** (`a9fcb42`) — 5 fixture iterations
each probed-then-benched, varying question shape (definitional,
how-use, plain-English, who-proposed, who-first-stated).
### Aggregate
| Investigation | Chain-root retrieval | Phase 3 fires | STRICT-rate Δ |
|---|---|---|---|
| Live validation | 6/7 → EVIDENCE-WARRANTED | n/a (smoke) | n/a (no A/B) |
| Aggressive A/B | 12/13 questions hit chain roots | **0 / 117 cells** | +0 (85/117 → 85/117) |
| Paraphrase v1-v3' | 12/13 → 1/8 (varies) | **0** across all iterations | not measured |
`cache_key parity 39/39` on aggressive A/B — Phase 3 is
mechanically observable but produces zero fires.
### Root cause — `arborist/qa/warrant.py:507`
`warrant_check` **vacuous-passes** (returns `(True, [])` without
anchor extraction) unless the question shape is one of:
```text
relation — proper-noun anchors via is_relation_question
date — 4-digit year in claim
entity-list — multi-entity question shape
count — count-shape question
why-cause — cause anchors gated on why-shape question
```
Definitional questions ("what is X", "define X") don't have
warrant shape → no `WARRANT_MISSING` to suppress → Phase 3 has
nothing to do.
### Structural reason for the dormancy
Claim-pack chunks (the 92 chain-root documents) are deliberately
**terse** — they state axioms / theorems precisely with
mathematical formalism but contain no biographical, historical,
or explanatory prose. So warrant-shape questions ("who" / "when"
/ "why") retrieve from Wikipedia (which has matching prose
context) instead of from claim-pack chunks.
The data condition Phase 3 needs — chain-backed chunk cited for a
warrant-shape claim — is **empirically unreachable through normal
retrieval pathways on this corpus shape**.
### Verdict
Phase 3's runtime fire rate is structurally bounded near zero on
the current corpus. **Mechanism is correct, non-regressing,
unit-tested** (`tests/test_warrant_chain.py`).
Two readings, both true:
1. **The lexical layer is strong enough** on clean axiom questions
that the chain-fallback isn't needed. The mechanism that lands
warrants on cheap lexical evidence does most of the work.
2. **Phase 3 is a latent capability.** It will activate when corpus
shape shifts — claim-pack content gaining prose context, or the
corpus mix changing toward warrant-shape questions. The audit-
line tail `· warrant proven via chain ×N` will surface fires
when they occur.
### Recommendation (per the investigation's own conclusion)
Don't engineer Phase 3 fires via fixture design. Track fire rate
as a **corpus-evolution signal**. If claim-pack content gains
prose context or the corpus mix changes, Phase 3 will start firing
organically and the audit-line tail will surface it.
This closes the three-thread investigation cleanly. The fixtures
+ reports remain as the empirical trail for future shifts.
### Adjacent landings (same-day infrastructure)
Two cryptographic-primitive Phase 1 deliverables landed
2026-05-10 ahead of v7 plastic-training deployment:
- **#000034 Phase 1a** (`1dfb8b9`): Hessian-alignment probe at
`bench/scripts/phi_alignment_probe.py` + 14 tests +
10-vector KAT fixture at
`bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl`.
- **#000035 Phase 1** (earlier today): φ_PRG reference impl at
`arborist/substrate/anchor_prg.py` (HMAC-SHA-512 counter-mode
KDF) + 20 tests + 10-vector KAT fixture at
`bench/fixtures/phi-prg/known-answer-tests.jsonl`.
Both are KAT-pinned + bench-pinned regression artifacts that
will surface signal the moment a real v7 checkpoint gives the
probes something to measure. Soft-hash analysis §9.1 + §9.2
were updated to point at these landings (`28b57da`, `0b038f1`).
### Distinct signal vs prior amends
| Source | Catches | Repair path |
|---|---|---|
| Random-word emergent | honesty failures (false-STRICTs) | verifier ladder hardening |
| Witness-sweep | capability failures (LLM gives wrong answer) | prompt eng or fine-tune |
| Phase 3 warrant-chain | rescue-mechanism activation conditions | corpus shape evolution (NOT fixture engineering) |
### Bench artifacts
- `bench/results/phase3-live-validation-2026-05-10.md`
- `bench/results/phase3-warrant-chain-A-B-2026-05-10.md` (initial)
- `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md`
- `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md`
Re-runnable via the bench-emergent harness when corpus shape
shifts and a measurable fire rate is expected.