ticket #000006: 2026-05-10 amend — Phase 3 warrant-chain dormancy
Third qualitatively different experimental shape in the rolling research log, parallel to the 2026-05-09 witness-sweep widening: - random-word emergent (honesty under no ground truth) - witness-sweep (capability under canonical ground truth) - warrant-chain (rescue-mechanism activation under chain-backed ground truth) ← new Captures the structurally-grounded dormancy of #000031 Phase 3 verifier — 0/117 fires across an aggressive A/B with 12/13 chain-root retrieval, plus 5 paraphrase fixture iterations none of which drive Phase 3 to fire empirically. Root cause pinned at arborist/qa/warrant.py:507: warrant_check vacuous-passes unless question shape is relation/date/entity-list/ count/why-cause. Definitional questions don't have warrant shape; claim-pack chunks are terse axioms without prose context that warrant-shape questions would retrieve. Phase 3 is structurally rescue-only on the current corpus. Mechanism correct + non-regressing + unit-tested. Recommendation: track fire rate as a corpus-evolution signal, not a fixture- engineering target. Audit-line tail will surface organic fires. Adjacent same-day landings noted: #000034 Phase 1a (Hessian- alignment probe) + #000035 Phase 1 (φ_PRG reference) — both KAT-pinned cryptographic-primitive infrastructure ahead of v7 plastic-training deployment. Doc-only update; no code change.
This commit is contained in:
parent
7ea0fdf2b1
commit
2c683de6d8
1 changed files with 135 additions and 0 deletions
|
|
@ -618,3 +618,138 @@ Re-runnable via `make bench-witness-sweep`.
|
|||
`bench/fixtures/5f/falsification-witness-v1.jsonl` — extracted
|
||||
divergence fixtures (10 rows after two extraction passes today).
|
||||
|
||||
|
||||
## Amend — 2026-05-10 (Phase 3 warrant-chain dormancy: structurally rescue-only)
|
||||
|
||||
A third qualitatively different experimental shape, parallel to
|
||||
the 2026-05-09 witness-sweep widening. The earlier amends measured
|
||||
honesty under no ground truth (random-word) and capability under
|
||||
canonical ground truth (witness-sweep). This amend measures
|
||||
**warrant-chain rescue activation under chain-backed ground truth**
|
||||
— the #000031 Phase 3 verifier (`a57b194`) running against fixtures
|
||||
explicitly targeting the 92 claim-pack-warrant-resolved records.
|
||||
|
||||
### Setup
|
||||
|
||||
#000031 Phase 3 (warrant-chain-aware verifier) landed 2026-05-10.
|
||||
Mechanism: when the LLM cites a chunk that has chain-root warrant
|
||||
backing AND the lexical `warrant_check` would otherwise emit
|
||||
`WARRANT_MISSING`, Phase 3 suppresses that violation by promoting
|
||||
via the chain.
|
||||
|
||||
Three sequential fixture iterations probed whether the mechanism
|
||||
fires empirically on the live corpus:
|
||||
|
||||
1. **Live validation** (`1d2a558`) — 7-question parallel-shift
|
||||
probe targeting chain-root documents.
|
||||
2. **Aggressive A/B** (`c5bc53f`) — 13 questions using exact
|
||||
claim-pack title phrasing; n=3, 117 cells per condition.
|
||||
3. **Paraphrase investigation** (`a9fcb42`) — 5 fixture iterations
|
||||
each probed-then-benched, varying question shape (definitional,
|
||||
how-use, plain-English, who-proposed, who-first-stated).
|
||||
|
||||
### Aggregate
|
||||
|
||||
| Investigation | Chain-root retrieval | Phase 3 fires | STRICT-rate Δ |
|
||||
|---|---|---|---|
|
||||
| Live validation | 6/7 → EVIDENCE-WARRANTED | n/a (smoke) | n/a (no A/B) |
|
||||
| Aggressive A/B | 12/13 questions hit chain roots | **0 / 117 cells** | +0 (85/117 → 85/117) |
|
||||
| Paraphrase v1-v3' | 12/13 → 1/8 (varies) | **0** across all iterations | not measured |
|
||||
|
||||
`cache_key parity 39/39` on aggressive A/B — Phase 3 is
|
||||
mechanically observable but produces zero fires.
|
||||
|
||||
### Root cause — `arborist/qa/warrant.py:507`
|
||||
|
||||
`warrant_check` **vacuous-passes** (returns `(True, [])` without
|
||||
anchor extraction) unless the question shape is one of:
|
||||
|
||||
```text
|
||||
relation — proper-noun anchors via is_relation_question
|
||||
date — 4-digit year in claim
|
||||
entity-list — multi-entity question shape
|
||||
count — count-shape question
|
||||
why-cause — cause anchors gated on why-shape question
|
||||
```
|
||||
|
||||
Definitional questions ("what is X", "define X") don't have
|
||||
warrant shape → no `WARRANT_MISSING` to suppress → Phase 3 has
|
||||
nothing to do.
|
||||
|
||||
### Structural reason for the dormancy
|
||||
|
||||
Claim-pack chunks (the 92 chain-root documents) are deliberately
|
||||
**terse** — they state axioms / theorems precisely with
|
||||
mathematical formalism but contain no biographical, historical,
|
||||
or explanatory prose. So warrant-shape questions ("who" / "when"
|
||||
/ "why") retrieve from Wikipedia (which has matching prose
|
||||
context) instead of from claim-pack chunks.
|
||||
|
||||
The data condition Phase 3 needs — chain-backed chunk cited for a
|
||||
warrant-shape claim — is **empirically unreachable through normal
|
||||
retrieval pathways on this corpus shape**.
|
||||
|
||||
### Verdict
|
||||
|
||||
Phase 3's runtime fire rate is structurally bounded near zero on
|
||||
the current corpus. **Mechanism is correct, non-regressing,
|
||||
unit-tested** (`tests/test_warrant_chain.py`).
|
||||
|
||||
Two readings, both true:
|
||||
|
||||
1. **The lexical layer is strong enough** on clean axiom questions
|
||||
that the chain-fallback isn't needed. The mechanism that lands
|
||||
warrants on cheap lexical evidence does most of the work.
|
||||
2. **Phase 3 is a latent capability.** It will activate when corpus
|
||||
shape shifts — claim-pack content gaining prose context, or the
|
||||
corpus mix changing toward warrant-shape questions. The audit-
|
||||
line tail `· warrant proven via chain ×N` will surface fires
|
||||
when they occur.
|
||||
|
||||
### Recommendation (per the investigation's own conclusion)
|
||||
|
||||
Don't engineer Phase 3 fires via fixture design. Track fire rate
|
||||
as a **corpus-evolution signal**. If claim-pack content gains
|
||||
prose context or the corpus mix changes, Phase 3 will start firing
|
||||
organically and the audit-line tail will surface it.
|
||||
|
||||
This closes the three-thread investigation cleanly. The fixtures
|
||||
+ reports remain as the empirical trail for future shifts.
|
||||
|
||||
### Adjacent landings (same-day infrastructure)
|
||||
|
||||
Two cryptographic-primitive Phase 1 deliverables landed
|
||||
2026-05-10 ahead of v7 plastic-training deployment:
|
||||
|
||||
- **#000034 Phase 1a** (`1dfb8b9`): Hessian-alignment probe at
|
||||
`bench/scripts/phi_alignment_probe.py` + 14 tests +
|
||||
10-vector KAT fixture at
|
||||
`bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl`.
|
||||
- **#000035 Phase 1** (earlier today): φ_PRG reference impl at
|
||||
`arborist/substrate/anchor_prg.py` (HMAC-SHA-512 counter-mode
|
||||
KDF) + 20 tests + 10-vector KAT fixture at
|
||||
`bench/fixtures/phi-prg/known-answer-tests.jsonl`.
|
||||
|
||||
Both are KAT-pinned + bench-pinned regression artifacts that
|
||||
will surface signal the moment a real v7 checkpoint gives the
|
||||
probes something to measure. Soft-hash analysis §9.1 + §9.2
|
||||
were updated to point at these landings (`28b57da`, `0b038f1`).
|
||||
|
||||
### Distinct signal vs prior amends
|
||||
|
||||
| Source | Catches | Repair path |
|
||||
|---|---|---|
|
||||
| Random-word emergent | honesty failures (false-STRICTs) | verifier ladder hardening |
|
||||
| Witness-sweep | capability failures (LLM gives wrong answer) | prompt eng or fine-tune |
|
||||
| Phase 3 warrant-chain | rescue-mechanism activation conditions | corpus shape evolution (NOT fixture engineering) |
|
||||
|
||||
### Bench artifacts
|
||||
|
||||
- `bench/results/phase3-live-validation-2026-05-10.md`
|
||||
- `bench/results/phase3-warrant-chain-A-B-2026-05-10.md` (initial)
|
||||
- `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md`
|
||||
- `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md`
|
||||
|
||||
Re-runnable via the bench-emergent harness when corpus shape
|
||||
shifts and a measurable fire rate is expected.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue