ticket #000006: 2026-05-10 amend — Phase 3 warrant-chain dormancy
Third qualitatively different experimental shape in the rolling research log, parallel to the 2026-05-09 witness-sweep widening: - random-word emergent (honesty under no ground truth) - witness-sweep (capability under canonical ground truth) - warrant-chain (rescue-mechanism activation under chain-backed ground truth) ← new Captures the structurally-grounded dormancy of #000031 Phase 3 verifier — 0/117 fires across an aggressive A/B with 12/13 chain-root retrieval, plus 5 paraphrase fixture iterations none of which drive Phase 3 to fire empirically. Root cause pinned at arborist/qa/warrant.py:507: warrant_check vacuous-passes unless question shape is relation/date/entity-list/ count/why-cause. Definitional questions don't have warrant shape; claim-pack chunks are terse axioms without prose context that warrant-shape questions would retrieve. Phase 3 is structurally rescue-only on the current corpus. Mechanism correct + non-regressing + unit-tested. Recommendation: track fire rate as a corpus-evolution signal, not a fixture- engineering target. Audit-line tail will surface organic fires. Adjacent same-day landings noted: #000034 Phase 1a (Hessian- alignment probe) + #000035 Phase 1 (φ_PRG reference) — both KAT-pinned cryptographic-primitive infrastructure ahead of v7 plastic-training deployment. Doc-only update; no code change.
This commit is contained in:
parent
7ea0fdf2b1
commit
2c683de6d8
1 changed files with 135 additions and 0 deletions
|
|
@ -618,3 +618,138 @@ Re-runnable via `make bench-witness-sweep`.
|
||||||
`bench/fixtures/5f/falsification-witness-v1.jsonl` — extracted
|
`bench/fixtures/5f/falsification-witness-v1.jsonl` — extracted
|
||||||
divergence fixtures (10 rows after two extraction passes today).
|
divergence fixtures (10 rows after two extraction passes today).
|
||||||
|
|
||||||
|
|
||||||
|
## Amend — 2026-05-10 (Phase 3 warrant-chain dormancy: structurally rescue-only)
|
||||||
|
|
||||||
|
A third qualitatively different experimental shape, parallel to
|
||||||
|
the 2026-05-09 witness-sweep widening. The earlier amends measured
|
||||||
|
honesty under no ground truth (random-word) and capability under
|
||||||
|
canonical ground truth (witness-sweep). This amend measures
|
||||||
|
**warrant-chain rescue activation under chain-backed ground truth**
|
||||||
|
— the #000031 Phase 3 verifier (`a57b194`) running against fixtures
|
||||||
|
explicitly targeting the 92 claim-pack-warrant-resolved records.
|
||||||
|
|
||||||
|
### Setup
|
||||||
|
|
||||||
|
#000031 Phase 3 (warrant-chain-aware verifier) landed 2026-05-10.
|
||||||
|
Mechanism: when the LLM cites a chunk that has chain-root warrant
|
||||||
|
backing AND the lexical `warrant_check` would otherwise emit
|
||||||
|
`WARRANT_MISSING`, Phase 3 suppresses that violation by promoting
|
||||||
|
via the chain.
|
||||||
|
|
||||||
|
Three sequential fixture iterations probed whether the mechanism
|
||||||
|
fires empirically on the live corpus:
|
||||||
|
|
||||||
|
1. **Live validation** (`1d2a558`) — 7-question parallel-shift
|
||||||
|
probe targeting chain-root documents.
|
||||||
|
2. **Aggressive A/B** (`c5bc53f`) — 13 questions using exact
|
||||||
|
claim-pack title phrasing; n=3, 117 cells per condition.
|
||||||
|
3. **Paraphrase investigation** (`a9fcb42`) — 5 fixture iterations
|
||||||
|
each probed-then-benched, varying question shape (definitional,
|
||||||
|
how-use, plain-English, who-proposed, who-first-stated).
|
||||||
|
|
||||||
|
### Aggregate
|
||||||
|
|
||||||
|
| Investigation | Chain-root retrieval | Phase 3 fires | STRICT-rate Δ |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Live validation | 6/7 → EVIDENCE-WARRANTED | n/a (smoke) | n/a (no A/B) |
|
||||||
|
| Aggressive A/B | 12/13 questions hit chain roots | **0 / 117 cells** | +0 (85/117 → 85/117) |
|
||||||
|
| Paraphrase v1-v3' | 12/13 → 1/8 (varies) | **0** across all iterations | not measured |
|
||||||
|
|
||||||
|
`cache_key parity 39/39` on aggressive A/B — Phase 3 is
|
||||||
|
mechanically observable but produces zero fires.
|
||||||
|
|
||||||
|
### Root cause — `arborist/qa/warrant.py:507`
|
||||||
|
|
||||||
|
`warrant_check` **vacuous-passes** (returns `(True, [])` without
|
||||||
|
anchor extraction) unless the question shape is one of:
|
||||||
|
|
||||||
|
```text
|
||||||
|
relation — proper-noun anchors via is_relation_question
|
||||||
|
date — 4-digit year in claim
|
||||||
|
entity-list — multi-entity question shape
|
||||||
|
count — count-shape question
|
||||||
|
why-cause — cause anchors gated on why-shape question
|
||||||
|
```
|
||||||
|
|
||||||
|
Definitional questions ("what is X", "define X") don't have
|
||||||
|
warrant shape → no `WARRANT_MISSING` to suppress → Phase 3 has
|
||||||
|
nothing to do.
|
||||||
|
|
||||||
|
### Structural reason for the dormancy
|
||||||
|
|
||||||
|
Claim-pack chunks (the 92 chain-root documents) are deliberately
|
||||||
|
**terse** — they state axioms / theorems precisely with
|
||||||
|
mathematical formalism but contain no biographical, historical,
|
||||||
|
or explanatory prose. So warrant-shape questions ("who" / "when"
|
||||||
|
/ "why") retrieve from Wikipedia (which has matching prose
|
||||||
|
context) instead of from claim-pack chunks.
|
||||||
|
|
||||||
|
The data condition Phase 3 needs — chain-backed chunk cited for a
|
||||||
|
warrant-shape claim — is **empirically unreachable through normal
|
||||||
|
retrieval pathways on this corpus shape**.
|
||||||
|
|
||||||
|
### Verdict
|
||||||
|
|
||||||
|
Phase 3's runtime fire rate is structurally bounded near zero on
|
||||||
|
the current corpus. **Mechanism is correct, non-regressing,
|
||||||
|
unit-tested** (`tests/test_warrant_chain.py`).
|
||||||
|
|
||||||
|
Two readings, both true:
|
||||||
|
|
||||||
|
1. **The lexical layer is strong enough** on clean axiom questions
|
||||||
|
that the chain-fallback isn't needed. The mechanism that lands
|
||||||
|
warrants on cheap lexical evidence does most of the work.
|
||||||
|
2. **Phase 3 is a latent capability.** It will activate when corpus
|
||||||
|
shape shifts — claim-pack content gaining prose context, or the
|
||||||
|
corpus mix changing toward warrant-shape questions. The audit-
|
||||||
|
line tail `· warrant proven via chain ×N` will surface fires
|
||||||
|
when they occur.
|
||||||
|
|
||||||
|
### Recommendation (per the investigation's own conclusion)
|
||||||
|
|
||||||
|
Don't engineer Phase 3 fires via fixture design. Track fire rate
|
||||||
|
as a **corpus-evolution signal**. If claim-pack content gains
|
||||||
|
prose context or the corpus mix changes, Phase 3 will start firing
|
||||||
|
organically and the audit-line tail will surface it.
|
||||||
|
|
||||||
|
This closes the three-thread investigation cleanly. The fixtures
|
||||||
|
+ reports remain as the empirical trail for future shifts.
|
||||||
|
|
||||||
|
### Adjacent landings (same-day infrastructure)
|
||||||
|
|
||||||
|
Two cryptographic-primitive Phase 1 deliverables landed
|
||||||
|
2026-05-10 ahead of v7 plastic-training deployment:
|
||||||
|
|
||||||
|
- **#000034 Phase 1a** (`1dfb8b9`): Hessian-alignment probe at
|
||||||
|
`bench/scripts/phi_alignment_probe.py` + 14 tests +
|
||||||
|
10-vector KAT fixture at
|
||||||
|
`bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl`.
|
||||||
|
- **#000035 Phase 1** (earlier today): φ_PRG reference impl at
|
||||||
|
`arborist/substrate/anchor_prg.py` (HMAC-SHA-512 counter-mode
|
||||||
|
KDF) + 20 tests + 10-vector KAT fixture at
|
||||||
|
`bench/fixtures/phi-prg/known-answer-tests.jsonl`.
|
||||||
|
|
||||||
|
Both are KAT-pinned + bench-pinned regression artifacts that
|
||||||
|
will surface signal the moment a real v7 checkpoint gives the
|
||||||
|
probes something to measure. Soft-hash analysis §9.1 + §9.2
|
||||||
|
were updated to point at these landings (`28b57da`, `0b038f1`).
|
||||||
|
|
||||||
|
### Distinct signal vs prior amends
|
||||||
|
|
||||||
|
| Source | Catches | Repair path |
|
||||||
|
|---|---|---|
|
||||||
|
| Random-word emergent | honesty failures (false-STRICTs) | verifier ladder hardening |
|
||||||
|
| Witness-sweep | capability failures (LLM gives wrong answer) | prompt eng or fine-tune |
|
||||||
|
| Phase 3 warrant-chain | rescue-mechanism activation conditions | corpus shape evolution (NOT fixture engineering) |
|
||||||
|
|
||||||
|
### Bench artifacts
|
||||||
|
|
||||||
|
- `bench/results/phase3-live-validation-2026-05-10.md`
|
||||||
|
- `bench/results/phase3-warrant-chain-A-B-2026-05-10.md` (initial)
|
||||||
|
- `bench/results/phase3-warrant-chain-A-B-aggressive-2026-05-10.md`
|
||||||
|
- `bench/results/phase3-warrant-chain-paraphrase-investigation-2026-05-10.md`
|
||||||
|
|
||||||
|
Re-runnable via the bench-emergent harness when corpus shape
|
||||||
|
shifts and a measurable fire rate is expected.
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue