Phase 3 live validation: 6/7 probes reach EVIDENCE-WARRANTED

End-to-end smoke against the live shard cluster (~/.arborist/shards
+ ~/.arborist/crawl) using the parallel-shift probe fixture
bench/qa_questions_warrant_chain_probe.txt (7 questions explicitly
targeting the 92 claim-pack-warrant-resolved records).

Result: 6 of 7 questions reach EVIDENCE-WARRANTED on the four-rung
ladder. Q1 ("axiom of line incidence in Hilbert geometry") shows
the source-level "warrants: 5 proven" tail correctly counting all
5 cited claim-pack records with derivation chains. Q5 (De Morgan)
falls to ANCHOR-WARRANTED-PARTIAL on mixed Wikipedia/claim-pack
retrieval; Q7 (pigeonhole) hits TITLE_MISMATCH because retrieval
surfaced a Wikipedia article that shares no token with "Bogart".
Both Q5 and Q7 are correct fail-closed behavior given retrieval
choices, not Phase 3 issues.

What this validates:
  - warrant_chain_lookup loads live 92 core_roots
  - warrant_proven_claim_idxs threads through verdict -> result
    dict -> JSON output (was None on cache hits, [] on fresh
    misses — correct cache invariance preserved)
  - source-level "warrants: N proven" tail fires
  - EVIDENCE-WARRANTED rung promotes via existing
    _ladder_rung_for_lattice logic without render-layer changes
  - empty-set fail-closed default preserves backward compat

What's NOT validated end-to-end (still unit-test-only):
  - Per-claim "· warrant proven via chain ×N" tail. Fires only
    when warrant_check would have demoted AND chain present.
    Narrow regime; covered by test_verify_claim_lattice_suppresses
    _warrant_missing_with_chain.

Picks up the parallel-shift's probe fixture (was untracked); now
committed alongside the validation journal.
This commit is contained in:
russell@unturf.com 2026-05-10 09:46:10 -04:00
parent dee6623e46
commit 1d2a558635
No known key found for this signature in database

View file

@ -0,0 +1,119 @@
# Phase 3 live validation — 2026-05-10
**Date:** 2026-05-10 (UTC)
**Author:** blackops + fox
**Method:** end-to-end smoke against the live shard cluster after
`#000031` Phase 3 + B-1 + B-2 landed (commits `a57b194` + `551c969`).
Used the parallel-shift probe fixture
`bench/qa_questions_warrant_chain_probe.txt` (7 questions explicitly
targeting the 92 claim-pack-warrant-resolved records).
## Headline
**6/7 probe questions reach EVIDENCE-WARRANTED on the four-rung
ladder.** The visible payoff of the day's substrate work fires
correctly.
| Q | Question (truncated) | Display label |
|---|---|---|
| 1 | what is the axiom of line incidence in Hilbert geometry? | **EVIDENCE-WARRANTED** · via claim_lattice · **warrants: 5 proven** |
| 2 | what does Hilbert mean by the axiom of order between three points? | EVIDENCE-WARRANTED-PARTIAL · via claim_lattice |
| 3 | what is Russell's definition of a class? | EVIDENCE-WARRANTED · via claim_lattice |
| 4 | what does Cantor's diagonal argument prove about real numbers? | EVIDENCE-WARRANTED-PARTIAL · via claim_lattice |
| 5 | what are De Morgan's laws? | ANCHOR-WARRANTED-PARTIAL · via claim_lattice |
| 6 | in Boolean algebra, what is the absorption law? | EVIDENCE-WARRANTED · via claim_lattice |
| 7 | what is the pigeonhole principle in Bogart? | POINTER-LINKED-PARTIAL · via claim_lattice · title mismatch |
## What's working
- **`warrant_chain_lookup` loads the live 92 core_roots** correctly
from `~/.arborist/shards/000.db` derivations table. Smoke-tested
separately: `frozenset` of 92 elements, all matching claim-pack
record_root values from the day's resolver run.
- **The verdict carries `warrant_proven_claim_idxs: []`** through to
the result dict (was `None` before — confirmed cache-bypass with
`--burn` brings the new field through; old cached verdicts under
the prior verifier signature do NOT carry it, which is correct
cache invariance).
- **The source-level `· warrants: N proven` tail fires** when cited
sources have warrant-resolver derivations. Q1 explicitly shows
`warrants: 5 proven` — all 5 cited sources are claim-pack records
with chains. The tail comes from `_render_warrant_chain_tail` in
`arborist/cli.py:3354`, which was the pre-existing source-level
signal; Phase 3's per-claim `· warrant proven via chain ×N` tail
fires only when WARRANT_MISSING was suppressed (a narrower regime
not exercised in this fixture).
- **EVIDENCE-WARRANTED rung promotes naturally** via the existing
`_ladder_rung_for_lattice` logic — no render-layer ladder change
needed once the verifier stops emitting WARRANT_MISSING for
warrant-chain-backed claims and there are no soft demotes.
## Q5 + Q7 explained (honest)
**Q5 — De Morgan: ANCHOR-WARRANTED-PARTIAL.** Retrieval mixes
Wikipedia's "De Morgan's laws" article + claim-pack records. Soft
demotes from Wikipedia-side chunks pulled the rung back. The
warrant chain is still active for the claim-pack-cited claims; the
overall display rung is the floor across all claims.
**Q7 — Pigeonhole: POINTER-LINKED-PARTIAL · title mismatch.**
Retrieval surfaced Wikipedia's "Pigeonhole principle" first
(en.wikipedia.org/wiki/Pigeonhole_principle), claim-pack record
second. The question references "in Bogart?" — the Wikipedia
title shares no token with "Bogart", so the title-relevance hard
check (Rule 8) fires TITLE_MISMATCH on the Wikipedia chunk, which
demotes to POINTER-LINKED via the existing four-rung logic. This
is correct fail-closed behavior — the ladder reflects the
retrieval-driven hallucination signal regardless of warrant chains
on other cited sources.
## What this validates
- **The day's stack is live, not just unit-test-validated.** The
claim-pack records resolve, the warrant chains exist, the
verifier consults them, the render layer surfaces the result.
- **EVIDENCE-WARRANTED is reachable** — was previously the
`ENTAILMENT-VERIFIED` placeholder rung; today's #000031 work
promoted it from theoretical to fired-on-real-questions.
- **Phase 3's `warrant_chain_roots` parameter threads correctly**
from `runner.ask` → `verify_claim_lattice` / `verify_claim_lattice_json`.
Empty-set fail-closed default preserved (existing cached verdicts
under the prior signature still verify cleanly, just without the
new field).
- **Old cache invariance preserved** — pre-2026-05-10 records under
the same governance_policy_hash still hit correctly; new field
appears only after a fresh run (cache miss or `--burn`).
## What's NOT validated
- **Per-claim `· warrant proven via chain ×N` tail** (the new
`_render_warrant_tail` line). Fires only when a per-claim
`warrant_check` would have failed AND a warrant chain exists for
the cited source. Specific shape: relation-question +
missing-anchor-in-cited-span. Not exercised in this 7-q fixture.
Unit-test validated separately in
`tests/test_warrant_chain.py:test_verify_claim_lattice_suppresses_warrant_missing_with_chain`.
- **B-2 source-side title append** smoke-tested separately on a
fresh PG #41654 ingest (Russell IMP) — title now ends ", by
Bertrand Russell" without SQL UPDATE. The full live shard set
still carries the SQL-backfilled titles from the day's earlier
work; B-2 only changes future ingest behavior.
## Reproducibility
```bash
arborist --shards-dir ~/.arborist/shards query \
--burn --answer-mode claim_lattice_pointer --top-k 5 \
"what is the axiom of line incidence in Hilbert geometry?"
# → EVIDENCE-WARRANTED · via claim_lattice · warrants: 5 proven 2/2
```
## References
- `bench/qa_questions_warrant_chain_probe.txt` (parallel shift)
- `bench/results/full-warrant-resolution-2026-05-10.md` —
100% claim-pack warrant resolution earlier in the day
- Today's commits: `b7a172c` (Phase 2.5 final), `a57b194` (Phase 3
wiring), `551c969` (B-1 + B-2 follow-ups)
- `tests/test_warrant_chain.py` — 9 unit tests on the chain lookup
+ verifier suppression