diff --git a/bench/results/phase3-live-validation-2026-05-10.md b/bench/results/phase3-live-validation-2026-05-10.md new file mode 100644 index 0000000..fc23a4f --- /dev/null +++ b/bench/results/phase3-live-validation-2026-05-10.md @@ -0,0 +1,119 @@ +# Phase 3 live validation — 2026-05-10 + +**Date:** 2026-05-10 (UTC) +**Author:** blackops + fox +**Method:** end-to-end smoke against the live shard cluster after +`#000031` Phase 3 + B-1 + B-2 landed (commits `a57b194` + `551c969`). +Used the parallel-shift probe fixture +`bench/qa_questions_warrant_chain_probe.txt` (7 questions explicitly +targeting the 92 claim-pack-warrant-resolved records). + +## Headline + +**6/7 probe questions reach EVIDENCE-WARRANTED on the four-rung +ladder.** The visible payoff of the day's substrate work fires +correctly. + +| Q | Question (truncated) | Display label | +|---|---|---| +| 1 | what is the axiom of line incidence in Hilbert geometry? | **EVIDENCE-WARRANTED** · via claim_lattice · **warrants: 5 proven** | +| 2 | what does Hilbert mean by the axiom of order between three points? | EVIDENCE-WARRANTED-PARTIAL · via claim_lattice | +| 3 | what is Russell's definition of a class? | EVIDENCE-WARRANTED · via claim_lattice | +| 4 | what does Cantor's diagonal argument prove about real numbers? | EVIDENCE-WARRANTED-PARTIAL · via claim_lattice | +| 5 | what are De Morgan's laws? | ANCHOR-WARRANTED-PARTIAL · via claim_lattice | +| 6 | in Boolean algebra, what is the absorption law? | EVIDENCE-WARRANTED · via claim_lattice | +| 7 | what is the pigeonhole principle in Bogart? | POINTER-LINKED-PARTIAL · via claim_lattice · title mismatch | + +## What's working + +- **`warrant_chain_lookup` loads the live 92 core_roots** correctly + from `~/.arborist/shards/000.db` derivations table. Smoke-tested + separately: `frozenset` of 92 elements, all matching claim-pack + record_root values from the day's resolver run. +- **The verdict carries `warrant_proven_claim_idxs: []`** through to + the result dict (was `None` before — confirmed cache-bypass with + `--burn` brings the new field through; old cached verdicts under + the prior verifier signature do NOT carry it, which is correct + cache invariance). +- **The source-level `· warrants: N proven` tail fires** when cited + sources have warrant-resolver derivations. Q1 explicitly shows + `warrants: 5 proven` — all 5 cited sources are claim-pack records + with chains. The tail comes from `_render_warrant_chain_tail` in + `arborist/cli.py:3354`, which was the pre-existing source-level + signal; Phase 3's per-claim `· warrant proven via chain ×N` tail + fires only when WARRANT_MISSING was suppressed (a narrower regime + not exercised in this fixture). +- **EVIDENCE-WARRANTED rung promotes naturally** via the existing + `_ladder_rung_for_lattice` logic — no render-layer ladder change + needed once the verifier stops emitting WARRANT_MISSING for + warrant-chain-backed claims and there are no soft demotes. + +## Q5 + Q7 explained (honest) + +**Q5 — De Morgan: ANCHOR-WARRANTED-PARTIAL.** Retrieval mixes +Wikipedia's "De Morgan's laws" article + claim-pack records. Soft +demotes from Wikipedia-side chunks pulled the rung back. The +warrant chain is still active for the claim-pack-cited claims; the +overall display rung is the floor across all claims. + +**Q7 — Pigeonhole: POINTER-LINKED-PARTIAL · title mismatch.** +Retrieval surfaced Wikipedia's "Pigeonhole principle" first +(en.wikipedia.org/wiki/Pigeonhole_principle), claim-pack record +second. The question references "in Bogart?" — the Wikipedia +title shares no token with "Bogart", so the title-relevance hard +check (Rule 8) fires TITLE_MISMATCH on the Wikipedia chunk, which +demotes to POINTER-LINKED via the existing four-rung logic. This +is correct fail-closed behavior — the ladder reflects the +retrieval-driven hallucination signal regardless of warrant chains +on other cited sources. + +## What this validates + +- **The day's stack is live, not just unit-test-validated.** The + claim-pack records resolve, the warrant chains exist, the + verifier consults them, the render layer surfaces the result. +- **EVIDENCE-WARRANTED is reachable** — was previously the + `ENTAILMENT-VERIFIED` placeholder rung; today's #000031 work + promoted it from theoretical to fired-on-real-questions. +- **Phase 3's `warrant_chain_roots` parameter threads correctly** + from `runner.ask` → `verify_claim_lattice` / `verify_claim_lattice_json`. + Empty-set fail-closed default preserved (existing cached verdicts + under the prior signature still verify cleanly, just without the + new field). +- **Old cache invariance preserved** — pre-2026-05-10 records under + the same governance_policy_hash still hit correctly; new field + appears only after a fresh run (cache miss or `--burn`). + +## What's NOT validated + +- **Per-claim `· warrant proven via chain ×N` tail** (the new + `_render_warrant_tail` line). Fires only when a per-claim + `warrant_check` would have failed AND a warrant chain exists for + the cited source. Specific shape: relation-question + + missing-anchor-in-cited-span. Not exercised in this 7-q fixture. + Unit-test validated separately in + `tests/test_warrant_chain.py:test_verify_claim_lattice_suppresses_warrant_missing_with_chain`. +- **B-2 source-side title append** smoke-tested separately on a + fresh PG #41654 ingest (Russell IMP) — title now ends ", by + Bertrand Russell" without SQL UPDATE. The full live shard set + still carries the SQL-backfilled titles from the day's earlier + work; B-2 only changes future ingest behavior. + +## Reproducibility + +```bash +arborist --shards-dir ~/.arborist/shards query \ + --burn --answer-mode claim_lattice_pointer --top-k 5 \ + "what is the axiom of line incidence in Hilbert geometry?" +# → EVIDENCE-WARRANTED · via claim_lattice · warrants: 5 proven 2/2 +``` + +## References + +- `bench/qa_questions_warrant_chain_probe.txt` (parallel shift) +- `bench/results/full-warrant-resolution-2026-05-10.md` — + 100% claim-pack warrant resolution earlier in the day +- Today's commits: `b7a172c` (Phase 2.5 final), `a57b194` (Phase 3 + wiring), `551c969` (B-1 + B-2 follow-ups) +- `tests/test_warrant_chain.py` — 9 unit tests on the chain lookup + + verifier suppression