bench/results: warrant-chain render tail measurement (2026-05-09)
Six-question Q&A sweep against the augmented shard cluster
post-#000031 Phase 3 commit (69e0a95). Looking for the new
`· warrant: N proven` tail surfaced by
_render_warrant_chain_tail when an answer's cited sources
include a claim-pack record with a warrant-resolver-v1
derivations row.
Result: 1 of 6 questions lit up the warrant tail.
Pasch's axiom → STRICT · via quote · warrant: 1 proven ✓
Hilbert geometry axioms → STRICT · via quote ✗
axiom of betweenness → HYBRID · via paraphrase ✗
Bayes theorem → STRICT · via quote ✗
modus tollens → UNGROUNDED · via quote ✗
binomial coefficient id. → HYBRID · via paraphrase ✗
1/6 matches the data substrate: 11 of 92 records have warrant
chains, all pillar-IV Hilbert axioms. The 5 unsigned cases are
honest:
- Hilbert / betweenness queries cited Wikipedia + crawled
textbook surfaces, NOT the claim-pack records — no claim-pack
record cited → no warrant chain to surface.
- Bayes theorem: even if cited, Kolmogorov isn't surface-
ingested → would stay at ANCHOR-WARRANTED ceiling.
- modus tollens UNGROUNDED: pre-warrant verifier issue,
orthogonal to this ticket.
- binomial-coefficient: claim-pack pillar VII cites Brualdi /
Stanley / Knuth, none surface-ingested.
No false positives observed — the render tail correctly stays
silent when chain-of-custody data is absent.
Roadmap: Adding any cited textbook to the manifest →
re-running warrant-resolve → more records resolve, more queries
light up the tail. The 74 still-unresolved records are a
prioritized acquisition list (Mendelson ~20, Enderton ~10,
Stanley ~5, Brualdi ~3, Knuth ~3). The substrate writes its
own roadmap.
This commit is contained in:
parent
aad24d3cfe
commit
8ba123dc17
1 changed files with 116 additions and 0 deletions
116
bench/results/warrant-chain-render-2026-05-09.md
Normal file
116
bench/results/warrant-chain-render-2026-05-09.md
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
# Warrant chain render tail bench — 2026-05-09
|
||||
|
||||
**Date:** 2026-05-09 (UTC)
|
||||
**Author:** blackops + fox
|
||||
**Commits:** `f0e6baf` (Phase 2 resolver + writer) + `69e0a95`
|
||||
(Phase 3 render tail + theorem-name resolver)
|
||||
|
||||
## Setup
|
||||
|
||||
- Shard cluster: `~/.arborist/shards/000-003.db` — 3.47M docs
|
||||
including 92 claim-pack records (78 v2 + 14 pillar VII)
|
||||
- Crawl shards: `~/.arborist/crawl/textbook_*.db` — 9 surface
|
||||
textbooks across 6 g4 pillars
|
||||
- 11 `derivations.process_id='warrant-resolver-v1'` rows in
|
||||
shard 000.db, all binding pillar-IV Hilbert axioms to chunks
|
||||
in the surface-ingested Hilbert *Foundations of Geometry*
|
||||
|
||||
## Method
|
||||
|
||||
Six representative questions through `arborist query --top-k 3`
|
||||
(full QA pipeline: retrieval + body-coverage rerank + title-
|
||||
relevance + concept-relations + verifier + render layer):
|
||||
|
||||
```bash
|
||||
arborist --shards-dir ~/.arborist/shards query "$q" --top-k 3
|
||||
```
|
||||
|
||||
Looking for the new `· warrant: N proven` tail surfaced by
|
||||
`_render_warrant_chain_tail` (commit `69e0a95`) when the
|
||||
answer's cited sources include a claim-pack record with a
|
||||
`warrant-resolver-v1` derivations row.
|
||||
|
||||
## Results
|
||||
|
||||
| # | Question | Audit line | Warrant tail? |
|
||||
|---|---|---|---|
|
||||
| 1 | `Pasch's axiom` | STRICT · via quote · **warrant: 1 proven** 1/1 2.9s | ✓ |
|
||||
| 2 | `Hilbert geometry axioms` | STRICT · via quote 13/13 13.9s | ✗ |
|
||||
| 3 | `axiom of betweenness` | HYBRID · via paraphrase 3/4 4.1s | ✗ |
|
||||
| 4 | `Bayes theorem` | STRICT · via quote 1/1 8.1s | ✗ |
|
||||
| 5 | `modus tollens` | UNGROUNDED · via quote 0/1 4.0s | ✗ |
|
||||
| 6 | `binomial coefficient identity` | HYBRID · via paraphrase 5/7 5.4s | ✗ |
|
||||
|
||||
**1 of 6 questions** lit up the warrant tail. That's the
|
||||
honest signal — only **11 of 92** claim-pack records (~12%)
|
||||
have warrant chains, and only the question that retrieved a
|
||||
Hilbert-pillar-IV record AS its cited source surfaced the tail.
|
||||
|
||||
## Why each case landed where it did
|
||||
|
||||
- **#1 Pasch's axiom (warrant: 1 proven)** — The retrieval
|
||||
pipeline cited *both* the Wikipedia "Pasch's axiom" article
|
||||
AND the claim-pack pillar-IV record. The claim-pack record
|
||||
has a `warrant-resolver-v1` derivations row binding it to
|
||||
chunk 1 of the Hilbert textbook. The render layer surfaces
|
||||
this chain.
|
||||
- **#2 Hilbert geometry axioms (no tail)** — STRICT verdict,
|
||||
but the cited sources were Wikipedia + crawled Hilbert
|
||||
surface — NOT the claim-pack records. The claim-pack records
|
||||
ARE in the cluster but didn't make the top-k for this
|
||||
multi-axiom question. No claim-pack record cited → no warrant
|
||||
chain to surface.
|
||||
- **#3 axiom of betweenness (no tail)** — Same as #2 — claim-
|
||||
pack record not in the cited set.
|
||||
- **#4 Bayes theorem (no tail)** — Different reason: even if
|
||||
the claim-pack Bayes record HAD been cited, Kolmogorov isn't
|
||||
surface-ingested, so its `source_reference` doesn't resolve
|
||||
to a Merkle-bound surface chunk. ANCHOR-WARRANTED ceiling per
|
||||
#000031 §1 honest-ceiling clause.
|
||||
- **#5 modus tollens (UNGROUNDED, no tail)** — The verifier
|
||||
couldn't substring-match the answer's quotes against any
|
||||
cited source. Pre-warrant-chain issue; orthogonal to this
|
||||
ticket's scope.
|
||||
- **#6 binomial coefficient identity (no tail)** — Same as #4
|
||||
— claim-pack pillar VII records cite Brualdi / Stanley /
|
||||
Knuth, none of which are surface-ingested under the current
|
||||
manifest.
|
||||
|
||||
## What this measurement actually shows
|
||||
|
||||
The render-layer tail correctly surfaces when a chain-of-custody
|
||||
exists from answer → cited source → primary-source surface
|
||||
chunk via Merkle-proven derivations. It correctly stays silent
|
||||
in absence of data:
|
||||
|
||||
1. **Honest signal floor.** 1/6 lift in the broad sweep matches
|
||||
the data substrate (11/92 records have chains). No false
|
||||
positives observed on Bayes / Combinatorics queries whose
|
||||
cited textbooks aren't ingested.
|
||||
2. **Targeted lift would be much higher.** A bench specifically
|
||||
on the 11 Hilbert-pillar-IV axioms with warrants, asked
|
||||
against the augmented shard, would light up the tail on
|
||||
~all of them (modulo which records the retrieval pipeline
|
||||
chooses to cite).
|
||||
3. **The path to higher coverage is content acquisition, not
|
||||
code.** Adding Mendelson + Enderton would resolve ~20 more
|
||||
records (logic pillar I); adding Stanley + Brualdi + Knuth
|
||||
would resolve ~7 pillar VII records; adding Jech would
|
||||
resolve ~7 set-theory records. The resolver code is done;
|
||||
the manifest needs more textbooks.
|
||||
|
||||
## Reproducibility
|
||||
|
||||
```bash
|
||||
# Clear + re-populate derivations
|
||||
sqlite3 ~/.arborist/shards/000.db \
|
||||
"DELETE FROM derivations WHERE process_id='warrant-resolver-v1';"
|
||||
arborist --shards-dir ~/.arborist/shards warrant-resolve --write
|
||||
# → records_total=92, records_resolved=11, derivations_written=11
|
||||
|
||||
# Run targeted query
|
||||
arborist --shards-dir ~/.arborist/shards query "Pasch's axiom" --top-k 3
|
||||
# Look for: "· warrant: 1 proven" on the audit line
|
||||
```
|
||||
|
||||
arborist commit at measurement: `69e0a95`.
|
||||
Loading…
Add table
Add a link
Reference in a new issue