russell@unturf.com
f0e6baf907
Closes the warrant-promotion data path: claim-pack records now
bind to surface-ingested textbook chunks via Merkle inclusion
proofs in the existing `derivations` table.
What landed
===========
arborist/qa/warrant_resolver.py — four pure-data steps + one DB
write:
1. parse_citation(s) — regex pipeline turning the claim-pack
`source_reference` string into structured Citation tuples.
Handles "Title by Author" (single + Oxford-comma multi +
et-al), semicolon-separated multi-cite ("Knuth §1.2.6;
Stanley §1.2; Brualdi §3.5"), and compact author-year
("Pascal 1654") forms.
2. resolve_chunks(c, shards_dir) — FTS5 search across sibling
crawl/ dir's textbook-surface shards. Skips the main numbered
shards (Wikipedia content; would be false positives). Per-
shard match filter requires BOTH author last name AND a title
token in the shard's title-haystack — honest "no match" for
textbooks not yet surface-ingested.
3. compute_proof(shard, doc_root, chunk_id) — reads
merkle_nodes, walks layer-by-layer to assemble siblings;
emits deterministic JSON proof_blob compatible with
arborist/merkle.py verification.
4. write_derivation(...) — INSERT OR IGNORE into the existing
derivations table with process_id="warrant-resolver-v1".
Idempotent at the database layer.
CLI surface
===========
- `arborist warrant-status --shards-dir ...` (read-only) —
emits per-record JSON: parsed citations, FTS5 candidates,
whether a derivations row exists.
- `arborist warrant-resolve --shards-dir ... [--write]` —
default dry-run summary; --write actually computes proofs
and inserts rows.
End-to-end verification
=======================
Real-shard run: `arborist warrant-resolve --shards-dir
~/.arborist/shards --write` →
records_total: 92
records_resolved: 18
derivations_written: 18
All 18 are pillar-IV Hilbert axioms citing "The Foundations of
Geometry by David Hilbert" — the only cited textbook fully
surface-ingested by Phase 1. The remaining 74 records cite
textbooks not in our shard cluster (Mendelson, Enderton,
Jech, Goldstein, Barendregt, Stanley, Brualdi, Knuth, …) and
correctly produce 0 matches; they stay at ANCHOR-WARRANTED
until those textbooks land via future Phase-1 manifest
expansions.
Re-running the writer is a no-op (PK collision on (core_root,
src_root, process_id) = INSERT OR IGNORE).
Drive-by fix
============
arborist/sources/textbook_tex.py — _extract_title now also
parses PG's plain-text `Author:` line and appends "by Author"
to the title, so the warrant resolver's author-last-name match
works against PG-ingested textbooks (Hilbert "The Foundations
of Geometry by David Hilbert" instead of just "The Foundations
of Geometry").
Test suite
==========
tests/test_warrant_resolver.py — 14 unit tests for the citation
parser (no DB / network). Full suite: 1588 passed / 28 skipped.
Phase 3 (verifier wiring)
=========================
NOT in this commit. The data substrate is in place; the
audit_mode upgrade path that lifts answers citing
claim-pack-records-with-derivations from ANCHOR-WARRANTED to
EVIDENCE-WARRANTED requires a verifier change — touches well-
tested code, worth its own ticket so the regression risk is
bounded.
141 lines
3.8 KiB
Python
141 lines
3.8 KiB
Python
"""Tests for the warrant resolver (#000031 Phase 2).
|
|
|
|
Pure unit tests over the citation parser; the FTS5 resolver +
|
|
Merkle proof writer are exercised end-to-end via the
|
|
``arborist warrant-resolve`` CLI on real shards. The CLI smoke test
|
|
is documented in the ticket; this file stays offline / DB-free.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from arborist.qa.warrant_resolver import (
|
|
Citation,
|
|
parse_citation,
|
|
)
|
|
|
|
|
|
# --- "Title by Author" pattern (most common) -------------------------
|
|
|
|
|
|
def test_simple_title_by_author():
|
|
out = parse_citation("Introduction to Mathematical Logic by Elliott Mendelson")
|
|
assert len(out) == 1
|
|
c = out[0]
|
|
assert c.title == "Introduction to Mathematical Logic"
|
|
assert c.authors == ("Elliott Mendelson",)
|
|
|
|
|
|
def test_title_with_punctuation():
|
|
out = parse_citation("The Lambda Calculus: Its Syntax and Semantics by H.P. Barendregt")
|
|
assert len(out) == 1
|
|
c = out[0]
|
|
assert c.title == "The Lambda Calculus: Its Syntax and Semantics"
|
|
assert c.authors == ("H.P. Barendregt",)
|
|
|
|
|
|
# --- multi-author Oxford comma --------------------------------------
|
|
|
|
|
|
def test_multi_author_oxford_comma():
|
|
out = parse_citation(
|
|
"Classical Mechanics by Herbert Goldstein, Charles P. Poole, and John L. Safko"
|
|
)
|
|
assert len(out) == 1
|
|
c = out[0]
|
|
assert c.title == "Classical Mechanics"
|
|
assert c.authors == ("Herbert Goldstein", "Charles P. Poole", "John L. Safko")
|
|
|
|
|
|
def test_multi_author_et_al():
|
|
out = parse_citation("Classical Mechanics by Herbert Goldstein et al.")
|
|
assert len(out) == 1
|
|
c = out[0]
|
|
assert c.title == "Classical Mechanics"
|
|
assert c.authors == ("Herbert Goldstein",)
|
|
|
|
|
|
# --- semicolon-separated multi-citation -----------------------------
|
|
|
|
|
|
def test_semicolon_split():
|
|
out = parse_citation(
|
|
"Knuth TAOCP Volume 1 §1.2.6; Stanley §1.2; Brualdi §3.5"
|
|
)
|
|
assert len(out) == 3
|
|
assert out[0].authors == ("Knuth",)
|
|
assert out[1].authors == ("Stanley",)
|
|
assert out[2].authors == ("Brualdi",)
|
|
|
|
|
|
def test_semicolon_with_year_and_section():
|
|
out = parse_citation(
|
|
"Knuth TAOCP Volume 1 §1.2.6 equation (13); Vandermonde 1772"
|
|
)
|
|
assert len(out) == 2
|
|
assert out[0].section
|
|
assert "§1.2.6" in out[0].section
|
|
assert out[1].year == "1772"
|
|
|
|
|
|
# --- compact form ---------------------------------------------------
|
|
|
|
|
|
def test_compact_year_form():
|
|
out = parse_citation("Pascal 1654")
|
|
assert len(out) == 1
|
|
c = out[0]
|
|
assert c.year == "1654"
|
|
assert "Pascal" in c.authors
|
|
|
|
|
|
def test_compact_section_only():
|
|
out = parse_citation("Brualdi §3.5")
|
|
assert len(out) == 1
|
|
c = out[0]
|
|
assert c.authors == ("Brualdi",)
|
|
assert "§3.5" in c.section
|
|
|
|
|
|
# --- edge cases ------------------------------------------------------
|
|
|
|
|
|
def test_empty_input():
|
|
assert parse_citation("") == []
|
|
assert parse_citation(" ") == []
|
|
|
|
|
|
def test_year_extraction():
|
|
out = parse_citation("Foundations of Probability by Andrey Kolmogorov 1933")
|
|
assert len(out) == 1
|
|
assert out[0].year == "1933"
|
|
|
|
|
|
def test_section_extraction():
|
|
out = parse_citation("Classical Mechanics by Goldstein Volume 1 §3.2")
|
|
assert len(out) == 1
|
|
assert "§3.2" in out[0].section
|
|
assert "Volume 1" in out[0].section
|
|
|
|
|
|
def test_oeis_identifier():
|
|
out = parse_citation("OEIS A000108")
|
|
assert len(out) == 1
|
|
# OEIS identifiers parse as compact form — first token is "author".
|
|
assert out[0].authors == ("OEIS",)
|
|
|
|
|
|
# --- Citation dataclass invariants ----------------------------------
|
|
|
|
|
|
def test_citation_is_empty():
|
|
assert Citation().is_empty()
|
|
assert not Citation(title="Foo").is_empty()
|
|
assert not Citation(authors=("Bar",)).is_empty()
|
|
|
|
|
|
def test_raw_field_preserved():
|
|
raw = "Some weird citation by Some Author"
|
|
out = parse_citation(raw)
|
|
assert out[0].raw == raw
|