engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

ticket #000031 follow-ups B-1 + B-2: alias attribution + source-side title author

Browse source 
B-1: via_citation_alias attribution — resolver no longer mislabels
citation-alias-substituted chains as DIRECT.

  - New Citation.via_citation_alias field (default False, preserves
    parse-from-source-ref path).
  - warrant_status sets via_citation_alias=True on substitute
    Citations from #000041 lookup_citation_aliases.
  - resolve_chunks reads citation.via_citation_alias as a "floor" for
    via_alias on every match it produces (Pass 1 hits inherit it
    too, not just Pass 2 term-alias hits). Audit-honest: matches
    from a substitute Citation are alias-driven regardless of which
    cascade pass found the chunk.

  Live re-resolve under the new attribution: 18 direct + 74 +alias
  (was 75/17 mis-labeled). The 18 direct = exactly Hilbert pillar IV
  records resolving on the literally-cited Hilbert textbook. All 74
  records resolved via citation-alias substitution now carry
  process_id="warrant-resolver-v1+alias" in derivations.

  3 new unit tests (test_warrant_resolver.py): default-False on
  parsed Citations, explicit-True construction works, resolve_chunks
  propagates the floor onto every ResolutionMatch.

B-2: source-side title-from-author backfill — eliminates the
per-shard SQL UPDATE workaround.

  - HtmlPageSource accepts default_author kwarg; appends ', by
    <author>' to ingested document titles when the <title> tag
    doesn't already include the surname.
  - TextbookTexSource accepts default_author kwarg; appends ' by
    <author>' to titles when the LaTeX has no \author{} macro AND
    no PG-style 'Author:' boilerplate.
  - _CrawledHtmlSource (BFS-crawler bridge) accepts default_author
    kwarg; same append logic. ingest_crawled() and arborist crawl
    --ingest plumb it through.
  - arborist ingest --author + arborist crawl --author CLI flags.
  - bench/scripts/textbooks_manifest.py:cmd_lookup emits the
    manifest's `author` field as a 7th tab column.
  - make textbook target reads the author column and threads
    --author into both crawl-ingest and shallow-ingest paths.

  Idempotency preserved — surname-already-in-title detection prevents
  double-stamping on re-ingest. Shards previously SQL-backfilled
  (Cantor / Russell IMP / Bogart / Judson / Levin / KT / Peano /
  Grinstead-Snell) keep their existing titles; new ingests pick up
  the author signal at source time.

  Live smoke: arborist ingest --source html --author "Bertrand
  Russell" against PG #41654 yields title "Introduction to
  Mathematical Philosophy | Project Gutenberg, by Bertrand Russell"
  with no SQL UPDATE needed.

Total: 1655 tests pass (was 1652). Both follow-ups land additive,
fail-closed, idempotent. The two cleanup items from #000031
Phase 3's commit message are now closed.
This commit is contained in:
russell@unturf.com 2026-05-10 09:35:49 -04:00
parent 7e81425d49
commit 551c9695e0
No known key found for this signature in database
8 changed files with 304 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

8
Makefile
View file

@ -766,20 +766,24 @@ textbook: bootstrap-crawler ## ingest one textbook by id: make textbook ID=bogar
max=$$(echo "$$row" | cut -f4); \
license=$$(echo "$$row" | cut -f5); \
domain=$$(echo "$$row" | cut -f6); \
author=$$(echo "$$row" | cut -f7); \
mkdir -p $(CRAWL_SHARDS_DIR); \
shard="$(CRAWL_SHARDS_DIR)/textbook_$${id}.db"; \
author_flag=""; \
if [ -n "$$author" ]; then author_flag="--author $$author"; fi; \
echo ">> $$id ($$license, $$domain)"; \
echo " crawl: $$url depth=$$depth max=$$max"; \
echo " shard: $$shard"; \
if [ -n "$$author" ]; then echo " author: $$author"; fi; \
if [ -n "$$url" ]; then \
$(ARBORIST) --db "$$shard" crawl --seed-url "$$url" --depth $$depth --max-pages $$max --ingest > /tmp/textbook-$${id}.log 2>&1 && \
$(ARBORIST) --db "$$shard" crawl --seed-url "$$url" --depth $$depth --max-pages $$max --ingest $$author_flag > /tmp/textbook-$${id}.log 2>&1 && \
docs=$$(.venv/bin/python -c "import sqlite3; print(sqlite3.connect(\"$$shard\").execute(\"SELECT COUNT(*) FROM documents\").fetchone()[0])"); \
chunks=$$(.venv/bin/python -c "import sqlite3; print(sqlite3.connect(\"$$shard\").execute(\"SELECT COUNT(*) FROM chunks\").fetchone()[0])"); \
echo " landed: $$docs docs / $$chunks chunks"; \
else \
echo " no crawl_url for $$id; using shallow ingest path"; \
echo "$$id" | $(PY) -c 'import sys, json, os; sys.path.insert(0, "."); ids = sys.stdin.read().split(); manifest = "$(TEXTBOOK_MANIFEST)"; [print(u) for line in open(manifest) for e in [json.loads(line)] if "_meta" not in e and e.get("id") in ids for u in e.get("urls", [])]' > /tmp/textbook-$${id}-urls.txt; \
$(ARBORIST) --db "$$shard" ingest --source html --urls-from /tmp/textbook-$${id}-urls.txt; \
$(ARBORIST) --db "$$shard" ingest --source html --urls-from /tmp/textbook-$${id}-urls.txt $$author_flag; \
rm -f /tmp/textbook-$${id}-urls.txt; \
fi

41
arborist/cli.py
View file

@ -56,7 +56,11 @@ def _cmd_ingest(args: argparse.Namespace) -> int:
if not urls:
print("html source needs --url or --urls-from", file=sys.stderr)
return 2
src = HtmlPageSource(urls, respect_robots=not args.no_robots)
src = HtmlPageSource(
urls,
respect_robots=not args.no_robots,
default_author=getattr(args, "author", None),
)
elif args.source in ("grok_export", "grok_media"):
if not args.path:
print(f"--path is required for {args.source}", file=sys.stderr)
@ -87,7 +91,10 @@ def _cmd_ingest(args: argparse.Namespace) -> int:
return 2
from arborist.sources import TextbookTexSource
src = TextbookTexSource(urls)
src = TextbookTexSource(
urls,
default_author=getattr(args, "author", None),
)
elif args.source == "claim_pack":
# Companion JSON bundles (axiom + theorem packs) — see ticket
# #000029. --bundle is repeatable so axiom-bundle and theorem-bundle
@ -4192,7 +4199,12 @@ def _cmd_crawl(args: argparse.Namespace) -> int:
ingest_progress = Progress(prefix="ingest ", total_estimate=len(urls))
conn = connect(args.db)
try:
result = ingest_crawled(conn, urls, progress=ingest_progress)
result = ingest_crawled(
conn,
urls,
progress=ingest_progress,
default_author=getattr(args, "author", None),
)
finally:
conn.close()
print(json.dumps(
@ -4320,6 +4332,18 @@ def build_parser() -> argparse.ArgumentParser:
action="store_true",
help="do not consult robots.txt (use only for explicitly authorized sites)",
)
ingest.add_argument(
"--author",
default=None,
help=(
"default author surname for the html / textbook_tex sources "
"(#000031 Phase 1 follow-up). Appended to document title as "
"'<title>, by <author>' so the warrant resolver's "
"_shard_matches_citation heuristic finds the surname in "
"the title haystack. Wikisource and PG HTML pages rarely "
"include author in <title>; the manifest carries it instead."
),
)
ingest.add_argument(
"--chunker", default=None, help="chunker name (default: tok-512-v1)"
)
@ -5673,6 +5697,17 @@ def build_parser() -> argparse.ArgumentParser:
"against domains where aggressive fetching is acceptable."
),
)
crawl_cmd.add_argument(
"--author",
default=None,
help=(
"default author surname (#000031 Phase 1 follow-up). "
"Appended to ingested document titles when the HTML's "
"<title> doesn't already carry the surname. Threads into "
"_shard_matches_citation's title-haystack for warrant "
"resolution. Only relevant with --ingest."
),
)
crawl_cmd.set_defaults(func=_cmd_crawl)
crawler_cmd = sub.add_parser(

20
arborist/qa/warrant_resolver.py
View file

@ -61,6 +61,14 @@ class Citation:
the fallback. ``section`` carries chapter / section / equation
refs when present (e.g., "§1.2.6 equation (13)") for downstream
chunk-resolution heuristics.
``via_citation_alias`` flags Citations that were appended to a
record's citation list via `#000041` lookup (rather than parsed
from the original ``source_reference``). When True, every
ResolutionMatch produced from this Citation inherits
``via_alias=True`` so the derivations `process_id` correctly
reflects the alias substitution. Defaults False to preserve the
parse-from-source path.
"""
title: str = ""
@ -68,6 +76,7 @@ class Citation:
year: str = ""
section: str = ""
raw: str = ""
via_citation_alias: bool = False
def is_empty(self) -> bool:
return not (self.title or self.authors)
@ -689,6 +698,14 @@ def resolve_chunks(
except Exception:
aliased_queries = []
# Citation-alias substitutions (#000041) propagate via_alias to
# every match they produce — Pass 1 hits from a substitute Citation
# are still alias-driven, just not via term-alias query expansion
# (#000042). Both alias mechanisms collapse to the same
# via_alias=True downstream so process_id correctly attributes
# the derivation as `warrant-resolver-v1+alias`.
citation_alias_floor = bool(getattr(citation, "via_citation_alias", False))
matches: list[ResolutionMatch] = []
for shard in candidate_shards:
if not _shard_matches_citation(str(shard), citation):
@ -710,7 +727,7 @@ def resolve_chunks(
chunk_idx=idx,
score=-float(rank or 0.0),
snippet=snippet or "",
via_alias=False,
via_alias=citation_alias_floor,
)
)
hit = True
@ -1105,6 +1122,7 @@ def warrant_status(
title=sub.substitute_title,
authors=sub.substitute_authors,
raw=sub.substitute_ref,
via_citation_alias=True,
)
)
except Exception:

31
arborist/sources/crawler/bridge.py
View file

@ -216,9 +216,16 @@ class _CrawledHtmlSource:
source_type = "html"
def __init__(self, urls: Iterable[str], *, timeout: float = 30.0):
def __init__(
self,
urls: Iterable[str],
*,
timeout: float = 30.0,
default_author: str | None = None,
):
self.urls = list(urls)
self.timeout = timeout
self.default_author = (default_author or "").strip()
# Filled in during iter_documents — keyed by document_uri (request
# URL pre-redirect) so the caller can map document_root → metadata
# via the documents.document_uri column after ingest.
@ -249,6 +256,25 @@ class _CrawledHtmlSource:
doc = parse_html(str(resp.url), resp.text, self.source_type)
if doc is None:
continue
# Phase 1 follow-up of #000031 — append manifest author
# to title when present and not already a substring.
if self.default_author:
title = (doc.title or "").strip()
title_l = title.lower()
has_surname = False
for tok in self.default_author.split():
tok_l = tok.lower()
if len(tok_l) >= 4 and tok_l in title_l:
has_surname = True
break
if not has_surname:
from dataclasses import replace
new_title = (
f"{title}, by {self.default_author}" if title
else f"by {self.default_author}"
)
doc = replace(doc, title=new_title)
self.http_meta[doc.uri] = {
"etag": resp.headers.get("etag"),
"last_modified": resp.headers.get("last-modified"),
@ -263,6 +289,7 @@ def ingest_crawled(
*,
timeout: float = 30.0,
progress: Progress | None = None,
default_author: str | None = None,
) -> dict:
"""Fetch each URL, ingest into `conn`, record ETag + Last-Modified.
@ -276,7 +303,7 @@ def ingest_crawled(
Pass ``progress`` for stderr heartbeats while ingest runs (the
underlying ingest_source supports it natively).
"""
src = _CrawledHtmlSource(urls, timeout=timeout)
src = _CrawledHtmlSource(urls, timeout=timeout, default_author=default_author)
stats = ingest_source(conn, src, progress=progress)
written: list[dict] = []
if not src.http_meta:

45
arborist/sources/html_page.py
View file

@ -169,13 +169,24 @@ class HtmlPageSource(Source):
loss_report_enabled: bool = True,
loss_report_excerpts: bool = True,
loss_report_max_excerpt_bytes: int = 200,
default_author: str | None = None,
):
"""``default_author`` (Phase 1 follow-up of #000031) — when the
ingested HTML's ``<title>`` doesn't already include the author
surname, this string is appended (``"<title>, by <author>"``).
Surfaces the author signal into the warrant resolver's
``_shard_matches_citation`` haystack without a per-shard SQL
UPDATE workaround. Wikisource and Project Gutenberg HTML pages
rarely include author in ``<title>`` — the manifest entry
carries it instead. Skipped when None or empty.
"""
self.urls = list(urls)
self.respect_robots = respect_robots
self.timeout = timeout
self.loss_report_enabled = loss_report_enabled
self.loss_report_excerpts = loss_report_excerpts
self.loss_report_max_excerpt_bytes = loss_report_max_excerpt_bytes
self.default_author = (default_author or "").strip()
self._robots_cache: dict[str, urllib.robotparser.RobotFileParser] = {}
@classmethod
@ -220,7 +231,7 @@ class HtmlPageSource(Source):
loss_collector=collector,
)
if doc is not None:
yield doc
yield self._with_author_appended(doc)
def _allowed(self, client: "httpx.Client", url: str) -> bool:
parsed = urllib.parse.urlparse(url)
@ -239,3 +250,35 @@ class HtmlPageSource(Source):
rp.allow_all = True
self._robots_cache[origin] = rp
return rp.can_fetch(USER_AGENT, url)
def _with_author_appended(self, doc):
"""Phase 1 follow-up of #000031 — when the source ingest
produces a Document whose title doesn't carry the author
surname, append ``", by <default_author>"`` so the resolver's
author-surname haystack check fires.
Suppress the append when ``default_author`` is empty OR when
the existing title already contains a surname token of length
>= 4 from the configured author (case-insensitive substring).
Idempotent — re-ingesting the same URL with the same config
produces the same Document content_root because content
doesn't change, only the title text appended.
"""
if not self.default_author or not doc:
return doc
title = (doc.title or "").strip()
# If any surname-shaped token from default_author already
# appears in title (case-insensitive), don't append.
title_l = title.lower()
for tok in self.default_author.split():
tok_l = tok.lower()
if len(tok_l) >= 4 and tok_l in title_l:
return doc
new_title = (
f"{title}, by {self.default_author}" if title
else f"by {self.default_author}"
)
# Document is a frozen-ish dataclass; replace title.
from dataclasses import replace
return replace(doc, title=new_title)

40
arborist/sources/textbook_tex.py
View file

@ -264,11 +264,24 @@ class TextbookTexSource(Source):
source_type = "textbook_tex"
def __init__(self, urls: list[str] | str, *, timeout: float = 60.0):
def __init__(
self,
urls: list[str] | str,
*,
timeout: float = 60.0,
default_author: str | None = None,
):
"""``default_author`` (Phase 1 follow-up of #000031) — fallback
when the LaTeX source has no ``\\author{}`` macro and no PG-
style ``Author:`` boilerplate (e.g., Peano's mdnahas/Peano_Book
TeX). Wired into the title via ``"<title> by <author>"`` for
the warrant resolver's author-surname haystack check. Skipped
when None or empty."""
if isinstance(urls, str):
urls = [urls]
self.urls = list(urls)
self.timeout = timeout
self.default_author = (default_author or "").strip()
def iter_documents(self) -> Iterator[Document]:
with httpx.Client(
@ -288,7 +301,16 @@ class TextbookTexSource(Source):
# Stripped output too thin to be useful — likely
# an error page or a non-TeX response.
continue
title = _extract_title(tex) or url
title = _extract_title(tex)
if not title:
# No \title{} or PG Title: line — fall back to URL,
# but still append default_author when configured
# so the haystack picks up the surname.
title = url
if self.default_author and not _title_has_author(
title, self.default_author
):
title = f"{title} by {self.default_author}"
yield Document(
uri=str(resp.url),
content=prose,
@ -314,6 +336,20 @@ _PG_TITLE_RE = re.compile(r"^Title:\s*(.+?)\s*$", re.MULTILINE)
_PG_AUTHOR_RE = re.compile(r"^Author:\s*(.+?)\s*$", re.MULTILINE)
def _title_has_author(title: str, author: str) -> bool:
"""True if any surname-shaped token (length >= 4) from ``author``
appears in ``title`` (case-insensitive substring). Used to decide
whether to append ``default_author`` — skip when the title
already carries the surname (avoids double-stamping).
"""
title_l = (title or "").lower()
for tok in (author or "").split():
tok_l = tok.lower()
if len(tok_l) >= 4 and tok_l in title_l:
return True
return False
def _extract_title(tex: str) -> str:
"""Pull the title — prefer ``\\title{...}`` if present (LaTeX
convention), fall back to ``Title:`` line (Project Gutenberg

12
bench/scripts/textbooks_manifest.py
View file

@ -180,9 +180,12 @@ def cmd_tex_targets(stream: TextIO) -> int:
def cmd_lookup(stream: TextIO) -> int:
"""Read entry id from sys.argv[2]; emit tab-separated
`id\\tcrawl_url\\tdepth\\tmax\\tlicense\\tdomain` for the matching
entry. Used by per-book make targets to look up one textbook's
crawl parameters from the manifest without grep wizardry.
`id\\tcrawl_url\\tdepth\\tmax\\tlicense\\tdomain\\tauthor` for the
matching entry. Used by per-book make targets to look up one
textbook's crawl parameters + author signal (#000031 Phase 1
follow-up — author flows into ``arborist crawl --author`` /
``arborist ingest --author`` so document titles carry the
surname downstream resolvers need).
"""
target_id = sys.argv[2] if len(sys.argv) > 2 else ""
if not target_id:
@ -199,9 +202,10 @@ def cmd_lookup(stream: TextIO) -> int:
crawl_url = entry.get("crawl_url", "")
depth = entry.get("crawl_depth", 2)
max_pages = entry.get("crawl_max", 80)
author = entry.get("author", "")
print(
f"{entry.get('id')}\t{crawl_url}\t{depth}\t{max_pages}\t"
f"{entry.get('license')}\t{entry.get('domain')}"
f"{entry.get('license')}\t{entry.get('domain')}\t{author}"
)
return 0
print(f"# id {target_id!r} not found in manifest", file=sys.stderr)

122
tests/test_warrant_resolver.py
View file

@ -216,3 +216,125 @@ def test_build_record_query_cascade_orders_correctly():
assert queries[0] == '"line incidence"'
# The content-token AND-join should appear in the cascade.
assert any(" AND " in q for q in queries[1:])
# --- B-1: via_citation_alias attribution -----------------------
def test_citation_via_citation_alias_default_false():
"""Citations parsed from the original source_reference start
with via_citation_alias=False — backward-compatible."""
from arborist.qa.warrant_resolver import Citation, parse_citation
cs = parse_citation("Foundations of Geometry by David Hilbert")
assert all(c.via_citation_alias is False for c in cs)
def test_citation_explicit_via_citation_alias_flag():
"""Construct a substitute Citation as if from #000041 lookup —
via_citation_alias=True. Used by warrant_resolve to flag chains
that came through a citation-alias substitution rather than the
parsed source_reference."""
from arborist.qa.warrant_resolver import Citation
sub = Citation(
title="Russell IMP",
authors=("Bertrand Russell",),
raw="Russell IMP by Bertrand Russell",
via_citation_alias=True,
)
assert sub.via_citation_alias is True
def test_resolve_chunks_propagates_via_citation_alias_to_match(tmp_path, monkeypatch):
"""When resolve_chunks is called with a Citation whose
via_citation_alias=True is set, every ResolutionMatch it produces
inherits via_alias=True so process_id correctly attributes the
derivation as `warrant-resolver-v1+alias`."""
import sqlite3
from pathlib import Path
from arborist.qa.warrant_resolver import (
Citation,
ResolutionMatch,
resolve_chunks,
)
# Build a tiny fake shard cluster: shards/000.db (empty) +
# crawl/textbook_test.db with one document + one chunk that the
# _shard_matches_citation heuristic can find by author surname.
shards_dir = tmp_path / "shards"
crawl_dir = tmp_path / "crawl"
shards_dir.mkdir()
crawl_dir.mkdir()
from arborist.store import SCHEMA_SQL
main_db = shards_dir / "000.db"
sqlite3.connect(str(main_db)).executescript(SCHEMA_SQL).close()
sub_db = crawl_dir / "textbook_test.db"
conn = sqlite3.connect(str(sub_db))
conn.executescript(SCHEMA_SQL)
conn.execute(
"INSERT INTO documents "
"(document_root, document_uri, source_type, kind, "
" compression_depth, title, chunking_version, "
" canonicalization_version, schema_version, ingest_ts) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
("a" * 64, "https://test/", "html", "surface", 0,
"Test Book by Bertrand Russell", "tok-512-v1",
"norm-v1", "v9.8.0", 0),
)
conn.execute(
"INSERT INTO chunks (chunk_id, document_root, idx, leaf_hash, content) "
"VALUES (1, ?, 0, ?, ?)",
("a" * 64, "h" * 64, "philosophy mathematics test sample"),
)
conn.execute(
"INSERT INTO chunks_fts (rowid, content) VALUES (1, ?)",
("philosophy mathematics test sample",),
)
conn.commit()
conn.close()
# Citation with via_citation_alias=True (mimics what #000041
# citation-alias lookup appends).
sub_cit = Citation(
title="Test Book",
authors=("Bertrand Russell",),
raw="Test Book by Bertrand Russell",
via_citation_alias=True,
)
matches = resolve_chunks(
sub_cit,
shards_dir,
theorem_name="Axiom of Test",
record_content="philosophy mathematics test sample axiom",
limit=3,
)
assert matches, "expected a match for the substitute citation"
assert all(
isinstance(m, ResolutionMatch) and m.via_alias is True
for m in matches
), "every match from a via_citation_alias=True Citation must be flagged via_alias"
# Sanity check the inverse — Citation without the flag → matches
# are via_alias=False (the existing direct-cascade path).
parsed_cit = Citation(
title="Test Book",
authors=("Bertrand Russell",),
raw="Test Book by Bertrand Russell",
)
plain_matches = resolve_chunks(
parsed_cit,
shards_dir,
theorem_name="Axiom of Test",
record_content="philosophy mathematics test sample axiom",
limit=3,
)
assert plain_matches
assert all(m.via_alias is False for m in plain_matches), (
"Citation parsed from source_ref (no via_citation_alias) "
"must NOT inherit via_alias=True from the citation_alias_floor"
)