engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

ticket #000031 follow-ups B-1 + B-2: alias attribution + source-side title author

Browse source 
B-1: via_citation_alias attribution — resolver no longer mislabels
citation-alias-substituted chains as DIRECT.

  - New Citation.via_citation_alias field (default False, preserves
    parse-from-source-ref path).
  - warrant_status sets via_citation_alias=True on substitute
    Citations from #000041 lookup_citation_aliases.
  - resolve_chunks reads citation.via_citation_alias as a "floor" for
    via_alias on every match it produces (Pass 1 hits inherit it
    too, not just Pass 2 term-alias hits). Audit-honest: matches
    from a substitute Citation are alias-driven regardless of which
    cascade pass found the chunk.

  Live re-resolve under the new attribution: 18 direct + 74 +alias
  (was 75/17 mis-labeled). The 18 direct = exactly Hilbert pillar IV
  records resolving on the literally-cited Hilbert textbook. All 74
  records resolved via citation-alias substitution now carry
  process_id="warrant-resolver-v1+alias" in derivations.

  3 new unit tests (test_warrant_resolver.py): default-False on
  parsed Citations, explicit-True construction works, resolve_chunks
  propagates the floor onto every ResolutionMatch.

B-2: source-side title-from-author backfill — eliminates the
per-shard SQL UPDATE workaround.

  - HtmlPageSource accepts default_author kwarg; appends ', by
    <author>' to ingested document titles when the <title> tag
    doesn't already include the surname.
  - TextbookTexSource accepts default_author kwarg; appends ' by
    <author>' to titles when the LaTeX has no \author{} macro AND
    no PG-style 'Author:' boilerplate.
  - _CrawledHtmlSource (BFS-crawler bridge) accepts default_author
    kwarg; same append logic. ingest_crawled() and arborist crawl
    --ingest plumb it through.
  - arborist ingest --author + arborist crawl --author CLI flags.
  - bench/scripts/textbooks_manifest.py:cmd_lookup emits the
    manifest's `author` field as a 7th tab column.
  - make textbook target reads the author column and threads
    --author into both crawl-ingest and shallow-ingest paths.

  Idempotency preserved — surname-already-in-title detection prevents
  double-stamping on re-ingest. Shards previously SQL-backfilled
  (Cantor / Russell IMP / Bogart / Judson / Levin / KT / Peano /
  Grinstead-Snell) keep their existing titles; new ingests pick up
  the author signal at source time.

  Live smoke: arborist ingest --source html --author "Bertrand
  Russell" against PG #41654 yields title "Introduction to
  Mathematical Philosophy | Project Gutenberg, by Bertrand Russell"
  with no SQL UPDATE needed.

Total: 1655 tests pass (was 1652). Both follow-ups land additive,
fail-closed, idempotent. The two cleanup items from #000031
Phase 3's commit message are now closed.
This commit is contained in:
russell@unturf.com 2026-05-10 09:35:49 -04:00
parent 7e81425d49
commit 551c9695e0
No known key found for this signature in database
8 changed files with 304 additions and 15 deletions
Download patch file Download diff file Expand all files Collapse all files

8
Makefile
View file

@ -766,20 +766,24 @@ textbook: bootstrap-crawler ## ingest one textbook by id: make textbook ID=bogar
max=$$(echo "$$row" | cut -f4); \
license=$$(echo "$$row" | cut -f5); \
domain=$$(echo "$$row" | cut -f6); \
author=$$(echo "$$row" | cut -f7); \
mkdir -p $(CRAWL_SHARDS_DIR); \
shard="$(CRAWL_SHARDS_DIR)/textbook_$${id}.db"; \
author_flag=""; \
if [ -n "$$author" ]; then author_flag="--author $$author"; fi; \
echo ">> $$id ($$license, $$domain)"; \
echo " crawl: $$url depth=$$depth max=$$max"; \
echo " shard: $$shard"; \
if [ -n "$$author" ]; then echo " author: $$author"; fi; \
if [ -n "$$url" ]; then \
$(ARBORIST) --db "$$shard" crawl --seed-url "$$url" --depth $$depth --max-pages $$max --ingest > /tmp/textbook-$${id}.log 2>&1 && \
$(ARBORIST) --db "$$shard" crawl --seed-url "$$url" --depth $$depth --max-pages $$max --ingest $$author_flag > /tmp/textbook-$${id}.log 2>&1 && \
docs=$$(.venv/bin/python -c "import sqlite3; print(sqlite3.connect(\"$$shard\").execute(\"SELECT COUNT(*) FROM documents\").fetchone()[0])"); \
chunks=$$(.venv/bin/python -c "import sqlite3; print(sqlite3.connect(\"$$shard\").execute(\"SELECT COUNT(*) FROM chunks\").fetchone()[0])"); \
echo " landed: $$docs docs / $$chunks chunks"; \
else \
echo " no crawl_url for $$id; using shallow ingest path"; \
echo "$$id" | $(PY) -c 'import sys, json, os; sys.path.insert(0, "."); ids = sys.stdin.read().split(); manifest = "$(TEXTBOOK_MANIFEST)"; [print(u) for line in open(manifest) for e in [json.loads(line)] if "_meta" not in e and e.get("id") in ids for u in e.get("urls", [])]' > /tmp/textbook-$${id}-urls.txt; \
$(ARBORIST) --db "$$shard" ingest --source html --urls-from /tmp/textbook-$${id}-urls.txt; \
$(ARBORIST) --db "$$shard" ingest --source html --urls-from /tmp/textbook-$${id}-urls.txt $$author_flag; \
rm -f /tmp/textbook-$${id}-urls.txt; \
fi