textbooks: TeX-source ingest closes pillars I + IV (Hilbert + Boole)

Two foundational PD textbooks ship from Project Gutenberg as
LaTeX source only — no clean HTML edition. Pandoc fails on PG's
custom preamble macros; a focused regex-based stripper is the
right amount of machinery for the well-known PG TeX format.

What landed
===========
- arborist/sources/textbook_tex.py — TextbookTexSource +
  strip_tex pipeline. Drops preamble + line comments + structural
  envs (tabular, figure, thebibliography, scshape, …); keeps the
  argument of structural-but-content-bearing single-arg commands
  (textbf, emph, section, chapter, paragraph, PG's custom \\rfa);
  drops zero-arg + brace-arg structural commands (noindent,
  thispagestyle, setcounter, label, index, …); substitutes
  symbol-level macros (\\to → →, \\neg → ¬, \\forall → ∀, \\S → §,
  Greek letters, etc.).

- arborist/cli.py — `--source textbook_tex` accepts --url,
  --bundle, or --urls-from. Reuses the existing fetch + chunker
  + Merkle commit + audit pipeline; idempotent at the database
  layer (same TeX → same prose → same document_root).

- bench/scripts/textbooks_manifest.py — gains `tex-targets`
  subcommand emitting tab-separated `<tex_url>\t<id>` rows for
  manifest entries with a `tex_url` field.

- Makefile targets:
    textbooks-tex         — ingest every entry with a tex_url
    textbook-hilbert      — convenience for PG #17384
    textbook-boole        — convenience for PG #15114
  Each writes to $(CRAWL_SHARDS_DIR)/textbook_<id>.db, idempotent
  on re-run.

- tests/test_textbook_tex.py — 20 unit tests covering preamble +
  postmatter stripping, line comments, env drops (tabular, figure),
  single-arg keepers (\\textbf, \\emph, \\section, \\rfa),
  symbol-level macro subs (10 paramerized cases), structural-cmd
  drops, whitespace cleanup, idempotence on already-stripped text.

End-to-end verification
=======================
Smoke test on PG #17384 + #15114:
  Hilbert Foundations of Geometry: 1 doc / 65 chunks (192K of
    plain prose). FTS5 finds "axiom of parallels" → real chapter
    content with axiom references intact (≡, §, math fragments).
  Boole Laws of Thought: 1 doc / 273 chunks (829K). FTS5 finds
    "law of contradiction" → "the principle of contradiction"
    passage from Chapter III of Boole's text.

Vital-books coverage now 6/7 pillars
====================================

  Pillar I   Logic        ✓ Levin + Aristotle Prior + Posterior + Boole
  Pillar II  Set Theory   ✓ Levin
  Pillar III Arithmetic   ✓ Levin
  Pillar IV  Geometry     ✓ Hilbert (PG TeX)
  Pillar V   Probability  ✗ Kolmogorov license analysis pending
  Pillar VI  Phys.        ✓ Newton Principia
  Pillar VII Combin.      ✓ Bogart + Keller-Trotter + Levin
  Pillar IX  λ-Calculus   ✗ Church + Turing 1936 papers pending

Test suite: 1574 passed / 28 skipped (was 1554 + 20 new TeX tests).

Out of scope: chunk-resolution + derivations.proof_blob warrant
promotion (#000031 follow-up; see also #000032).
This commit is contained in:
russell@unturf.com 2026-05-09 16:06:53 -04:00
parent 57dafbdc3d
commit 514e07d7c2
No known key found for this signature in database
7 changed files with 644 additions and 6 deletions

View file

@ -770,6 +770,40 @@ textbook-newton: ## Newton Principia (PD, Wikisource)
textbook-morin: ## Morin Open Data Structures (CC-BY)
$(MAKE) textbook ID=morin-open-data-structures
# TeX-source textbooks: Project Gutenberg eBooks that ship as
# LaTeX source only (no clean HTML edition). The textbook_tex
# source strips the LaTeX into plain prose via a focused PG-aware
# pipeline. Idempotent at the database layer like every other
# ingest path.
.PHONY: textbooks-tex textbook-hilbert textbook-boole
textbooks-tex: bootstrap ## ingest every manifest entry that declares a tex_url (Hilbert + Boole)
@mkdir -p $(CRAWL_SHARDS_DIR)
@$(PY) -m bench.scripts.textbooks_manifest tex-targets < $(TEXTBOOK_MANIFEST) | \
while IFS=$$'\t' read -r url id; do \
shard="$(CRAWL_SHARDS_DIR)/textbook_$${id}.db"; \
echo ">> $$id (textbook_tex)"; \
echo " url: $$url"; \
echo " shard: $$shard"; \
$(ARBORIST) --db "$$shard" ingest --source textbook_tex --url "$$url"; \
done
textbook-hilbert: ## Hilbert Foundations of Geometry (PD, PG TeX)
@row=$$(grep '"hilbert-foundations-geometry-1902"' $(TEXTBOOK_MANIFEST) | head -1); \
url=$$(echo "$$row" | $(PY) -c "import json,sys; print(json.loads(sys.stdin.read()).get('tex_url',''))"); \
if [ -z "$$url" ]; then echo "no tex_url for Hilbert" >&2; exit 1; fi; \
mkdir -p $(CRAWL_SHARDS_DIR); \
$(ARBORIST) --db "$(CRAWL_SHARDS_DIR)/textbook_hilbert-foundations-geometry-1902.db" \
ingest --source textbook_tex --url "$$url"
textbook-boole: ## Boole Laws of Thought (PD, PG TeX)
@row=$$(grep '"boole-laws-of-thought-1854"' $(TEXTBOOK_MANIFEST) | head -1); \
url=$$(echo "$$row" | $(PY) -c "import json,sys; print(json.loads(sys.stdin.read()).get('tex_url',''))"); \
if [ -z "$$url" ]; then echo "no tex_url for Boole" >&2; exit 1; fi; \
mkdir -p $(CRAWL_SHARDS_DIR); \
$(ARBORIST) --db "$(CRAWL_SHARDS_DIR)/textbook_boole-laws-of-thought-1854.db" \
ingest --source textbook_tex --url "$$url"
test-crawler: bootstrap-crawler ## run only the lifted crawler tests
$(VENV)/bin/pytest -q tests/crawler