textbooks: TeX-source ingest closes pillars I + IV (Hilbert + Boole)
Two foundational PD textbooks ship from Project Gutenberg as
LaTeX source only — no clean HTML edition. Pandoc fails on PG's
custom preamble macros; a focused regex-based stripper is the
right amount of machinery for the well-known PG TeX format.
What landed
===========
- arborist/sources/textbook_tex.py — TextbookTexSource +
strip_tex pipeline. Drops preamble + line comments + structural
envs (tabular, figure, thebibliography, scshape, …); keeps the
argument of structural-but-content-bearing single-arg commands
(textbf, emph, section, chapter, paragraph, PG's custom \\rfa);
drops zero-arg + brace-arg structural commands (noindent,
thispagestyle, setcounter, label, index, …); substitutes
symbol-level macros (\\to → →, \\neg → ¬, \\forall → ∀, \\S → §,
Greek letters, etc.).
- arborist/cli.py — `--source textbook_tex` accepts --url,
--bundle, or --urls-from. Reuses the existing fetch + chunker
+ Merkle commit + audit pipeline; idempotent at the database
layer (same TeX → same prose → same document_root).
- bench/scripts/textbooks_manifest.py — gains `tex-targets`
subcommand emitting tab-separated `<tex_url>\t<id>` rows for
manifest entries with a `tex_url` field.
- Makefile targets:
textbooks-tex — ingest every entry with a tex_url
textbook-hilbert — convenience for PG #17384
textbook-boole — convenience for PG #15114
Each writes to $(CRAWL_SHARDS_DIR)/textbook_<id>.db, idempotent
on re-run.
- tests/test_textbook_tex.py — 20 unit tests covering preamble +
postmatter stripping, line comments, env drops (tabular, figure),
single-arg keepers (\\textbf, \\emph, \\section, \\rfa),
symbol-level macro subs (10 paramerized cases), structural-cmd
drops, whitespace cleanup, idempotence on already-stripped text.
End-to-end verification
=======================
Smoke test on PG #17384 + #15114:
Hilbert Foundations of Geometry: 1 doc / 65 chunks (192K of
plain prose). FTS5 finds "axiom of parallels" → real chapter
content with axiom references intact (≡, §, math fragments).
Boole Laws of Thought: 1 doc / 273 chunks (829K). FTS5 finds
"law of contradiction" → "the principle of contradiction"
passage from Chapter III of Boole's text.
Vital-books coverage now 6/7 pillars
====================================
Pillar I Logic ✓ Levin + Aristotle Prior + Posterior + Boole
Pillar II Set Theory ✓ Levin
Pillar III Arithmetic ✓ Levin
Pillar IV Geometry ✓ Hilbert (PG TeX)
Pillar V Probability ✗ Kolmogorov license analysis pending
Pillar VI Phys. ✓ Newton Principia
Pillar VII Combin. ✓ Bogart + Keller-Trotter + Levin
Pillar IX λ-Calculus ✗ Church + Turing 1936 papers pending
Test suite: 1574 passed / 28 skipped (was 1554 + 20 new TeX tests).
Out of scope: chunk-resolution + derivations.proof_blob warrant
promotion (#000031 follow-up; see also #000032).
This commit is contained in:
parent
57dafbdc3d
commit
514e07d7c2
7 changed files with 644 additions and 6 deletions
34
Makefile
34
Makefile
|
|
@ -770,6 +770,40 @@ textbook-newton: ## Newton Principia (PD, Wikisource)
|
|||
textbook-morin: ## Morin Open Data Structures (CC-BY)
|
||||
$(MAKE) textbook ID=morin-open-data-structures
|
||||
|
||||
# TeX-source textbooks: Project Gutenberg eBooks that ship as
|
||||
# LaTeX source only (no clean HTML edition). The textbook_tex
|
||||
# source strips the LaTeX into plain prose via a focused PG-aware
|
||||
# pipeline. Idempotent at the database layer like every other
|
||||
# ingest path.
|
||||
.PHONY: textbooks-tex textbook-hilbert textbook-boole
|
||||
|
||||
textbooks-tex: bootstrap ## ingest every manifest entry that declares a tex_url (Hilbert + Boole)
|
||||
@mkdir -p $(CRAWL_SHARDS_DIR)
|
||||
@$(PY) -m bench.scripts.textbooks_manifest tex-targets < $(TEXTBOOK_MANIFEST) | \
|
||||
while IFS=$$'\t' read -r url id; do \
|
||||
shard="$(CRAWL_SHARDS_DIR)/textbook_$${id}.db"; \
|
||||
echo ">> $$id (textbook_tex)"; \
|
||||
echo " url: $$url"; \
|
||||
echo " shard: $$shard"; \
|
||||
$(ARBORIST) --db "$$shard" ingest --source textbook_tex --url "$$url"; \
|
||||
done
|
||||
|
||||
textbook-hilbert: ## Hilbert Foundations of Geometry (PD, PG TeX)
|
||||
@row=$$(grep '"hilbert-foundations-geometry-1902"' $(TEXTBOOK_MANIFEST) | head -1); \
|
||||
url=$$(echo "$$row" | $(PY) -c "import json,sys; print(json.loads(sys.stdin.read()).get('tex_url',''))"); \
|
||||
if [ -z "$$url" ]; then echo "no tex_url for Hilbert" >&2; exit 1; fi; \
|
||||
mkdir -p $(CRAWL_SHARDS_DIR); \
|
||||
$(ARBORIST) --db "$(CRAWL_SHARDS_DIR)/textbook_hilbert-foundations-geometry-1902.db" \
|
||||
ingest --source textbook_tex --url "$$url"
|
||||
|
||||
textbook-boole: ## Boole Laws of Thought (PD, PG TeX)
|
||||
@row=$$(grep '"boole-laws-of-thought-1854"' $(TEXTBOOK_MANIFEST) | head -1); \
|
||||
url=$$(echo "$$row" | $(PY) -c "import json,sys; print(json.loads(sys.stdin.read()).get('tex_url',''))"); \
|
||||
if [ -z "$$url" ]; then echo "no tex_url for Boole" >&2; exit 1; fi; \
|
||||
mkdir -p $(CRAWL_SHARDS_DIR); \
|
||||
$(ARBORIST) --db "$(CRAWL_SHARDS_DIR)/textbook_boole-laws-of-thought-1854.db" \
|
||||
ingest --source textbook_tex --url "$$url"
|
||||
|
||||
test-crawler: bootstrap-crawler ## run only the lifted crawler tests
|
||||
$(VENV)/bin/pytest -q tests/crawler
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue