|
|
514e07d7c2
|
textbooks: TeX-source ingest closes pillars I + IV (Hilbert + Boole)
Two foundational PD textbooks ship from Project Gutenberg as
LaTeX source only — no clean HTML edition. Pandoc fails on PG's
custom preamble macros; a focused regex-based stripper is the
right amount of machinery for the well-known PG TeX format.
What landed
===========
- arborist/sources/textbook_tex.py — TextbookTexSource +
strip_tex pipeline. Drops preamble + line comments + structural
envs (tabular, figure, thebibliography, scshape, …); keeps the
argument of structural-but-content-bearing single-arg commands
(textbf, emph, section, chapter, paragraph, PG's custom \\rfa);
drops zero-arg + brace-arg structural commands (noindent,
thispagestyle, setcounter, label, index, …); substitutes
symbol-level macros (\\to → →, \\neg → ¬, \\forall → ∀, \\S → §,
Greek letters, etc.).
- arborist/cli.py — `--source textbook_tex` accepts --url,
--bundle, or --urls-from. Reuses the existing fetch + chunker
+ Merkle commit + audit pipeline; idempotent at the database
layer (same TeX → same prose → same document_root).
- bench/scripts/textbooks_manifest.py — gains `tex-targets`
subcommand emitting tab-separated `<tex_url>\t<id>` rows for
manifest entries with a `tex_url` field.
- Makefile targets:
textbooks-tex — ingest every entry with a tex_url
textbook-hilbert — convenience for PG #17384
textbook-boole — convenience for PG #15114
Each writes to $(CRAWL_SHARDS_DIR)/textbook_<id>.db, idempotent
on re-run.
- tests/test_textbook_tex.py — 20 unit tests covering preamble +
postmatter stripping, line comments, env drops (tabular, figure),
single-arg keepers (\\textbf, \\emph, \\section, \\rfa),
symbol-level macro subs (10 paramerized cases), structural-cmd
drops, whitespace cleanup, idempotence on already-stripped text.
End-to-end verification
=======================
Smoke test on PG #17384 + #15114:
Hilbert Foundations of Geometry: 1 doc / 65 chunks (192K of
plain prose). FTS5 finds "axiom of parallels" → real chapter
content with axiom references intact (≡, §, math fragments).
Boole Laws of Thought: 1 doc / 273 chunks (829K). FTS5 finds
"law of contradiction" → "the principle of contradiction"
passage from Chapter III of Boole's text.
Vital-books coverage now 6/7 pillars
====================================
Pillar I Logic ✓ Levin + Aristotle Prior + Posterior + Boole
Pillar II Set Theory ✓ Levin
Pillar III Arithmetic ✓ Levin
Pillar IV Geometry ✓ Hilbert (PG TeX)
Pillar V Probability ✗ Kolmogorov license analysis pending
Pillar VI Phys. ✓ Newton Principia
Pillar VII Combin. ✓ Bogart + Keller-Trotter + Levin
Pillar IX λ-Calculus ✗ Church + Turing 1936 papers pending
Test suite: 1574 passed / 28 skipped (was 1554 + 20 new TeX tests).
Out of scope: chunk-resolution + derivations.proof_blob warrant
promotion (#000031 follow-up; see also #000032).
|
2026-05-09 16:06:53 -04:00 |
|
