arborist/bench/fixtures/textbooks

Textbook surface-ingest manifest

Public-domain and copyleft-redistributable math / logic / computer-science textbooks fetched into a dedicated arborist shard via the existing HtmlPageSource ingest pipeline.

Files

• manifest-v1.jsonl — one JSON record per textbook. First line is a _meta block; remaining lines are entries.

Make targets

Three ingest paths, idempotent at the database layer:

# Manifest inspection
make textbooks-summary    # license + URL counts per entry
make textbooks-urls       # one URL per line on stdout
make textbook-list        # ingestable textbook ids

# All-in-one: shallow URL ingest into a single shard
make fetch-textbooks      # → $(TEXTBOOK_DB) (default ~/.arborist/textbooks.db)
make textbooks-stats
make textbooks-verify

# All-in-one: deep BFS crawl, one shard per textbook id
make crawl-textbooks      # → $(CRAWL_SHARDS_DIR)/textbook_<id>.db each
make crawl-textbooks-stats

# Per-book: refresh / ingest one textbook
make textbook ID=<id>     # generic; works on any manifest entry id
make textbook-bogart           # convenience target
make textbook-keller-trotter   # convenience target (slow ~25min)
make textbook-levin            # convenience target
make textbook-aristotle-prior  # convenience target
make textbook-aristotle-posterior  # convenience target
make textbook-newton           # convenience target
make textbook-morin            # convenience target (Open Data Structures)

Idempotency

Database-layer: every ingest path is content-addressed. Same content → same document_root → no-op insert. Re-running any target on the same data is safe — no duplicates, no schema changes, no audit-event proliferation.

Network-layer: make textbook-* and make crawl-textbooks do refetch HTTP every run. To probe freshness without paying network cost, use the existing make recrawl-check target — it sends conditional HEAD requests (If-None-Match, If-Modified-Since) and classifies each ingested doc as fresh / stale / gone. Pair with selective re-ingest based on that report.

The shallow fetch-textbooks path is fast but limited to the URLs you list in the manifest's urls field. The deep crawl-textbooks path uses the existing crawler per textbook home, honors robots.txt

• crawl delays, and yields full-book coverage. The per-book targets (textbook-*) call into the deep path with one entry's parameters.

Shard layout

Target	Shard path
fetch-textbooks	$(TEXTBOOK_DB) (single) — default ~/.arborist/textbooks.db
crawl-textbooks / textbook ID=...	$(CRAWL_SHARDS_DIR)/textbook_<id>.db (one per textbook) — default ~/.arborist/crawl/

Crawl shards live in ~/.arborist/crawl/ to keep them out of the main ~/.arborist/shards/ directory — SQLite caps simultaneous attached databases at 10, so co-mingling 4 main shards + N crawl shards trips the limit on cross-shard queries.

Crawl-Delay caveat

appliedcombinatorics.org's robots.txt declares Crawl-Delay: 20 — full BFS of Keller-Trotter takes ~25 minutes. Background it (make textbook-keller-trotter &) and let it run; the others all complete in seconds-to-minutes.

License discipline (fail-closed)

The helper bench/scripts/textbooks_manifest.py refuses to emit URLs from entries with missing or disallowed license tokens. The allow-list:

Token	Meaning
PD	Public domain by age or explicit dedication
CC0-1.0	Public domain dedication
CC-BY-2.5 / CC-BY-3.0 / CC-BY-4.0	Creative Commons Attribution
CC-BY-SA-2.5 / CC-BY-SA-3.0 / CC-BY-SA-4.0	Attribution-ShareAlike
GFDL-1.2 / GFDL-1.3	GNU Free Documentation License
AGPL-3.0-only / Apache-2.0 / MIT	OSI-approved redistributable

Excluded: any CC-BY-NC-* (non-commercial clauses are incompatible with arborist's AGPLv3 distribution profile), any CC-BY-ND-* (no-derivatives prevents chunking), and any proprietary / educational- use-only license. These belong on a separate "private research only" manifest if needed; do not mix them in.

Schema

Required fields per entry:

Field	Type	Notes
id	string	Hyphen-lowercase slug, must be unique
title	string	Human-readable book title
author	string	Authors / translators
year	string	Publication year (any reasonable string)
license	string	Token from the allow-list above
license_url	string	Canonical URL of the license text
domain	string	Domain tag (logic, combinatorics, …)
urls	array	Shallow-fetch seed URLs (HtmlPageSource path)

Crawler fields (used by make crawl-textbooks; entries without crawl_url are skipped on the deep path):

Field	Type	Notes
crawl_url	string	Seed URL for BFS — should land directly on substantive content, NOT on a meta-refresh redirect (e.g., frontmatter.html, not index.html)
crawl_depth	int	BFS depth (default 2)
crawl_max	int	Max pages to fetch (default 80)

Optional but recommended:

Field	Type	Notes
home_url	string	Top-level human-facing entry point of the text
pillar_targets	array	Roman-numeral pillar IDs the text supports (e.g., ["I", "VII"])
notes	string	Free-text caveats / version pinning
tex_url	string	Direct LaTeX source URL (e.g., Project Gutenberg /files/N/N-t/N-t.tex) — awaits PDF/TeX source ticket
pdf_url	string	Direct PDF URL — awaits PDF source ticket

How the consistent process works

1. make textbooks-urls invokes bench/scripts/textbooks_manifest.py urls, which validates every entry's license + emits its seed URLs (deduped).
2. make fetch-textbooks pipes that URL list into arborist ingest --source html --urls-from.
3. arborist/sources/html_page.py (HtmlPageSource) handles each URL:
   • Honors robots.txt (default: respect_robots=True).
   • Strips noise (script / style / nav / footer / header / aside).
   • Extracts main body text + outbound <a href> links as Edges.
   • Records loss reports for stripped-noise spans (#000022).
4. The standard ingest path takes over: 512-token chunker + per-doc Merkle root + per-chunk inclusion proofs + audit-event with chained sha256.
5. Idempotent re-ingest: same URL + same body → same document_root → no-op insert. Same URL + different body → new doc + supersedes edge (lossless history).

This is the same pipeline every other surface uses; no new ingest machinery was added for textbooks.

Adding a textbook

1. Confirm the license. If PD / CC0 / CC-BY / CC-BY-SA / GFDL / OSI-approved → proceed. If CC-BY-NC / ND / proprietary → stop; do not add to this manifest.
2. Append a JSON record to manifest-v1.jsonl with the schema above.
3. Run make textbooks-summary — confirm your entry appears.
4. Run make fetch-textbooks — ingest into $(TEXTBOOK_DB).
5. make textbooks-verify — confirm Merkle proofs pass.

Going deeper on one textbook

The manifest's urls field is a small seed list per entry — usually just a TOC or representative chapter. To pull a full book:

• Append more chapter URLs to the manifest entry (cheapest path).
• OR use make crawl-ingest URL=<home_url> DEPTH=N (the existing crawler) with the textbook's home URL. The crawler does BFS, honors robots.txt, dedups by URL, and writes to a separate crawl_<domain>.db shard.

Out of scope

• PDF processing. The Internet Archive hosts PD textbooks as PDFs (MacMahon's Combinatory Analysis 1915, Whitworth's Choice and Chance 1867, the Motte 1729 Principia). Adding a TextbookPdfSource with pdftotext or pypdf extraction is a separate ticket; the current scope is HTML-shaped sources.
• Mendelson + Enderton (proprietary logic textbooks cited by the claim-pack pillar I records). Out by license discipline; awaits a separate decision per #000031 §2.1.
• Wilf's generatingfunctionology. The free PDF from UPenn forbids commercial use and rehosting. Citable but not ingestible here under our redistribution profile.

Vital-books checklist

Coverage by g4 pillar (per the claim-pack source #000029):

Pillar	Domain	Status	Source
I	Logic	✓ landed	Levin (CC-BY-SA), Aristotle Prior (PD), Aristotle Posterior (PD), Boole (PD, PG TeX)
II	Set Theory	✓ landed	Levin (CC-BY-SA) — covers ZF basics
III	Arithmetic (Peano)	✓ landed	Levin (CC-BY-SA) — covers Peano basics
IV	Geometry	✓ landed	Hilbert (PD, PG TeX)
V	Probability	✗ pending	Kolmogorov 1933 license analysis pending
VI	Classical Physics	✓ landed	Newton Principia Motte (PD, Wikisource)
VII	Combinatorics	✓ landed	Bogart (GFDL), Keller-Trotter (CC-BY-SA, slow), Levin
IX	λ-Calculus	✗ pending	Church 1936 + Turing 1936 papers; need separate ingest path

6 of 7 pillars have surface coverage. Remaining gaps:

• Pillar V (Probability) — Kolmogorov Foundations of the Theory of Probability license analysis pending (German original PD-by- age in EU; US copyright restored via URAA through 2058).
• Pillar IX (λ-Calculus) — Church 1936 + Turing 1936 papers rather than full books; awaits a paper-level ingest helper.

TeX-source ingest path

Two PD textbooks ship from Project Gutenberg as LaTeX source only (no clean HTML edition):

make textbook-hilbert  # PG #17384 — Foundations of Geometry
make textbook-boole    # PG #15114 — Laws of Thought
make textbooks-tex     # all manifest entries with a tex_url field

These use arborist ingest --source textbook_tex which fetches the .tex URL, runs a focused PG-aware LaTeX-strip pipeline (drops preamble + comments + tabular/figure envs; keeps argument of \textbf / \emph / \section etc.; substitutes \to → →, \neg → ¬, \forall → ∀, etc.), and produces plain prose for the standard 512-token chunker.

See also

• Ticket #000031 — surface-ingest cited textbooks (warrant promotion for claim-pack records).
• Ticket #000033 — claim-pack pillar VII (combinatorics) which cites several entries on this manifest as primary sources.
• arborist/sources/html_page.py — the consistent-process source.