# Textbook surface-ingest manifest

Public-domain and copyleft-redistributable math / logic / computer-science
textbooks fetched into a dedicated arborist shard via the existing
`HtmlPageSource` ingest pipeline.

## Files

- `manifest-v1.jsonl` — one JSON record per textbook. First line is a
  `_meta` block; remaining lines are entries.

## Make targets

Three ingest paths, idempotent at the database layer:

```bash
# Manifest inspection
make textbooks-summary    # license + URL counts per entry
make textbooks-urls       # one URL per line on stdout
make textbook-list        # ingestable textbook ids

# All-in-one: shallow URL ingest into a single shard
make fetch-textbooks      # → $(TEXTBOOK_DB) (default ~/.arborist/textbooks.db)
make textbooks-stats
make textbooks-verify

# All-in-one: deep BFS crawl, one shard per textbook id
make crawl-textbooks      # → $(CRAWL_SHARDS_DIR)/textbook_<id>.db each
make crawl-textbooks-stats

# Per-book: refresh / ingest one textbook
make textbook ID=<id>     # generic; works on any manifest entry id
make textbook-bogart           # convenience target
make textbook-keller-trotter   # convenience target (slow ~25min)
make textbook-levin            # convenience target
make textbook-aristotle-prior  # convenience target
make textbook-aristotle-posterior  # convenience target
make textbook-newton           # convenience target
make textbook-morin            # convenience target (Open Data Structures)
```

## Idempotency

**Database-layer:** every ingest path is content-addressed. Same
content → same `document_root` → no-op insert. Re-running any
target on the same data is safe — no duplicates, no schema
changes, no audit-event proliferation.

**Network-layer:** `make textbook-*` and `make crawl-textbooks`
do refetch HTTP every run. To probe freshness without paying
network cost, use the existing `make recrawl-check` target — it
sends conditional HEAD requests (`If-None-Match`,
`If-Modified-Since`) and classifies each ingested doc as
fresh / stale / gone. Pair with selective re-ingest based on
that report.

The shallow `fetch-textbooks` path is fast but limited to the URLs
you list in the manifest's `urls` field. The deep `crawl-textbooks`
path uses the existing crawler per textbook home, honors robots.txt
+ crawl delays, and yields full-book coverage. The per-book targets
(`textbook-*`) call into the deep path with one entry's parameters.

## Shard layout

| Target | Shard path |
|---|---|
| `fetch-textbooks` | `$(TEXTBOOK_DB)` (single) — default `~/.arborist/textbooks.db` |
| `crawl-textbooks` / `textbook ID=...` | `$(CRAWL_SHARDS_DIR)/textbook_<id>.db` (one per textbook) — default `~/.arborist/crawl/` |

Crawl shards live in `~/.arborist/crawl/` to keep them out of the
main `~/.arborist/shards/` directory — SQLite caps simultaneous
attached databases at 10, so co-mingling 4 main shards + N crawl
shards trips the limit on cross-shard queries.

## Crawl-Delay caveat

`appliedcombinatorics.org`'s `robots.txt` declares
`Crawl-Delay: 20` — full BFS of Keller-Trotter takes ~25 minutes.
Background it (`make textbook-keller-trotter &`) and let it run;
the others all complete in seconds-to-minutes.

## License discipline (fail-closed)

The helper `bench/scripts/textbooks_manifest.py` refuses to emit URLs
from entries with missing or disallowed license tokens. The allow-list:

| Token | Meaning |
|---|---|
| `PD` | Public domain by age or explicit dedication |
| `CC0-1.0` | Public domain dedication |
| `CC-BY-2.5` / `CC-BY-3.0` / `CC-BY-4.0` | Creative Commons Attribution |
| `CC-BY-SA-2.5` / `CC-BY-SA-3.0` / `CC-BY-SA-4.0` | Attribution-ShareAlike |
| `GFDL-1.2` / `GFDL-1.3` | GNU Free Documentation License |
| `AGPL-3.0-only` / `Apache-2.0` / `MIT` | OSI-approved redistributable |

**Excluded:** any `CC-BY-NC-*` (non-commercial clauses are incompatible
with arborist's AGPLv3 distribution profile), any `CC-BY-ND-*`
(no-derivatives prevents chunking), and any proprietary / educational-
use-only license. These belong on a separate "private research only"
manifest if needed; do not mix them in.

## Schema

Required fields per entry:

| Field | Type | Notes |
|---|---|---|
| `id` | string | Hyphen-lowercase slug, must be unique |
| `title` | string | Human-readable book title |
| `author` | string | Authors / translators |
| `year` | string | Publication year (any reasonable string) |
| `license` | string | Token from the allow-list above |
| `license_url` | string | Canonical URL of the license text |
| `domain` | string | Domain tag (`logic`, `combinatorics`, …) |
| `urls` | array | Shallow-fetch seed URLs (HtmlPageSource path) |

Crawler fields (used by `make crawl-textbooks`; entries without
`crawl_url` are skipped on the deep path):

| Field | Type | Notes |
|---|---|---|
| `crawl_url` | string | Seed URL for BFS — should land directly on substantive content, NOT on a meta-refresh redirect (e.g., `frontmatter.html`, not `index.html`) |
| `crawl_depth` | int | BFS depth (default 2) |
| `crawl_max` | int | Max pages to fetch (default 80) |

Optional but recommended:

| Field | Type | Notes |
|---|---|---|
| `home_url` | string | Top-level human-facing entry point of the text |
| `pillar_targets` | array | Roman-numeral pillar IDs the text supports (e.g., `["I", "VII"]`) |
| `notes` | string | Free-text caveats / version pinning |
| `tex_url` | string | Direct LaTeX source URL (e.g., Project Gutenberg `/files/N/N-t/N-t.tex`) — awaits PDF/TeX source ticket |
| `pdf_url` | string | Direct PDF URL — awaits PDF source ticket |

## How the consistent process works

1. `make textbooks-urls` invokes
   `bench/scripts/textbooks_manifest.py urls`, which validates every
   entry's license + emits its seed URLs (deduped).
2. `make fetch-textbooks` pipes that URL list into
   `arborist ingest --source html --urls-from`.
3. `arborist/sources/html_page.py` (`HtmlPageSource`) handles each URL:
   - Honors `robots.txt` (default: `respect_robots=True`).
   - Strips noise (script / style / nav / footer / header / aside).
   - Extracts main body text + outbound `<a href>` links as `Edge`s.
   - Records loss reports for stripped-noise spans (#000022).
4. The standard ingest path takes over: 512-token chunker + per-doc
   Merkle root + per-chunk inclusion proofs + audit-event with
   chained sha256.
5. Idempotent re-ingest: same URL + same body → same `document_root`
   → no-op insert. Same URL + different body → new doc + `supersedes`
   edge (lossless history).

This is the same pipeline every other surface uses; no new ingest
machinery was added for textbooks.

## Adding a textbook

1. Confirm the license. If `PD` / `CC0` / `CC-BY` / `CC-BY-SA` / `GFDL`
   / OSI-approved → proceed. If `CC-BY-NC` / `ND` / proprietary →
   stop; do not add to this manifest.
2. Append a JSON record to `manifest-v1.jsonl` with the schema above.
3. Run `make textbooks-summary` — confirm your entry appears.
4. Run `make fetch-textbooks` — ingest into `$(TEXTBOOK_DB)`.
5. `make textbooks-verify` — confirm Merkle proofs pass.

## Going deeper on one textbook

The manifest's `urls` field is a small seed list per entry — usually
just a TOC or representative chapter. To pull a full book:

- **Append more chapter URLs to the manifest entry** (cheapest path).
- **OR** use `make crawl-ingest URL=<home_url> DEPTH=N` (the existing
  crawler) with the textbook's home URL. The crawler does BFS,
  honors robots.txt, dedups by URL, and writes to a separate
  `crawl_<domain>.db` shard.

## Out of scope

- **PDF processing.** The Internet Archive hosts PD textbooks as PDFs
  (MacMahon's *Combinatory Analysis* 1915, Whitworth's *Choice and
  Chance* 1867, the Motte 1729 *Principia*). Adding a `TextbookPdfSource`
  with `pdftotext` or `pypdf` extraction is a separate ticket; the
  current scope is HTML-shaped sources.
- **Mendelson + Enderton** (proprietary logic textbooks cited by the
  claim-pack pillar I records). Out by license discipline; awaits a
  separate decision per #000031 §2.1.
- **Wilf's *generatingfunctionology*.** The free PDF from UPenn
  forbids commercial use and rehosting. Citable but not ingestible
  here under our redistribution profile.

## Vital-books checklist

Coverage by g4 pillar (per the claim-pack source #000029):

| Pillar | Domain | Status | Source |
|---|---|---|---|
| I | Logic | ✓ landed | Levin (CC-BY-SA), Aristotle Prior (PD), Aristotle Posterior (PD), Boole (PD, PG TeX) |
| II | Set Theory | ✓ landed | Levin (CC-BY-SA) — covers ZF basics |
| III | Arithmetic (Peano) | ✓ landed | Levin (CC-BY-SA) — covers Peano basics |
| IV | Geometry | ✓ landed | Hilbert (PD, PG TeX) |
| V | Probability | ✗ pending | Kolmogorov 1933 license analysis pending |
| VI | Classical Physics | ✓ landed | Newton Principia Motte (PD, Wikisource) |
| VII | Combinatorics | ✓ landed | Bogart (GFDL), Keller-Trotter (CC-BY-SA, slow), Levin |
| IX | λ-Calculus | ✗ pending | Church 1936 + Turing 1936 papers; need separate ingest path |

**6 of 7 pillars** have surface coverage. Remaining gaps:

- **Pillar V (Probability)** — Kolmogorov *Foundations of the Theory
  of Probability* license analysis pending (German original PD-by-
  age in EU; US copyright restored via URAA through 2058).
- **Pillar IX (λ-Calculus)** — Church 1936 + Turing 1936 papers
  rather than full books; awaits a paper-level ingest helper.

## TeX-source ingest path

Two PD textbooks ship from Project Gutenberg as LaTeX source only
(no clean HTML edition):

```bash
make textbook-hilbert  # PG #17384 — Foundations of Geometry
make textbook-boole    # PG #15114 — Laws of Thought
make textbooks-tex     # all manifest entries with a tex_url field
```

These use `arborist ingest --source textbook_tex` which fetches
the `.tex` URL, runs a focused PG-aware LaTeX-strip pipeline
(drops preamble + comments + `tabular`/`figure` envs;
keeps argument of `\textbf` / `\emph` / `\section` etc.;
substitutes `\to` → →, `\neg` → ¬, `\forall` → ∀, etc.), and
produces plain prose for the standard 512-token chunker.

## See also

- Ticket #000031 — surface-ingest cited textbooks (warrant promotion
  for claim-pack records).
- Ticket #000033 — claim-pack pillar VII (combinatorics) which cites
  several entries on this manifest as primary sources.
- `arborist/sources/html_page.py` — the consistent-process source.
