textbooks: per-book make targets + idempotency docs + vital-books checklist

Three concrete improvements to the textbook ingest layer:

Per-book make targets
=====================
- `make textbook ID=<id>` — generic, looks up one entry from the
  manifest and runs the right ingest path (BFS crawl if
  `crawl_url` is set, shallow URL ingest otherwise).
- `make textbook-list` — list ingestable textbook ids.
- Convenience targets per active manifest entry:
    textbook-bogart, textbook-keller-trotter, textbook-levin,
    textbook-aristotle-prior, textbook-aristotle-posterior,
    textbook-newton, textbook-morin

Each per-book target writes to a dedicated shard at
`$(CRAWL_SHARDS_DIR)/textbook_<id>.db` — one shard per textbook
id rather than one per host (which would conflate multiple
Wikisource books — Aristotle Prior, Posterior, Newton — into
one shard).

The manifest helper script gains `ids` and `lookup <id>`
subcommands so the Makefile can introspect manifest entries
without grep gymnastics.

Idempotency contract documented
================================
Every ingest path is content-addressed. Same content →
same `document_root` → no-op insert. Re-running `make textbook-*`
or `make crawl-textbooks` on the same network state is safe at
the database layer — no duplicates, no schema drift, no
audit-event proliferation. HTTP refetch happens every run; pair
with `make recrawl-check` for a HEAD-only freshness probe.

README rewritten to make this contract explicit, with shard-
layout table + crawl-delay caveats consolidated.

Vital-books checklist
=====================
The README now carries a g4-pillar coverage table:

  Pillar I   Logic        ✓ Levin + Aristotle Prior + Posterior
  Pillar II  Set Theory   ✓ Levin
  Pillar III Arithmetic   ✓ Levin
  Pillar IV  Geometry     ✗ Hilbert PG #17384 (PDF/TeX only)
  Pillar V   Probability  ✗ Kolmogorov license analysis pending
  Pillar VI  Phys. Newton ✓ Newton Principia Motte
  Pillar VII Combin.      ✓ Bogart + Keller-Trotter + Levin
  Pillar IX  λ-Calculus   ✗ Church + Turing 1936 papers pending

5 of 7 pillars now have surface coverage. The remaining gaps
are explicit license (V) or format (IV PDF/TeX, IX paper-level)
issues, each documented in the corresponding manifest entry's
notes field.

Test suite stays at 1554 passed / 28 skipped — no source-code
changes to arborist itself; the textbook layer is pure tooling
on top of the existing pipeline.
This commit is contained in:
russell@unturf.com 2026-05-09 15:59:31 -04:00
parent 2acbc0e691
commit 57dafbdc3d
No known key found for this signature in database
3 changed files with 200 additions and 23 deletions

View file

@ -127,9 +127,9 @@ def cmd_summary(stream: TextIO) -> int:
def cmd_crawl_targets(stream: TextIO) -> int:
"""Emit one tab-separated `crawl_url\\tdepth\\tmax` triple per entry
"""Emit one tab-separated `crawl_url\\tdepth\\tmax\\tid` row per entry
that declares a `crawl_url`. Used by `make crawl-textbooks` to drive
the BFS crawler, one shard per host.
the BFS crawler, one shard per textbook id.
"""
for entry in iter_entries(stream):
_validate(entry)
@ -142,7 +142,58 @@ def cmd_crawl_targets(stream: TextIO) -> int:
return 0
_COMMANDS = {"urls": cmd_urls, "summary": cmd_summary, "crawl-targets": cmd_crawl_targets}
def cmd_ids(stream: TextIO) -> int:
"""Emit one entry id per line for entries that have either `urls` or
`crawl_url`. License-fail placeholders (no fetchable URLs) are
excluded — they have nothing to ingest.
"""
for entry in iter_entries(stream):
try:
_validate(entry)
except ValueError:
continue
if entry.get("urls") or entry.get("crawl_url"):
print(entry.get("id", ""))
return 0
def cmd_lookup(stream: TextIO) -> int:
"""Read entry id from sys.argv[2]; emit tab-separated
`id\\tcrawl_url\\tdepth\\tmax\\tlicense\\tdomain` for the matching
entry. Used by per-book make targets to look up one textbook's
crawl parameters from the manifest without grep wizardry.
"""
target_id = sys.argv[2] if len(sys.argv) > 2 else ""
if not target_id:
print("usage: lookup <id>", file=sys.stderr)
return 2
for entry in iter_entries(stream):
if entry.get("id") != target_id:
continue
try:
_validate(entry)
except ValueError as exc:
print(f"# license-fail placeholder: {exc}", file=sys.stderr)
return 3
crawl_url = entry.get("crawl_url", "")
depth = entry.get("crawl_depth", 2)
max_pages = entry.get("crawl_max", 80)
print(
f"{entry.get('id')}\t{crawl_url}\t{depth}\t{max_pages}\t"
f"{entry.get('license')}\t{entry.get('domain')}"
)
return 0
print(f"# id {target_id!r} not found in manifest", file=sys.stderr)
return 4
_COMMANDS = {
"urls": cmd_urls,
"summary": cmd_summary,
"crawl-targets": cmd_crawl_targets,
"ids": cmd_ids,
"lookup": cmd_lookup,
}
def main(argv: list[str]) -> int: