textbooks: per-book make targets + idempotency docs + vital-books checklist
Three concrete improvements to the textbook ingest layer:
Per-book make targets
=====================
- `make textbook ID=<id>` — generic, looks up one entry from the
manifest and runs the right ingest path (BFS crawl if
`crawl_url` is set, shallow URL ingest otherwise).
- `make textbook-list` — list ingestable textbook ids.
- Convenience targets per active manifest entry:
textbook-bogart, textbook-keller-trotter, textbook-levin,
textbook-aristotle-prior, textbook-aristotle-posterior,
textbook-newton, textbook-morin
Each per-book target writes to a dedicated shard at
`$(CRAWL_SHARDS_DIR)/textbook_<id>.db` — one shard per textbook
id rather than one per host (which would conflate multiple
Wikisource books — Aristotle Prior, Posterior, Newton — into
one shard).
The manifest helper script gains `ids` and `lookup <id>`
subcommands so the Makefile can introspect manifest entries
without grep gymnastics.
Idempotency contract documented
================================
Every ingest path is content-addressed. Same content →
same `document_root` → no-op insert. Re-running `make textbook-*`
or `make crawl-textbooks` on the same network state is safe at
the database layer — no duplicates, no schema drift, no
audit-event proliferation. HTTP refetch happens every run; pair
with `make recrawl-check` for a HEAD-only freshness probe.
README rewritten to make this contract explicit, with shard-
layout table + crawl-delay caveats consolidated.
Vital-books checklist
=====================
The README now carries a g4-pillar coverage table:
Pillar I Logic ✓ Levin + Aristotle Prior + Posterior
Pillar II Set Theory ✓ Levin
Pillar III Arithmetic ✓ Levin
Pillar IV Geometry ✗ Hilbert PG #17384 (PDF/TeX only)
Pillar V Probability ✗ Kolmogorov license analysis pending
Pillar VI Phys. Newton ✓ Newton Principia Motte
Pillar VII Combin. ✓ Bogart + Keller-Trotter + Levin
Pillar IX λ-Calculus ✗ Church + Turing 1936 papers pending
5 of 7 pillars now have surface coverage. The remaining gaps
are explicit license (V) or format (IV PDF/TeX, IX paper-level)
issues, each documented in the corresponding manifest entry's
notes field.
Test suite stays at 1554 passed / 28 skipped — no source-code
changes to arborist itself; the textbook layer is pure tooling
on top of the existing pipeline.
This commit is contained in:
parent
2acbc0e691
commit
57dafbdc3d
3 changed files with 200 additions and 23 deletions
|
|
@ -127,9 +127,9 @@ def cmd_summary(stream: TextIO) -> int:
|
|||
|
||||
|
||||
def cmd_crawl_targets(stream: TextIO) -> int:
|
||||
"""Emit one tab-separated `crawl_url\\tdepth\\tmax` triple per entry
|
||||
"""Emit one tab-separated `crawl_url\\tdepth\\tmax\\tid` row per entry
|
||||
that declares a `crawl_url`. Used by `make crawl-textbooks` to drive
|
||||
the BFS crawler, one shard per host.
|
||||
the BFS crawler, one shard per textbook id.
|
||||
"""
|
||||
for entry in iter_entries(stream):
|
||||
_validate(entry)
|
||||
|
|
@ -142,7 +142,58 @@ def cmd_crawl_targets(stream: TextIO) -> int:
|
|||
return 0
|
||||
|
||||
|
||||
_COMMANDS = {"urls": cmd_urls, "summary": cmd_summary, "crawl-targets": cmd_crawl_targets}
|
||||
def cmd_ids(stream: TextIO) -> int:
|
||||
"""Emit one entry id per line for entries that have either `urls` or
|
||||
`crawl_url`. License-fail placeholders (no fetchable URLs) are
|
||||
excluded — they have nothing to ingest.
|
||||
"""
|
||||
for entry in iter_entries(stream):
|
||||
try:
|
||||
_validate(entry)
|
||||
except ValueError:
|
||||
continue
|
||||
if entry.get("urls") or entry.get("crawl_url"):
|
||||
print(entry.get("id", ""))
|
||||
return 0
|
||||
|
||||
|
||||
def cmd_lookup(stream: TextIO) -> int:
|
||||
"""Read entry id from sys.argv[2]; emit tab-separated
|
||||
`id\\tcrawl_url\\tdepth\\tmax\\tlicense\\tdomain` for the matching
|
||||
entry. Used by per-book make targets to look up one textbook's
|
||||
crawl parameters from the manifest without grep wizardry.
|
||||
"""
|
||||
target_id = sys.argv[2] if len(sys.argv) > 2 else ""
|
||||
if not target_id:
|
||||
print("usage: lookup <id>", file=sys.stderr)
|
||||
return 2
|
||||
for entry in iter_entries(stream):
|
||||
if entry.get("id") != target_id:
|
||||
continue
|
||||
try:
|
||||
_validate(entry)
|
||||
except ValueError as exc:
|
||||
print(f"# license-fail placeholder: {exc}", file=sys.stderr)
|
||||
return 3
|
||||
crawl_url = entry.get("crawl_url", "")
|
||||
depth = entry.get("crawl_depth", 2)
|
||||
max_pages = entry.get("crawl_max", 80)
|
||||
print(
|
||||
f"{entry.get('id')}\t{crawl_url}\t{depth}\t{max_pages}\t"
|
||||
f"{entry.get('license')}\t{entry.get('domain')}"
|
||||
)
|
||||
return 0
|
||||
print(f"# id {target_id!r} not found in manifest", file=sys.stderr)
|
||||
return 4
|
||||
|
||||
|
||||
_COMMANDS = {
|
||||
"urls": cmd_urls,
|
||||
"summary": cmd_summary,
|
||||
"crawl-targets": cmd_crawl_targets,
|
||||
"ids": cmd_ids,
|
||||
"lookup": cmd_lookup,
|
||||
}
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue