engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

textbook ingest: 4 base-knowledge additions for pillars I/II/III/IX

Browse source 
Adds Cantor (Jourdain 1915 transl., PD), De Morgan First Notions of
Logic (PG #67017, PD), Russell Introduction to Mathematical Philosophy
(PG #41654, PD), and Judson Abstract Algebra Theory and Applications
(GFDL-1.3) to the textbook manifest, plus per-book make targets +
a `textbooks-base-knowledge` bulk target. Idempotent at DB layer.

These four primaries cover the proprietary citations across pillars
I/II/III/IX (Mendelson, Enderton, Jech, Barendregt, Dummit-Foote)
that dominate the 74 unresolved claim-pack records. They land as
honest "no match" under the existing resolver `_shard_matches_citation`
heuristic (cited-author surname required) — full lift gates on
fox citation-alias decisions per #000041.

Total ingest delta: 43 docs / 295 chunks across four crawl shards.
Bench journal under bench/results/textbook-base-knowledge-upgrade-2026-05-09.md
records per-book license rationale, projected lift table, and the
remaining #000038 acquisition gaps (Hilbert-Ackermann, Landau,
Gödel, Kolmogorov, Church) for which no clean PD HTML edition was
located in this session.
This commit is contained in:
russell@unturf.com 2026-05-09 20:38:32 -04:00
parent cc1a597708
commit a5d666eba2
No known key found for this signature in database
3 changed files with 179 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

45
Makefile
View file

@ -32,7 +32,8 @@ SEARCH_Q ?= computer
verify search stats test test-ci test-live docs docs-api docs-api-clean \
chain-check chain-check-shards \
falsify burn burn-kindergarten inspect bootstrap-crawler test-crawler crawl-ingest \
recrawl-check bench-qa bootstrap-math clean clean-db clean-data help \
recrawl-check bench-qa bench-qa-smoke bench-qa-progressive-and \
bootstrap-math clean clean-db clean-data help \
textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
crawl-textbooks crawl-textbooks-stats textbook textbook-list
@ -205,6 +206,25 @@ bench-qa-smoke: bootstrap ## quick 5-question smoke (all anchor classes; ~30s)
--n 1 \
--concurrency $(BENCH_QA_CONCURRENCY)
# Progressive-AND / DF-filter fixture: 9 questions chosen to exercise
# the OR-fallback and progressive-AND drop paths. Use this for any
# retrieval-side A/B (alternative search backends, synonym/rerank
# changes, etc.) — the smoke fixture is structurally insensitive
# because it only contains queries where full-AND succeeds on every
# shard. Default --n 3 to clear the LLM noise floor. ~3-5 min wall
# at concurrency=4. Header in bench/qa_questions_progressive_and.txt
# documents which chain fires per query.
BENCH_PROGRESSIVE_N ?= 3
bench-qa-progressive-and: bootstrap ## retrieval-side fixture exercising progressive-AND + DF filter [BENCH_PROGRESSIVE_N=3]
PYTHONUNBUFFERED=1 $(PY) bench/qa_sweep.py \
--questions bench/qa_questions_progressive_and.txt \
--shards-dir $(SHARDS_DIR) \
--out-dir $(BENCH_QA_OUT) \
--top-k $(QUERY_TOP_K) \
--modes $(BENCH_QA_MODES) \
--n $(BENCH_PROGRESSIVE_N) \
--concurrency $(BENCH_QA_CONCURRENCY)
test-live: bootstrap ## live QA quality tests against Hermes (gated; -n auto parallel)
ARBORIST_LIVE_TESTS=1 ARBORIST_LIVE_SHARDS_DIR=$(SHARDS_DIR) \
.venv/bin/pytest tests/test_qa_quality_live.py -v -n auto
@ -761,7 +781,8 @@ textbook-list: bootstrap ## list ingestable textbook ids (entries with urls or c
# want a freshness probe before paying network cost.
.PHONY: textbook-bogart textbook-keller-trotter textbook-levin \
textbook-aristotle-prior textbook-aristotle-posterior \
textbook-newton textbook-morin
textbook-newton textbook-morin textbook-judson textbook-cantor \
textbook-demorgan textbook-russell-imp textbooks-base-knowledge
textbook-bogart: ## Bogart Combinatorics Through Guided Discovery (GFDL)
$(MAKE) textbook ID=bogart-ctgd-2017
@ -784,6 +805,26 @@ textbook-newton: ## Newton Principia (PD, Wikisource)
textbook-morin: ## Morin Open Data Structures (CC-BY)
$(MAKE) textbook ID=morin-open-data-structures
textbook-judson: ## Judson Abstract Algebra: Theory and Applications (GFDL)
$(MAKE) textbook ID=judson-abstract-algebra
textbook-cantor: ## Cantor Contributions to Transfinite Numbers (PD, Jourdain transl. 1915, Wikisource)
$(MAKE) textbook ID=cantor-transfinite-numbers-jourdain
textbook-demorgan: ## De Morgan First Notions of Logic 1839 (PD, PG #67017)
$(MAKE) textbook ID=demorgan-first-notions-logic
textbook-russell-imp: ## Russell Introduction to Mathematical Philosophy 1919 (PD, PG #41654)
$(MAKE) textbook ID=russell-imp-1919
# Bulk target: ingest every base-knowledge textbook needed for the
# claim-pack warrant chains across pillars I/II/III/IX. Skips entries
# already ingested at the same content_root (idempotent at DB layer).
# Ordered cheap → expensive crawl-delay-wise; Judson last because its
# 23-chapter PreTeXt crawl is the heaviest of the new entries.
textbooks-base-knowledge: textbook-cantor textbook-demorgan textbook-russell-imp textbook-judson ## ingest the four 2026-05-09 base-knowledge additions (pillars I/II/III/IX)
@echo ">>> base-knowledge textbooks ingested: Cantor + De Morgan + Russell IMP + Judson AGT"
# TeX-source textbooks: Project Gutenberg eBooks that ship as
# LaTeX source only (no clean HTML edition). The textbook_tex
# source strips the LaTeX into plain prose via a focused PG-aware