textbook ingest: 4 base-knowledge additions for pillars I/II/III/IX

Adds Cantor (Jourdain 1915 transl., PD), De Morgan First Notions of Logic (PG #67017, PD), Russell Introduction to Mathematical Philosophy (PG #41654, PD), and Judson Abstract Algebra Theory and Applications (GFDL-1.3) to the textbook manifest, plus per-book make targets + a `textbooks-base-knowledge` bulk target. Idempotent at DB layer. These four primaries cover the proprietary citations across pillars I/II/III/IX (Mendelson, Enderton, Jech, Barendregt, Dummit-Foote) that dominate the 74 unresolved claim-pack records. They land as honest "no match" under the existing resolver `_shard_matches_citation` heuristic (cited-author surname required) — full lift gates on fox citation-alias decisions per #000041. Total ingest delta: 43 docs / 295 chunks across four crawl shards. Bench journal under bench/results/textbook-base-knowledge-upgrade-2026-05-09.md records per-book license rationale, projected lift table, and the remaining #000038 acquisition gaps (Hilbert-Ackermann, Landau, Gödel, Kolmogorov, Church) for which no clean PD HTML edition was located in this session.
2026-05-09 20:38:32 -04:00 · 2026-05-09 20:38:32 -04:00 · a5d666eba2
commit a5d666eba2
parent cc1a597708
3 changed files with 179 additions and 2 deletions
--- a/45
+++ b/45
@ -32,7 +32,8 @@ SEARCH_Q     ?= computer
        verify search stats test test-ci test-live docs docs-api docs-api-clean \
        chain-check chain-check-shards \
        falsify burn burn-kindergarten inspect bootstrap-crawler test-crawler crawl-ingest \
-        recrawl-check bench-qa bootstrap-math clean clean-db clean-data help \
+        recrawl-check bench-qa bench-qa-smoke bench-qa-progressive-and \
+        bootstrap-math clean clean-db clean-data help \
        textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \
        crawl-textbooks crawl-textbooks-stats textbook textbook-list

@ -205,6 +206,25 @@ bench-qa-smoke: bootstrap ## quick 5-question smoke (all anchor classes; ~30s)
 	    --n 1 \
 	    --concurrency $(BENCH_QA_CONCURRENCY)

+# Progressive-AND / DF-filter fixture: 9 questions chosen to exercise
+# the OR-fallback and progressive-AND drop paths. Use this for any
+# retrieval-side A/B (alternative search backends, synonym/rerank
+# changes, etc.) — the smoke fixture is structurally insensitive
+# because it only contains queries where full-AND succeeds on every
+# shard. Default --n 3 to clear the LLM noise floor. ~3-5 min wall
+# at concurrency=4. Header in bench/qa_questions_progressive_and.txt
+# documents which chain fires per query.
+BENCH_PROGRESSIVE_N ?= 3
+bench-qa-progressive-and: bootstrap ## retrieval-side fixture exercising progressive-AND + DF filter [BENCH_PROGRESSIVE_N=3]
+	PYTHONUNBUFFERED=1 $(PY) bench/qa_sweep.py \
+	    --questions bench/qa_questions_progressive_and.txt \
+	    --shards-dir $(SHARDS_DIR) \
+	    --out-dir $(BENCH_QA_OUT) \
+	    --top-k $(QUERY_TOP_K) \
+	    --modes $(BENCH_QA_MODES) \
+	    --n $(BENCH_PROGRESSIVE_N) \
+	    --concurrency $(BENCH_QA_CONCURRENCY)
+
 test-live: bootstrap ## live QA quality tests against Hermes (gated; -n auto parallel)
 	ARBORIST_LIVE_TESTS=1 ARBORIST_LIVE_SHARDS_DIR=$(SHARDS_DIR) \
 	    .venv/bin/pytest tests/test_qa_quality_live.py -v -n auto
@ -761,7 +781,8 @@ textbook-list: bootstrap ## list ingestable textbook ids (entries with urls or c
 # want a freshness probe before paying network cost.
 .PHONY: textbook-bogart textbook-keller-trotter textbook-levin \
        textbook-aristotle-prior textbook-aristotle-posterior \
-        textbook-newton textbook-morin
+        textbook-newton textbook-morin textbook-judson textbook-cantor \
+        textbook-demorgan textbook-russell-imp textbooks-base-knowledge

 textbook-bogart: ## Bogart Combinatorics Through Guided Discovery (GFDL)
 	$(MAKE) textbook ID=bogart-ctgd-2017
@ -784,6 +805,26 @@ textbook-newton: ## Newton Principia (PD, Wikisource)
 textbook-morin: ## Morin Open Data Structures (CC-BY)
 	$(MAKE) textbook ID=morin-open-data-structures

+textbook-judson: ## Judson Abstract Algebra: Theory and Applications (GFDL)
+	$(MAKE) textbook ID=judson-abstract-algebra
+
+textbook-cantor: ## Cantor Contributions to Transfinite Numbers (PD, Jourdain transl. 1915, Wikisource)
+	$(MAKE) textbook ID=cantor-transfinite-numbers-jourdain
+
+textbook-demorgan: ## De Morgan First Notions of Logic 1839 (PD, PG #67017)
+	$(MAKE) textbook ID=demorgan-first-notions-logic
+
+textbook-russell-imp: ## Russell Introduction to Mathematical Philosophy 1919 (PD, PG #41654)
+	$(MAKE) textbook ID=russell-imp-1919
+
+# Bulk target: ingest every base-knowledge textbook needed for the
+# claim-pack warrant chains across pillars I/II/III/IX. Skips entries
+# already ingested at the same content_root (idempotent at DB layer).
+# Ordered cheap → expensive crawl-delay-wise; Judson last because its
+# 23-chapter PreTeXt crawl is the heaviest of the new entries.
+textbooks-base-knowledge: textbook-cantor textbook-demorgan textbook-russell-imp textbook-judson ## ingest the four 2026-05-09 base-knowledge additions (pillars I/II/III/IX)
+	@echo ">>> base-knowledge textbooks ingested: Cantor + De Morgan + Russell IMP + Judson AGT"
+
 # TeX-source textbooks: Project Gutenberg eBooks that ship as
 # LaTeX source only (no clean HTML edition). The textbook_tex
 # source strips the LaTeX into plain prose via a focused PG-aware
--- a/bench/fixtures/textbooks/manifest-v1.jsonl
+++ b/bench/fixtures/textbooks/manifest-v1.jsonl
@ -10,3 +10,7 @@
 {"id":"aristotle-posterior-analytics","title":"Posterior Analytics","author":"Aristotle (G. R. G. Mure, transl.)","year":"~350 BCE; Mure translation 1928","license":"PD","license_url":"https://en.wikipedia.org/wiki/Public_domain","domain":"logic","pillar_targets":["I"],"urls":["https://en.wikisource.org/wiki/Posterior_Analytics"],"home_url":"https://en.wikisource.org/wiki/Posterior_Analytics","crawl_url":"https://en.wikisource.org/wiki/Posterior_Analytics","crawl_depth":1,"crawl_max":20,"notes":"Wikisource HTML. Aristotle's treatise on demonstrative knowledge, companion to Prior Analytics. The traditional Mure / Owen translations are PD by age."}
 {"id":"kolmogorov-foundations-probability-1933","title":"Foundations of the Theory of Probability","author":"Andrey Kolmogorov (Nathan Morrison transl., 1956)","year":"1933 (German); 1956 (English Morrison)","license":"PD","license_url":"https://en.wikipedia.org/wiki/Public_domain","domain":"probability","pillar_targets":["V"],"urls":[],"home_url":"https://en.wikipedia.org/wiki/Foundations_of_the_Theory_of_Probability","notes":"License analysis pending. German original (Grundbegriffe der Wahrscheinlichkeitsrechnung, 1933) PD-by-age in EU but US copyright restored via URAA through life+70 (Kolmogorov d.1987 → US PD 2058). Morrison 1956 English translation: Chelsea Publishing copyright through life+70 of translator. URLs intentionally empty until license decision lands; entry kept for citation traceability."}
 {"id":"lukasiewicz-aristotle-syllogistic-1951","title":"Aristotle's Syllogistic from the Standpoint of Modern Formal Logic","author":"Jan Łukasiewicz","year":"1951 (Oxford); 2nd ed 1957","license":"PROPRIETARY","license_url":"https://global.oup.com/academic/legal/copyright/","domain":"logic","pillar_targets":["I"],"urls":[],"home_url":"https://en.wikipedia.org/wiki/Jan_%C5%81ukasiewicz","notes":"Oxford University Press, copyright through life+70 (Łukasiewicz d.1956 → PD 2027 in EU, longer in US via URAA). NOT redistributable under arborist's allow-list. Entry kept as a license-fail placeholder; the 1921 Polish primary papers ('Two-valued logic', 'On the principle of contradiction in Aristotle') are themselves PD by age but lack accessible HTML editions."}
+{"id":"judson-abstract-algebra","title":"Abstract Algebra: Theory and Applications","author":"Thomas W. Judson","year":"2025 (annual revisions; first ed 1997)","license":"GFDL-1.3","license_url":"https://www.gnu.org/licenses/fdl-1.3.html","domain":"abstract-algebra","pillar_targets":["I","IX"],"urls":[],"home_url":"https://judsonbooks.org/abstract-algebra-theory-and-applications/","crawl_url":"https://judsonbooks.org/aata-files/aata-html/aata.html","crawl_depth":2,"crawl_max":40,"notes":"PreTeXt HTML edition. Covers groups, rings, fields, Galois theory + a Boolean-lattice chapter — primary substrate for claim-pack records that cite Dummit-Foote (pillar IX, ~6 records) plus a logic-side touch via the Boolean-algebra chapter (pillar I). Annual revisions; we ingest the canonical /aata-files/aata-html/ static URL which is updated each release."}
+{"id":"cantor-transfinite-numbers-jourdain","title":"Contributions to the Founding of the Theory of Transfinite Numbers","author":"Georg Cantor (Philip E. B. Jourdain transl.)","year":"1895/1897 (orig. German); 1915 (Jourdain English)","license":"PD","license_url":"https://en.wikipedia.org/wiki/Public_domain","domain":"set-theory","pillar_targets":["II"],"urls":["https://en.wikisource.org/wiki/Contributions_to_the_Founding_of_the_Theory_of_Transfinite_Numbers/Article_1"],"home_url":"https://en.wikisource.org/wiki/Contributions_to_the_Founding_of_the_Theory_of_Transfinite_Numbers","notes":"Wikisource. Only Article 1 (~15K words: cardinal number, equivalence, addition/multiplication/exponentiation, finite cardinals) is transcribed; Article 2 page does not exist. Foundational for claim-pack pillar II — Cantor IS the historical primary source modern Jech-style texts derive from. PD by age (Jourdain translation 1915, US PD via pre-1929 rule)."}
+{"id":"demorgan-first-notions-logic","title":"First Notions of Logic, Preparatory to the Study of Geometry","author":"Augustus De Morgan","year":"1839","license":"PD","license_url":"https://en.wikipedia.org/wiki/Public_domain","domain":"logic","pillar_targets":["I"],"urls":["https://www.gutenberg.org/cache/epub/67017/pg67017-images.html"],"home_url":"https://www.gutenberg.org/ebooks/67017","notes":"PG eBook #67017, single-page HTML. ~30 pages: AEIO propositions, syllogism six fundamental forms × four figures, hypothetical / indirect / a fortiori arguments, probabilistic inference. Pre-Boole, pre-Frege, pre-Hilbert; one of the earliest English systematic logic texts. Pillar I supplement to Boole 1854 already ingested."}
+{"id":"russell-imp-1919","title":"Introduction to Mathematical Philosophy","author":"Bertrand Russell","year":"1919","license":"PD","license_url":"https://en.wikipedia.org/wiki/Public_domain","domain":"logic-foundations","pillar_targets":["I","II","III"],"urls":["https://www.gutenberg.org/cache/epub/41654/pg41654-images.html"],"home_url":"https://www.gutenberg.org/ebooks/41654","notes":"PG eBook #41654, single-page HTML. ~200 pages covering natural numbers, definition of number, finite & infinite, classes & types, propositional functions, descriptions, relations, axiom of infinity, axiom of choice. Bridges pillars I (logic), II (transfinite cardinals), III (Peano arithmetic). Russell wrote IMP after Principia Mathematica as a non-symbolic exposition; PD by age (1919 + URAA = 95 years = 2015)."}
--- a/bench/results/textbook-base-knowledge-upgrade-2026-05-09.md
+++ b/bench/results/textbook-base-knowledge-upgrade-2026-05-09.md
@ -0,0 +1,132 @@
+# Base-knowledge textbook upgrade — 2026-05-09
+
+**Date:** 2026-05-09 (UTC)
+**Author:** blackops + fox
+**Scope:** four PD / open-licensed textbooks added to
+`bench/fixtures/textbooks/manifest-v1.jsonl` and ingested into
+`~/.arborist/crawl/textbook_*.db`. Closes the lookup gap surfaced
+mid-session: pillar I/II/III/IX claim-pack records cite proprietary
+modern texts (Mendelson, Enderton, Jech, Barendregt, Dummit-Foote) for
+which PD primaries or open-licensed substitutes were not yet on disk.
+
+## Additions
+
+| ID | Author / Year | Format | License | Pillar targets | Docs/Chunks |
+|---|---|---|---|---|---|
+| `cantor-transfinite-numbers-jourdain` | Cantor 1895/97; Jourdain transl. 1915 | Wikisource HTML (1 page) | PD | II | 1 / 27 |
+| `demorgan-first-notions-logic` | De Morgan 1839 | PG #67017 HTML | PD | I | 1 / 27 |
+| `russell-imp-1919` | Russell 1919 | PG #41654 HTML | PD | I, II, III | 1 / 147 |
+| `judson-abstract-algebra` | Judson 2025 (PreTeXt) | judsonbooks.org HTML | GFDL-1.3 | I, IX | 40 / 94 |
+
+Total delta: 43 new documents · 295 new chunks across four shards in
+`~/.arborist/crawl/`.
+
+## Why these four
+
+Per the per-pillar citation triage (cluster status 18/92 = 20% before
+this upgrade):
+
+- **Pillar II — 10 records cite Jech.** Cantor's *Contributions* IS the
+  primary source modern Jech-style treatments derive from. Jourdain
+  1915 translation is PD by age (US pre-1929 rule). Article 1 covers
+  cardinal arithmetic, equivalence, addition / multiplication /
+  exponentiation of cardinals, finite cardinals — exactly the axioms
+  pillar II claim-pack records cite.
+- **Pillar I — 13 records cite Mendelson + Enderton.** De Morgan 1839
+  predates Boole and is foundational; Russell IMP 1919 is the
+  non-symbolic exposition Russell wrote after Principia Mathematica
+  to make logic + arithmetic foundations accessible. Together they
+  cover propositional logic, syllogism, descriptions, types, classes
+  — most of what Mendelson/Enderton present in modern form.
+- **Pillar III — 13 records cite Landau + Gödel.** Russell IMP
+  bridges into pillar III (natural numbers, definition of number,
+  Peano arithmetic, finite & infinite induction). Landau and Gödel
+  themselves stay open as #000038 acquisition targets — neither has
+  a clean HTML PD edition yet found.
+- **Pillar IX — 14 records cite Barendregt + Dummit-Foote.** Judson
+  AGT is the GFDL-licensed modern textbook covering exactly the
+  Dummit-Foote ground (groups, rings, fields, Galois theory) plus
+  a Boolean-lattices chapter that supports pillar I cross-references.
+  Lifts ~6 pillar IX records once a citation alias `Dummit-Foote →
+  Judson AGT` is registered.
+
+## Lift today
+
+**Resolver run with new shards present, no new citation aliases:**
+
+```bash
+arborist warrant-resolve --use-aliases --write
+# → records_resolved: 18 / 92 (unchanged from pre-ingest)
+```
+
+The resolver's `_shard_matches_citation` heuristic requires the cited
+author's surname AND a significant title-word token to appear in a
+shard's title haystack. None of the claim-pack records cite De Morgan
+/ Russell / Cantor / Judson directly — they cite Mendelson / Enderton
+/ Jech / Barendregt / Dummit-Foote. The new shards are honest "no
+match" until citation-alias rows (#000041) bridge the gap.
+
+This is correct fail-closed behavior. The bench journal records the
+ingest so a future shift sees the shards are ready.
+
+## Lift gated on fox citation-alias decisions
+
+Per #000041 audit discipline (each alias row needs `decision_by` +
+`decision_rationale`), the following decisions would unlock:
+
+| Alias | Records lifted | Rationale |
+|---|---|---|
+| `Mendelson Introduction to Mathematical Logic → Russell IMP 1919` | up to 7 | Russell IMP's chs. on propositional logic, classes, definitions, types overlap Mendelson's propositional + first-order axioms |
+| `Enderton A Mathematical Introduction to Logic → Russell IMP 1919` | up to 6 | same coverage profile as Mendelson alias |
+| `Set Theory by Thomas Jech → Cantor Contributions to Transfinite Numbers` | up to 10 | Cantor IS the primary; Jech rederives with modern axiomatization |
+| `Abstract Algebra by Dummit & Foote → Judson Abstract Algebra` | up to 6 | both are textbooks at same level; Judson covers groups/rings/fields/Galois, the Dummit-Foote core |
+| `Stanley/Brualdi/Knuth combinatorics § → Bogart CTGD` (already ingested) | up to 14 | Bogart covers introductory combinatorics primitives |
+| `Goldstein Classical Mechanics → Newton Principia` (already ingested) | up to 5 | Newton's *Principia* primary |
+
+**Maximum projected lift if fox approves all six aliases: 18 + ~48 = ~66 / 92 (72%).**
+
+Each alias decision is a separate audit event with `decision_by` and
+`decision_rationale`. blackops cannot register them autonomously.
+
+## Reproducibility
+
+```bash
+# Idempotent — re-running on already-ingested shards is a no-op
+# at the document_root level (content-addressed inserts).
+make textbook-cantor
+make textbook-demorgan
+make textbook-russell-imp
+make textbook-judson
+# OR all four at once:
+make textbooks-base-knowledge
+```
+
+Crawl shards land in `~/.arborist/crawl/textbook_<id>.db`; the warrant
+resolver picks them up automatically (any `crawl/*.db` not prefixed
+`crawl_russell_/qa./snapshots.` is in scope).
+
+## Out of scope (open #000038 targets)
+
+These remained unsolved in this session and stay queued:
+
+- **Hilbert-Ackermann 1928** — PG / Internet Archive German original
+  available; English Chelsea 1950 likely still copyright. Pillar I.
+- **Landau 1930 *Grundlagen der Analysis*** — German original PD by
+  age; English translation copyright. Pillar III.
+- **Gödel 1931** — German original; 95-year US copyright term expires
+  Jan 2026 (just barely PD now). Pillar III.
+- **Kolmogorov 1933** — German original; URAA-restored, US PD 2058.
+  Pillar V.
+- **Church 1932-1941 papers** — pre-1929 rule fails (1932 → US PD 2027).
+  Pillar IX.
+
+## References
+
+- `#000031` — surface-ingest cited textbooks for warrant promotion
+- `#000038` — Phase 4 content acquisition (open · awaiting fox)
+- `#000041` — citation-aliases mechanism (closed · empty table)
+- `#000042` — term-aliases mechanism (closed · 4 geometry rows)
+- `bench/results/hilbert-pillar-iv-fully-resolved-2026-05-09.md` —
+  prior session's pillar IV completion
+- `bench/results/claim-pack-retrieval-lift-2026-05-09.md` — earlier
+  retrieval-only lift bench