From a5d666eba202962ef6c228344bdd32c936f7cbbb Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sat, 9 May 2026 20:38:32 -0400 Subject: [PATCH] textbook ingest: 4 base-knowledge additions for pillars I/II/III/IX MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds Cantor (Jourdain 1915 transl., PD), De Morgan First Notions of Logic (PG #67017, PD), Russell Introduction to Mathematical Philosophy (PG #41654, PD), and Judson Abstract Algebra Theory and Applications (GFDL-1.3) to the textbook manifest, plus per-book make targets + a `textbooks-base-knowledge` bulk target. Idempotent at DB layer. These four primaries cover the proprietary citations across pillars I/II/III/IX (Mendelson, Enderton, Jech, Barendregt, Dummit-Foote) that dominate the 74 unresolved claim-pack records. They land as honest "no match" under the existing resolver `_shard_matches_citation` heuristic (cited-author surname required) — full lift gates on fox citation-alias decisions per #000041. Total ingest delta: 43 docs / 295 chunks across four crawl shards. Bench journal under bench/results/textbook-base-knowledge-upgrade-2026-05-09.md records per-book license rationale, projected lift table, and the remaining #000038 acquisition gaps (Hilbert-Ackermann, Landau, Gödel, Kolmogorov, Church) for which no clean PD HTML edition was located in this session. --- Makefile | 45 +++++- bench/fixtures/textbooks/manifest-v1.jsonl | 4 + ...tbook-base-knowledge-upgrade-2026-05-09.md | 132 ++++++++++++++++++ 3 files changed, 179 insertions(+), 2 deletions(-) create mode 100644 bench/results/textbook-base-knowledge-upgrade-2026-05-09.md diff --git a/Makefile b/Makefile index 24dc43a..4cc3f28 100644 --- a/Makefile +++ b/Makefile @@ -32,7 +32,8 @@ SEARCH_Q ?= computer verify search stats test test-ci test-live docs docs-api docs-api-clean \ chain-check chain-check-shards \ falsify burn burn-kindergarten inspect bootstrap-crawler test-crawler crawl-ingest \ - recrawl-check bench-qa bootstrap-math clean clean-db clean-data help \ + recrawl-check bench-qa bench-qa-smoke bench-qa-progressive-and \ + bootstrap-math clean clean-db clean-data help \ textbooks-summary textbooks-urls fetch-textbooks textbooks-stats textbooks-verify \ crawl-textbooks crawl-textbooks-stats textbook textbook-list @@ -205,6 +206,25 @@ bench-qa-smoke: bootstrap ## quick 5-question smoke (all anchor classes; ~30s) --n 1 \ --concurrency $(BENCH_QA_CONCURRENCY) +# Progressive-AND / DF-filter fixture: 9 questions chosen to exercise +# the OR-fallback and progressive-AND drop paths. Use this for any +# retrieval-side A/B (alternative search backends, synonym/rerank +# changes, etc.) — the smoke fixture is structurally insensitive +# because it only contains queries where full-AND succeeds on every +# shard. Default --n 3 to clear the LLM noise floor. ~3-5 min wall +# at concurrency=4. Header in bench/qa_questions_progressive_and.txt +# documents which chain fires per query. +BENCH_PROGRESSIVE_N ?= 3 +bench-qa-progressive-and: bootstrap ## retrieval-side fixture exercising progressive-AND + DF filter [BENCH_PROGRESSIVE_N=3] + PYTHONUNBUFFERED=1 $(PY) bench/qa_sweep.py \ + --questions bench/qa_questions_progressive_and.txt \ + --shards-dir $(SHARDS_DIR) \ + --out-dir $(BENCH_QA_OUT) \ + --top-k $(QUERY_TOP_K) \ + --modes $(BENCH_QA_MODES) \ + --n $(BENCH_PROGRESSIVE_N) \ + --concurrency $(BENCH_QA_CONCURRENCY) + test-live: bootstrap ## live QA quality tests against Hermes (gated; -n auto parallel) ARBORIST_LIVE_TESTS=1 ARBORIST_LIVE_SHARDS_DIR=$(SHARDS_DIR) \ .venv/bin/pytest tests/test_qa_quality_live.py -v -n auto @@ -761,7 +781,8 @@ textbook-list: bootstrap ## list ingestable textbook ids (entries with urls or c # want a freshness probe before paying network cost. .PHONY: textbook-bogart textbook-keller-trotter textbook-levin \ textbook-aristotle-prior textbook-aristotle-posterior \ - textbook-newton textbook-morin + textbook-newton textbook-morin textbook-judson textbook-cantor \ + textbook-demorgan textbook-russell-imp textbooks-base-knowledge textbook-bogart: ## Bogart Combinatorics Through Guided Discovery (GFDL) $(MAKE) textbook ID=bogart-ctgd-2017 @@ -784,6 +805,26 @@ textbook-newton: ## Newton Principia (PD, Wikisource) textbook-morin: ## Morin Open Data Structures (CC-BY) $(MAKE) textbook ID=morin-open-data-structures +textbook-judson: ## Judson Abstract Algebra: Theory and Applications (GFDL) + $(MAKE) textbook ID=judson-abstract-algebra + +textbook-cantor: ## Cantor Contributions to Transfinite Numbers (PD, Jourdain transl. 1915, Wikisource) + $(MAKE) textbook ID=cantor-transfinite-numbers-jourdain + +textbook-demorgan: ## De Morgan First Notions of Logic 1839 (PD, PG #67017) + $(MAKE) textbook ID=demorgan-first-notions-logic + +textbook-russell-imp: ## Russell Introduction to Mathematical Philosophy 1919 (PD, PG #41654) + $(MAKE) textbook ID=russell-imp-1919 + +# Bulk target: ingest every base-knowledge textbook needed for the +# claim-pack warrant chains across pillars I/II/III/IX. Skips entries +# already ingested at the same content_root (idempotent at DB layer). +# Ordered cheap → expensive crawl-delay-wise; Judson last because its +# 23-chapter PreTeXt crawl is the heaviest of the new entries. +textbooks-base-knowledge: textbook-cantor textbook-demorgan textbook-russell-imp textbook-judson ## ingest the four 2026-05-09 base-knowledge additions (pillars I/II/III/IX) + @echo ">>> base-knowledge textbooks ingested: Cantor + De Morgan + Russell IMP + Judson AGT" + # TeX-source textbooks: Project Gutenberg eBooks that ship as # LaTeX source only (no clean HTML edition). The textbook_tex # source strips the LaTeX into plain prose via a focused PG-aware diff --git a/bench/fixtures/textbooks/manifest-v1.jsonl b/bench/fixtures/textbooks/manifest-v1.jsonl index 05c416b..9b18560 100644 --- a/bench/fixtures/textbooks/manifest-v1.jsonl +++ b/bench/fixtures/textbooks/manifest-v1.jsonl @@ -10,3 +10,7 @@ {"id":"aristotle-posterior-analytics","title":"Posterior Analytics","author":"Aristotle (G. R. G. Mure, transl.)","year":"~350 BCE; Mure translation 1928","license":"PD","license_url":"https://en.wikipedia.org/wiki/Public_domain","domain":"logic","pillar_targets":["I"],"urls":["https://en.wikisource.org/wiki/Posterior_Analytics"],"home_url":"https://en.wikisource.org/wiki/Posterior_Analytics","crawl_url":"https://en.wikisource.org/wiki/Posterior_Analytics","crawl_depth":1,"crawl_max":20,"notes":"Wikisource HTML. Aristotle's treatise on demonstrative knowledge, companion to Prior Analytics. The traditional Mure / Owen translations are PD by age."} {"id":"kolmogorov-foundations-probability-1933","title":"Foundations of the Theory of Probability","author":"Andrey Kolmogorov (Nathan Morrison transl., 1956)","year":"1933 (German); 1956 (English Morrison)","license":"PD","license_url":"https://en.wikipedia.org/wiki/Public_domain","domain":"probability","pillar_targets":["V"],"urls":[],"home_url":"https://en.wikipedia.org/wiki/Foundations_of_the_Theory_of_Probability","notes":"License analysis pending. German original (Grundbegriffe der Wahrscheinlichkeitsrechnung, 1933) PD-by-age in EU but US copyright restored via URAA through life+70 (Kolmogorov d.1987 → US PD 2058). Morrison 1956 English translation: Chelsea Publishing copyright through life+70 of translator. URLs intentionally empty until license decision lands; entry kept for citation traceability."} {"id":"lukasiewicz-aristotle-syllogistic-1951","title":"Aristotle's Syllogistic from the Standpoint of Modern Formal Logic","author":"Jan Łukasiewicz","year":"1951 (Oxford); 2nd ed 1957","license":"PROPRIETARY","license_url":"https://global.oup.com/academic/legal/copyright/","domain":"logic","pillar_targets":["I"],"urls":[],"home_url":"https://en.wikipedia.org/wiki/Jan_%C5%81ukasiewicz","notes":"Oxford University Press, copyright through life+70 (Łukasiewicz d.1956 → PD 2027 in EU, longer in US via URAA). NOT redistributable under arborist's allow-list. Entry kept as a license-fail placeholder; the 1921 Polish primary papers ('Two-valued logic', 'On the principle of contradiction in Aristotle') are themselves PD by age but lack accessible HTML editions."} +{"id":"judson-abstract-algebra","title":"Abstract Algebra: Theory and Applications","author":"Thomas W. Judson","year":"2025 (annual revisions; first ed 1997)","license":"GFDL-1.3","license_url":"https://www.gnu.org/licenses/fdl-1.3.html","domain":"abstract-algebra","pillar_targets":["I","IX"],"urls":[],"home_url":"https://judsonbooks.org/abstract-algebra-theory-and-applications/","crawl_url":"https://judsonbooks.org/aata-files/aata-html/aata.html","crawl_depth":2,"crawl_max":40,"notes":"PreTeXt HTML edition. Covers groups, rings, fields, Galois theory + a Boolean-lattice chapter — primary substrate for claim-pack records that cite Dummit-Foote (pillar IX, ~6 records) plus a logic-side touch via the Boolean-algebra chapter (pillar I). Annual revisions; we ingest the canonical /aata-files/aata-html/ static URL which is updated each release."} +{"id":"cantor-transfinite-numbers-jourdain","title":"Contributions to the Founding of the Theory of Transfinite Numbers","author":"Georg Cantor (Philip E. B. Jourdain transl.)","year":"1895/1897 (orig. German); 1915 (Jourdain English)","license":"PD","license_url":"https://en.wikipedia.org/wiki/Public_domain","domain":"set-theory","pillar_targets":["II"],"urls":["https://en.wikisource.org/wiki/Contributions_to_the_Founding_of_the_Theory_of_Transfinite_Numbers/Article_1"],"home_url":"https://en.wikisource.org/wiki/Contributions_to_the_Founding_of_the_Theory_of_Transfinite_Numbers","notes":"Wikisource. Only Article 1 (~15K words: cardinal number, equivalence, addition/multiplication/exponentiation, finite cardinals) is transcribed; Article 2 page does not exist. Foundational for claim-pack pillar II — Cantor IS the historical primary source modern Jech-style texts derive from. PD by age (Jourdain translation 1915, US PD via pre-1929 rule)."} +{"id":"demorgan-first-notions-logic","title":"First Notions of Logic, Preparatory to the Study of Geometry","author":"Augustus De Morgan","year":"1839","license":"PD","license_url":"https://en.wikipedia.org/wiki/Public_domain","domain":"logic","pillar_targets":["I"],"urls":["https://www.gutenberg.org/cache/epub/67017/pg67017-images.html"],"home_url":"https://www.gutenberg.org/ebooks/67017","notes":"PG eBook #67017, single-page HTML. ~30 pages: AEIO propositions, syllogism six fundamental forms × four figures, hypothetical / indirect / a fortiori arguments, probabilistic inference. Pre-Boole, pre-Frege, pre-Hilbert; one of the earliest English systematic logic texts. Pillar I supplement to Boole 1854 already ingested."} +{"id":"russell-imp-1919","title":"Introduction to Mathematical Philosophy","author":"Bertrand Russell","year":"1919","license":"PD","license_url":"https://en.wikipedia.org/wiki/Public_domain","domain":"logic-foundations","pillar_targets":["I","II","III"],"urls":["https://www.gutenberg.org/cache/epub/41654/pg41654-images.html"],"home_url":"https://www.gutenberg.org/ebooks/41654","notes":"PG eBook #41654, single-page HTML. ~200 pages covering natural numbers, definition of number, finite & infinite, classes & types, propositional functions, descriptions, relations, axiom of infinity, axiom of choice. Bridges pillars I (logic), II (transfinite cardinals), III (Peano arithmetic). Russell wrote IMP after Principia Mathematica as a non-symbolic exposition; PD by age (1919 + URAA = 95 years = 2015)."} diff --git a/bench/results/textbook-base-knowledge-upgrade-2026-05-09.md b/bench/results/textbook-base-knowledge-upgrade-2026-05-09.md new file mode 100644 index 0000000..6e79430 --- /dev/null +++ b/bench/results/textbook-base-knowledge-upgrade-2026-05-09.md @@ -0,0 +1,132 @@ +# Base-knowledge textbook upgrade — 2026-05-09 + +**Date:** 2026-05-09 (UTC) +**Author:** blackops + fox +**Scope:** four PD / open-licensed textbooks added to +`bench/fixtures/textbooks/manifest-v1.jsonl` and ingested into +`~/.arborist/crawl/textbook_*.db`. Closes the lookup gap surfaced +mid-session: pillar I/II/III/IX claim-pack records cite proprietary +modern texts (Mendelson, Enderton, Jech, Barendregt, Dummit-Foote) for +which PD primaries or open-licensed substitutes were not yet on disk. + +## Additions + +| ID | Author / Year | Format | License | Pillar targets | Docs/Chunks | +|---|---|---|---|---|---| +| `cantor-transfinite-numbers-jourdain` | Cantor 1895/97; Jourdain transl. 1915 | Wikisource HTML (1 page) | PD | II | 1 / 27 | +| `demorgan-first-notions-logic` | De Morgan 1839 | PG #67017 HTML | PD | I | 1 / 27 | +| `russell-imp-1919` | Russell 1919 | PG #41654 HTML | PD | I, II, III | 1 / 147 | +| `judson-abstract-algebra` | Judson 2025 (PreTeXt) | judsonbooks.org HTML | GFDL-1.3 | I, IX | 40 / 94 | + +Total delta: 43 new documents · 295 new chunks across four shards in +`~/.arborist/crawl/`. + +## Why these four + +Per the per-pillar citation triage (cluster status 18/92 = 20% before +this upgrade): + +- **Pillar II — 10 records cite Jech.** Cantor's *Contributions* IS the + primary source modern Jech-style treatments derive from. Jourdain + 1915 translation is PD by age (US pre-1929 rule). Article 1 covers + cardinal arithmetic, equivalence, addition / multiplication / + exponentiation of cardinals, finite cardinals — exactly the axioms + pillar II claim-pack records cite. +- **Pillar I — 13 records cite Mendelson + Enderton.** De Morgan 1839 + predates Boole and is foundational; Russell IMP 1919 is the + non-symbolic exposition Russell wrote after Principia Mathematica + to make logic + arithmetic foundations accessible. Together they + cover propositional logic, syllogism, descriptions, types, classes + — most of what Mendelson/Enderton present in modern form. +- **Pillar III — 13 records cite Landau + Gödel.** Russell IMP + bridges into pillar III (natural numbers, definition of number, + Peano arithmetic, finite & infinite induction). Landau and Gödel + themselves stay open as #000038 acquisition targets — neither has + a clean HTML PD edition yet found. +- **Pillar IX — 14 records cite Barendregt + Dummit-Foote.** Judson + AGT is the GFDL-licensed modern textbook covering exactly the + Dummit-Foote ground (groups, rings, fields, Galois theory) plus + a Boolean-lattices chapter that supports pillar I cross-references. + Lifts ~6 pillar IX records once a citation alias `Dummit-Foote → + Judson AGT` is registered. + +## Lift today + +**Resolver run with new shards present, no new citation aliases:** + +```bash +arborist warrant-resolve --use-aliases --write +# → records_resolved: 18 / 92 (unchanged from pre-ingest) +``` + +The resolver's `_shard_matches_citation` heuristic requires the cited +author's surname AND a significant title-word token to appear in a +shard's title haystack. None of the claim-pack records cite De Morgan +/ Russell / Cantor / Judson directly — they cite Mendelson / Enderton +/ Jech / Barendregt / Dummit-Foote. The new shards are honest "no +match" until citation-alias rows (#000041) bridge the gap. + +This is correct fail-closed behavior. The bench journal records the +ingest so a future shift sees the shards are ready. + +## Lift gated on fox citation-alias decisions + +Per #000041 audit discipline (each alias row needs `decision_by` + +`decision_rationale`), the following decisions would unlock: + +| Alias | Records lifted | Rationale | +|---|---|---| +| `Mendelson Introduction to Mathematical Logic → Russell IMP 1919` | up to 7 | Russell IMP's chs. on propositional logic, classes, definitions, types overlap Mendelson's propositional + first-order axioms | +| `Enderton A Mathematical Introduction to Logic → Russell IMP 1919` | up to 6 | same coverage profile as Mendelson alias | +| `Set Theory by Thomas Jech → Cantor Contributions to Transfinite Numbers` | up to 10 | Cantor IS the primary; Jech rederives with modern axiomatization | +| `Abstract Algebra by Dummit & Foote → Judson Abstract Algebra` | up to 6 | both are textbooks at same level; Judson covers groups/rings/fields/Galois, the Dummit-Foote core | +| `Stanley/Brualdi/Knuth combinatorics § → Bogart CTGD` (already ingested) | up to 14 | Bogart covers introductory combinatorics primitives | +| `Goldstein Classical Mechanics → Newton Principia` (already ingested) | up to 5 | Newton's *Principia* primary | + +**Maximum projected lift if fox approves all six aliases: 18 + ~48 = ~66 / 92 (72%).** + +Each alias decision is a separate audit event with `decision_by` and +`decision_rationale`. blackops cannot register them autonomously. + +## Reproducibility + +```bash +# Idempotent — re-running on already-ingested shards is a no-op +# at the document_root level (content-addressed inserts). +make textbook-cantor +make textbook-demorgan +make textbook-russell-imp +make textbook-judson +# OR all four at once: +make textbooks-base-knowledge +``` + +Crawl shards land in `~/.arborist/crawl/textbook_.db`; the warrant +resolver picks them up automatically (any `crawl/*.db` not prefixed +`crawl_russell_/qa./snapshots.` is in scope). + +## Out of scope (open #000038 targets) + +These remained unsolved in this session and stay queued: + +- **Hilbert-Ackermann 1928** — PG / Internet Archive German original + available; English Chelsea 1950 likely still copyright. Pillar I. +- **Landau 1930 *Grundlagen der Analysis*** — German original PD by + age; English translation copyright. Pillar III. +- **Gödel 1931** — German original; 95-year US copyright term expires + Jan 2026 (just barely PD now). Pillar III. +- **Kolmogorov 1933** — German original; URAA-restored, US PD 2058. + Pillar V. +- **Church 1932-1941 papers** — pre-1929 rule fails (1932 → US PD 2027). + Pillar IX. + +## References + +- `#000031` — surface-ingest cited textbooks for warrant promotion +- `#000038` — Phase 4 content acquisition (open · awaiting fox) +- `#000041` — citation-aliases mechanism (closed · empty table) +- `#000042` — term-aliases mechanism (closed · 4 geometry rows) +- `bench/results/hilbert-pillar-iv-fully-resolved-2026-05-09.md` — + prior session's pillar IV completion +- `bench/results/claim-pack-retrieval-lift-2026-05-09.md` — earlier + retrieval-only lift bench