arborist/docs/tickets/ticket-000038-phase-4-content-acquisition.md

Ticket #000038 — Phase 4 content acquisition for warrant coverage

Status: closed · most scope obviated 2026-05-10 by the citation-alias mechanism (#000041) + 18-substrate ingest sprint under #000031 Phase 2.5. 92/92 (100%) claim-pack records now resolve — every cited proprietary author was either substituted to a peer-level open-licensed work (Mendelson/Enderton → Russell IMP / De Morgan / Boole / SF-LF; Jech → Cantor / Russell PoM / PM Vol 1; Stanley/Brualdi/Knuth → Bogart + Levin + Keller-Trotter; Dummit-Foote → Judson AGT; Goldstein → Newton Principia; Barendregt → PLFA + SF-LF; Landau/Gödel → Peano + Dedekind + Russell IMP + SF-LF; Kolmogorov → Grinstead-Snell + Laplace; Böhm-Jacopini → SF-LF) or directly aliased. §3.2 option (b) "citation_aliases table" implemented as #000041 (74 rows live as of 2026-05-10; count grew 40 → 54 → 74 across the day). §6 Hilbert resolver-miss closed via #000040 cascade tuning + 4 #000042 term aliases (geometry domain). Genuinely-open residue documented in §8 below — primarily multilingual PD originals + Hilbert-Ackermann 1928 OCR + personal-copy path B; none currently blocking warrant coverage. Opened: 2026-05-09 Scope: Widen warrant-resolver coverage from 11/92 records (Hilbert pillar IV) to a substantively larger fraction of the claim-pack corpus by ingesting the cited textbooks the substrate itself flagged as priorities. License-gated per textbook; proprietary works require fox's explicit decision and a clear acquisition path (purchased PDF / library e-book / written permission / PD substitute). Audience: fox (license decisions); future blackops shifts running the next round of warrant promotion. Hard constraint: same fail-closed license discipline as #000031 §2.1. Allow-list: PD, CC0, CC-BY-, CC-BY-SA-, GFDL-*, AGPL-3.0, Apache-2.0, MIT. Anything CC-BY-NC, CC-BY-ND, or proprietary requires explicit fox approval AND a documented acquisition path before it enters the manifest with a non-empty URL set.

---

1. Problem statement

#000031 Phase 1+2+3 landed today with 11 of 92 claim-pack records bound by Merkle proof to a primary-source surface chunk — all pillar-IV Hilbert axioms citing The Foundations of Geometry by David Hilbert (PG #17384, PD via age, ingested through the new textbook_tex source).

The remaining 81 records sit at ANCHOR-WARRANTED because their cited textbooks are not in our shard cluster. The warrant resolver correctly returns no match, no derivations row, no upgrade. Adding those textbooks (where licensing permits) is the path to widen coverage.

The substrate produced its own priority list: walking every unresolved record's parsed citation and counting which authors

• titles appear most frequently tells us EXACTLY which acquisitions would move the dial most.

2. Substrate-derived priority list (2026-05-09 measurement)

Top unresolved authors by record-count impact:

Author	Records	Pillar(s)	License status
Stanley Enumerative Combinatorics	11	VII	proprietary (Cambridge)
Jech Set Theory	10	II	proprietary (Springer)
Brualdi Introductory Combinatorics	9	VII	proprietary (Pearson)
Knuth TAOCP	9	VII	proprietary (Addison-Wesley)
Mendelson Introduction to Mathematical Logic	7	I	proprietary (CRC)
Landau Foundations of Analysis	7	III	PD-by-age (1930; English translation 1951 still in copyright)
Hilbert Foundations of Geometry	7	IV	PD ingested, but resolver fails on records whose discriminating token (Incidence, Plane, Line) is too common in the text — see §6
Barendregt Lambda Calculus	7	IX	proprietary (Elsevier)
Enderton Mathematical Introduction to Logic	6	I	proprietary (Academic)
Gödel On Formally Undecidable Propositions	6	III	PD-by-age (1931; van Heijenoort 1967 translation in copyright)
Dummit + Foote Abstract Algebra	6	(algebra)	proprietary (Wiley)
Kolmogorov Foundations of Probability	5	V	German PD-by-age; Morrison 1956 English in copyright
Goldstein Classical Mechanics	5	VI	proprietary (Pearson)

Three buckets:

1. PD originals (Landau German 1930, Gödel German 1931, Kolmogorov German 1933, Hilbert German 1899). All foreign-language; ingestion would be in the original language. Possibly useful for some kinds of retrieval but our claim-pack records cite ENGLISH titles, so the resolver's title-matching would need a multilingual extension OR we'd need PD English translations (often the binding constraint).
2. Proprietary modern textbooks (Stanley, Jech, Brualdi, Knuth, Mendelson, Enderton, Barendregt, Dummit-Foote, Goldstein). License-gated per text; require fox's decision.
3. Already-ingested but resolver-misses (the 7 Hilbert records). A resolver fix, not a content acquisition. See §6.

3. Design choices

3.1 License-decision matrix

For each proprietary textbook, fox needs to pick:

Option	What it means	Cost	Provenance
A. Skip	Stay at ANCHOR-WARRANTED for those records permanently	Zero	Honest; substrate stays under-warranted
B. Personal-copy ingest, single machine	Ingest a purchased / library-checked-out PDF as a private surface; record never leaves fox's machine	Cost of book(s); acquisition time	Strong (we own/borrow the source)
C. PD substitute	Find a comparable PD work; ingest that; rewrite claim-pack source_reference to point at it	Authoring effort; loses fidelity to original cite	Weakened (substituted source is similar but not identical to what claim-pack records cite)
D. Negotiate redistribution rights	Get explicit permission from rights-holder	Months; only viable for a few specific works	Strongest

Recommended path per textbook:

• Knuth TAOCP — option B (Knuth's known-to-permit educational redistribution; contact the rights holder).
• Mendelson + Enderton — option C (PD substitute: Hilbert-Ackermann 1928 Principles of Mathematical Logic, PD by age, covers the same ground).
• Stanley + Brualdi — option B (purchase or library; these are in college library e-book catalogs almost universally).
• Jech — option B (library e-book) or option C (Halmos Naive Set Theory PD if 1960; needs verification).
• Barendregt — option A (defer; pillar IX is paper-level not book-level anyway).
• Goldstein — option C (Newton's Principia via Motte is already ingested; add a citation-rewrite step for pillar VI records).
• Dummit + Foote — option C if a PD substitute exists; otherwise A.

3.2 Citation-rewrite mechanism (for option C)

The claim-pack source_reference is currently a string field. Rewriting it requires either:

(a) Re-author the bundle — produce a new axiomsclaude-vii-v2.json (and v2 of the other Claude bundles) with substituted citations; re-ingest. Old records stay in shard 000.db unless explicitly burned. (b) Add an alias table — arborist citation_aliases table mapping "Original Cite by X" → "Substitute Cite by Y", read at warrant-resolve time. Cleaner — original citations stay intact; resolver knows which surface to look in.

→ (b) is the cleaner architectural fit. Proposal: add to the existing arborist concept family or as a new top-level table citation_aliases(original_ref, substitute_ref, substitute_authors, substitute_title, decision_at, decision_by).

3.3 What this ticket does NOT do

• Does not buy textbooks. That's a fox-decision-driven acquisition outside the codebase.
• Does not implement the Hilbert resolver fix (§6) — the warrant resolver's discriminating-token strategy is too strict for axioms whose discriminating words ("Incidence", "Plane") are common in the text. That's a #000031 Phase 5 follow-up.
• Does not add multilingual support for ingesting PD German originals. Same scope rationale.

4. Recommendation

Sequence:

1. Fox picks an acquisition path per textbook from the table above. Single message; low blackops effort.
2. For each fox-approved textbook:
   • Land in bench/fixtures/textbooks/manifest-v1.jsonl with license token + tex_url / urls populated, OR
   • Land in a private manifest-private-v1.jsonl (gitignored) for personal-copy ingest.
3. Run make textbook-<id> per the existing per-book convention; let arborist warrant-resolve --write re-run.
4. Each iteration: track derivations-rows count and the warrant tail's appearance rate on the bench questions.

Targets: each bucket of acquisitions should bring 5-10 more records into warrant coverage. End state after all "B" options ingest: ~50/92 records resolved (vs 11 today).

5. Out of scope

• Resolver fix for too-generic discriminating tokens (Hilbert §6 case) — separate Phase 5 ticket.
• Ingesting the 1899 German Hilbert original or 1933 German Kolmogorov — multilingual scope.
• Citation-aliases schema implementation — depends on §3.2 decision; separate ticket.
• The 6 records citing Gödel's On Formally Undecidable Propositions — original 1931 paper is 25 pages, paper-level not book-level; depends on a paper-ingest helper that's out of scope for any current ticket.

6. The Hilbert resolver-miss issue (footnote)

7 of the 18 Hilbert pillar-IV axiom records currently fail to resolve, even though the textbook IS in the shard. The records are:

Axiom of Line Incidence Axiom of Plane Incidence Axiom of Point-Line Incidence Axiom of Point-Plane Incidence Axiom of Non-Triviality Axiom of Side-Angle-Side (SAS) Euclidean Parallel Postulate

The discriminating-token strategy filters out generic terms ("axiom", "theorem", "principle") then AND-joins the rest. For "Axiom of Line Incidence" the discriminating tokens are "Line" + "Incidence". Both are ALL OVER Hilbert's geometry text — so FTS5 returns too many candidate chunks and BM25 ranking can't pin a winning one. The matcher backs off and returns no match.

This is a known limitation noted in the bench journal. Fix candidates:

• Use TF-IDF instead of BM25 over the whole textbook for ranking — favors discriminating CHUNKS, not just discriminating tokens.
• Boost chunks that contain the EXACT axiom name as a phrase ("Axiom I,1" or "Group I" labels in Hilbert).
• Use the CLAIM-PACK record's chunk content (the Δ formula or ∇verbose prose) as the FTS5 query, not just the title.

That's a Phase 5 follow-up. Folding into this ticket would overload its scope.

7. Status

Closed 2026-05-10 — most scope obviated by alias-substitution sprint. Walking the §2 priority table against current state:

Original cite	Records	Resolution path actually taken
Stanley Enumerative Combinatorics	11	aliased → Bogart CTGD + Levin + KT (3 substrates)
Jech Set Theory	10	aliased → Cantor 1915 + Russell PoM + PM Vol 1
Brualdi Introductory Combinatorics	9	aliased → Bogart + Levin + KT
Knuth TAOCP	9	aliased → Bogart + Levin + KT
Mendelson Introduction to Mathematical Logic	7	aliased → Russell IMP + De Morgan + Boole + Russell PoM + SF-LF (5 substrates)
Landau Foundations of Analysis	7	aliased → Russell IMP + Dedekind + Peano + SF-LF
Hilbert Foundations of Geometry	7 (deficient)	resolver fixed via #000040 cascade tuning + 4 #000042 term-aliases (incidence ↔ connection, euclidean parallel postulate ↔ axiom of parallels, non triviality ↔ space axiom, side angle side ↔ included angle)
Barendregt Lambda Calculus	7	aliased → PLFA (Wadler/Kokke/Siek) + SF-LF
Enderton Mathematical Introduction to Logic	6	aliased → Russell IMP + De Morgan + Russell PoM + SF-LF
Gödel 1931 (× 2 variants)	6	aliased → Peano + Russell IMP + Dedekind + SF-LF
Dummit + Foote Abstract Algebra	6	aliased → Judson AGT
Kolmogorov Foundations of Probability	5	aliased → Grinstead-Snell + Laplace
Goldstein Classical Mechanics (× 2 variants)	5	aliased → Newton Principia (Motte 1729)
Böhm-Jacopini 1966 (CACM)	1	aliased → SF-LF (Imp.v sequence/conditional/while)

§3.2 option (b) "citation_aliases table" landed as #000041 (closed). 74 rows live as of 2026-05-10 (count grew 40 → 54 → 74 across the day; decision_by="fox 2026-05-10"); rationale + caveats per row.

§6 Hilbert resolver-miss landed via:

• #000040 Phase 5 resolver (phrase + content-token cascade)
• #000042 term-aliases for the 1902 Townsend vs modern vocabulary gap
• Phase 2.5 cascade tuning in _build_record_query_cascade (parenthetical-phrase variant + OR-of-top-3 content-tokens fallback)

8. Residue (genuinely-open items NOT obviated)

These remain as design follow-ups but no longer block warrant coverage (the substitutes above carry the substrate):

1. Hilbert-Ackermann 1928 — OCR pipeline for Internet Archive scan. The §3.1 §B path "personal-copy ingest" was scoped here but the Mendelson/Enderton aliases to Russell IMP + De Morgan
   • Boole + SF-LF cover the same territory. Follow-up only if a future audit demands the historically-precise primary.
2. Multilingual PD originals (Landau German 1930, Gödel German 1931, Kolmogorov German 1933, Hilbert German 1899) — ingestion would need cross-language title matching since claim-pack records cite English titles. Out of scope unless warrant coverage of foreign-language editions becomes a compliance / academic-citation requirement.
3. Knuth TAOCP redistribution permission negotiation — §2 §A path. Aliased to Bogart + Levin + KT in the meantime; could pursue if a future bench shows TAOCP-specific axioms underserved by the substrates.
4. Personal-copy single-machine path B (Stanley, Brualdi, Jech specific texts) — fox-decision-driven outside codebase. Aliases hold the warrant coverage today; revisit only if a record's claim-pack content is unservable by the open substrates (none currently are).

None of (1)-(4) is required for downstream consumers. Closing #000038 as obviated; preserving §1-§6 above as the design log.

References

• #000041 — citation-aliases mechanism (40 → 54 → 74 rows live across 2026-05-10)
• #000042 — term-aliases mechanism (13 rows live)
• #000040 — Phase 5 resolver phrase + content-token cascade
• #000031 — full surface-ingest + warrant-resolver chain (Phase 1 + 2 + 2.5 + 3 + B-1 + B-2)
• bench/results/full-warrant-resolution-2026-05-10.md — 100% milestone bench journal
• docs/warrant-substrate-cookbook.md — per-pillar substrate map + ingest pattern catalog