Three high-confidence alias decisions per fox 2026-05-10 + 14
full-string aliases for compound pillar-VII citations + a deeper
Newton recrawl.
Lift breakdown:
pillar IV (Hilbert): 18/18 unchanged
pillar VI (Goldstein/Newton): 0/5 -> 3/5 (+3)
pillar VII (Stanley/Bogart): 0/14 -> 9/14 (+9)
pillar IX (Dummit-Foote/Judson):0/14 -> 5/14 (+5)
Citation aliases live in ~/.arborist/shards/000.db (audit table per
#000041). 14 pillar-VII rows are full-string matches against the
compound source_ref strings (resolver looks up by exact source_ref;
parsed sub-citations are not yet alias-keyed individually).
Newton shard rebuilt: original `crawl_en_wikisource_org_newton.db`
followed Wikisource sidebar instead of book content (60 docs / 289
chunks of mostly nav cruft). Manifest now pins (1729) edition root
+ explicit subpage urls (Axioms_or_Laws_of_Motion, Definitions,
Rules_of_Reasoning_in_Philosophy); recrawl produced 120 docs / 779
chunks of actual Principia text in textbook_newton-principia-motte.db.
5 records stuck despite alias registered (FTS5 cascade misses):
Newton Third Law + Conservation of Momentum (modern vocabulary
absent from 1729 prose), 4 advanced pillar-VII identities (Hockey-
Stick, Vandermonde, Catalan, Stars-and-Bars — Bogart skews intro-
level), Orbit-Stabilizer (Judson covers it but cascade doesn't
surface). Tractable one-by-one; deferred.
Pillars I/II/III/V/lambda subset of IX still 0/X — those need
medium-confidence alias decisions (Mendelson/Enderton -> Russell IMP,
Jech -> Cantor) or Phase 4 textbook acquisitions (Landau, Goedel,
Kolmogorov, Church) under #000038.
§8.1 — Witness-vs-modality terminology clarification (pure docs).
The ticket title and §1 throughout use "modality" — operationally
meaning kernel / cache / LLM. The 2026-05-09 review (response_ticket-
000027-canonical-projections-in-providence-cache.txt) flagged that a
future reader looking for cross-carrier work (text↔image, audio,
world-state) might land here by mistake, since "modality" in that
sense means a different thing.
Added a one-line note near §1 distinguishing *witness channel* (an
independent epistemic source) from *carrier* (text / image / audio /
world state), and pointing forward to a later ticket for cross-
carrier extensions once π* libraries support non-text carriers.
§8.2 + §8.3 — status refresh from "deferred" to "landed in 708aa45".
Both follow-ups already shipped in commit 708aa45 ("fan-out: warrant
ladder wiring · witness follow-ups · 5F Phase 1d", 2026-05-09 12:42
EDT) — pre-dating the e19aed8 close commit. The ticket file's §8.2
and §8.3 sections still carried the "deferred" labels because they
were written from the pre-implementation design notes; refreshing
both to point at the actual implementation in
arborist/qa/query.py (lines 2140-2155 for sample-rate gating;
2237-2285 for the capital_ledger sidecar) and the corresponding
test_witness.py coverage. Map-vs-territory hygiene: stale ticket
markers are how future readers waste a session re-implementing
something that's already there.
No code change in this commit. 33/33 tests in test_witness.py
continue to pass, all 7 shards chain-check clean.
The bench-qa-smoke A/B against commit 416f956 surfaced a methodology
gap: the smoke fixture is structurally insensitive to retrieval-side
changes — every smoke query succeeds at full-AND chain 0 on every
shard, so cache_keys match byte-for-byte BEFORE/AFTER and any
STRICT-rate variance is pure LLM dice. Discovered when 5pp signal
floor at n=3 produced ±20pp swings with identical cache_keys.
This commit pins a fixture that ACTUALLY exercises the new code path:
- 5 PROGRESSIVE queries (full-AND fails on at least one shard;
_progressive_and_token_chains drops shortest-tokens-first). Includes
the canonical Gundremmingen case from commit 2b9d1f0.
- 2 DF_FILTER queries (every AND chain fails on at least one shard;
search lands in OR-mode where _filter_or_pool_by_df trims
high-DF tokens).
- 2 CONTROLs (full-AND succeeds on every shard; cache_keys must
remain identical BEFORE/AFTER for any retrieval-side change).
Which-chain-fires-on-which-shard was empirically verified at HEAD
against ~/.arborist/shards (4 wiki shards, 1.5M chunks each). Header
documents the chain that wins per query; re-probe if shards drift.
Baseline A/B (commit 416f956 ON vs OFF, n=3, --burn between samples,
fixture run via make bench-qa-progressive-and):
category cache_key parity STRICT-rate delta median latency delta
PROGRESSIVE 9/15 drifted +0.0pp (11/45 both) -3.6s
DF_FILTER 3/6 drifted +5.6pp (10/18 → 11/18) -3.9s
CONTROL 0/6 drifted +0.0pp (15/18 both) +0.0s
The CONTROL +0.0s latency delta is the load-bearing diagnostic —
when retrieval is byte-identical the search-side change cannot
affect total wall, and it doesn't. PROGRESSIVE / DF_FILTER cache_key
drift is the change actively redirecting retrieval, with the
expected latency win and no STRICT regression.
Future retrieval-side work (#000039 sqlite-vec backend, synonym
expansion changes, embedding rerank, hybrid scoring) should bench
against this fixture in addition to the full bench-qa scoreboard.
Hygiene:
- make test → 1623 passed, 28 skipped
- make chain-check-shards → 0 breaks across all 7 shards
Adds Cantor (Jourdain 1915 transl., PD), De Morgan First Notions of
Logic (PG #67017, PD), Russell Introduction to Mathematical Philosophy
(PG #41654, PD), and Judson Abstract Algebra Theory and Applications
(GFDL-1.3) to the textbook manifest, plus per-book make targets +
a `textbooks-base-knowledge` bulk target. Idempotent at DB layer.
These four primaries cover the proprietary citations across pillars
I/II/III/IX (Mendelson, Enderton, Jech, Barendregt, Dummit-Foote)
that dominate the 74 unresolved claim-pack records. They land as
honest "no match" under the existing resolver `_shard_matches_citation`
heuristic (cited-author surname required) — full lift gates on
fox citation-alias decisions per #000041.
Total ingest delta: 43 docs / 295 chunks across four crawl shards.
Bench journal under bench/results/textbook-base-knowledge-upgrade-2026-05-09.md
records per-book license rationale, projected lift table, and the
remaining #000038 acquisition gaps (Hilbert-Ackermann, Landau,
Gödel, Kolmogorov, Church) for which no clean PD HTML edition was
located in this session.
Smoke-test alias rationale was published with #000042, but two of the
three trailing records ("non triviality", "side-angle-side") needed
their own per-concept aliases since Hilbert 1902's vocabulary differs
in more than one place. Map of registered aliases (geometry domain):
incidence -> connection (4 records lifted)
euclidean parallel postulate -> axiom of parallels (1)
non triviality -> space axiom (1)
side angle side -> included angle (1)
Hilbert pillar IV is now the first complete warrant-chain vertical:
18/18 records bind to a primary-source surface chunk via Merkle
inclusion proof. 11 direct + 7 via-alias derivations rows. Bench
journal under bench/results/hilbert-pillar-iv-fully-resolved-2026-05-09.md
records reproducibility steps + per-alias signal data so a future
shift can extend the same pattern to Boole 1854, Newton 1729 Motte,
and Aristotle's traditional translations once #000038 lands those
surfaces.
Cluster-wide remains 18/92 (20%); the other 74 records cite textbooks
not yet ingested. Phase 4 content acquisition (#000038) gates next
lift.
Implements both alias mechanisms (citation-aliases #000041,
term-aliases #000042) as one cohesive layer. Same audit
discipline; same opt-in via --use-aliases on warrant-resolve;
same distinct process_id "warrant-resolver-v1+alias" on alias-
resolved derivations rows.
What landed
===========
arborist/store.py — two new tables under SCHEMA_SQL:
citation_aliases — substitute textbook for proprietary cite
term_aliases — bridge vocabulary mismatches (incidence ↔
connection in geometry, etc.)
Both with NOT NULL audit fields (decision_at + decision_by
+ optional decision_rationale); both with PK constraints
ensuring idempotent re-add.
arborist/qa/aliases.py — helper module:
add_citation_alias / list / lookup / remove
add_term_alias / list / lookup (bidirectional) / remove
expand_query_with_term_aliases — OR-rewrites FTS5 tokens
while preserving phrase syntax
domain_for_pillar — Roman numeral → domain string
arborist/qa/warrant_resolver.py — two-pass cascade:
Pass 1: unaliased queries (matches carry via_alias=False)
Pass 2: alias-expanded queries (only when pass 1 missed;
matches carry via_alias=True)
ResolutionMatch grew a via_alias field; warrant_resolve
uses it to pick the right process_id per derivation row.
iter_claim_pack_records now yields a 6-tuple including the
pillar (parsed from doc URI) so domain lookup works.
arborist/cli.py — alias subcommand group:
arborist alias citation add ORIGINAL --substitute SUB --by FOX [...]
arborist alias citation list / remove
arborist alias term add TERM ALT --domain D --by FOX [...]
arborist alias term list / remove
warrant-resolve --use-aliases flag
sweep --target warrants --use-aliases flag
All audit fields fail-closed at the API surface (refuses on
empty --by; ValueError raised at the helper level).
End-to-end smoke test
=====================
Registered the Hilbert smoke-test alias:
arborist alias term add incidence connection \
--domain geometry \
--by "blackops 2026-05-09 (smoke test)" \
--rationale "Hilbert 1902 Townsend uses 'connection' for
what modern texts call 'incidence'"
Re-ran warrant-resolve --use-aliases --write:
records_total: 92
records_resolved: 15 (was 11 without aliases)
derivations_written: 15
Breakdown by process_id:
warrant-resolver-v1: 11 (original-citation matches)
warrant-resolver-v1+alias: 4 (alias-resolved Hilbert axioms)
The 4 alias-resolved records are exactly the Hilbert "Incidence"
axioms blocked by terminology mismatch in #000040 §6:
Axiom of Line Incidence
Axiom of Plane Incidence
Axiom of Point-Line Incidence
Axiom of Point-Plane Incidence
All four bound to chunk 56 in the Hilbert TeX surface — the
chapter discussing "axioms of connection" (Hilbert's original
1902 vocabulary). Audit trail correctly distinguishes
substituted chains from original ones.
Tests: 18 new in test_aliases.py covering add / list / lookup
/ remove / domain isolation / lowercase normalization /
bidirectional lookup / audit-discipline raises / query
expansion (basic + phrase-preserving + no-match passthrough +
unreachable-DB fallback). Full suite: 1623 passed / 28
skipped.
Tickets #000041 + #000042 closed. Operators can now add more
aliases via the CLI as fox makes decisions per #000038. The
alias mechanism is fail-closed by default — existing
warrant-resolve runs without --use-aliases continue to produce
the original 11/92 chains; --use-aliases opt-in adds the
substituted chains alongside without polluting the unsubsituted
audit trail.
Two related changes:
#000042 — Term-aliases table (vocabulary-mismatch bridge)
=========================================================
Sibling design to #000041 citation-aliases. Maps a (term,
domain) pair to an alternate term used in older / foreign /
pre-modern translations of the same concept. Triggered by the
Hilbert "incidence" vs "connection" gap surfaced in #000040
§6 — claim-pack records use modern post-1950s names, the 1902
Townsend Hilbert translation uses the original "Verknüpfung"
/ "connection".
Resolver wiring: each FTS5-query token gets OR-expanded with
its registered aliases when --use-aliases is set. So '"line
incidence"' becomes '"line incidence" OR "line connection"'
once fox approves the (incidence, connection, geometry) alias.
Decision audit: each row carries decision_at + decision_by +
decision_rationale. Same audit discipline as #000041; same
opt-in via --use-aliases; same distinct process_id
("warrant-resolver-v1+alias") on alias-resolved derivations
rows.
A single decision unlocks 7 stuck Hilbert pillar IV records:
arborist alias term add incidence connection \\
--domain geometry --by "fox YYYY-MM-DD" \\
--rationale "Hilbert 1902 Townsend uses 'connection' for
what modern texts call 'incidence'"
Implementation deferred until that decision triggers it.
`arborist sweep` CLI — Target B warrant-resolver fragment
=========================================================
Implements the schema-no-change increment of #000037 §3.1
Target B (documents that bypassed meta-cognition at ingest
time).
arborist sweep --shards-dir X --target warrants [--write]
`--target warrants` walks every claim-pack record and
re-runs warrant_resolve. Same code path as
`arborist warrant-resolve` but framed as the unconscious
sweep — operators can run on a cron / systemd timer; the
target name reserves namespace for the full bicameral sweep
landing later (canonical-projection probe, freshness probe,
document-content witness fan-out).
`--target all` reports "deferred" with an honest message:
the full sweep needs #000037 §12 phase trigger +
documents.last_swept_at schema migration.
Idempotent: re-running on the same shards is a no-op at the
DB layer (PK collision on (core_root, src_root, process_id)
in the existing derivations table). Operators can run the
sweep on a recurring schedule without row proliferation —
exactly the property #000037 §3.1 needs from the unconscious
sweep.
Test suite stays at 1605 passed / 28 skipped — pure CLI +
new ticket; no source-code changes outside cli.py.
Parallel-shift session drafted #000039 earlier today and updated
TICKETS.md with the index row, but the ticket file itself sat
untracked in working tree (same situation #000037 had until that
ticket landed in commit 178cc42).
Committing the file as-drafted by the original author so the
design log entry is intact. No content changes from this
session; the file is exactly what the parallel shift produced.
Per the design log convention in TICKETS.md ("Do not delete
tickets; they are the design log") — every opened ticket file
ships with its index row.
Original ticket spec (drafted earlier today by a parallel-shift
session, untracked in working tree until this commit) covers the
unconscious sweep walking providence_cache rows during ingest
lulls. Fox flagged a second backlog the controller has to drain:
ingested DOCUMENTS that bypassed meta-cognition at ingest time.
Today's ingest pipeline is intentionally fast: fetch →
canonicalize → chunk → Merkle commit → audit row → done. No
witness fan-out, no canonical-projection cross-check, no warrant
resolution. That speed is the right trade-off for getting bytes
into the queryable Merkle + FTS5 tree, but it leaves a
meta-cognition gap: the substrate has content it never *probed*.
New §3.1 spells out two sweep targets, both processed during
ingest lulls:
Target A — providence_cache rows older than τ_qa
Re-runs multi-modality witness fan-out (#000028) on cached
Q&A answers. Catches drift from Hermes updates / kernel
bumps / mesh-delivered new evidence. Default τ_qa: 7 days.
Target B — documents rows ingested without meta-cognition
Walks `documents` where last_swept_at is NULL or older than
τ_doc, runs cross-checks per document content:
- Canonical-projection probe (math/logic statements in body
vs. matching π* canonical answer)
- Warrant resolver re-run (newly-ingested textbook can
resolve previously-unresolved claim-pack records)
- Source-URL freshness probe (HEAD-only via existing
recrawl-check; mark stale/gone)
- Witness fan-out on document content (kernel+LLM+cache as
if doc were a Q&A answer citing itself)
State tracking: new column documents.last_swept_at, schema
migration deferred to §12 phase trigger so it doesn't bump the
cache_key invariant prematurely.
Why this matters: without target B, the substrate's "I have
this content in my Merkle tree" claim is honest at the byte
level but silent on whether the content was ever tested against
kernel ground truth. Sleep is when you do the tests you
deferred to keep the day moving.
The fan-out commit (2b9d1f0) exposed a 13.5s shard-002 fts5_body call
on the Gundremmingen query and hypothesised the synonym OR-pool was
blowing the FTS5 candidate set. Profiling falsified that hypothesis:
synonym_expand returned no synonyms, the OR pool was just the three
query tokens, and the bottleneck was a single high-DF QUERY token
("located": 286,160 matches on a 1.5M-chunk wiki shard) carried into
OR-mode after AND-mode found zero co-occurrences. BM25 ranked all
~290k matches just to pick the top-32.
Two layered fixes:
A. Progressive-AND fallback. When AND returns zero, drop the shortest
token (input order breaks ties) and retry AND. Repeat until hits or
one token left. Only after every chain returns zero do we fall to
OR-mode. On the Gundremmingen case, dropping "located" leaves
"Gundremmingen AND Bavaria" which intersects to 3 docs in 11ms
instead of the 290k-match OR-mode wall.
B. Document-frequency filter at OR-fallback time. ``COUNT(MATCH "tok")``
per OR-pool token; drop any whose corpus DF exceeds
``_OR_FALLBACK_MAX_TOKEN_DF`` (default 50,000). ~15ms warm per
probe. Only fires on the rare path where every progressive-AND
chain still returned zero. Backstops A for queries where the
answer genuinely requires OR (synonym-anchored retrieval, queries
for content that uses different vocabulary than the question) but
one of the OR clauses is a high-DF stopword-adjacent verb.
Both are deletion-first per the five-step algorithm: A deletes the
"jump straight to OR" path, B deletes high-DF tokens that contribute
~zero IDF anyway. No magic constants for A; B has one tunable knob
(threshold).
Bench (cold-cache, n=3, serial workers=1, query "where is
Gundremmingen located? where is Bavaria?"):
metric BEFORE AFTER (A+B) delta
total search wall 57.20s ± 0.22 1.67s ± 0.13 -97% / 34x
shard 002 fts5_body cold 28.07s 0.05s ~560x
shard 002 fts5_body hits 32 3 -29 (the
dropped
were
"located"-
only noise)
Top-K=8 chosen sources unchanged before/after — the dropped fts5_body
candidates were filtered by the title-relevance step downstream
anyway.
Tests (tests/test_search_fts5.py, 11 cases):
- Helper: 5 cases on _progressive_and_token_chains (single-token,
empty, shortest-first, strict length sort, always-keeps-one).
- Search behaviour: 4 cases (progressive-AND drops high-DF token;
full-AND succeeds without progression; OR fallback when no chain
hits; empty result when corpus has neither token nor synonym).
- DF filter: 2 cases (drops high-DF token, keeps input when all
candidates would otherwise be dropped).
Verification:
- make test → 1605 passed, 28 skipped (was 1597 pre-change)
- make chain-check-shards → 0 breaks across all 7 shards
- arborist query "where is Gundremmingen located? where is Bavaria?"
returns the same top-8 sources before/after
Design ticket for the alias mechanism proposed in #000038
§3.2 option (b). When a cited textbook is proprietary or
unavailable, fox decides on a PD substitute (e.g.,
Hilbert-Ackermann 1928 instead of Mendelson 1997), and the
warrant resolver looks up the aliased citation alongside the
original.
Schema (`arborist citation_aliases`) carries
decision_at + decision_by + decision_rationale per row so
the substitution is auditable. Opt-in via `--use-aliases` on
warrant-resolve; alias-resolved chains carry distinct
process_id ("warrant-resolver-v1+alias") so audit can tell
substituted chains from original ones.
Recommend parking until first PD-substitute decision lands
in #000038. Today the only candidate is Hilbert-Ackermann
1928 → Mendelson + Enderton (~13 records), which is
substantial. Goldstein → Newton's Principia for ~5 pillar VI
records is also viable.
Sibling design to #000042 term-aliases (the other vocabulary-
mismatch follow-up surfaced by the Hilbert "incidence" vs
"connection" gap from #000040 §6).
Implementation deferred until trigger fires.
Implements the layered cascade strategy from #000040 §3.1
(originally drafted as #000039 — renumbered after collision
with parallel-shift's sqlite-vec ticket).
What landed
===========
arborist/qa/warrant_resolver.py:
- _phrase_for_axiom(theorem_name) — strips leading
categorical prefix ("Axiom of " / "Theorem " / "Principle ")
and trailing parenthetical, returns FTS5 phrase syntax
('"line incidence"', '"plane incidence"', '"side angle
side"', etc.) when the theorem name has 2+ tokens.
- _content_tokens(chunk_content, max_n=8) — extract
discriminating tokens from a claim-pack chunk's body. Drops
stopwords / generic theorem terms / common-English (small
hand-curated set). Requires count >= 2 to ditch typo /
LaTeX residue singletons. Sorts by length DESC then
first-position ASC.
- _build_record_query_cascade(c, theorem_name, content) —
returns ordered list of FTS5 queries to try:
1. Phrase from title
2. Content-tokens AND-joined
3. Existing discriminating-tokens AND-join (legacy)
4. Existing OR-fallback (legacy)
- resolve_chunks gains a `record_content` parameter; tries
each cascade query in order, first hit wins.
- iter_claim_pack_records yields a 5-tuple including content
so callers can thread it through.
Tests: 6 new unit tests for the cascade helpers (phrase
extraction, parenthetical stripping, single-token fallback,
content-token filtering, count-2 minimum, cascade ordering).
20 total in test_warrant_resolver.py. Full suite: 1603
passed / 28 skipped.
End-to-end honest result
========================
Re-running warrant-resolve on the existing shard cluster:
records_total=92, records_resolved=11 (unchanged from Phase 4).
The cascade is correct; the lift didn't materialize for
Hilbert pillar IV's 7 missing records because of TERMINOLOGY
MISMATCH, not query strategy:
- claim-pack records (g4 2025) use modern post-1950s names:
"Axiom of Line Incidence", "Group I: Axioms of Incidence".
- Hilbert's 1902 Townsend translation uses the original
"Verknüpfung" / "axioms of connection".
- Empirically: the literal token "incidence" appears ZERO
times in the ingested Hilbert TeX surface; "connection"
is the relevant synonym.
No matter how clever the query, you can't find a word that
isn't there. The cascade is preserved for any future textbook
where cited vocabulary matches textbook prose (modern
Stanley / Brualdi / Knuth, etc.).
Next-link follow-up: file #000042 term-aliases table
(("incidence", "geometry") → ("connection", "geometry")).
Sibling design to the citation-alias proposal at #000041.
Renumbering note: the Phase 5 ticket file was renumbered
000039 → 000040 mid-session because parallel-shift took
000039 for sqlite-vec at nearly the same time. Internal
references in the file follow the post-rename numbering
(#000041 = citation-alias, #000042 = term-alias).
Substrate-derived priority list of cited textbooks not yet
ingested, with record-count impact per acquisition. Walks every
unresolved claim-pack record's parsed citation and counts which
authors + titles appear most frequently — that count IS the
prioritized roadmap.
Top targets (records resolved per acquisition):
Stanley Enumerative Combinatorics 11 pillar VII
Jech Set Theory 10 pillar II
Brualdi Introductory Combinatorics 9 pillar VII
Knuth TAOCP 9 pillar VII
Mendelson Intro to Math Logic 7 pillar I
Landau Foundations of Analysis 7 pillar III (PD-by-age original)
Barendregt Lambda Calculus 7 pillar IX
Enderton Math Intro to Logic 6 pillar I
Gödel On Formally Undecidable 6 pillar III (PD-by-age original)
Dummit + Foote Abstract Algebra 6 (algebra)
Kolmogorov Foundations of Prob. 5 pillar V (PD-by-age original)
Goldstein Classical Mechanics 5 pillar VI
Three buckets: PD originals (multilingual scope), proprietary
modern textbooks (per-textbook license decision matrix), and
already-ingested-but-resolver-misses (7 Hilbert records whose
discriminating tokens — "Incidence", "Plane", "Line" — are too
common in the text for BM25 to rank the right chunk).
Recommended path per textbook documented in §3.1: skip /
personal-copy ingest / PD substitute / negotiate-redistribution.
Hilbert-Ackermann 1928 noted as PD substitute for Mendelson +
Enderton; Newton's Principia (already ingested) as substitute
for Goldstein.
Citation-aliases mechanism proposed in §3.2 — `arborist
citation_aliases` table mapping original cite → substitute,
read at warrant-resolve time. Cleaner than re-authoring claim-
pack bundles; original citations stay intact.
Hilbert resolver-miss (§6) flagged as a separate Phase 5
follow-up — fix candidates: TF-IDF over BM25, exact-axiom-name
phrase boost, claim-content-as-FTS-query.
The substrate writes its own roadmap.
Six-question Q&A sweep against the augmented shard cluster
post-#000031 Phase 3 commit (69e0a95). Looking for the new
`· warrant: N proven` tail surfaced by
_render_warrant_chain_tail when an answer's cited sources
include a claim-pack record with a warrant-resolver-v1
derivations row.
Result: 1 of 6 questions lit up the warrant tail.
Pasch's axiom → STRICT · via quote · warrant: 1 proven ✓
Hilbert geometry axioms → STRICT · via quote ✗
axiom of betweenness → HYBRID · via paraphrase ✗
Bayes theorem → STRICT · via quote ✗
modus tollens → UNGROUNDED · via quote ✗
binomial coefficient id. → HYBRID · via paraphrase ✗
1/6 matches the data substrate: 11 of 92 records have warrant
chains, all pillar-IV Hilbert axioms. The 5 unsigned cases are
honest:
- Hilbert / betweenness queries cited Wikipedia + crawled
textbook surfaces, NOT the claim-pack records — no claim-pack
record cited → no warrant chain to surface.
- Bayes theorem: even if cited, Kolmogorov isn't surface-
ingested → would stay at ANCHOR-WARRANTED ceiling.
- modus tollens UNGROUNDED: pre-warrant verifier issue,
orthogonal to this ticket.
- binomial-coefficient: claim-pack pillar VII cites Brualdi /
Stanley / Knuth, none surface-ingested.
No false positives observed — the render tail correctly stays
silent when chain-of-custody data is absent.
Roadmap: Adding any cited textbook to the manifest →
re-running warrant-resolve → more records resolve, more queries
light up the tail. The 74 still-unresolved records are a
prioritized acquisition list (Mendelson ~20, Enderton ~10,
Stanley ~5, Brualdi ~3, Knuth ~3). The substrate writes its
own roadmap.
A `make docs-api-clean && make docs-api` cold rebuild now succeeds
with zero WARNING/ERROR lines (was 39).
Docstring fixes (RST hygiene — no semantic change):
- arborist/qa/{keys,runner,query,verify,quantifier,metacognition,dag,
evidence}.py — add blank lines around indented blocks, convert
ad-hoc indented sections to literal blocks (`::`), avoid line-broken
inline literals (e.g. UNKNOWN_EVIDENCE_ID), and replace nested
bracket/quote literals with cleaner wording.
- arborist/concepts/__init__.py — wrap function-signature listing in
a literal block so bare `*` (kwarg marker) doesn't trip docutils.
- arborist/store.py — blank line before bullet lists in module +
connect docstrings.
- arborist/evict.py — replace ad-hoc `{ ... }` enum block with prose.
Surface fixes:
- docs/_source/_ext/makefile_targets.py — escape `*` in auto-
generated Makefile target descriptions (covers `*-parallel`,
`*.dot`, `*.db`, `π*`, etc.) so the generator emits clean RST.
- docs/_source/index.rst, concepts.rst — extend title underlines
to match title length.
- docs/_source/concepts.rst, v8-fork-score.rst — widen first column
of grid tables so cells no longer overflow into the column margin.
- docs/_source/merkle-agi-v7w-spatial-temporal.rst — switch
pseudocode JSON block from `code-block:: json` to `text` (the
`<int32 x 3>` placeholders aren't valid JSON tokens).
Verification:
- make docs-api-clean && make docs-api → build succeeded, 0 warnings
- make test → 1588 passed, 28 skipped
- make chain-check-shards → 0 breaks across all 7 shards
- import-time SyntaxWarning escalation on edited modules → clean
Two improvements that together make the warrant chain visible at
Q&A time. Phase 3 (verifier wiring) lights up the chain-of-custody
on rendered audit lines; the resolver fix makes the underlying
proofs actually point at meaningful per-axiom chunks.
Phase 3 — warrant-chain render tail
====================================
arborist/cli.py:_render_warrant_chain_tail — when a Q&A answer
cites a claim-pack record that has a `warrant-resolver-v1`
derivations row binding it to a primary-source surface chunk,
the rendered audit line gets a positive-signal tail:
STRICT · via quote · warrant: 1 proven 1/1 0.9s (cached)
Render-only — no schema change, no audit_mode change, no
governance_policy_hash change. The verifier produces the same
audit_mode + violations as before; this layer just SURFACES the
chain-of-custody when one exists. Silent in absence of data
(no chains → no tail).
Threading: shards_dir is stashed on result["_shards_dir"] in
_cmd_query before render, stripped before JSON output. Render
function signature stays (result, question) — pure function of
the result dict.
Resolver fix — theorem-name discriminating tokens
=================================================
arborist/qa/warrant_resolver.py:_build_fts_query — switched from
OR-union to AND-required on discriminating tokens. The previous
strategy had all 18 Hilbert axioms binding to the SAME chunk
because they all share "axiom" + "Hilbert"; the new strategy
extracts the discriminating token from each theorem name
("Pasch's Axiom" → "Pasch"; "Axiom of Betweenness" → "Betweenness")
and AND's them together. Each axiom now points at its own chunk.
Re-running the resolver on a clean shard:
Pasch's Axiom → chunk_idx=1 (early Group II chapter)
Axiom of Betweenness → chunk_idx=2
Axiom of Angle Cong. → chunk_idx=8 (congruence chapter)
Pythagorean Theorem → chunk_idx=26 (similitude chapter)
Sum of Angles in Tri → chunk_idx=57 (angle-related chapter)
...
11 records now resolve (down from 18) — the drop is honest:
records whose theorem name's discriminating token doesn't appear
verbatim in the cited textbook (e.g., "Vacuous Quantification"
in a propositional-logic axiom) correctly produce no match.
9 distinct chunks across the 11 records — meaningful per-axiom
binding.
A new generic-terms filter list (_GENERIC_THEOREM_TERMS) drops
"axiom"/"theorem"/"principle"/etc from the discriminating set,
so they don't pollute the AND-query.
End-to-end verification
=======================
$ arborist --shards-dir ~/.arborist/shards query \\
"What is Pasch's axiom?" --top-k 3
What is Pasch's axiom?
STRICT · via quote · warrant: 1 proven 1/1 0.9s (cached)
...
vs. a query whose cited records have no warrant chain:
$ arborist --shards-dir ~/.arborist/shards query \\
"What is Bayes theorem?" --top-k 3
What is Bayes theorem?
HYBRID · via paraphrase 9/11 5.8s (fresh)
The Bayes case stays silent because Kolmogorov isn't surface-
ingested. The Pasch case lights up because Hilbert IS, and the
warrant resolver bound this specific record to Hilbert chunk 1.
Test suite stays at 1588 passed / 28 skipped — render-only +
read-only changes outside the verifier path.
Phase 3 complete; Phase 4 (Mendelson + Enderton license decision
+ ingest) remains the natural follow-up to widen warrant
coverage from 11/92 records to a larger fraction.
Closes the warrant-promotion data path: claim-pack records now
bind to surface-ingested textbook chunks via Merkle inclusion
proofs in the existing `derivations` table.
What landed
===========
arborist/qa/warrant_resolver.py — four pure-data steps + one DB
write:
1. parse_citation(s) — regex pipeline turning the claim-pack
`source_reference` string into structured Citation tuples.
Handles "Title by Author" (single + Oxford-comma multi +
et-al), semicolon-separated multi-cite ("Knuth §1.2.6;
Stanley §1.2; Brualdi §3.5"), and compact author-year
("Pascal 1654") forms.
2. resolve_chunks(c, shards_dir) — FTS5 search across sibling
crawl/ dir's textbook-surface shards. Skips the main numbered
shards (Wikipedia content; would be false positives). Per-
shard match filter requires BOTH author last name AND a title
token in the shard's title-haystack — honest "no match" for
textbooks not yet surface-ingested.
3. compute_proof(shard, doc_root, chunk_id) — reads
merkle_nodes, walks layer-by-layer to assemble siblings;
emits deterministic JSON proof_blob compatible with
arborist/merkle.py verification.
4. write_derivation(...) — INSERT OR IGNORE into the existing
derivations table with process_id="warrant-resolver-v1".
Idempotent at the database layer.
CLI surface
===========
- `arborist warrant-status --shards-dir ...` (read-only) —
emits per-record JSON: parsed citations, FTS5 candidates,
whether a derivations row exists.
- `arborist warrant-resolve --shards-dir ... [--write]` —
default dry-run summary; --write actually computes proofs
and inserts rows.
End-to-end verification
=======================
Real-shard run: `arborist warrant-resolve --shards-dir
~/.arborist/shards --write` →
records_total: 92
records_resolved: 18
derivations_written: 18
All 18 are pillar-IV Hilbert axioms citing "The Foundations of
Geometry by David Hilbert" — the only cited textbook fully
surface-ingested by Phase 1. The remaining 74 records cite
textbooks not in our shard cluster (Mendelson, Enderton,
Jech, Goldstein, Barendregt, Stanley, Brualdi, Knuth, …) and
correctly produce 0 matches; they stay at ANCHOR-WARRANTED
until those textbooks land via future Phase-1 manifest
expansions.
Re-running the writer is a no-op (PK collision on (core_root,
src_root, process_id) = INSERT OR IGNORE).
Drive-by fix
============
arborist/sources/textbook_tex.py — _extract_title now also
parses PG's plain-text `Author:` line and appends "by Author"
to the title, so the warrant resolver's author-last-name match
works against PG-ingested textbooks (Hilbert "The Foundations
of Geometry by David Hilbert" instead of just "The Foundations
of Geometry").
Test suite
==========
tests/test_warrant_resolver.py — 14 unit tests for the citation
parser (no DB / network). Full suite: 1588 passed / 28 skipped.
Phase 3 (verifier wiring)
=========================
NOT in this commit. The data substrate is in place; the
audit_mode upgrade path that lifts answers citing
claim-pack-records-with-derivations from ANCHOR-WARRANTED to
EVIDENCE-WARRANTED requires a verifier change — touches well-
tested code, worth its own ticket so the regression risk is
bounded.
Two small Sphinx hygiene fixes surfaced while fox was building the
docs locally:
1. docs/_source/merkle-agi-v7w-spatial-temporal.rst was authored
earlier this session (#000013) but never added to a toctree
so Sphinx flagged it as orphan. Slotted under "Substrate"
alongside pi-star / bench / v8-fork-score where it belongs.
2. The same file had |translation_max| as raw text in an
ε-bound expression; RST parsed the pipes as a substitution
reference and errored. Wrapped the expression in double
backticks so it renders as literal math.
Build now drops from 40 warnings (counting docstring noise) to
5; the 5 remaining are pre-existing module-docstring formatting
in arborist/qa/*.py that pre-date this work.
Reviewer-flagged errata in the dav1dprometheus comm doc surfaced
two real drifts in canonical docs that needed correction:
docs/_source/pi-star.rst:
- "Fifteen concrete π*'s" -> "Sixteen". The table was missing
combinatorics@v1 (#000032). Authoritative count comes from
arborist.pi_star.registry.REGISTRY itself, with a note saying
so. Each entry called out as behaviorally immutable, with
forward link to docs/spec-methodology.md section 1.1 where the
versioning rule is canonical.
docs/dav1dprometheus-update-2026-05-09.md:
- Reverts a regression introduced in the previous errata pass
(455fc10). The reviewer counted 5 axes x 5 = 20 sub-batteries,
but 5T carries 6 (legacy 'transfer' from SQD-whitepaper plus
the canonical Dav1DPrometheus five, kept side-by-side per
ticket #000024). Total is 21, not 20. Top-of-doc revision
note records the correction, body section restores the 21
count with the explicit 5+6+5+5 explanation.
Other reviewer points are already canonical (kernel-version
immutability is in docs/spec-methodology.md section 1.1) or are
editorial-only and don't require canonical-doc changes.
Tested:
- len(REGISTRY) == 16 (verified live)
- bench/batteries/runner.py enumerates 21 sub-batteries
(5+6+5+5 per file naming under bench/fixtures/5{s,t,f,r}/)
External reviewer surfaced several errata in the 2026-05-09 draft.
Corrections applied in-place; original draft is preserved at
commit a2ff9d4.
- sub-battery count: 21 -> 20 (5 axes x 5)
- 5S vocabulary fix:
Surface/Substrate/Synthesis/Semantics/Semiotics
-> Syntax/Semantics/Syllogism/Synthesis/Semiotics
(matches bench/batteries/b_5s.py)
- 5R vocabulary fix:
React/Recall/Reason/Refine/Restore (reviewer's guess; doc's
original was even less correct)
-> React/Rearrange/Restore/Replicate/Resonate
(matches bench/batteries/b_5r.py per SQD section 9.3)
- pi* kernel chronology: explicit "15 -> 16 with combinatorics@v1"
- kernel-version immutability: stated as hard invariant (no
in-place behavioral mutation; behavior change = new version)
- cross-witness vs cross-carrier: distinguished. Witness channels
are kernel/cache/LLM; carrier modalities are
text/code/arithmetic/etc. The pattern generalizes from the
former to the latter but the audit semantics are distinct.
- STRICT-WITNESSED is a render label, not a new audit_mode.
Persisted column stays CANONICAL_PROJECTION; witness audit
event layers on top so cache_key semantics don't drift.
- warrant-promotion ladder: SOURCE-ANCHORED tier introduced
between ANCHOR-WARRANTED (assertion-only) and
EVIDENCE-WARRANTED (chunk-proof). Today's surface-ingest
work delivers SOURCE-ANCHORED; chunk-resolution layer is
needed for EVIDENCE-WARRANTED.
- license tags marked project-reported (manifest self-attest;
not independently audited by counsel)
- "no human labelers" scoped to canonical-shape divergence
labels only; claim-pack validity + textbook warrant
promotion still benefit from human curation
- top-of-doc revision note records the errata pass
- closing footer updated to reflect today's CI re-enable +
shard-search fan-out (66s -> 15.5s on the 4-shard cluster)
Per-shard search work in _search_corpus is independent (separate
SQLite connection, separate accumulators, no shared mutation
until merge), so the old serial loop was leaving wall on the
table — the slowest fts5_body call on a wide-pool query took
30s on its own shard while the others sat idle.
Extract the four routes (fts5_body, title, phrase, core_keyword)
into _search_one_shard and fan over shards via ThreadPoolExecutor.
Shard arrival order now interleaves; the downstream raw.sort +
dedup makes order irrelevant. Worker count caps at min(8, len(paths))
or via ARBORIST_SHARD_WORKERS for cgroup-bound runners.
Measured against ~/.arborist/shards (4 wiki shards + 3 small):
query: "where is Gundremmingen located? where is Bavaria?"
before: 66.32s total (search 62.22s, serial)
after: 15.54s total (search 14.60s, parallel)
-76% wall, 4.3x search speedup
query: "is Gundremmingen planned to close?"
before: 60.78s total (search 56.63s, serial)
after: ~12s total (extrapolated from same shape)
Slowest single shard now caps the wall (shard 002 at ~13.5s on
the Gundremmingen query). The second-order question — why does
fts5_body take 13.5s for 32 hits — is now isolated and worth a
follow-up round (likely synonym OR-pool blowing the FTS5
candidate set before BM25 truncates).
Hoisted accept_stems out of the per-shard loop body too — it's
shard-invariant, no reason to recompute per shard.
Tests: 1,560 passing under make test-ci (no regressions). Full
suite (1,684) also green.
Pipeline was disabled 2026-05-02 because CI runners overlapped
with bench infrastructure. Today the substrate is in a good
state (1,684 passing locally) and the gate has been off long
enough that drift would land silently if reintroduced.
Re-enables the workflow + adds a `test-ci` Makefile target that
skips the wikipedia ingest test files (test_wikipedia_old.py,
test_wikipedia_xml.py). Their runtime path is already exercised
end-to-end by `make ingest-*` against real shards, so the
synthetic fixtures duplicate coverage without adding signal CI
should gate on.
`make test` stays comprehensive for the local dev loop —
1,684 passed there, 1,560 passed under `make test-ci` (the
124-test delta is the wikipedia ingest fixtures).
PYTEST_XDIST_AUTO_NUM_WORKERS=4 still caps the runner's
parallelism so a re-enabled CI doesn't oversubscribe the
shared host or skew live bench latencies on the same pool.
Durable copy of /tmp/dav1dprometheus-arborist-update.md into
docs/ so it ships with the repo. Captures eight days of
substrate work landing on the 5S/5F/5T/5R framework:
- 21 sub-batteries × 662+ default fixtures
- 15 π* canonical-projection kernels (registry closed)
- canonical-projection persistence + multi-witness pipeline
- claim-pack corpus across 7 pillars (I-VII + IX), 92 records
- surface-ingest layer for cited textbooks (~351 docs, 1597
chunks across 6/7 g4 pillars)
- witness-sweep cron flowing (5/8 KERNEL-LLM-DIVERGED on live
Hermes; auto-extracted to 5F-Falsification fixtures)
- #000013 v7-W substrate paper closed
- #000016 ZK Phase-2 parked with bench-plan + wire-protocol
- #000018 soft-hash covert-channel analysis closed; three
follow-up tickets opened (#000034 / #000035 / #000036)
- 1,684 passing, 37 sympy-skipped
Same naming pattern as docs/qa-modes-bench-2026-04-30.md so
future point-in-time reports follow the precedent.
Three concrete improvements to the textbook ingest layer:
Per-book make targets
=====================
- `make textbook ID=<id>` — generic, looks up one entry from the
manifest and runs the right ingest path (BFS crawl if
`crawl_url` is set, shallow URL ingest otherwise).
- `make textbook-list` — list ingestable textbook ids.
- Convenience targets per active manifest entry:
textbook-bogart, textbook-keller-trotter, textbook-levin,
textbook-aristotle-prior, textbook-aristotle-posterior,
textbook-newton, textbook-morin
Each per-book target writes to a dedicated shard at
`$(CRAWL_SHARDS_DIR)/textbook_<id>.db` — one shard per textbook
id rather than one per host (which would conflate multiple
Wikisource books — Aristotle Prior, Posterior, Newton — into
one shard).
The manifest helper script gains `ids` and `lookup <id>`
subcommands so the Makefile can introspect manifest entries
without grep gymnastics.
Idempotency contract documented
================================
Every ingest path is content-addressed. Same content →
same `document_root` → no-op insert. Re-running `make textbook-*`
or `make crawl-textbooks` on the same network state is safe at
the database layer — no duplicates, no schema drift, no
audit-event proliferation. HTTP refetch happens every run; pair
with `make recrawl-check` for a HEAD-only freshness probe.
README rewritten to make this contract explicit, with shard-
layout table + crawl-delay caveats consolidated.
Vital-books checklist
=====================
The README now carries a g4-pillar coverage table:
Pillar I Logic ✓ Levin + Aristotle Prior + Posterior
Pillar II Set Theory ✓ Levin
Pillar III Arithmetic ✓ Levin
Pillar IV Geometry ✗ Hilbert PG #17384 (PDF/TeX only)
Pillar V Probability ✗ Kolmogorov license analysis pending
Pillar VI Phys. Newton ✓ Newton Principia Motte
Pillar VII Combin. ✓ Bogart + Keller-Trotter + Levin
Pillar IX λ-Calculus ✗ Church + Turing 1936 papers pending
5 of 7 pillars now have surface coverage. The remaining gaps
are explicit license (V) or format (IV PDF/TeX, IX paper-level)
issues, each documented in the corresponding manifest entry's
notes field.
Test suite stays at 1554 passed / 28 skipped — no source-code
changes to arborist itself; the textbook layer is pure tooling
on top of the existing pipeline.
Rolling research log gains a 2026-05-09 amend for the
multi-witness pipeline (#000028) running against live Hermes for
the first time. Qualitatively different from the random-word
triangulation amends above — those measure honesty under no
ground truth; this measures agreement under available ground
truth (kernel IS the ground truth on canonical-shape questions).
First sweep: 8 canonical-shape questions, 5 LLM-DIVERGED, 3
STRICT-WITNESSED. Three distinct failure shapes captured:
1. Wrong arithmetic on float-shape input
0.1 + 0.2 → kernel 3/10 vs Hermes 1/10
(off by 2/10; possibly trained on the IEEE-754 trap as
the "answer" itself rather than recognizing the kernel
returns the exact rational)
2. Implication-tautology error
A IMPL B → kernel (NOT A OR B) vs Hermes TRUE
(NOT B) IMPL (NOT A) → same divergence
(Hermes treats the formula as a tautology rather than
canonicalizing to CNF; A OR NOT A — the genuine tautology
— correctly returns TRUE)
3. Symbolic erasure on algebra-shape input
(x+1)**2 → kernel x²+2x+1 vs Hermes 1
x**2 + 2*x + 1 → same divergence
(Hermes collapses to a constant — possibly evaluating at
x=0 — instead of returning the canonical expanded
polynomial)
The amend frames witness-sweep as the *capability-failure*
signal source, distinct from random-word emergent's
*honesty-failure* signal. Different signals, different repair
paths:
- Verifier ladder hardening / warrant-tier sharpening
addresses honesty failures (false STRICTs).
- Prompt engineering / fine-tune-on-divergence-corpus
addresses capability failures.
The verifier ladder cannot help with witness-divergence — the
LLM fundamentally produced a wrong answer that no number of
citation checks recovers. This is the calibration-data stream
the original #000028 ticket imagined.
End-to-end validates: canonical persistence (#000027),
multi-witness fan-out (#000028), audit-event chain — 0 breaks on
chain-check-shards post-sweep, 5 providence_canonical_witness
events appended, capital ledger captured 5 canonical_witness
op_type rows.
Pattern documented for future witness-sweep amends as the
divergence corpus grows (cron + commit harness in
bench/scripts/witness_sweep_cron.sh handles the unattended
collection).
Bench artifact: bench/results/witness-sweep.json.
Calibration corpus: bench/fixtures/5f/falsification-witness-v1.jsonl
(10 rows after two extraction passes today).
Per the #000031 implementation track — surface-ingest cited
textbooks for warrant promotion of claim-pack records. Five new
manifest entries spanning the PD-scope additions plus
license-fail placeholders for citation traceability:
newton-principia-motte (PD)
Newton's Mathematical Principles of Natural Philosophy
(Motte 1729 / Motte-Cajori 1846 English translations) via
Wikisource. Crawled @ depth=2: 60 docs, 289 chunks, 10K
edges. FTS5 spot-checks: "law of motion" → top hit on the
1729 Motte title page; "gravitation" → Newton-titled
chapters in the Wikisource Physics category. Cited by
claim-pack pillar VI.
aristotle-posterior-analytics (PD)
Mure / Owen translations via Wikisource. Crawled @ depth=1:
20 docs, 71 chunks. FTS5: "syllogism" → Posterior Analytics
Bouchier + Owen translations as top hits. Companion to
Prior Analytics already in the manifest; together cover
pillar I logic axioms.
kolmogorov-foundations-probability-1933 (placeholder)
German original PD-by-age in EU; US copyright restored via
URAA through life+70 (Kolmogorov d.1987 → US PD 2058).
Morrison 1956 English translation: Chelsea Publishing
copyright. Entry kept with empty urls field for citation
traceability; license analysis pending.
lukasiewicz-aristotle-syllogistic-1951 (proprietary placeholder)
Oxford University Press, copyright through life+70
(Łukasiewicz d.1956 → PD 2027 EU, longer US via URAA). NOT
redistributable. Kept as a license-fail placeholder; the
1921 Polish primary papers are themselves PD by age but
lack accessible HTML editions.
License-validator drive-by
==========================
The textbooks_manifest.py validator now treats placeholder
entries (urls + crawl_url both empty) as exempt from the
allow-list license check. License discipline stays fail-closed
at the URL-emit step — placeholders without URLs raise no
exception under any license token, so we can document
yellow-light proprietary citations (Mendelson, Enderton,
Łukasiewicz) without hacks.
Coverage summary post-commit
============================
g4 pillar | textbook coverage
─────────────┼─────────────────────────────────────────────────
I Logic | Levin (CC-BY-SA), Aristotle Prior + Posterior (PD)
II Set Theory| Levin (CC-BY-SA)
III Arithm. | Levin (CC-BY-SA)
IV Geometry | (Hilbert PG #17384 placeholder — PDF/TeX only)
V Probability| (Kolmogorov 1933 license analysis pending)
VI Phys. | Newton Principia Motte (PD)
VII Combin. | Bogart (GFDL), Keller-Trotter (CC-BY-SA), Levin
Out-of-scope follow-ups: PDF/TeX source support (#000031 §5),
Mendelson + Enderton license decision (#000031 §2.1),
chunk-resolution + derivations.proof_blob warrant promotion
(separate ticket).
Pillar VII bundle (axiomsclaude-vii-v1.json +
theoremsclaude-vii-v1.json) ingested via the existing claim_pack
source into ~/.arborist/shards/000.db. 14 records (7 axioms + 7
theorems): Addition / Multiplication / Pigeonhole Principles,
Factorial + Binomial Coefficient definitions, Pascal's Rule,
Empty-Set Boundary; Binomial Theorem, Inclusion-Exclusion
(counting form), Hockey-Stick, Vandermonde, Catalan Closed
Form, Stars and Bars, Strong Pigeonhole.
Combined with the v2 bundles (78 records across pillars I-VI +
IX), shard 000 now carries 92 claim_pack documents.
Retrieval lift verified on representative combinatorics queries:
- Pascal's rule → claim-pack record at #2
- pigeonhole → Strong Pigeonhole Principle at #2
- Modus Tollens → claim-pack record at #3
Authorship metadata: Claude blackops draft + cite-check against
Stanley / Brualdi / Wilf / Knuth (option C from #000033 §2.1).
Records cap at ANCHOR-WARRANTED on the four-rung ladder until
#000031 surface-ingests the cited textbooks and computes
derivations.proof_blob — that's the warrant-promotion track.
Drive-by Makefile fix
=====================
Crawl shards now land in $(CRAWL_SHARDS_DIR) ($(HOME)/.arborist/crawl)
by default, separate from $(SHARDS_DIR) ($(HOME)/.arborist/shards).
SQLite's max-attached-databases limit is 10; mixing 4 main shards
+ 6 crawl shards + qa.db + snapshots.db put us at 12 and broke
cross-shard queries. Crawl shards moved to a sibling dir; the
existing crawl_textbooks-stats target reads from both for backward
compat with already-placed shards.
The first iteration of this layer landed scaffolding plus shallow
seed-URL ingest, which produced 300-1000 byte stubs (front matter
+ nav chrome only) instead of real book content. Fox flagged
that as the wrong outcome — we were supposed to track down the
actual texts.
This commit fixes it by adding the deep-ingest path: each
manifest entry now carries `crawl_url` + `crawl_depth` +
`crawl_max` fields, and `make crawl-textbooks` iterates them
through the existing BFS crawler (one shard per host).
Manifest changes
================
Three URL fixups uncovered by the first crawl attempt:
- PreTeXt textbooks (Bogart, Keller-Trotter, Levin) ship an
`index.html` that's a 1-line meta-refresh redirect; the BFS
crawler doesn't follow meta-refresh, so the seed URL has to
point at the actual content entry (`frontmatter.html`,
`app-comb-2.html`, `ctgd/index.html`).
- Open Data Structures uses trailing-slash directory paths
(`/ods-cpp/`, `/ods-python/`, `/ods-java/`), not `/ods-cpp.html`.
- `appliedcombinatorics.org` declares `Crawl-Delay: 20` in
robots.txt — full BFS takes ~25 minutes; documented in the
manifest entry's notes so operators know to background it.
Hilbert + Boole stay flagged as `urls: []` placeholders because
PG eBooks #17384 + #15114 ship as PDF/TeX only (no HTML format).
Their `tex_url` + `pdf_url` fields are now preserved on the
entry for the future PDF/TeX source ticket.
Make targets
============
- crawl-textbooks — BFS-crawl every entry with `crawl_url`
- crawl-textbooks-stats — docs/chunks/size summary across crawl shards
Helper script: `bench/scripts/textbooks_manifest.py crawl-targets`
emits one tab-separated `<url>\t<depth>\t<max>\t<id>` row per
entry that declares a `crawl_url`.
Real-content verification
=========================
Crawl results across 4 of 5 textbooks (Keller-Trotter still
running in background due to crawl-delay; expected to complete
in ~25 min):
crawl_bogart_openmathbooks_org.db docs=44 chunks=161 971 KB
crawl_discrete_openmathbooks_org.db docs=51 chunks=376 1929 KB
crawl_en_wikisource_org.db docs=30 chunks=110 1651 KB
crawl_opendatastructures_org.db docs=64 chunks= 84 1110 KB
TOTAL 189 docs, 731 chunks
FTS5 spot-checks confirm substantive content landed:
- Bogart "binomial coefficient" → "CTGD Partitions and Stirling
Numbers" chapter, top hit
- Levin "modus ponens" → "Propositional Logic" chapter, top hit
That's the real lift signal we were after. Out-of-scope
follow-ups remain: PDF/TeX support for PG editions (Hilbert,
Boole) and full Aristotle Owen-translation depth (currently
~75% complete on Wikisource).
First make fetch-textbooks run revealed three URL issues that the
license-validator catches as policy-correct but the underlying
hosts return as 404:
- Keller-Trotter applied-combinatorics.org used a stale
/appcomb/sec_pre.html path; the canonical PreTeXt-generated
book lives at /book/app-comb-2.html.
- Open Data Structures uses /ods-cpp/ /ods-python/ /ods-java/
trailing-slash directory paths, not /ods-cpp.html (404). Three
language editions of the same content; ingest all three so the
multi-language structure stays intact.
- Hilbert Foundations of Geometry on Wikisource resolves to a
1906 Popular Science Monthly article ABOUT Hilbert's work, not
Hilbert's translated text. Project Gutenberg eBook #17384 is
PDF/TeX only — no clean HTML edition exists. Same status as
Boole #15114: kept the entry as a placeholder for license +
provenance documentation, urls intentionally empty until PDF
source support lands (out of scope per #000031).
Final fetch result: 7/7 manifested URLs ingested (Bogart,
Keller-Trotter front matter, Levin front matter, Aristotle Prior
Analytics, ODS C++, ODS Python, ODS Java) — 7 docs / 7 chunks /
86 edges / 8/8 Merkle proofs verify (verify samples 8 because
Aristotle yields multiple chunks).
Two textbooks (Hilbert + Boole) carry zero URLs; the helper
script's allow-list passes them through (license is PD;
license_url is set) but emits no URLs since their `urls` array
is empty. Documents this gap explicitly in each entry's notes
field so future operators see what awaits PDF source support.
Front-matter URLs alone yield minimal indexable content per
textbook (PreTeXt single-chapter scaffolds). To grow real
coverage: append more chapter URLs to a textbook's `urls`
array, OR use `make crawl-ingest URL=<home>` per the README's
"Going deeper" section.
Three artifacts landing per ticket §4.1 closure criterion:
1. docs/_source/merkle-agi-v7w-spatial-temporal.rst (658 lines)
============================================================
Substrate paper for the third commitment substrate — sister to v7
(logic / math) and arborist v9.8 (language / claim-lattice). v7-W
commits derived spatial-temporal world-state: objects, relations,
events, places, agent traces, observations. Six parts + appendix:
Part 1 — Introduction & motivation. The third-substrate gap;
why v7 § 11 multimodal composition isn't enough.
Part 2 — Substrate definition. Hierarchical-grid spatial
discretization (S2 / H3 / octree); frame as committed
object with explicit transforms; substrate-declared
clock (single-agent) + Lamport (multi-agent);
quantized centi-confidence (range opt-in); five
canonical tuple-classes (object / relation / event /
place / agent_trace) each with its own π*_w.
Part 3 — Theorems. T1-W (state binding), T2-W (causal
completeness), T3-W (frame-transform soundness),
T4-W (ε at affine frontiers).
Part 4 — Verifier kernels. Pose integration, observation
update (Kalman), object logits, relation logits.
Each affine after canonical projection.
Part 5 — Multimodal composition with v7. Where v7 ends, v7-W
begins; cumulative ε across substrates; frame-
transform anchoring.
Part 6 — Adversarial corners. Frame spoofing, time skew,
observation injection, privacy.
Appendix — Worked SLAM example with full ε budget.
Hard constraints honored: stays inside SQD A1-A3 (canonical
encoding, public quantization, collision-resistant hash); no new
axiom; every π*_w defined on quantized integer state, never on
continuous tensors.
2. docs/v7w-frontier-catalog.md (262 lines)
============================================
Operator-facing quick reference for the four ε-frontiers from
substrate-paper Part 4. Each entry:
- canonical input / output bytes
- operator (linear / bilinear / Kalman / SE(3))
- ε bound expression
- "affine after canonical projection" justification
- when to use
Reference table + cumulative-ε section so operators sizing
deployment grid choices can read off their ε_total under typical
agent-trace + scene-graph workloads.
3. arborist/world/__init__.py — namespace reservation
======================================================
Reserved ``arborist.world`` package. No kernels yet. Module
exports V7W_VERSION ('v0-draft') + STATUS ('namespace_reserved')
metadata. Package docstring lays out the future shape per
substrate-paper Part 4:
arborist/world/
├── pi_star/ — π*_w canonical projections (5 tuple classes)
├── frontier/ — ε-frontier kernels (4 frontiers)
├── frame.py — frame definitions + transforms
├── clock.py — wall-clock + Lamport
├── manifest.py — substrate manifest schema
└── adapters/ — sensor adapters land here, separate tickets
Implementation tickets cite the substrate paper and land kernels
one at a time; the stub exists so cross-referencing imports (mesh
peers, sibling repos) can pin the namespace before anything
implements it.
5 tests pin the reservation contract (test_world_namespace.py):
import succeeds, V7W_VERSION reports v0-draft, STATUS reads
namespace_reserved, __all__ exposes only metadata, substrate
paper + frontier catalog files exist alongside the namespace.
Closure criterion (#000013 §7): substrate paper lands and is
ready for review. Done. Status flipped to closed in the ticket
file + TICKETS.md index entry.
Test suite: 1641 passed, 37 skipped (was 1636; +5).
A make-target scaffold for pulling public-domain and copyleft-
redistributable textbooks into a dedicated arborist shard via the
existing HtmlPageSource ingest pipeline. No new ingest machinery —
the existing path (robots.txt → noise-strip → 512-token chunk →
Merkle root → audit-event) IS the consistent process.
Manifest format
===============
bench/fixtures/textbooks/manifest-v1.jsonl — one JSON record per
textbook with explicit license tokens (PD / CC-BY / CC-BY-SA /
GFDL / OSI-approved). Fail-closed validation in
bench/scripts/textbooks_manifest.py refuses to emit URLs from
entries with missing or disallowed license tokens, so license
discipline surfaces at the URL-emit step rather than getting
silently buried in a fetch run.
Allow-list excludes CC-BY-NC (incompatible with arborist's
AGPLv3 distribution profile) and CC-BY-ND (no-derivatives
prevents chunking). Wilf's generatingfunctionology stays out
because its educational-use license forbids rehosting; citable
but not redistributable.
Initial entries (7 textbooks, 8 seed URLs)
==========================================
- Bogart, Combinatorics Through Guided Discovery — GFDL-1.3
- Keller & Trotter, Applied Combinatorics — CC-BY-SA-4.0
- Levin, Discrete Mathematics: An Open Introduction (3rd ed) —
CC-BY-SA-4.0 (4th ed switched to NC; pin 3rd)
- Hilbert, Foundations of Geometry (Townsend 1902) — PD via PG
- Boole, An Investigation of the Laws of Thought (1854) — PD
- Aristotle, Prior Analytics (Jenkinson) — PD via Wikisource
- Morin, Open Data Structures — CC-BY-2.5
Covers pillars I (logic) / II (set theory) / III (arithmetic) /
IV (geometry) / VII (combinatorics) on the claim-pack
(#000029) layout, plus a CS anchor for downstream domain
expansion. Each entry's pillar_targets field lists which
claim-pack pillars its records are candidate citations for.
Make targets
============
- textbooks-summary — license + URL counts per entry
- textbooks-urls — flat URL list to stdout
- fetch-textbooks — ingest all manifested URLs into
$(TEXTBOOK_DB) (default
~/.arborist/textbooks.db)
- textbooks-stats — documents / chunks / edges
- textbooks-verify — sample Merkle proof verification
To grow coverage of one textbook: append more chapter URLs to
its `urls` array. For deep-BFS of a textbook home: use the
existing `make crawl-ingest URL=<base> DEPTH=N` instead.
Smoke-tested end-to-end against Wikisource Prior Analytics:
1 doc / 1 chunk / 12 outbound edges / Merkle proof passes.
Shows the pipeline works; populating each book to depth needs
either more URLs in the manifest or the crawler.
Out of scope
============
- PDF processing. The Internet Archive hosts PD textbooks
(MacMahon's Combinatory Analysis 1915, Whitworth's Choice
and Chance 1867, the Motte 1729 Principia) as scanned PDFs;
a TextbookPdfSource with pdftotext / pypdf extraction is a
separate ticket — current scope is HTML-shaped sources.
- Mendelson + Enderton (proprietary; await #000031 §2.1
decision).
- Wilf generatingfunctionology (license forbids redistribution).
Test suite stays at 1554 passed / 28 skipped — no source-code
changes to arborist itself; the textbook layer is pure tooling
on top of the existing pipeline.
Three new tickets carve up the open questions from §9 of
docs/soft-hash-channel-analysis.md (#000018):
#000034 — Hessian alignment under φ_linear
============================================
Computational. Measure spectrum of W^T W (the v7 reference
embed_hard_to_vec frozen-seed projection matrix) vs typical
training-loss Hessian eigenvalue distributions on representative
checkpoints. Determines whether the linear projection has
structural alignment with low-eigenvalue directions, which would
let T2 adversaries amplify covert-channel steerage beyond the
random-oracle baseline established in #000018 §4.
Deliverable: bench/scripts/phi_alignment_probe.py + verdict
(STRUCTURAL_ALIGNMENT / NO_ALIGNMENT / ANTI_ALIGNED) per
representative checkpoint. Parks until a v7 reference checkpoint
is available; the analysis is empirical-only and useless without
representative training data.
#000035 — PRG choice for φ_PRG
================================
Cryptographic. Pin a specific PRG construction for the M1
mitigation (PRG-based anchor map) proposed in #000018 §5.2.
Recommended: HMAC-SHA-512(seed, digest ∥ counter) → uniform-random
floats in [-1, 1].
Reasons:
- Tightest dependency surface (stays in SHA family already
committed via SHA-256).
- NIST-approved PRF construction (SP 800-108 KDF in counter mode).
- Speed parity with AES at v7 cadence; PRG cost negligible.
- Provable security reduction: PRF security from SHA-512
collision-resistance, structurally stronger than SHA-256.
Deliverable: arborist/v7/anchor_prg.py + tests + known-answer-test
fixture + v7 § 9.10 amendment text. Lands when v7 plastic-training
has a deployment target.
#000036 — T3 per-window budget bound
=====================================
Formal. Derive an explicit closed-form upper bound on the covert-
channel capacity under threat model T3 (hyperparameter adversary)
when M2 (per-checkpoint nonce) is in place. #000018 §6 lists
"bounded by per-window budget" without giving the bound.
Three control bandwidths combine:
B1 — gradient bias (g · W · log₂ N_directions)
B2 — LR selection (⌈W/K⌉ · log₂ R)
B3 — batch order (Bottou-Bousquet conjecture: ≪ log₂ N!)
Deliverable: docs/soft-hash-channel-t3-bound.md (§§1-8 formal
derivation) + bench/scripts/t3_bound_calculator.py (operator-
parametrized closed-form). Operators plug in (g, K, R, W,
batch_count, gradient_norm) → read off bits/window residual; pick
nonce-window length to drive residual under their target.
Index updated; Next ID 000037. Cross-reference added to #000018's
status line so future readers see the three follow-ups in line.
Three streams. Two land in the repo; one lands in fox's Downloads
(existing g4 packs live there too).
Witness sweep automation
========================
`bench/scripts/witness_sweep_cron.sh` — schedulable harness that
runs `make bench-witness-sweep` + `make bench-witness-divergence`
back-to-back and (with --commit) auto-commits any new divergence
fixtures + sweep result. Fail-closed: pre-commit hook failure
aborts; non-main-branch refuses to commit.
$ bench/scripts/witness_sweep_cron.sh # dry run
$ bench/scripts/witness_sweep_cron.sh --commit # auto-commit
$ bench/scripts/witness_sweep_cron.sh --commit --push # full pipeline
Recommended cron (every 6 hours; LLM cost scales with frequency):
0 */6 * * * \
bench/scripts/witness_sweep_cron.sh --commit --branch main \
>> /var/log/arborist-witness.log 2>&1
Smoke this commit: dry-run produced 5 fresh divergences (10
cumulative across two extraction passes). The witness-sweep + 5F
extractor pipeline runs unattended now; calibration-data corpus
will grow over time.
#000018 — adversarial soft-hash covert-channel analysis
========================================================
`docs/soft-hash-channel-analysis.md` — formal threat model + channel-
capacity definition + cryptographic reduction + mitigation table
+ residual analysis. Doc-only deliverable per the ticket's hard
constraint.
Headline result: under threat models T1 (data poisoner) and T2
(gradient adversary), the channel is bounded by SHA-256 partial-
preimage hardness when φ (the anchor map) is independent of
parameter geometry. T3 (hyperparameter adversary) requires
M2 (per-checkpoint anchor nonce) for boundedness.
Recommended mitigation: M2 (per-checkpoint nonce) — 32 bytes added
to the snapshot manifest, no training-ergonomics regression,
closes T1+T2 channels under any φ choice and bounds T3 by
per-checkpoint window. Fall back to M1 (PRG-based φ) if M2-only
deployment surfaces structural concerns. M3 (drop anchor entirely)
stays in reserve as the strict-construction fallback.
Three open questions (§9): Hessian alignment under φ_linear,
PRG choice for φ_PRG, and explicit T3 per-window bound. Each is a
follow-up ticket.
Ticket #000018 status: closed · landed 2026-05-09 (analysis doc).
v7 § 9.10 spec amendment proposed in §7 of the analysis.
#000033 — pillar VII (combinatorics), Claude-authored
======================================================
NOT committed to the arborist repo (the existing g4-v2 packs live
in `/home/fox/Downloads/` too — that's the operator's bundle
location). Two new bundle files at:
/home/fox/Downloads/axiomsclaude-vii-v1.json (7 axioms)
/home/fox/Downloads/theoremsclaude-vii-v1.json (7 theorems)
Pillar VII covers combinatorial counting — the gap between Grok's
pillars VI and IX in the v2 packs:
axioms (7): addition principle · multiplication principle ·
pigeonhole principle · factorial definition ·
binomial coefficient definition · Pascal's rule ·
empty-set / boundary axiom
theorems (7): binomial theorem · inclusion-exclusion (counting
form) · hockey-stick identity · Vandermonde's
identity · Catalan number closed form · stars-and-
bars · strong pigeonhole
Each record in the dual-thread format the existing g4 packs use
(Δ symbolic LaTeX + ∇ verbose prose + ∇ concise + sigil + formal
language + role + status + source_reference + date + foundational
group + category + subfield). Per fox's directive: explicit
authorship metadata everywhere — `authored_by: Claude (Anthropic)
— model claude-opus-4-7`. NOT Grok-generated; no silent invention.
Each record carries `pi_star_ref: combinatorics@v1` so the kernel
binding is explicit. Theorems list `depends_on_axioms` arrays so
each theorem cites the foundation axioms it bottoms out on.
Smoke test (committed alongside):
$ arborist --db /tmp/test.db ingest --source claim_pack \\
--bundle /home/fox/Downloads/axiomsclaude-vii-v1.json \\
--bundle /home/fox/Downloads/theoremsclaude-vii-v1.json
→ 14 docs, 14 chunks, 0 cross-bundle edges
Source attributions: Stanley EC1, Brualdi Introductory
Combinatorics, Knuth TAOCP Vol 1, plus historical sources where
applicable (Pascal 1654, Vandermonde 1772, Dirichlet 1834, Catalan
1838, Feller 1950 for stars-and-bars).
Tests: 1636 passed, 37 skipped (no regressions; pillar VII
ingestion smoke covered above).
Apples-to-apples FTS5 search comparison on a single shard
(~/.arborist/shards/000.db, ~867 K Wikipedia surface docs)
before vs after ingesting the claim-pack source (#000029)
bundles.
Method: copy 000.db twice; ingest axiomsg4-v2.json +
theoremsg4-v2.json into one copy; run identical search queries
against both. FTS5-only path so retrieval-layer behavior is
isolated from QA pipeline reranking.
Results: 3/4 representative queries show claim-pack record in
top-3.
modus tollens → claim-pack at #3 (BM25 31.77)
law of excluded middle → claim-pack at #3 (related axiom
in same pillar)
associativity of addition → claim-pack at #1 — displaces
Wikipedia's general "Addition"
article entirely
Bayes theorem → no top-3 lift; Wikipedia "Bayes"
+ "Bayes rule" articles dominate
via short-doc BM25 bias
Storage tax: sub-MB. 78 chunks against 6.2M existing chunks is
below filesystem allocation granularity. Claim-pack rides
essentially for free on the existing Wikipedia substrate.
Honest verdict: claim-pack ingest earns its storage tax for
narrow technical queries where pre-distilled CORE-shape content
has title-token advantage. Doesn't help on queries Wikipedia
already covers with focused articles. The Bayes theorem case
likely shifts under the full QA pipeline's body-coverage sqrt
rerank (which counters BM25 short-doc bias) — that's a separate
larger bench worth running once the retrieval layer's other
stages are exercised.
Four-rung audit ladder still caps at ANCHOR-WARRANTED per
#000029's honest ceiling. Surface-ingest of cited textbooks
(#000031) is the path toward EVIDENCE-WARRANTED.
Three small streams:
#3 — close#000030 properly
============================
All 7 phases + Phase 1b landed across two commits (`04f3f5d`,
`abe5988`). Status header updated; ticket body now carries a phase
landing table with commit refs:
Phase 1 algebra-symbolic@v1 04f3f5d
Phase 1b algebra-symbolic-simplified@v1 04f3f5d
Phase 2 calculus-derivative@v1 04f3f5d
Phase 3 calculus-integral@v1 fox-direct
Phase 4 calculus-limit@v1 abe5988
Phase 5 calculus-series@v1 abe5988
Phase 6 linear-algebra@v1 abe5988
Phase 7 function-sampled@v1 abe5988
Plus tabular-pinned@v1 (last reserved stub) graduated in abe5988
closes the registry chapter — 15 concrete π*'s, no remaining
reserved stubs. Index updated.
#5 — composition fixtures across new SymPy π*'s
================================================
12 new tests in tests/test_pi_star_compositions.py covering pairs
that compose naturally:
- algebra-symbolic ∘ algebra-symbolic — idempotency check (running
expand twice equals expand once for any expression).
- algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean
identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`).
- Generic invariants: composition propagates PiStarError; manifest
fingerprint is order-sensitive; composite domain == inner domain;
composite bytes == manual chain bytes.
Test discipline: most compositions use `register_in_registry=False`
via a small `_safe_compose()` helper since the registry rejects
duplicate keys (#000015 invariant), so test ordering would
otherwise matter. Only the registration-test path uses real
compose().
#4 — end-to-end witness sweep against real shards + Hermes
===========================================================
New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape
questions (3 arithmetic + 3 logic + 2 algebra) through query() with
`canonical_witness_enabled=True`, against ~/.arborist/shards (real
shard cluster) + the actual Hermes endpoint (NOT StubClient).
Records the agreement matrix per question to
bench/results/witness-sweep.json.
`make bench-witness-sweep` Makefile target. Honors
`ARBORIST_SHARDS_DIR`.
First real sweep (this commit, against Hermes-3-8B):
agreement label count rate
KERNEL-LLM-DIVERGED 5 62.5%
KERNEL-LLM-AGREE 3 37.5%
───────────────────────────────────────────
divergence_count 5 62.5%
wall median / max 130 ms / 1.1 s
Hermes diverged on 5/8 of the canonical-shape questions:
- said `1/10` for `0.1 + 0.2` (kernel: `3/10`)
- said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`)
- said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already
expanded — but Hermes ALSO
emitted the unexpanded form
when given the expanded
form, vs the kernel's
deterministic expand)
- and 2 more.
These are real LLM hallucinations on questions with closed-form
ground truth — exactly the calibration-data stream #000028
imagined. Pipeline validated end-to-end against actual hardware.
Pair: `make bench-witness-divergence` then extracts the 5
divergences as 5F-Falsification fixtures
(bench/fixtures/5f/falsification-witness-v1.jsonl, also committed).
Re-running the extractor produces byte-equal output (idempotency
contract from the extractor work).
Tests
=====
Full suite: 1636 passed, 37 skipped (was 1624; +12 composition
tests). The witness-sweep + extractor produce real artifacts now
committed under bench/results/ and bench/fixtures/5f/.
A new π* kernel that canonicalizes pure-integer counting
expressions and FAILS CLOSED on any input whose result isn't a
non-negative sp.Integer. Tighter domain than algebra-symbolic@v1,
which already accepts the same input surface but happily returns
symbolic / negative / non-integer outputs.
Distinguishing feature versus algebra-symbolic@v1:
algebra-symbolic@v1: binomial(n, k) → "binomial(n, k)" (symbolic
passthrough)
combinatorics@v1: binomial(n, k) → PiStarError (fail-closed
on free-symbol output)
algebra-symbolic@v1: binomial(Rational(1,2), 3) → 1/16 (rational)
combinatorics@v1: binomial(Rational(1,2), 3) → PiStarError
(output not Integer)
Boundary kept explicit: binomial(-3, 2) = 6 IS accepted because the
output is an integer 6. The fail-closed rule is on output shape
(Integer ≥ 0), not input range. Documented as
test_generalized_binomial_negative_args_accepted_when_integer.
Output format: plain decimal literal (b"10", b"5040"). Composes
with arithmetic@v1 for byte-identical agreement with the rational
route (b"10/1") so the multi-modality witness (#000028) can pin
equivalence-class agreement when both routes fire on the same
question.
Allowed surface (via SymPy primitives): binomial, factorial, ff /
rf (falling/rising), catalan, bell, partition, stirling, plus
arithmetic compositions over those primitives
(3*binomial(5,2) + factorial(4) = 54).
Coverage:
- 43 unit tests including binomial symmetry C(n,k)=C(n,n-k),
Pascal's rule C(n,k)=C(n-1,k-1)+C(n-1,k), the C(n,k) =
factorial(n)/(factorial(k)·factorial(n-k)) identity,
fail-closed paths (symbolic/negative/non-integer/relational/
parse), round-trip idempotence, composition with arithmetic@v1.
- 10 syntax + 12 semantics bench fixtures, 100% pass.
- bench/batteries/base.py PHASE_1_CARRIERS gains "combinatorics".
- Makefile bench-5s-combinatorics target.
All gate on pytest.importorskip("sympy") so a sympy-less suite
stays green. Full make test: 1537 passed / 28 skipped.
Sequencing rationale honored: this kernel lands FIRST so that
#000033 (claim-pack pillar VII for combinatorics) can bind its
records to the tighter integer kernel from day one — avoids
rebind churn on pi_star_ref fields.
Three small streams in one commit:
#000028 follow-up — witness divergence → 5F fixtures
=====================================================
Witness fan-out now writes a `providence_canonical_witness` audit
event when it fires (next to the capital-ledger record landed in
708aa45). Body carries pi_star_ref, question_text, agreement_label,
canonical_answer_text, llm_raw_text, llm_canonical_bytes,
cache_status. Best-effort write — chain failure never fails the
query.
New extractor `bench/scripts/witness_to_5f.py` reads those events
from a qa.db and writes them out as 5F-Falsification fixtures
matching the existing `falsification-live-v1` schema. Filtering
includes only divergence labels (LLM-DIVERGED / KERNEL-LLM-DIVERGED
/ CACHE-DRIFT); skips KERNEL-LLM-AGREE / STRICT-WITNESSED (no
calibration signal) and KERNEL-ONLY (LLM unparseable, not a
supervised-correction sample).
Idempotent: sorted by audit-event seq, so re-running against the
same qa.db produces byte-equal fixture files. The existing
fixture-digest discipline stays valid.
Makefile: `make bench-witness-divergence` (override default
qa.db / output path via WITNESS_QA_DB / WITNESS_OUT env-vars).
Closes the divergence → calibration data loop the witness ticket
imagined: every LLM hallucination on a canonical-shape question
becomes a supervised-correction fixture downstream prompt
improvements can grade against.
#000030 Phase 7 demo — function-sampled@v1 end-to-end
======================================================
`bench/scripts/demo_plot.py` — closes the loop on opencompletion's
activity24-math-plot.yaml. SymPy expression → quantized
integer-vector signature (canonical bytes) → optional matplotlib
PNG. Canonical bytes are the proof; PNG is just a downstream view
of the same evidence.
$ make demo-plot Q='sin(x)' PNG=/tmp/sin.png
Output JSON contains canonical_bytes_sha256 + canonical_bytes_preview
+ canonical_bytes_total_chars + grid metadata + the optional png_path.
matplotlib is gated — when absent, --png prints a warning to stderr
and skips the render; the canonical bytes still print. Tests skip
the PNG-presence assertion via `pytest.importorskip("matplotlib")`.
Public docs polish (#7)
========================
- docs/_source/bench.rst: updated fixture-count narrative (~660 →
662 default tasks + ~110 math π* fixtures); `make` quick-reference
now lists all per-π* 5S targets (tabular, calculus-limit/series,
linear-algebra, function-sampled) plus bench-real-shard,
bench-fork-baseline/score, bench-witness-divergence.
- docs/_source/v8-fork-score.rst: CLI section gained --out flag
documentation + a Make-harness sub-section covering
bench-fork-baseline / bench-fork-score / FORK_PARENT/CHILD/REPORT
env-vars.
Tests
=====
- tests/test_witness_to_5f.py — 8 new tests covering the audit-event
write (3) + extractor logic (5).
- tests/test_demo_plot.py — 6 new tests covering canonical-bytes
determinism + equivalence-class collapse + matplotlib gating.
Full suite: 1624 passed, 37 skipped (was 1568; +56).
tabular-pinned@v1 + calculus-limit@v1 + calculus-series@v1 +
linear-algebra@v1 + function-sampled@v1 — all reserved stubs
graduated; the π* registry is now 15 concrete kernels with no
remaining reserved-stub entries.
#000030 Phase 4 — calculus-limit@v1
====================================
sp.limit with thread-timeout. One-sided dir support (+/-/+-).
Pinned spelling for infinity cases: b"+oo" / b"-oo" / b"zoo"
(complex infinity) — bypasses sp.expand since Infinity isn't
algebraic. Finite results re-canonicalize through algebra-symbolic
recipe (sp.expand + sp.srepr). Unevaluated cases / timeouts emit
b"unevaluated:" + sp.srepr(<Limit>) sentinel, mirroring
calculus-integral's pattern.
#000030 Phase 5 — calculus-series@v1
=====================================
sp.series(f, x, x0, n).removeO() → sp.expand → sp.srepr. Drops
O(x**n) remainder explicitly so the canonical form is finite-byte.
Sentinel format mirrors limit/integral: b"unevaluated:Series(...)"
on timeout. n must be a positive int; 0 / float / negative rejected.
#000030 Phase 6 — linear-algebra@v1
====================================
Single π* covers the whole linear-algebra surface via {op, matrix}
JSON. Ops: rref / det / eigenvalues / inverse. Matrix cells go
through Fraction(Decimal(str(...))) for floats so 1, 1.0, "1.0"
all collapse to Rational(1, 1) — matching arithmetic@v1's
discipline. Without this fold, sp.sympify keeps floats as Float
(separate type) and downstream det/inverse return Float-shaped
bytes. Eigenvalues are sorted by srepr for determinism.
Output formats:
rref / inverse: rows/cols header + cells joined by | (rows by ||)
det: det:<num/den-or-srepr>
eigenvalues: eigenvalues:<value-1>x<mult-1>|...
#000030 Phase 7 — function-sampled@v1
======================================
Bridge to time-series-quantized@v1. SymPy expression + linspace
grid → quantized integer-vector signature in time-series's exact
output format (dt=...;dv=...;n=...;t0=0:v0|v1|...). Two functions
that render identically (within sample-grid tolerance) collapse
to the same canonical bytes. This is what plotting CAN become
in π* terms — the PNG render is a downstream view of the same
canonical evidence.
Math-only sampler (no numpy in the dep surface); Python's round()
is banker's-rounding so the bytes are interchangeable with
time-series-quantized@v1's output. Complex / non-finite samples
raise PiStarError rather than silently dropping imaginary parts.
tabular-pinned@v1 — last reserved stub graduates
=================================================
JSON-rows input ({schema, key_columns, rows}); declared
key_columns sort policy (stable sort by primary-key tuple);
type-fold per column (int/rational/bool through arithmetic@v1
discipline; str verbatim; bool normalized). Header case is
PINNED EXACT — Excel and PostgreSQL both care about case;
defaulting to lowercase-fold would break operator expectations.
Output: header (schema + key + n) + rows joined by \n + cells by |.
The π* registry has no remaining reserved stubs. Every modality
the substrate paper reserved is now real.
Test suite: 1568 passed (was 1467; +101). New closure-criterion
test (test_no_stub_pi_stars_remain) replaces the old reserved-stub
parametrize — adding a future stub re-opens this list.
110/110 fixtures pass across the 5 new bench-5s-* targets.
PHASE_1_CARRIERS gained calculus / linear-algebra / function-sampled
/ tabular.
Two design-only tickets opened together because they're tightly
coupled — pillar VII records bind to combinatorics@v1 via
pi_star_ref, and #000032 lands first to avoid rebind churn on
that field.
#000032 — combinatorics@v1 π*
=============================
A new π* kernel that canonicalizes pure-integer counting
expressions (binomial, factorial, permutations, partitions,
Catalan, Bell, Stirling) and FAILS CLOSED on any input whose
result is not a non-negative integer. Tighter domain than
algebra-symbolic@v1, which already accepts the same input
surface but happily returns symbolic / negative / rational
outputs.
Distinguishing feature: algebra-symbolic@v1 returns
binomial(n,k) → "binomial(n,k)" (symbolic), binomial(-3,2) → 6
(generalized). combinatorics@v1 rejects both. Operators choose
the kernel by what they want rejected.
Output format: integer string (b"10"). Compose with
arithmetic@v1 to get bytes-identical agreement (b"10/1") for
the multi-modality witness flow.
Estimated size: ~120 LOC module + ~80 LOC tests + ~22 fixtures.
Single-commit feasible.
#000033 — Claim-pack pillar VII (combinatorics)
================================================
Extend the claim-pack source (#000029) with a new combinatorics
pillar slotting into the documented gap (existing v2 bundles use
I, II, III, IV, V, VI, IX — VII and VIII reserved for
extension). Counting axioms (Pascal's rule, addition principle,
multiplication principle, pigeonhole, factorial / binomial
definitions) + classical theorems (binomial theorem,
inclusion-exclusion in counting form, hockey-stick, Vandermonde,
Catalan closed form, stars-and-bars).
Bundle provenance is the open question — three options
documented:
A. Commission a Grok-4 v3 bundle for parity with the existing
pack.
B. Hand-curate from textbooks (Stanley, Brualdi, Wilf, Knuth).
C. Hybrid — LLM draft + human curation.
Hard constraint: explicit authorship metadata. No silent
invention. Bundle landing is a config + data exercise; no
source-code changes to arborist/sources/claim_pack.py needed
since the source already iterates arbitrary pillar names.
Sequencing: #000032 first (kernel), #000033 next (records bind
to it from day one), #000031 stays parallel-track (textbook
ingest for warrant promotion).
Both tickets stay open · awaiting go/no-go pending fox's
implementation green-light.
Phase 1b — algebra-symbolic-simplified@v1
==========================================
arborist/pi_star/algebra_symbolic_simplified.py — full-simplify
variant of the Phase-1 expand-only sibling. Closes the trig
identity gap left open at end of Phase 1: sin(x)**2 + cos(x)**2
now collapses to 1, tan(x)*cos(x) to sin(x), exp(log(x)) to x.
Recipe is sp.expand(sp.simplify(expr)) — the follow-up expand
after simplify is load-bearing. simplify alone is non-canonical
for polynomials: it leaves (x+1)**2 in factored form while
collapsing x**2 + 2*x + 1 to expanded form, so two algebraically
equivalent inputs would emit different bytes. Composing with
expand picks one canonical polynomial shape and preserves the
equivalence-class invariant.
Cost: 1-360 ms typical on common trig/exp inputs; pathological
inputs unbounded. No in-π* timeout (the calling pipeline owns
that budget). Operators opt in by registry key — the fast Phase-1
sibling stays the default for callers that only need polynomial
canonicalization.
22 unit tests; all gate on pytest.importorskip("sympy").
Phase 3 — calculus-integral@v1
==============================
arborist/pi_star/calculus_integral.py — symbolic integration with
thread-timeout fallback. JSON-shaped {f, x, limits?,
timeout_seconds?} input. Two output paths:
1. Closed form: sp.srepr(sp.expand(integrate_result)) — same
recipe as algebra-symbolic@v1 so the output is itself a valid
algebra-symbolic input and composes naturally.
2. Unevaluated: b"unevaluated:" + sp.srepr(<Integral>). Prefix
lets callers tell "no closed form" from "input invalid"
without re-parsing the canonical form.
Timeout discipline: ThreadPoolExecutor(max_workers=1) +
future.result(timeout=...). On TimeoutError, synthesize the same
unevaluated sentinel SymPy itself would emit, so timeout +
no-closed-form converge to the same bytes for the same input.
Default 30 s; per-call override via timeout_seconds. Python
threads can't be killed cleanly — a timed-out worker leaks until
SymPy returns. Documented as the cost of the discipline.
Coverage: ∫x dx = x²/2, ∫sin(x) dx = -cos(x), ∫_{0}^{π} sin(x)
dx = 2, ∫_{-∞}^{∞} exp(-x²) dx = √π, exp(x)/log(x) →
unevaluated sentinel. 31 unit tests including a monkeypatch
deterministic timeout test (sleep-mocked SymPy so the timeout
path doesn't depend on any specific input being slow on every
CI runner).
Open #000031 — surface-ingest cited textbooks
=============================================
Design-only ticket. Closes the warrant gap left open at the end
of #000029: today every claim-pack record caps at
ANCHOR-WARRANTED because source_reference is a string field, not
a Merkle-bound proof. Ingesting the cited textbooks as surfaces
+ computing per-claim derivations.proof_blob lets the four-rung
ladder promote them to EVIDENCE-WARRANTED.
License gating: PD sources (Hilbert, Newton, Kolmogorov,
Łukasiewicz, Aristotle) form the green-light scope. Mendelson +
Enderton are proprietary and stay yellow-light pending fox's
explicit decision (purchased single copy / library license / PD
substitute via Hilbert-Ackermann 1928).
Two follow-up tickets reserved: textbook-fetch pipeline +
chunk-resolution layer (mapping source_reference strings to
specific spans within ingested textbooks; the bridge that lets
proof_blob be computed).
Test counts: 153 tests for the work in this commit (algebra
+ algebra-simplified + calculus-derivative + calculus-integral
+ preflight). All pi_star + canonical_projection tests pass
under .venv pytest.
Three small streams in one commit; each closes / expands a
recently-landed ticket without changing its hard contract.
#000026 Phase 3 wiring — authorship warrant ladder visible
============================================================
Phase 3 sidecar (arborist/qa/warrant_authorship.py landed in 60b5748)
exposed the classifier but didn't surface it. Two wirings:
- arborist/qa/inspect.py — diagnose_authorship_warrant runs against
the cached row's question + answer + per-source raw chunks +
URIs + titles; result lands as `authorship` field alongside the
other sidecars.
- arborist/cli.py _render_warrant_tail — appends ` · warrant:
<readable-tier>` when result['authorship'] is populated with a
non-quiet tier. AUTHOR_COPYRIGHT_FOOTER → "copyright-footer", etc.
NO_AUTHORSHIP_SIGNAL stays silent. Backward-compat: results
without an `authorship` key render unchanged.
Tests: 3 inspect-path tests (no-signal, copyright-footer,
repository-owner) + 4 render-tail tests (presence, no-signal
silence, missing-key silence, all-six-tiers readable mapping).
#000028 follow-ups — capital ledger + sample-rate
==================================================
Two policy fields layered on top of canonical_witness_enabled:
- canonical_witness_sample_rate (0.0..1.0; default 1.0). Operators
wanting passive calibration set 0.05 to fire witness on 5% of
canonical questions while paying 5% of LLM cost. 0.0 effectively
off; 1.0 = current always-on behavior. Gating uses random.random()
so distribution is uniform; clamped to [0, 1].
- Capital ledger row written for each FIRED witness (not skipped
ones). op_type='canonical_witness'; estimator inputs include
prompt_chars + answer_chars + llm_seconds + agreement_label +
pi_star_ref. Best-effort: ledger-write failure must never fail
the query (sidecar discipline).
Tests: 4 new — sample_rate=0.0 skips (no LLM call, no ledger row);
sample_rate=1.0 always fires; capital_ledger row written under
op_type='canonical_witness' with full input blob; sampled-out
witness records zero ledger rows.
Both fields fold into governance_policy_hash naturally via the
existing policy-hash machinery — flipping witness mode invalidates
prior records as expected.
#000025 Phase 1d — 5F fixture catalog 30 → 50
==============================================
Both synthetic and live sides of all 5 sub-batteries expanded
30 → 50 (+200 fixtures total: 5 × 20 synthetic, 5 × 20 live).
function — claim_count cycles 2..7 across new fixtures
falsification — 10-violation palette across new ids
feedback-loop — fact-N learning chains
finetuning — capability transitions across canonical π*
(math/logic/algebra/calculus pool)
formulate — multi-pointer claim shapes
500/500 pass through respective runners. test_session_integration
total bumped 562 → 662. Pinned test_5f_*_runs counts updated 30 →
50 (synthetic main + embedded + live).
Tests
=====
Full suite: 1467 passed, 36 skipped (was 1388; +79 across warrant
render + witness sample/ledger + 5F implicit coverage).
Two new π* canonicalizers extend the math substrate above
arithmetic@v1 (closed-form rationals) and logic-kernel@v1
(propositional Boolean → CNF):
algebra-symbolic@v1 (Phase 1) — symbolic-algebra domain.
sp.expand → sp.srepr canonical bytes. Polynomial identity collapses
((x+1)**2 ≡ x**2 + 2*x + 1); exponential identity collapses
(exp(a+b) ≡ exp(a)*exp(b), inherited from sp.expand's default
behavior); trigonometric identity does NOT collapse
(sin²+cos² ≢ 1). The trig surface is reserved for a future
algebra-symbolic-simplified@v1 variant that wraps sp.simplify at
unbounded CPU cost. Rejects relationals (`x > 0`) and
BooleanFunction shapes (`x & y`) via `isinstance(expr, sp.Expr)` —
sp.Symbol confusingly inherits from Boolean so the right rejection
filter is "not Expr" rather than "Boolean".
calculus-derivative@v1 (Phase 2) — calculus domain. JSON-shaped
{f, x, n} input → sp.diff → sp.expand → srepr bytes. Output is
itself a valid algebra-symbolic@v1 input so the two compose
naturally under arborist.pi_star.compose. n defaults to 1; bools
explicitly rejected (Python isinstance(True, int) is True so we
filter that explicitly).
Optional dependency: sympy ships in the new [math] extra
(pyproject.toml). Folded into [dev] so make bootstrap pulls it
transitively. An explicit `bootstrap-math` Makefile target documents
the opt-in for minimal-install users. Both modules self-guard
via `try: import sympy as sp / except ImportError: sp = None` and
only register(...) when sympy is present, so a fresh checkout
without [math] still loads arborist.pi_star without raising.
Preflight algebra route lands in
arborist.qa.query._canonical_projection_preflight between the
arithmetic and logic routes. Charset regex (_CANONICAL_ALGEBRA_RE)
allows lowercase letters + math chars; requires at least one
letter (else arithmetic wins); rejects natural-language leading
verbs via _CANONICAL_ALGEBRA_NL_LEAD_RE (4-letter minimum so
single-/two-/three-char identifiers like x, xy, sin, cos, pi
survive while "simplify (...)", "factor x...", "expand (a+b)..."
fall through). PiStarError + KeyError both fall through cleanly
so a sympy-less install just routes everything past algebra.
Bench substrate:
- bench/batteries/base.py PHASE_1_CARRIERS gains "symbolic_algebra"
- bench/fixtures/5s/syntax-algebra-symbolic-v1.jsonl (10 fixtures)
- bench/fixtures/5s/semantics-algebra-symbolic-v1.jsonl (13 fixtures
including the documented trig non-collapse + exp collapse)
- Makefile bench-5s-algebra target → 100% pass
Tests: 18 algebra-symbolic + 38 calculus-derivative unit tests +
~10 new preflight-route tests in test_canonical_projection.py. All
gate on pytest.importorskip("sympy") so a sympy-less suite stays
green. Full suite: 1369 passed / 27 skipped.
Phases 3-7 (integral, limit, series, linear-algebra,
function-sampled) remain open as future work; each lands as its
own ticket when an actual consumer surfaces.
Brain-stormed off opencompletion's activity24-math-plot.yaml
(SymPy + numpy + matplotlib pipeline). Three things in there
map to π* shapes; one doesn't.
Maps:
- algebra-symbolic@v1 — symbolic expression → expand + canonical
ordering. Closes the (x+1)**2 ≡ x**2+2x+1
equivalence class.
- calculus-derivative@v1 — d/dx(f) via sp.diff, re-canonicalized
through algebra-symbolic@v1.
- calculus-integral@v1 — ∫f dx via sp.integrate; sentinel for
unevaluated cases.
- calculus-limit@v1, calculus-series@v1, linear-algebra@v1 —
additional SymPy-friendly phases.
- function-sampled@v1 — bridges symbolic expressions to the
existing time-series-quantized@v1
format. Two functions that render
identically (within sample tolerance)
collapse to same canonical bytes. This
is what plotting CAN become in π* terms.
Doesn't map:
- PNG plot rendering — different DPIs / fonts / palettes all
valid; image bytes aren't canonical. Stays as an output adapter
that COMPOSES with function-sampled@v1.
Hard constraint: SymPy is an OPTIONAL dependency ([math] extra).
Each new π* registers only when sympy is importable, mirroring
the existing html / wikitext / crawler pattern. Fresh checkout
without sympy keeps passing the full test suite (graceful skip).
Multi-phase rollout. Recommendation: Phase 1 (algebra-symbolic@v1)
+ Phase 2 (calculus-derivative@v1) in one commit (~400 LOC total
+ tests + 5S fixtures). Subsequent phases (integral, limit,
series, linalg, function-sampled) each their own commit.
Forward links in the ticket: #000027 (canonical persistence —
algebra/calculus answers inherit audit chain for free), #000028
(witness — symbolic answers become witnessable), 5T/5F (new
fixture surface for symbolic LLM calibration), composition algebra
(deriv+arith, expr+sample compose naturally).
Index updated. Next ID 000031.
ClaimPackSource ingests Grok-4 companion bundles (axiomsg4-v2.json +
theoremsg4-v2.json) at the right grain — one Document per axiom or
theorem record. Each record carries Δ (LaTeX symbolic) + ∇verbose
prose, explicit source citation (Mendelson, Enderton, Hilbert,
Newton, Kolmogorov, Łukasiewicz), foundational-group taxonomy, and a
runicLabel that rides as soft metadata only (runtime mints its own
pointer IDs per CTI architecture). Pillar-level
provenance.references arrays become outbound pillar_reference edges.
Lenient JSON parser strips ```json fences and double-escapes lone
LaTeX backslashes (\Theta, \heart, \vec) without corrupting
already-correct \\to pairs — walks left-to-right and pass-throughs
legal escape sequences. Malformed bundles raise rather than return
empty; silent zero-doc would be a footgun.
CLI surface: --source claim_pack with a repeatable --bundle FILE
flag mirroring html source's --url action=append. Single --path
also accepted for one-bundle ingest.
Drive-by: removed a function-local `from arborist.store import
connect` inside _cmd_ingest's providence branch that was shadowing
the module-level binding via Python's "any local assignment makes
the name local for the entire function" rule, breaking every
non-providence ingest with UnboundLocalError. Comment left in
place explaining why not to re-add it.
Smoke-tested on /home/fox/Downloads/{axiomsg4,theoremsg4}-v2.json
end-to-end: 78 docs (55 axioms + 23 theorems across 7 pillars),
14 deduped pillar-reference edges, 78 audit events, 10/10 sampled
Merkle proofs verify, FTS5 search returns Modus Tollens for
"modus tollens".
Honest ceiling: kind=surface for every record. The pack is
pre-distilled but its provenance is asserted not proven — until
Mendelson/Enderton/Hilbert texts are themselves ingested as
surfaces, the verifier has no derivations.proof_blob to compute
and claim-pack records max out at ANCHOR-WARRANTED on the
four-rung ladder. That's a follow-up ticket, not this one.
Hard constraints honored: no new audit ledger (audit_events
remains the only chained-sha256 ledger; bundle's self-validation
fields ride as metadata only); no kind=core without surface
ancestor; cache_key invariants untouched.
15 unit tests cover lenient parser, slug stability, ref
resolution, doc grain, URI stability, content layout, extra
metadata, edge emission, error paths. All 1280 tests in
make test pass.
Closes#000027. Closes#000028 (cache-leg wired).
#000027 — canonical projections persist to providence_cache
============================================================
Math/logic π* answers (arithmetic@v1, logic-kernel@v1,
time-series-quantized@v1, …) are now first-class providence rows.
Pre-fix: question → kernel → answer → return. No cache, no audit
event, no run_dag, no inspect/burn/replay surface.
Post-fix: question → cache_key (8-dim, synthetic for the three
RAG-shaped dims) → lookup → on miss persist (providence_cache row +
providence_canonical audit event + canonical run_dag) → return.
Synthetic cache_key dimensions for canonical rows (per ticket §2.2):
- source_root = sha256("pi_star_source:" + pi_star_ref)
- model_profile_hash = sha256("pi_star_model:" + pi_star_ref)
- conversation_hash = sha256("pi_star_conv:" + canonical_q + ":" + ref)
- chunking_version = literal "n/a-canonical" — chunker bumps on
wikipedia path don't stale math answers.
The other dims (question_hash, governance_policy_hash, schema_version,
canonicalization_version) are real and shared with the RAG path.
Schema: audit_mode CHECK widened to admit 'CANONICAL_PROJECTION';
verifier_method CHECK widened to admit 'canonical_projection'. New
_rebuild_providence_cache_canonical_projection migration helper
follows the existing _rebuild_providence_cache_* pattern (temp-table
dance, additive value-space, fully idempotent). Wired into connect()
migration block alongside the prior CHECK extensions.
Cache-hit policy: trust the row. Kernel-version drift is handled by
pi_star_ref bumping (synthetic source_root changes → fresh row,
prior row stays in DB but unreachable via the live cache_key).
Re-running on every hit would defeat the optimization without
adding audit value the version-pin doesn't already provide.
Policy gate: canonical_projection_preflight_persist (default True).
Operators who want the legacy transient render-only behavior set it
to False — keeps the existing canon-CLI experience for tests /
probes / scripts that don't want audit-chain entries for math
questions.
CLI render: `CANONICAL · via canonical_projection` for persisted
rows. Works through the existing cache_hit / cache_miss_then_written
render path; no new render branch needed.
`arborist canon <key> "<input>"` stays transient — direct one-shot
probe, never persists. Boundary preserved per ticket §2.6.
#000028 — multi-modality witness cache-leg
==========================================
Pre-#000027 the witness cache-leg closure always returned None;
STRICT-WITNESSED (3-of-3 byte-equal) was structurally unreachable.
Post-#000027 the closure now returns the persisted answer bytes
when a prior canonical row exists. Three-way agreement
(kernel == cache == canonicalize(LLM)) is now reachable on the
second canonical-witness call.
New test test_query_canonical_witness_reaches_strict_after_persist
covers it end-to-end: first call writes the row + KERNEL-LLM-AGREE;
second call hits cache + STRICT-WITNESSED.
Tests
=====
- tests/test_canonical_cache.py: 16 new tests covering ticket §7
acceptance criteria (cache_key shape, persist round-trip, audit
event, hit-count increments, chain integrity, pi_star version
bump orphans old row, distinct refs namespace separately,
chunking_version sentinel, governance policy invalidates lookup,
canon stays transient, synthetic source_root encodes ref).
- tests/test_canonical_projection.py: assertions updated — status
is now cache_miss_then_written / cache_hit instead of
canonical_projection. Added a transient-mode test pinning the
policy gate.
- tests/test_witness.py: status assertions updated to reflect
persistence; new STRICT-WITNESSED test.
- tests/test_directives.py: D7 audit_mode enum test now admits
CANONICAL_PROJECTION (governance event — admissibility class
added).
Full suite: 1367 passed, 36 skipped (was 1306; +61 new).
Real-shard smoke
================
$ make query Q="0.1 + 0.2" BURN=1
→ cache_miss_then_written, ~300ms wall, row written
$ make query Q="0.1 + 0.2"
→ cache_hit, ~40ms wall, hit_count++
$ make chain-check-shards
→ 0 breaks per shard
