Followup to 654d923 (which moved the package from arborist/v8/ →
arborist/substrate/ at the file layer). The CLI surface still baked
in `v8` so a new operator running `--help` would see
``arborist v8 score`` and ask the same "what's v8 vs v9.8?"
naming-confusion question that drove the package rename in the
first place. Closing the loop end-to-end.
arborist/cli.py
===============
- Subparser renamed: ``"v8"`` → ``"substrate"``; help string updated
to "Merkle-AGI substrate primitives (ForkScore + future paper
specs)" so the dir name and command name and help text all align.
- Inner subparser dest renamed: ``v8_op`` → ``substrate_op``.
- Function renamed: ``_cmd_v8_score`` → ``_cmd_substrate_score``;
docstring updated.
- All ``v8_score`` local variables renamed to ``substrate_score``.
- New comment block above the subparser block explains the rename
+ why the v-prefix was retired (substrate-paper version vs v9.8
schema version naming collision).
The old ``arborist v8 score`` is gone — no alias preserved. CI + ops
scripts must update; today's earlier commit chain has been the only
place using it and that's been refreshed in lock-step.
tests/test_v8_fork_score.py
===========================
- 4 ``parser.parse_args(["v8", "score", ...])`` calls → ``["substrate", ...]``.
- 4 test functions renamed: ``test_cli_v8_score_*`` →
``test_cli_substrate_score_*``.
- Module docstring + section comment + helper docstring updated.
Filename intentionally kept as ``test_v8_fork_score.py`` for git
history continuity; pytest discovers by ``test_*`` content, not
filename. Renaming the file would muddle ``git log --follow`` for
the test surface.
Docs refreshed
==============
- docs/v8-fork-score.md — §5 CLI block invocation.
- docs/_source/v8-fork-score.rst — :code-block:: bash invocation.
- docs/_source/bench.rst — invocation in `### v8 ForkScore` section.
- docs/tickets/ticket-000012-selection-consensus-protocol.md —
three references in §7 close-out + §7 Phase 1c proposal +
§7 future-CLI-shape note.
- docs/dav1dprometheus-update-2026-05-09.md — bench journal mention.
Doc filenames (``v8-fork-score.{md,rst}``) kept stable since they
are URL identities; the file content explains the v8→substrate
rename internally. ``index.rst`` toctree references unchanged.
Hygiene
=======
- ``.venv/bin/arborist substrate score --help`` → 0 + valid usage.
- ``.venv/bin/arborist v8 score`` → exits non-zero (subcommand
removed, surfaced cleanly in ``argparse`` error).
- ``make test`` → 1643 passed, 45 skipped.
- ``make chain-check-shards`` → 0 across all 7 shards.
- fox's parallel work in arborist/qa/{runner,verify}.py +
arborist/qa/warrant_chain.py left untouched.
fox's read: the version-prefixed namespace pattern (`arborist/v7/`,
`arborist/v8/`) coupled module location to the substrate-paper
version. That collided with the live SQLite schema version (v9.8)
and made readers ask "is this dir tracking schema or paper?" —
a real onboarding hazard surfaced when the v7 dir landed earlier
today (06c95a0) for #000035 Phase 1.
Resolution: collapse v7+v8 into one topic-named dir,
``arborist/substrate/``, which holds Merkle-AGI substrate primitives
that future paper specs require — decoupled from the paper version.
Moves
=====
arborist/v7/anchor_prg.py → arborist/substrate/anchor_prg.py
arborist/v8/fork_score.py → arborist/substrate/fork_score.py
arborist/v8/weights.py → arborist/substrate/weights.py
Empty v7/ + v8/ dirs deleted; their __init__.py docstrings folded
into the new arborist/substrate/__init__.py with an explanation of
why the version-prefixed pattern was retired.
Imports updated
===============
- arborist/cli.py:_cmd_v8_score — arborist.v8 → arborist.substrate
- arborist/substrate/fork_score.py — internal weights import
- tests/test_anchor_prg.py — module + module-docstring
- tests/test_v8_fork_score.py — three import lines
Docs updated
============
- docs/v8-fork-score.md — header note explaining the move
- docs/_source/v8-fork-score.rst — :class: ref updated
- docs/tickets/ticket-000012-selection-consensus-protocol.md — §7
Phase 1a close-out paths refreshed (kept "Originally landed at
arborist/v8/..." parenthetical so the historical record survives);
§7 Phase 1b consensus-paper reference; §7 Phase 1c proposal §3
read-API path
- docs/tickets/ticket-000035-prg-choice-phi-prg.md — §7 Phase 1
close-out path refreshed (with full path-note explaining the
move); §3.1 + §5 left as the original design log per CLAUDE.md
"closed tickets stay in place as design log"
Left untouched
==============
- arborist/world/ — already topic-named; not version-prefixed; the
v7-W reservation lives there with its own planned subdir layout.
- docs/tickets/ticket-000037-prometheus-sigma-...md §13 still refs
``arborist/v9/prometheus.py`` and ``arborist/v8/fork_score.py`` —
fox has 792 lines of in-flight modifications on this file; those
refs should refresh to ``arborist/substrate/`` when the in-flight
edit lands. Avoiding interleaved edits.
Hygiene
=======
- make test → 1643 passed, 45 skipped (was 1643; refactor preserved)
- make chain-check-shards → 0 across all 7 shards
- arborist.substrate namespace picked up by the existing
pyproject.toml ``include = ["arborist*"]`` glob; no setup change.
Internal architecture reference written 2026-05-10 after the day's
18/92 -> 92/92 push under #000031. Covers:
- per-pillar substrate map (which open textbook covers which
pillar; license + ingest path for each of the 18 substrates)
- five proven ingest patterns (HTML single-URL, HTML BFS,
textbook_tex LaTeX-source, PDF -> localhost-HTML, direct
Python API)
- discipline patterns: title-from-author backfill workaround
(until the source-side fix in #000031 Phase 1 lands), alias
audit-fail-closed (decision_by + decision_rationale per row),
multi-substitute pattern, cascade tuning
- honest tier breakdown of the 92 chains (~25 direct primary,
~50 substrate substitution, ~17 soft-fallback OR-of-3 match)
- what the substrate doesn't yet do (render layer doesn't read
derivations, process_id under-attributes alias chains as DIRECT,
no per-record tier classification in the schema)
- re-running steps for future shifts (idempotent at DB layer)
Format follows other docs/ references (cti-architecture,
concept-relations-design, tool-action-dag-design) — describes
state of the world, not proposing change.
CLAUDE.md and TICKETS.md updated to point at the cookbook from
the docs index.
No undefect/whitepaper publication — this stays internal as
requested. Future blackops shifts re-discovering the substrate
map shouldn't have to walk five bench journals.
Keller-Trotter Applied Combinatorics (CC-BY-SA-4.0) re-crawled
successfully on the 38-min retry — earlier 25-min timeout was
too tight against appliedcombinatorics.org's 20-second crawl-delay.
Result: 80 docs / 168 chunks of advanced enumerative combinatorics
(generating-functions, partitions, Polya enumeration) that Bogart
skips and Levin treats lightly.
Title-backfilled the 79 newly-ingested docs with author surname
'Keller and William T. Trotter' for resolver _shard_matches_citation
heuristic.
Registered 14 KT alias rows for pillar VII compound source_refs as
a third substitute alongside Bogart + Levin. Audit-disciplined
(decision_by="fox 2026-05-10", decision_rationale notes Georgia
Tech AIM-approved provenance + advanced-chapter coverage).
Total citation_aliases now 54 rows (was 40); coverage stays at
92/92 (100%) — pillar VII chains gain peer-level redundancy
without changing the per-record resolution status.
Cleaned up orphan crawl_appliedcombinatorics_org.db from the
earlier failed crawl (no schema; tripped the resolver's
_shard_title_haystack with a missing-table OperationalError).
Doc updates (TICKETS.md, #000031, #000041) reflect 54-row count
and KT as third pillar-VII substrate.
Lands the M1 mitigation cryptographic primitive that ticket #000018
§5.2 + §9.2 specified, scoped per ticket #000035 §3.1-§3.3. Pure
stdlib (hashlib + hmac); no third-party dependency.
arborist/v7/__init__.py
=======================
First module landed under the v7 namespace. v7 plastic-training is
currently paper-stage (per #000037 §17.2); this is where its
deterministic primitives accumulate ahead of an active deployment
target so the building blocks are unit-tested + KAT-pinned the
moment v7 needs them.
arborist/v7/anchor_prg.py
=========================
Implements ``phi_prg(hard_hash_32, dim_h, *, seed) -> list[float]``
per #000035 §3.1. Construction is SP 800-108 KDF in counter mode
over HMAC-SHA-512:
Output(SEED, C(M), n_bytes) :=
i = 0
out = b""
while len(out) < n_bytes:
out += HMAC-SHA-512(SEED, C(M) || i.to_bytes(4, 'big'))
i += 1
return out[:n_bytes]
Float conversion: f(u32) := 2 * (u32 / 2**32) - 1
↑ uniform on [-1, 1)
Security argument from #000035 §2.1: HMAC-SHA-512 is a PRF under
the standard SHA-512 + HMAC assumption; distinguishing advantage
from random bounded by SHA-512 collision-resistance (~2^256), which
structurally matches the substrate's SHA-256 hard-hash family. The
seed is published, not secret — secrecy is not the security
property; the property is computational indistinguishability of the
output from random, which holds even when the seed is public.
Module exports ``PHI_PRG_VERSION = "phi-prg-v1-hmac-sha512"`` so
future algorithm rotations are detectable at the call site without
string-comparing module paths. Per #000035 risk §6.1, a
``phi_prg_version`` field in the v7 manifest will let future
deployments swap to a successor PRF without breaking historical
replay; this version string is the runtime-side mirror.
Hard-hash input length checked exactly at 32 bytes — silently
padding shorter input would break the PRF security argument.
``dim_h`` validated as positive int.
tests/test_anchor_prg.py
========================
20 tests covering #000035 §3.2 acceptance criteria + the strict
range invariant + the input-validation surface:
- Determinism: same (seed, hard_hash, dim_h) → identical floats.
- Range: every output in [-1, 1) with strict upper bound. Three
edge cases pinned: u32=0 → -1.0, u32=2^31 → 0.0,
u32=2^32-1 → just below 1.0.
- Chi² loose-uniformity: 4096-sample bin-test (df=15) with a
generous threshold (60); catches catastrophic PRG bugs (counter
cycling, mis-keyed HMAC) without claiming cryptographic-grade
evidence.
- Boundary: dim_h=1 + dim_h=16384 both produce sensible output.
- Avalanche, seed-bit: flip top bit of seed[0]; require 35-65% of
output bits flipped (PRF avalanche property).
- Avalanche, hash-bit: same surface for the hard-hash input.
- Validation rejects: short hashes, long hashes, non-bytes hashes,
zero / negative / non-int dim_h.
- Module export shape: PHI_PRG_VERSION + PLACEHOLDER_SEED.
- KAT regression: pinned vectors verified against
``bench/fixtures/phi-prg/known-answer-tests.jsonl``.
bench/fixtures/phi-prg/known-answer-tests.jsonl
================================================
10 KAT vectors generated against the placeholder seed + custom
seed/hash combinations; covers smoke (placeholder seed × small
dim_h), block boundaries (HMAC-SHA-512 blocks are 64 bytes, so
dim_h=16 is exactly one block, dim_h=17 is two blocks with
truncation), seed/hash one-bit-flip variants, and a 4096-element
stress sample.
Each row pins the SHA-256 of the raw byte stream (not the float
list) — that's the durable contract; switching from list[float] to
array.array('f', ...) or numpy arrays at the float layer would not
invalidate the fixture. Algorithm changes MUST bump
PHI_PRG_VERSION and create a new fixture file under
bench/fixtures/phi-prg/; old runs replay against old data per the
v7 spec replay discipline.
#000035 status flip
===================
docs/tickets/ticket-000035-prg-choice-phi-prg.md §7 updated from
"open · awaiting go/no-go" to "in progress · Phase 1 landed
2026-05-10". §7 now carries Phase 1 close-out details + Phase 2
gating criteria (active v7 deployment target + spec maintainer
review of §3.4 amendment text). The §3.4 v7 §9.10 amendment text
stays as the draft awaiting Phase 2 landing.
docs/TICKETS.md index row was already updated by fox in commit
ed470dc; my edit was idempotent.
Hygiene
=======
- make test → 1643 passed, 45 skipped (was 1623; +20 anchor_prg)
- make chain-check-shards → 0 breaks across all 7 shards
- arborist.v7 namespace picked up automatically by the existing
pyproject.toml [tool.setuptools.packages.find] include="arborist*"
glob; no setup change required.
- fox's in-flight #000037 ticket modifications + a parallel
#000031 ticket update left untouched.
Update #000031 ticket with the 2026-05-10 push details: 18 textbook
substrates ingested, 40 citation-alias rows + 13 term-alias rows,
cascade tuning in _build_record_query_cascade. Per-pillar end state
recorded (13/13 · 10/10 · 13/13 · 18/18 · 5/5 · 5/5 · 14/14 · 14/14).
Honest tier breakdown of the 92 chains added. Phase 1 follow-up
(source-side title-from-author backfill) and via_citation_alias
attribution fix surfaced as open follow-ups under the same ticket.
Update #000041 with full registry of 40 citation aliases now live
(distribution by substrate; per-pillar breakdown). Update #000042
with 13-row count + acknowledgement that Newton/arithmetic rows are
mostly unused (cascade picked different tokens).
Update CLAUDE.md with the 18 per-textbook make targets + the 100%
warrant-resolve milestone.
Update TICKETS.md index status for #000031, #000041, #000042.
No code change in this commit; documentation refresh only.
Broader stale-map sweep across docs/, bench/, scripts/, Makefile, and
top-level surfaced one hit: the witness-mode example sketch in §2.6
of ticket #000028 still showed a no-op cache_lookup closure with
"TODO: wire post-#000027" — even though the real implementation in
arborist/qa/query.py:2158-2172 wired the cache leg in commit e19aed8
(2026-05-09), the same commit that closed both #000027 and #000028.
Updated the sketch to match the actual implementation: the cache
lookup closes over the prior persisted row's answer bytes (encoded
with surrogatepass for UTF safety), and the comparison is non-
tautological because cached_row is the row picked up BEFORE we'd
write a new one in this same call.
Same drift pattern as the #000028 §8 follow-up refresh (6d20aeb)
and the #000010/§000021/#000040 sweep (e84f453) and the repair.py
TODOs (8980e64): pre-implementation design notes don't get refreshed
after the implementation lands. Each instance is one session-waste
saved.
After this, both surfaces are clean:
- arborist/ + tests/ stale TODOs: zero (8980e64)
- docs/ + bench/ + scripts/ + Makefile + top-level: zero (this commit)
The remaining matches are all genuine future-work markers (raw_html
cache in async_web_fetcher.py:2309) or false-positives (\\uXXXX
escape-pattern docs in mesh/wire.py and tests/test_cli_render.py).
Three threads landed in a single commit because they share the same
substrate (the #000037 §12 trigger probe shipped in c422216):
A — code-side stale-map sweep (arborist/qa/repair.py)
=====================================================
Walked inline TODO/FIXME/XXX markers across arborist/ + tests/. Five
hits: two false-positives (\\uXXXX in escape pattern docs), one
genuine deferral (raw_html cache in async_web_fetcher), and **two
stale TODOs in arborist/qa/repair.py** referring to "re-prompt
feedback path is future work" — even though `reprompt_repair` is
fully implemented (lines 140+ at the time of this commit), wired
through CLI `--repair-reprompts N` flag (cli.py:4574), and gated by
`policy["repair_max_reprompts"]`. Refreshed the docstring header
and the in-body comment to point at the actual function.
Same drift pattern as today's earlier ticket sweep (e84f453):
implementation lands, the TODO doesn't get refreshed, future readers
re-implement what's already there.
B — N-power follow-up to the §12 trigger probe
==============================================
c422216's first probe run reported trigger 2 (divergence variance)
at N=16, σ=0.5, ratio=0.8 — both threshold conditions would fire if
N reached the 30-sample N_min. To validate that the variance signal
holds at N≥30 (rather than vanishing on a wider sample), drove the
canonical-witness path 20 additional times via a new
bench/qa_questions_canonical_witness_npower.txt fixture
(10 arithmetic@v1 + 10 logic-kernel@v1 questions; all
canonicalize-then-LLM-witness without errors).
Result at N=37: trigger 2 fires. Ratio 0.575 > 0.5, abs σ 0.435 > 0.10.
Variance signal is real at the floor. Captured in
bench/results/prometheus-sigma-triggers-2026-05-10-npower.md with a
loud caveat at the top: this is N-power validation, NOT a measure of
real workload pressure. The §12 phase-1 go/no-go decision should
still come from natural workload N or operator-stated need.
The prior baseline (prometheus-sigma-triggers-2026-05-10.md) stays
in place as the workload-state-at-time-of-ticket-c422216 record;
this new -npower.md report is the statistical-power follow-up.
C — ForkScore Phase 1c proposal (#000012)
=========================================
#000037 §12 Trigger 1 ("ForkScore receives ≥4 candidate branches per
checkpoint") gates on multi-branch persistence — but #000012 Phase 1a
(landed 2026-05-08) is single-validator scoring of one fork at a
time, and Phase 1b (still open) is the consensus paper. Neither
persists branch-sets. So Trigger 1 structurally cannot fire today,
which the probe correctly reports as "no fork_score branch-set table
found".
Added Phase 1c to #000012 as a doc-only proposal (no code in this
commit):
- New `fork_score_branches` table (sibling to capital_ledger; does
NOT enter audit_events.event_hash preimage).
- Optional `--branch-set <ID>` flag on `arborist v8 score`.
- One read API: `branch_set_density(conn, branch_set_id)`.
- #000037 probe wires through the new function once it lands.
Not opened as its own ticket because operator pressure for it
hasn't surfaced naturally — gates on either #000012 Phase 1b
landing OR #000030 algebra/calc kernel expansion producing
competing-kernel branches an operator wants to compare. Captured
here so a future shift doesn't re-discover the gap.
Hygiene
=======
- make test → 1623 passed, 45 skipped
- make chain-check-shards → 0 across all 7 shards
- fox's in-flight #000037 ticket modifications left untouched
Pattern from #000028 §8.2/§8.3 (commit 6d20aeb): tickets carry
"deferred" markers in follow-up sections written from pre-
implementation design notes; code lands but the markers don't get
refreshed; future readers waste a session re-implementing what's
already there. Walked all 19 closed tickets carrying deferred-class
mentions and cross-checked each one against current code.
Three confirmed stale markers refreshed:
#000010 §12.2 — "First pass does NOT bind into run_dag_root —
deferred to ticket #000009 (Phase 5)". The metacog binding DID
land. Commit 111dda6 ("qa(#000009): §8 corrections — reject-path
DAG + nested CTI clauses", 2026-05-04) extended
arborist.qa.dag.build_preflight_node_payload with a question_state
clause carrying the metacog to_dict() payload. Today every cache
write threads question_state.to_dict() into the preflight node
(arborist/qa/query.py:3296-3297). Refreshed the §12.2 paragraph
to point at that commit + call site.
#000021 Phase 1b "(deferred)" — items "Synthesis, Syllogism,
Semiotics under 5S; Triangulate, Timing, Transitivity, Truth under
5T". Phase 1b was scope-split into per-sub-battery tickets:
#000023 (5S) and #000024 (5T), both closed 2026-05-08. The Closure
section in this ticket already reflects the resulting fixture
totals (108 5S + 154 5T) but the Phase 1b sub-section still said
"deferred". Refreshed to point at #000023 / #000024.
#000040 §6 — "Cascade landed 2026-05-09 (commit TBD)". TBD was a
placeholder while the commit was in flight; landed as b9e5bbd
("ticket #000040 Phase 5: phrase + content-token resolver
cascade"). Replaced TBD with the real hash.
Items confirmed genuinely deferred (no refresh needed) — recorded
here so the next sweep doesn't re-walk them: #000010 reference-
frame plumbing / SOFT_PREFLIGHT_HINT / metacog-trigger bench;
#000011 §13 bench plan; #000014 selfmodel_binding cache-key
folding (only docstring mentions; no actual gating wiring) /
selfmodel diff CLI / cross-shard reconciliation; #000015 / #000017
out-of-scope items; #000019 auto-canonicalizer / cross-language /
source-adapter methodology; #000020 stewardship-halt / conversion
tables / real-time instrumentation; #000023 multimodal execution.
Pure docs change; no code touched.
§8.1 — Witness-vs-modality terminology clarification (pure docs).
The ticket title and §1 throughout use "modality" — operationally
meaning kernel / cache / LLM. The 2026-05-09 review (response_ticket-
000027-canonical-projections-in-providence-cache.txt) flagged that a
future reader looking for cross-carrier work (text↔image, audio,
world-state) might land here by mistake, since "modality" in that
sense means a different thing.
Added a one-line note near §1 distinguishing *witness channel* (an
independent epistemic source) from *carrier* (text / image / audio /
world state), and pointing forward to a later ticket for cross-
carrier extensions once π* libraries support non-text carriers.
§8.2 + §8.3 — status refresh from "deferred" to "landed in 708aa45".
Both follow-ups already shipped in commit 708aa45 ("fan-out: warrant
ladder wiring · witness follow-ups · 5F Phase 1d", 2026-05-09 12:42
EDT) — pre-dating the e19aed8 close commit. The ticket file's §8.2
and §8.3 sections still carried the "deferred" labels because they
were written from the pre-implementation design notes; refreshing
both to point at the actual implementation in
arborist/qa/query.py (lines 2140-2155 for sample-rate gating;
2237-2285 for the capital_ledger sidecar) and the corresponding
test_witness.py coverage. Map-vs-territory hygiene: stale ticket
markers are how future readers waste a session re-implementing
something that's already there.
No code change in this commit. 33/33 tests in test_witness.py
continue to pass, all 7 shards chain-check clean.
Smoke-test alias rationale was published with #000042, but two of the
three trailing records ("non triviality", "side-angle-side") needed
their own per-concept aliases since Hilbert 1902's vocabulary differs
in more than one place. Map of registered aliases (geometry domain):
incidence -> connection (4 records lifted)
euclidean parallel postulate -> axiom of parallels (1)
non triviality -> space axiom (1)
side angle side -> included angle (1)
Hilbert pillar IV is now the first complete warrant-chain vertical:
18/18 records bind to a primary-source surface chunk via Merkle
inclusion proof. 11 direct + 7 via-alias derivations rows. Bench
journal under bench/results/hilbert-pillar-iv-fully-resolved-2026-05-09.md
records reproducibility steps + per-alias signal data so a future
shift can extend the same pattern to Boole 1854, Newton 1729 Motte,
and Aristotle's traditional translations once #000038 lands those
surfaces.
Cluster-wide remains 18/92 (20%); the other 74 records cite textbooks
not yet ingested. Phase 4 content acquisition (#000038) gates next
lift.
Implements both alias mechanisms (citation-aliases #000041,
term-aliases #000042) as one cohesive layer. Same audit
discipline; same opt-in via --use-aliases on warrant-resolve;
same distinct process_id "warrant-resolver-v1+alias" on alias-
resolved derivations rows.
What landed
===========
arborist/store.py — two new tables under SCHEMA_SQL:
citation_aliases — substitute textbook for proprietary cite
term_aliases — bridge vocabulary mismatches (incidence ↔
connection in geometry, etc.)
Both with NOT NULL audit fields (decision_at + decision_by
+ optional decision_rationale); both with PK constraints
ensuring idempotent re-add.
arborist/qa/aliases.py — helper module:
add_citation_alias / list / lookup / remove
add_term_alias / list / lookup (bidirectional) / remove
expand_query_with_term_aliases — OR-rewrites FTS5 tokens
while preserving phrase syntax
domain_for_pillar — Roman numeral → domain string
arborist/qa/warrant_resolver.py — two-pass cascade:
Pass 1: unaliased queries (matches carry via_alias=False)
Pass 2: alias-expanded queries (only when pass 1 missed;
matches carry via_alias=True)
ResolutionMatch grew a via_alias field; warrant_resolve
uses it to pick the right process_id per derivation row.
iter_claim_pack_records now yields a 6-tuple including the
pillar (parsed from doc URI) so domain lookup works.
arborist/cli.py — alias subcommand group:
arborist alias citation add ORIGINAL --substitute SUB --by FOX [...]
arborist alias citation list / remove
arborist alias term add TERM ALT --domain D --by FOX [...]
arborist alias term list / remove
warrant-resolve --use-aliases flag
sweep --target warrants --use-aliases flag
All audit fields fail-closed at the API surface (refuses on
empty --by; ValueError raised at the helper level).
End-to-end smoke test
=====================
Registered the Hilbert smoke-test alias:
arborist alias term add incidence connection \
--domain geometry \
--by "blackops 2026-05-09 (smoke test)" \
--rationale "Hilbert 1902 Townsend uses 'connection' for
what modern texts call 'incidence'"
Re-ran warrant-resolve --use-aliases --write:
records_total: 92
records_resolved: 15 (was 11 without aliases)
derivations_written: 15
Breakdown by process_id:
warrant-resolver-v1: 11 (original-citation matches)
warrant-resolver-v1+alias: 4 (alias-resolved Hilbert axioms)
The 4 alias-resolved records are exactly the Hilbert "Incidence"
axioms blocked by terminology mismatch in #000040 §6:
Axiom of Line Incidence
Axiom of Plane Incidence
Axiom of Point-Line Incidence
Axiom of Point-Plane Incidence
All four bound to chunk 56 in the Hilbert TeX surface — the
chapter discussing "axioms of connection" (Hilbert's original
1902 vocabulary). Audit trail correctly distinguishes
substituted chains from original ones.
Tests: 18 new in test_aliases.py covering add / list / lookup
/ remove / domain isolation / lowercase normalization /
bidirectional lookup / audit-discipline raises / query
expansion (basic + phrase-preserving + no-match passthrough +
unreachable-DB fallback). Full suite: 1623 passed / 28
skipped.
Tickets #000041 + #000042 closed. Operators can now add more
aliases via the CLI as fox makes decisions per #000038. The
alias mechanism is fail-closed by default — existing
warrant-resolve runs without --use-aliases continue to produce
the original 11/92 chains; --use-aliases opt-in adds the
substituted chains alongside without polluting the unsubsituted
audit trail.
Two related changes:
#000042 — Term-aliases table (vocabulary-mismatch bridge)
=========================================================
Sibling design to #000041 citation-aliases. Maps a (term,
domain) pair to an alternate term used in older / foreign /
pre-modern translations of the same concept. Triggered by the
Hilbert "incidence" vs "connection" gap surfaced in #000040
§6 — claim-pack records use modern post-1950s names, the 1902
Townsend Hilbert translation uses the original "Verknüpfung"
/ "connection".
Resolver wiring: each FTS5-query token gets OR-expanded with
its registered aliases when --use-aliases is set. So '"line
incidence"' becomes '"line incidence" OR "line connection"'
once fox approves the (incidence, connection, geometry) alias.
Decision audit: each row carries decision_at + decision_by +
decision_rationale. Same audit discipline as #000041; same
opt-in via --use-aliases; same distinct process_id
("warrant-resolver-v1+alias") on alias-resolved derivations
rows.
A single decision unlocks 7 stuck Hilbert pillar IV records:
arborist alias term add incidence connection \\
--domain geometry --by "fox YYYY-MM-DD" \\
--rationale "Hilbert 1902 Townsend uses 'connection' for
what modern texts call 'incidence'"
Implementation deferred until that decision triggers it.
`arborist sweep` CLI — Target B warrant-resolver fragment
=========================================================
Implements the schema-no-change increment of #000037 §3.1
Target B (documents that bypassed meta-cognition at ingest
time).
arborist sweep --shards-dir X --target warrants [--write]
`--target warrants` walks every claim-pack record and
re-runs warrant_resolve. Same code path as
`arborist warrant-resolve` but framed as the unconscious
sweep — operators can run on a cron / systemd timer; the
target name reserves namespace for the full bicameral sweep
landing later (canonical-projection probe, freshness probe,
document-content witness fan-out).
`--target all` reports "deferred" with an honest message:
the full sweep needs #000037 §12 phase trigger +
documents.last_swept_at schema migration.
Idempotent: re-running on the same shards is a no-op at the
DB layer (PK collision on (core_root, src_root, process_id)
in the existing derivations table). Operators can run the
sweep on a recurring schedule without row proliferation —
exactly the property #000037 §3.1 needs from the unconscious
sweep.
Test suite stays at 1605 passed / 28 skipped — pure CLI +
new ticket; no source-code changes outside cli.py.
Parallel-shift session drafted #000039 earlier today and updated
TICKETS.md with the index row, but the ticket file itself sat
untracked in working tree (same situation #000037 had until that
ticket landed in commit 178cc42).
Committing the file as-drafted by the original author so the
design log entry is intact. No content changes from this
session; the file is exactly what the parallel shift produced.
Per the design log convention in TICKETS.md ("Do not delete
tickets; they are the design log") — every opened ticket file
ships with its index row.
Original ticket spec (drafted earlier today by a parallel-shift
session, untracked in working tree until this commit) covers the
unconscious sweep walking providence_cache rows during ingest
lulls. Fox flagged a second backlog the controller has to drain:
ingested DOCUMENTS that bypassed meta-cognition at ingest time.
Today's ingest pipeline is intentionally fast: fetch →
canonicalize → chunk → Merkle commit → audit row → done. No
witness fan-out, no canonical-projection cross-check, no warrant
resolution. That speed is the right trade-off for getting bytes
into the queryable Merkle + FTS5 tree, but it leaves a
meta-cognition gap: the substrate has content it never *probed*.
New §3.1 spells out two sweep targets, both processed during
ingest lulls:
Target A — providence_cache rows older than τ_qa
Re-runs multi-modality witness fan-out (#000028) on cached
Q&A answers. Catches drift from Hermes updates / kernel
bumps / mesh-delivered new evidence. Default τ_qa: 7 days.
Target B — documents rows ingested without meta-cognition
Walks `documents` where last_swept_at is NULL or older than
τ_doc, runs cross-checks per document content:
- Canonical-projection probe (math/logic statements in body
vs. matching π* canonical answer)
- Warrant resolver re-run (newly-ingested textbook can
resolve previously-unresolved claim-pack records)
- Source-URL freshness probe (HEAD-only via existing
recrawl-check; mark stale/gone)
- Witness fan-out on document content (kernel+LLM+cache as
if doc were a Q&A answer citing itself)
State tracking: new column documents.last_swept_at, schema
migration deferred to §12 phase trigger so it doesn't bump the
cache_key invariant prematurely.
Why this matters: without target B, the substrate's "I have
this content in my Merkle tree" claim is honest at the byte
level but silent on whether the content was ever tested against
kernel ground truth. Sleep is when you do the tests you
deferred to keep the day moving.
Design ticket for the alias mechanism proposed in #000038
§3.2 option (b). When a cited textbook is proprietary or
unavailable, fox decides on a PD substitute (e.g.,
Hilbert-Ackermann 1928 instead of Mendelson 1997), and the
warrant resolver looks up the aliased citation alongside the
original.
Schema (`arborist citation_aliases`) carries
decision_at + decision_by + decision_rationale per row so
the substitution is auditable. Opt-in via `--use-aliases` on
warrant-resolve; alias-resolved chains carry distinct
process_id ("warrant-resolver-v1+alias") so audit can tell
substituted chains from original ones.
Recommend parking until first PD-substitute decision lands
in #000038. Today the only candidate is Hilbert-Ackermann
1928 → Mendelson + Enderton (~13 records), which is
substantial. Goldstein → Newton's Principia for ~5 pillar VI
records is also viable.
Sibling design to #000042 term-aliases (the other vocabulary-
mismatch follow-up surfaced by the Hilbert "incidence" vs
"connection" gap from #000040 §6).
Implementation deferred until trigger fires.
Implements the layered cascade strategy from #000040 §3.1
(originally drafted as #000039 — renumbered after collision
with parallel-shift's sqlite-vec ticket).
What landed
===========
arborist/qa/warrant_resolver.py:
- _phrase_for_axiom(theorem_name) — strips leading
categorical prefix ("Axiom of " / "Theorem " / "Principle ")
and trailing parenthetical, returns FTS5 phrase syntax
('"line incidence"', '"plane incidence"', '"side angle
side"', etc.) when the theorem name has 2+ tokens.
- _content_tokens(chunk_content, max_n=8) — extract
discriminating tokens from a claim-pack chunk's body. Drops
stopwords / generic theorem terms / common-English (small
hand-curated set). Requires count >= 2 to ditch typo /
LaTeX residue singletons. Sorts by length DESC then
first-position ASC.
- _build_record_query_cascade(c, theorem_name, content) —
returns ordered list of FTS5 queries to try:
1. Phrase from title
2. Content-tokens AND-joined
3. Existing discriminating-tokens AND-join (legacy)
4. Existing OR-fallback (legacy)
- resolve_chunks gains a `record_content` parameter; tries
each cascade query in order, first hit wins.
- iter_claim_pack_records yields a 5-tuple including content
so callers can thread it through.
Tests: 6 new unit tests for the cascade helpers (phrase
extraction, parenthetical stripping, single-token fallback,
content-token filtering, count-2 minimum, cascade ordering).
20 total in test_warrant_resolver.py. Full suite: 1603
passed / 28 skipped.
End-to-end honest result
========================
Re-running warrant-resolve on the existing shard cluster:
records_total=92, records_resolved=11 (unchanged from Phase 4).
The cascade is correct; the lift didn't materialize for
Hilbert pillar IV's 7 missing records because of TERMINOLOGY
MISMATCH, not query strategy:
- claim-pack records (g4 2025) use modern post-1950s names:
"Axiom of Line Incidence", "Group I: Axioms of Incidence".
- Hilbert's 1902 Townsend translation uses the original
"Verknüpfung" / "axioms of connection".
- Empirically: the literal token "incidence" appears ZERO
times in the ingested Hilbert TeX surface; "connection"
is the relevant synonym.
No matter how clever the query, you can't find a word that
isn't there. The cascade is preserved for any future textbook
where cited vocabulary matches textbook prose (modern
Stanley / Brualdi / Knuth, etc.).
Next-link follow-up: file #000042 term-aliases table
(("incidence", "geometry") → ("connection", "geometry")).
Sibling design to the citation-alias proposal at #000041.
Renumbering note: the Phase 5 ticket file was renumbered
000039 → 000040 mid-session because parallel-shift took
000039 for sqlite-vec at nearly the same time. Internal
references in the file follow the post-rename numbering
(#000041 = citation-alias, #000042 = term-alias).
Substrate-derived priority list of cited textbooks not yet
ingested, with record-count impact per acquisition. Walks every
unresolved claim-pack record's parsed citation and counts which
authors + titles appear most frequently — that count IS the
prioritized roadmap.
Top targets (records resolved per acquisition):
Stanley Enumerative Combinatorics 11 pillar VII
Jech Set Theory 10 pillar II
Brualdi Introductory Combinatorics 9 pillar VII
Knuth TAOCP 9 pillar VII
Mendelson Intro to Math Logic 7 pillar I
Landau Foundations of Analysis 7 pillar III (PD-by-age original)
Barendregt Lambda Calculus 7 pillar IX
Enderton Math Intro to Logic 6 pillar I
Gödel On Formally Undecidable 6 pillar III (PD-by-age original)
Dummit + Foote Abstract Algebra 6 (algebra)
Kolmogorov Foundations of Prob. 5 pillar V (PD-by-age original)
Goldstein Classical Mechanics 5 pillar VI
Three buckets: PD originals (multilingual scope), proprietary
modern textbooks (per-textbook license decision matrix), and
already-ingested-but-resolver-misses (7 Hilbert records whose
discriminating tokens — "Incidence", "Plane", "Line" — are too
common in the text for BM25 to rank the right chunk).
Recommended path per textbook documented in §3.1: skip /
personal-copy ingest / PD substitute / negotiate-redistribution.
Hilbert-Ackermann 1928 noted as PD substitute for Mendelson +
Enderton; Newton's Principia (already ingested) as substitute
for Goldstein.
Citation-aliases mechanism proposed in §3.2 — `arborist
citation_aliases` table mapping original cite → substitute,
read at warrant-resolve time. Cleaner than re-authoring claim-
pack bundles; original citations stay intact.
Hilbert resolver-miss (§6) flagged as a separate Phase 5
follow-up — fix candidates: TF-IDF over BM25, exact-axiom-name
phrase boost, claim-content-as-FTS-query.
The substrate writes its own roadmap.
A `make docs-api-clean && make docs-api` cold rebuild now succeeds
with zero WARNING/ERROR lines (was 39).
Docstring fixes (RST hygiene — no semantic change):
- arborist/qa/{keys,runner,query,verify,quantifier,metacognition,dag,
evidence}.py — add blank lines around indented blocks, convert
ad-hoc indented sections to literal blocks (`::`), avoid line-broken
inline literals (e.g. UNKNOWN_EVIDENCE_ID), and replace nested
bracket/quote literals with cleaner wording.
- arborist/concepts/__init__.py — wrap function-signature listing in
a literal block so bare `*` (kwarg marker) doesn't trip docutils.
- arborist/store.py — blank line before bullet lists in module +
connect docstrings.
- arborist/evict.py — replace ad-hoc `{ ... }` enum block with prose.
Surface fixes:
- docs/_source/_ext/makefile_targets.py — escape `*` in auto-
generated Makefile target descriptions (covers `*-parallel`,
`*.dot`, `*.db`, `π*`, etc.) so the generator emits clean RST.
- docs/_source/index.rst, concepts.rst — extend title underlines
to match title length.
- docs/_source/concepts.rst, v8-fork-score.rst — widen first column
of grid tables so cells no longer overflow into the column margin.
- docs/_source/merkle-agi-v7w-spatial-temporal.rst — switch
pseudocode JSON block from `code-block:: json` to `text` (the
`<int32 x 3>` placeholders aren't valid JSON tokens).
Verification:
- make docs-api-clean && make docs-api → build succeeded, 0 warnings
- make test → 1588 passed, 28 skipped
- make chain-check-shards → 0 breaks across all 7 shards
- import-time SyntaxWarning escalation on edited modules → clean
Closes the warrant-promotion data path: claim-pack records now
bind to surface-ingested textbook chunks via Merkle inclusion
proofs in the existing `derivations` table.
What landed
===========
arborist/qa/warrant_resolver.py — four pure-data steps + one DB
write:
1. parse_citation(s) — regex pipeline turning the claim-pack
`source_reference` string into structured Citation tuples.
Handles "Title by Author" (single + Oxford-comma multi +
et-al), semicolon-separated multi-cite ("Knuth §1.2.6;
Stanley §1.2; Brualdi §3.5"), and compact author-year
("Pascal 1654") forms.
2. resolve_chunks(c, shards_dir) — FTS5 search across sibling
crawl/ dir's textbook-surface shards. Skips the main numbered
shards (Wikipedia content; would be false positives). Per-
shard match filter requires BOTH author last name AND a title
token in the shard's title-haystack — honest "no match" for
textbooks not yet surface-ingested.
3. compute_proof(shard, doc_root, chunk_id) — reads
merkle_nodes, walks layer-by-layer to assemble siblings;
emits deterministic JSON proof_blob compatible with
arborist/merkle.py verification.
4. write_derivation(...) — INSERT OR IGNORE into the existing
derivations table with process_id="warrant-resolver-v1".
Idempotent at the database layer.
CLI surface
===========
- `arborist warrant-status --shards-dir ...` (read-only) —
emits per-record JSON: parsed citations, FTS5 candidates,
whether a derivations row exists.
- `arborist warrant-resolve --shards-dir ... [--write]` —
default dry-run summary; --write actually computes proofs
and inserts rows.
End-to-end verification
=======================
Real-shard run: `arborist warrant-resolve --shards-dir
~/.arborist/shards --write` →
records_total: 92
records_resolved: 18
derivations_written: 18
All 18 are pillar-IV Hilbert axioms citing "The Foundations of
Geometry by David Hilbert" — the only cited textbook fully
surface-ingested by Phase 1. The remaining 74 records cite
textbooks not in our shard cluster (Mendelson, Enderton,
Jech, Goldstein, Barendregt, Stanley, Brualdi, Knuth, …) and
correctly produce 0 matches; they stay at ANCHOR-WARRANTED
until those textbooks land via future Phase-1 manifest
expansions.
Re-running the writer is a no-op (PK collision on (core_root,
src_root, process_id) = INSERT OR IGNORE).
Drive-by fix
============
arborist/sources/textbook_tex.py — _extract_title now also
parses PG's plain-text `Author:` line and appends "by Author"
to the title, so the warrant resolver's author-last-name match
works against PG-ingested textbooks (Hilbert "The Foundations
of Geometry by David Hilbert" instead of just "The Foundations
of Geometry").
Test suite
==========
tests/test_warrant_resolver.py — 14 unit tests for the citation
parser (no DB / network). Full suite: 1588 passed / 28 skipped.
Phase 3 (verifier wiring)
=========================
NOT in this commit. The data substrate is in place; the
audit_mode upgrade path that lifts answers citing
claim-pack-records-with-derivations from ANCHOR-WARRANTED to
EVIDENCE-WARRANTED requires a verifier change — touches well-
tested code, worth its own ticket so the regression risk is
bounded.
Two small Sphinx hygiene fixes surfaced while fox was building the
docs locally:
1. docs/_source/merkle-agi-v7w-spatial-temporal.rst was authored
earlier this session (#000013) but never added to a toctree
so Sphinx flagged it as orphan. Slotted under "Substrate"
alongside pi-star / bench / v8-fork-score where it belongs.
2. The same file had |translation_max| as raw text in an
ε-bound expression; RST parsed the pipes as a substitution
reference and errored. Wrapped the expression in double
backticks so it renders as literal math.
Build now drops from 40 warnings (counting docstring noise) to
5; the 5 remaining are pre-existing module-docstring formatting
in arborist/qa/*.py that pre-date this work.
Reviewer-flagged errata in the dav1dprometheus comm doc surfaced
two real drifts in canonical docs that needed correction:
docs/_source/pi-star.rst:
- "Fifteen concrete π*'s" -> "Sixteen". The table was missing
combinatorics@v1 (#000032). Authoritative count comes from
arborist.pi_star.registry.REGISTRY itself, with a note saying
so. Each entry called out as behaviorally immutable, with
forward link to docs/spec-methodology.md section 1.1 where the
versioning rule is canonical.
docs/dav1dprometheus-update-2026-05-09.md:
- Reverts a regression introduced in the previous errata pass
(455fc10). The reviewer counted 5 axes x 5 = 20 sub-batteries,
but 5T carries 6 (legacy 'transfer' from SQD-whitepaper plus
the canonical Dav1DPrometheus five, kept side-by-side per
ticket #000024). Total is 21, not 20. Top-of-doc revision
note records the correction, body section restores the 21
count with the explicit 5+6+5+5 explanation.
Other reviewer points are already canonical (kernel-version
immutability is in docs/spec-methodology.md section 1.1) or are
editorial-only and don't require canonical-doc changes.
Tested:
- len(REGISTRY) == 16 (verified live)
- bench/batteries/runner.py enumerates 21 sub-batteries
(5+6+5+5 per file naming under bench/fixtures/5{s,t,f,r}/)
External reviewer surfaced several errata in the 2026-05-09 draft.
Corrections applied in-place; original draft is preserved at
commit a2ff9d4.
- sub-battery count: 21 -> 20 (5 axes x 5)
- 5S vocabulary fix:
Surface/Substrate/Synthesis/Semantics/Semiotics
-> Syntax/Semantics/Syllogism/Synthesis/Semiotics
(matches bench/batteries/b_5s.py)
- 5R vocabulary fix:
React/Recall/Reason/Refine/Restore (reviewer's guess; doc's
original was even less correct)
-> React/Rearrange/Restore/Replicate/Resonate
(matches bench/batteries/b_5r.py per SQD section 9.3)
- pi* kernel chronology: explicit "15 -> 16 with combinatorics@v1"
- kernel-version immutability: stated as hard invariant (no
in-place behavioral mutation; behavior change = new version)
- cross-witness vs cross-carrier: distinguished. Witness channels
are kernel/cache/LLM; carrier modalities are
text/code/arithmetic/etc. The pattern generalizes from the
former to the latter but the audit semantics are distinct.
- STRICT-WITNESSED is a render label, not a new audit_mode.
Persisted column stays CANONICAL_PROJECTION; witness audit
event layers on top so cache_key semantics don't drift.
- warrant-promotion ladder: SOURCE-ANCHORED tier introduced
between ANCHOR-WARRANTED (assertion-only) and
EVIDENCE-WARRANTED (chunk-proof). Today's surface-ingest
work delivers SOURCE-ANCHORED; chunk-resolution layer is
needed for EVIDENCE-WARRANTED.
- license tags marked project-reported (manifest self-attest;
not independently audited by counsel)
- "no human labelers" scoped to canonical-shape divergence
labels only; claim-pack validity + textbook warrant
promotion still benefit from human curation
- top-of-doc revision note records the errata pass
- closing footer updated to reflect today's CI re-enable +
shard-search fan-out (66s -> 15.5s on the 4-shard cluster)
Durable copy of /tmp/dav1dprometheus-arborist-update.md into
docs/ so it ships with the repo. Captures eight days of
substrate work landing on the 5S/5F/5T/5R framework:
- 21 sub-batteries × 662+ default fixtures
- 15 π* canonical-projection kernels (registry closed)
- canonical-projection persistence + multi-witness pipeline
- claim-pack corpus across 7 pillars (I-VII + IX), 92 records
- surface-ingest layer for cited textbooks (~351 docs, 1597
chunks across 6/7 g4 pillars)
- witness-sweep cron flowing (5/8 KERNEL-LLM-DIVERGED on live
Hermes; auto-extracted to 5F-Falsification fixtures)
- #000013 v7-W substrate paper closed
- #000016 ZK Phase-2 parked with bench-plan + wire-protocol
- #000018 soft-hash covert-channel analysis closed; three
follow-up tickets opened (#000034 / #000035 / #000036)
- 1,684 passing, 37 sympy-skipped
Same naming pattern as docs/qa-modes-bench-2026-04-30.md so
future point-in-time reports follow the precedent.
Rolling research log gains a 2026-05-09 amend for the
multi-witness pipeline (#000028) running against live Hermes for
the first time. Qualitatively different from the random-word
triangulation amends above — those measure honesty under no
ground truth; this measures agreement under available ground
truth (kernel IS the ground truth on canonical-shape questions).
First sweep: 8 canonical-shape questions, 5 LLM-DIVERGED, 3
STRICT-WITNESSED. Three distinct failure shapes captured:
1. Wrong arithmetic on float-shape input
0.1 + 0.2 → kernel 3/10 vs Hermes 1/10
(off by 2/10; possibly trained on the IEEE-754 trap as
the "answer" itself rather than recognizing the kernel
returns the exact rational)
2. Implication-tautology error
A IMPL B → kernel (NOT A OR B) vs Hermes TRUE
(NOT B) IMPL (NOT A) → same divergence
(Hermes treats the formula as a tautology rather than
canonicalizing to CNF; A OR NOT A — the genuine tautology
— correctly returns TRUE)
3. Symbolic erasure on algebra-shape input
(x+1)**2 → kernel x²+2x+1 vs Hermes 1
x**2 + 2*x + 1 → same divergence
(Hermes collapses to a constant — possibly evaluating at
x=0 — instead of returning the canonical expanded
polynomial)
The amend frames witness-sweep as the *capability-failure*
signal source, distinct from random-word emergent's
*honesty-failure* signal. Different signals, different repair
paths:
- Verifier ladder hardening / warrant-tier sharpening
addresses honesty failures (false STRICTs).
- Prompt engineering / fine-tune-on-divergence-corpus
addresses capability failures.
The verifier ladder cannot help with witness-divergence — the
LLM fundamentally produced a wrong answer that no number of
citation checks recovers. This is the calibration-data stream
the original #000028 ticket imagined.
End-to-end validates: canonical persistence (#000027),
multi-witness fan-out (#000028), audit-event chain — 0 breaks on
chain-check-shards post-sweep, 5 providence_canonical_witness
events appended, capital ledger captured 5 canonical_witness
op_type rows.
Pattern documented for future witness-sweep amends as the
divergence corpus grows (cron + commit harness in
bench/scripts/witness_sweep_cron.sh handles the unattended
collection).
Bench artifact: bench/results/witness-sweep.json.
Calibration corpus: bench/fixtures/5f/falsification-witness-v1.jsonl
(10 rows after two extraction passes today).
Pillar VII bundle (axiomsclaude-vii-v1.json +
theoremsclaude-vii-v1.json) ingested via the existing claim_pack
source into ~/.arborist/shards/000.db. 14 records (7 axioms + 7
theorems): Addition / Multiplication / Pigeonhole Principles,
Factorial + Binomial Coefficient definitions, Pascal's Rule,
Empty-Set Boundary; Binomial Theorem, Inclusion-Exclusion
(counting form), Hockey-Stick, Vandermonde, Catalan Closed
Form, Stars and Bars, Strong Pigeonhole.
Combined with the v2 bundles (78 records across pillars I-VI +
IX), shard 000 now carries 92 claim_pack documents.
Retrieval lift verified on representative combinatorics queries:
- Pascal's rule → claim-pack record at #2
- pigeonhole → Strong Pigeonhole Principle at #2
- Modus Tollens → claim-pack record at #3
Authorship metadata: Claude blackops draft + cite-check against
Stanley / Brualdi / Wilf / Knuth (option C from #000033 §2.1).
Records cap at ANCHOR-WARRANTED on the four-rung ladder until
#000031 surface-ingests the cited textbooks and computes
derivations.proof_blob — that's the warrant-promotion track.
Drive-by Makefile fix
=====================
Crawl shards now land in $(CRAWL_SHARDS_DIR) ($(HOME)/.arborist/crawl)
by default, separate from $(SHARDS_DIR) ($(HOME)/.arborist/shards).
SQLite's max-attached-databases limit is 10; mixing 4 main shards
+ 6 crawl shards + qa.db + snapshots.db put us at 12 and broke
cross-shard queries. Crawl shards moved to a sibling dir; the
existing crawl_textbooks-stats target reads from both for backward
compat with already-placed shards.
Three artifacts landing per ticket §4.1 closure criterion:
1. docs/_source/merkle-agi-v7w-spatial-temporal.rst (658 lines)
============================================================
Substrate paper for the third commitment substrate — sister to v7
(logic / math) and arborist v9.8 (language / claim-lattice). v7-W
commits derived spatial-temporal world-state: objects, relations,
events, places, agent traces, observations. Six parts + appendix:
Part 1 — Introduction & motivation. The third-substrate gap;
why v7 § 11 multimodal composition isn't enough.
Part 2 — Substrate definition. Hierarchical-grid spatial
discretization (S2 / H3 / octree); frame as committed
object with explicit transforms; substrate-declared
clock (single-agent) + Lamport (multi-agent);
quantized centi-confidence (range opt-in); five
canonical tuple-classes (object / relation / event /
place / agent_trace) each with its own π*_w.
Part 3 — Theorems. T1-W (state binding), T2-W (causal
completeness), T3-W (frame-transform soundness),
T4-W (ε at affine frontiers).
Part 4 — Verifier kernels. Pose integration, observation
update (Kalman), object logits, relation logits.
Each affine after canonical projection.
Part 5 — Multimodal composition with v7. Where v7 ends, v7-W
begins; cumulative ε across substrates; frame-
transform anchoring.
Part 6 — Adversarial corners. Frame spoofing, time skew,
observation injection, privacy.
Appendix — Worked SLAM example with full ε budget.
Hard constraints honored: stays inside SQD A1-A3 (canonical
encoding, public quantization, collision-resistant hash); no new
axiom; every π*_w defined on quantized integer state, never on
continuous tensors.
2. docs/v7w-frontier-catalog.md (262 lines)
============================================
Operator-facing quick reference for the four ε-frontiers from
substrate-paper Part 4. Each entry:
- canonical input / output bytes
- operator (linear / bilinear / Kalman / SE(3))
- ε bound expression
- "affine after canonical projection" justification
- when to use
Reference table + cumulative-ε section so operators sizing
deployment grid choices can read off their ε_total under typical
agent-trace + scene-graph workloads.
3. arborist/world/__init__.py — namespace reservation
======================================================
Reserved ``arborist.world`` package. No kernels yet. Module
exports V7W_VERSION ('v0-draft') + STATUS ('namespace_reserved')
metadata. Package docstring lays out the future shape per
substrate-paper Part 4:
arborist/world/
├── pi_star/ — π*_w canonical projections (5 tuple classes)
├── frontier/ — ε-frontier kernels (4 frontiers)
├── frame.py — frame definitions + transforms
├── clock.py — wall-clock + Lamport
├── manifest.py — substrate manifest schema
└── adapters/ — sensor adapters land here, separate tickets
Implementation tickets cite the substrate paper and land kernels
one at a time; the stub exists so cross-referencing imports (mesh
peers, sibling repos) can pin the namespace before anything
implements it.
5 tests pin the reservation contract (test_world_namespace.py):
import succeeds, V7W_VERSION reports v0-draft, STATUS reads
namespace_reserved, __all__ exposes only metadata, substrate
paper + frontier catalog files exist alongside the namespace.
Closure criterion (#000013 §7): substrate paper lands and is
ready for review. Done. Status flipped to closed in the ticket
file + TICKETS.md index entry.
Test suite: 1641 passed, 37 skipped (was 1636; +5).
Three new tickets carve up the open questions from §9 of
docs/soft-hash-channel-analysis.md (#000018):
#000034 — Hessian alignment under φ_linear
============================================
Computational. Measure spectrum of W^T W (the v7 reference
embed_hard_to_vec frozen-seed projection matrix) vs typical
training-loss Hessian eigenvalue distributions on representative
checkpoints. Determines whether the linear projection has
structural alignment with low-eigenvalue directions, which would
let T2 adversaries amplify covert-channel steerage beyond the
random-oracle baseline established in #000018 §4.
Deliverable: bench/scripts/phi_alignment_probe.py + verdict
(STRUCTURAL_ALIGNMENT / NO_ALIGNMENT / ANTI_ALIGNED) per
representative checkpoint. Parks until a v7 reference checkpoint
is available; the analysis is empirical-only and useless without
representative training data.
#000035 — PRG choice for φ_PRG
================================
Cryptographic. Pin a specific PRG construction for the M1
mitigation (PRG-based anchor map) proposed in #000018 §5.2.
Recommended: HMAC-SHA-512(seed, digest ∥ counter) → uniform-random
floats in [-1, 1].
Reasons:
- Tightest dependency surface (stays in SHA family already
committed via SHA-256).
- NIST-approved PRF construction (SP 800-108 KDF in counter mode).
- Speed parity with AES at v7 cadence; PRG cost negligible.
- Provable security reduction: PRF security from SHA-512
collision-resistance, structurally stronger than SHA-256.
Deliverable: arborist/v7/anchor_prg.py + tests + known-answer-test
fixture + v7 § 9.10 amendment text. Lands when v7 plastic-training
has a deployment target.
#000036 — T3 per-window budget bound
=====================================
Formal. Derive an explicit closed-form upper bound on the covert-
channel capacity under threat model T3 (hyperparameter adversary)
when M2 (per-checkpoint nonce) is in place. #000018 §6 lists
"bounded by per-window budget" without giving the bound.
Three control bandwidths combine:
B1 — gradient bias (g · W · log₂ N_directions)
B2 — LR selection (⌈W/K⌉ · log₂ R)
B3 — batch order (Bottou-Bousquet conjecture: ≪ log₂ N!)
Deliverable: docs/soft-hash-channel-t3-bound.md (§§1-8 formal
derivation) + bench/scripts/t3_bound_calculator.py (operator-
parametrized closed-form). Operators plug in (g, K, R, W,
batch_count, gradient_norm) → read off bits/window residual; pick
nonce-window length to drive residual under their target.
Index updated; Next ID 000037. Cross-reference added to #000018's
status line so future readers see the three follow-ups in line.
Three streams. Two land in the repo; one lands in fox's Downloads
(existing g4 packs live there too).
Witness sweep automation
========================
`bench/scripts/witness_sweep_cron.sh` — schedulable harness that
runs `make bench-witness-sweep` + `make bench-witness-divergence`
back-to-back and (with --commit) auto-commits any new divergence
fixtures + sweep result. Fail-closed: pre-commit hook failure
aborts; non-main-branch refuses to commit.
$ bench/scripts/witness_sweep_cron.sh # dry run
$ bench/scripts/witness_sweep_cron.sh --commit # auto-commit
$ bench/scripts/witness_sweep_cron.sh --commit --push # full pipeline
Recommended cron (every 6 hours; LLM cost scales with frequency):
0 */6 * * * \
bench/scripts/witness_sweep_cron.sh --commit --branch main \
>> /var/log/arborist-witness.log 2>&1
Smoke this commit: dry-run produced 5 fresh divergences (10
cumulative across two extraction passes). The witness-sweep + 5F
extractor pipeline runs unattended now; calibration-data corpus
will grow over time.
#000018 — adversarial soft-hash covert-channel analysis
========================================================
`docs/soft-hash-channel-analysis.md` — formal threat model + channel-
capacity definition + cryptographic reduction + mitigation table
+ residual analysis. Doc-only deliverable per the ticket's hard
constraint.
Headline result: under threat models T1 (data poisoner) and T2
(gradient adversary), the channel is bounded by SHA-256 partial-
preimage hardness when φ (the anchor map) is independent of
parameter geometry. T3 (hyperparameter adversary) requires
M2 (per-checkpoint anchor nonce) for boundedness.
Recommended mitigation: M2 (per-checkpoint nonce) — 32 bytes added
to the snapshot manifest, no training-ergonomics regression,
closes T1+T2 channels under any φ choice and bounds T3 by
per-checkpoint window. Fall back to M1 (PRG-based φ) if M2-only
deployment surfaces structural concerns. M3 (drop anchor entirely)
stays in reserve as the strict-construction fallback.
Three open questions (§9): Hessian alignment under φ_linear,
PRG choice for φ_PRG, and explicit T3 per-window bound. Each is a
follow-up ticket.
Ticket #000018 status: closed · landed 2026-05-09 (analysis doc).
v7 § 9.10 spec amendment proposed in §7 of the analysis.
#000033 — pillar VII (combinatorics), Claude-authored
======================================================
NOT committed to the arborist repo (the existing g4-v2 packs live
in `/home/fox/Downloads/` too — that's the operator's bundle
location). Two new bundle files at:
/home/fox/Downloads/axiomsclaude-vii-v1.json (7 axioms)
/home/fox/Downloads/theoremsclaude-vii-v1.json (7 theorems)
Pillar VII covers combinatorial counting — the gap between Grok's
pillars VI and IX in the v2 packs:
axioms (7): addition principle · multiplication principle ·
pigeonhole principle · factorial definition ·
binomial coefficient definition · Pascal's rule ·
empty-set / boundary axiom
theorems (7): binomial theorem · inclusion-exclusion (counting
form) · hockey-stick identity · Vandermonde's
identity · Catalan number closed form · stars-and-
bars · strong pigeonhole
Each record in the dual-thread format the existing g4 packs use
(Δ symbolic LaTeX + ∇ verbose prose + ∇ concise + sigil + formal
language + role + status + source_reference + date + foundational
group + category + subfield). Per fox's directive: explicit
authorship metadata everywhere — `authored_by: Claude (Anthropic)
— model claude-opus-4-7`. NOT Grok-generated; no silent invention.
Each record carries `pi_star_ref: combinatorics@v1` so the kernel
binding is explicit. Theorems list `depends_on_axioms` arrays so
each theorem cites the foundation axioms it bottoms out on.
Smoke test (committed alongside):
$ arborist --db /tmp/test.db ingest --source claim_pack \\
--bundle /home/fox/Downloads/axiomsclaude-vii-v1.json \\
--bundle /home/fox/Downloads/theoremsclaude-vii-v1.json
→ 14 docs, 14 chunks, 0 cross-bundle edges
Source attributions: Stanley EC1, Brualdi Introductory
Combinatorics, Knuth TAOCP Vol 1, plus historical sources where
applicable (Pascal 1654, Vandermonde 1772, Dirichlet 1834, Catalan
1838, Feller 1950 for stars-and-bars).
Tests: 1636 passed, 37 skipped (no regressions; pillar VII
ingestion smoke covered above).
Three small streams:
#3 — close#000030 properly
============================
All 7 phases + Phase 1b landed across two commits (`04f3f5d`,
`abe5988`). Status header updated; ticket body now carries a phase
landing table with commit refs:
Phase 1 algebra-symbolic@v1 04f3f5d
Phase 1b algebra-symbolic-simplified@v1 04f3f5d
Phase 2 calculus-derivative@v1 04f3f5d
Phase 3 calculus-integral@v1 fox-direct
Phase 4 calculus-limit@v1 abe5988
Phase 5 calculus-series@v1 abe5988
Phase 6 linear-algebra@v1 abe5988
Phase 7 function-sampled@v1 abe5988
Plus tabular-pinned@v1 (last reserved stub) graduated in abe5988
closes the registry chapter — 15 concrete π*'s, no remaining
reserved stubs. Index updated.
#5 — composition fixtures across new SymPy π*'s
================================================
12 new tests in tests/test_pi_star_compositions.py covering pairs
that compose naturally:
- algebra-symbolic ∘ algebra-symbolic — idempotency check (running
expand twice equals expand once for any expression).
- algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean
identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`).
- Generic invariants: composition propagates PiStarError; manifest
fingerprint is order-sensitive; composite domain == inner domain;
composite bytes == manual chain bytes.
Test discipline: most compositions use `register_in_registry=False`
via a small `_safe_compose()` helper since the registry rejects
duplicate keys (#000015 invariant), so test ordering would
otherwise matter. Only the registration-test path uses real
compose().
#4 — end-to-end witness sweep against real shards + Hermes
===========================================================
New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape
questions (3 arithmetic + 3 logic + 2 algebra) through query() with
`canonical_witness_enabled=True`, against ~/.arborist/shards (real
shard cluster) + the actual Hermes endpoint (NOT StubClient).
Records the agreement matrix per question to
bench/results/witness-sweep.json.
`make bench-witness-sweep` Makefile target. Honors
`ARBORIST_SHARDS_DIR`.
First real sweep (this commit, against Hermes-3-8B):
agreement label count rate
KERNEL-LLM-DIVERGED 5 62.5%
KERNEL-LLM-AGREE 3 37.5%
───────────────────────────────────────────
divergence_count 5 62.5%
wall median / max 130 ms / 1.1 s
Hermes diverged on 5/8 of the canonical-shape questions:
- said `1/10` for `0.1 + 0.2` (kernel: `3/10`)
- said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`)
- said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already
expanded — but Hermes ALSO
emitted the unexpanded form
when given the expanded
form, vs the kernel's
deterministic expand)
- and 2 more.
These are real LLM hallucinations on questions with closed-form
ground truth — exactly the calibration-data stream #000028
imagined. Pipeline validated end-to-end against actual hardware.
Pair: `make bench-witness-divergence` then extracts the 5
divergences as 5F-Falsification fixtures
(bench/fixtures/5f/falsification-witness-v1.jsonl, also committed).
Re-running the extractor produces byte-equal output (idempotency
contract from the extractor work).
Tests
=====
Full suite: 1636 passed, 37 skipped (was 1624; +12 composition
tests). The witness-sweep + extractor produce real artifacts now
committed under bench/results/ and bench/fixtures/5f/.
A new π* kernel that canonicalizes pure-integer counting
expressions and FAILS CLOSED on any input whose result isn't a
non-negative sp.Integer. Tighter domain than algebra-symbolic@v1,
which already accepts the same input surface but happily returns
symbolic / negative / non-integer outputs.
Distinguishing feature versus algebra-symbolic@v1:
algebra-symbolic@v1: binomial(n, k) → "binomial(n, k)" (symbolic
passthrough)
combinatorics@v1: binomial(n, k) → PiStarError (fail-closed
on free-symbol output)
algebra-symbolic@v1: binomial(Rational(1,2), 3) → 1/16 (rational)
combinatorics@v1: binomial(Rational(1,2), 3) → PiStarError
(output not Integer)
Boundary kept explicit: binomial(-3, 2) = 6 IS accepted because the
output is an integer 6. The fail-closed rule is on output shape
(Integer ≥ 0), not input range. Documented as
test_generalized_binomial_negative_args_accepted_when_integer.
Output format: plain decimal literal (b"10", b"5040"). Composes
with arithmetic@v1 for byte-identical agreement with the rational
route (b"10/1") so the multi-modality witness (#000028) can pin
equivalence-class agreement when both routes fire on the same
question.
Allowed surface (via SymPy primitives): binomial, factorial, ff /
rf (falling/rising), catalan, bell, partition, stirling, plus
arithmetic compositions over those primitives
(3*binomial(5,2) + factorial(4) = 54).
Coverage:
- 43 unit tests including binomial symmetry C(n,k)=C(n,n-k),
Pascal's rule C(n,k)=C(n-1,k-1)+C(n-1,k), the C(n,k) =
factorial(n)/(factorial(k)·factorial(n-k)) identity,
fail-closed paths (symbolic/negative/non-integer/relational/
parse), round-trip idempotence, composition with arithmetic@v1.
- 10 syntax + 12 semantics bench fixtures, 100% pass.
- bench/batteries/base.py PHASE_1_CARRIERS gains "combinatorics".
- Makefile bench-5s-combinatorics target.
All gate on pytest.importorskip("sympy") so a sympy-less suite
stays green. Full make test: 1537 passed / 28 skipped.
Sequencing rationale honored: this kernel lands FIRST so that
#000033 (claim-pack pillar VII for combinatorics) can bind its
records to the tighter integer kernel from day one — avoids
rebind churn on pi_star_ref fields.
Three small streams in one commit:
#000028 follow-up — witness divergence → 5F fixtures
=====================================================
Witness fan-out now writes a `providence_canonical_witness` audit
event when it fires (next to the capital-ledger record landed in
708aa45). Body carries pi_star_ref, question_text, agreement_label,
canonical_answer_text, llm_raw_text, llm_canonical_bytes,
cache_status. Best-effort write — chain failure never fails the
query.
New extractor `bench/scripts/witness_to_5f.py` reads those events
from a qa.db and writes them out as 5F-Falsification fixtures
matching the existing `falsification-live-v1` schema. Filtering
includes only divergence labels (LLM-DIVERGED / KERNEL-LLM-DIVERGED
/ CACHE-DRIFT); skips KERNEL-LLM-AGREE / STRICT-WITNESSED (no
calibration signal) and KERNEL-ONLY (LLM unparseable, not a
supervised-correction sample).
Idempotent: sorted by audit-event seq, so re-running against the
same qa.db produces byte-equal fixture files. The existing
fixture-digest discipline stays valid.
Makefile: `make bench-witness-divergence` (override default
qa.db / output path via WITNESS_QA_DB / WITNESS_OUT env-vars).
Closes the divergence → calibration data loop the witness ticket
imagined: every LLM hallucination on a canonical-shape question
becomes a supervised-correction fixture downstream prompt
improvements can grade against.
#000030 Phase 7 demo — function-sampled@v1 end-to-end
======================================================
`bench/scripts/demo_plot.py` — closes the loop on opencompletion's
activity24-math-plot.yaml. SymPy expression → quantized
integer-vector signature (canonical bytes) → optional matplotlib
PNG. Canonical bytes are the proof; PNG is just a downstream view
of the same evidence.
$ make demo-plot Q='sin(x)' PNG=/tmp/sin.png
Output JSON contains canonical_bytes_sha256 + canonical_bytes_preview
+ canonical_bytes_total_chars + grid metadata + the optional png_path.
matplotlib is gated — when absent, --png prints a warning to stderr
and skips the render; the canonical bytes still print. Tests skip
the PNG-presence assertion via `pytest.importorskip("matplotlib")`.
Public docs polish (#7)
========================
- docs/_source/bench.rst: updated fixture-count narrative (~660 →
662 default tasks + ~110 math π* fixtures); `make` quick-reference
now lists all per-π* 5S targets (tabular, calculus-limit/series,
linear-algebra, function-sampled) plus bench-real-shard,
bench-fork-baseline/score, bench-witness-divergence.
- docs/_source/v8-fork-score.rst: CLI section gained --out flag
documentation + a Make-harness sub-section covering
bench-fork-baseline / bench-fork-score / FORK_PARENT/CHILD/REPORT
env-vars.
Tests
=====
- tests/test_witness_to_5f.py — 8 new tests covering the audit-event
write (3) + extractor logic (5).
- tests/test_demo_plot.py — 6 new tests covering canonical-bytes
determinism + equivalence-class collapse + matplotlib gating.
Full suite: 1624 passed, 37 skipped (was 1568; +56).
tabular-pinned@v1 + calculus-limit@v1 + calculus-series@v1 +
linear-algebra@v1 + function-sampled@v1 — all reserved stubs
graduated; the π* registry is now 15 concrete kernels with no
remaining reserved-stub entries.
#000030 Phase 4 — calculus-limit@v1
====================================
sp.limit with thread-timeout. One-sided dir support (+/-/+-).
Pinned spelling for infinity cases: b"+oo" / b"-oo" / b"zoo"
(complex infinity) — bypasses sp.expand since Infinity isn't
algebraic. Finite results re-canonicalize through algebra-symbolic
recipe (sp.expand + sp.srepr). Unevaluated cases / timeouts emit
b"unevaluated:" + sp.srepr(<Limit>) sentinel, mirroring
calculus-integral's pattern.
#000030 Phase 5 — calculus-series@v1
=====================================
sp.series(f, x, x0, n).removeO() → sp.expand → sp.srepr. Drops
O(x**n) remainder explicitly so the canonical form is finite-byte.
Sentinel format mirrors limit/integral: b"unevaluated:Series(...)"
on timeout. n must be a positive int; 0 / float / negative rejected.
#000030 Phase 6 — linear-algebra@v1
====================================
Single π* covers the whole linear-algebra surface via {op, matrix}
JSON. Ops: rref / det / eigenvalues / inverse. Matrix cells go
through Fraction(Decimal(str(...))) for floats so 1, 1.0, "1.0"
all collapse to Rational(1, 1) — matching arithmetic@v1's
discipline. Without this fold, sp.sympify keeps floats as Float
(separate type) and downstream det/inverse return Float-shaped
bytes. Eigenvalues are sorted by srepr for determinism.
Output formats:
rref / inverse: rows/cols header + cells joined by | (rows by ||)
det: det:<num/den-or-srepr>
eigenvalues: eigenvalues:<value-1>x<mult-1>|...
#000030 Phase 7 — function-sampled@v1
======================================
Bridge to time-series-quantized@v1. SymPy expression + linspace
grid → quantized integer-vector signature in time-series's exact
output format (dt=...;dv=...;n=...;t0=0:v0|v1|...). Two functions
that render identically (within sample-grid tolerance) collapse
to the same canonical bytes. This is what plotting CAN become
in π* terms — the PNG render is a downstream view of the same
canonical evidence.
Math-only sampler (no numpy in the dep surface); Python's round()
is banker's-rounding so the bytes are interchangeable with
time-series-quantized@v1's output. Complex / non-finite samples
raise PiStarError rather than silently dropping imaginary parts.
tabular-pinned@v1 — last reserved stub graduates
=================================================
JSON-rows input ({schema, key_columns, rows}); declared
key_columns sort policy (stable sort by primary-key tuple);
type-fold per column (int/rational/bool through arithmetic@v1
discipline; str verbatim; bool normalized). Header case is
PINNED EXACT — Excel and PostgreSQL both care about case;
defaulting to lowercase-fold would break operator expectations.
Output: header (schema + key + n) + rows joined by \n + cells by |.
The π* registry has no remaining reserved stubs. Every modality
the substrate paper reserved is now real.
Test suite: 1568 passed (was 1467; +101). New closure-criterion
test (test_no_stub_pi_stars_remain) replaces the old reserved-stub
parametrize — adding a future stub re-opens this list.
110/110 fixtures pass across the 5 new bench-5s-* targets.
PHASE_1_CARRIERS gained calculus / linear-algebra / function-sampled
/ tabular.
Two design-only tickets opened together because they're tightly
coupled — pillar VII records bind to combinatorics@v1 via
pi_star_ref, and #000032 lands first to avoid rebind churn on
that field.
#000032 — combinatorics@v1 π*
=============================
A new π* kernel that canonicalizes pure-integer counting
expressions (binomial, factorial, permutations, partitions,
Catalan, Bell, Stirling) and FAILS CLOSED on any input whose
result is not a non-negative integer. Tighter domain than
algebra-symbolic@v1, which already accepts the same input
surface but happily returns symbolic / negative / rational
outputs.
Distinguishing feature: algebra-symbolic@v1 returns
binomial(n,k) → "binomial(n,k)" (symbolic), binomial(-3,2) → 6
(generalized). combinatorics@v1 rejects both. Operators choose
the kernel by what they want rejected.
Output format: integer string (b"10"). Compose with
arithmetic@v1 to get bytes-identical agreement (b"10/1") for
the multi-modality witness flow.
Estimated size: ~120 LOC module + ~80 LOC tests + ~22 fixtures.
Single-commit feasible.
#000033 — Claim-pack pillar VII (combinatorics)
================================================
Extend the claim-pack source (#000029) with a new combinatorics
pillar slotting into the documented gap (existing v2 bundles use
I, II, III, IV, V, VI, IX — VII and VIII reserved for
extension). Counting axioms (Pascal's rule, addition principle,
multiplication principle, pigeonhole, factorial / binomial
definitions) + classical theorems (binomial theorem,
inclusion-exclusion in counting form, hockey-stick, Vandermonde,
Catalan closed form, stars-and-bars).
Bundle provenance is the open question — three options
documented:
A. Commission a Grok-4 v3 bundle for parity with the existing
pack.
B. Hand-curate from textbooks (Stanley, Brualdi, Wilf, Knuth).
C. Hybrid — LLM draft + human curation.
Hard constraint: explicit authorship metadata. No silent
invention. Bundle landing is a config + data exercise; no
source-code changes to arborist/sources/claim_pack.py needed
since the source already iterates arbitrary pillar names.
Sequencing: #000032 first (kernel), #000033 next (records bind
to it from day one), #000031 stays parallel-track (textbook
ingest for warrant promotion).
Both tickets stay open · awaiting go/no-go pending fox's
implementation green-light.
Phase 1b — algebra-symbolic-simplified@v1
==========================================
arborist/pi_star/algebra_symbolic_simplified.py — full-simplify
variant of the Phase-1 expand-only sibling. Closes the trig
identity gap left open at end of Phase 1: sin(x)**2 + cos(x)**2
now collapses to 1, tan(x)*cos(x) to sin(x), exp(log(x)) to x.
Recipe is sp.expand(sp.simplify(expr)) — the follow-up expand
after simplify is load-bearing. simplify alone is non-canonical
for polynomials: it leaves (x+1)**2 in factored form while
collapsing x**2 + 2*x + 1 to expanded form, so two algebraically
equivalent inputs would emit different bytes. Composing with
expand picks one canonical polynomial shape and preserves the
equivalence-class invariant.
Cost: 1-360 ms typical on common trig/exp inputs; pathological
inputs unbounded. No in-π* timeout (the calling pipeline owns
that budget). Operators opt in by registry key — the fast Phase-1
sibling stays the default for callers that only need polynomial
canonicalization.
22 unit tests; all gate on pytest.importorskip("sympy").
Phase 3 — calculus-integral@v1
==============================
arborist/pi_star/calculus_integral.py — symbolic integration with
thread-timeout fallback. JSON-shaped {f, x, limits?,
timeout_seconds?} input. Two output paths:
1. Closed form: sp.srepr(sp.expand(integrate_result)) — same
recipe as algebra-symbolic@v1 so the output is itself a valid
algebra-symbolic input and composes naturally.
2. Unevaluated: b"unevaluated:" + sp.srepr(<Integral>). Prefix
lets callers tell "no closed form" from "input invalid"
without re-parsing the canonical form.
Timeout discipline: ThreadPoolExecutor(max_workers=1) +
future.result(timeout=...). On TimeoutError, synthesize the same
unevaluated sentinel SymPy itself would emit, so timeout +
no-closed-form converge to the same bytes for the same input.
Default 30 s; per-call override via timeout_seconds. Python
threads can't be killed cleanly — a timed-out worker leaks until
SymPy returns. Documented as the cost of the discipline.
Coverage: ∫x dx = x²/2, ∫sin(x) dx = -cos(x), ∫_{0}^{π} sin(x)
dx = 2, ∫_{-∞}^{∞} exp(-x²) dx = √π, exp(x)/log(x) →
unevaluated sentinel. 31 unit tests including a monkeypatch
deterministic timeout test (sleep-mocked SymPy so the timeout
path doesn't depend on any specific input being slow on every
CI runner).
Open #000031 — surface-ingest cited textbooks
=============================================
Design-only ticket. Closes the warrant gap left open at the end
of #000029: today every claim-pack record caps at
ANCHOR-WARRANTED because source_reference is a string field, not
a Merkle-bound proof. Ingesting the cited textbooks as surfaces
+ computing per-claim derivations.proof_blob lets the four-rung
ladder promote them to EVIDENCE-WARRANTED.
License gating: PD sources (Hilbert, Newton, Kolmogorov,
Łukasiewicz, Aristotle) form the green-light scope. Mendelson +
Enderton are proprietary and stay yellow-light pending fox's
explicit decision (purchased single copy / library license / PD
substitute via Hilbert-Ackermann 1928).
Two follow-up tickets reserved: textbook-fetch pipeline +
chunk-resolution layer (mapping source_reference strings to
specific spans within ingested textbooks; the bridge that lets
proof_blob be computed).
Test counts: 153 tests for the work in this commit (algebra
+ algebra-simplified + calculus-derivative + calculus-integral
+ preflight). All pi_star + canonical_projection tests pass
under .venv pytest.
Three small streams in one commit; each closes / expands a
recently-landed ticket without changing its hard contract.
#000026 Phase 3 wiring — authorship warrant ladder visible
============================================================
Phase 3 sidecar (arborist/qa/warrant_authorship.py landed in 60b5748)
exposed the classifier but didn't surface it. Two wirings:
- arborist/qa/inspect.py — diagnose_authorship_warrant runs against
the cached row's question + answer + per-source raw chunks +
URIs + titles; result lands as `authorship` field alongside the
other sidecars.
- arborist/cli.py _render_warrant_tail — appends ` · warrant:
<readable-tier>` when result['authorship'] is populated with a
non-quiet tier. AUTHOR_COPYRIGHT_FOOTER → "copyright-footer", etc.
NO_AUTHORSHIP_SIGNAL stays silent. Backward-compat: results
without an `authorship` key render unchanged.
Tests: 3 inspect-path tests (no-signal, copyright-footer,
repository-owner) + 4 render-tail tests (presence, no-signal
silence, missing-key silence, all-six-tiers readable mapping).
#000028 follow-ups — capital ledger + sample-rate
==================================================
Two policy fields layered on top of canonical_witness_enabled:
- canonical_witness_sample_rate (0.0..1.0; default 1.0). Operators
wanting passive calibration set 0.05 to fire witness on 5% of
canonical questions while paying 5% of LLM cost. 0.0 effectively
off; 1.0 = current always-on behavior. Gating uses random.random()
so distribution is uniform; clamped to [0, 1].
- Capital ledger row written for each FIRED witness (not skipped
ones). op_type='canonical_witness'; estimator inputs include
prompt_chars + answer_chars + llm_seconds + agreement_label +
pi_star_ref. Best-effort: ledger-write failure must never fail
the query (sidecar discipline).
Tests: 4 new — sample_rate=0.0 skips (no LLM call, no ledger row);
sample_rate=1.0 always fires; capital_ledger row written under
op_type='canonical_witness' with full input blob; sampled-out
witness records zero ledger rows.
Both fields fold into governance_policy_hash naturally via the
existing policy-hash machinery — flipping witness mode invalidates
prior records as expected.
#000025 Phase 1d — 5F fixture catalog 30 → 50
==============================================
Both synthetic and live sides of all 5 sub-batteries expanded
30 → 50 (+200 fixtures total: 5 × 20 synthetic, 5 × 20 live).
function — claim_count cycles 2..7 across new fixtures
falsification — 10-violation palette across new ids
feedback-loop — fact-N learning chains
finetuning — capability transitions across canonical π*
(math/logic/algebra/calculus pool)
formulate — multi-pointer claim shapes
500/500 pass through respective runners. test_session_integration
total bumped 562 → 662. Pinned test_5f_*_runs counts updated 30 →
50 (synthetic main + embedded + live).
Tests
=====
Full suite: 1467 passed, 36 skipped (was 1388; +79 across warrant
render + witness sample/ledger + 5F implicit coverage).
Two new π* canonicalizers extend the math substrate above
arithmetic@v1 (closed-form rationals) and logic-kernel@v1
(propositional Boolean → CNF):
algebra-symbolic@v1 (Phase 1) — symbolic-algebra domain.
sp.expand → sp.srepr canonical bytes. Polynomial identity collapses
((x+1)**2 ≡ x**2 + 2*x + 1); exponential identity collapses
(exp(a+b) ≡ exp(a)*exp(b), inherited from sp.expand's default
behavior); trigonometric identity does NOT collapse
(sin²+cos² ≢ 1). The trig surface is reserved for a future
algebra-symbolic-simplified@v1 variant that wraps sp.simplify at
unbounded CPU cost. Rejects relationals (`x > 0`) and
BooleanFunction shapes (`x & y`) via `isinstance(expr, sp.Expr)` —
sp.Symbol confusingly inherits from Boolean so the right rejection
filter is "not Expr" rather than "Boolean".
calculus-derivative@v1 (Phase 2) — calculus domain. JSON-shaped
{f, x, n} input → sp.diff → sp.expand → srepr bytes. Output is
itself a valid algebra-symbolic@v1 input so the two compose
naturally under arborist.pi_star.compose. n defaults to 1; bools
explicitly rejected (Python isinstance(True, int) is True so we
filter that explicitly).
Optional dependency: sympy ships in the new [math] extra
(pyproject.toml). Folded into [dev] so make bootstrap pulls it
transitively. An explicit `bootstrap-math` Makefile target documents
the opt-in for minimal-install users. Both modules self-guard
via `try: import sympy as sp / except ImportError: sp = None` and
only register(...) when sympy is present, so a fresh checkout
without [math] still loads arborist.pi_star without raising.
Preflight algebra route lands in
arborist.qa.query._canonical_projection_preflight between the
arithmetic and logic routes. Charset regex (_CANONICAL_ALGEBRA_RE)
allows lowercase letters + math chars; requires at least one
letter (else arithmetic wins); rejects natural-language leading
verbs via _CANONICAL_ALGEBRA_NL_LEAD_RE (4-letter minimum so
single-/two-/three-char identifiers like x, xy, sin, cos, pi
survive while "simplify (...)", "factor x...", "expand (a+b)..."
fall through). PiStarError + KeyError both fall through cleanly
so a sympy-less install just routes everything past algebra.
Bench substrate:
- bench/batteries/base.py PHASE_1_CARRIERS gains "symbolic_algebra"
- bench/fixtures/5s/syntax-algebra-symbolic-v1.jsonl (10 fixtures)
- bench/fixtures/5s/semantics-algebra-symbolic-v1.jsonl (13 fixtures
including the documented trig non-collapse + exp collapse)
- Makefile bench-5s-algebra target → 100% pass
Tests: 18 algebra-symbolic + 38 calculus-derivative unit tests +
~10 new preflight-route tests in test_canonical_projection.py. All
gate on pytest.importorskip("sympy") so a sympy-less suite stays
green. Full suite: 1369 passed / 27 skipped.
Phases 3-7 (integral, limit, series, linear-algebra,
function-sampled) remain open as future work; each lands as its
own ticket when an actual consumer surfaces.
Brain-stormed off opencompletion's activity24-math-plot.yaml
(SymPy + numpy + matplotlib pipeline). Three things in there
map to π* shapes; one doesn't.
Maps:
- algebra-symbolic@v1 — symbolic expression → expand + canonical
ordering. Closes the (x+1)**2 ≡ x**2+2x+1
equivalence class.
- calculus-derivative@v1 — d/dx(f) via sp.diff, re-canonicalized
through algebra-symbolic@v1.
- calculus-integral@v1 — ∫f dx via sp.integrate; sentinel for
unevaluated cases.
- calculus-limit@v1, calculus-series@v1, linear-algebra@v1 —
additional SymPy-friendly phases.
- function-sampled@v1 — bridges symbolic expressions to the
existing time-series-quantized@v1
format. Two functions that render
identically (within sample tolerance)
collapse to same canonical bytes. This
is what plotting CAN become in π* terms.
Doesn't map:
- PNG plot rendering — different DPIs / fonts / palettes all
valid; image bytes aren't canonical. Stays as an output adapter
that COMPOSES with function-sampled@v1.
Hard constraint: SymPy is an OPTIONAL dependency ([math] extra).
Each new π* registers only when sympy is importable, mirroring
the existing html / wikitext / crawler pattern. Fresh checkout
without sympy keeps passing the full test suite (graceful skip).
Multi-phase rollout. Recommendation: Phase 1 (algebra-symbolic@v1)
+ Phase 2 (calculus-derivative@v1) in one commit (~400 LOC total
+ tests + 5S fixtures). Subsequent phases (integral, limit,
series, linalg, function-sampled) each their own commit.
Forward links in the ticket: #000027 (canonical persistence —
algebra/calculus answers inherit audit chain for free), #000028
(witness — symbolic answers become witnessable), 5T/5F (new
fixture surface for symbolic LLM calibration), composition algebra
(deriv+arith, expr+sample compose naturally).
Index updated. Next ID 000031.
ClaimPackSource ingests Grok-4 companion bundles (axiomsg4-v2.json +
theoremsg4-v2.json) at the right grain — one Document per axiom or
theorem record. Each record carries Δ (LaTeX symbolic) + ∇verbose
prose, explicit source citation (Mendelson, Enderton, Hilbert,
Newton, Kolmogorov, Łukasiewicz), foundational-group taxonomy, and a
runicLabel that rides as soft metadata only (runtime mints its own
pointer IDs per CTI architecture). Pillar-level
provenance.references arrays become outbound pillar_reference edges.
Lenient JSON parser strips ```json fences and double-escapes lone
LaTeX backslashes (\Theta, \heart, \vec) without corrupting
already-correct \\to pairs — walks left-to-right and pass-throughs
legal escape sequences. Malformed bundles raise rather than return
empty; silent zero-doc would be a footgun.
CLI surface: --source claim_pack with a repeatable --bundle FILE
flag mirroring html source's --url action=append. Single --path
also accepted for one-bundle ingest.
Drive-by: removed a function-local `from arborist.store import
connect` inside _cmd_ingest's providence branch that was shadowing
the module-level binding via Python's "any local assignment makes
the name local for the entire function" rule, breaking every
non-providence ingest with UnboundLocalError. Comment left in
place explaining why not to re-add it.
Smoke-tested on /home/fox/Downloads/{axiomsg4,theoremsg4}-v2.json
end-to-end: 78 docs (55 axioms + 23 theorems across 7 pillars),
14 deduped pillar-reference edges, 78 audit events, 10/10 sampled
Merkle proofs verify, FTS5 search returns Modus Tollens for
"modus tollens".
Honest ceiling: kind=surface for every record. The pack is
pre-distilled but its provenance is asserted not proven — until
Mendelson/Enderton/Hilbert texts are themselves ingested as
surfaces, the verifier has no derivations.proof_blob to compute
and claim-pack records max out at ANCHOR-WARRANTED on the
four-rung ladder. That's a follow-up ticket, not this one.
Hard constraints honored: no new audit ledger (audit_events
remains the only chained-sha256 ledger; bundle's self-validation
fields ride as metadata only); no kind=core without surface
ancestor; cache_key invariants untouched.
15 unit tests cover lenient parser, slug stability, ref
resolution, doc grain, URI stability, content layout, extra
metadata, edge emission, error paths. All 1280 tests in
make test pass.
Closes#000027. Closes#000028 (cache-leg wired).
#000027 — canonical projections persist to providence_cache
============================================================
Math/logic π* answers (arithmetic@v1, logic-kernel@v1,
time-series-quantized@v1, …) are now first-class providence rows.
Pre-fix: question → kernel → answer → return. No cache, no audit
event, no run_dag, no inspect/burn/replay surface.
Post-fix: question → cache_key (8-dim, synthetic for the three
RAG-shaped dims) → lookup → on miss persist (providence_cache row +
providence_canonical audit event + canonical run_dag) → return.
Synthetic cache_key dimensions for canonical rows (per ticket §2.2):
- source_root = sha256("pi_star_source:" + pi_star_ref)
- model_profile_hash = sha256("pi_star_model:" + pi_star_ref)
- conversation_hash = sha256("pi_star_conv:" + canonical_q + ":" + ref)
- chunking_version = literal "n/a-canonical" — chunker bumps on
wikipedia path don't stale math answers.
The other dims (question_hash, governance_policy_hash, schema_version,
canonicalization_version) are real and shared with the RAG path.
Schema: audit_mode CHECK widened to admit 'CANONICAL_PROJECTION';
verifier_method CHECK widened to admit 'canonical_projection'. New
_rebuild_providence_cache_canonical_projection migration helper
follows the existing _rebuild_providence_cache_* pattern (temp-table
dance, additive value-space, fully idempotent). Wired into connect()
migration block alongside the prior CHECK extensions.
Cache-hit policy: trust the row. Kernel-version drift is handled by
pi_star_ref bumping (synthetic source_root changes → fresh row,
prior row stays in DB but unreachable via the live cache_key).
Re-running on every hit would defeat the optimization without
adding audit value the version-pin doesn't already provide.
Policy gate: canonical_projection_preflight_persist (default True).
Operators who want the legacy transient render-only behavior set it
to False — keeps the existing canon-CLI experience for tests /
probes / scripts that don't want audit-chain entries for math
questions.
CLI render: `CANONICAL · via canonical_projection` for persisted
rows. Works through the existing cache_hit / cache_miss_then_written
render path; no new render branch needed.
`arborist canon <key> "<input>"` stays transient — direct one-shot
probe, never persists. Boundary preserved per ticket §2.6.
#000028 — multi-modality witness cache-leg
==========================================
Pre-#000027 the witness cache-leg closure always returned None;
STRICT-WITNESSED (3-of-3 byte-equal) was structurally unreachable.
Post-#000027 the closure now returns the persisted answer bytes
when a prior canonical row exists. Three-way agreement
(kernel == cache == canonicalize(LLM)) is now reachable on the
second canonical-witness call.
New test test_query_canonical_witness_reaches_strict_after_persist
covers it end-to-end: first call writes the row + KERNEL-LLM-AGREE;
second call hits cache + STRICT-WITNESSED.
Tests
=====
- tests/test_canonical_cache.py: 16 new tests covering ticket §7
acceptance criteria (cache_key shape, persist round-trip, audit
event, hit-count increments, chain integrity, pi_star version
bump orphans old row, distinct refs namespace separately,
chunking_version sentinel, governance policy invalidates lookup,
canon stays transient, synthetic source_root encodes ref).
- tests/test_canonical_projection.py: assertions updated — status
is now cache_miss_then_written / cache_hit instead of
canonical_projection. Added a transient-mode test pinning the
policy gate.
- tests/test_witness.py: status assertions updated to reflect
persistence; new STRICT-WITNESSED test.
- tests/test_directives.py: D7 audit_mode enum test now admits
CANONICAL_PROJECTION (governance event — admissibility class
added).
Full suite: 1367 passed, 36 skipped (was 1306; +61 new).
Real-shard smoke
================
$ make query Q="0.1 + 0.2" BURN=1
→ cache_miss_then_written, ~300ms wall, row written
$ make query Q="0.1 + 0.2"
→ cache_hit, ~40ms wall, hit_count++
$ make chain-check-shards
→ 0 breaks per shard
Acted on response review (2026-05-09,
response_ticket-000027-canonical-projections-in-providence-cache.txt).
#000027 — Hard-constraint phrasing corrected. Original draft said
"no schema changes / no new admissibility mode," which would
mislead an implementer into skipping the CHECK-constraint
extension. The current SCHEMA_SQL CHECK rejects both
'CANONICAL_PROJECTION' (audit_mode) and 'canonical_projection'
(verifier_method) — five locations in store.py listed in §9.1.
Updated to: "no new columns / no new tables / 8-dim cache_key
preserved / CHECK widened via existing rebuild migration." Same
spirit as the original draft, but unambiguous. §2.6 amended to
note the CHECK widening is value-space, not column/table change.
Forward-link added pointing at #000028 as the witness layer that
runs ON TOP of this persistence primitive (not a Phase 2).
#000028 — Post-MVP follow-up appendix. MVP shipped in 656b573
(witness.py 394 LOC + tests 354 LOC, 28/28 passing). Five items
captured for follow-up:
- terminology: ticket uses "modality" but kernel/cache/LLM are
epistemic witness channels, not carrier modalities; suggest a
one-line clarification near §1 to prevent cross-carrier
misreads.
- capital ledger integration (#000020): MVP records LLM latency
on the result dict but doesn't thread cost into the ledger;
~15 LOC follow-up to wire it; ForkScore (#000012) needs this
to compare witness-on vs witness-off forks honestly.
- sample-rate policy field (canonical_witness_sample_rate):
explicit out-of-scope per §2.4, but flagged so it isn't
re-discovered when calibration-data hunger appears.
- cache-leg dependency on #000027: MVP works with stub closure
returning None; STRICT-WITNESSED is unreachable until #000027
lands; swap is one-line when persistence ships.
- 5F / 5T bench-data integration: divergence events are
supervised calibration data; suggest a future
`make bench-witness-divergence` target.
- threat-model caveats from review §7: correct claim is "no
clean single-channel adversarial path," not "no adversarial
path"; future dual-kernel witness option captured.
No code changes in this commit — ticket text only.
Design doc only; no code in this commit. Spec the persistence pattern
for canonical-projection answers — same providence_cache table, same
audit chain, same admissibility ledger, distinguished by
verifier_method="canonical_projection" and audit_mode=
"CANONICAL_PROJECTION". Synthetic-but-deterministic values for the
LLM-only cache_key dimensions (source_root, model_profile_hash,
conversation_hash) bound to pi_star_ref so kernel-version migrations
work via the existing falsification pipeline.
chunking_version = "n/a-canonical" sentinel keeps RAG chunker bumps
from mass-staling unrelated math/logic answers.
Builds on existing _canonical_projection_preflight() short-circuit
(currently transient compute, no row written). Ships with #000027 and
fox confirmation on §8 open questions before code lands.
Two fan-out streams. Both bear directly on the ticket's "search
latency on real shards" headline finding.
## Lazy concept_relations loading
Phase 1 (migration memoization) cut SQLite executes 65% but warm-
cache wall barely moved. cProfile pinned the next hotspot:
synonym_expand 2.8 s × 2 calls + _load_token_idf 0.3 s. The eager
loader dumped all ~290 K concept_relations rows on first call —
the price of being able to answer ANY future question without
re-querying. Wrong tradeoff for single-query CLI use.
Refactored arborist/concepts/query.py:
- New _load_neighbors_for(shards_dir, tokens) — targeted
WHERE token IN (...) OR target IN (...) query. Returns just the
direct synonym neighborhood for the given tokens (~300 rows for
a typical 5-token question, vs 290 K for the full table).
- New _load_rivalry_rows(shards_dir) — process-wide cache of the
~2-row rivalry-relation set; near-zero cost.
- New _load_idf_for(shards_dir, tokens) — IDF only fetched when
expansion exceeds max_total (the cap). Most queries never reach
the truncation branch and skip IDF entirely.
- Per-token process-wide neighbor cache so multi-query bench scripts
don't re-query tokens already seen.
- synonym_expand and rivalry_excluded refactored to use the lazy
loaders. Eager _load_indices / _get_indices kept for any
back-compat caller; not used by hot paths.
- invalidate_cache() clears all three caches.
All 14 concept tests pass unchanged — the contract is preserved.
Re-profile of `who wrote virt-back?` against ~38 GB of real shards
(warm cache):
metric pre-fix post-Phase-1 post-lazy-concepts
wall_ms 14,500 13,400 9,300 (-36%)
search_ms 9,900 10,600 5,000 (-49%)
SQLite executes 10,623 3,687 3,708 ~same
synonym_expand 2,966 2,840 ~0 (lazy hit)
Search target was <5 s; we hit 5.0 s on the warm path. Cold cache
should drop further (the 290 K-row dump was disk-bound).
## Real-shard baseline artifact (Phase 2)
bench/scripts/real_shard_baseline.py — runs an 8-question fixture
through the full query() pipeline and emits:
- bench/results/real-shard-baseline.json (durable; commit_sha,
shards_fingerprint, per-query rows, summary)
- bench/results/real-shard-baseline.md (human-readable summary)
Question set in bench/fixtures/real-shard-baseline-v1.jsonl:
virt-back, France, Mac OS X, Linux, Microsoft, AMD/Intel
(rivalry path), and two canonical-projection cases (math + logic
preflight short-circuit).
First baseline run (commit e78814c plus this fan-out, BURN=1):
audit_mode n notes
STRICT 4 (virt-back, France, Mac OS X, Linux)
HYBRID 2 (Microsoft founder, AMD/Intel)
CANONICAL 2 (0.1+0.2, A IMPL B; <1 ms each)
wall median 4.1 s (range 0.6 ms – 7.2 s)
primary used 4 / 8
`who wrote virt-back?` lands at 6.3 s wall, audit STRICT, primary
source #1, cited evidence still includes a copyright footer
(reviewer's warrant-quality finding — deferred to follow-up ticket
since the latency fix was the gating concern).
Hard constraint preserved: baselines NEVER gate CI. The artifact
is for confirming a fix moved the needle, seeding ForkScore
comparisons, and noting findings worth tickets.
`make bench-real-shard` wires it. Honors ARBORIST_SHARDS_DIR.
## Test status
Full suite: 1306 passed, 36 skipped (no regressions from the
concept refactor; 14 concept tests cover the lazy/eager
equivalence).
