Closes four gaps in the initial Phase 1 landing (commit f625cac):
1. §14 row 4 — Hermes-saturation guard. `hermes_utilization` was on
the input contract but never consumed. Now: `utilization >=
budget` → DEFERRED with HERMES_SATURATED note + advisory event
carrying (utilization, budget) for Phase 2 audit. Three new
tests (saturation-equal, saturation-overflow, headroom-exists).
2. §13 step 11 — falsification-fixture proposal emission.
New `FalsificationFixtureProposal` dataclass; controller now
emits one per branch whose `witness_divergence >=
falsification_divergence_threshold` (weight-tunable, default
0.5). Emission fires BEFORE the all-vetoed cascade so vetoed-
AND-diverged branches still surface as 5F-fixture candidates
per #000025. Four new tests (high-divergence emits, low-
divergence stays silent, threshold is weight-tunable, vetoed-
diverged emits anyway).
3. Entropy + memory gating moved from module-level constants
(`H_LOW=0.3`, `H_HIGH=0.7`, `KAPPA_MEMORY=0.5`) to weight
fields (`h_low`, `h_high`, `kappa_memory`). Constants stay as
back-compat exports; defaults match exactly so byte-identical
behavior when neither override fires. Three new tests
(h_low/h_high tunable, kappa_memory tunable, back-compat
match).
4. Veto-class cascade hardening. Added explicit tests for the
`replay_window_unbounded` → ESCALATE path and the
`soft_hash_signal` → QUARANTINE path (§6 + §14 documented but
previously untested). Plus tests for the §6 fail-loud
priority ordering: ESCALATE > QUARANTINE > REJECT when mixed.
Dry-run regenerated against ~/.arborist/shards corpus:
Target A at τ_qa=1d surfaces **576 FalsificationFixture
proposals** from witness_divergence >= 0.5 (24% of swept
candidates). These rows are now an actionable funnel for 5F
fixture mining under #000025 §3.
Module: arborist/substrate/prometheus.py (+117 LOC, 899 total)
Tests: tests/test_prometheus.py (+196 LOC, 22 → 36 tests)
Dryrun: bench/scripts/prometheus_sigma_sweep_dryrun.py +
bench/results/prometheus-sigma-sweep-dryrun-2026-05-10.md
track the new falsification_proposals_total counter.
Full suite: 2326 passed, 37 skipped (+14 net from the new tests).
Pure-function recursive-falsification controller implementing
ticket #000037 §13 algorithm (steps 1-9 + 12), §5 Shannon-entropy
fork-selection with stable softmax, §6 8-class hard-veto order,
§7 Kelly-bounded allocation with 4 safety guards, §7.1 EMA-smoothed
difficulty update, §14 exception-matrix dispatch, and §15 three
named weight profiles (safe/conservative/exploratory).
No DB, no LLM, no scheduler — advisory pure function over already-
committed state. Phase 2 (controller_events sibling table) lands
in companion commit a786d6d. Phase 3 (sleep sweep scheduler) is
not in this commit.
Module: arborist/substrate/prometheus.py (782 lines)
Tests: tests/test_prometheus.py (22 passing tests — the 17 named
contracts from §16.2 plus 5 boundary cases for the §6 veto-class
priority dispatch and §7.1 EMA stability).
Also removes obsolete tests/test_prometheus_sigma.py — pre-Phase-1
scaffolding placeholder whose 17 tests all called pytest.fail()
with "Phase 1 implementation pending" and the @skip_until_phase_1
decorator never auto-flipped to pass-on-import. The contract is
now in tests/test_prometheus.py.
Sibling table `controller_events` for advisory persistence of
ControllerDecision output from #000037 Phase 1. Same pattern as
`capital_ledger` — forward-migrated, indexed, but does NOT enter
audit_events.event_hash preimage. Audit chain semantics are
unaffected.
Three event kinds:
controller_decision — one per ControllerDecision
controller_difficulty — one per ControllerDecision (records
the difficulty_next value)
controller_budget_allocation — one per branch with nonzero
allocation in decision.allocations
Idempotent on (event_kind, body_hash) UNIQUE constraint.
sha256 of canonical-JSON-encoded body is the dedupe key.
Tests: tests/test_prometheus_audit.py — 14 cases covering
migration, idempotency, no-chain-mutation invariant, query paths.
Uses stub ControllerDecision so tests run independently of Phase
1's controller_decide implementation.
Fan-out follow-up to ``fc5ba50``. Two thrusts in one commit since
they exercise the same surface:
== Task 3: extend AUTOCOUNT with db-rows metric ==
New metric ``db-rows`` for tagging live SQLite row counts (alias
tables, claim-pack records, etc — operator state that drifted on
``30a9488`` and earlier). Target syntax::
<!--AUTOCOUNT:db-rows:citation_aliases-->74<!--/AUTOCOUNT-->
<!--AUTOCOUNT:db-rows:002.db:concept_relations-->1234<!--/AUTOCOUNT-->
Default shard: ``~/.arborist/shards/000.db`` (where the alias
tables live per ``arborist.cli._aliases_db_path``). Operator state
is graceful-skip semantics: when DB or table is absent (CI, fresh
checkout, sibling repo), the claim is logged as skipped and the
test still passes. Drift only fires when the DB IS present and
the count diverged.
Sentinel returns:
- ``_DB_MISSING`` (-2): shards dir not present → skip
- ``_TABLE_MISSING`` (-3): DB present but table absent → skip
- ``_DB_ERROR`` (-4): malformed table name or sqlite error → skip
Table name validated against ``[A-Za-z_][A-Za-z0-9_]*`` regex
before string-interpolating into ``SELECT COUNT(*) FROM <table>``;
this is belt-and-suspenders since AUTOCOUNT tags are author-
controlled, but the dynamic SQL surface deserves a bouncer.
Smoke verified under HOME redirect to ``/tmp/<empty>``: 3 db-rows
claims gracefully skip with informative line-numbered messages,
suite still passes.
== Task 2: backfill 15 tags ==
Cookbook test/code-density table (lines 569-579, 10 rows) — every
``(N tests)`` cell now machine-checked:
| aliases.py | 512 | 469 (28 tests) | 0.92 |
→
| aliases.py | 512 | 469 (<!--AUTOCOUNT:tests:tests/test_aliases.py-->28<!--/AUTOCOUNT--> tests) | 0.92 |
Markdown renderers strip HTML comments — table cells display
``28 tests`` unchanged. The ``warrant_resolver.py`` row stays
untagged because its test count is split across two test files
(verifier + parser) and the cell encodes a combined "~430"
instead of one collected count.
Cookbook alias-count surfaces (3 db-rows tags):
- L364 ``citation_aliases (74 rows live as of 2026-05-10)``
- L437 ``#000041 — citation-aliases table + 74 live rows``
- L438 ``#000042 — term-aliases table + 13 live rows``
Ticket #000035 (in progress, line 274) — refresh ``20 tests``
→ ``27 tests`` for ``test_anchor_prg.py`` + tag. Same drift
pattern as ``5c21e83``: ticket prose was written before the
``de997f7`` 2026-05-10 pattern backfill that added 7 tests
(prefix-extension closure, hand-formula, parametrized
invalid-input cones). Also tagged ``L279``'s 10-vector KAT
fixture claim with ``fixture-rows`` metric.
== Closed-ticket counts deliberately not tagged ==
#000028, #000030, #000042, #000031, #000004, #000026, #000009,
#000032, #000008 all carry historical "N tests pass" snapshots
from their landing date. Those are point-in-time records, not
live claims — drifting from current state is BY DESIGN. Tagging
them would fire the test on every successive change to the
codebase. Closed tickets are the design log; we don't backfill
them.
== Coverage summary ==
Total tags after this commit: 44 (was 29; +15)
Tags by metric:
tests: 39
fixture-rows: 2
db-rows: 3
Files with tags:
docs/warrant-substrate-cookbook.md 27 (was 14)
docs/soft-hash-channel-analysis.md 5
docs/tickets/ticket-000006-bench-emergent... 4
docs/seven-point-program.md 3
docs/calculator-test-patterns.md 3
docs/tickets/ticket-000035-prg-choice-phi-prg.md 2 (new)
== Verification ==
$ .venv/bin/pytest tests/test_doc_counts.py -v
3 passed in 4.32s
$ .venv/bin/pytest -q
2276 passed, 54 skipped in 168.34s
$ HOME=/tmp/empty pytest tests/test_doc_counts.py -v -s
3 db-rows AUTOCOUNT claim(s) skipped:
docs/warrant-substrate-cookbook.md:364 db-rows:citation_aliases skipped — /tmp/empty/.arborist/shards not present (CI / fresh checkout)
docs/warrant-substrate-cookbook.md:437 db-rows:citation_aliases skipped — /tmp/empty/.arborist/shards not present (CI / fresh checkout)
docs/warrant-substrate-cookbook.md:438 db-rows:term_aliases skipped — /tmp/empty/.arborist/shards not present (CI / fresh checkout)
3 passed in 4.78s
No new dependencies. No schema changes.
The doc-drift pattern recurred four times today on 2026-05-10
(commits 6cbbf95, 14bcb99, 5c21e83, 30a9488). Each fix was the
same shape: walk a doc, find a count that drifted from live truth
during the hours after the doc was written, refresh it. Cost: ~5
min per drift × 4 = 20 min of manual catching, with no guarantee
the next drift gets caught before someone external reads it.
Per fox's selection: regression test that makes drift loud at
test time instead of relying on visual catching.
== Mechanism ==
`tests/test_doc_counts.py` scans `docs/**/*.md` for AUTOCOUNT
tags of the form:
<!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->
Two metrics supported:
- `tests` — pytest collected count for path. Batches every
tagged path into one `pytest --collect-only` subprocess
(~0.5s total).
- `fixture-rows` — non-blank-non-comment line count in a JSONL
fixture.
GitHub and most markdown renderers strip HTML comments, so
readers see only `N`. The tags are invisible in rendered output
but make the claim machine-checkable. Three tests in the file:
1. `test_doc_autocount_claims_match_live` — the core invariant
2. `test_autocount_tags_are_well_formed` — open/close balance
3. `test_autocount_metric_names_are_documented` — fail-closed on
undocumented metrics (catches typos)
Failure message names the doc file, line number, and the
claimed-vs-live diff. Example:
`docs/foo.md:42 AUTOCOUNT(tests:tests/test_x.py) claims 23, live is 27`
== 29 tags installed across 5 docs ==
While installing tags I had to read the surrounding prose, which
surfaced six stale counts that had drifted same-day:
`docs/soft-hash-channel-analysis.md`:
- L392 14 → 23 tests for phi_alignment_probe
- L417 20 → 27 tests for anchor_prg
- L463 14 → 23 tests for phi_alignment_probe (status section)
`docs/seven-point-program.md`:
- L77 68 → 58 tests for metacognition (drift -10; the file
shed tests during a refactor and the doc didn't catch up)
- L78 9 tests for `test_dag.py::test_preflight_*` — removed
count entirely; pytest selector subsets aren't currently
supported by the AUTOCOUNT metric set (would need a
`tests-matching` metric; not worth the surface for one claim).
- L110 24 → 33 tests for test_dag.py
`docs/calculator-test-patterns.md`:
- L35 33 → 23 tests for warrant_resolver
- L35 10 → 9 tests for warrant_chain
- L16, L265 51 → 53 tests for t3_bound_calculator (kept
initial-shipment provenance in prose)
== Coverage installed ==
calculator-test-patterns.md 3 tagged claims
soft-hash-channel-analysis.md 5 tagged claims
warrant-substrate-cookbook.md 14 tagged claims
seven-point-program.md 3 tagged claims
tickets/ticket-000006-bench-... 4 tagged claims
---
29 tagged claims
Every count that drifted today is now tagged. Future drift
fires the regression test at the next pytest run instead of
waiting for human catching.
== Discipline pattern ==
Walk this pattern for any new doc that names a count:
1. Surround the number with the tag pair:
`<!--AUTOCOUNT:tests:tests/test_foo.py-->N<!--/AUTOCOUNT-->`
2. Run `pytest tests/test_doc_counts.py` (~3.5s)
3. If it passes, the claim is now machine-verified
Aim to tag counts on first authorship. Retrofitting is cheap
but only catches drift after the fact.
== Out of scope ==
Test counts inside source code (docstrings, CLI --help) are not
scanned — would expand the test surface significantly and the
drift pattern hasn't manifested there. Add `**/*.py` scope when
that pattern surfaces.
Alias-row counts and claim-pack-record counts could be tagged
with new `db-rows:<table>` and `db-where:<sql>` metrics; deferred
until the next drift on those numbers (none caught today after
30a9488's cookbook refresh).
== Verification ==
$ .venv/bin/pytest tests/test_doc_counts.py -v
3 passed in 3.89s
$ .venv/bin/pytest -q
2276 passed, 54 skipped in 153.21s
No new dependencies. No schema changes. No source-code changes.
Pre-Phase-1e: falsification-v1.jsonl covered 10 motif tags across 50
fixtures (the high-traffic warrant/title/anchor/format set). Phase
1e adds 12 fixtures (5f-fal-051..062) for the previously-uncovered
motifs from the verifier+soft-demote registries:
CITATION_MISMATCH DEFLECTION_DETECTED
MANUAL_QUOTE_VIOLATION SCHEMA_INVALID
SOURCE_ROLE_BLOCKED SUBJECT_TOKENS_ABSENT
TOO_MANY_EVIDENCE_IDS UNKNOWN_EVIDENCE_ID
BROAD_QUANTIFIER_RUNAWAY BROAD_QUANTIFIER_CAP_APPLIED
BROAD_QUANTIFIER_SCOPE_UNBOUND BROAD_QUANTIFIER_REJECTED
Coverage now: 22 unique motif tags across 62 fixtures.
Harness changes:
- test_bench_batteries.py: bump pass_count assertion 50 → 62 in both
falsification tests; add test_5f_falsification_covers_every_documented_motif
that pins the motif set against the verifier+soft-demote registries
so adding a new violation upstream surfaces here as a missing
fixture (loud signal, no silent drift).
- test_session_integration.py: bump full-suite total 662 → 674.
Closes#000025 §10.12 (every documented failure-motif tag).
Still open in Phase 1b: §10.11 (real shard finetuning chains),
§10.13 (Feedback Loop latency/efficiency against real workload),
§10.14 (threshold handoff to #000012).
Backfills zero dedicated coverage on arborist/pi_star/logic.py
(371 lines, 2026-05-10 zero-coverage sweep). KATs mirror the
docstring's equivalence classes: commutativity, associativity,
IMPL/IFF/XOR rewrites, De Morgan, double negation, distribution,
idempotence, within-clause tautology drop. Pins v1's documented
limitation: A AND NOT A is NOT collapsed (multi-clause
contradiction detection is out of scope; only empty clauses
surface as FALSE). Negative cones: >8 atoms, unrecognized token,
unexpected char, unbalanced paren, unconsumed tokens, dangling
operator, empty, non-bytes. Plus determinism + lexical-sort
canonicalization.
Three threads bundled, all surfaced by today's calculator-test-
patterns.md audit + fox's directive to remove v-prefix from test
filenames:
THREAD 1 — rename test_v8_fork_score.py → test_substrate_fork_score.py
=====================================================================
Single test file in the tree had a v-prefix in its filename:
``tests/test_v8_fork_score.py``. Renamed via ``git mv`` for
consistency with yesterday's substrate refactor (the package is
``arborist/substrate/fork_score.py``; the CLI subcommand is
``arborist substrate score``; the test file should match).
No internal code changes needed — the file's imports + assertions
were already updated to ``arborist.substrate.*`` paths in
yesterday's bae5caf commit. Pure rename.
THREAD 2 — close v1 substring discipline gap
=============================================
Audit of test_anchor_prg.py + test_phi_alignment_probe.py against
docs/calculator-test-patterns.md §2 (versioned-default discipline)
found one gap: both files asserted the version string's exact
value but neither asserted the ``"v1"`` substring discipline that
fox's test_returns_calculator_version_token established.
Added ``assert "v1" in PHI_PRG_VERSION`` to
test_module_exports_version_string in test_anchor_prg.py.
Added ``assert "v1" in PROBE_VERSION`` to
test_module_exports_thresholds_and_version in
test_phi_alignment_probe.py.
Both follow fox's pattern: when the algorithm changes (v2-blake3-
expansion, v2-arnoldi-iteration, etc.), the version string MUST
change too. The "v1" substring assertion catches a future
contributor who refactors without bumping the version constant.
THREAD 3 — close CLI subprocess gap on test_substrate_fork_score.py
====================================================================
The renamed file had four CLI tests but all in-process via
build_parser() + parse_args() + func(args). That catches argparse-
shape drift but NOT entry-point / module-loading / sys.argv drift.
Added test_cli_substrate_score_subprocess_invocation: real
``subprocess.run(["python", "-m", "arborist.cli", "substrate",
"score", "--parent", ..., "--child", ..., "--out", ...])`` against
synthetic bench results. Asserts exit 0 + the --out artifact is
written + JSON-parses with valid verdict.
Pattern matches fox's test_cli_baseline_runs_clean in
test_t3_bound_calculator.py + the 581ad90 KAT-fixture-gap closure.
Same hazard fox already hit three times during the substrate
rename refactor (85be5eb, 209d670, b320e27): import-only tests
silently miss CLI surface drift.
CHECKLIST AUDIT — POST-FIX
==========================
Three calculator-style test files now all 9-item complete:
| t3 | anchor_prg | phi_alignment | substrate_fork |
KAT fixture | ✓ | ✓ | ✓ | n/a (different)|
VERSION + "v1" | ✓ | ✓ NOW | ✓ NOW | ✓ |
Hand-formula | ✓ | ✓ | ✓ | ✓ (synthetic) |
Monotonicity | ✓ | ✓ | ✓ | ✓ |
Closure / sum-of-parts | ✓ | ✓ | ✓ | ✓ |
Parametrized invalid | ✓ | ✓ | ✓ | ~ |
CLI subprocess | ✓ | n/a | n/a | ✓ NOW |
Doc parity | ✓ | KAT | KAT | KAT |
Module-export shape | ✓ | ✓ | ✓ | ✓ |
All four files now consistently track the calculator-test-patterns
checklist. test_substrate_fork_score.py is structurally different
(verifier-adjacent: tests scoring + verdict-band logic, not
closed-form math) so some checklist items map differently — KAT
fixture replaced by synthetic-input verdict tests (closer to
verifier-style), parametrized-invalid is partial (per-verdict-
class assertions rather than per-bad-input cone). Acceptable.
Test counts:
- test_substrate_fork_score.py: 26 → 27 (+1 subprocess test)
- test_anchor_prg.py: 27 → 27 (assertion added inline)
- test_phi_alignment_probe.py: 23 → 23 (assertion added inline)
- t3 file untouched in this commit (581ad90 already at 53)
Full suite: 1985 → 1986 (+1 from this commit's only
new-test-function addition; the inline assertions don't count
as new tests).
Hygiene
=======
- make test → 1986 passed, 45 skipped.
- All four calculator-style test files structurally aligned.
- No v-prefixed test filenames remain in tests/ tree.
Audited fox's exemplar 51-test file against the 9-item checklist
in docs/calculator-test-patterns.md (codified earlier today in
0725eb4). Two gaps found, both additive:
GAP 1: no KAT fixture
=====================
Checklist item 1: "KAT fixture under bench/fixtures/<module>/
(≥ 5 cases)". Fox's tests cover the doc's worked-example numbers
inline (test_baseline_matches_section_11_doc) but no separate
fixture file existed for off-the-baseline regression coverage.
Generated bench/fixtures/t3-bound/known-answer-tests.jsonl with
8 KATs:
- small-deployment-§7.1, medium-deployment-§7.2,
hardened-deployment-§7.3 (the doc's three worked examples)
- extreme-low-g (g=0.001, σ=0.5 — exercises low-SNR regime)
- tight-window-W=100 (small-W ceiling-rounding edge case)
- tightened-c-b1 (override-constants path)
- all-constants-tight (all three c_b1/c_b2/c_b3 overridden)
- b3-floor-regime (factor < 1, B3 floors to 0)
Each entry pins (calculator_version, inputs, expected_total,
expected_b1, expected_b2, expected_b3, expected_snr_grad).
Algorithm change MUST bump CALCULATOR_VERSION + emit new fixture
file under bench/fixtures/t3-bound/ — old runs replay against
old data per §1 discipline.
GAP 2: no B3 hand-computed formula test
=======================================
Checklist item 3: "Hand-computed formula tests — at least one
per independent contribution / output field". Fox had
test_b1_exact_formula + test_b2_exact_formula + test_snr_grad_formula
covering three of the five output fields. B3 had only
test_b3_floor_at_zero_when_factor_below_one (an edge case),
not a closed-form check on the general formula.
Added test_b3_exact_formula: hand-computes
``C_B3 · ⌈W/E⌉ · log₂(N_b · σ_grad / ‖∇L_max‖) / 2`` per #000036
§5 (Bottou-Bousquet refinement), asserts agreement with the
function's B3_contribution. Pairs cleanly with the B1/B2 hand-
formula tests fox had.
CHECKLIST AUDIT — POST-FIX
==========================
1. KAT fixture ✓ NOW (was ❌; 8 entries)
2. VERSION + "v1" ✓ test_returns_calculator_version_token
3. Hand-formula ✓ NOW B1/B2/B3/snr_grad all covered
(was ⚠️ partial; B3 had floor-only)
4. Monotonicity ✓ test_monotone_in_window_length /
gradient_fraction
5. Closure ✓ test_total_equals_sum_of_three_contributions
6. Parametrized invalid ✓ four @pytest.mark.parametrize blocks
7. CLI subprocess ✓ test_cli_baseline_runs_clean +
test_cli_invalid_input_exits_2
8. Doc parity ✓ test_baseline_matches_section_11_doc
(caught today's §11 calibration drift)
9. Module-export shape ✓ test_returns_calculator_version_token +
test_constants_recorded
All nine items now ✓. test_t3_bound_calculator.py is the
exemplar for calculator-style test discipline.
Test count: 51 → 53 (+2 from this commit). Full suite:
1915 → 1985 (+70 from fox's parallel work + this commit's +2;
partial cycle effects).
Hygiene
=======
- make test → 1985 passed, 45 skipped.
- KAT fixture is JSONL with header comment naming
CALCULATOR_VERSION; future drift caught at the test level.
- Eat-my-own-dogfood: applied my docs/calculator-test-patterns.md
checklist to fox's exemplar test file. The fact that gaps
surfaced (even on fox's substantive 51-test surface) validates
that the checklist has real reviewer value, not just guideline
signaling.
Catches the regression class where a PR breaks argparse setup
(duplicate flag, broken set_defaults reference, renamed leaf
verb) without breaking any existing test. ``--help`` exercises
the parser-construction path without invoking any handler.
Coverage:
- top-level: arborist --help / --version / no-subcommand error
- 30 top-level subcommand --help calls (parametrized)
- 38 nested-subcommand --help calls across 9 verbs that have
their own subcommand groups (alias / memory / capital /
selfmodel / mesh / crawler / canon / substrate / snapshot)
- completeness check: top-level --help mentions every verb in
_TOP_LEVEL_SUBCOMMANDS (regression guard against silent
removal)
All 68 pass. ~20s wall via subprocess (~150ms per --help call x 68
calls + Python startup overhead). Fixture is hand-curated so a
new subcommand requires fixture update — that's the discipline
which surfaces the regression as a test failure rather than as a
missed --help test.
Pattern after fox: contract phrases pinned as test invariants
(here, the subcommand verb names) so silent renames fire the
test loud + force the fixture update + force the docstring +
help-text update too.
CLI version-bake sweep surfaced one real defect: the Phase 1 test
scaffolding I shipped in c422216 imported ``from arborist.v9
import prometheus``, baking in the version-prefixed namespace path
that yesterday's substrate refactor (654d923) abolished.
The skip mechanism (try/except ImportError → CONTROLLER_AVAILABLE
= False) was masking the issue: when fox lands Phase 1 of #000037
at ``arborist/substrate/prometheus.py`` (per the topic-named
convention), my tests would CONTINUE to skip with
"controller module arborist/v9/prometheus.py absent" because the
import target itself is wrong. The skip-stub becomes permanent
dormancy instead of activating when the module ships.
Two changes:
- Import line: ``from arborist.v9 import prometheus`` →
``from arborist.substrate import prometheus``.
- Skip-reason text + module-docstring: ``arborist/v9/prometheus.py``
→ ``arborist/substrate/prometheus.py``, with a parenthetical
noting the post-2026-05-10 topic-named convention and that
§13's original sketch predated the v-dir retirement.
The fix is a real one — when fox's Phase 1 of #000037 lands, my
17 skip-stubs now activate against the correct module path.
Without this fix, they'd silently stay dormant.
Sweep summary
=============
Walked every CLI subcommand --help (arborist top-level + nested
substrate / memory / capital / selfmodel / warrant-resolve / sweep
/ alias / mesh / crawl / providence) plus full-tree grep for
``arborist v[789]`` / ``arborist\\.v[789]`` / ``arborist/v[789]``.
Remaining v-prefix mentions across the tree (all intentional):
- arborist/cli.py:5166-5169 — historical-note comment for the
v8→substrate rename
- arborist/substrate/__init__.py:8 — same convention-explanation note
- arborist/substrate/anchor_prg.py:59 — bytestring inside SHA-256
derivation of placeholder seed; can't change without breaking KAT
- bench/fixtures/phi-prg/known-answer-tests.jsonl:1 — fixture header
naming the v7 paper section §9.10 (paper version, correct)
- docs/v8-fork-score.md:4 — historical-note ("module moved from")
- docs/tickets/ticket-000019, ticket-000013 — "arborist v9.8" schema
references (schema version, correct)
- docs/_source/merkle-agi-v7w-spatial-temporal.rst — v7 paper +
v9.8 schema references (both correct)
Tests: 1915 passing, 45 skipped (was 1872; +43 from fox's parallel
test additions during this commit's prep + the 17 prometheus
skips activating against the correct path stub).
CLI version-bake sweep complete. The substrate refactor is now
surface-clean end-to-end.
Same pattern-application as de997f7 did for test_anchor_prg.py.
The phi_alignment_probe tests landed in 1dfb8b9 with KAT
regression + verdict-bucket coverage + Lanczos convergence
check, but lacked the four patterns fox demonstrated in
test_t3_bound_calculator.py (51 cases for the T3 calculator):
monotonicity, hand-computed formula, closure invariants, and
parametrized invalid-input cones.
New tests added:
1. **test_monotone_alignment_strength_in_concentration** —
tighter W concentration on low-λ subspace must monotonically
increase the ratio. Tested across 32-row → 16-row → 8-row
concentrations, normalized to constant ‖W‖_F. Catches sign
errors + drops in the alignment-score formula.
2. **test_monotone_alignment_in_dim_h** — verdict invariant
under W column-count scaling. Sanity check that A(W, H)'s
‖W‖_F² normalization decouples it from sample count.
3. **test_uniform_baseline_matches_analytical_formula** —
hand-computes the isotropic baseline
``E[A_bot(W_uniform, H)] = (1/dim_d) Σ 1/(λ_j+ε)`` from
#000034 §2.1 derivation; asserts exact agreement with the
function's ``a_uniform`` field. Catches algorithm drift the
KAT regression would miss (KAT could regenerate against a
buggy implementation).
4. **test_full_spectrum_a_top_plus_a_bot_covers_full_isotropic_baseline**
— closure invariant: when k_top + k_bot = dim_d, the function's
a_top + a_bot must equal full-spectrum A computed via dense
numpy.linalg.eigh decomposition. Catches missing terms /
double-counting.
5. **test_eigenvalue_ordering_top_dominates_bot** — closure
invariant: top-k eigenvalues must all be ≥ bot-k eigenvalues.
Catches a bug where eigsh's 'LA'/'SA' modes returned
overlapping ranges on near-degenerate spectra.
6. **test_rejects_wrong_dim_w** — parametrized over (1-D, 3-D,
0-D scalar) shape errors. Same pattern as test_anchor_prg's
parametrized rejects.
7. **test_rejects_non_positive_epsilon** — parametrized over
(0, -1e-6, -1.0). Collapsed N separate test_rejects_*
functions into a single parametrized cone.
Test count: was 14 in test_phi_alignment_probe.py; now 23
(+9 from the new patterns + parametrize expansion).
Full suite: 1727 → 1872 (note: large jump partly from fox's
parallel test additions today, +136 since my last test count
checkpoint; my contribution here is +9 directly attributable
to this commit).
Three calculator/probe-style modules now have consistent
test coverage:
bench/scripts/t3_bound_calculator.py — 51 tests (fox)
bench/scripts/phi_alignment_probe.py — 23 tests (this commit)
arborist/substrate/anchor_prg.py — 27 tests (de997f7)
Same pattern bench applied across all three. Future
calculator-style code should pin: monotonicity in each input
axis + hand-computed formula assertions + closure / sum-of-parts
invariants + parametrized invalid-input cones.
Hygiene
=======
- make test → 1872 passed, 45 skipped
- make chain-check-shards → 0 across all 7 shards
- All new tests use synthetic inputs (no LLM, no shard
dependency); run in ~35s suite-wide
arborist/qa/prompts.py — 4 string constants that ARE the contract
with the LLM. Silent edits drop bench STRICT-rate by tens of
points; per CLAUDE.md "bench-maxing" discipline, prompt
regressions need a per-PR guard, not just bench surfacing after
the fact.
This test file pins the load-bearing phrases as regression
guards for both claim_lattice modes:
CLAIM_LATTICE_SYSTEM_PROMPT (pointer mode)
- non-empty + substantial
- both worked examples present (Apple founders + Mars descriptive)
- two-pointer cap rule (folds into claim_lattice_max_pointers_per_claim)
- pointer ID shape teaching (E1, E2, E3, [E#] / [E#,E#])
- one-claim-per-line rule (parser splits on newlines)
- synthetic-elision-by-construction-impossible: NO instruction
to wrap claims in double quotes (that's legacy quote-mode)
CLAIM_LATTICE_GROUNDING_REMINDER (pointer mode user-turn)
- "REMINDER" prefix; "pointer-line" format restated
- two-pointer cap restated
CLAIM_LATTICE_JSON_SYSTEM_PROMPT (JSON mode)
- schema shape: claims/text/evidence_ids
- first-char-`{` / last-char-`}` discipline
- two-evidence-id cap
CLAIM_LATTICE_JSON_GROUNDING_REMINDER (JSON mode user-turn)
- schema restated; cap restated
Cross-prompt parity
- both modes reference the two-pointer cap
- both reminders end with "next message" (handoff to question)
- all 4 constants importable + non-empty strings
Full suite: 1872 passed, 45 skipped.
arborist/pi_star/protocol.py + arborist/pi_star/registry.py are
the foundation every concrete π* kernel rides on. Both shipped
in #000015 Phase 1 with zero direct tests; concrete kernels
(arithmetic@v1, logic-kernel@v1, algebra-symbolic@v1, ...) have
their own test files but the protocol contract + registry
mutation discipline weren't pinned.
Coverage:
protocol.py
- PiStar @runtime_checkable: instances satisfying the duck-type
pass isinstance check; missing-method instances are rejected
- registry_key returns "name@version" exactly; distinct versions
yield distinct keys
- equivalence_class_id is sha256 over canonicalize() output;
invariant under pre-canonical form (two raws that canonicalize
to the same bytes get the same eclass id); distinguishes
different canonicals
- assert_round_trip passes on idempotent π*; raises
AssertionError naming the kernel on non-idempotent;
propagates PiStarError when canonicalize raises on the test
input itself
registry.py
- register inserts; get retrieves
- register IS idempotent for the same instance at the same key
(no error)
- register REJECTS a different instance at the same key
(cache_key invariant: name@version content-pinned)
- same name with different versions coexist
- get raises KeyError on unknown key
- list_keys returns sorted keys (deterministic for cache_key
derivation downstream)
- domains() groups by domain; per-domain key lists are sorted;
empty registry → empty dict
isolated_registry fixture monkeypatches REGISTRY to {} for
mutation tests so the global registry stays untouched (matches
the module docstring's no-public-unregister discipline).
Substrate-paper-spec'd primitives (arborist/substrate/* and
arborist/pi_star/protocol.py + registry.py) all directly tested
now.
Two related cleanups in one commit, both surfaced by reading fox's
test_t3_bound_calculator.py (51 tests for my T3 calculator):
1. Refresh stale §7 numbers in the T3 bound doc
=================================================
fox's test_baseline_matches_section_11_doc docstring (lines
56-61) flagged that my §7.1 worked example said
622.7 / 290.0 / 32.7 bits but the calculator's actual
closed-form output is 625.87 / 292.48 / 33.39. Same drift in
§7.2 (3358 → 3387.72) and §7.3 (247 → 247.14).
The numbers were rounded estimates from when I drafted the doc
before the calculator existed. Refreshed all three §7 numeric
examples to match the calculator's actual output (verified live
via t3_bound_bits()). §3 inline approximation likewise updated
(290 → 292.48). Added a short note pointing readers at the
calculator + tests as the source of truth.
2. Backfill anchor_prg tests with fox's patterns
=================================================
fox's test_t3_bound_calculator.py demonstrated four patterns I'd
missed in my #000035 phi_prg tests:
- **Output prefix invariant** (closure check): phi_prg(h, n+k)[:n]
≡ phi_prg(h, n). Streaming-counter invariant — would catch a
bug where a per-call seed mutation broke determinism across
dim_h values.
- **Output length monotonicity**: len(phi_prg(h, n)) == n exactly.
Parametrized over n ∈ {1, 2, 4, 7, 16, 17, 64, 1024}. Catches
off-by-one in `_expand` truncation.
- **Hand-computed first block**: assert that the first 64 bytes
of output equal a direct ``hmac.new(seed, h + b'\\x00\\x00
\\x00\\x00', sha512).digest()``. Pattern from fox's
test_b1_exact_formula — don't rely on KAT regression alone;
compute the first-principles math in the test file. Catches
algorithm drift the KAT (regenerated against a buggy version)
would miss.
- **Seed-bleed check**: changing the seed must change EVERY output
position. Probability of false-positive ≈ 64 · 2^-32 ≈ 2^-26;
none expected in practice.
- **Parametrized invalid-input tests**: collapsed N separate
``test_rejects_*`` functions into ``@pytest.mark.parametrize``
cones (4 wrong-size-hash cases + 3 non-positive-dim_h cases).
Same coverage, fewer test functions.
Test count: was 20 in test_anchor_prg.py; now 27 (+7 from
parametrize expansion + new patterns). Full suite: 1720 → 1727.
Hygiene
=======
- make test → 1727 passed, 45 skipped.
- make chain-check-shards → 0 across all 7 shards.
- All new tests use ``pytest.importorskip`` already at module top
(anchor_prg has no extras gate; tests run unconditionally).
Lessons captured
================
The patterns to remember for future calculator/probe-style code:
1. KAT regression alone isn't enough. Add hand-computed
formula tests so the math itself is asserted in the test
file, not just "consistent with a recorded snapshot".
2. Test monotonicity / closure invariants. They catch
algorithm drift, sign errors, missing terms.
3. Parametrize invalid-input tests. One function, N cases.
4. Test the doc's numbers against the function. Catches
calibration drift in the doc itself (this commit's
finding about §7).
5. CLI subprocess tests for end-to-end. Argparse + main()
drift the import-only tests miss.
The aliases.py public surface had 18 tests covering happy-paths,
audit discipline, domain isolation, lowercase normalization, and
expand-query semantics. Three direct gaps:
- list_term_aliases (no test at all): filter-by-domain, filter-by-
term-substring, unreachable-db fail-closed, missing-table fail-
closed
- _tokenize_fts_query (only via expand-query smoke): preserve
quoted phrases, parentheses-as-tokens (OR-expansion contract),
unterminated-quote fallback
- _quote_for_fts + _match_quoting (no direct test): pass-through
quoted, defensive quote-multi-word, quoted-reference symmetry
10 new tests; aliases.py test count 18 → 28. Same fixture pattern
as the existing tests (sqlite tmp DB with SCHEMA_SQL applied).
Adds tests/test_t3_bound_calculator.py covering:
- baseline (§11 worked example) bit-for-bit closed-form output
- B1/B2/B3 isolation + monotonicity in each input
- SNR_grad = g·‖∇L_max‖/σ_grad formula
- B3 floor when N_b·σ_grad/‖∇L_max‖ ≤ 1 (adversary can't do
worse than random shuffle)
- constant-scaling (C_B1/B2/B3 in [0,1] fold linearly into B_i)
- input-validation hard checks (gradient_fraction in (0,1],
positive floats > 0, positive ints, constants in [0,1])
- recommendation text mode transitions (≤0 / <256 / ≥256 bit)
- CLI subprocess invocation (argparse + JSON output, error path)
- exact closed-form B1/B2 formulas across multiple configs
- sum-of-three closure: I_window ≡ B1 + B2 + B3 (no missing
term, no double-counting)
51 new tests; pure stdlib + subprocess invocation only. Full
suite now 1720 passed / 45 skipped.
Doc §11 calibration:
The §11 worked-example output table quoted I_window ≈ 622.7,
B1 ≈ 290.0, B3 ≈ 32.7. The closed-form actuals are 625.8716 /
292.4813 / 33.3904 — a ~3-bit total drift from rounding in the
first-cut spec. Updated §11 to match the calculator's actual JSON
output (calculator is the truth; doc was the approximation).
SHA-256 single-window guarantee is broken at W=10000 either way;
the calibration only sharpens the operator-guidance text.
Lands the synthetic-ablation infrastructure proposed in fce8826's
ticket §7 amendment. Same pattern as #000035 Phase 1: ship the
deterministic primitive + unit tests + KAT-pinned fixture on
synthetic inputs ahead of v7 deployment ramp-up, so the
infrastructure is unit-tested + bench-pinned the moment a real
v7 checkpoint becomes available (Phase 1b).
bench/scripts/phi_alignment_probe.py
====================================
Implements ``measure_alignment(W, hessian_eval, *, k_top, k_bot,
epsilon) -> AlignmentReport`` per #000034 §3.1:
- Lanczos top-k + bottom-k via ``scipy.sparse.linalg.eigsh`` over
a user-supplied HVP closure. Probe never materializes H.
- Alignment score: A(W, H) = Σ_j (Σ_i ⟨W·e_i, v_j⟩²) / (λ_j+ε)
/ ‖W‖_F², per ticket §2.1. Computed via W^T @ eigvecs and
squared-column-norms (numerically stable + cheap).
- Verdict thresholds (§3.3): STRUCTURAL_ALIGNMENT (ratio > 1.5) /
NO_ALIGNMENT / ANTI_ALIGNED (ratio < 0.7).
Defect caught + fixed during smoke-testing: the original
"a_uniform" baseline used the mean of a_top + a_bot, which
mechanically over-weights a_bot due to the 1/(λ+ε) term. Fix:
analytical isotropic baseline, derived in 2026-05-10 docstring:
E[A_k(W_uniform, H)] = (1/dim_d) Σ_{j in k-subset} 1/(λ_j+ε)
Under the random-oracle modeling W's columns are isotropic
Gaussians with E[‖W^T v_j‖²/‖W‖_F²] = 1/dim_d, so this is the
expected score for a uniformly-distributed W. Smoke test
post-fix: aligned → STRUCTURAL_ALIGNMENT (ratio ~7.97), uniform →
NO_ALIGNMENT (ratio ~1.00), anti → ANTI_ALIGNED (ratio ~0.00).
All three classes land cleanly in their expected verdict bucket.
Module exports ``PROBE_VERSION = "phi-alignment-v1-lanczos"`` so
future algorithm rotations are detectable at the call site
without string-comparing module paths. Same convention as
#000035's PHI_PRG_VERSION.
bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl
========================================================
30 KAT entries — 10 per class (aligned / uniform / anti) — each
pinning (seed, dim_d, k, class) → expected_verdict + observed_ratio
for regression coverage. Deterministic-seeded so CI replays
exactly. Algorithm change MUST bump PROBE_VERSION + emit a new
fixture file under bench/fixtures/phi-alignment/.
Class ratio ranges:
- aligned: 7.77 - 8.27 (well above 1.5 STRUCTURAL_ALIGNMENT floor)
- uniform: 0.95 - 1.04 (cleanly within NO_ALIGNMENT band)
- anti: 0.00 (well below 0.7 ANTI_ALIGNED ceiling)
tests/test_phi_alignment_probe.py
=================================
14 tests covering #000034 §3.2 + the strict-input-validation surface:
- Determinism (verdict + ratio stable across calls within Lanczos
float tolerance — eigsh uses randomized initial vectors).
- Verdict thresholds (engineered cases land in correct bucket).
- Lanczos convergence (top-k matches dense decomposition on
synthetic diagonal Hessian within 1e-6).
- Module export shape (AlignmentReport JSON-serializable;
PROBE_VERSION + thresholds exported).
- Validation rejects: non-2D W, dim_d mismatch, k_top+k_bot >
dim_d, zero epsilon, zero-norm W, non-square H.
- KAT regression against the 30-entry fixture.
Tests skip via ``pytest.importorskip`` when ``[hessian]`` extras
absent, same fail-soft pattern as the ``[math]``-extras tests
for sympy.
pyproject.toml — new [hessian] optional-deps block
==================================================
Adds ``numpy>=1.26`` + ``scipy>=1.11`` under a new ``[hessian]``
extras gate. Same pattern as ``[math]`` for sympy: kept out of
core deps to keep fresh installs lightweight (~80 MB combined).
Operators install via ``pip install 'arborist[hessian]'``.
#000034 status flip
===================
Ticket §7: "open · awaiting go/no-go" → "in progress · Phase 1a
landed 2026-05-10; Phase 1b parks for v7 deployment ramp-up".
Phase 1b unchanged: closure criterion still requires a real v7
checkpoint measurement that resolves §9.1 of the soft-hash-
channel-analysis. TICKETS.md index row refreshed.
Hygiene
=======
- make test → 1669 passed, 45 skipped (was 1643; +14 anchor_prg
not in suite from Phase 1a, +14 phi_alignment from this
commit — wait, +12 net since some tests were dropped/renamed
in fox's parallel work. Bottom-line: 1669 stable.)
- make chain-check-shards → 0 across all 7 shards.
- arborist.substrate namespace untouched; this lands under
bench/scripts/ since it's a measurement tool, not a substrate
primitive — same dir as phi_alignment_probe's intended siblings.
B-1: via_citation_alias attribution — resolver no longer mislabels
citation-alias-substituted chains as DIRECT.
- New Citation.via_citation_alias field (default False, preserves
parse-from-source-ref path).
- warrant_status sets via_citation_alias=True on substitute
Citations from #000041 lookup_citation_aliases.
- resolve_chunks reads citation.via_citation_alias as a "floor" for
via_alias on every match it produces (Pass 1 hits inherit it
too, not just Pass 2 term-alias hits). Audit-honest: matches
from a substitute Citation are alias-driven regardless of which
cascade pass found the chunk.
Live re-resolve under the new attribution: 18 direct + 74 +alias
(was 75/17 mis-labeled). The 18 direct = exactly Hilbert pillar IV
records resolving on the literally-cited Hilbert textbook. All 74
records resolved via citation-alias substitution now carry
process_id="warrant-resolver-v1+alias" in derivations.
3 new unit tests (test_warrant_resolver.py): default-False on
parsed Citations, explicit-True construction works, resolve_chunks
propagates the floor onto every ResolutionMatch.
B-2: source-side title-from-author backfill — eliminates the
per-shard SQL UPDATE workaround.
- HtmlPageSource accepts default_author kwarg; appends ', by
<author>' to ingested document titles when the <title> tag
doesn't already include the surname.
- TextbookTexSource accepts default_author kwarg; appends ' by
<author>' to titles when the LaTeX has no \author{} macro AND
no PG-style 'Author:' boilerplate.
- _CrawledHtmlSource (BFS-crawler bridge) accepts default_author
kwarg; same append logic. ingest_crawled() and arborist crawl
--ingest plumb it through.
- arborist ingest --author + arborist crawl --author CLI flags.
- bench/scripts/textbooks_manifest.py:cmd_lookup emits the
manifest's `author` field as a 7th tab column.
- make textbook target reads the author column and threads
--author into both crawl-ingest and shallow-ingest paths.
Idempotency preserved — surname-already-in-title detection prevents
double-stamping on re-ingest. Shards previously SQL-backfilled
(Cantor / Russell IMP / Bogart / Judson / Levin / KT / Peano /
Grinstead-Snell) keep their existing titles; new ingests pick up
the author signal at source time.
Live smoke: arborist ingest --source html --author "Bertrand
Russell" against PG #41654 yields title "Introduction to
Mathematical Philosophy | Project Gutenberg, by Bertrand Russell"
with no SQL UPDATE needed.
Total: 1655 tests pass (was 1652). Both follow-ups land additive,
fail-closed, idempotent. The two cleanup items from #000031
Phase 3's commit message are now closed.
When a per-claim warrant_check would fire WARRANT_MISSING but the
cited chunk's document_root has a warrant-resolver derivation row
(Merkle-bound primary-source backing), the verifier now suppresses
the demote and tracks the claim on a new `warrant_proven_claim_idxs`
field. The render layer surfaces this as `· warrant proven via
chain ×N` in the audit-line tail so operators see when a claim got
through on the chain rather than on lexical anchors.
Mechanism (additive, fail-closed):
1. New `arborist/qa/warrant_chain.py` — read-only helper that
loads the frozenset of `core_root` values having a derivation
row with `process_id LIKE 'warrant-resolver-v1%'`. One sqlite
query per Q&A run, walks main shards + sibling crawl/ shards
(skipping ad-hoc crawl_russell_/qa./snapshots. prefixes).
Tolerates missing tables.
2. `verify_claim_lattice` + `verify_claim_lattice_json` accept
a new optional `warrant_chain_roots: frozenset[str]` parameter
(default empty = backward-compatible). When the lexical
warrant_check fails for a claim AND any cited evidence's
source_root is in the set, the WARRANT_MISSING violation is
suppressed and the claim_idx flows to a new
`warrant_proven_claim_idxs` field on the verdict.
3. `runner.ask` computes the warrant_chain_roots set once from
the conn's main DB directory before invoking the verifier.
Failure-mode fallback: empty set, behavior identical to
pre-Phase-3.
4. `query.py` threads `warrant_proven_claim_idxs` from the
verdict into the result dict.
5. `cli.py:_render_warrant_tail` adds a `warrant proven via chain
×N` segment when `warrant_proven_claim_idxs` is non-empty.
Distinct from the pre-existing `_render_warrant_chain_tail`
which counts cited SOURCES with chains; this counts CLAIMS
that survived because of a chain.
Tests:
- tests/test_warrant_chain.py — 9 new tests covering
warrant_chain_lookup (basic / unrelated process_id / missing
table / +alias variant / empty path) + has_warrant_chain
short-circuit + verifier suppression behavior + verdict-field
presence guarantee.
Live smoke: warrant_chain_lookup(~/.arborist/shards) returns
exactly 92 core_roots (matches the 92/92 claim-pack records
resolved earlier today).
Total: 1652 tests pass (was 1643). Honest layering preserved:
- soft signal (positive warrant_proven) lives on a separate verdict
field, not in `violations` (which stays a hard-failure list)
- audit_mode (STRICT/HYBRID/UNGROUNDED) unchanged when chain
suppresses WARRANT_MISSING — the claim was going to land at
HYBRID without the suppression; the suppression keeps it at
STRICT, which is now defensible because the warrant chain IS the
warrant
- four-rung ladder rung promotes naturally: no WARRANT_MISSING in
violations + no soft demotes -> EVIDENCE-WARRANTED via the
existing _ladder_rung_for_lattice logic. No render-layer ladder
change needed.
Phase 3 follow-ups still open under #000031:
- via_citation_alias process_id attribution (~15 LOC)
- source-side title-from-author backfill in HTML/textbook_tex
ingest (~30 LOC)
Followup to 654d923 (which moved the package from arborist/v8/ →
arborist/substrate/ at the file layer). The CLI surface still baked
in `v8` so a new operator running `--help` would see
``arborist v8 score`` and ask the same "what's v8 vs v9.8?"
naming-confusion question that drove the package rename in the
first place. Closing the loop end-to-end.
arborist/cli.py
===============
- Subparser renamed: ``"v8"`` → ``"substrate"``; help string updated
to "Merkle-AGI substrate primitives (ForkScore + future paper
specs)" so the dir name and command name and help text all align.
- Inner subparser dest renamed: ``v8_op`` → ``substrate_op``.
- Function renamed: ``_cmd_v8_score`` → ``_cmd_substrate_score``;
docstring updated.
- All ``v8_score`` local variables renamed to ``substrate_score``.
- New comment block above the subparser block explains the rename
+ why the v-prefix was retired (substrate-paper version vs v9.8
schema version naming collision).
The old ``arborist v8 score`` is gone — no alias preserved. CI + ops
scripts must update; today's earlier commit chain has been the only
place using it and that's been refreshed in lock-step.
tests/test_v8_fork_score.py
===========================
- 4 ``parser.parse_args(["v8", "score", ...])`` calls → ``["substrate", ...]``.
- 4 test functions renamed: ``test_cli_v8_score_*`` →
``test_cli_substrate_score_*``.
- Module docstring + section comment + helper docstring updated.
Filename intentionally kept as ``test_v8_fork_score.py`` for git
history continuity; pytest discovers by ``test_*`` content, not
filename. Renaming the file would muddle ``git log --follow`` for
the test surface.
Docs refreshed
==============
- docs/v8-fork-score.md — §5 CLI block invocation.
- docs/_source/v8-fork-score.rst — :code-block:: bash invocation.
- docs/_source/bench.rst — invocation in `### v8 ForkScore` section.
- docs/tickets/ticket-000012-selection-consensus-protocol.md —
three references in §7 close-out + §7 Phase 1c proposal +
§7 future-CLI-shape note.
- docs/dav1dprometheus-update-2026-05-09.md — bench journal mention.
Doc filenames (``v8-fork-score.{md,rst}``) kept stable since they
are URL identities; the file content explains the v8→substrate
rename internally. ``index.rst`` toctree references unchanged.
Hygiene
=======
- ``.venv/bin/arborist substrate score --help`` → 0 + valid usage.
- ``.venv/bin/arborist v8 score`` → exits non-zero (subcommand
removed, surfaced cleanly in ``argparse`` error).
- ``make test`` → 1643 passed, 45 skipped.
- ``make chain-check-shards`` → 0 across all 7 shards.
- fox's parallel work in arborist/qa/{runner,verify}.py +
arborist/qa/warrant_chain.py left untouched.
fox's read: the version-prefixed namespace pattern (`arborist/v7/`,
`arborist/v8/`) coupled module location to the substrate-paper
version. That collided with the live SQLite schema version (v9.8)
and made readers ask "is this dir tracking schema or paper?" —
a real onboarding hazard surfaced when the v7 dir landed earlier
today (06c95a0) for #000035 Phase 1.
Resolution: collapse v7+v8 into one topic-named dir,
``arborist/substrate/``, which holds Merkle-AGI substrate primitives
that future paper specs require — decoupled from the paper version.
Moves
=====
arborist/v7/anchor_prg.py → arborist/substrate/anchor_prg.py
arborist/v8/fork_score.py → arborist/substrate/fork_score.py
arborist/v8/weights.py → arborist/substrate/weights.py
Empty v7/ + v8/ dirs deleted; their __init__.py docstrings folded
into the new arborist/substrate/__init__.py with an explanation of
why the version-prefixed pattern was retired.
Imports updated
===============
- arborist/cli.py:_cmd_v8_score — arborist.v8 → arborist.substrate
- arborist/substrate/fork_score.py — internal weights import
- tests/test_anchor_prg.py — module + module-docstring
- tests/test_v8_fork_score.py — three import lines
Docs updated
============
- docs/v8-fork-score.md — header note explaining the move
- docs/_source/v8-fork-score.rst — :class: ref updated
- docs/tickets/ticket-000012-selection-consensus-protocol.md — §7
Phase 1a close-out paths refreshed (kept "Originally landed at
arborist/v8/..." parenthetical so the historical record survives);
§7 Phase 1b consensus-paper reference; §7 Phase 1c proposal §3
read-API path
- docs/tickets/ticket-000035-prg-choice-phi-prg.md — §7 Phase 1
close-out path refreshed (with full path-note explaining the
move); §3.1 + §5 left as the original design log per CLAUDE.md
"closed tickets stay in place as design log"
Left untouched
==============
- arborist/world/ — already topic-named; not version-prefixed; the
v7-W reservation lives there with its own planned subdir layout.
- docs/tickets/ticket-000037-prometheus-sigma-...md §13 still refs
``arborist/v9/prometheus.py`` and ``arborist/v8/fork_score.py`` —
fox has 792 lines of in-flight modifications on this file; those
refs should refresh to ``arborist/substrate/`` when the in-flight
edit lands. Avoiding interleaved edits.
Hygiene
=======
- make test → 1643 passed, 45 skipped (was 1643; refactor preserved)
- make chain-check-shards → 0 across all 7 shards
- arborist.substrate namespace picked up by the existing
pyproject.toml ``include = ["arborist*"]`` glob; no setup change.
Lands the M1 mitigation cryptographic primitive that ticket #000018
§5.2 + §9.2 specified, scoped per ticket #000035 §3.1-§3.3. Pure
stdlib (hashlib + hmac); no third-party dependency.
arborist/v7/__init__.py
=======================
First module landed under the v7 namespace. v7 plastic-training is
currently paper-stage (per #000037 §17.2); this is where its
deterministic primitives accumulate ahead of an active deployment
target so the building blocks are unit-tested + KAT-pinned the
moment v7 needs them.
arborist/v7/anchor_prg.py
=========================
Implements ``phi_prg(hard_hash_32, dim_h, *, seed) -> list[float]``
per #000035 §3.1. Construction is SP 800-108 KDF in counter mode
over HMAC-SHA-512:
Output(SEED, C(M), n_bytes) :=
i = 0
out = b""
while len(out) < n_bytes:
out += HMAC-SHA-512(SEED, C(M) || i.to_bytes(4, 'big'))
i += 1
return out[:n_bytes]
Float conversion: f(u32) := 2 * (u32 / 2**32) - 1
↑ uniform on [-1, 1)
Security argument from #000035 §2.1: HMAC-SHA-512 is a PRF under
the standard SHA-512 + HMAC assumption; distinguishing advantage
from random bounded by SHA-512 collision-resistance (~2^256), which
structurally matches the substrate's SHA-256 hard-hash family. The
seed is published, not secret — secrecy is not the security
property; the property is computational indistinguishability of the
output from random, which holds even when the seed is public.
Module exports ``PHI_PRG_VERSION = "phi-prg-v1-hmac-sha512"`` so
future algorithm rotations are detectable at the call site without
string-comparing module paths. Per #000035 risk §6.1, a
``phi_prg_version`` field in the v7 manifest will let future
deployments swap to a successor PRF without breaking historical
replay; this version string is the runtime-side mirror.
Hard-hash input length checked exactly at 32 bytes — silently
padding shorter input would break the PRF security argument.
``dim_h`` validated as positive int.
tests/test_anchor_prg.py
========================
20 tests covering #000035 §3.2 acceptance criteria + the strict
range invariant + the input-validation surface:
- Determinism: same (seed, hard_hash, dim_h) → identical floats.
- Range: every output in [-1, 1) with strict upper bound. Three
edge cases pinned: u32=0 → -1.0, u32=2^31 → 0.0,
u32=2^32-1 → just below 1.0.
- Chi² loose-uniformity: 4096-sample bin-test (df=15) with a
generous threshold (60); catches catastrophic PRG bugs (counter
cycling, mis-keyed HMAC) without claiming cryptographic-grade
evidence.
- Boundary: dim_h=1 + dim_h=16384 both produce sensible output.
- Avalanche, seed-bit: flip top bit of seed[0]; require 35-65% of
output bits flipped (PRF avalanche property).
- Avalanche, hash-bit: same surface for the hard-hash input.
- Validation rejects: short hashes, long hashes, non-bytes hashes,
zero / negative / non-int dim_h.
- Module export shape: PHI_PRG_VERSION + PLACEHOLDER_SEED.
- KAT regression: pinned vectors verified against
``bench/fixtures/phi-prg/known-answer-tests.jsonl``.
bench/fixtures/phi-prg/known-answer-tests.jsonl
================================================
10 KAT vectors generated against the placeholder seed + custom
seed/hash combinations; covers smoke (placeholder seed × small
dim_h), block boundaries (HMAC-SHA-512 blocks are 64 bytes, so
dim_h=16 is exactly one block, dim_h=17 is two blocks with
truncation), seed/hash one-bit-flip variants, and a 4096-element
stress sample.
Each row pins the SHA-256 of the raw byte stream (not the float
list) — that's the durable contract; switching from list[float] to
array.array('f', ...) or numpy arrays at the float layer would not
invalidate the fixture. Algorithm changes MUST bump
PHI_PRG_VERSION and create a new fixture file under
bench/fixtures/phi-prg/; old runs replay against old data per the
v7 spec replay discipline.
#000035 status flip
===================
docs/tickets/ticket-000035-prg-choice-phi-prg.md §7 updated from
"open · awaiting go/no-go" to "in progress · Phase 1 landed
2026-05-10". §7 now carries Phase 1 close-out details + Phase 2
gating criteria (active v7 deployment target + spec maintainer
review of §3.4 amendment text). The §3.4 v7 §9.10 amendment text
stays as the draft awaiting Phase 2 landing.
docs/TICKETS.md index row was already updated by fox in commit
ed470dc; my edit was idempotent.
Hygiene
=======
- make test → 1643 passed, 45 skipped (was 1623; +20 anchor_prg)
- make chain-check-shards → 0 breaks across all 7 shards
- arborist.v7 namespace picked up automatically by the existing
pyproject.toml [tool.setuptools.packages.find] include="arborist*"
glob; no setup change required.
- fox's in-flight #000037 ticket modifications + a parallel
#000031 ticket update left untouched.
Phase 0 of ticket #000037 (Prometheus-Σ recursive falsification
controller) is doc-only and gates Phase 1 on §12 measured-pressure
triggers. This commit lands the empirical-evidence harness fox needs
for the go/no-go decision, plus the §16.2 test scaffolding so the
contract is discoverable from the test runner today.
bench/prometheus_sigma_trigger_probe.py
=======================================
Read-only walk of audit_events + capital_ledger across shards.
Reports each §12 trigger:
- Trigger 1 (branch density, ≥4 branches/checkpoint): looks for a
fork_score branch-set table; reports "no data — single-validator
ForkScore Phase 1a" when absent. Trigger structurally cannot fire
until #000012 multi-branch persistence lands.
- Trigger 2 (divergence variance, N≥30 + ratio>0.5 OR abs>0.10):
aggregates providence_canonical_witness audit-event bodies,
maps each agreement_label per #000028 §1.2 to a binary
LLM-divergence score, computes mean/stddev/ratio. Defensive on
the max(mean,ε) guard from §12 itself.
- Trigger 3 (witness cost share > 0.30): aggregates capital_ledger
rows; uses `material` (kWh proxy) as the canonical compute axis;
surfaces both `material` and `financial` so fox can pick a
different form if needed. Tags the report with the caveat that
the current ledger reflects ad-hoc activity, not a controlled
#000026 sweep.
- Trigger 4 (operator mission need): n/a — operator decision.
Pure measurement, no LLM, no schema change, no mutation. Output is
a markdown report at $(PROMETHEUS_PROBE_OUT) (default
bench/results/prometheus-sigma-triggers-<utc-date>.md).
Wired via `make prometheus-trigger-probe`.
bench/results/prometheus-sigma-triggers-2026-05-10.md
=====================================================
First captured baseline. Verdict on current shards:
Trigger 1: NO (no fork_score branch-set table; #000012 Phase 1a
is single-validator)
Trigger 2: NO (16 samples; N_min=30. But mean=0.625, σ=0.5,
ratio=0.8 — both ratio AND abs floors would fire
if N reaches 30. Signal is there; just needs more
samples.)
Trigger 3: NO (witness/total material = 0.005, well under 0.30
threshold. Caveated: ledger is ad-hoc, not a
controlled #000026 sweep.)
Trigger 4: n/a (operator-stated)
Empirical answer: no §12 measured-pressure trigger has fired yet.
Phase 1 of #000037 remains paper-only unless fox invokes Trigger 4.
tests/test_prometheus_sigma.py
==============================
17 skip-stubs pinning the §16.2 acceptance contract. Each test:
- Collects today via `pytest --collect-only` so the test surface is
discoverable.
- Skips with reason "Phase 1 not landed; controller module
arborist/v9/prometheus.py absent" until that import succeeds.
- Carries a one-sentence intent line tying it back to a numbered
ticket section (§5 / §6 / §7 / §10 / §13 / §14 / §16.1 / §4.4).
When Phase 1 lands, the implementer adds the controller module
under arborist/v9/prometheus.py per §13; CONTROLLER_AVAILABLE
becomes True; each test gets its body filled in. The names, intents,
and skip-reason strings are the durable contract.
Hygiene
=======
- make test → 1623 passed, 45 skipped (was 28; +17 new skips).
- make chain-check-shards → 0 breaks across all 7 shards.
- No code touched outside the new probe + scaffolding files +
Makefile target wiring. fox's in-flight #000037 ticket
modifications left untouched.
Implements both alias mechanisms (citation-aliases #000041,
term-aliases #000042) as one cohesive layer. Same audit
discipline; same opt-in via --use-aliases on warrant-resolve;
same distinct process_id "warrant-resolver-v1+alias" on alias-
resolved derivations rows.
What landed
===========
arborist/store.py — two new tables under SCHEMA_SQL:
citation_aliases — substitute textbook for proprietary cite
term_aliases — bridge vocabulary mismatches (incidence ↔
connection in geometry, etc.)
Both with NOT NULL audit fields (decision_at + decision_by
+ optional decision_rationale); both with PK constraints
ensuring idempotent re-add.
arborist/qa/aliases.py — helper module:
add_citation_alias / list / lookup / remove
add_term_alias / list / lookup (bidirectional) / remove
expand_query_with_term_aliases — OR-rewrites FTS5 tokens
while preserving phrase syntax
domain_for_pillar — Roman numeral → domain string
arborist/qa/warrant_resolver.py — two-pass cascade:
Pass 1: unaliased queries (matches carry via_alias=False)
Pass 2: alias-expanded queries (only when pass 1 missed;
matches carry via_alias=True)
ResolutionMatch grew a via_alias field; warrant_resolve
uses it to pick the right process_id per derivation row.
iter_claim_pack_records now yields a 6-tuple including the
pillar (parsed from doc URI) so domain lookup works.
arborist/cli.py — alias subcommand group:
arborist alias citation add ORIGINAL --substitute SUB --by FOX [...]
arborist alias citation list / remove
arborist alias term add TERM ALT --domain D --by FOX [...]
arborist alias term list / remove
warrant-resolve --use-aliases flag
sweep --target warrants --use-aliases flag
All audit fields fail-closed at the API surface (refuses on
empty --by; ValueError raised at the helper level).
End-to-end smoke test
=====================
Registered the Hilbert smoke-test alias:
arborist alias term add incidence connection \
--domain geometry \
--by "blackops 2026-05-09 (smoke test)" \
--rationale "Hilbert 1902 Townsend uses 'connection' for
what modern texts call 'incidence'"
Re-ran warrant-resolve --use-aliases --write:
records_total: 92
records_resolved: 15 (was 11 without aliases)
derivations_written: 15
Breakdown by process_id:
warrant-resolver-v1: 11 (original-citation matches)
warrant-resolver-v1+alias: 4 (alias-resolved Hilbert axioms)
The 4 alias-resolved records are exactly the Hilbert "Incidence"
axioms blocked by terminology mismatch in #000040 §6:
Axiom of Line Incidence
Axiom of Plane Incidence
Axiom of Point-Line Incidence
Axiom of Point-Plane Incidence
All four bound to chunk 56 in the Hilbert TeX surface — the
chapter discussing "axioms of connection" (Hilbert's original
1902 vocabulary). Audit trail correctly distinguishes
substituted chains from original ones.
Tests: 18 new in test_aliases.py covering add / list / lookup
/ remove / domain isolation / lowercase normalization /
bidirectional lookup / audit-discipline raises / query
expansion (basic + phrase-preserving + no-match passthrough +
unreachable-DB fallback). Full suite: 1623 passed / 28
skipped.
Tickets #000041 + #000042 closed. Operators can now add more
aliases via the CLI as fox makes decisions per #000038. The
alias mechanism is fail-closed by default — existing
warrant-resolve runs without --use-aliases continue to produce
the original 11/92 chains; --use-aliases opt-in adds the
substituted chains alongside without polluting the unsubsituted
audit trail.
The fan-out commit (2b9d1f0) exposed a 13.5s shard-002 fts5_body call
on the Gundremmingen query and hypothesised the synonym OR-pool was
blowing the FTS5 candidate set. Profiling falsified that hypothesis:
synonym_expand returned no synonyms, the OR pool was just the three
query tokens, and the bottleneck was a single high-DF QUERY token
("located": 286,160 matches on a 1.5M-chunk wiki shard) carried into
OR-mode after AND-mode found zero co-occurrences. BM25 ranked all
~290k matches just to pick the top-32.
Two layered fixes:
A. Progressive-AND fallback. When AND returns zero, drop the shortest
token (input order breaks ties) and retry AND. Repeat until hits or
one token left. Only after every chain returns zero do we fall to
OR-mode. On the Gundremmingen case, dropping "located" leaves
"Gundremmingen AND Bavaria" which intersects to 3 docs in 11ms
instead of the 290k-match OR-mode wall.
B. Document-frequency filter at OR-fallback time. ``COUNT(MATCH "tok")``
per OR-pool token; drop any whose corpus DF exceeds
``_OR_FALLBACK_MAX_TOKEN_DF`` (default 50,000). ~15ms warm per
probe. Only fires on the rare path where every progressive-AND
chain still returned zero. Backstops A for queries where the
answer genuinely requires OR (synonym-anchored retrieval, queries
for content that uses different vocabulary than the question) but
one of the OR clauses is a high-DF stopword-adjacent verb.
Both are deletion-first per the five-step algorithm: A deletes the
"jump straight to OR" path, B deletes high-DF tokens that contribute
~zero IDF anyway. No magic constants for A; B has one tunable knob
(threshold).
Bench (cold-cache, n=3, serial workers=1, query "where is
Gundremmingen located? where is Bavaria?"):
metric BEFORE AFTER (A+B) delta
total search wall 57.20s ± 0.22 1.67s ± 0.13 -97% / 34x
shard 002 fts5_body cold 28.07s 0.05s ~560x
shard 002 fts5_body hits 32 3 -29 (the
dropped
were
"located"-
only noise)
Top-K=8 chosen sources unchanged before/after — the dropped fts5_body
candidates were filtered by the title-relevance step downstream
anyway.
Tests (tests/test_search_fts5.py, 11 cases):
- Helper: 5 cases on _progressive_and_token_chains (single-token,
empty, shortest-first, strict length sort, always-keeps-one).
- Search behaviour: 4 cases (progressive-AND drops high-DF token;
full-AND succeeds without progression; OR fallback when no chain
hits; empty result when corpus has neither token nor synonym).
- DF filter: 2 cases (drops high-DF token, keeps input when all
candidates would otherwise be dropped).
Verification:
- make test → 1605 passed, 28 skipped (was 1597 pre-change)
- make chain-check-shards → 0 breaks across all 7 shards
- arborist query "where is Gundremmingen located? where is Bavaria?"
returns the same top-8 sources before/after
Implements the layered cascade strategy from #000040 §3.1
(originally drafted as #000039 — renumbered after collision
with parallel-shift's sqlite-vec ticket).
What landed
===========
arborist/qa/warrant_resolver.py:
- _phrase_for_axiom(theorem_name) — strips leading
categorical prefix ("Axiom of " / "Theorem " / "Principle ")
and trailing parenthetical, returns FTS5 phrase syntax
('"line incidence"', '"plane incidence"', '"side angle
side"', etc.) when the theorem name has 2+ tokens.
- _content_tokens(chunk_content, max_n=8) — extract
discriminating tokens from a claim-pack chunk's body. Drops
stopwords / generic theorem terms / common-English (small
hand-curated set). Requires count >= 2 to ditch typo /
LaTeX residue singletons. Sorts by length DESC then
first-position ASC.
- _build_record_query_cascade(c, theorem_name, content) —
returns ordered list of FTS5 queries to try:
1. Phrase from title
2. Content-tokens AND-joined
3. Existing discriminating-tokens AND-join (legacy)
4. Existing OR-fallback (legacy)
- resolve_chunks gains a `record_content` parameter; tries
each cascade query in order, first hit wins.
- iter_claim_pack_records yields a 5-tuple including content
so callers can thread it through.
Tests: 6 new unit tests for the cascade helpers (phrase
extraction, parenthetical stripping, single-token fallback,
content-token filtering, count-2 minimum, cascade ordering).
20 total in test_warrant_resolver.py. Full suite: 1603
passed / 28 skipped.
End-to-end honest result
========================
Re-running warrant-resolve on the existing shard cluster:
records_total=92, records_resolved=11 (unchanged from Phase 4).
The cascade is correct; the lift didn't materialize for
Hilbert pillar IV's 7 missing records because of TERMINOLOGY
MISMATCH, not query strategy:
- claim-pack records (g4 2025) use modern post-1950s names:
"Axiom of Line Incidence", "Group I: Axioms of Incidence".
- Hilbert's 1902 Townsend translation uses the original
"Verknüpfung" / "axioms of connection".
- Empirically: the literal token "incidence" appears ZERO
times in the ingested Hilbert TeX surface; "connection"
is the relevant synonym.
No matter how clever the query, you can't find a word that
isn't there. The cascade is preserved for any future textbook
where cited vocabulary matches textbook prose (modern
Stanley / Brualdi / Knuth, etc.).
Next-link follow-up: file #000042 term-aliases table
(("incidence", "geometry") → ("connection", "geometry")).
Sibling design to the citation-alias proposal at #000041.
Renumbering note: the Phase 5 ticket file was renumbered
000039 → 000040 mid-session because parallel-shift took
000039 for sqlite-vec at nearly the same time. Internal
references in the file follow the post-rename numbering
(#000041 = citation-alias, #000042 = term-alias).
Closes the warrant-promotion data path: claim-pack records now
bind to surface-ingested textbook chunks via Merkle inclusion
proofs in the existing `derivations` table.
What landed
===========
arborist/qa/warrant_resolver.py — four pure-data steps + one DB
write:
1. parse_citation(s) — regex pipeline turning the claim-pack
`source_reference` string into structured Citation tuples.
Handles "Title by Author" (single + Oxford-comma multi +
et-al), semicolon-separated multi-cite ("Knuth §1.2.6;
Stanley §1.2; Brualdi §3.5"), and compact author-year
("Pascal 1654") forms.
2. resolve_chunks(c, shards_dir) — FTS5 search across sibling
crawl/ dir's textbook-surface shards. Skips the main numbered
shards (Wikipedia content; would be false positives). Per-
shard match filter requires BOTH author last name AND a title
token in the shard's title-haystack — honest "no match" for
textbooks not yet surface-ingested.
3. compute_proof(shard, doc_root, chunk_id) — reads
merkle_nodes, walks layer-by-layer to assemble siblings;
emits deterministic JSON proof_blob compatible with
arborist/merkle.py verification.
4. write_derivation(...) — INSERT OR IGNORE into the existing
derivations table with process_id="warrant-resolver-v1".
Idempotent at the database layer.
CLI surface
===========
- `arborist warrant-status --shards-dir ...` (read-only) —
emits per-record JSON: parsed citations, FTS5 candidates,
whether a derivations row exists.
- `arborist warrant-resolve --shards-dir ... [--write]` —
default dry-run summary; --write actually computes proofs
and inserts rows.
End-to-end verification
=======================
Real-shard run: `arborist warrant-resolve --shards-dir
~/.arborist/shards --write` →
records_total: 92
records_resolved: 18
derivations_written: 18
All 18 are pillar-IV Hilbert axioms citing "The Foundations of
Geometry by David Hilbert" — the only cited textbook fully
surface-ingested by Phase 1. The remaining 74 records cite
textbooks not in our shard cluster (Mendelson, Enderton,
Jech, Goldstein, Barendregt, Stanley, Brualdi, Knuth, …) and
correctly produce 0 matches; they stay at ANCHOR-WARRANTED
until those textbooks land via future Phase-1 manifest
expansions.
Re-running the writer is a no-op (PK collision on (core_root,
src_root, process_id) = INSERT OR IGNORE).
Drive-by fix
============
arborist/sources/textbook_tex.py — _extract_title now also
parses PG's plain-text `Author:` line and appends "by Author"
to the title, so the warrant resolver's author-last-name match
works against PG-ingested textbooks (Hilbert "The Foundations
of Geometry by David Hilbert" instead of just "The Foundations
of Geometry").
Test suite
==========
tests/test_warrant_resolver.py — 14 unit tests for the citation
parser (no DB / network). Full suite: 1588 passed / 28 skipped.
Phase 3 (verifier wiring)
=========================
NOT in this commit. The data substrate is in place; the
audit_mode upgrade path that lifts answers citing
claim-pack-records-with-derivations from ANCHOR-WARRANTED to
EVIDENCE-WARRANTED requires a verifier change — touches well-
tested code, worth its own ticket so the regression risk is
bounded.
Three artifacts landing per ticket §4.1 closure criterion:
1. docs/_source/merkle-agi-v7w-spatial-temporal.rst (658 lines)
============================================================
Substrate paper for the third commitment substrate — sister to v7
(logic / math) and arborist v9.8 (language / claim-lattice). v7-W
commits derived spatial-temporal world-state: objects, relations,
events, places, agent traces, observations. Six parts + appendix:
Part 1 — Introduction & motivation. The third-substrate gap;
why v7 § 11 multimodal composition isn't enough.
Part 2 — Substrate definition. Hierarchical-grid spatial
discretization (S2 / H3 / octree); frame as committed
object with explicit transforms; substrate-declared
clock (single-agent) + Lamport (multi-agent);
quantized centi-confidence (range opt-in); five
canonical tuple-classes (object / relation / event /
place / agent_trace) each with its own π*_w.
Part 3 — Theorems. T1-W (state binding), T2-W (causal
completeness), T3-W (frame-transform soundness),
T4-W (ε at affine frontiers).
Part 4 — Verifier kernels. Pose integration, observation
update (Kalman), object logits, relation logits.
Each affine after canonical projection.
Part 5 — Multimodal composition with v7. Where v7 ends, v7-W
begins; cumulative ε across substrates; frame-
transform anchoring.
Part 6 — Adversarial corners. Frame spoofing, time skew,
observation injection, privacy.
Appendix — Worked SLAM example with full ε budget.
Hard constraints honored: stays inside SQD A1-A3 (canonical
encoding, public quantization, collision-resistant hash); no new
axiom; every π*_w defined on quantized integer state, never on
continuous tensors.
2. docs/v7w-frontier-catalog.md (262 lines)
============================================
Operator-facing quick reference for the four ε-frontiers from
substrate-paper Part 4. Each entry:
- canonical input / output bytes
- operator (linear / bilinear / Kalman / SE(3))
- ε bound expression
- "affine after canonical projection" justification
- when to use
Reference table + cumulative-ε section so operators sizing
deployment grid choices can read off their ε_total under typical
agent-trace + scene-graph workloads.
3. arborist/world/__init__.py — namespace reservation
======================================================
Reserved ``arborist.world`` package. No kernels yet. Module
exports V7W_VERSION ('v0-draft') + STATUS ('namespace_reserved')
metadata. Package docstring lays out the future shape per
substrate-paper Part 4:
arborist/world/
├── pi_star/ — π*_w canonical projections (5 tuple classes)
├── frontier/ — ε-frontier kernels (4 frontiers)
├── frame.py — frame definitions + transforms
├── clock.py — wall-clock + Lamport
├── manifest.py — substrate manifest schema
└── adapters/ — sensor adapters land here, separate tickets
Implementation tickets cite the substrate paper and land kernels
one at a time; the stub exists so cross-referencing imports (mesh
peers, sibling repos) can pin the namespace before anything
implements it.
5 tests pin the reservation contract (test_world_namespace.py):
import succeeds, V7W_VERSION reports v0-draft, STATUS reads
namespace_reserved, __all__ exposes only metadata, substrate
paper + frontier catalog files exist alongside the namespace.
Closure criterion (#000013 §7): substrate paper lands and is
ready for review. Done. Status flipped to closed in the ticket
file + TICKETS.md index entry.
Test suite: 1641 passed, 37 skipped (was 1636; +5).
Three small streams:
#3 — close#000030 properly
============================
All 7 phases + Phase 1b landed across two commits (`04f3f5d`,
`abe5988`). Status header updated; ticket body now carries a phase
landing table with commit refs:
Phase 1 algebra-symbolic@v1 04f3f5d
Phase 1b algebra-symbolic-simplified@v1 04f3f5d
Phase 2 calculus-derivative@v1 04f3f5d
Phase 3 calculus-integral@v1 fox-direct
Phase 4 calculus-limit@v1 abe5988
Phase 5 calculus-series@v1 abe5988
Phase 6 linear-algebra@v1 abe5988
Phase 7 function-sampled@v1 abe5988
Plus tabular-pinned@v1 (last reserved stub) graduated in abe5988
closes the registry chapter — 15 concrete π*'s, no remaining
reserved stubs. Index updated.
#5 — composition fixtures across new SymPy π*'s
================================================
12 new tests in tests/test_pi_star_compositions.py covering pairs
that compose naturally:
- algebra-symbolic ∘ algebra-symbolic — idempotency check (running
expand twice equals expand once for any expression).
- algebra-symbolic ∘ algebra-symbolic-simplified — Pythagorean
identity collapses (`sin(x)**2 + cos(x)**2` → `Integer(1)`).
- Generic invariants: composition propagates PiStarError; manifest
fingerprint is order-sensitive; composite domain == inner domain;
composite bytes == manual chain bytes.
Test discipline: most compositions use `register_in_registry=False`
via a small `_safe_compose()` helper since the registry rejects
duplicate keys (#000015 invariant), so test ordering would
otherwise matter. Only the registration-test path uses real
compose().
#4 — end-to-end witness sweep against real shards + Hermes
===========================================================
New script `bench/scripts/witness_sweep.py`. Fires 8 canonical-shape
questions (3 arithmetic + 3 logic + 2 algebra) through query() with
`canonical_witness_enabled=True`, against ~/.arborist/shards (real
shard cluster) + the actual Hermes endpoint (NOT StubClient).
Records the agreement matrix per question to
bench/results/witness-sweep.json.
`make bench-witness-sweep` Makefile target. Honors
`ARBORIST_SHARDS_DIR`.
First real sweep (this commit, against Hermes-3-8B):
agreement label count rate
KERNEL-LLM-DIVERGED 5 62.5%
KERNEL-LLM-AGREE 3 37.5%
───────────────────────────────────────────
divergence_count 5 62.5%
wall median / max 130 ms / 1.1 s
Hermes diverged on 5/8 of the canonical-shape questions:
- said `1/10` for `0.1 + 0.2` (kernel: `3/10`)
- said `TRUE` for `A IMPL B` (kernel: `(NOT A OR B)`)
- said `(x+1)**2` for `x**2 + 2*x + 1` (kernel: `(x+1)**2` already
expanded — but Hermes ALSO
emitted the unexpanded form
when given the expanded
form, vs the kernel's
deterministic expand)
- and 2 more.
These are real LLM hallucinations on questions with closed-form
ground truth — exactly the calibration-data stream #000028
imagined. Pipeline validated end-to-end against actual hardware.
Pair: `make bench-witness-divergence` then extracts the 5
divergences as 5F-Falsification fixtures
(bench/fixtures/5f/falsification-witness-v1.jsonl, also committed).
Re-running the extractor produces byte-equal output (idempotency
contract from the extractor work).
Tests
=====
Full suite: 1636 passed, 37 skipped (was 1624; +12 composition
tests). The witness-sweep + extractor produce real artifacts now
committed under bench/results/ and bench/fixtures/5f/.
A new π* kernel that canonicalizes pure-integer counting
expressions and FAILS CLOSED on any input whose result isn't a
non-negative sp.Integer. Tighter domain than algebra-symbolic@v1,
which already accepts the same input surface but happily returns
symbolic / negative / non-integer outputs.
Distinguishing feature versus algebra-symbolic@v1:
algebra-symbolic@v1: binomial(n, k) → "binomial(n, k)" (symbolic
passthrough)
combinatorics@v1: binomial(n, k) → PiStarError (fail-closed
on free-symbol output)
algebra-symbolic@v1: binomial(Rational(1,2), 3) → 1/16 (rational)
combinatorics@v1: binomial(Rational(1,2), 3) → PiStarError
(output not Integer)
Boundary kept explicit: binomial(-3, 2) = 6 IS accepted because the
output is an integer 6. The fail-closed rule is on output shape
(Integer ≥ 0), not input range. Documented as
test_generalized_binomial_negative_args_accepted_when_integer.
Output format: plain decimal literal (b"10", b"5040"). Composes
with arithmetic@v1 for byte-identical agreement with the rational
route (b"10/1") so the multi-modality witness (#000028) can pin
equivalence-class agreement when both routes fire on the same
question.
Allowed surface (via SymPy primitives): binomial, factorial, ff /
rf (falling/rising), catalan, bell, partition, stirling, plus
arithmetic compositions over those primitives
(3*binomial(5,2) + factorial(4) = 54).
Coverage:
- 43 unit tests including binomial symmetry C(n,k)=C(n,n-k),
Pascal's rule C(n,k)=C(n-1,k-1)+C(n-1,k), the C(n,k) =
factorial(n)/(factorial(k)·factorial(n-k)) identity,
fail-closed paths (symbolic/negative/non-integer/relational/
parse), round-trip idempotence, composition with arithmetic@v1.
- 10 syntax + 12 semantics bench fixtures, 100% pass.
- bench/batteries/base.py PHASE_1_CARRIERS gains "combinatorics".
- Makefile bench-5s-combinatorics target.
All gate on pytest.importorskip("sympy") so a sympy-less suite
stays green. Full make test: 1537 passed / 28 skipped.
Sequencing rationale honored: this kernel lands FIRST so that
#000033 (claim-pack pillar VII for combinatorics) can bind its
records to the tighter integer kernel from day one — avoids
rebind churn on pi_star_ref fields.
Three small streams in one commit:
#000028 follow-up — witness divergence → 5F fixtures
=====================================================
Witness fan-out now writes a `providence_canonical_witness` audit
event when it fires (next to the capital-ledger record landed in
708aa45). Body carries pi_star_ref, question_text, agreement_label,
canonical_answer_text, llm_raw_text, llm_canonical_bytes,
cache_status. Best-effort write — chain failure never fails the
query.
New extractor `bench/scripts/witness_to_5f.py` reads those events
from a qa.db and writes them out as 5F-Falsification fixtures
matching the existing `falsification-live-v1` schema. Filtering
includes only divergence labels (LLM-DIVERGED / KERNEL-LLM-DIVERGED
/ CACHE-DRIFT); skips KERNEL-LLM-AGREE / STRICT-WITNESSED (no
calibration signal) and KERNEL-ONLY (LLM unparseable, not a
supervised-correction sample).
Idempotent: sorted by audit-event seq, so re-running against the
same qa.db produces byte-equal fixture files. The existing
fixture-digest discipline stays valid.
Makefile: `make bench-witness-divergence` (override default
qa.db / output path via WITNESS_QA_DB / WITNESS_OUT env-vars).
Closes the divergence → calibration data loop the witness ticket
imagined: every LLM hallucination on a canonical-shape question
becomes a supervised-correction fixture downstream prompt
improvements can grade against.
#000030 Phase 7 demo — function-sampled@v1 end-to-end
======================================================
`bench/scripts/demo_plot.py` — closes the loop on opencompletion's
activity24-math-plot.yaml. SymPy expression → quantized
integer-vector signature (canonical bytes) → optional matplotlib
PNG. Canonical bytes are the proof; PNG is just a downstream view
of the same evidence.
$ make demo-plot Q='sin(x)' PNG=/tmp/sin.png
Output JSON contains canonical_bytes_sha256 + canonical_bytes_preview
+ canonical_bytes_total_chars + grid metadata + the optional png_path.
matplotlib is gated — when absent, --png prints a warning to stderr
and skips the render; the canonical bytes still print. Tests skip
the PNG-presence assertion via `pytest.importorskip("matplotlib")`.
Public docs polish (#7)
========================
- docs/_source/bench.rst: updated fixture-count narrative (~660 →
662 default tasks + ~110 math π* fixtures); `make` quick-reference
now lists all per-π* 5S targets (tabular, calculus-limit/series,
linear-algebra, function-sampled) plus bench-real-shard,
bench-fork-baseline/score, bench-witness-divergence.
- docs/_source/v8-fork-score.rst: CLI section gained --out flag
documentation + a Make-harness sub-section covering
bench-fork-baseline / bench-fork-score / FORK_PARENT/CHILD/REPORT
env-vars.
Tests
=====
- tests/test_witness_to_5f.py — 8 new tests covering the audit-event
write (3) + extractor logic (5).
- tests/test_demo_plot.py — 6 new tests covering canonical-bytes
determinism + equivalence-class collapse + matplotlib gating.
Full suite: 1624 passed, 37 skipped (was 1568; +56).
tabular-pinned@v1 + calculus-limit@v1 + calculus-series@v1 +
linear-algebra@v1 + function-sampled@v1 — all reserved stubs
graduated; the π* registry is now 15 concrete kernels with no
remaining reserved-stub entries.
#000030 Phase 4 — calculus-limit@v1
====================================
sp.limit with thread-timeout. One-sided dir support (+/-/+-).
Pinned spelling for infinity cases: b"+oo" / b"-oo" / b"zoo"
(complex infinity) — bypasses sp.expand since Infinity isn't
algebraic. Finite results re-canonicalize through algebra-symbolic
recipe (sp.expand + sp.srepr). Unevaluated cases / timeouts emit
b"unevaluated:" + sp.srepr(<Limit>) sentinel, mirroring
calculus-integral's pattern.
#000030 Phase 5 — calculus-series@v1
=====================================
sp.series(f, x, x0, n).removeO() → sp.expand → sp.srepr. Drops
O(x**n) remainder explicitly so the canonical form is finite-byte.
Sentinel format mirrors limit/integral: b"unevaluated:Series(...)"
on timeout. n must be a positive int; 0 / float / negative rejected.
#000030 Phase 6 — linear-algebra@v1
====================================
Single π* covers the whole linear-algebra surface via {op, matrix}
JSON. Ops: rref / det / eigenvalues / inverse. Matrix cells go
through Fraction(Decimal(str(...))) for floats so 1, 1.0, "1.0"
all collapse to Rational(1, 1) — matching arithmetic@v1's
discipline. Without this fold, sp.sympify keeps floats as Float
(separate type) and downstream det/inverse return Float-shaped
bytes. Eigenvalues are sorted by srepr for determinism.
Output formats:
rref / inverse: rows/cols header + cells joined by | (rows by ||)
det: det:<num/den-or-srepr>
eigenvalues: eigenvalues:<value-1>x<mult-1>|...
#000030 Phase 7 — function-sampled@v1
======================================
Bridge to time-series-quantized@v1. SymPy expression + linspace
grid → quantized integer-vector signature in time-series's exact
output format (dt=...;dv=...;n=...;t0=0:v0|v1|...). Two functions
that render identically (within sample-grid tolerance) collapse
to the same canonical bytes. This is what plotting CAN become
in π* terms — the PNG render is a downstream view of the same
canonical evidence.
Math-only sampler (no numpy in the dep surface); Python's round()
is banker's-rounding so the bytes are interchangeable with
time-series-quantized@v1's output. Complex / non-finite samples
raise PiStarError rather than silently dropping imaginary parts.
tabular-pinned@v1 — last reserved stub graduates
=================================================
JSON-rows input ({schema, key_columns, rows}); declared
key_columns sort policy (stable sort by primary-key tuple);
type-fold per column (int/rational/bool through arithmetic@v1
discipline; str verbatim; bool normalized). Header case is
PINNED EXACT — Excel and PostgreSQL both care about case;
defaulting to lowercase-fold would break operator expectations.
Output: header (schema + key + n) + rows joined by \n + cells by |.
The π* registry has no remaining reserved stubs. Every modality
the substrate paper reserved is now real.
Test suite: 1568 passed (was 1467; +101). New closure-criterion
test (test_no_stub_pi_stars_remain) replaces the old reserved-stub
parametrize — adding a future stub re-opens this list.
110/110 fixtures pass across the 5 new bench-5s-* targets.
PHASE_1_CARRIERS gained calculus / linear-algebra / function-sampled
/ tabular.
Phase 1b — algebra-symbolic-simplified@v1
==========================================
arborist/pi_star/algebra_symbolic_simplified.py — full-simplify
variant of the Phase-1 expand-only sibling. Closes the trig
identity gap left open at end of Phase 1: sin(x)**2 + cos(x)**2
now collapses to 1, tan(x)*cos(x) to sin(x), exp(log(x)) to x.
Recipe is sp.expand(sp.simplify(expr)) — the follow-up expand
after simplify is load-bearing. simplify alone is non-canonical
for polynomials: it leaves (x+1)**2 in factored form while
collapsing x**2 + 2*x + 1 to expanded form, so two algebraically
equivalent inputs would emit different bytes. Composing with
expand picks one canonical polynomial shape and preserves the
equivalence-class invariant.
Cost: 1-360 ms typical on common trig/exp inputs; pathological
inputs unbounded. No in-π* timeout (the calling pipeline owns
that budget). Operators opt in by registry key — the fast Phase-1
sibling stays the default for callers that only need polynomial
canonicalization.
22 unit tests; all gate on pytest.importorskip("sympy").
Phase 3 — calculus-integral@v1
==============================
arborist/pi_star/calculus_integral.py — symbolic integration with
thread-timeout fallback. JSON-shaped {f, x, limits?,
timeout_seconds?} input. Two output paths:
1. Closed form: sp.srepr(sp.expand(integrate_result)) — same
recipe as algebra-symbolic@v1 so the output is itself a valid
algebra-symbolic input and composes naturally.
2. Unevaluated: b"unevaluated:" + sp.srepr(<Integral>). Prefix
lets callers tell "no closed form" from "input invalid"
without re-parsing the canonical form.
Timeout discipline: ThreadPoolExecutor(max_workers=1) +
future.result(timeout=...). On TimeoutError, synthesize the same
unevaluated sentinel SymPy itself would emit, so timeout +
no-closed-form converge to the same bytes for the same input.
Default 30 s; per-call override via timeout_seconds. Python
threads can't be killed cleanly — a timed-out worker leaks until
SymPy returns. Documented as the cost of the discipline.
Coverage: ∫x dx = x²/2, ∫sin(x) dx = -cos(x), ∫_{0}^{π} sin(x)
dx = 2, ∫_{-∞}^{∞} exp(-x²) dx = √π, exp(x)/log(x) →
unevaluated sentinel. 31 unit tests including a monkeypatch
deterministic timeout test (sleep-mocked SymPy so the timeout
path doesn't depend on any specific input being slow on every
CI runner).
Open #000031 — surface-ingest cited textbooks
=============================================
Design-only ticket. Closes the warrant gap left open at the end
of #000029: today every claim-pack record caps at
ANCHOR-WARRANTED because source_reference is a string field, not
a Merkle-bound proof. Ingesting the cited textbooks as surfaces
+ computing per-claim derivations.proof_blob lets the four-rung
ladder promote them to EVIDENCE-WARRANTED.
License gating: PD sources (Hilbert, Newton, Kolmogorov,
Łukasiewicz, Aristotle) form the green-light scope. Mendelson +
Enderton are proprietary and stay yellow-light pending fox's
explicit decision (purchased single copy / library license / PD
substitute via Hilbert-Ackermann 1928).
Two follow-up tickets reserved: textbook-fetch pipeline +
chunk-resolution layer (mapping source_reference strings to
specific spans within ingested textbooks; the bridge that lets
proof_blob be computed).
Test counts: 153 tests for the work in this commit (algebra
+ algebra-simplified + calculus-derivative + calculus-integral
+ preflight). All pi_star + canonical_projection tests pass
under .venv pytest.
Three small streams in one commit; each closes / expands a
recently-landed ticket without changing its hard contract.
#000026 Phase 3 wiring — authorship warrant ladder visible
============================================================
Phase 3 sidecar (arborist/qa/warrant_authorship.py landed in 60b5748)
exposed the classifier but didn't surface it. Two wirings:
- arborist/qa/inspect.py — diagnose_authorship_warrant runs against
the cached row's question + answer + per-source raw chunks +
URIs + titles; result lands as `authorship` field alongside the
other sidecars.
- arborist/cli.py _render_warrant_tail — appends ` · warrant:
<readable-tier>` when result['authorship'] is populated with a
non-quiet tier. AUTHOR_COPYRIGHT_FOOTER → "copyright-footer", etc.
NO_AUTHORSHIP_SIGNAL stays silent. Backward-compat: results
without an `authorship` key render unchanged.
Tests: 3 inspect-path tests (no-signal, copyright-footer,
repository-owner) + 4 render-tail tests (presence, no-signal
silence, missing-key silence, all-six-tiers readable mapping).
#000028 follow-ups — capital ledger + sample-rate
==================================================
Two policy fields layered on top of canonical_witness_enabled:
- canonical_witness_sample_rate (0.0..1.0; default 1.0). Operators
wanting passive calibration set 0.05 to fire witness on 5% of
canonical questions while paying 5% of LLM cost. 0.0 effectively
off; 1.0 = current always-on behavior. Gating uses random.random()
so distribution is uniform; clamped to [0, 1].
- Capital ledger row written for each FIRED witness (not skipped
ones). op_type='canonical_witness'; estimator inputs include
prompt_chars + answer_chars + llm_seconds + agreement_label +
pi_star_ref. Best-effort: ledger-write failure must never fail
the query (sidecar discipline).
Tests: 4 new — sample_rate=0.0 skips (no LLM call, no ledger row);
sample_rate=1.0 always fires; capital_ledger row written under
op_type='canonical_witness' with full input blob; sampled-out
witness records zero ledger rows.
Both fields fold into governance_policy_hash naturally via the
existing policy-hash machinery — flipping witness mode invalidates
prior records as expected.
#000025 Phase 1d — 5F fixture catalog 30 → 50
==============================================
Both synthetic and live sides of all 5 sub-batteries expanded
30 → 50 (+200 fixtures total: 5 × 20 synthetic, 5 × 20 live).
function — claim_count cycles 2..7 across new fixtures
falsification — 10-violation palette across new ids
feedback-loop — fact-N learning chains
finetuning — capability transitions across canonical π*
(math/logic/algebra/calculus pool)
formulate — multi-pointer claim shapes
500/500 pass through respective runners. test_session_integration
total bumped 562 → 662. Pinned test_5f_*_runs counts updated 30 →
50 (synthetic main + embedded + live).
Tests
=====
Full suite: 1467 passed, 36 skipped (was 1388; +79 across warrant
render + witness sample/ledger + 5F implicit coverage).
Two new π* canonicalizers extend the math substrate above
arithmetic@v1 (closed-form rationals) and logic-kernel@v1
(propositional Boolean → CNF):
algebra-symbolic@v1 (Phase 1) — symbolic-algebra domain.
sp.expand → sp.srepr canonical bytes. Polynomial identity collapses
((x+1)**2 ≡ x**2 + 2*x + 1); exponential identity collapses
(exp(a+b) ≡ exp(a)*exp(b), inherited from sp.expand's default
behavior); trigonometric identity does NOT collapse
(sin²+cos² ≢ 1). The trig surface is reserved for a future
algebra-symbolic-simplified@v1 variant that wraps sp.simplify at
unbounded CPU cost. Rejects relationals (`x > 0`) and
BooleanFunction shapes (`x & y`) via `isinstance(expr, sp.Expr)` —
sp.Symbol confusingly inherits from Boolean so the right rejection
filter is "not Expr" rather than "Boolean".
calculus-derivative@v1 (Phase 2) — calculus domain. JSON-shaped
{f, x, n} input → sp.diff → sp.expand → srepr bytes. Output is
itself a valid algebra-symbolic@v1 input so the two compose
naturally under arborist.pi_star.compose. n defaults to 1; bools
explicitly rejected (Python isinstance(True, int) is True so we
filter that explicitly).
Optional dependency: sympy ships in the new [math] extra
(pyproject.toml). Folded into [dev] so make bootstrap pulls it
transitively. An explicit `bootstrap-math` Makefile target documents
the opt-in for minimal-install users. Both modules self-guard
via `try: import sympy as sp / except ImportError: sp = None` and
only register(...) when sympy is present, so a fresh checkout
without [math] still loads arborist.pi_star without raising.
Preflight algebra route lands in
arborist.qa.query._canonical_projection_preflight between the
arithmetic and logic routes. Charset regex (_CANONICAL_ALGEBRA_RE)
allows lowercase letters + math chars; requires at least one
letter (else arithmetic wins); rejects natural-language leading
verbs via _CANONICAL_ALGEBRA_NL_LEAD_RE (4-letter minimum so
single-/two-/three-char identifiers like x, xy, sin, cos, pi
survive while "simplify (...)", "factor x...", "expand (a+b)..."
fall through). PiStarError + KeyError both fall through cleanly
so a sympy-less install just routes everything past algebra.
Bench substrate:
- bench/batteries/base.py PHASE_1_CARRIERS gains "symbolic_algebra"
- bench/fixtures/5s/syntax-algebra-symbolic-v1.jsonl (10 fixtures)
- bench/fixtures/5s/semantics-algebra-symbolic-v1.jsonl (13 fixtures
including the documented trig non-collapse + exp collapse)
- Makefile bench-5s-algebra target → 100% pass
Tests: 18 algebra-symbolic + 38 calculus-derivative unit tests +
~10 new preflight-route tests in test_canonical_projection.py. All
gate on pytest.importorskip("sympy") so a sympy-less suite stays
green. Full suite: 1369 passed / 27 skipped.
Phases 3-7 (integral, limit, series, linear-algebra,
function-sampled) remain open as future work; each lands as its
own ticket when an actual consumer surfaces.
ClaimPackSource ingests Grok-4 companion bundles (axiomsg4-v2.json +
theoremsg4-v2.json) at the right grain — one Document per axiom or
theorem record. Each record carries Δ (LaTeX symbolic) + ∇verbose
prose, explicit source citation (Mendelson, Enderton, Hilbert,
Newton, Kolmogorov, Łukasiewicz), foundational-group taxonomy, and a
runicLabel that rides as soft metadata only (runtime mints its own
pointer IDs per CTI architecture). Pillar-level
provenance.references arrays become outbound pillar_reference edges.
Lenient JSON parser strips ```json fences and double-escapes lone
LaTeX backslashes (\Theta, \heart, \vec) without corrupting
already-correct \\to pairs — walks left-to-right and pass-throughs
legal escape sequences. Malformed bundles raise rather than return
empty; silent zero-doc would be a footgun.
CLI surface: --source claim_pack with a repeatable --bundle FILE
flag mirroring html source's --url action=append. Single --path
also accepted for one-bundle ingest.
Drive-by: removed a function-local `from arborist.store import
connect` inside _cmd_ingest's providence branch that was shadowing
the module-level binding via Python's "any local assignment makes
the name local for the entire function" rule, breaking every
non-providence ingest with UnboundLocalError. Comment left in
place explaining why not to re-add it.
Smoke-tested on /home/fox/Downloads/{axiomsg4,theoremsg4}-v2.json
end-to-end: 78 docs (55 axioms + 23 theorems across 7 pillars),
14 deduped pillar-reference edges, 78 audit events, 10/10 sampled
Merkle proofs verify, FTS5 search returns Modus Tollens for
"modus tollens".
Honest ceiling: kind=surface for every record. The pack is
pre-distilled but its provenance is asserted not proven — until
Mendelson/Enderton/Hilbert texts are themselves ingested as
surfaces, the verifier has no derivations.proof_blob to compute
and claim-pack records max out at ANCHOR-WARRANTED on the
four-rung ladder. That's a follow-up ticket, not this one.
Hard constraints honored: no new audit ledger (audit_events
remains the only chained-sha256 ledger; bundle's self-validation
fields ride as metadata only); no kind=core without surface
ancestor; cache_key invariants untouched.
15 unit tests cover lenient parser, slug stability, ref
resolution, doc grain, URI stability, content layout, extra
metadata, edge emission, error paths. All 1280 tests in
make test pass.
Closes#000027. Closes#000028 (cache-leg wired).
#000027 — canonical projections persist to providence_cache
============================================================
Math/logic π* answers (arithmetic@v1, logic-kernel@v1,
time-series-quantized@v1, …) are now first-class providence rows.
Pre-fix: question → kernel → answer → return. No cache, no audit
event, no run_dag, no inspect/burn/replay surface.
Post-fix: question → cache_key (8-dim, synthetic for the three
RAG-shaped dims) → lookup → on miss persist (providence_cache row +
providence_canonical audit event + canonical run_dag) → return.
Synthetic cache_key dimensions for canonical rows (per ticket §2.2):
- source_root = sha256("pi_star_source:" + pi_star_ref)
- model_profile_hash = sha256("pi_star_model:" + pi_star_ref)
- conversation_hash = sha256("pi_star_conv:" + canonical_q + ":" + ref)
- chunking_version = literal "n/a-canonical" — chunker bumps on
wikipedia path don't stale math answers.
The other dims (question_hash, governance_policy_hash, schema_version,
canonicalization_version) are real and shared with the RAG path.
Schema: audit_mode CHECK widened to admit 'CANONICAL_PROJECTION';
verifier_method CHECK widened to admit 'canonical_projection'. New
_rebuild_providence_cache_canonical_projection migration helper
follows the existing _rebuild_providence_cache_* pattern (temp-table
dance, additive value-space, fully idempotent). Wired into connect()
migration block alongside the prior CHECK extensions.
Cache-hit policy: trust the row. Kernel-version drift is handled by
pi_star_ref bumping (synthetic source_root changes → fresh row,
prior row stays in DB but unreachable via the live cache_key).
Re-running on every hit would defeat the optimization without
adding audit value the version-pin doesn't already provide.
Policy gate: canonical_projection_preflight_persist (default True).
Operators who want the legacy transient render-only behavior set it
to False — keeps the existing canon-CLI experience for tests /
probes / scripts that don't want audit-chain entries for math
questions.
CLI render: `CANONICAL · via canonical_projection` for persisted
rows. Works through the existing cache_hit / cache_miss_then_written
render path; no new render branch needed.
`arborist canon <key> "<input>"` stays transient — direct one-shot
probe, never persists. Boundary preserved per ticket §2.6.
#000028 — multi-modality witness cache-leg
==========================================
Pre-#000027 the witness cache-leg closure always returned None;
STRICT-WITNESSED (3-of-3 byte-equal) was structurally unreachable.
Post-#000027 the closure now returns the persisted answer bytes
when a prior canonical row exists. Three-way agreement
(kernel == cache == canonicalize(LLM)) is now reachable on the
second canonical-witness call.
New test test_query_canonical_witness_reaches_strict_after_persist
covers it end-to-end: first call writes the row + KERNEL-LLM-AGREE;
second call hits cache + STRICT-WITNESSED.
Tests
=====
- tests/test_canonical_cache.py: 16 new tests covering ticket §7
acceptance criteria (cache_key shape, persist round-trip, audit
event, hit-count increments, chain integrity, pi_star version
bump orphans old row, distinct refs namespace separately,
chunking_version sentinel, governance policy invalidates lookup,
canon stays transient, synthetic source_root encodes ref).
- tests/test_canonical_projection.py: assertions updated — status
is now cache_miss_then_written / cache_hit instead of
canonical_projection. Added a transient-mode test pinning the
policy gate.
- tests/test_witness.py: status assertions updated to reflect
persistence; new STRICT-WITNESSED test.
- tests/test_directives.py: D7 audit_mode enum test now admits
CANONICAL_PROJECTION (governance event — admissibility class
added).
Full suite: 1367 passed, 36 skipped (was 1306; +61 new).
Real-shard smoke
================
$ make query Q="0.1 + 0.2" BURN=1
→ cache_miss_then_written, ~300ms wall, row written
$ make query Q="0.1 + 0.2"
→ cache_hit, ~40ms wall, hit_count++
$ make chain-check-shards
→ 0 breaks per shard
`connect()` used to run executescript(SCHEMA_SQL) + 7 forward-
migration probes on every open. Profile of `who wrote virt-back?`
on 38 GB of real shards (warm cache) showed 588 connect() calls
per query, each running the full probe sequence — 10,623 total
SQLite executes. Migrations are forward-only and idempotent within
a code version, so once we've run them on a path in this process
there's no work to do on subsequent opens.
Cache shape: `set[str]` keyed by `str(Path(p).resolve())`.
Migration block runs once per (path, process); subsequent calls on
the same shard skip it entirely. Per-connection PRAGMAs
(foreign_keys=ON, synchronous=NORMAL, cache_size, temp_store,
mmap_size) still run every time — SQLite scopes foreign_keys
per-connection and our schema's FK CASCADE behavior depends on it.
That's why `PRAGMA foreign_keys = ON` moved out of the cached
SCHEMA_SQL block into the always-run pragma section.
Cache invalidation: explicit only.
`store.invalidate_migration_cache(path)` for callers who replace a
shard at the same path (snapshot-restore flows). `_clear_migration_
cache()` for tests. We don't auto-detect file replacement —
(dev, inode) is unreliable under tmpfs inode reuse, and (mtime,
size) drifts naturally as SQLite operates on the file (WAL
checkpoints, page growth). Path-only with explicit invalidation
is the honest contract.
Re-profile (same query, same shards, warm cache):
metric before after
_migrate_* (each function) 586 7 ← per shard
executescript 586 7
SQLite executes 10,623 3,687 (-65%)
wall (warm) 14.5 s 13.4 s
The warm-cache wall delta is small because the probes were many-
but-cheap; residual cost lives in FTS5 search (6.7 s) and
synonym_expand (2.8 s, both separate concerns). The 65% execute
drop is the cold-cache win — each redundant executescript() had
been triggering disk reads at the 75 s scale the reviewer reported.
Tests (6, all green): first connect runs all 7 probes, second
connect runs zero, schema integrity preserved across re-opens,
explicit invalidation re-probes, distinct paths each get one probe,
clear-cache helper works.
Full suite: 1306 passed, 36 skipped. Found and fixed an FK CASCADE
regression mid-implementation: PRAGMA foreign_keys = ON was inside
SCHEMA_SQL, so memoization was silently turning it off on subsequent
opens. test_burn_doc.py caught it. Moved to the per-connection
pragma block.
Ticket #000026 status: Phase 1 landed; Phase 2 (baseline artifact)
and Phase 3 (warrant-quality finding) queued.
