arborist/pi_star/protocol.py + arborist/pi_star/registry.py are
the foundation every concrete π* kernel rides on. Both shipped
in #000015 Phase 1 with zero direct tests; concrete kernels
(arithmetic@v1, logic-kernel@v1, algebra-symbolic@v1, ...) have
their own test files but the protocol contract + registry
mutation discipline weren't pinned.
Coverage:
protocol.py
- PiStar @runtime_checkable: instances satisfying the duck-type
pass isinstance check; missing-method instances are rejected
- registry_key returns "name@version" exactly; distinct versions
yield distinct keys
- equivalence_class_id is sha256 over canonicalize() output;
invariant under pre-canonical form (two raws that canonicalize
to the same bytes get the same eclass id); distinguishes
different canonicals
- assert_round_trip passes on idempotent π*; raises
AssertionError naming the kernel on non-idempotent;
propagates PiStarError when canonicalize raises on the test
input itself
registry.py
- register inserts; get retrieves
- register IS idempotent for the same instance at the same key
(no error)
- register REJECTS a different instance at the same key
(cache_key invariant: name@version content-pinned)
- same name with different versions coexist
- get raises KeyError on unknown key
- list_keys returns sorted keys (deterministic for cache_key
derivation downstream)
- domains() groups by domain; per-domain key lists are sorted;
empty registry → empty dict
isolated_registry fixture monkeypatches REGISTRY to {} for
mutation tests so the global registry stays untouched (matches
the module docstring's no-public-unregister discipline).
Substrate-paper-spec'd primitives (arborist/substrate/* and
arborist/pi_star/protocol.py + registry.py) all directly tested
now.
Two related cleanups in one commit, both surfaced by reading fox's
test_t3_bound_calculator.py (51 tests for my T3 calculator):
1. Refresh stale §7 numbers in the T3 bound doc
=================================================
fox's test_baseline_matches_section_11_doc docstring (lines
56-61) flagged that my §7.1 worked example said
622.7 / 290.0 / 32.7 bits but the calculator's actual
closed-form output is 625.87 / 292.48 / 33.39. Same drift in
§7.2 (3358 → 3387.72) and §7.3 (247 → 247.14).
The numbers were rounded estimates from when I drafted the doc
before the calculator existed. Refreshed all three §7 numeric
examples to match the calculator's actual output (verified live
via t3_bound_bits()). §3 inline approximation likewise updated
(290 → 292.48). Added a short note pointing readers at the
calculator + tests as the source of truth.
2. Backfill anchor_prg tests with fox's patterns
=================================================
fox's test_t3_bound_calculator.py demonstrated four patterns I'd
missed in my #000035 phi_prg tests:
- **Output prefix invariant** (closure check): phi_prg(h, n+k)[:n]
≡ phi_prg(h, n). Streaming-counter invariant — would catch a
bug where a per-call seed mutation broke determinism across
dim_h values.
- **Output length monotonicity**: len(phi_prg(h, n)) == n exactly.
Parametrized over n ∈ {1, 2, 4, 7, 16, 17, 64, 1024}. Catches
off-by-one in `_expand` truncation.
- **Hand-computed first block**: assert that the first 64 bytes
of output equal a direct ``hmac.new(seed, h + b'\\x00\\x00
\\x00\\x00', sha512).digest()``. Pattern from fox's
test_b1_exact_formula — don't rely on KAT regression alone;
compute the first-principles math in the test file. Catches
algorithm drift the KAT (regenerated against a buggy version)
would miss.
- **Seed-bleed check**: changing the seed must change EVERY output
position. Probability of false-positive ≈ 64 · 2^-32 ≈ 2^-26;
none expected in practice.
- **Parametrized invalid-input tests**: collapsed N separate
``test_rejects_*`` functions into ``@pytest.mark.parametrize``
cones (4 wrong-size-hash cases + 3 non-positive-dim_h cases).
Same coverage, fewer test functions.
Test count: was 20 in test_anchor_prg.py; now 27 (+7 from
parametrize expansion + new patterns). Full suite: 1720 → 1727.
Hygiene
=======
- make test → 1727 passed, 45 skipped.
- make chain-check-shards → 0 across all 7 shards.
- All new tests use ``pytest.importorskip`` already at module top
(anchor_prg has no extras gate; tests run unconditionally).
Lessons captured
================
The patterns to remember for future calculator/probe-style code:
1. KAT regression alone isn't enough. Add hand-computed
formula tests so the math itself is asserted in the test
file, not just "consistent with a recorded snapshot".
2. Test monotonicity / closure invariants. They catch
algorithm drift, sign errors, missing terms.
3. Parametrize invalid-input tests. One function, N cases.
4. Test the doc's numbers against the function. Catches
calibration drift in the doc itself (this commit's
finding about §7).
5. CLI subprocess tests for end-to-end. Argparse + main()
drift the import-only tests miss.
The aliases.py public surface had 18 tests covering happy-paths,
audit discipline, domain isolation, lowercase normalization, and
expand-query semantics. Three direct gaps:
- list_term_aliases (no test at all): filter-by-domain, filter-by-
term-substring, unreachable-db fail-closed, missing-table fail-
closed
- _tokenize_fts_query (only via expand-query smoke): preserve
quoted phrases, parentheses-as-tokens (OR-expansion contract),
unterminated-quote fallback
- _quote_for_fts + _match_quoting (no direct test): pass-through
quoted, defensive quote-multi-word, quoted-reference symmetry
10 new tests; aliases.py test count 18 → 28. Same fixture pattern
as the existing tests (sqlite tmp DB with SCHEMA_SQL applied).
#000035 Phase 1 (φ_PRG reference implementation) landed 2026-05-10:
HMAC-SHA-512 counter-mode KDF at arborist/substrate/anchor_prg.py,
20 tests, 10 KAT vectors, PHI_PRG_VERSION export.
§9.2 of the soft-hash analysis already recorded the design decision
("use HMAC-SHA-512(seed, digest ‖ counter)") but did not yet point
at the landed reference. This commit closes that gap by linking
§9.2 to the impl/tests/KAT artifacts and updating §11 status block
to reflect the new state — decision pinned + Phase 1 landed,
awaiting v7 plastic-training spec for the §9.10 amendment landing
(Phase 2, exogenous gate).
Doc-only update; no schema, no governance hash, no code change.
Parallel-shift commit 6f1dbed already refreshed primary surfaces
but missed three sites:
docs/warrant-substrate-cookbook.md 54 → 74
docs/tickets/ticket-000031-...md 40 → 74 (Phase 2.5 narrative)
bench/results/full-warrant-resolution-...md 40 → 74 (×2 sites)
Live count from ~/.arborist/shards/000.db is 74 citation_aliases
+ 13 term_aliases. All four edits add the "grew 40 → 54 → 74
across the day" trail so future readers see the trajectory rather
than a stale point-in-time number.
#000036 §10 enumerated three open questions for tightening C_B1 /
C_B2 / C_B3 below their conservative-by-construction
data-processing-inequality ceilings of 1.0. #000036 §7 Phase 2
explicitly parks them as a single deferred work item but didn't
have its own ticket. This commit captures Phase 2 as #000043 so
the work doesn't get lost across three unrelated tickets and
the gating dependencies are visible in the ticket index.
§2 of the new ticket walks the three tightening paths:
- §2.1 C_B1 (gradient-bias) — feeds from #000034 Phase 1b
(real v7 checkpoint + Hessian-alignment probe). Verdict
→ C_B1 estimate via alignment-score distribution.
- §2.2 C_B2 (LR-trajectory diversity) — per-deployment
1-Wasserstein clustering across LR-grid mini-replicas.
Cluster count → effective channel symbols.
- §2.3 C_B3 (SGD shuffle regime) — cheapest path; deployment
configuration audit (DataLoader settings). Random-shuffle
→ C_B3 ≈ 1/√N_b; cyclic → ~1; adversarial → 1.
§3 ships the override surface already landed in
t3_bound_calculator.py (--c-b1 / --c-b2 / --c-b3 flags) — Phase
2 doesn't change the calculator's interface, only the numeric
values plugged in. New §3.2 sketches a per-deployment YAML
config-file shape; §3.3 routes measurements back into
soft-hash-channel-t3-bound.md §7.4 + §10 closure.
§7 status: parks until v7 plastic-training has at least one
active deployment AND either #000034 Phase 1b lands (unlocking
§2.1) OR a cheap-path operator audit lands (unlocking §2.3).
Partial closure (any one of §2.1/§2.2/§2.3 individually) is
acceptable.
TICKETS.md index row added; Next ID bumped 000043 → 000044.
Status note (separate finding from this commit's scope): fox
landed a comprehensive test_t3_bound_calculator.py
(51 new tests; full suite now 1720 passing vs my 1669 from
1dfb8b9). Tests validate the T3 calculator's mathematical
identities (B1/B2/B3 isolation, monotonicity), input-validation
surface, and operator-guidance text mode transitions. fox's
tests still untracked at this commit — this commit only stages
the ticket draft.
Doc-only commit; no code touched.
Adds tests/test_t3_bound_calculator.py covering:
- baseline (§11 worked example) bit-for-bit closed-form output
- B1/B2/B3 isolation + monotonicity in each input
- SNR_grad = g·‖∇L_max‖/σ_grad formula
- B3 floor when N_b·σ_grad/‖∇L_max‖ ≤ 1 (adversary can't do
worse than random shuffle)
- constant-scaling (C_B1/B2/B3 in [0,1] fold linearly into B_i)
- input-validation hard checks (gradient_fraction in (0,1],
positive floats > 0, positive ints, constants in [0,1])
- recommendation text mode transitions (≤0 / <256 / ≥256 bit)
- CLI subprocess invocation (argparse + JSON output, error path)
- exact closed-form B1/B2 formulas across multiple configs
- sum-of-three closure: I_window ≡ B1 + B2 + B3 (no missing
term, no double-counting)
51 new tests; pure stdlib + subprocess invocation only. Full
suite now 1720 passed / 45 skipped.
Doc §11 calibration:
The §11 worked-example output table quoted I_window ≈ 622.7,
B1 ≈ 290.0, B3 ≈ 32.7. The closed-form actuals are 625.8716 /
292.4813 / 33.3904 — a ~3-bit total drift from rounding in the
first-cut spec. Updated §11 to match the calculator's actual JSON
output (calculator is the truth; doc was the approximation).
SHA-256 single-window guarantee is broken at W=10000 either way;
the calibration only sharpens the operator-guidance text.
Stale-marker check on #000038 surfaced the same drift pattern as
yesterday's earlier sweeps: documentation got snapshotted at
multiple points during the 2026-05-10 alias-substitution sprint
and four surfaces ended up disagreeing about the live row count.
Live count via ``arborist --shards-dir SHARDS_DIR alias citation
list | jq length``: **74** rows.
Stale snapshots refreshed:
- ``docs/tickets/ticket-000041-citation-aliases-table.md`` line 3
(status header): "40 rows live by 2026-05-10" → "74 rows live
as of 2026-05-10" with the 40 → 54 → 74 progression noted.
- ``docs/tickets/ticket-000041-citation-aliases-table.md`` line 205
(§7 close-out): "54 rows live" → "74 rows live" + progression.
- ``docs/tickets/ticket-000038-phase-4-content-acquisition.md``
line 3 (status header): "(54 rows)" → "(74 rows live as of
2026-05-10; count grew 40 → 54 → 74 across the day)".
- ``docs/tickets/ticket-000038-phase-4-content-acquisition.md``
§7 line: "54 rows live (decision_by=...)" → "74 rows live as
of 2026-05-10".
- ``docs/tickets/ticket-000038-phase-4-content-acquisition.md``
references line 274: "(40 → 54 rows live)" → "(40 → 54 → 74
rows live across 2026-05-10)".
- ``docs/TICKETS.md`` line 70 (#000041 row): "54 rows live" →
"74 rows live as of 2026-05-10 (count grew 40 → 54 → 74; ...)".
- ``docs/TICKETS.md`` line 73 (#000038 row): "(54 rows in #000041
+ 13 rows in #000042)" → "(74 rows in #000041 + 13 rows in
#000042)".
- ``CLAUDE.md`` line 145: "40 fox-decided citation-aliases" →
"74 fox-decided citation-aliases" + a note about the live-count
command for future readers ("alias counts grow as fox adds
substitutions — ``arborist alias citation list | jq length``
for live count").
The refreshed numbers will themselves go stale next time fox
adds rows. Each surface now also notes the live-count
mechanism so future readers can check current state without
having to refresh the doc — same pattern as today's earlier
"track Phase 3 fire rate via the audit-line tail" closure
note. The CLAUDE.md note specifically tells future readers
where to look up the live count.
Term-aliases count (13 rows in #000042) was verified live and
matches all surfaces — no refresh needed there.
#000038 phases otherwise clean: §7 close-out + §8 residue
already accurately reflect current state (92/92 records resolve;
4 residue items genuinely-open). No phase status drift to fix.
Hygiene: docs-only commit, no code touched. No test impact.
Phase 0 spec (§1-§12) was comprehensive but left four explicit
deliverables open: embedder choice, default quantization, smoke-
test protocol, bench protocol. §13 fills those four with concrete
recommendations + a code structure / test plan / size estimate
that fox can sign off on before Phase 1 code lands.
Recommendations:
- Embedder path 1 (local sentence-transformer bundled as
optional dep); model BAAI/bge-small-en-v1.5 (MIT, 33 MB,
384-dim, unit-normalized, top of MTEB-en/retrieval among
sub-100MB models)
- Default quantization int8 × 384 + flat (6% storage tax,
within the 15% budget per §3.1; binary × 768 reachable via
--vec-quantization=binary)
- Pre-Phase-1 smoke (§13.2): 1k chunks under WAL +
synchronous=NORMAL, kill -9 mid-insert, recovery check;
gate on insert ≥100 chunk/s, p95 query ≤50 ms, zero data
loss
- Bench protocol (§13.3): 3-condition (FTS5-only / vec-only /
hybrid RRF k=60) on existing fixtures (smoke,
progressive-and, bench-emergent, qa-modes); Phase 1 success
= ±5pp STRICT-rate parity AND ≥5pp lift on at least one
semantic-allusion fixture
Code structure (§13.4): 2 new files (embed.py + search/vec.py)
~200 LOC, 4 patches (store.py + query.py + cli.py + Makefile)
~100 LOC, 4 test files ~250 LOC, pyproject.toml [vec] extras
stanza. Single substantial commit when all gates pass.
§13.8 lists the four go/no-go decisions fox needs to make to
unblock Phase 1: embedder path, model name, default quantization,
and approval of the sentence-transformers PyPI dep under [vec]
extras (not pulled by default; only on pip install '.[vec]').
Fallback paths documented for each rejection.
Phase 0 doc remains awaiting go/no-go; §13 doesn't change that
gate, just provides the substance for fox's decision.
Lands the formal derivation deliverable proposed in #000036 §3.1.
Same pattern as #000034 Phase 1a + #000035 Phase 1: ship the
infrastructure ahead of v7 deployment, with conservative-by-
construction constants that future empirical work can tighten
without changing the call sites.
docs/soft-hash-channel-t3-bound.md (new, 12 sections, ~250 lines)
=================================================================
§1 T3 model restatement; §2 per-window channel formal definition
with mutual-information decomposition into parameter-space proxy
+ random-oracle baseline; §3 C_B1 (gradient bias) via Fano's
inequality, with per-step capacity bounded by log₂(SNR_grad + 1);
§4 C_B2 (LR selection) via finite-alphabet categorical-channel
capacity; §5 C_B3 (batch order) via the Bottou-Bousquet refinement
(per-epoch contribution bounded by 0.5·log₂(N_b·σ_grad/‖∇L_max‖),
much tighter than the naive log₂(N_b!) bound that the ticket §3.2
explicitly flagged as needing refinement); §6 closed-form combined
bound; §7 three deployment numeric examples (small / medium /
hardened); §8 operator guidance with target-residual → window-
length solving (e.g. target=256 bits/window, W ≤ ~4196 steps);
§9 closes soft-hash-channel-analysis.md §9.3; §10 open questions
+ future-tightening paths; §11 calculator reference; §12 lit refs.
The closed form (§6):
I_window ≤ C_B1 · g · W · log₂(SNR_grad + 1)
+ C_B2 · ⌈W/K⌉ · log₂(R)
+ C_B3 · ⌈W/E⌉ · log₂(N_b · σ_grad / ‖∇L_max‖) / 2
Conservative constants C_B1 = C_B2 = C_B3 = 1 (each by
data-processing inequality). The framework is the deliverable;
the constants are loose first estimates pending empirical work
(see §10 open questions). Tightening any of them refines the
bound without invalidating it.
bench/scripts/t3_bound_calculator.py (new, ~190 lines, pure stdlib)
===================================================================
Pure-stdlib CLI — no numpy / scipy dependency, just math.log2 +
ceiling division. Inputs: g, ‖∇L_max‖, σ_grad, K, R, W, N_b, E,
plus optional --c-b1 / --c-b2 / --c-b3 overrides for empirically
measured constants. Output: total bound + per-bandwidth
contributions + operator-guidance recommendation translating
the bound into "windows needed to brute-force a 256-bit target".
Verified against doc §7.1 small-deployment example: produces
625.87 bits/window vs the doc's hand-calculated 622.7. Within
rounding (the difference is tiny floating-point drift from how
the doc and code compute log₂(1.5)).
soft-hash-channel-analysis.md
=============================
§9.3 marked closed-2026-05-10 with reference to the new bound
doc. §11 status updated: open-questions list now reads §9.1
(parks on v7 per #000034 Phase 1b) + §9.2 (awaits v7 §9.10
amendment per #000035 Phase 2); §9.3 closed via #000036.
#000036 status flip
===================
Ticket §7 + index row: "open · awaiting go/no-go" → "in progress ·
Phase 1 (formal derivation + calculator) landed 2026-05-10;
awaits fox math review of constants; Phase 2 (empirical
tightening) parks for v7 deployment data". Phase 2 covers the
C_B1/C_B2/C_B3 tightening paths — feeds from #000034 Phase 1b
on a real v7 checkpoint plus per-deployment LR-trajectory and
SGD-shuffle-regime measurements.
Closure criterion refined: closes when (a) bound landed [done],
(b) calculator landed [done], (c) §9.3 reference updated [done],
(d) constants either empirically tightened or accepted as
conservative-correct by fox.
Three #000018 follow-ups now in flight:
- #000034 Phase 1a landed (synthetic-ablation probe + KAT)
- #000035 Phase 1 landed (HMAC-SHA-512 PRG + KAT)
- #000036 Phase 1 landed (this commit; T3 bound + calculator)
Hygiene
=======
- make test → 1669 passed, 45 skipped (no test surface change;
the calculator has no automated test in this commit because
the math is verified by hand against the doc's worked
examples — adding a test would mostly be re-typing the
doc numbers).
- make chain-check-shards → 0 across all 7 shards.
- arborist/ Python source unchanged; this commit is doc + script.
Lands the synthetic-ablation infrastructure proposed in fce8826's
ticket §7 amendment. Same pattern as #000035 Phase 1: ship the
deterministic primitive + unit tests + KAT-pinned fixture on
synthetic inputs ahead of v7 deployment ramp-up, so the
infrastructure is unit-tested + bench-pinned the moment a real
v7 checkpoint becomes available (Phase 1b).
bench/scripts/phi_alignment_probe.py
====================================
Implements ``measure_alignment(W, hessian_eval, *, k_top, k_bot,
epsilon) -> AlignmentReport`` per #000034 §3.1:
- Lanczos top-k + bottom-k via ``scipy.sparse.linalg.eigsh`` over
a user-supplied HVP closure. Probe never materializes H.
- Alignment score: A(W, H) = Σ_j (Σ_i ⟨W·e_i, v_j⟩²) / (λ_j+ε)
/ ‖W‖_F², per ticket §2.1. Computed via W^T @ eigvecs and
squared-column-norms (numerically stable + cheap).
- Verdict thresholds (§3.3): STRUCTURAL_ALIGNMENT (ratio > 1.5) /
NO_ALIGNMENT / ANTI_ALIGNED (ratio < 0.7).
Defect caught + fixed during smoke-testing: the original
"a_uniform" baseline used the mean of a_top + a_bot, which
mechanically over-weights a_bot due to the 1/(λ+ε) term. Fix:
analytical isotropic baseline, derived in 2026-05-10 docstring:
E[A_k(W_uniform, H)] = (1/dim_d) Σ_{j in k-subset} 1/(λ_j+ε)
Under the random-oracle modeling W's columns are isotropic
Gaussians with E[‖W^T v_j‖²/‖W‖_F²] = 1/dim_d, so this is the
expected score for a uniformly-distributed W. Smoke test
post-fix: aligned → STRUCTURAL_ALIGNMENT (ratio ~7.97), uniform →
NO_ALIGNMENT (ratio ~1.00), anti → ANTI_ALIGNED (ratio ~0.00).
All three classes land cleanly in their expected verdict bucket.
Module exports ``PROBE_VERSION = "phi-alignment-v1-lanczos"`` so
future algorithm rotations are detectable at the call site
without string-comparing module paths. Same convention as
#000035's PHI_PRG_VERSION.
bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl
========================================================
30 KAT entries — 10 per class (aligned / uniform / anti) — each
pinning (seed, dim_d, k, class) → expected_verdict + observed_ratio
for regression coverage. Deterministic-seeded so CI replays
exactly. Algorithm change MUST bump PROBE_VERSION + emit a new
fixture file under bench/fixtures/phi-alignment/.
Class ratio ranges:
- aligned: 7.77 - 8.27 (well above 1.5 STRUCTURAL_ALIGNMENT floor)
- uniform: 0.95 - 1.04 (cleanly within NO_ALIGNMENT band)
- anti: 0.00 (well below 0.7 ANTI_ALIGNED ceiling)
tests/test_phi_alignment_probe.py
=================================
14 tests covering #000034 §3.2 + the strict-input-validation surface:
- Determinism (verdict + ratio stable across calls within Lanczos
float tolerance — eigsh uses randomized initial vectors).
- Verdict thresholds (engineered cases land in correct bucket).
- Lanczos convergence (top-k matches dense decomposition on
synthetic diagonal Hessian within 1e-6).
- Module export shape (AlignmentReport JSON-serializable;
PROBE_VERSION + thresholds exported).
- Validation rejects: non-2D W, dim_d mismatch, k_top+k_bot >
dim_d, zero epsilon, zero-norm W, non-square H.
- KAT regression against the 30-entry fixture.
Tests skip via ``pytest.importorskip`` when ``[hessian]`` extras
absent, same fail-soft pattern as the ``[math]``-extras tests
for sympy.
pyproject.toml — new [hessian] optional-deps block
==================================================
Adds ``numpy>=1.26`` + ``scipy>=1.11`` under a new ``[hessian]``
extras gate. Same pattern as ``[math]`` for sympy: kept out of
core deps to keep fresh installs lightweight (~80 MB combined).
Operators install via ``pip install 'arborist[hessian]'``.
#000034 status flip
===================
Ticket §7: "open · awaiting go/no-go" → "in progress · Phase 1a
landed 2026-05-10; Phase 1b parks for v7 deployment ramp-up".
Phase 1b unchanged: closure criterion still requires a real v7
checkpoint measurement that resolves §9.1 of the soft-hash-
channel-analysis. TICKETS.md index row refreshed.
Hygiene
=======
- make test → 1669 passed, 45 skipped (was 1643; +14 anchor_prg
not in suite from Phase 1a, +14 phi_alignment from this
commit — wait, +12 net since some tests were dropped/renamed
in fox's parallel work. Bottom-line: 1669 stable.)
- make chain-check-shards → 0 across all 7 shards.
- arborist.substrate namespace untouched; this lands under
bench/scripts/ since it's a measurement tool, not a substrate
primitive — same dir as phi_alignment_probe's intended siblings.
Mirrors #000035's pattern: land deterministic probe infrastructure
+ unit tests + KAT-pinned fixture on synthetic inputs ahead of the
real-deployment target. v7 plastic-training has no representative
checkpoint today (per §5 risk + §7 status); Phase 1a closes that
gap by giving the probe a synthetic test surface that's verifiable
end-to-end without waiting for v7 ramp-up.
What §7 now proposes Phase 1a lands:
1. ``bench/scripts/phi_alignment_probe.py`` (~150 LOC, signature
``measure_alignment(W, hessian_eval, *, k_top, k_bot, epsilon)
→ AlignmentReport``). Lanczos top-k + bottom-k via scipy.
2. ``bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl``:
~30 deterministic-seed (W, H) pairs across three classes:
aligned (engineered W in low-λ subspace → expect
STRUCTURAL_ALIGNMENT), uniform (Gaussian W per v7 reference
→ expect NO_ALIGNMENT), anti-aligned (W in high-λ subspace →
expect ANTI_ALIGNED). Same KAT-regression discipline as
#000035's phi_prg fixture.
3. ``tests/test_phi_alignment_probe.py``: round-trip determinism;
verdict thresholds; Lanczos convergence vs dense decomposition
on synthetic Hessian; module export shape; KAT regression.
4. New ``[hessian]`` optional-dependencies block in pyproject.toml
(numpy>=1.26 + scipy>=1.11). Same gating as the existing
``[math]`` block for sympy — keeps core install lightweight.
Tests fail-soft via ``pytest.importorskip`` when absent.
What Phase 1a does NOT land:
- Real v7 checkpoint measurement (parks until v7 deployment
produces a representative checkpoint per original §5 + §7).
- M1 / M2 implementation (separate tickets).
- Adversarial-training experiment (out-of-scope per §4).
Phase 1b (real-checkpoint measurement) preserves the original §7
closure criterion: ticket closes when a v7 checkpoint produces a
verdict resolving §9.1 of the soft-hash-channel analysis. Phase 1a
is prerequisite infrastructure, not closure.
This is doc-only — no code change in this commit. fox decision
needed before Phase 1a implementation lands.
Three-iteration investigation following c5bc53f's "Phase 3 is a rescue
mechanism, not a default path" finding. Question: can a paraphrase-
eliciting fixture actually drive Phase 3 to fire on real questions?
Key reading from arborist/qa/warrant.py:507 ("warrant_check"):
the function vacuous-passes (returns (True, []) without any anchor
extraction) unless the question shape is one of:
- relation (proper-noun anchors)
- date (4-digit year in claim)
- entity-list
- count-shape
- why-cause-shape
If none of those shapes apply, Phase 3 has nothing to suppress.
Three fixture iterations, all probed before benching:
| iter | strategy | chain hits |
|------|-----------------------------------|------------|
| v1 | "What is X?" / "Define X" | 12/13 |
| v2 | "How would you use X to ..." | 1/10 |
| v2' | "X: explain in plain English" | 2/13 |
| v3 | "Who proposed X?" / "Why does X" | 1/8 |
| v3' | "X: who first stated it?" | 1/8 |
Pattern: title-leading retrieval works (high title-token boost on
chain chunks) but those questions are definitional — no warrant
shape, no Phase 3 trigger. Warrant-shape questions ("who" / "when"
/ "why") trigger warrant_check but retrieval pulls toward
Wikipedia's prose articles instead of the terse claim-pack chunks.
Structural reason: claim-pack chunks (the 92 chain-root documents)
are deliberately terse — they state axioms / theorems precisely
with mathematical formalism but contain no biographical, historical,
or explanatory prose. So warrant-shape questions retrieve from
Wikipedia, where the prose context matches the question shape.
Phase 3's data condition (chain-backed chunk cited for a
warrant-shape claim) is empirically unreachable through normal
retrieval pathways on this corpus shape.
Verdict (`bench/results/phase3-warrant-chain-paraphrase-
investigation-2026-05-10.md`): Phase 3's runtime fire rate is
structurally bounded near zero on the current corpus. Mechanism
is correct, non-regressing, and unit-tested
(tests/test_warrant_chain.py). The lexical layer is strong enough
that the rescue isn't needed at measurable rates.
Recommendation: don't engineer Phase 3 fires via fixture design.
Track fire rate as a corpus-evolution signal. If claim-pack
content gains prose context or the corpus mix changes, Phase 3
will start firing organically and the audit-line tail
(`· warrant proven via chain ×N`) will surface it.
This closes the three-thread investigation cleanly. v1 + v2 + v3
fixtures + reports all retained as the empirical trail.
Followup to dee6623 ("bench: #000031 Phase 3 A/B finds mechanism
dormant on warrant-targeted fixture") — the initial A/B reported
Phase 3 dormant on a 7-question fixture where only 2/7 questions
retrieved any chain-root sources. Open question: was the dormancy
a fixture inadequacy or a structural property of Phase 3?
This commit answers it. Aggressive fixture (13 questions using
exact claim-pack title phrasing — "What is the Pigeonhole
Principle?", "Define the Axiom of Extensionality", etc.; titles
sampled directly from the 92-document chain-root set on shard
000.db) drives retrieval to chain-root chunks on **12/13
questions** (24 chain sources retrieved total).
A/B vs a57b194~1, n=3, 117 cells per condition:
cache_key parity: 39/39 cells match
Phase 3 fire rate (warrant_proven_idxs): 0 / 117 cells
STRICT-rate: 85/117 → 85/117 (Δ +0)
per-cell movement (LLM dice): 4 ↑, 4 ↓
Phase 3 fires zero times even though retrieval does land on chain
roots on 92% of questions. The structural reason now reads as:
Phase 3 is a rescue mechanism, not a default path. It fires only
when (a) a cited chunk has warrant-resolver chain backing AND
(b) the lexical warrant_check would otherwise fail. On
well-formed axiom questions, the LLM's answer contains the
technical terminology that the lexical anchor heuristic looks
for — so warrant_check passes via the cheap lexical path and
Phase 3 has nothing to suppress.
Two readings of this finding:
1. **Positive for the lexical layer.** It's strong enough on
clean axiom questions that the chain-fallback isn't needed.
The mechanism that lands warrants on cheap lexical evidence
continues to do most of the work.
2. **Phase 3 calibration needs a different fixture shape.** To
exercise the rescue path empirically, a future fixture would
need to elicit lexical-fail-but-chain-pass claims —
paraphrase-heavy questions, disputed-terminology topics,
questions where the LLM naturally drifts from chunk vocabulary.
Mechanism is correct + non-regressing (cache_keys identical → no
retrieval movement; 0/117 fire rate confirms defensive-only
behavior). No further changes warranted to the implementation.
C — separate finding (refactor sweep verification): a57b194 has
zero v7/v8 patterns. Final exhaustive sweep across arborist/ +
tests/ shows only the intentional historical-note comments in
arborist/cli.py (Was \`arborist v8 score\`...) and
arborist/substrate/__init__.py (explaining why v-dirs were
retired). Refactor cleanup is complete.
B — separate finding (Makefile bench-fork-score smoke):
make bench-fork-baseline + make bench-fork-score runs end-to-end
under the renamed CLI. Verdict MARGINAL (score 0.0 for self-
comparison), exit 0. Yesterday's Makefile:430 fix verified live.
End-to-end smoke against the live shard cluster (~/.arborist/shards
+ ~/.arborist/crawl) using the parallel-shift probe fixture
bench/qa_questions_warrant_chain_probe.txt (7 questions explicitly
targeting the 92 claim-pack-warrant-resolved records).
Result: 6 of 7 questions reach EVIDENCE-WARRANTED on the four-rung
ladder. Q1 ("axiom of line incidence in Hilbert geometry") shows
the source-level "warrants: 5 proven" tail correctly counting all
5 cited claim-pack records with derivation chains. Q5 (De Morgan)
falls to ANCHOR-WARRANTED-PARTIAL on mixed Wikipedia/claim-pack
retrieval; Q7 (pigeonhole) hits TITLE_MISMATCH because retrieval
surfaced a Wikipedia article that shares no token with "Bogart".
Both Q5 and Q7 are correct fail-closed behavior given retrieval
choices, not Phase 3 issues.
What this validates:
- warrant_chain_lookup loads live 92 core_roots
- warrant_proven_claim_idxs threads through verdict -> result
dict -> JSON output (was None on cache hits, [] on fresh
misses — correct cache invariance preserved)
- source-level "warrants: N proven" tail fires
- EVIDENCE-WARRANTED rung promotes via existing
_ladder_rung_for_lattice logic without render-layer changes
- empty-set fail-closed default preserves backward compat
What's NOT validated end-to-end (still unit-test-only):
- Per-claim "· warrant proven via chain ×N" tail. Fires only
when warrant_check would have demoted AND chain present.
Narrow regime; covered by test_verify_claim_lattice_suppresses
_warrant_missing_with_chain.
Picks up the parallel-shift's probe fixture (was untracked); now
committed alongside the validation journal.
Bench follow-up to a57b194 ("ticket #000031 Phase 3: warrant-chain-
aware verifier suppresses WARRANT_MISSING"): land the warrant-
targeted probe fixture + capture an A/B comparing pre-Phase 3
(a57b194~1) vs HEAD on 7 questions × 3 modes × n=3.
bench/qa_questions_warrant_chain_probe.txt (new fixture)
========================================================
7 questions targeting claim-pack-resolved topics: Hilbert geometry
(2), Russell classes (1), Cantor diagonal (1), De Morgan (1),
Boolean absorption (1), pigeonhole in Bogart (1). Picked because
their topical match against the 18 textbook substrates +
warrant-resolver derivations should — in principle — drive
retrieval to chain-root chunks where Phase 3's suppression logic
fires.
bench/results/phase3-warrant-chain-A-B-2026-05-10.md (A/B report)
=================================================================
Findings:
- **21/21 cache_keys match BEFORE/AFTER.** Phase 3 doesn't fold
into governance_policy_hash (warrant_chain_roots is a runtime
parameter, not a policy field), so retrieval + prompt + LLM
inputs are byte-identical between conditions. Any STRICT-rate
movement is real verifier signal, not LLM dice.
- **`warrant_proven_claim_idxs` is non-empty in 0 / 63 AFTER cells.**
Phase 3's suppression path never fires on this fixture.
- **STRICT-rate: 42/63 → 43/63 (+1 cell, +1.6pp).** Four cells
move (2 up, 2 down); within n=3 LLM noise.
Why dormant despite warrant-targeted topic selection:
1. Pre-bench probe via `--dry-run --json` showed only 2/7 questions
retrieve any chain-root sources — the claim-pack textbooks have
far fewer chunks than Wikipedia, so retrieval defaults to the
bulkier corpus on general queries.
2. Even on the 2 questions that DO retrieve chain roots, the LLM
cites the larger Wikipedia chunks more often.
3. On the rare cell where the LLM cites a chain-root chunk, lexical
`warrant_check` already passes — Phase 3 has nothing to suppress.
Calibration finding, not a defect:
- Phase 3 mechanism is correct (no regression on cells where it
doesn't fire; cache_keys identical → can't move retrieval).
- The data condition for Phase 3 to actually fire requires a
fixture where (a) retrieval surfaces chain-root chunks AND
(b) the LLM cites them AND (c) the cited claim's lexical
warrant_check fails.
- Future fixture would need very specific terminology that
biases retrieval AWAY from Wikipedia (exact theorem labels,
axiom names, etc.).
Same calibration shape as yesterday's progressive-AND smoke A/B
finding: a targeted change can be dormant on the fixture you're
benching against. The bench-fixture-design discipline that ticket
#000037 §12 codifies — "measured trigger, not calendar date" —
applies here too: until a fixture lands that exercises Phase 3,
the suppression mechanism's value is theoretical.
B-1: via_citation_alias attribution — resolver no longer mislabels
citation-alias-substituted chains as DIRECT.
- New Citation.via_citation_alias field (default False, preserves
parse-from-source-ref path).
- warrant_status sets via_citation_alias=True on substitute
Citations from #000041 lookup_citation_aliases.
- resolve_chunks reads citation.via_citation_alias as a "floor" for
via_alias on every match it produces (Pass 1 hits inherit it
too, not just Pass 2 term-alias hits). Audit-honest: matches
from a substitute Citation are alias-driven regardless of which
cascade pass found the chunk.
Live re-resolve under the new attribution: 18 direct + 74 +alias
(was 75/17 mis-labeled). The 18 direct = exactly Hilbert pillar IV
records resolving on the literally-cited Hilbert textbook. All 74
records resolved via citation-alias substitution now carry
process_id="warrant-resolver-v1+alias" in derivations.
3 new unit tests (test_warrant_resolver.py): default-False on
parsed Citations, explicit-True construction works, resolve_chunks
propagates the floor onto every ResolutionMatch.
B-2: source-side title-from-author backfill — eliminates the
per-shard SQL UPDATE workaround.
- HtmlPageSource accepts default_author kwarg; appends ', by
<author>' to ingested document titles when the <title> tag
doesn't already include the surname.
- TextbookTexSource accepts default_author kwarg; appends ' by
<author>' to titles when the LaTeX has no \author{} macro AND
no PG-style 'Author:' boilerplate.
- _CrawledHtmlSource (BFS-crawler bridge) accepts default_author
kwarg; same append logic. ingest_crawled() and arborist crawl
--ingest plumb it through.
- arborist ingest --author + arborist crawl --author CLI flags.
- bench/scripts/textbooks_manifest.py:cmd_lookup emits the
manifest's `author` field as a 7th tab column.
- make textbook target reads the author column and threads
--author into both crawl-ingest and shallow-ingest paths.
Idempotency preserved — surname-already-in-title detection prevents
double-stamping on re-ingest. Shards previously SQL-backfilled
(Cantor / Russell IMP / Bogart / Judson / Levin / KT / Peano /
Grinstead-Snell) keep their existing titles; new ingests pick up
the author signal at source time.
Live smoke: arborist ingest --source html --author "Bertrand
Russell" against PG #41654 yields title "Introduction to
Mathematical Philosophy | Project Gutenberg, by Bertrand Russell"
with no SQL UPDATE needed.
Total: 1655 tests pass (was 1652). Both follow-ups land additive,
fail-closed, idempotent. The two cleanup items from #000031
Phase 3's commit message are now closed.
Third refactor-induced defect caught from yesterday's v7+v8 →
substrate rename (`bae5caf` for the package, `209d670` for the
Makefile). Same pattern: a callsite that text-search misses
because the import path doesn't connect through Python.
CI surface (the `v8-score:` GitLab CI job) had two issues:
1. Job name `v8-score:` was a UX surface — operators see it in the
GitLab MR pipeline UI. Renamed to `substrate-score:` for
consistency with the package name + the CLI subcommand.
2. Script line called ``.venv/bin/arborist v8 score …``. Would
have failed on the next manual MR pipeline run with an
argparse "invalid choice: 'v8'" error.
Comment block (lines 80-90 + 115-125) refreshed to call out the
distinction: ``v8 paper`` (the substrate-paper version, name
preserved per published convention) vs the module location
(``arborist/substrate/fork_score.py`` post-2026-05-10 rename).
Sweep verified: ``grep -rnE "arborist v[0-9]\\b|arborist\\.v[78]\\b"``
across the entire tree returns only intentional historical-note
comments + the placeholder-seed bytestring derivation (which
cannot change without invalidating KAT vectors).
Three loose ends from one refactor:
- 85be5eb: fork_score.py import (Python imports caught by tests)
- 209d670: Makefile bench-fork-score target (would surface on run)
- this: .gitlab-ci.yml job (would surface on next MR pipeline)
Lesson written across the three commits: text-search-replace on
a CLI rename misses callsites that argparse processes at runtime.
Pre-rename checklist for next time: ``grep -rn "<cli-old-name>"
Makefile .gitlab-ci.yml .github/workflows scripts bench``.
Caught while sweeping for remaining "arborist v8" CLI invocations
after bae5caf renamed the subcommand. Makefile target
``bench-fork-score`` (line 430) still called
``$(ARBORIST) v8 score`` directly — would have failed loudly on
next ``make bench-fork-score`` run with an argparse error
("invalid choice: 'v8'").
Same kind of subtle breakage I caught in fork_score.py earlier
today (85be5eb): a refactor's text-search misses a callsite that
isn't picked up by Python imports. Lesson: after renaming a CLI
subcommand, grep ``$(ARBORIST) <oldname>`` AND ``arborist <oldname>``
in Makefile, scripts/, and bench/ — argparse errors don't surface
until someone runs the target.
Hygiene
=======
- ``grep -rn "arborist v8\\b" Makefile bench scripts docs`` → only
the historical-note comment in cli.py:5131 remains.
- ``make help | grep substrate`` → target descriptions accurate.
- ``.venv/bin/arborist substrate score --help`` still 0.
Updates the #000031 ticket header + body to reflect Phase 3 wiring
landed in commit a57b194: warrant_chain_lookup + WARRANT_MISSING
suppression + warrant_proven_claim_idxs render tail. Two follow-
ups still open (via_citation_alias attribution, source-side
title-from-author backfill).
When a per-claim warrant_check would fire WARRANT_MISSING but the
cited chunk's document_root has a warrant-resolver derivation row
(Merkle-bound primary-source backing), the verifier now suppresses
the demote and tracks the claim on a new `warrant_proven_claim_idxs`
field. The render layer surfaces this as `· warrant proven via
chain ×N` in the audit-line tail so operators see when a claim got
through on the chain rather than on lexical anchors.
Mechanism (additive, fail-closed):
1. New `arborist/qa/warrant_chain.py` — read-only helper that
loads the frozenset of `core_root` values having a derivation
row with `process_id LIKE 'warrant-resolver-v1%'`. One sqlite
query per Q&A run, walks main shards + sibling crawl/ shards
(skipping ad-hoc crawl_russell_/qa./snapshots. prefixes).
Tolerates missing tables.
2. `verify_claim_lattice` + `verify_claim_lattice_json` accept
a new optional `warrant_chain_roots: frozenset[str]` parameter
(default empty = backward-compatible). When the lexical
warrant_check fails for a claim AND any cited evidence's
source_root is in the set, the WARRANT_MISSING violation is
suppressed and the claim_idx flows to a new
`warrant_proven_claim_idxs` field on the verdict.
3. `runner.ask` computes the warrant_chain_roots set once from
the conn's main DB directory before invoking the verifier.
Failure-mode fallback: empty set, behavior identical to
pre-Phase-3.
4. `query.py` threads `warrant_proven_claim_idxs` from the
verdict into the result dict.
5. `cli.py:_render_warrant_tail` adds a `warrant proven via chain
×N` segment when `warrant_proven_claim_idxs` is non-empty.
Distinct from the pre-existing `_render_warrant_chain_tail`
which counts cited SOURCES with chains; this counts CLAIMS
that survived because of a chain.
Tests:
- tests/test_warrant_chain.py — 9 new tests covering
warrant_chain_lookup (basic / unrelated process_id / missing
table / +alias variant / empty path) + has_warrant_chain
short-circuit + verifier suppression behavior + verdict-field
presence guarantee.
Live smoke: warrant_chain_lookup(~/.arborist/shards) returns
exactly 92 core_roots (matches the 92/92 claim-pack records
resolved earlier today).
Total: 1652 tests pass (was 1643). Honest layering preserved:
- soft signal (positive warrant_proven) lives on a separate verdict
field, not in `violations` (which stays a hard-failure list)
- audit_mode (STRICT/HYBRID/UNGROUNDED) unchanged when chain
suppresses WARRANT_MISSING — the claim was going to land at
HYBRID without the suppression; the suppression keeps it at
STRICT, which is now defensible because the warrant chain IS the
warrant
- four-rung ladder rung promotes naturally: no WARRANT_MISSING in
violations + no soft demotes -> EVIDENCE-WARRANTED via the
existing _ladder_rung_for_lattice logic. No render-layer ladder
change needed.
Phase 3 follow-ups still open under #000031:
- via_citation_alias process_id attribution (~15 LOC)
- source-side title-from-author backfill in HTML/textbook_tex
ingest (~30 LOC)
Followup to 654d923 (which moved the package from arborist/v8/ →
arborist/substrate/ at the file layer). The CLI surface still baked
in `v8` so a new operator running `--help` would see
``arborist v8 score`` and ask the same "what's v8 vs v9.8?"
naming-confusion question that drove the package rename in the
first place. Closing the loop end-to-end.
arborist/cli.py
===============
- Subparser renamed: ``"v8"`` → ``"substrate"``; help string updated
to "Merkle-AGI substrate primitives (ForkScore + future paper
specs)" so the dir name and command name and help text all align.
- Inner subparser dest renamed: ``v8_op`` → ``substrate_op``.
- Function renamed: ``_cmd_v8_score`` → ``_cmd_substrate_score``;
docstring updated.
- All ``v8_score`` local variables renamed to ``substrate_score``.
- New comment block above the subparser block explains the rename
+ why the v-prefix was retired (substrate-paper version vs v9.8
schema version naming collision).
The old ``arborist v8 score`` is gone — no alias preserved. CI + ops
scripts must update; today's earlier commit chain has been the only
place using it and that's been refreshed in lock-step.
tests/test_v8_fork_score.py
===========================
- 4 ``parser.parse_args(["v8", "score", ...])`` calls → ``["substrate", ...]``.
- 4 test functions renamed: ``test_cli_v8_score_*`` →
``test_cli_substrate_score_*``.
- Module docstring + section comment + helper docstring updated.
Filename intentionally kept as ``test_v8_fork_score.py`` for git
history continuity; pytest discovers by ``test_*`` content, not
filename. Renaming the file would muddle ``git log --follow`` for
the test surface.
Docs refreshed
==============
- docs/v8-fork-score.md — §5 CLI block invocation.
- docs/_source/v8-fork-score.rst — :code-block:: bash invocation.
- docs/_source/bench.rst — invocation in `### v8 ForkScore` section.
- docs/tickets/ticket-000012-selection-consensus-protocol.md —
three references in §7 close-out + §7 Phase 1c proposal +
§7 future-CLI-shape note.
- docs/dav1dprometheus-update-2026-05-09.md — bench journal mention.
Doc filenames (``v8-fork-score.{md,rst}``) kept stable since they
are URL identities; the file content explains the v8→substrate
rename internally. ``index.rst`` toctree references unchanged.
Hygiene
=======
- ``.venv/bin/arborist substrate score --help`` → 0 + valid usage.
- ``.venv/bin/arborist v8 score`` → exits non-zero (subcommand
removed, surfaced cleanly in ``argparse`` error).
- ``make test`` → 1643 passed, 45 skipped.
- ``make chain-check-shards`` → 0 across all 7 shards.
- fox's parallel work in arborist/qa/{runner,verify}.py +
arborist/qa/warrant_chain.py left untouched.
Fox 2026-05-10: blackops kept padding decision menus with a "C — stop
here" option. Encode the discipline directly: when fox asks "next?"
list the actual moves (e.g. high-payoff Phase 3 vs low-risk cleanup
batch), don't dress up "let's call it" as a third choice. Fox keeps
going. If a task is genuinely complete, say so flat instead of as a
proposal option.
CLAUDE.md
=========
Architecture diagram refreshed to include every current arborist/
subdir (was missing capital/, concepts/, memory/, mesh/, pi_star/,
selfmodel/, substrate/, world/ and several top-level modules) and
the qa/ tree got the recently-landed members (canonical_cache.py
#000027, witness.py #000028, warrant_resolver.py #000031). New
explanatory paragraph names the topic-named convention explicitly:
> Topic-named, never version-prefixed. The substrate-paper version
> (v7 plastic-training, v8 selection/consensus, v9 falsification
> controller, …) and the live SQLite schema version (v9.8) are two
> unrelated numbering schemes that share decimals; version-prefixed
> dirs were tried 2026-05-10 and retired the same day because
> readers asked "is this schema-v7 or paper-v7?".
Future blackops shifts looking at where to put a new module now find
the answer in CLAUDE.md instead of having to reverse-engineer the
convention.
arborist/substrate/fork_score.py — fix broken import
====================================================
Previous commit 654d923 (the v7+v8 → substrate refactor) shipped
broken. Sequence:
1. ``git mv arborist/v8/fork_score.py arborist/substrate/fork_score.py``
— staged the rename.
2. Edited the file to flip ``from arborist.v8.weights ...`` →
``from arborist.substrate.weights ...`` — modified the working
tree but did NOT re-stage.
3. Selective ``git add`` of the other touched files (cli.py,
substrate/__init__.py, docs, tests) — fork_score.py's import
edit was NOT in the staged change set.
4. ``git commit`` shipped the rename WITHOUT the import fix.
HEAD's fork_score.py imports ``arborist.v8.weights`` — a path that
654d923 itself deleted. Tests passed locally because the disk had
the edit; a fresh checkout from origin/main breaks at import time.
This commit lands the import edit. Verified: ``git show
HEAD:arborist/substrate/fork_score.py`` now has
``arborist.substrate.weights``; full suite re-runs at 1643 passed.
Lesson: ``git mv`` stages the rename only. Modifications after that
need their own ``git add``. Selective-add commits should run
``git diff --cached`` (not just ``git diff``) before commit. Adding
this to my own habit list.
fox's read: the version-prefixed namespace pattern (`arborist/v7/`,
`arborist/v8/`) coupled module location to the substrate-paper
version. That collided with the live SQLite schema version (v9.8)
and made readers ask "is this dir tracking schema or paper?" —
a real onboarding hazard surfaced when the v7 dir landed earlier
today (06c95a0) for #000035 Phase 1.
Resolution: collapse v7+v8 into one topic-named dir,
``arborist/substrate/``, which holds Merkle-AGI substrate primitives
that future paper specs require — decoupled from the paper version.
Moves
=====
arborist/v7/anchor_prg.py → arborist/substrate/anchor_prg.py
arborist/v8/fork_score.py → arborist/substrate/fork_score.py
arborist/v8/weights.py → arborist/substrate/weights.py
Empty v7/ + v8/ dirs deleted; their __init__.py docstrings folded
into the new arborist/substrate/__init__.py with an explanation of
why the version-prefixed pattern was retired.
Imports updated
===============
- arborist/cli.py:_cmd_v8_score — arborist.v8 → arborist.substrate
- arborist/substrate/fork_score.py — internal weights import
- tests/test_anchor_prg.py — module + module-docstring
- tests/test_v8_fork_score.py — three import lines
Docs updated
============
- docs/v8-fork-score.md — header note explaining the move
- docs/_source/v8-fork-score.rst — :class: ref updated
- docs/tickets/ticket-000012-selection-consensus-protocol.md — §7
Phase 1a close-out paths refreshed (kept "Originally landed at
arborist/v8/..." parenthetical so the historical record survives);
§7 Phase 1b consensus-paper reference; §7 Phase 1c proposal §3
read-API path
- docs/tickets/ticket-000035-prg-choice-phi-prg.md — §7 Phase 1
close-out path refreshed (with full path-note explaining the
move); §3.1 + §5 left as the original design log per CLAUDE.md
"closed tickets stay in place as design log"
Left untouched
==============
- arborist/world/ — already topic-named; not version-prefixed; the
v7-W reservation lives there with its own planned subdir layout.
- docs/tickets/ticket-000037-prometheus-sigma-...md §13 still refs
``arborist/v9/prometheus.py`` and ``arborist/v8/fork_score.py`` —
fox has 792 lines of in-flight modifications on this file; those
refs should refresh to ``arborist/substrate/`` when the in-flight
edit lands. Avoiding interleaved edits.
Hygiene
=======
- make test → 1643 passed, 45 skipped (was 1643; refactor preserved)
- make chain-check-shards → 0 across all 7 shards
- arborist.substrate namespace picked up by the existing
pyproject.toml ``include = ["arborist*"]`` glob; no setup change.
Internal architecture reference written 2026-05-10 after the day's
18/92 -> 92/92 push under #000031. Covers:
- per-pillar substrate map (which open textbook covers which
pillar; license + ingest path for each of the 18 substrates)
- five proven ingest patterns (HTML single-URL, HTML BFS,
textbook_tex LaTeX-source, PDF -> localhost-HTML, direct
Python API)
- discipline patterns: title-from-author backfill workaround
(until the source-side fix in #000031 Phase 1 lands), alias
audit-fail-closed (decision_by + decision_rationale per row),
multi-substitute pattern, cascade tuning
- honest tier breakdown of the 92 chains (~25 direct primary,
~50 substrate substitution, ~17 soft-fallback OR-of-3 match)
- what the substrate doesn't yet do (render layer doesn't read
derivations, process_id under-attributes alias chains as DIRECT,
no per-record tier classification in the schema)
- re-running steps for future shifts (idempotent at DB layer)
Format follows other docs/ references (cti-architecture,
concept-relations-design, tool-action-dag-design) — describes
state of the world, not proposing change.
CLAUDE.md and TICKETS.md updated to point at the cookbook from
the docs index.
No undefect/whitepaper publication — this stays internal as
requested. Future blackops shifts re-discovering the substrate
map shouldn't have to walk five bench journals.
Keller-Trotter Applied Combinatorics (CC-BY-SA-4.0) re-crawled
successfully on the 38-min retry — earlier 25-min timeout was
too tight against appliedcombinatorics.org's 20-second crawl-delay.
Result: 80 docs / 168 chunks of advanced enumerative combinatorics
(generating-functions, partitions, Polya enumeration) that Bogart
skips and Levin treats lightly.
Title-backfilled the 79 newly-ingested docs with author surname
'Keller and William T. Trotter' for resolver _shard_matches_citation
heuristic.
Registered 14 KT alias rows for pillar VII compound source_refs as
a third substitute alongside Bogart + Levin. Audit-disciplined
(decision_by="fox 2026-05-10", decision_rationale notes Georgia
Tech AIM-approved provenance + advanced-chapter coverage).
Total citation_aliases now 54 rows (was 40); coverage stays at
92/92 (100%) — pillar VII chains gain peer-level redundancy
without changing the per-record resolution status.
Cleaned up orphan crawl_appliedcombinatorics_org.db from the
earlier failed crawl (no schema; tripped the resolver's
_shard_title_haystack with a missing-table OperationalError).
Doc updates (TICKETS.md, #000031, #000041) reflect 54-row count
and KT as third pillar-VII substrate.
Lands the M1 mitigation cryptographic primitive that ticket #000018
§5.2 + §9.2 specified, scoped per ticket #000035 §3.1-§3.3. Pure
stdlib (hashlib + hmac); no third-party dependency.
arborist/v7/__init__.py
=======================
First module landed under the v7 namespace. v7 plastic-training is
currently paper-stage (per #000037 §17.2); this is where its
deterministic primitives accumulate ahead of an active deployment
target so the building blocks are unit-tested + KAT-pinned the
moment v7 needs them.
arborist/v7/anchor_prg.py
=========================
Implements ``phi_prg(hard_hash_32, dim_h, *, seed) -> list[float]``
per #000035 §3.1. Construction is SP 800-108 KDF in counter mode
over HMAC-SHA-512:
Output(SEED, C(M), n_bytes) :=
i = 0
out = b""
while len(out) < n_bytes:
out += HMAC-SHA-512(SEED, C(M) || i.to_bytes(4, 'big'))
i += 1
return out[:n_bytes]
Float conversion: f(u32) := 2 * (u32 / 2**32) - 1
↑ uniform on [-1, 1)
Security argument from #000035 §2.1: HMAC-SHA-512 is a PRF under
the standard SHA-512 + HMAC assumption; distinguishing advantage
from random bounded by SHA-512 collision-resistance (~2^256), which
structurally matches the substrate's SHA-256 hard-hash family. The
seed is published, not secret — secrecy is not the security
property; the property is computational indistinguishability of the
output from random, which holds even when the seed is public.
Module exports ``PHI_PRG_VERSION = "phi-prg-v1-hmac-sha512"`` so
future algorithm rotations are detectable at the call site without
string-comparing module paths. Per #000035 risk §6.1, a
``phi_prg_version`` field in the v7 manifest will let future
deployments swap to a successor PRF without breaking historical
replay; this version string is the runtime-side mirror.
Hard-hash input length checked exactly at 32 bytes — silently
padding shorter input would break the PRF security argument.
``dim_h`` validated as positive int.
tests/test_anchor_prg.py
========================
20 tests covering #000035 §3.2 acceptance criteria + the strict
range invariant + the input-validation surface:
- Determinism: same (seed, hard_hash, dim_h) → identical floats.
- Range: every output in [-1, 1) with strict upper bound. Three
edge cases pinned: u32=0 → -1.0, u32=2^31 → 0.0,
u32=2^32-1 → just below 1.0.
- Chi² loose-uniformity: 4096-sample bin-test (df=15) with a
generous threshold (60); catches catastrophic PRG bugs (counter
cycling, mis-keyed HMAC) without claiming cryptographic-grade
evidence.
- Boundary: dim_h=1 + dim_h=16384 both produce sensible output.
- Avalanche, seed-bit: flip top bit of seed[0]; require 35-65% of
output bits flipped (PRF avalanche property).
- Avalanche, hash-bit: same surface for the hard-hash input.
- Validation rejects: short hashes, long hashes, non-bytes hashes,
zero / negative / non-int dim_h.
- Module export shape: PHI_PRG_VERSION + PLACEHOLDER_SEED.
- KAT regression: pinned vectors verified against
``bench/fixtures/phi-prg/known-answer-tests.jsonl``.
bench/fixtures/phi-prg/known-answer-tests.jsonl
================================================
10 KAT vectors generated against the placeholder seed + custom
seed/hash combinations; covers smoke (placeholder seed × small
dim_h), block boundaries (HMAC-SHA-512 blocks are 64 bytes, so
dim_h=16 is exactly one block, dim_h=17 is two blocks with
truncation), seed/hash one-bit-flip variants, and a 4096-element
stress sample.
Each row pins the SHA-256 of the raw byte stream (not the float
list) — that's the durable contract; switching from list[float] to
array.array('f', ...) or numpy arrays at the float layer would not
invalidate the fixture. Algorithm changes MUST bump
PHI_PRG_VERSION and create a new fixture file under
bench/fixtures/phi-prg/; old runs replay against old data per the
v7 spec replay discipline.
#000035 status flip
===================
docs/tickets/ticket-000035-prg-choice-phi-prg.md §7 updated from
"open · awaiting go/no-go" to "in progress · Phase 1 landed
2026-05-10". §7 now carries Phase 1 close-out details + Phase 2
gating criteria (active v7 deployment target + spec maintainer
review of §3.4 amendment text). The §3.4 v7 §9.10 amendment text
stays as the draft awaiting Phase 2 landing.
docs/TICKETS.md index row was already updated by fox in commit
ed470dc; my edit was idempotent.
Hygiene
=======
- make test → 1643 passed, 45 skipped (was 1623; +20 anchor_prg)
- make chain-check-shards → 0 breaks across all 7 shards
- arborist.v7 namespace picked up automatically by the existing
pyproject.toml [tool.setuptools.packages.find] include="arborist*"
glob; no setup change required.
- fox's in-flight #000037 ticket modifications + a parallel
#000031 ticket update left untouched.
Update #000031 ticket with the 2026-05-10 push details: 18 textbook
substrates ingested, 40 citation-alias rows + 13 term-alias rows,
cascade tuning in _build_record_query_cascade. Per-pillar end state
recorded (13/13 · 10/10 · 13/13 · 18/18 · 5/5 · 5/5 · 14/14 · 14/14).
Honest tier breakdown of the 92 chains added. Phase 1 follow-up
(source-side title-from-author backfill) and via_citation_alias
attribution fix surfaced as open follow-ups under the same ticket.
Update #000041 with full registry of 40 citation aliases now live
(distribution by substrate; per-pillar breakdown). Update #000042
with 13-row count + acknowledgement that Newton/arithmetic rows are
mostly unused (cascade picked different tokens).
Update CLAUDE.md with the 18 per-textbook make targets + the 100%
warrant-resolve milestone.
Update TICKETS.md index status for #000031, #000041, #000042.
No code change in this commit; documentation refresh only.
The +14-record close-out: most stuck records had substrate content
in already-ingested shards but the FTS5 cascade picked discriminator
tokens the substrate text didn't use as titles. Two surgical
additions to _build_record_query_cascade:
(1) parenthetical-phrase variant via new _phrase_from_parenthetical
helper. The discriminating term often lives INSIDE a trailing
parenthetical that _phrase_for_axiom was stripping:
"Kolmogorov's First Axiom (Non-Negativity)" -> '"non negativity"'
"Newton's First Law (Law of Inertia)" -> '"inertia"'
"Axiom of Side-Angle-Side (SAS)" -> '"sas"' (already had this via prior path)
(2) content-tokens OR-join (top 3) as soft fallback after the AND
join. Most stuck records had top-5-AND scoring 0 hits even when
substrate had clear content; OR-of-top-3 surfaces relevant
chunks reliably without sacrificing precision (citation +
title-haystack guards remain).
Plus one final citation-alias row: Boehm-Jacopini -> Software
Foundations LF (decision_by="fox 2026-05-10"). SF-LF Imp.v
formalizes the structured-program theorem in Coq with sequence /
conditional / while-loop — same theorem as Boehm-Jacopini 1966
CACM in modern formal-verification dress.
Per-pillar:
I: 12/13 -> 13/13
II: 9/10 -> 10/10
III: 11/13 -> 13/13
IV: 18/18 stable
V: 2/5 -> 5/5
VI: 3/5 -> 5/5
VII: 11/14 -> 14/14
IX: 11/14 -> 14/14
Total: 92/92 (100%)
Cumulative day-of-2026-05-10 lift: 18/92 -> 92/92 (+74 records,
+80 pp absolute). Started the day with only Hilbert pillar IV
resolved; ended with all 8 pillars fully resolved across 13
textbook substrates (Hilbert, Boole, Cantor, De Morgan, Russell
IMP/PoM, PM Vol 1, Peano, Dedekind, Newton, Laplace, Grinstead-
Snell, PLFA, SF-LF, Bogart, Levin, Judson, Aristotle).
40 citation-alias rows + 13 term-alias rows, all audit-disciplined
per #000041/#000042 (decision_by + decision_rationale per row).
All 1623 tests pass.
Under fox "work all paths" directive, four orthogonal lifts:
(1) 14 Levin alias rows for pillar VII compound citations — each
Stanley/Brualdi/Knuth ref now has BOTH Bogart AND Levin as
substitute. Lifted +2 (hockey-stick + stars-and-bars).
(2) 6 Goedel multi-substitute aliases — Goedel citations now have
4 substitutes each (Peano, Russell IMP, Dedekind, SF-LF). Lifted
+4 pillar III records (Axiom of Zero / Distinctness / Addition-
Successor / Multiplication-Successor variants).
(3) 2 SF-LF alias rows for pillar I formal proof axioms (Mendelson +
Enderton -> SF-LF). SF Logic.v has 12 hits "excluded middle",
40 "existential", 1 "double negation". Lifted +2 (Existential
Introduction, Double Negation Elimination).
(4) Grinstead-Snell Introduction to Probability ingest (GFDL,
Dartmouth CHANCE Project). Initial LibreTexts BFS crawl thin
(24 chunks nav-only at depth=2). Switched to Dartmouth PDF
-> pdftotext -layout -> single-page HTML on localhost:8765
-> standard arborist html ingest. Result: 339 chunks of
substantive probability content. Lifted +1 (Inclusion-Exclusion).
New ingest pattern useful for any PDF-only textbook.
Per-pillar:
I: 10/13 -> 12/13 (+2)
II: 9/10 stable
III: 7/13 -> 11/13 (+4 — biggest win)
IV: 18/18 stable
V: 1/5 -> 2/5 (+1)
VI: 3/5 stable
VII: 9/14 -> 11/14 (+2)
IX: 11/14 stable
Total: 77/92 (84%)
7 term aliases registered (physics Newton-vocab + arithmetic Peano-
vocab) but most didn't lift — cascade picks different tokens than
my aliases target. Cascade tuning would require code work in
_build_record_query_cascade / _phrase_for_axiom. Deferred.
Cumulative day-of-2026-05-10: 18 -> 77 / 92 (+59 records, +64 pp).
15 records still stuck. Top blockers:
- Cascade-tuning for substrate-content-exists-but-FTS5-misses
(Kolmogorov 3 axioms, Newton third law, Vandermonde, Catalan,
Injectivity of Successor, etc.) — code work
- PM *24+ not transcribed (Russell's Paradox Resolution)
- Boehm-Jacopini esoteric (no PD substitute)
Broader stale-map sweep across docs/, bench/, scripts/, Makefile, and
top-level surfaced one hit: the witness-mode example sketch in §2.6
of ticket #000028 still showed a no-op cache_lookup closure with
"TODO: wire post-#000027" — even though the real implementation in
arborist/qa/query.py:2158-2172 wired the cache leg in commit e19aed8
(2026-05-09), the same commit that closed both #000027 and #000028.
Updated the sketch to match the actual implementation: the cache
lookup closes over the prior persisted row's answer bytes (encoded
with surrogatepass for UTF safety), and the comparison is non-
tautological because cached_row is the row picked up BEFORE we'd
write a new one in this same call.
Same drift pattern as the #000028 §8 follow-up refresh (6d20aeb)
and the #000010/§000021/#000040 sweep (e84f453) and the repair.py
TODOs (8980e64): pre-implementation design notes don't get refreshed
after the implementation lands. Each instance is one session-waste
saved.
After this, both surfaces are clean:
- arborist/ + tests/ stale TODOs: zero (8980e64)
- docs/ + bench/ + scripts/ + Makefile + top-level: zero (this commit)
The remaining matches are all genuine future-work markers (raw_html
cache in async_web_fetcher.py:2309) or false-positives (\\uXXXX
escape-pattern docs in mesh/wire.py and tests/test_cli_render.py).
Three threads landed in a single commit because they share the same
substrate (the #000037 §12 trigger probe shipped in c422216):
A — code-side stale-map sweep (arborist/qa/repair.py)
=====================================================
Walked inline TODO/FIXME/XXX markers across arborist/ + tests/. Five
hits: two false-positives (\\uXXXX in escape pattern docs), one
genuine deferral (raw_html cache in async_web_fetcher), and **two
stale TODOs in arborist/qa/repair.py** referring to "re-prompt
feedback path is future work" — even though `reprompt_repair` is
fully implemented (lines 140+ at the time of this commit), wired
through CLI `--repair-reprompts N` flag (cli.py:4574), and gated by
`policy["repair_max_reprompts"]`. Refreshed the docstring header
and the in-body comment to point at the actual function.
Same drift pattern as today's earlier ticket sweep (e84f453):
implementation lands, the TODO doesn't get refreshed, future readers
re-implement what's already there.
B — N-power follow-up to the §12 trigger probe
==============================================
c422216's first probe run reported trigger 2 (divergence variance)
at N=16, σ=0.5, ratio=0.8 — both threshold conditions would fire if
N reached the 30-sample N_min. To validate that the variance signal
holds at N≥30 (rather than vanishing on a wider sample), drove the
canonical-witness path 20 additional times via a new
bench/qa_questions_canonical_witness_npower.txt fixture
(10 arithmetic@v1 + 10 logic-kernel@v1 questions; all
canonicalize-then-LLM-witness without errors).
Result at N=37: trigger 2 fires. Ratio 0.575 > 0.5, abs σ 0.435 > 0.10.
Variance signal is real at the floor. Captured in
bench/results/prometheus-sigma-triggers-2026-05-10-npower.md with a
loud caveat at the top: this is N-power validation, NOT a measure of
real workload pressure. The §12 phase-1 go/no-go decision should
still come from natural workload N or operator-stated need.
The prior baseline (prometheus-sigma-triggers-2026-05-10.md) stays
in place as the workload-state-at-time-of-ticket-c422216 record;
this new -npower.md report is the statistical-power follow-up.
C — ForkScore Phase 1c proposal (#000012)
=========================================
#000037 §12 Trigger 1 ("ForkScore receives ≥4 candidate branches per
checkpoint") gates on multi-branch persistence — but #000012 Phase 1a
(landed 2026-05-08) is single-validator scoring of one fork at a
time, and Phase 1b (still open) is the consensus paper. Neither
persists branch-sets. So Trigger 1 structurally cannot fire today,
which the probe correctly reports as "no fork_score branch-set table
found".
Added Phase 1c to #000012 as a doc-only proposal (no code in this
commit):
- New `fork_score_branches` table (sibling to capital_ledger; does
NOT enter audit_events.event_hash preimage).
- Optional `--branch-set <ID>` flag on `arborist v8 score`.
- One read API: `branch_set_density(conn, branch_set_id)`.
- #000037 probe wires through the new function once it lands.
Not opened as its own ticket because operator pressure for it
hasn't surfaced naturally — gates on either #000012 Phase 1b
landing OR #000030 algebra/calc kernel expansion producing
competing-kernel branches an operator wants to compare. Captured
here so a future shift doesn't re-discover the gap.
Hygiene
=======
- make test → 1623 passed, 45 skipped
- make chain-check-shards → 0 across all 7 shards
- fox's in-flight #000037 ticket modifications left untouched
Phase 0 of ticket #000037 (Prometheus-Σ recursive falsification
controller) is doc-only and gates Phase 1 on §12 measured-pressure
triggers. This commit lands the empirical-evidence harness fox needs
for the go/no-go decision, plus the §16.2 test scaffolding so the
contract is discoverable from the test runner today.
bench/prometheus_sigma_trigger_probe.py
=======================================
Read-only walk of audit_events + capital_ledger across shards.
Reports each §12 trigger:
- Trigger 1 (branch density, ≥4 branches/checkpoint): looks for a
fork_score branch-set table; reports "no data — single-validator
ForkScore Phase 1a" when absent. Trigger structurally cannot fire
until #000012 multi-branch persistence lands.
- Trigger 2 (divergence variance, N≥30 + ratio>0.5 OR abs>0.10):
aggregates providence_canonical_witness audit-event bodies,
maps each agreement_label per #000028 §1.2 to a binary
LLM-divergence score, computes mean/stddev/ratio. Defensive on
the max(mean,ε) guard from §12 itself.
- Trigger 3 (witness cost share > 0.30): aggregates capital_ledger
rows; uses `material` (kWh proxy) as the canonical compute axis;
surfaces both `material` and `financial` so fox can pick a
different form if needed. Tags the report with the caveat that
the current ledger reflects ad-hoc activity, not a controlled
#000026 sweep.
- Trigger 4 (operator mission need): n/a — operator decision.
Pure measurement, no LLM, no schema change, no mutation. Output is
a markdown report at $(PROMETHEUS_PROBE_OUT) (default
bench/results/prometheus-sigma-triggers-<utc-date>.md).
Wired via `make prometheus-trigger-probe`.
bench/results/prometheus-sigma-triggers-2026-05-10.md
=====================================================
First captured baseline. Verdict on current shards:
Trigger 1: NO (no fork_score branch-set table; #000012 Phase 1a
is single-validator)
Trigger 2: NO (16 samples; N_min=30. But mean=0.625, σ=0.5,
ratio=0.8 — both ratio AND abs floors would fire
if N reaches 30. Signal is there; just needs more
samples.)
Trigger 3: NO (witness/total material = 0.005, well under 0.30
threshold. Caveated: ledger is ad-hoc, not a
controlled #000026 sweep.)
Trigger 4: n/a (operator-stated)
Empirical answer: no §12 measured-pressure trigger has fired yet.
Phase 1 of #000037 remains paper-only unless fox invokes Trigger 4.
tests/test_prometheus_sigma.py
==============================
17 skip-stubs pinning the §16.2 acceptance contract. Each test:
- Collects today via `pytest --collect-only` so the test surface is
discoverable.
- Skips with reason "Phase 1 not landed; controller module
arborist/v9/prometheus.py absent" until that import succeeds.
- Carries a one-sentence intent line tying it back to a numbered
ticket section (§5 / §6 / §7 / §10 / §13 / §14 / §16.1 / §4.4).
When Phase 1 lands, the implementer adds the controller module
under arborist/v9/prometheus.py per §13; CONTROLLER_AVAILABLE
becomes True; each test gets its body filled in. The names, intents,
and skip-reason strings are the durable contract.
Hygiene
=======
- make test → 1623 passed, 45 skipped (was 28; +17 new skips).
- make chain-check-shards → 0 breaks across all 7 shards.
- No code touched outside the new probe + scaffolding files +
Makefile target wiring. fox's in-flight #000037 ticket
modifications left untouched.
Three orthogonal lifts under fox 2026-05-10 "all of the above"
directive:
(1) 7 medium-confidence citation aliases per #000041:
Mendelson -> Russell IMP + De Morgan FNL + Boole Laws of Thought
Enderton -> Russell IMP + De Morgan FNL
Jech -> Cantor 1915 Contributions
Landau -> Russell IMP
Each row carries partial-coverage caveat in decision_rationale.
(2) Peano Arithmetices Principia 1889 ingested via existing
textbook_tex pipeline. Source: mdnahas/Peano_Book GitHub
repo (CC-BY-SA-4.0, Verheyen+Nahas English+Latin LaTeX).
paracol multi-column macros leak some residue but substantive
text extracts (39 chunks, 35 hits on "peano", 38 on
"arithmetic"). Make target textbook-peano. Manifest entry
peano-arithmetices-principia-1889. Plus 3 substitute-citation
aliases:
Landau -> Peano (lifts 2 pillar III records)
Goedel -> Peano (2 variant strings)
(3) Title-from-author backfill on Cantor / Russell IMP / Bogart /
Judson / Peano shards. Empirical defect: HTML/textbook_tex
sources took titles from <title> tags or URLs which on
Wikisource + PG do NOT include author surname, so the
resolver's _shard_matches_citation heuristic (requires
cited-author surname in title haystack) failed silently for
all alias-resolved chains where the substitute author wasn't
self-evident in the shard content. Surgical SQL UPDATE on
documents.title fixed it. Single biggest lift in this turn:
+13 records.
Per-pillar:
I: 4/13 -> 7/13 (+3, Russell IMP / Boole / De Morgan)
II: 0/10 -> 5/10 (+5, Cantor)
III: 0/13 -> 7/13 (+7, Peano + Russell IMP)
IV: 18/18 stable
V: 0/5 stable (Kolmogorov URAA-blocked)
VI: 3/5 stable
VII: 9/14 stable
IX: 5/14 stable
Total: 54/92 (59%)
Title backfill is a one-off SQL; the proper long-term fix is in
the source ingest pipelines (HTML / textbook_tex should pull
author from manifest's `author` field at ingest time and prepend
to document.title). Tracked under #000031 Phase 1 follow-up.
Pattern from #000028 §8.2/§8.3 (commit 6d20aeb): tickets carry
"deferred" markers in follow-up sections written from pre-
implementation design notes; code lands but the markers don't get
refreshed; future readers waste a session re-implementing what's
already there. Walked all 19 closed tickets carrying deferred-class
mentions and cross-checked each one against current code.
Three confirmed stale markers refreshed:
#000010 §12.2 — "First pass does NOT bind into run_dag_root —
deferred to ticket #000009 (Phase 5)". The metacog binding DID
land. Commit 111dda6 ("qa(#000009): §8 corrections — reject-path
DAG + nested CTI clauses", 2026-05-04) extended
arborist.qa.dag.build_preflight_node_payload with a question_state
clause carrying the metacog to_dict() payload. Today every cache
write threads question_state.to_dict() into the preflight node
(arborist/qa/query.py:3296-3297). Refreshed the §12.2 paragraph
to point at that commit + call site.
#000021 Phase 1b "(deferred)" — items "Synthesis, Syllogism,
Semiotics under 5S; Triangulate, Timing, Transitivity, Truth under
5T". Phase 1b was scope-split into per-sub-battery tickets:
#000023 (5S) and #000024 (5T), both closed 2026-05-08. The Closure
section in this ticket already reflects the resulting fixture
totals (108 5S + 154 5T) but the Phase 1b sub-section still said
"deferred". Refreshed to point at #000023 / #000024.
#000040 §6 — "Cascade landed 2026-05-09 (commit TBD)". TBD was a
placeholder while the commit was in flight; landed as b9e5bbd
("ticket #000040 Phase 5: phrase + content-token resolver
cascade"). Replaced TBD with the real hash.
Items confirmed genuinely deferred (no refresh needed) — recorded
here so the next sweep doesn't re-walk them: #000010 reference-
frame plumbing / SOFT_PREFLIGHT_HINT / metacog-trigger bench;
#000011 §13 bench plan; #000014 selfmodel_binding cache-key
folding (only docstring mentions; no actual gating wiring) /
selfmodel diff CLI / cross-shard reconciliation; #000015 / #000017
out-of-scope items; #000019 auto-canonicalizer / cross-language /
source-adapter methodology; #000020 stewardship-halt / conversion
tables / real-time instrumentation; #000023 multimodal execution.
Pure docs change; no code touched.
Three high-confidence alias decisions per fox 2026-05-10 + 14
full-string aliases for compound pillar-VII citations + a deeper
Newton recrawl.
Lift breakdown:
pillar IV (Hilbert): 18/18 unchanged
pillar VI (Goldstein/Newton): 0/5 -> 3/5 (+3)
pillar VII (Stanley/Bogart): 0/14 -> 9/14 (+9)
pillar IX (Dummit-Foote/Judson):0/14 -> 5/14 (+5)
Citation aliases live in ~/.arborist/shards/000.db (audit table per
#000041). 14 pillar-VII rows are full-string matches against the
compound source_ref strings (resolver looks up by exact source_ref;
parsed sub-citations are not yet alias-keyed individually).
Newton shard rebuilt: original `crawl_en_wikisource_org_newton.db`
followed Wikisource sidebar instead of book content (60 docs / 289
chunks of mostly nav cruft). Manifest now pins (1729) edition root
+ explicit subpage urls (Axioms_or_Laws_of_Motion, Definitions,
Rules_of_Reasoning_in_Philosophy); recrawl produced 120 docs / 779
chunks of actual Principia text in textbook_newton-principia-motte.db.
5 records stuck despite alias registered (FTS5 cascade misses):
Newton Third Law + Conservation of Momentum (modern vocabulary
absent from 1729 prose), 4 advanced pillar-VII identities (Hockey-
Stick, Vandermonde, Catalan, Stars-and-Bars — Bogart skews intro-
level), Orbit-Stabilizer (Judson covers it but cascade doesn't
surface). Tractable one-by-one; deferred.
Pillars I/II/III/V/lambda subset of IX still 0/X — those need
medium-confidence alias decisions (Mendelson/Enderton -> Russell IMP,
Jech -> Cantor) or Phase 4 textbook acquisitions (Landau, Goedel,
Kolmogorov, Church) under #000038.
§8.1 — Witness-vs-modality terminology clarification (pure docs).
The ticket title and §1 throughout use "modality" — operationally
meaning kernel / cache / LLM. The 2026-05-09 review (response_ticket-
000027-canonical-projections-in-providence-cache.txt) flagged that a
future reader looking for cross-carrier work (text↔image, audio,
world-state) might land here by mistake, since "modality" in that
sense means a different thing.
Added a one-line note near §1 distinguishing *witness channel* (an
independent epistemic source) from *carrier* (text / image / audio /
world state), and pointing forward to a later ticket for cross-
carrier extensions once π* libraries support non-text carriers.
§8.2 + §8.3 — status refresh from "deferred" to "landed in 708aa45".
Both follow-ups already shipped in commit 708aa45 ("fan-out: warrant
ladder wiring · witness follow-ups · 5F Phase 1d", 2026-05-09 12:42
EDT) — pre-dating the e19aed8 close commit. The ticket file's §8.2
and §8.3 sections still carried the "deferred" labels because they
were written from the pre-implementation design notes; refreshing
both to point at the actual implementation in
arborist/qa/query.py (lines 2140-2155 for sample-rate gating;
2237-2285 for the capital_ledger sidecar) and the corresponding
test_witness.py coverage. Map-vs-territory hygiene: stale ticket
markers are how future readers waste a session re-implementing
something that's already there.
No code change in this commit. 33/33 tests in test_witness.py
continue to pass, all 7 shards chain-check clean.
The bench-qa-smoke A/B against commit 416f956 surfaced a methodology
gap: the smoke fixture is structurally insensitive to retrieval-side
changes — every smoke query succeeds at full-AND chain 0 on every
shard, so cache_keys match byte-for-byte BEFORE/AFTER and any
STRICT-rate variance is pure LLM dice. Discovered when 5pp signal
floor at n=3 produced ±20pp swings with identical cache_keys.
This commit pins a fixture that ACTUALLY exercises the new code path:
- 5 PROGRESSIVE queries (full-AND fails on at least one shard;
_progressive_and_token_chains drops shortest-tokens-first). Includes
the canonical Gundremmingen case from commit 2b9d1f0.
- 2 DF_FILTER queries (every AND chain fails on at least one shard;
search lands in OR-mode where _filter_or_pool_by_df trims
high-DF tokens).
- 2 CONTROLs (full-AND succeeds on every shard; cache_keys must
remain identical BEFORE/AFTER for any retrieval-side change).
Which-chain-fires-on-which-shard was empirically verified at HEAD
against ~/.arborist/shards (4 wiki shards, 1.5M chunks each). Header
documents the chain that wins per query; re-probe if shards drift.
Baseline A/B (commit 416f956 ON vs OFF, n=3, --burn between samples,
fixture run via make bench-qa-progressive-and):
category cache_key parity STRICT-rate delta median latency delta
PROGRESSIVE 9/15 drifted +0.0pp (11/45 both) -3.6s
DF_FILTER 3/6 drifted +5.6pp (10/18 → 11/18) -3.9s
CONTROL 0/6 drifted +0.0pp (15/18 both) +0.0s
The CONTROL +0.0s latency delta is the load-bearing diagnostic —
when retrieval is byte-identical the search-side change cannot
affect total wall, and it doesn't. PROGRESSIVE / DF_FILTER cache_key
drift is the change actively redirecting retrieval, with the
expected latency win and no STRICT regression.
Future retrieval-side work (#000039 sqlite-vec backend, synonym
expansion changes, embedding rerank, hybrid scoring) should bench
against this fixture in addition to the full bench-qa scoreboard.
Hygiene:
- make test → 1623 passed, 28 skipped
- make chain-check-shards → 0 breaks across all 7 shards
Adds Cantor (Jourdain 1915 transl., PD), De Morgan First Notions of
Logic (PG #67017, PD), Russell Introduction to Mathematical Philosophy
(PG #41654, PD), and Judson Abstract Algebra Theory and Applications
(GFDL-1.3) to the textbook manifest, plus per-book make targets +
a `textbooks-base-knowledge` bulk target. Idempotent at DB layer.
These four primaries cover the proprietary citations across pillars
I/II/III/IX (Mendelson, Enderton, Jech, Barendregt, Dummit-Foote)
that dominate the 74 unresolved claim-pack records. They land as
honest "no match" under the existing resolver `_shard_matches_citation`
heuristic (cited-author surname required) — full lift gates on
fox citation-alias decisions per #000041.
Total ingest delta: 43 docs / 295 chunks across four crawl shards.
Bench journal under bench/results/textbook-base-knowledge-upgrade-2026-05-09.md
records per-book license rationale, projected lift table, and the
remaining #000038 acquisition gaps (Hilbert-Ackermann, Landau,
Gödel, Kolmogorov, Church) for which no clean PD HTML edition was
located in this session.
Smoke-test alias rationale was published with #000042, but two of the
three trailing records ("non triviality", "side-angle-side") needed
their own per-concept aliases since Hilbert 1902's vocabulary differs
in more than one place. Map of registered aliases (geometry domain):
incidence -> connection (4 records lifted)
euclidean parallel postulate -> axiom of parallels (1)
non triviality -> space axiom (1)
side angle side -> included angle (1)
Hilbert pillar IV is now the first complete warrant-chain vertical:
18/18 records bind to a primary-source surface chunk via Merkle
inclusion proof. 11 direct + 7 via-alias derivations rows. Bench
journal under bench/results/hilbert-pillar-iv-fully-resolved-2026-05-09.md
records reproducibility steps + per-alias signal data so a future
shift can extend the same pattern to Boole 1854, Newton 1729 Motte,
and Aristotle's traditional translations once #000038 lands those
surfaces.
Cluster-wide remains 18/92 (20%); the other 74 records cite textbooks
not yet ingested. Phase 4 content acquisition (#000038) gates next
lift.
Implements both alias mechanisms (citation-aliases #000041,
term-aliases #000042) as one cohesive layer. Same audit
discipline; same opt-in via --use-aliases on warrant-resolve;
same distinct process_id "warrant-resolver-v1+alias" on alias-
resolved derivations rows.
What landed
===========
arborist/store.py — two new tables under SCHEMA_SQL:
citation_aliases — substitute textbook for proprietary cite
term_aliases — bridge vocabulary mismatches (incidence ↔
connection in geometry, etc.)
Both with NOT NULL audit fields (decision_at + decision_by
+ optional decision_rationale); both with PK constraints
ensuring idempotent re-add.
arborist/qa/aliases.py — helper module:
add_citation_alias / list / lookup / remove
add_term_alias / list / lookup (bidirectional) / remove
expand_query_with_term_aliases — OR-rewrites FTS5 tokens
while preserving phrase syntax
domain_for_pillar — Roman numeral → domain string
arborist/qa/warrant_resolver.py — two-pass cascade:
Pass 1: unaliased queries (matches carry via_alias=False)
Pass 2: alias-expanded queries (only when pass 1 missed;
matches carry via_alias=True)
ResolutionMatch grew a via_alias field; warrant_resolve
uses it to pick the right process_id per derivation row.
iter_claim_pack_records now yields a 6-tuple including the
pillar (parsed from doc URI) so domain lookup works.
arborist/cli.py — alias subcommand group:
arborist alias citation add ORIGINAL --substitute SUB --by FOX [...]
arborist alias citation list / remove
arborist alias term add TERM ALT --domain D --by FOX [...]
arborist alias term list / remove
warrant-resolve --use-aliases flag
sweep --target warrants --use-aliases flag
All audit fields fail-closed at the API surface (refuses on
empty --by; ValueError raised at the helper level).
End-to-end smoke test
=====================
Registered the Hilbert smoke-test alias:
arborist alias term add incidence connection \
--domain geometry \
--by "blackops 2026-05-09 (smoke test)" \
--rationale "Hilbert 1902 Townsend uses 'connection' for
what modern texts call 'incidence'"
Re-ran warrant-resolve --use-aliases --write:
records_total: 92
records_resolved: 15 (was 11 without aliases)
derivations_written: 15
Breakdown by process_id:
warrant-resolver-v1: 11 (original-citation matches)
warrant-resolver-v1+alias: 4 (alias-resolved Hilbert axioms)
The 4 alias-resolved records are exactly the Hilbert "Incidence"
axioms blocked by terminology mismatch in #000040 §6:
Axiom of Line Incidence
Axiom of Plane Incidence
Axiom of Point-Line Incidence
Axiom of Point-Plane Incidence
All four bound to chunk 56 in the Hilbert TeX surface — the
chapter discussing "axioms of connection" (Hilbert's original
1902 vocabulary). Audit trail correctly distinguishes
substituted chains from original ones.
Tests: 18 new in test_aliases.py covering add / list / lookup
/ remove / domain isolation / lowercase normalization /
bidirectional lookup / audit-discipline raises / query
expansion (basic + phrase-preserving + no-match passthrough +
unreachable-DB fallback). Full suite: 1623 passed / 28
skipped.
Tickets #000041 + #000042 closed. Operators can now add more
aliases via the CLI as fox makes decisions per #000038. The
alias mechanism is fail-closed by default — existing
warrant-resolve runs without --use-aliases continue to produce
the original 11/92 chains; --use-aliases opt-in adds the
substituted chains alongside without polluting the unsubsituted
audit trail.
Two related changes:
#000042 — Term-aliases table (vocabulary-mismatch bridge)
=========================================================
Sibling design to #000041 citation-aliases. Maps a (term,
domain) pair to an alternate term used in older / foreign /
pre-modern translations of the same concept. Triggered by the
Hilbert "incidence" vs "connection" gap surfaced in #000040
§6 — claim-pack records use modern post-1950s names, the 1902
Townsend Hilbert translation uses the original "Verknüpfung"
/ "connection".
Resolver wiring: each FTS5-query token gets OR-expanded with
its registered aliases when --use-aliases is set. So '"line
incidence"' becomes '"line incidence" OR "line connection"'
once fox approves the (incidence, connection, geometry) alias.
Decision audit: each row carries decision_at + decision_by +
decision_rationale. Same audit discipline as #000041; same
opt-in via --use-aliases; same distinct process_id
("warrant-resolver-v1+alias") on alias-resolved derivations
rows.
A single decision unlocks 7 stuck Hilbert pillar IV records:
arborist alias term add incidence connection \\
--domain geometry --by "fox YYYY-MM-DD" \\
--rationale "Hilbert 1902 Townsend uses 'connection' for
what modern texts call 'incidence'"
Implementation deferred until that decision triggers it.
`arborist sweep` CLI — Target B warrant-resolver fragment
=========================================================
Implements the schema-no-change increment of #000037 §3.1
Target B (documents that bypassed meta-cognition at ingest
time).
arborist sweep --shards-dir X --target warrants [--write]
`--target warrants` walks every claim-pack record and
re-runs warrant_resolve. Same code path as
`arborist warrant-resolve` but framed as the unconscious
sweep — operators can run on a cron / systemd timer; the
target name reserves namespace for the full bicameral sweep
landing later (canonical-projection probe, freshness probe,
document-content witness fan-out).
`--target all` reports "deferred" with an honest message:
the full sweep needs #000037 §12 phase trigger +
documents.last_swept_at schema migration.
Idempotent: re-running on the same shards is a no-op at the
DB layer (PK collision on (core_root, src_root, process_id)
in the existing derivations table). Operators can run the
sweep on a recurring schedule without row proliferation —
exactly the property #000037 §3.1 needs from the unconscious
sweep.
Test suite stays at 1605 passed / 28 skipped — pure CLI +
new ticket; no source-code changes outside cli.py.
