Three threads bundled, all surfaced by today's calculator-test-
patterns.md audit + fox's directive to remove v-prefix from test
filenames:
THREAD 1 — rename test_v8_fork_score.py → test_substrate_fork_score.py
=====================================================================
Single test file in the tree had a v-prefix in its filename:
``tests/test_v8_fork_score.py``. Renamed via ``git mv`` for
consistency with yesterday's substrate refactor (the package is
``arborist/substrate/fork_score.py``; the CLI subcommand is
``arborist substrate score``; the test file should match).
No internal code changes needed — the file's imports + assertions
were already updated to ``arborist.substrate.*`` paths in
yesterday's bae5caf commit. Pure rename.
THREAD 2 — close v1 substring discipline gap
=============================================
Audit of test_anchor_prg.py + test_phi_alignment_probe.py against
docs/calculator-test-patterns.md §2 (versioned-default discipline)
found one gap: both files asserted the version string's exact
value but neither asserted the ``"v1"`` substring discipline that
fox's test_returns_calculator_version_token established.
Added ``assert "v1" in PHI_PRG_VERSION`` to
test_module_exports_version_string in test_anchor_prg.py.
Added ``assert "v1" in PROBE_VERSION`` to
test_module_exports_thresholds_and_version in
test_phi_alignment_probe.py.
Both follow fox's pattern: when the algorithm changes (v2-blake3-
expansion, v2-arnoldi-iteration, etc.), the version string MUST
change too. The "v1" substring assertion catches a future
contributor who refactors without bumping the version constant.
THREAD 3 — close CLI subprocess gap on test_substrate_fork_score.py
====================================================================
The renamed file had four CLI tests but all in-process via
build_parser() + parse_args() + func(args). That catches argparse-
shape drift but NOT entry-point / module-loading / sys.argv drift.
Added test_cli_substrate_score_subprocess_invocation: real
``subprocess.run(["python", "-m", "arborist.cli", "substrate",
"score", "--parent", ..., "--child", ..., "--out", ...])`` against
synthetic bench results. Asserts exit 0 + the --out artifact is
written + JSON-parses with valid verdict.
Pattern matches fox's test_cli_baseline_runs_clean in
test_t3_bound_calculator.py + the 581ad90 KAT-fixture-gap closure.
Same hazard fox already hit three times during the substrate
rename refactor (85be5eb, 209d670, b320e27): import-only tests
silently miss CLI surface drift.
CHECKLIST AUDIT — POST-FIX
==========================
Three calculator-style test files now all 9-item complete:
| t3 | anchor_prg | phi_alignment | substrate_fork |
KAT fixture | ✓ | ✓ | ✓ | n/a (different)|
VERSION + "v1" | ✓ | ✓ NOW | ✓ NOW | ✓ |
Hand-formula | ✓ | ✓ | ✓ | ✓ (synthetic) |
Monotonicity | ✓ | ✓ | ✓ | ✓ |
Closure / sum-of-parts | ✓ | ✓ | ✓ | ✓ |
Parametrized invalid | ✓ | ✓ | ✓ | ~ |
CLI subprocess | ✓ | n/a | n/a | ✓ NOW |
Doc parity | ✓ | KAT | KAT | KAT |
Module-export shape | ✓ | ✓ | ✓ | ✓ |
All four files now consistently track the calculator-test-patterns
checklist. test_substrate_fork_score.py is structurally different
(verifier-adjacent: tests scoring + verdict-band logic, not
closed-form math) so some checklist items map differently — KAT
fixture replaced by synthetic-input verdict tests (closer to
verifier-style), parametrized-invalid is partial (per-verdict-
class assertions rather than per-bad-input cone). Acceptable.
Test counts:
- test_substrate_fork_score.py: 26 → 27 (+1 subprocess test)
- test_anchor_prg.py: 27 → 27 (assertion added inline)
- test_phi_alignment_probe.py: 23 → 23 (assertion added inline)
- t3 file untouched in this commit (581ad90 already at 53)
Full suite: 1985 → 1986 (+1 from this commit's only
new-test-function addition; the inline assertions don't count
as new tests).
Hygiene
=======
- make test → 1986 passed, 45 skipped.
- All four calculator-style test files structurally aligned.
- No v-prefixed test filenames remain in tests/ tree.
Same pattern-application as de997f7 did for test_anchor_prg.py.
The phi_alignment_probe tests landed in 1dfb8b9 with KAT
regression + verdict-bucket coverage + Lanczos convergence
check, but lacked the four patterns fox demonstrated in
test_t3_bound_calculator.py (51 cases for the T3 calculator):
monotonicity, hand-computed formula, closure invariants, and
parametrized invalid-input cones.
New tests added:
1. **test_monotone_alignment_strength_in_concentration** —
tighter W concentration on low-λ subspace must monotonically
increase the ratio. Tested across 32-row → 16-row → 8-row
concentrations, normalized to constant ‖W‖_F. Catches sign
errors + drops in the alignment-score formula.
2. **test_monotone_alignment_in_dim_h** — verdict invariant
under W column-count scaling. Sanity check that A(W, H)'s
‖W‖_F² normalization decouples it from sample count.
3. **test_uniform_baseline_matches_analytical_formula** —
hand-computes the isotropic baseline
``E[A_bot(W_uniform, H)] = (1/dim_d) Σ 1/(λ_j+ε)`` from
#000034 §2.1 derivation; asserts exact agreement with the
function's ``a_uniform`` field. Catches algorithm drift the
KAT regression would miss (KAT could regenerate against a
buggy implementation).
4. **test_full_spectrum_a_top_plus_a_bot_covers_full_isotropic_baseline**
— closure invariant: when k_top + k_bot = dim_d, the function's
a_top + a_bot must equal full-spectrum A computed via dense
numpy.linalg.eigh decomposition. Catches missing terms /
double-counting.
5. **test_eigenvalue_ordering_top_dominates_bot** — closure
invariant: top-k eigenvalues must all be ≥ bot-k eigenvalues.
Catches a bug where eigsh's 'LA'/'SA' modes returned
overlapping ranges on near-degenerate spectra.
6. **test_rejects_wrong_dim_w** — parametrized over (1-D, 3-D,
0-D scalar) shape errors. Same pattern as test_anchor_prg's
parametrized rejects.
7. **test_rejects_non_positive_epsilon** — parametrized over
(0, -1e-6, -1.0). Collapsed N separate test_rejects_*
functions into a single parametrized cone.
Test count: was 14 in test_phi_alignment_probe.py; now 23
(+9 from the new patterns + parametrize expansion).
Full suite: 1727 → 1872 (note: large jump partly from fox's
parallel test additions today, +136 since my last test count
checkpoint; my contribution here is +9 directly attributable
to this commit).
Three calculator/probe-style modules now have consistent
test coverage:
bench/scripts/t3_bound_calculator.py — 51 tests (fox)
bench/scripts/phi_alignment_probe.py — 23 tests (this commit)
arborist/substrate/anchor_prg.py — 27 tests (de997f7)
Same pattern bench applied across all three. Future
calculator-style code should pin: monotonicity in each input
axis + hand-computed formula assertions + closure / sum-of-parts
invariants + parametrized invalid-input cones.
Hygiene
=======
- make test → 1872 passed, 45 skipped
- make chain-check-shards → 0 across all 7 shards
- All new tests use synthetic inputs (no LLM, no shard
dependency); run in ~35s suite-wide
Lands the synthetic-ablation infrastructure proposed in fce8826's
ticket §7 amendment. Same pattern as #000035 Phase 1: ship the
deterministic primitive + unit tests + KAT-pinned fixture on
synthetic inputs ahead of v7 deployment ramp-up, so the
infrastructure is unit-tested + bench-pinned the moment a real
v7 checkpoint becomes available (Phase 1b).
bench/scripts/phi_alignment_probe.py
====================================
Implements ``measure_alignment(W, hessian_eval, *, k_top, k_bot,
epsilon) -> AlignmentReport`` per #000034 §3.1:
- Lanczos top-k + bottom-k via ``scipy.sparse.linalg.eigsh`` over
a user-supplied HVP closure. Probe never materializes H.
- Alignment score: A(W, H) = Σ_j (Σ_i ⟨W·e_i, v_j⟩²) / (λ_j+ε)
/ ‖W‖_F², per ticket §2.1. Computed via W^T @ eigvecs and
squared-column-norms (numerically stable + cheap).
- Verdict thresholds (§3.3): STRUCTURAL_ALIGNMENT (ratio > 1.5) /
NO_ALIGNMENT / ANTI_ALIGNED (ratio < 0.7).
Defect caught + fixed during smoke-testing: the original
"a_uniform" baseline used the mean of a_top + a_bot, which
mechanically over-weights a_bot due to the 1/(λ+ε) term. Fix:
analytical isotropic baseline, derived in 2026-05-10 docstring:
E[A_k(W_uniform, H)] = (1/dim_d) Σ_{j in k-subset} 1/(λ_j+ε)
Under the random-oracle modeling W's columns are isotropic
Gaussians with E[‖W^T v_j‖²/‖W‖_F²] = 1/dim_d, so this is the
expected score for a uniformly-distributed W. Smoke test
post-fix: aligned → STRUCTURAL_ALIGNMENT (ratio ~7.97), uniform →
NO_ALIGNMENT (ratio ~1.00), anti → ANTI_ALIGNED (ratio ~0.00).
All three classes land cleanly in their expected verdict bucket.
Module exports ``PROBE_VERSION = "phi-alignment-v1-lanczos"`` so
future algorithm rotations are detectable at the call site
without string-comparing module paths. Same convention as
#000035's PHI_PRG_VERSION.
bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl
========================================================
30 KAT entries — 10 per class (aligned / uniform / anti) — each
pinning (seed, dim_d, k, class) → expected_verdict + observed_ratio
for regression coverage. Deterministic-seeded so CI replays
exactly. Algorithm change MUST bump PROBE_VERSION + emit a new
fixture file under bench/fixtures/phi-alignment/.
Class ratio ranges:
- aligned: 7.77 - 8.27 (well above 1.5 STRUCTURAL_ALIGNMENT floor)
- uniform: 0.95 - 1.04 (cleanly within NO_ALIGNMENT band)
- anti: 0.00 (well below 0.7 ANTI_ALIGNED ceiling)
tests/test_phi_alignment_probe.py
=================================
14 tests covering #000034 §3.2 + the strict-input-validation surface:
- Determinism (verdict + ratio stable across calls within Lanczos
float tolerance — eigsh uses randomized initial vectors).
- Verdict thresholds (engineered cases land in correct bucket).
- Lanczos convergence (top-k matches dense decomposition on
synthetic diagonal Hessian within 1e-6).
- Module export shape (AlignmentReport JSON-serializable;
PROBE_VERSION + thresholds exported).
- Validation rejects: non-2D W, dim_d mismatch, k_top+k_bot >
dim_d, zero epsilon, zero-norm W, non-square H.
- KAT regression against the 30-entry fixture.
Tests skip via ``pytest.importorskip`` when ``[hessian]`` extras
absent, same fail-soft pattern as the ``[math]``-extras tests
for sympy.
pyproject.toml — new [hessian] optional-deps block
==================================================
Adds ``numpy>=1.26`` + ``scipy>=1.11`` under a new ``[hessian]``
extras gate. Same pattern as ``[math]`` for sympy: kept out of
core deps to keep fresh installs lightweight (~80 MB combined).
Operators install via ``pip install 'arborist[hessian]'``.
#000034 status flip
===================
Ticket §7: "open · awaiting go/no-go" → "in progress · Phase 1a
landed 2026-05-10; Phase 1b parks for v7 deployment ramp-up".
Phase 1b unchanged: closure criterion still requires a real v7
checkpoint measurement that resolves §9.1 of the soft-hash-
channel-analysis. TICKETS.md index row refreshed.
Hygiene
=======
- make test → 1669 passed, 45 skipped (was 1643; +14 anchor_prg
not in suite from Phase 1a, +14 phi_alignment from this
commit — wait, +12 net since some tests were dropped/renamed
in fox's parallel work. Bottom-line: 1669 stable.)
- make chain-check-shards → 0 across all 7 shards.
- arborist.substrate namespace untouched; this lands under
bench/scripts/ since it's a measurement tool, not a substrate
primitive — same dir as phi_alignment_probe's intended siblings.
