arborist/docs/calculator-test-patterns.md

Calculator / probe test patterns

Architecture reference for testing calculator-style Python modules: pure functions over numeric / cryptographic inputs that emit JSON-serializable structured output and (often) a CLI surface. The patterns codified here surfaced from the 2026-05-10 pattern bench across three modules:

• arborist/substrate/anchor_prg.py — HMAC-SHA-512 PRG for #000035
• bench/scripts/phi_alignment_probe.py — Lanczos alignment probe for #000034 Phase 1a
• bench/scripts/t3_bound_calculator.py — T3 channel-capacity closed-form bound for #000036

The exemplar test file is tests/test_t3_bound_calculator.py (53 cases as of 2026-05-10; fox shipped 51 in the initial cut and the +2 KAT-fixture-gap closure landed in 581ad90). The other two test files (tests/test_anchor_prg.py and tests/test_phi_alignment_probe.py) were retroactively backfilled with the same patterns in commits de997f7 and a4b3056.

---

When to use these patterns

Apply to: pure functions over numeric input, math-heavy code, modules that expose a structured-output dataclass / JSON-emitting CLI, anything with closed-form formulas the test can hand-compute.

Don't apply to: parser / state-machine code (e.g. arborist/qa/warrant_resolver.py), verifier code (e.g. arborist/qa/warrant_chain.py), retrieval code, anything where inputs and outputs lack a hand-derivable mathematical relationship. The verifier-style tests in tests/test_warrant_resolver.py (23 tests) and tests/test_warrant_chain.py (9 tests) are appropriately structured for their domain — descriptive function-name-per-input-shape works better than parametrized tables for parsers, and there's no formula to hand-check.

---

The patterns

1. KAT (known-answer-test) regression

Pin a small fixture of (input, expected_output) tuples into bench/fixtures/<module>/known-answer-tests.jsonl. Tests replay every fixture row against the live module; mismatches fail. Catches unintentional algorithm changes between releases.

Examples:

• bench/fixtures/phi-prg/known-answer-tests.jsonl (10 vectors)
• bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl (30 vectors)

Discipline: bump the module's VERSION constant + emit a new fixture file when the algorithm changes. Old runs replay against old data; new runs replay against new data.

Caveat: KAT regression alone can't catch bugs in the original implementation that were baked into the fixture. Always pair with hand-computed formula tests below.

2. Hand-computed formula assertion

For closed-form math, compute the expected output from first principles inside the test file using stdlib (math, hashlib) or numpy. Assert agreement with the function output.

Why: catches algorithm drift that KAT regression misses (KAT could have been generated against a buggy version).

Example — fox's test_b1_exact_formula in test_t3_bound_calculator.py:

def test_b1_exact_formula():
    g, gnm, sigma, W = 0.1, 2.0, 0.5, 5000
    snr = g * gnm / sigma  # 0.4
    expected = 1.0 * g * W * math.log2(snr + 1)
    r = t3_bound_bits(**{**_BASELINE, "gradient_fraction": g, ...})
    assert r["B1_contribution"] == pytest.approx(expected, abs=1e-3)

The test computes B1 by hand using the §3 formula from the source spec, then asserts the function returns the same value. Catches a sign error or constant drop that no KAT regression would.

Example — test_phi_prg_first_block_matches_direct_hmac in test_anchor_prg.py: directly invokes hmac.new(seed, h + counter_bytes, sha512).digest() and asserts the function's first 64 bytes match.

Example — test_uniform_baseline_matches_analytical_formula in test_phi_alignment_probe.py: hand-computes (1/dim_d) Σ 1/(λ_j+ε) from the §2.1 derivation and asserts agreement with the function's a_uniform field.

3. Monotonicity in each input axis

For each input the function takes, assert that scaling it monotonically moves the output in the expected direction. Holding others fixed.

Why: catches sign errors, dropped terms, missed dependencies.

Examples:

• T3 calculator: doubling window_length doubles B1 exactly (linear); raising gradient_fraction raises I_window while leaving B2 and B3 untouched.
• phi_prg: increasing dim_h from N to N+k yields output whose first N entries are byte-identical to the dim_h=N output (prefix-extension / streaming-counter invariant).
• phi_alignment_probe: tighter W concentration on low-λ subspace produces strictly larger ratio.

Pattern code:

def test_monotone_in_X():
    base = func(**baseline)
    higher_x = func(**{**baseline, "X": baseline["X"] * 2})
    assert higher_x["output"] > base["output"]
    # Bonus: exact relationship if the formula is linear in X
    assert higher_x["output"] == pytest.approx(2 * base["output"], rel=1e-6)

4. Closure / sum-of-parts invariants

If the function returns a structured output with multiple contributions, assert the contributions sum / compose to the total. No missing terms, no double-counting.

Examples:

• T3: I_window ≡ B1 + B2 + B3 (test_total_equals_sum_of_three_contributions)
• phi_alignment_probe: when k_top + k_bot = dim_d (full spectrum), a_top + a_bot must equal full-spectrum A from dense numpy.linalg.eigh decomposition (test_full_spectrum_a_top_plus_a_bot_covers_full_isotropic_baseline)
• phi_alignment_probe: top-k eigenvalue min ≥ bot-k eigenvalue max (no overlap; test_eigenvalue_ordering_top_dominates_bot)

Why: these are mechanical sanity invariants. A pure implementation passes them automatically; a buggy implementation that drops a term, double-counts, or has overlapping ranges fails loudly.

5. Parametrized invalid-input cones

Use @pytest.mark.parametrize to spawn one test function per invalid-input class.

Why: cleaner than N separate test_rejects_X / test_rejects_Y functions for the same validation surface.

Pattern:

@pytest.mark.parametrize("g", [0.0, -0.1, 1.5, 2.0])
def test_invalid_gradient_fraction_rejected(g):
    args = {**_BASELINE, "gradient_fraction": g}
    with pytest.raises(ValueError, match="gradient_fraction"):
        func(**args)

Don't over-parametrize: if each case has a unique error message that the test asserts specifically, separate functions remain readable. The cone form works best when the assertion shape is identical across cases.

6. CLI subprocess end-to-end

For modules that expose a CLI surface, test argparse + main() via subprocess.run. Parse the JSON output; check exit codes on validation errors.

Why: import-only tests miss argparse + main() drift. Refactors that break the CLI surface (rename flag, change exit code) won't surface until someone runs the command. fox's test_cli_baseline_runs_clean and test_cli_invalid_input_exits_2 catch exactly this.

Example:

def test_cli_baseline_runs_clean(tmp_path):
    cmd = [
        sys.executable, "-m", "bench.scripts.t3_bound_calculator",
        "--gradient-fraction", "0.05", ...
    ]
    out = subprocess.run(cmd, capture_output=True, text=True,
                         check=True, cwd="/home/fox/git/arborist")
    j = json.loads(out.stdout)
    assert j["calculator_version"] == CALCULATOR_VERSION
    assert j["I_window_bits_upper_bound"] == pytest.approx(625.87, abs=0.05)

Caveat: the cwd argument matters — invoking python -m bench.scripts.<X> requires the project root as cwd so the package import resolves.

7. Doc parity (and what fox's pattern caught)

Test that the function's actual output matches the numbers in the spec doc's worked-example sections. Catches doc drift — when a doc was hand-written before the implementation crystallized and approximate rounding got baked into prose.

Example finding (2026-05-10): fox's test_baseline_matches_section_11_doc flagged that soft-hash-channel-t3-bound.md §11 said 622.7 / 290.0 / 32.7 bits but the calculator produced 625.87 / 292.48 / 33.39. The test's docstring (lines 56-61) explicitly tracks the calibration-pass follow-up. Refresh landed in commit de997f7.

Why: docs go stale faster than code. A test that pins the spec-quoted numbers fails noisily when the doc drifts; the prose in the doc gets refreshed (or the function gets fixed, depending on which side has the bug).

8. Module-export shape

Test the public exports: version constants, threshold constants, dataclass round-trip via asdict, JSON-serializability.

Example — fox's test_returns_calculator_version_token:

def test_returns_calculator_version_token():
    r = func(**_BASELINE)
    assert r["calculator_version"] == CALCULATOR_VERSION
    assert "v1" in CALCULATOR_VERSION

The "v1" assertion encodes the discipline: when the algorithm changes, the version string MUST change too. A future contributor who refactors the algorithm without bumping CALCULATOR_VERSION gets caught here.

Example — test_alignment_report_round_trips_via_asdict: asserts asdict(report) is JSON-serializable. Catches non-stdlib types leaking into the dataclass that would break downstream consumers.

---

Checklist for new calculator-style code

When opening a new bench/scripts/<X>.py or substrate primitive:

1. KAT fixture under bench/fixtures/<module>/ (≥ 5 cases)
2. Module-level VERSION constant; tests assert its presence + the v1 discipline
3. Hand-computed formula tests — at least one per independent contribution / output field
4. Monotonicity test per input axis (or invariance assertion if the function is meant to be insensitive to that axis)
5. Closure / sum-of-parts invariant if the output decomposes
6. Parametrized invalid-input cone (one @parametrize per validation class — wrong dim, out-of-range value, wrong type)
7. CLI subprocess test if the module exposes a CLI
8. Doc-parity test if the spec doc has worked-example numbers
9. Module-export shape test (asdict round-trip, JSON-serializability, version token presence)

The 53 tests fox shipped for t3_bound_calculator are the exemplar; new modules should aim for similar coverage density on their own surface.

---

What this doc is NOT

• A general testing tutorial. Pytest mechanics + fixture scoping
  • tmp_path use are documented in pytest's own docs.
• A code-style guide. Ruff / black / type-hints discipline lives in CLAUDE.md.
• A list of every pattern in the repo. Verifier tests, parser tests, retrieval tests have their own structures appropriate to their domains; this doc is calculator-style only.
• A coverage inventory. The per-discipline test-file index lives at docs/warrant-substrate-cookbook.md § "Appendix — test-coverage cross-reference". This doc is the checklist for new tests; the cookbook appendix is the index of existing tests keyed by substrate discipline. Walk both: this doc designs the test file, the cookbook appendix records it under the matching discipline.

The "When to use" caveat at the top is the load-bearing line: don't retrofit these patterns onto verifier-style tests. Calculator code has math; verifier code has state machines. Different shapes need different test patterns.