arborist/docs/calculator-test-patterns.md

# Calculator / probe test patterns

Architecture reference for testing **calculator-style** Python
modules: pure functions over numeric / cryptographic inputs that
emit JSON-serializable structured output and (often) a CLI surface.
The patterns codified here surfaced from the 2026-05-10 pattern
bench across three modules:

- ``arborist/substrate/anchor_prg.py`` — HMAC-SHA-512 PRG for #000035
- ``bench/scripts/phi_alignment_probe.py`` — Lanczos alignment probe
  for #000034 Phase 1a
- ``bench/scripts/t3_bound_calculator.py`` — T3 channel-capacity
  closed-form bound for #000036

The exemplar test file is ``tests/test_t3_bound_calculator.py``
(<!--AUTOCOUNT:tests:tests/test_t3_bound_calculator.py-->83<!--/AUTOCOUNT--> cases as of 2026-05-10; fox shipped 51 in the
initial cut and the +2 KAT-fixture-gap closure landed in
``581ad90``). The other two test files
(``tests/test_anchor_prg.py`` and
``tests/test_phi_alignment_probe.py``) were retroactively
backfilled with the same patterns in commits
``de997f7`` and ``a4b3056``.

---

## When to use these patterns

**Apply to**: pure functions over numeric input, math-heavy code,
modules that expose a structured-output dataclass / JSON-emitting
CLI, anything with closed-form formulas the test can hand-compute.

**Don't apply to**: parser / state-machine code (e.g.
``arborist/qa/warrant_resolver.py``), verifier code (e.g.
``arborist/qa/warrant_chain.py``), retrieval code, anything where
inputs and outputs lack a hand-derivable mathematical relationship.
The verifier-style tests in ``tests/test_warrant_resolver.py``
(<!--AUTOCOUNT:tests:tests/test_warrant_resolver.py-->23<!--/AUTOCOUNT--> tests) and ``tests/test_warrant_chain.py`` (<!--AUTOCOUNT:tests:tests/test_warrant_chain.py-->9<!--/AUTOCOUNT--> tests) are
appropriately structured for their domain — descriptive
function-name-per-input-shape works better than parametrized
tables for parsers, and there's no formula to hand-check.

---

## The patterns

### 1. KAT (known-answer-test) regression

Pin a small fixture of (input, expected_output) tuples into
``bench/fixtures/<module>/known-answer-tests.jsonl``. Tests replay
every fixture row against the live module; mismatches fail. Catches
unintentional algorithm changes between releases.

**Examples**:
- ``bench/fixtures/phi-prg/known-answer-tests.jsonl`` (10 vectors)
- ``bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl`` (30 vectors)

**Discipline**: bump the module's ``VERSION`` constant + emit a new
fixture file when the algorithm changes. Old runs replay against
old data; new runs replay against new data.

**Caveat**: KAT regression alone can't catch bugs in the original
implementation that were baked into the fixture. Always pair with
hand-computed formula tests below.

### 2. Hand-computed formula assertion

For closed-form math, compute the expected output from first
principles inside the test file using stdlib (``math``, ``hashlib``)
or numpy. Assert agreement with the function output.

**Why**: catches algorithm drift that KAT regression misses (KAT
could have been generated against a buggy version).

**Example** — fox's ``test_b1_exact_formula`` in
``test_t3_bound_calculator.py``:

```python
def test_b1_exact_formula():
    g, gnm, sigma, W = 0.1, 2.0, 0.5, 5000
    snr = g * gnm / sigma  # 0.4
    expected = 1.0 * g * W * math.log2(snr + 1)
    r = t3_bound_bits(**{**_BASELINE, "gradient_fraction": g, ...})
    assert r["B1_contribution"] == pytest.approx(expected, abs=1e-3)
```

The test computes B1 by hand using the §3 formula from the source
spec, then asserts the function returns the same value. Catches a
sign error or constant drop that no KAT regression would.

**Example** — ``test_phi_prg_first_block_matches_direct_hmac`` in
``test_anchor_prg.py``: directly invokes
``hmac.new(seed, h + counter_bytes, sha512).digest()`` and asserts
the function's first 64 bytes match.

**Example** — ``test_uniform_baseline_matches_analytical_formula``
in ``test_phi_alignment_probe.py``: hand-computes
``(1/dim_d) Σ 1/(λ_j+ε)`` from the §2.1 derivation and asserts
agreement with the function's ``a_uniform`` field.

### 3. Monotonicity in each input axis

For each input the function takes, assert that scaling it
monotonically moves the output in the expected direction.
Holding others fixed.

**Why**: catches sign errors, dropped terms, missed dependencies.

**Examples**:
- T3 calculator: doubling ``window_length`` doubles ``B1`` exactly
  (linear); raising ``gradient_fraction`` raises ``I_window``
  while leaving ``B2`` and ``B3`` untouched.
- phi_prg: increasing ``dim_h`` from N to N+k yields output whose
  first N entries are byte-identical to the dim_h=N output
  (prefix-extension / streaming-counter invariant).
- phi_alignment_probe: tighter W concentration on low-λ subspace
  produces strictly larger ratio.

**Pattern code**:

```python
def test_monotone_in_X():
    base = func(**baseline)
    higher_x = func(**{**baseline, "X": baseline["X"] * 2})
    assert higher_x["output"] > base["output"]
    # Bonus: exact relationship if the formula is linear in X
    assert higher_x["output"] == pytest.approx(2 * base["output"], rel=1e-6)
```

### 4. Closure / sum-of-parts invariants

If the function returns a structured output with multiple
contributions, assert the contributions sum / compose to the
total. No missing terms, no double-counting.

**Examples**:
- T3: ``I_window ≡ B1 + B2 + B3`` (test_total_equals_sum_of_three_contributions)
- phi_alignment_probe: when ``k_top + k_bot = dim_d`` (full
  spectrum), ``a_top + a_bot`` must equal full-spectrum A from
  dense ``numpy.linalg.eigh`` decomposition
  (test_full_spectrum_a_top_plus_a_bot_covers_full_isotropic_baseline)
- phi_alignment_probe: top-k eigenvalue min ≥ bot-k eigenvalue
  max (no overlap; test_eigenvalue_ordering_top_dominates_bot)

**Why**: these are mechanical sanity invariants. A pure
implementation passes them automatically; a buggy implementation
that drops a term, double-counts, or has overlapping ranges fails
loudly.

### 5. Parametrized invalid-input cones

Use ``@pytest.mark.parametrize`` to spawn one test function per
invalid-input class.

**Why**: cleaner than N separate ``test_rejects_X`` /
``test_rejects_Y`` functions for the same validation surface.

**Pattern**:

```python
@pytest.mark.parametrize("g", [0.0, -0.1, 1.5, 2.0])
def test_invalid_gradient_fraction_rejected(g):
    args = {**_BASELINE, "gradient_fraction": g}
    with pytest.raises(ValueError, match="gradient_fraction"):
        func(**args)
```

**Don't over-parametrize**: if each case has a unique error
message that the test asserts specifically, separate functions
remain readable. The cone form works best when the assertion
shape is identical across cases.

### 6. CLI subprocess end-to-end

For modules that expose a CLI surface, test ``argparse + main()``
via ``subprocess.run``. Parse the JSON output; check exit codes
on validation errors.

**Why**: import-only tests miss argparse + main() drift. Refactors
that break the CLI surface (rename flag, change exit code) won't
surface until someone runs the command. fox's
``test_cli_baseline_runs_clean`` and ``test_cli_invalid_input_exits_2``
catch exactly this.

**Example**:

```python
def test_cli_baseline_runs_clean(tmp_path):
    cmd = [
        sys.executable, "-m", "bench.scripts.t3_bound_calculator",
        "--gradient-fraction", "0.05", ...
    ]
    out = subprocess.run(cmd, capture_output=True, text=True,
                         check=True, cwd="/home/fox/git/arborist")
    j = json.loads(out.stdout)
    assert j["calculator_version"] == CALCULATOR_VERSION
    assert j["I_window_bits_upper_bound"] == pytest.approx(625.87, abs=0.05)
```

**Caveat**: the ``cwd`` argument matters — invoking
``python -m bench.scripts.<X>`` requires the project root as cwd
so the package import resolves.

### 7. Doc parity (and what fox's pattern caught)

Test that the function's actual output matches the numbers in the
spec doc's worked-example sections. Catches **doc drift** —
when a doc was hand-written before the implementation crystallized
and approximate rounding got baked into prose.

**Example finding (2026-05-10)**: fox's
``test_baseline_matches_section_11_doc`` flagged that
``soft-hash-channel-t3-bound.md §11`` said 622.7 / 290.0 / 32.7
bits but the calculator produced 625.87 / 292.48 / 33.39. The
test's docstring (lines 56-61) explicitly tracks the
calibration-pass follow-up. Refresh landed in commit ``de997f7``.

**Why**: docs go stale faster than code. A test that pins the
spec-quoted numbers fails noisily when the doc drifts; the prose
in the doc gets refreshed (or the function gets fixed, depending
on which side has the bug).

### 8. Module-export shape

Test the public exports: version constants, threshold constants,
dataclass round-trip via ``asdict``, JSON-serializability.

**Example** — fox's ``test_returns_calculator_version_token``:

```python
def test_returns_calculator_version_token():
    r = func(**_BASELINE)
    assert r["calculator_version"] == CALCULATOR_VERSION
    assert "v1" in CALCULATOR_VERSION
```

The "v1" assertion encodes the discipline: when the algorithm
changes, the version string MUST change too. A future contributor
who refactors the algorithm without bumping ``CALCULATOR_VERSION``
gets caught here.

**Example** — ``test_alignment_report_round_trips_via_asdict``:
asserts ``asdict(report)`` is JSON-serializable. Catches
non-stdlib types leaking into the dataclass that would break
downstream consumers.

### 9. Projective-contract pin (one-way canonicalizers)

For canonicalizers whose output type ≠ input type — output is
**not** in the input domain by design — assert that re-applying
the function raises. Catches a future "make it round-trippable"
PR that silently breaks the projection.

**Why**: the contract is "one-way projection," not "fixed-point
normalization." If the output were re-canonicalizable, downstream
code might come to depend on idempotence; the test pins the
boundary.

**Examples** —

- ``arborist/pi_star/code_py_ast.py`` (commit ``593550b``):
  Python source → S-expression. Re-canonicalizing the
  S-expression text raises ``PiStarError`` because it isn't
  valid Python source.
- ``arborist/pi_star/time_series_quantized.py`` (commit
  ``2585d3c``): JSON ``{dt, dv, samples}`` → quantized text
  ``dt=...;dv=...;n=...;t0=...:v0|v1|...``. The quantized text
  isn't JSON; re-canonicalizing raises.

**Pattern code**::

    def test_output_is_projective_not_invertible():
        canonical = canonicalize(input_value)
        with pytest.raises(PiStarError):
            canonicalize(canonical)

When the output type **does** equal the input type (e.g. the
arithmetic kernel's ℚ → ℚ rationals), use the dual pattern —
**round-trip idempotence**: ``canonicalize(canonicalize(x)) ==
canonicalize(x)``. Both patterns pin a contract; pick whichever
matches the kernel's type signature.

### 10. Dispatch-order pin (Python type-hierarchy gotchas)

For functions branching on ``isinstance`` over numeric types,
pin the branch order. ``bool`` subclasses ``int`` in Python, so
this naive chain never reaches the bool branch::

    if isinstance(x, int):    # True is also an int
        ...
    elif isinstance(x, bool):  # unreachable
        ...

A kernel that distinguishes ``True`` from ``1`` (e.g. a Python
AST normalizer) must check ``bool`` first. Pin the branch order
in a test so a future "simplify the dispatch" PR fires loud.

**Why**: subtle Python type-hierarchy gotcha that no other test
shape catches. Determinism + KAT tests pass either way at the
boundary case ``True``; only an explicit dispatch-order pin
exposes the regression.

**Example** — ``arborist/pi_star/code_py_ast.py`` (commit
``593550b``)::

    def test_bool_dispatched_before_int():
        assert canonicalize("True") != canonicalize("1")
        # Both are int-ish; bool-first dispatch keeps them distinct.

**How to apply**: any kernel that branches on ``isinstance`` for
numeric types should have a paired test pinning the branch order.
Same applies to ``Decimal`` vs ``float``, ``bytes`` vs ``bytearray``,
``int`` vs ``numpy.int64`` — anywhere Python's MRO crosses a
distinguishing boundary.

### 11. Tie-breaking-rule pin (ties-to-even / banker's rounding)

For functions using ``round()``, pin the tie-breaking direction.
Python's ``round()`` uses **banker's rounding** by default
(ties-to-even, PEP 3141): 0.5→0, 1.5→2, 2.5→2. A naive switch to
``math.floor(x + 0.5)`` ("round half up": 0.5→1, 1.5→2, 2.5→3)
silently changes output for tie inputs without breaking
non-tie tests.

**Why**: many programmers expect round-half-up; a contributor
unfamiliar with PEP 3141 will refactor to ``floor(x + 0.5)``
"to be explicit" and silently corrupt quantization. Pin the rule
with the actual tie inputs.

**Example** — ``arborist/pi_star/time_series_quantized.py``
(commit ``2585d3c``)::

    @pytest.mark.parametrize("v_in,v_out", [
        (0.5, 0),   # ties-to-even: 0 is even
        (1.5, 2),   # ties-to-even: 2 is even
        (2.5, 2),   # ties-to-even: 2 is even (NOT 3)
        (3.5, 4),   # ties-to-even: 4 is even
    ])
    def test_quantization_uses_bankers_rounding(v_in, v_out):
        assert quantize_value(v_in, dv=1) == v_out

**How to apply**: any kernel using ``round()`` or any quantization
that sits on a discrete grid should have a parametrized test
covering at least 4 ties (mix of round-up-to-even and
round-down-to-even cases).

---

## Checklist for new calculator-style code

When opening a new ``bench/scripts/<X>.py`` or substrate primitive:

1. [ ] KAT fixture under ``bench/fixtures/<module>/`` (≥ 5 cases)
2. [ ] Module-level ``VERSION`` constant; tests assert its
   presence + the ``v1`` discipline
3. [ ] Hand-computed formula tests — at least one per
   independent contribution / output field
4. [ ] Monotonicity test per input axis (or invariance assertion
   if the function is meant to be insensitive to that axis)
5. [ ] Closure / sum-of-parts invariant if the output decomposes
6. [ ] Parametrized invalid-input cone (one ``@parametrize`` per
   validation class — wrong dim, out-of-range value, wrong type)
7. [ ] CLI subprocess test if the module exposes a CLI
8. [ ] Doc-parity test if the spec doc has worked-example numbers
9. [ ] Module-export shape test (asdict round-trip,
   JSON-serializability, version token presence)
10. [ ] Projective-contract pin if output type ≠ input type
   (re-canonicalizing the canonical output raises) — OR
   round-trip idempotence pin if output type = input type
11. [ ] Dispatch-order pin if the kernel branches on
   ``isinstance`` for numeric types (bool-before-int, etc)
12. [ ] Tie-breaking-rule pin if the kernel uses ``round()`` or
   sits on a discrete quantization grid

Items 10/11/12 are conditional — only apply when the kernel's
shape exposes the corresponding surface. Many calculator modules
(e.g. T3 bound, anchor PRG) need only items 1-9.

Exemplar test files:

- ``tests/test_t3_bound_calculator.py`` —
  <!--AUTOCOUNT:tests:tests/test_t3_bound_calculator.py-->83<!--/AUTOCOUNT--> tests covering
  items 1-9 for the T3 closed-form bound.
- ``tests/test_pi_star_arithmetic.py`` —
  <!--AUTOCOUNT:tests:tests/test_pi_star_arithmetic.py-->56<!--/AUTOCOUNT--> tests covering items
  1-6 + 9 + round-trip idempotence (output type = input type =
  rationals). From ``6c9bc04`` 2026-05-10.
- ``tests/test_pi_star_logic.py`` —
  <!--AUTOCOUNT:tests:tests/test_pi_star_logic.py-->53<!--/AUTOCOUNT--> tests covering items
  1-6 + 9 + 11 (tie-breaking on multi-tautology equivalence-class
  collapse). From ``e7bef5f`` 2026-05-10.
- ``tests/test_pi_star_code.py`` —
  <!--AUTOCOUNT:tests:tests/test_pi_star_code.py-->32<!--/AUTOCOUNT--> tests covering items
  1-6 + 9 + 10 (projective: Python → S-expr) + 11 (dispatch:
  bool-before-int). From ``593550b`` 2026-05-10.
- ``tests/test_pi_star_time_series.py`` —
  <!--AUTOCOUNT:tests:tests/test_pi_star_time_series.py-->35<!--/AUTOCOUNT--> tests covering
  items 1-6 + 9 + 10 (projective: JSON → quantized text) + 12
  (banker's rounding pin). From ``2585d3c`` 2026-05-10.

New modules should aim for similar coverage density on their
own surface.

---

## What this doc is NOT

- A general testing tutorial. Pytest mechanics + fixture scoping
  + ``tmp_path`` use are documented in pytest's own docs.
- A code-style guide. Ruff / black / type-hints discipline lives
  in CLAUDE.md.
- A list of every pattern in the repo. Verifier tests, parser
  tests, retrieval tests have their own structures appropriate
  to their domains; this doc is calculator-style only.
- A coverage inventory. The per-discipline test-file index lives
  at `docs/warrant-substrate-cookbook.md` § "Appendix —
  test-coverage cross-reference". This doc is the **checklist
  for new tests**; the cookbook appendix is the **index of
  existing tests** keyed by substrate discipline. Walk both:
  this doc designs the test file, the cookbook appendix records
  it under the matching discipline.

The "When to use" caveat at the top is the load-bearing line:
**don't retrofit these patterns onto verifier-style tests**.
Calculator code has math; verifier code has state machines.
Different shapes need different test patterns.