tests/t3_bound_calculator: close the two checklist gaps from §3 audit
Audited fox's exemplar 51-test file against the 9-item checklist
in docs/calculator-test-patterns.md (codified earlier today in
0725eb4). Two gaps found, both additive:
GAP 1: no KAT fixture
=====================
Checklist item 1: "KAT fixture under bench/fixtures/<module>/
(≥ 5 cases)". Fox's tests cover the doc's worked-example numbers
inline (test_baseline_matches_section_11_doc) but no separate
fixture file existed for off-the-baseline regression coverage.
Generated bench/fixtures/t3-bound/known-answer-tests.jsonl with
8 KATs:
- small-deployment-§7.1, medium-deployment-§7.2,
hardened-deployment-§7.3 (the doc's three worked examples)
- extreme-low-g (g=0.001, σ=0.5 — exercises low-SNR regime)
- tight-window-W=100 (small-W ceiling-rounding edge case)
- tightened-c-b1 (override-constants path)
- all-constants-tight (all three c_b1/c_b2/c_b3 overridden)
- b3-floor-regime (factor < 1, B3 floors to 0)
Each entry pins (calculator_version, inputs, expected_total,
expected_b1, expected_b2, expected_b3, expected_snr_grad).
Algorithm change MUST bump CALCULATOR_VERSION + emit new fixture
file under bench/fixtures/t3-bound/ — old runs replay against
old data per §1 discipline.
GAP 2: no B3 hand-computed formula test
=======================================
Checklist item 3: "Hand-computed formula tests — at least one
per independent contribution / output field". Fox had
test_b1_exact_formula + test_b2_exact_formula + test_snr_grad_formula
covering three of the five output fields. B3 had only
test_b3_floor_at_zero_when_factor_below_one (an edge case),
not a closed-form check on the general formula.
Added test_b3_exact_formula: hand-computes
``C_B3 · ⌈W/E⌉ · log₂(N_b · σ_grad / ‖∇L_max‖) / 2`` per #000036
§5 (Bottou-Bousquet refinement), asserts agreement with the
function's B3_contribution. Pairs cleanly with the B1/B2 hand-
formula tests fox had.
CHECKLIST AUDIT — POST-FIX
==========================
1. KAT fixture ✓ NOW (was ❌; 8 entries)
2. VERSION + "v1" ✓ test_returns_calculator_version_token
3. Hand-formula ✓ NOW B1/B2/B3/snr_grad all covered
(was ⚠️ partial; B3 had floor-only)
4. Monotonicity ✓ test_monotone_in_window_length /
gradient_fraction
5. Closure ✓ test_total_equals_sum_of_three_contributions
6. Parametrized invalid ✓ four @pytest.mark.parametrize blocks
7. CLI subprocess ✓ test_cli_baseline_runs_clean +
test_cli_invalid_input_exits_2
8. Doc parity ✓ test_baseline_matches_section_11_doc
(caught today's §11 calibration drift)
9. Module-export shape ✓ test_returns_calculator_version_token +
test_constants_recorded
All nine items now ✓. test_t3_bound_calculator.py is the
exemplar for calculator-style test discipline.
Test count: 51 → 53 (+2 from this commit). Full suite:
1915 → 1985 (+70 from fox's parallel work + this commit's +2;
partial cycle effects).
Hygiene
=======
- make test → 1985 passed, 45 skipped.
- KAT fixture is JSONL with header comment naming
CALCULATOR_VERSION; future drift caught at the test level.
- Eat-my-own-dogfood: applied my docs/calculator-test-patterns.md
checklist to fox's exemplar test file. The fact that gaps
surfaced (even on fox's substantive 51-test surface) validates
that the checklist has real reviewer value, not just guideline
signaling.
This commit is contained in:
parent
d720b73d91
commit
581ad908f0
2 changed files with 107 additions and 0 deletions
12
bench/fixtures/t3-bound/known-answer-tests.jsonl
Normal file
12
bench/fixtures/t3-bound/known-answer-tests.jsonl
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
# T3 bound calculator known-answer tests — version t3-bound-v1-bottou-refinement
|
||||
# Generated 2026-05-10 from docs/soft-hash-channel-t3-bound.md §7
|
||||
# worked examples + override-constant variants + edge regimes.
|
||||
# Algorithm change MUST bump CALCULATOR_VERSION + emit new fixture.
|
||||
{"label": "small-deployment-\u00a77.1", "calculator_version": "t3-bound-v1-bottou-refinement", "inputs": {"gradient_fraction": 0.05, "gradient_norm_max": 1.0, "gradient_noise_stddev": 0.1, "lr_decision_interval": 100, "lr_grid_size": 8, "window_length": 10000, "batches_per_epoch": 1024, "steps_per_epoch": 1024}, "expected_total": 625.8716, "expected_b1": 292.4813, "expected_b2": 300.0, "expected_b3": 33.3904, "expected_snr_grad": 0.5}
|
||||
{"label": "medium-deployment-\u00a77.2", "calculator_version": "t3-bound-v1-bottou-refinement", "inputs": {"gradient_fraction": 0.05, "gradient_norm_max": 1.0, "gradient_noise_stddev": 0.1, "lr_decision_interval": 1000, "lr_grid_size": 16, "window_length": 100000, "batches_per_epoch": 8192, "steps_per_epoch": 8192}, "expected_total": 3387.72, "expected_b1": 2924.8125, "expected_b2": 400.0, "expected_b3": 62.9075, "expected_snr_grad": 0.5}
|
||||
{"label": "hardened-deployment-\u00a77.3", "calculator_version": "t3-bound-v1-bottou-refinement", "inputs": {"gradient_fraction": 0.01, "gradient_norm_max": 1.0, "gradient_noise_stddev": 0.1, "lr_decision_interval": 100, "lr_grid_size": 4, "window_length": 10000, "batches_per_epoch": 1024, "steps_per_epoch": 1024}, "expected_total": 247.1407, "expected_b1": 13.7504, "expected_b2": 200.0, "expected_b3": 33.3904, "expected_snr_grad": 0.1}
|
||||
{"label": "extreme-low-g", "calculator_version": "t3-bound-v1-bottou-refinement", "inputs": {"gradient_fraction": 0.001, "gradient_norm_max": 1.0, "gradient_noise_stddev": 0.5, "lr_decision_interval": 500, "lr_grid_size": 2, "window_length": 5000, "batches_per_epoch": 256, "steps_per_epoch": 256}, "expected_total": 80.0144, "expected_b1": 0.0144, "expected_b2": 10.0, "expected_b3": 70.0, "expected_snr_grad": 0.002}
|
||||
{"label": "tight-window-W=100", "calculator_version": "t3-bound-v1-bottou-refinement", "inputs": {"gradient_fraction": 0.05, "gradient_norm_max": 2.0, "gradient_noise_stddev": 0.1, "lr_decision_interval": 50, "lr_grid_size": 4, "window_length": 100, "batches_per_epoch": 64, "steps_per_epoch": 64}, "expected_total": 10.6781, "expected_b1": 5.0, "expected_b2": 4.0, "expected_b3": 1.6781, "expected_snr_grad": 1.0}
|
||||
{"label": "tightened-c-b1", "calculator_version": "t3-bound-v1-bottou-refinement", "inputs": {"gradient_fraction": 0.05, "gradient_norm_max": 1.0, "gradient_noise_stddev": 0.1, "lr_decision_interval": 100, "lr_grid_size": 8, "window_length": 10000, "batches_per_epoch": 1024, "steps_per_epoch": 1024, "c_b1": 0.5}, "expected_total": 479.631, "expected_b1": 146.2406, "expected_b2": 300.0, "expected_b3": 33.3904, "expected_snr_grad": 0.5}
|
||||
{"label": "all-constants-tight", "calculator_version": "t3-bound-v1-bottou-refinement", "inputs": {"gradient_fraction": 0.05, "gradient_norm_max": 1.0, "gradient_noise_stddev": 0.1, "lr_decision_interval": 100, "lr_grid_size": 8, "window_length": 10000, "batches_per_epoch": 1024, "steps_per_epoch": 1024, "c_b1": 0.1, "c_b2": 0.1, "c_b3": 0.1}, "expected_total": 62.5872, "expected_b1": 29.2481, "expected_b2": 30.0, "expected_b3": 3.339, "expected_snr_grad": 0.5}
|
||||
{"label": "b3-floor-regime", "calculator_version": "t3-bound-v1-bottou-refinement", "inputs": {"gradient_fraction": 0.05, "gradient_norm_max": 100.0, "gradient_noise_stddev": 0.001, "lr_decision_interval": 100, "lr_grid_size": 8, "window_length": 10000, "batches_per_epoch": 4, "steps_per_epoch": 4}, "expected_total": 6444.0004, "expected_b1": 6144.0004, "expected_b2": 300.0, "expected_b3": 0.0, "expected_snr_grad": 5000.0}
|
||||
|
|
@ -18,6 +18,7 @@ from __future__ import annotations
|
|||
|
||||
import json
|
||||
import math
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
|
@ -349,3 +350,97 @@ def test_total_equals_sum_of_three_contributions():
|
|||
total = r["B1_contribution"] + r["B2_contribution"] + r["B3_contribution"]
|
||||
# Allow small rounding (round to 4 dp on each contribution).
|
||||
assert r["I_window_bits_upper_bound"] == pytest.approx(total, abs=1e-3)
|
||||
|
||||
|
||||
# --- B3 hand-computed formula (gap caught by the docs/calculator-test-
|
||||
# patterns.md checklist, 2026-05-10) ---------------------------------
|
||||
|
||||
|
||||
def test_b3_exact_formula():
|
||||
"""B3 = C_B3 · ⌈W/E⌉ · log₂(N_b · σ_grad / ‖∇L_max‖) / 2 per
|
||||
#000036 §5 (Bottou-Bousquet refinement).
|
||||
|
||||
Pairs the existing test_b1_exact_formula + test_b2_exact_formula
|
||||
by hand-computing B3 from the spec formula. Closes the
|
||||
calculator-test-patterns.md §3 checklist gap (item 3:
|
||||
hand-formula assertions per output field).
|
||||
"""
|
||||
# Pick a config where the B3 factor (N_b · σ / ‖∇L_max‖) > 1
|
||||
# so the floor doesn't kick in — that case is covered separately
|
||||
# by test_b3_floor_at_zero_when_factor_below_one.
|
||||
g, gnm, sigma, W = 0.05, 1.0, 0.5, 4096
|
||||
K, R = 100, 8
|
||||
N_b, E = 512, 512
|
||||
factor = N_b * sigma / gnm # 256.0 > 1
|
||||
epochs = -(-W // E) # ceil
|
||||
expected_b3 = 1.0 * epochs * math.log2(factor) / 2.0
|
||||
r = t3_bound_bits(
|
||||
gradient_fraction=g, gradient_norm_max=gnm,
|
||||
gradient_noise_stddev=sigma, lr_decision_interval=K,
|
||||
lr_grid_size=R, window_length=W,
|
||||
batches_per_epoch=N_b, steps_per_epoch=E,
|
||||
)
|
||||
assert r["B3_contribution"] == pytest.approx(expected_b3, abs=1e-3), (
|
||||
f"B3 mismatch: function returned {r['B3_contribution']}, "
|
||||
f"hand-computed {expected_b3} via "
|
||||
f"⌈{W}/{E}⌉ · log₂({factor})/2 = {epochs} · {math.log2(factor):.4f}/2"
|
||||
)
|
||||
|
||||
|
||||
# --- KAT regression (gap caught by checklist, 2026-05-10) -----------
|
||||
|
||||
|
||||
KAT_FIXTURE = (
|
||||
pathlib.Path(__file__).parent.parent
|
||||
/ "bench" / "fixtures" / "t3-bound"
|
||||
/ "known-answer-tests.jsonl"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
not KAT_FIXTURE.exists(),
|
||||
reason="KAT fixture not yet generated",
|
||||
)
|
||||
def test_t3_bound_known_answer_tests():
|
||||
"""Replay the pinned t3-bound/known-answer-tests.jsonl fixture;
|
||||
every entry's per-contribution + total must match what the
|
||||
function produces today.
|
||||
|
||||
Algorithm change MUST bump ``CALCULATOR_VERSION`` and emit a new
|
||||
fixture file under ``bench/fixtures/t3-bound/`` — old runs
|
||||
replay against old data per the calculator-test-patterns.md §1
|
||||
discipline.
|
||||
|
||||
Closes the calculator-test-patterns.md §1 checklist gap (no
|
||||
KAT fixture for this calculator).
|
||||
"""
|
||||
n_kats = 0
|
||||
for line in KAT_FIXTURE.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
kat = json.loads(line)
|
||||
n_kats += 1
|
||||
r = t3_bound_bits(**kat["inputs"])
|
||||
assert r["calculator_version"] == kat["calculator_version"], (
|
||||
f"KAT version mismatch on {kat['label']!r}: "
|
||||
f"recorded {kat['calculator_version']}, "
|
||||
f"function {r['calculator_version']}"
|
||||
)
|
||||
for field, key in [
|
||||
("expected_total", "I_window_bits_upper_bound"),
|
||||
("expected_b1", "B1_contribution"),
|
||||
("expected_b2", "B2_contribution"),
|
||||
("expected_b3", "B3_contribution"),
|
||||
("expected_snr_grad", "snr_grad"),
|
||||
]:
|
||||
recorded = float(kat[field])
|
||||
observed = float(r[key])
|
||||
assert observed == pytest.approx(recorded, abs=1e-3), (
|
||||
f"KAT {field} mismatch on {kat['label']!r}: "
|
||||
f"recorded {recorded}, observed {observed}"
|
||||
)
|
||||
assert n_kats >= 5, (
|
||||
f"KAT fixture seems incomplete: only {n_kats} entries; "
|
||||
"calculator-test-patterns.md §1 wants ≥ 5"
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue