From fc5ba507dc0b50a3fbae23cdfd2116660dcaa7d6 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sun, 10 May 2026 16:15:52 -0400 Subject: [PATCH] tests/doc_counts: regression test for numeric claims in docs/ (4x drift fix) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The doc-drift pattern recurred four times today on 2026-05-10 (commits 6cbbf95, 14bcb99, 5c21e83, 30a9488). Each fix was the same shape: walk a doc, find a count that drifted from live truth during the hours after the doc was written, refresh it. Cost: ~5 min per drift × 4 = 20 min of manual catching, with no guarantee the next drift gets caught before someone external reads it. Per fox's selection: regression test that makes drift loud at test time instead of relying on visual catching. == Mechanism == `tests/test_doc_counts.py` scans `docs/**/*.md` for AUTOCOUNT tags of the form: N Two metrics supported: - `tests` — pytest collected count for path. Batches every tagged path into one `pytest --collect-only` subprocess (~0.5s total). - `fixture-rows` — non-blank-non-comment line count in a JSONL fixture. GitHub and most markdown renderers strip HTML comments, so readers see only `N`. The tags are invisible in rendered output but make the claim machine-checkable. Three tests in the file: 1. `test_doc_autocount_claims_match_live` — the core invariant 2. `test_autocount_tags_are_well_formed` — open/close balance 3. `test_autocount_metric_names_are_documented` — fail-closed on undocumented metrics (catches typos) Failure message names the doc file, line number, and the claimed-vs-live diff. Example: `docs/foo.md:42 AUTOCOUNT(tests:tests/test_x.py) claims 23, live is 27` == 29 tags installed across 5 docs == While installing tags I had to read the surrounding prose, which surfaced six stale counts that had drifted same-day: `docs/soft-hash-channel-analysis.md`: - L392 14 → 23 tests for phi_alignment_probe - L417 20 → 27 tests for anchor_prg - L463 14 → 23 tests for phi_alignment_probe (status section) `docs/seven-point-program.md`: - L77 68 → 58 tests for metacognition (drift -10; the file shed tests during a refactor and the doc didn't catch up) - L78 9 tests for `test_dag.py::test_preflight_*` — removed count entirely; pytest selector subsets aren't currently supported by the AUTOCOUNT metric set (would need a `tests-matching` metric; not worth the surface for one claim). - L110 24 → 33 tests for test_dag.py `docs/calculator-test-patterns.md`: - L35 33 → 23 tests for warrant_resolver - L35 10 → 9 tests for warrant_chain - L16, L265 51 → 53 tests for t3_bound_calculator (kept initial-shipment provenance in prose) == Coverage installed == calculator-test-patterns.md 3 tagged claims soft-hash-channel-analysis.md 5 tagged claims warrant-substrate-cookbook.md 14 tagged claims seven-point-program.md 3 tagged claims tickets/ticket-000006-bench-... 4 tagged claims --- 29 tagged claims Every count that drifted today is now tagged. Future drift fires the regression test at the next pytest run instead of waiting for human catching. == Discipline pattern == Walk this pattern for any new doc that names a count: 1. Surround the number with the tag pair: `N` 2. Run `pytest tests/test_doc_counts.py` (~3.5s) 3. If it passes, the claim is now machine-verified Aim to tag counts on first authorship. Retrofitting is cheap but only catches drift after the fact. == Out of scope == Test counts inside source code (docstrings, CLI --help) are not scanned — would expand the test surface significantly and the drift pattern hasn't manifested there. Add `**/*.py` scope when that pattern surfaces. Alias-row counts and claim-pack-record counts could be tagged with new `db-rows:` and `db-where:` metrics; deferred until the next drift on those numbers (none caught today after 30a9488's cookbook refresh). == Verification == $ .venv/bin/pytest tests/test_doc_counts.py -v 3 passed in 3.89s $ .venv/bin/pytest -q 2276 passed, 54 skipped in 153.21s No new dependencies. No schema changes. No source-code changes. --- docs/calculator-test-patterns.md | 8 +- docs/seven-point-program.md | 9 +- docs/soft-hash-channel-analysis.md | 19 +- .../ticket-000006-bench-emergent-findings.md | 8 +- docs/warrant-substrate-cookbook.md | 28 +-- tests/test_doc_counts.py | 180 ++++++++++++++++++ 6 files changed, 220 insertions(+), 32 deletions(-) create mode 100644 tests/test_doc_counts.py diff --git a/docs/calculator-test-patterns.md b/docs/calculator-test-patterns.md index d5c2438..e966ba7 100644 --- a/docs/calculator-test-patterns.md +++ b/docs/calculator-test-patterns.md @@ -13,7 +13,9 @@ bench across three modules: closed-form bound for #000036 The exemplar test file is ``tests/test_t3_bound_calculator.py`` -(51 cases, written by fox 2026-05-10). The other two test files +(53 cases as of 2026-05-10; fox shipped 51 in the +initial cut and the +2 KAT-fixture-gap closure landed in +``581ad90``). The other two test files (``tests/test_anchor_prg.py`` and ``tests/test_phi_alignment_probe.py``) were retroactively backfilled with the same patterns in commits @@ -32,7 +34,7 @@ CLI, anything with closed-form formulas the test can hand-compute. ``arborist/qa/warrant_chain.py``), retrieval code, anything where inputs and outputs lack a hand-derivable mathematical relationship. The verifier-style tests in ``tests/test_warrant_resolver.py`` -(33 tests) and ``tests/test_warrant_chain.py`` (10 tests) are +(23 tests) and ``tests/test_warrant_chain.py`` (9 tests) are appropriately structured for their domain — descriptive function-name-per-input-shape works better than parametrized tables for parsers, and there's no formula to hand-check. @@ -262,7 +264,7 @@ When opening a new ``bench/scripts/.py`` or substrate primitive: 9. [ ] Module-export shape test (asdict round-trip, JSON-serializability, version token presence) -The 51 tests fox shipped for ``t3_bound_calculator`` are the +The 53 tests fox shipped for ``t3_bound_calculator`` are the exemplar; new modules should aim for similar coverage density on their own surface. diff --git a/docs/seven-point-program.md b/docs/seven-point-program.md index 0ac4a7d..21ea52b 100644 --- a/docs/seven-point-program.md +++ b/docs/seven-point-program.md @@ -73,9 +73,10 @@ frame and discarding the others. clauses bind into the run-DAG `preflight` stage). Future: runtime-side polarity contract emission for multi-frame answers. - **Pinning tests**: `tests/test_quantifier_classifier.py` - (10-rung intensity ladder, 70 tests), `tests/test_metacognition.py` - (4 detectors + governance + audit-line tails, 68 tests), - `tests/test_dag.py::test_preflight_*` (DAG binding, 9 tests). + (10-rung intensity ladder, 70 tests), `tests/test_metacognition.py` + (4 detectors + governance + audit-line tails, 58 tests), + `tests/test_dag.py::test_preflight_*` (DAG binding; subset of + the `test_dag.py` suite tagged below). Future: per-shape frame-detector tests + multi-frame answer-shape live fixtures. - **Open ticket**: @@ -107,7 +108,7 @@ cap / reminder / reject decisions hash-bound). preflight-extended shapes), `build_reject_run_dag` (3-stage), `preflight_node_hash` / `build_preflight_node_payload` (5-clause payload), `arborist/qa/evidence.py:evidence_map_root`. -- **Pinning tests**: `tests/test_dag.py` (24 tests including +- **Pinning tests**: `tests/test_dag.py` (33 tests including preflight stage + reject path), `tests/test_evidence.py`. - **Open ticket**: [#000001 Retrieval-keywords audit gap](ticket-000001-retrieval-keywords-audit-gap.md) closed; future retrieval-side refinement still possible. diff --git a/docs/soft-hash-channel-analysis.md b/docs/soft-hash-channel-analysis.md index 51704ab..984d265 100644 --- a/docs/soft-hash-channel-analysis.md +++ b/docs/soft-hash-channel-analysis.md @@ -389,11 +389,14 @@ The reduction in §4 leaves three loose threads: ablation infrastructure at `bench/scripts/phi_alignment_probe.py` (`measure_alignment(W, hessian_eval, *, k_top, k_bot, epsilon) → AlignmentReport`; Lanczos top-k + bottom-k via - `scipy.sparse.linalg.eigsh`). 14 tests in + `scipy.sparse.linalg.eigsh`). 23 tests in `tests/test_phi_alignment_probe.py` covering verdict classification (`STRUCTURAL_ALIGNMENT`, `NO_ALIGNMENT`, - `ANTI_ALIGNED`), boundary cases, and KAT regression. KAT - fixture at `bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl` + `ANTI_ALIGNED`), boundary cases, monotonicity, closure + (full-spectrum sum), parametrized invalid-input cones, and + KAT regression. KAT fixture (30 vectors: + 10 aligned + 10 uniform + 10 anti-aligned synthetic + checkpoints) at `bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl` pins synthetic-checkpoint verdicts so the probe is regression- guarded ahead of v7 deployment ramp-up. Lands under `bench/scripts/` (measurement tool, not a substrate primitive @@ -414,11 +417,13 @@ The reduction in §4 leaves three loose threads: **Phase 1 landed 2026-05-10** under #000035: reference implementation at `arborist/substrate/anchor_prg.py` (HMAC-SHA-512 counter-mode KDF, pure stdlib — `hashlib` + - `hmac`, no third-party crypto dep). 20 tests in + `hmac`, no third-party crypto dep). 27 tests in `tests/test_anchor_prg.py` covering determinism, range invariants, chi² uniformity, dim_h boundary (1, 16384), - seed-bit-flip and hash-bit-flip avalanche, input validation, - module-export shape, and KAT regression. 10 pinned KAT + seed-bit-flip and hash-bit-flip avalanche, hand-computed + HMAC-SHA-512 first-block formula, prefix-extension closure, + parametrized invalid-input cones, input validation, + module-export shape, and KAT regression. 10 pinned KAT vectors at `bench/fixtures/phi-prg/known-answer-tests.jsonl` covering block-boundary cases (dim_h=16 = one HMAC block; dim_h=17 = two blocks with truncation), one-bit-flip @@ -460,7 +465,7 @@ The reduction in §4 leaves three loose threads: **Open questions:** §9.1 (Hessian alignment under `φ_linear`: synthetic-ablation infrastructure landed 2026-05-10 per #000034 -Phase 1a — probe + 14 tests + KAT fixture; parks on v7 +Phase 1a — probe + 23 tests + KAT fixture; parks on v7 deployment data per #000034 Phase 1b for the actual checkpoint measurement), §9.2 (PRG cryptographic strength for `φ_PRG`: decision pinned + diff --git a/docs/tickets/ticket-000006-bench-emergent-findings.md b/docs/tickets/ticket-000006-bench-emergent-findings.md index 95ff5a1..3969d4b 100644 --- a/docs/tickets/ticket-000006-bench-emergent-findings.md +++ b/docs/tickets/ticket-000006-bench-emergent-findings.md @@ -722,19 +722,19 @@ Two cryptographic-primitive Phase 1 deliverables landed 2026-05-10 ahead of v7 plastic-training deployment: - **#000034 Phase 1a** (`1dfb8b9`): Hessian-alignment probe at - `bench/scripts/phi_alignment_probe.py` + 23 tests (was 14; + `bench/scripts/phi_alignment_probe.py` + 23 tests (was 14; +9 from `a4b3056` 2026-05-10 pattern backfill applying the monotonicity / hand-formula / closure / parametrized-invalid patterns from `docs/calculator-test-patterns.md`) + - 30-vector KAT fixture (10 aligned + 10 uniform + 10 + 30-vector KAT fixture (10 aligned + 10 uniform + 10 anti-aligned synthetic checkpoints, deterministic-seeded) at `bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl`. - **#000035 Phase 1** (earlier today): φ_PRG reference impl at `arborist/substrate/anchor_prg.py` (HMAC-SHA-512 counter-mode - KDF) + 27 tests (was 20; +7 from `de997f7` 2026-05-10 pattern + KDF) + 27 tests (was 20; +7 from `de997f7` 2026-05-10 pattern backfill — prefix-extension closure invariant, hand-computed HMAC-SHA-512 first-block formula, parametrized invalid-input - cones) + 10-vector KAT fixture at + cones) + 10-vector KAT fixture at `bench/fixtures/phi-prg/known-answer-tests.jsonl`. Both are KAT-pinned + bench-pinned regression artifacts that diff --git a/docs/warrant-substrate-cookbook.md b/docs/warrant-substrate-cookbook.md index d496c26..5c38992 100644 --- a/docs/warrant-substrate-cookbook.md +++ b/docs/warrant-substrate-cookbook.md @@ -459,7 +459,7 @@ than waiting for bench-time STRICT-rate drift to surface it. ### Citation-alias / term-alias mechanism (#000041 + #000042) -- `tests/test_aliases.py` — 28 tests covering +- `tests/test_aliases.py` — 28 tests covering `add_citation_alias` / `add_term_alias` audit-fail-closed (empty `decision_by` raises ValueError), domain isolation (same term different domain stays distinct), lowercase @@ -472,13 +472,13 @@ than waiting for bench-time STRICT-rate drift to surface it. ### Warrant-resolver chain (#000031 Phase 1+2+3) -- `tests/test_warrant_resolver.py` — 23 tests covering citation +- `tests/test_warrant_resolver.py` — 23 tests covering citation parsing (3 patterns: "Title by Author", multi-author Oxford comma, semicolon-separated multi-cite); cascade builder (5-query variants: title-phrase, parenthetical-phrase, AND-top-5, OR-top-3, legacy); `via_citation_alias` floor propagation through ResolutionMatch (B-1 attribution fix). -- `tests/test_warrant_chain.py` — 9 tests covering +- `tests/test_warrant_chain.py` — 9 tests covering `warrant_chain_lookup` (process_id LIKE filter, missing-table fail-closed, +alias variant matching), `has_warrant_chain` empty-set short-circuit, verifier suppression of @@ -487,7 +487,7 @@ than waiting for bench-time STRICT-rate drift to surface it. ### Textbook ingest license-discipline gate -- `tests/test_textbooks_manifest.py` — 43 tests covering every +- `tests/test_textbooks_manifest.py` — 43 tests covering every license token in `_ALLOWED_LICENSES` (parametrized 12-token sweep), placeholder rows allowed without URLs, disallowed license + emit URLs raises, CLI dispatch, return-code @@ -505,49 +505,49 @@ than waiting for bench-time STRICT-rate drift to surface it. ### Substrate-paper-spec'd primitives (#000012 + #000018 + #000034) -- `tests/test_anchor_prg.py` — 27 tests for φ_PRG HMAC-SHA-512 +- `tests/test_anchor_prg.py` — 27 tests for φ_PRG HMAC-SHA-512 expansion (#000035 Phase 1). Covers KAT regression, hand-formula (first-block matches direct ``hmac.new``), prefix-extending closure invariant, output-length monotonicity per dim_h. -- `tests/test_phi_alignment_probe.py` — **23 tests** (was 14; +- `tests/test_phi_alignment_probe.py` — **23 tests** (was 14; +9 from `a4b3056` 2026-05-10 pattern backfill) for φ_linear Hessian-alignment probe (#000034 Phase 1a). Covers KAT regression, hand-formula (uniform baseline), monotonicity in W concentration + dim_h, closure (a_top + a_bot ≡ full-spectrum on dense decomposition), Lanczos eigenvalue ordering invariant. -- `tests/test_t3_bound_calculator.py` — **53 tests** (was 51; +- `tests/test_t3_bound_calculator.py` — **53 tests** (was 51; +2 from `581ad90` 2026-05-10 KAT-fixture-gap closure) for the T3 per-window covert-channel bound calculator (#000036 §11); pins the closed-form B1/B2/B3 formulas, monotonicity in each input, recommendation-text mode transitions, and the §11 worked-example bit-for-bit (with doc-calibration update surfaced through the test). -- `tests/test_fork_score.py` — 18 tests for v8 ForkScore +- `tests/test_fork_score.py` — 18 tests for v8 ForkScore (#000012 Phase 1a); pins SIGNAL_FLOOR (5pp) + HARD_REGRESSION_FLOOR (5pp), score = sum-of-breakdown closure, security_risk inert under default iota=0 (opt-in), NEG_INF_REGRESSION hard-reject. -- `tests/test_substrate_fork_score.py` — 27 tests covering the +- `tests/test_substrate_fork_score.py` — 27 tests covering the ``arborist substrate score`` CLI surface (renamed from ``test_v8_fork_score.py`` in `a4058a4` per the 2026-05-10 v-prefix retirement). Adapter tests + 4 in-process CLI tests via ``build_parser()`` + 1 real subprocess invocation catching entry-point / sys.argv drift the in-process tests miss. -- `tests/test_weights.py` — 16 tests for WeightSet defaults +- `tests/test_weights.py` — 16 tests for WeightSet defaults (each weight value pinned to its docstring rationale; PR that flips alpha=1.0→0.5 fires this test), greek-letter and Python-safe key aliases, frozen-dataclass invariant. -- `tests/test_pi_star_protocol_and_registry.py` — 21 tests for +- `tests/test_pi_star_protocol_and_registry.py` — 21 tests for `PiStar` Protocol contract + registry mutation discipline (no public unregister; `name@version` content-pinned). ### Q&A / verifier scaffolding -- `tests/test_qa_progress.py` — 31 tests for the Progress +- `tests/test_qa_progress.py` — 31 tests for the Progress emitter (env / TTY / cli-override precedence; truthy/falsy spelling matrix; fail-closed on missing `.isatty`). -- `tests/test_qa_prompts.py` — 20 tests pinning load-bearing +- `tests/test_qa_prompts.py` — 20 tests pinning load-bearing system prompts (worked-example presence, two-pointer cap, pointer-mode no-quote-instruction discipline, JSON-mode first-char-`{` / last-char-`}` rule); a silent @@ -556,7 +556,7 @@ than waiting for bench-time STRICT-rate drift to surface it. ### Concept-relations write-side -- `tests/test_concepts_extract.py` — 20 tests for synonym / +- `tests/test_concepts_extract.py` — 20 tests for synonym / IDF / FTS5-titles extractors (`_title_tokens` stopword + length-floor + dedupe; `EXTRACTORS` registry contract; `link_reciprocity_synonym` idempotency + self-overlap diff --git a/tests/test_doc_counts.py b/tests/test_doc_counts.py new file mode 100644 index 0000000..3fee722 --- /dev/null +++ b/tests/test_doc_counts.py @@ -0,0 +1,180 @@ +"""Regression test for AUTOCOUNT-tagged numeric claims in ``docs/``. + +The doc-drift pattern recurred four times on 2026-05-10 (commits +``6cbbf95``, ``14bcb99``, ``5c21e83``, ``30a9488``). Test counts and +alias counts in docs went stale within hours of fox writing them +because more work landed in parallel. Each refresh cost a +Read + Edit + commit cycle. + +This test makes drift loud at test time instead of catching it +visually. Tag any numeric claim in ``docs/**/*.md`` with the +HTML-comment pair below; the test compares the claimed number to +the live value and fails with a clear diff message. + +Format:: + + N + +Where ``metric`` is one of: + +- ``tests`` — ``path`` is a pytest-collectible file/directory; the + live value is ``pytest --collect-only`` count. +- ``fixture-rows`` — ``path`` is a JSONL fixture; the live value + is the non-blank-non-comment line count. + +GitHub and most markdown renderers strip HTML comments, so readers +see only ``N``. The tags are invisible in rendered output but +make the claim machine-checkable. + +Adding a new claim: write the surrounding prose with the number, +wrap the number in the AUTOCOUNT comment pair, save, run +``pytest tests/test_doc_counts.py``. If it passes, ship. +""" + +from __future__ import annotations + +import re +import subprocess +import sys +from collections import defaultdict +from pathlib import Path +from typing import Callable, Iterator + +import pytest + +REPO_ROOT = Path(__file__).parent.parent +DOCS_DIR = REPO_ROOT / "docs" + +AUTOCOUNT_RE = re.compile( + r"" + r"(?P\d+)" + r"" +) + + +def _live_test_count_batch(paths: list[str]) -> dict[str, int]: + """Run ``pytest --collect-only`` once for every requested path. + + Returns a {path: count} map. We invoke pytest as a subprocess so + we don't recursively collect ourselves. Single batch keeps cost + near constant (~0.5s) regardless of how many docs reference + test files. + """ + if not paths: + return {} + out = subprocess.run( + [sys.executable, "-m", "pytest", "--collect-only", "-q", *paths], + capture_output=True, + text=True, + cwd=str(REPO_ROOT), + check=False, + ) + counts: dict[str, int] = defaultdict(int) + for line in out.stdout.splitlines(): + if "::" not in line: + continue + # Lines look like: tests/test_foo.py::test_bar + # or: tests/test_foo.py::TestClass::test_bar[case] + file_part = line.split("::", 1)[0].strip() + counts[file_part] += 1 + # Make sure every requested path got a key (zero if collected nothing). + for p in paths: + counts.setdefault(p, 0) + return dict(counts) + + +def _live_fixture_rows(path: str) -> int: + full = REPO_ROOT / path + if not full.exists(): + return -1 + return sum( + 1 + for ln in full.read_text().splitlines() + if ln.strip() and not ln.strip().startswith("#") + ) + + +def _iter_claims() -> Iterator[tuple[Path, int, str, str, int]]: + """Yield (doc_path, lineno, metric, target_path, claimed_n).""" + for md in sorted(DOCS_DIR.rglob("*.md")): + text = md.read_text() + for m in AUTOCOUNT_RE.finditer(text): + lineno = text.count("\n", 0, m.start()) + 1 + yield ( + md, + lineno, + m.group("metric"), + m.group("path"), + int(m.group("n")), + ) + + +def test_doc_autocount_claims_match_live() -> None: + """Every ```` claim in ``docs/`` matches live.""" + claims = list(_iter_claims()) + # Batch the pytest-collect call across every tests:* claim — one + # subprocess instead of N. + test_paths = sorted({p for _, _, m, p, _ in claims if m == "tests"}) + test_counts = _live_test_count_batch(test_paths) + + drifts: list[str] = [] + for doc, lineno, metric, target, claimed in claims: + rel = doc.relative_to(REPO_ROOT) + if metric == "tests": + live = test_counts.get(target, -1) + elif metric == "fixture-rows": + live = _live_fixture_rows(target) + else: + drifts.append(f"{rel}:{lineno} unknown AUTOCOUNT metric {metric!r}") + continue + if live < 0: + drifts.append( + f"{rel}:{lineno} AUTOCOUNT({metric}:{target}) target " + f"missing or uncollectable" + ) + elif live != claimed: + drifts.append( + f"{rel}:{lineno} AUTOCOUNT({metric}:{target}) claims " + f"{claimed}, live is {live}" + ) + assert not drifts, "Doc count drift detected:\n " + "\n ".join(drifts) + + +def test_autocount_tags_are_well_formed() -> None: + """Catch typos in the comment pair (open without close, etc.).""" + open_re = re.compile(r"") + close_re = re.compile(r"") + issues: list[str] = [] + for md in sorted(DOCS_DIR.rglob("*.md")): + text = md.read_text() + opens = len(open_re.findall(text)) + closes = len(close_re.findall(text)) + full = len(AUTOCOUNT_RE.findall(text)) + rel = md.relative_to(REPO_ROOT) + if opens != closes: + issues.append( + f"{rel}: {opens} open AUTOCOUNT tag(s) but " + f"{closes} close tag(s)" + ) + if opens != full: + issues.append( + f"{rel}: {opens} open tag(s) but only {full} " + f"matched complete AUTOCOUNT pair(s) — check for " + f"missing digit, malformed close, or stray markup" + ) + assert not issues, "Malformed AUTOCOUNT tags:\n " + "\n ".join(issues) + + +def test_autocount_metric_names_are_documented() -> None: + """Fail-closed if a doc uses an undocumented metric.""" + known = {"tests", "fixture-rows"} + seen: set[str] = set() + for _, _, metric, _, _ in _iter_claims(): + seen.add(metric) + unknown = seen - known + assert not unknown, ( + f"AUTOCOUNT uses undocumented metric(s) {sorted(unknown)}. " + f"Add the metric to test_doc_counts.py and to the module " + f"docstring's metric list before tagging docs with it." + )