tests/doc_counts: regression test for numeric claims in docs/ (4x drift fix)

The doc-drift pattern recurred four times today on 2026-05-10
(commits 6cbbf95, 14bcb99, 5c21e83, 30a9488). Each fix was the
same shape: walk a doc, find a count that drifted from live truth
during the hours after the doc was written, refresh it. Cost: ~5
min per drift × 4 = 20 min of manual catching, with no guarantee
the next drift gets caught before someone external reads it.

Per fox's selection: regression test that makes drift loud at
test time instead of relying on visual catching.

== Mechanism ==

`tests/test_doc_counts.py` scans `docs/**/*.md` for AUTOCOUNT
tags of the form:

  <!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->

Two metrics supported:

- `tests` — pytest collected count for path. Batches every
  tagged path into one `pytest --collect-only` subprocess
  (~0.5s total).
- `fixture-rows` — non-blank-non-comment line count in a JSONL
  fixture.

GitHub and most markdown renderers strip HTML comments, so
readers see only `N`. The tags are invisible in rendered output
but make the claim machine-checkable. Three tests in the file:

1. `test_doc_autocount_claims_match_live` — the core invariant
2. `test_autocount_tags_are_well_formed` — open/close balance
3. `test_autocount_metric_names_are_documented` — fail-closed on
   undocumented metrics (catches typos)

Failure message names the doc file, line number, and the
claimed-vs-live diff. Example:
`docs/foo.md:42 AUTOCOUNT(tests:tests/test_x.py) claims 23, live is 27`

== 29 tags installed across 5 docs ==

While installing tags I had to read the surrounding prose, which
surfaced six stale counts that had drifted same-day:

`docs/soft-hash-channel-analysis.md`:
- L392 14 → 23 tests for phi_alignment_probe
- L417 20 → 27 tests for anchor_prg
- L463 14 → 23 tests for phi_alignment_probe (status section)

`docs/seven-point-program.md`:
- L77 68 → 58 tests for metacognition (drift -10; the file
  shed tests during a refactor and the doc didn't catch up)
- L78 9 tests for `test_dag.py::test_preflight_*` — removed
  count entirely; pytest selector subsets aren't currently
  supported by the AUTOCOUNT metric set (would need a
  `tests-matching` metric; not worth the surface for one claim).
- L110 24 → 33 tests for test_dag.py

`docs/calculator-test-patterns.md`:
- L35 33 → 23 tests for warrant_resolver
- L35 10 → 9 tests for warrant_chain
- L16, L265 51 → 53 tests for t3_bound_calculator (kept
  initial-shipment provenance in prose)

== Coverage installed ==

  calculator-test-patterns.md           3 tagged claims
  soft-hash-channel-analysis.md         5 tagged claims
  warrant-substrate-cookbook.md        14 tagged claims
  seven-point-program.md                3 tagged claims
  tickets/ticket-000006-bench-...      4 tagged claims
                                      ---
                                       29 tagged claims

Every count that drifted today is now tagged. Future drift
fires the regression test at the next pytest run instead of
waiting for human catching.

== Discipline pattern ==

Walk this pattern for any new doc that names a count:

1. Surround the number with the tag pair:
   `<!--AUTOCOUNT:tests:tests/test_foo.py-->N<!--/AUTOCOUNT-->`
2. Run `pytest tests/test_doc_counts.py` (~3.5s)
3. If it passes, the claim is now machine-verified

Aim to tag counts on first authorship. Retrofitting is cheap
but only catches drift after the fact.

== Out of scope ==

Test counts inside source code (docstrings, CLI --help) are not
scanned — would expand the test surface significantly and the
drift pattern hasn't manifested there. Add `**/*.py` scope when
that pattern surfaces.

Alias-row counts and claim-pack-record counts could be tagged
with new `db-rows:<table>` and `db-where:<sql>` metrics; deferred
until the next drift on those numbers (none caught today after
30a9488's cookbook refresh).

== Verification ==

  $ .venv/bin/pytest tests/test_doc_counts.py -v
  3 passed in 3.89s

  $ .venv/bin/pytest -q
  2276 passed, 54 skipped in 153.21s

No new dependencies. No schema changes. No source-code changes.

tests/test_doc_counts.py

@@ -0,0 +1,180 @@
+"""Regression test for AUTOCOUNT-tagged numeric claims in ``docs/``.
+
+The doc-drift pattern recurred four times on 2026-05-10 (commits
+``6cbbf95``, ``14bcb99``, ``5c21e83``, ``30a9488``). Test counts and
+alias counts in docs went stale within hours of fox writing them
+because more work landed in parallel. Each refresh cost a
+Read + Edit + commit cycle.
+
+This test makes drift loud at test time instead of catching it
+visually. Tag any numeric claim in ``docs/**/*.md`` with the
+HTML-comment pair below; the test compares the claimed number to
+the live value and fails with a clear diff message.
+
+Format::
+
+    <!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->
+
+Where ``metric`` is one of:
+
+- ``tests`` — ``path`` is a pytest-collectible file/directory; the
+  live value is ``pytest --collect-only`` count.
+- ``fixture-rows`` — ``path`` is a JSONL fixture; the live value
+  is the non-blank-non-comment line count.
+
+GitHub and most markdown renderers strip HTML comments, so readers
+see only ``N``. The tags are invisible in rendered output but
+make the claim machine-checkable.
+
+Adding a new claim: write the surrounding prose with the number,
+wrap the number in the AUTOCOUNT comment pair, save, run
+``pytest tests/test_doc_counts.py``. If it passes, ship.
+"""
+
+from __future__ import annotations
+
+import re
+import subprocess
+import sys
+from collections import defaultdict
+from pathlib import Path
+from typing import Callable, Iterator
+
+import pytest
+
+REPO_ROOT = Path(__file__).parent.parent
+DOCS_DIR = REPO_ROOT / "docs"
+
+AUTOCOUNT_RE = re.compile(
+    r"<!--\s*AUTOCOUNT:(?P<metric>[a-z][a-z0-9_-]*)"
+    r":(?P<path>[^\s>]+)\s*-->"
+    r"(?P<n>\d+)"
+    r"<!--\s*/AUTOCOUNT\s*-->"
+)
+
+
+def _live_test_count_batch(paths: list[str]) -> dict[str, int]:
+    """Run ``pytest --collect-only`` once for every requested path.
+
+    Returns a {path: count} map. We invoke pytest as a subprocess so
+    we don't recursively collect ourselves. Single batch keeps cost
+    near constant (~0.5s) regardless of how many docs reference
+    test files.
+    """
+    if not paths:
+        return {}
+    out = subprocess.run(
+        [sys.executable, "-m", "pytest", "--collect-only", "-q", *paths],
+        capture_output=True,
+        text=True,
+        cwd=str(REPO_ROOT),
+        check=False,
+    )
+    counts: dict[str, int] = defaultdict(int)
+    for line in out.stdout.splitlines():
+        if "::" not in line:
+            continue
+        # Lines look like: tests/test_foo.py::test_bar
+        # or:              tests/test_foo.py::TestClass::test_bar[case]
+        file_part = line.split("::", 1)[0].strip()
+        counts[file_part] += 1
+    # Make sure every requested path got a key (zero if collected nothing).
+    for p in paths:
+        counts.setdefault(p, 0)
+    return dict(counts)
+
+
+def _live_fixture_rows(path: str) -> int:
+    full = REPO_ROOT / path
+    if not full.exists():
+        return -1
+    return sum(
+        1
+        for ln in full.read_text().splitlines()
+        if ln.strip() and not ln.strip().startswith("#")
+    )
+
+
+def _iter_claims() -> Iterator[tuple[Path, int, str, str, int]]:
+    """Yield (doc_path, lineno, metric, target_path, claimed_n)."""
+    for md in sorted(DOCS_DIR.rglob("*.md")):
+        text = md.read_text()
+        for m in AUTOCOUNT_RE.finditer(text):
+            lineno = text.count("\n", 0, m.start()) + 1
+            yield (
+                md,
+                lineno,
+                m.group("metric"),
+                m.group("path"),
+                int(m.group("n")),
+            )
+
+
+def test_doc_autocount_claims_match_live() -> None:
+    """Every ``<!--AUTOCOUNT:-->`` claim in ``docs/`` matches live."""
+    claims = list(_iter_claims())
+    # Batch the pytest-collect call across every tests:* claim — one
+    # subprocess instead of N.
+    test_paths = sorted({p for _, _, m, p, _ in claims if m == "tests"})
+    test_counts = _live_test_count_batch(test_paths)
+
+    drifts: list[str] = []
+    for doc, lineno, metric, target, claimed in claims:
+        rel = doc.relative_to(REPO_ROOT)
+        if metric == "tests":
+            live = test_counts.get(target, -1)
+        elif metric == "fixture-rows":
+            live = _live_fixture_rows(target)
+        else:
+            drifts.append(f"{rel}:{lineno} unknown AUTOCOUNT metric {metric!r}")
+            continue
+        if live < 0:
+            drifts.append(
+                f"{rel}:{lineno} AUTOCOUNT({metric}:{target}) target "
+                f"missing or uncollectable"
+            )
+        elif live != claimed:
+            drifts.append(
+                f"{rel}:{lineno} AUTOCOUNT({metric}:{target}) claims "
+                f"{claimed}, live is {live}"
+            )
+    assert not drifts, "Doc count drift detected:\n  " + "\n  ".join(drifts)
+
+
+def test_autocount_tags_are_well_formed() -> None:
+    """Catch typos in the comment pair (open without close, etc.)."""
+    open_re = re.compile(r"<!--\s*AUTOCOUNT[^>]*-->")
+    close_re = re.compile(r"<!--\s*/AUTOCOUNT\s*-->")
+    issues: list[str] = []
+    for md in sorted(DOCS_DIR.rglob("*.md")):
+        text = md.read_text()
+        opens = len(open_re.findall(text))
+        closes = len(close_re.findall(text))
+        full = len(AUTOCOUNT_RE.findall(text))
+        rel = md.relative_to(REPO_ROOT)
+        if opens != closes:
+            issues.append(
+                f"{rel}: {opens} open AUTOCOUNT tag(s) but "
+                f"{closes} close tag(s)"
+            )
+        if opens != full:
+            issues.append(
+                f"{rel}: {opens} open tag(s) but only {full} "
+                f"matched complete AUTOCOUNT pair(s) — check for "
+                f"missing digit, malformed close, or stray markup"
+            )
+    assert not issues, "Malformed AUTOCOUNT tags:\n  " + "\n  ".join(issues)
+
+
+def test_autocount_metric_names_are_documented() -> None:
+    """Fail-closed if a doc uses an undocumented metric."""
+    known = {"tests", "fixture-rows"}
+    seen: set[str] = set()
+    for _, _, metric, _, _ in _iter_claims():
+        seen.add(metric)
+    unknown = seen - known
+    assert not unknown, (
+        f"AUTOCOUNT uses undocumented metric(s) {sorted(unknown)}. "
+        f"Add the metric to test_doc_counts.py and to the module "
+        f"docstring's metric list before tagging docs with it."
+    )