tests/doc_counts: regression test for numeric claims in docs/ (4x drift fix)

The doc-drift pattern recurred four times today on 2026-05-10
(commits 6cbbf95, 14bcb99, 5c21e83, 30a9488). Each fix was the
same shape: walk a doc, find a count that drifted from live truth
during the hours after the doc was written, refresh it. Cost: ~5
min per drift × 4 = 20 min of manual catching, with no guarantee
the next drift gets caught before someone external reads it.

Per fox's selection: regression test that makes drift loud at
test time instead of relying on visual catching.

== Mechanism ==

`tests/test_doc_counts.py` scans `docs/**/*.md` for AUTOCOUNT
tags of the form:

  <!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->

Two metrics supported:

- `tests` — pytest collected count for path. Batches every
  tagged path into one `pytest --collect-only` subprocess
  (~0.5s total).
- `fixture-rows` — non-blank-non-comment line count in a JSONL
  fixture.

GitHub and most markdown renderers strip HTML comments, so
readers see only `N`. The tags are invisible in rendered output
but make the claim machine-checkable. Three tests in the file:

1. `test_doc_autocount_claims_match_live` — the core invariant
2. `test_autocount_tags_are_well_formed` — open/close balance
3. `test_autocount_metric_names_are_documented` — fail-closed on
   undocumented metrics (catches typos)

Failure message names the doc file, line number, and the
claimed-vs-live diff. Example:
`docs/foo.md:42 AUTOCOUNT(tests:tests/test_x.py) claims 23, live is 27`

== 29 tags installed across 5 docs ==

While installing tags I had to read the surrounding prose, which
surfaced six stale counts that had drifted same-day:

`docs/soft-hash-channel-analysis.md`:
- L392 14 → 23 tests for phi_alignment_probe
- L417 20 → 27 tests for anchor_prg
- L463 14 → 23 tests for phi_alignment_probe (status section)

`docs/seven-point-program.md`:
- L77 68 → 58 tests for metacognition (drift -10; the file
  shed tests during a refactor and the doc didn't catch up)
- L78 9 tests for `test_dag.py::test_preflight_*` — removed
  count entirely; pytest selector subsets aren't currently
  supported by the AUTOCOUNT metric set (would need a
  `tests-matching` metric; not worth the surface for one claim).
- L110 24 → 33 tests for test_dag.py

`docs/calculator-test-patterns.md`:
- L35 33 → 23 tests for warrant_resolver
- L35 10 → 9 tests for warrant_chain
- L16, L265 51 → 53 tests for t3_bound_calculator (kept
  initial-shipment provenance in prose)

== Coverage installed ==

  calculator-test-patterns.md           3 tagged claims
  soft-hash-channel-analysis.md         5 tagged claims
  warrant-substrate-cookbook.md        14 tagged claims
  seven-point-program.md                3 tagged claims
  tickets/ticket-000006-bench-...      4 tagged claims
                                      ---
                                       29 tagged claims

Every count that drifted today is now tagged. Future drift
fires the regression test at the next pytest run instead of
waiting for human catching.

== Discipline pattern ==

Walk this pattern for any new doc that names a count:

1. Surround the number with the tag pair:
   `<!--AUTOCOUNT:tests:tests/test_foo.py-->N<!--/AUTOCOUNT-->`
2. Run `pytest tests/test_doc_counts.py` (~3.5s)
3. If it passes, the claim is now machine-verified

Aim to tag counts on first authorship. Retrofitting is cheap
but only catches drift after the fact.

== Out of scope ==

Test counts inside source code (docstrings, CLI --help) are not
scanned — would expand the test surface significantly and the
drift pattern hasn't manifested there. Add `**/*.py` scope when
that pattern surfaces.

Alias-row counts and claim-pack-record counts could be tagged
with new `db-rows:<table>` and `db-where:<sql>` metrics; deferred
until the next drift on those numbers (none caught today after
30a9488's cookbook refresh).

== Verification ==

  $ .venv/bin/pytest tests/test_doc_counts.py -v
  3 passed in 3.89s

  $ .venv/bin/pytest -q
  2276 passed, 54 skipped in 153.21s

No new dependencies. No schema changes. No source-code changes.

docs/calculator-test-patterns.md

@@ -13,7 +13,9 @@ bench across three modules:
   closed-form bound for #000036
 
 The exemplar test file is ``tests/test_t3_bound_calculator.py``
-(51 cases, written by fox 2026-05-10). The other two test files
+(<!--AUTOCOUNT:tests:tests/test_t3_bound_calculator.py-->53<!--/AUTOCOUNT--> cases as of 2026-05-10; fox shipped 51 in the
+initial cut and the +2 KAT-fixture-gap closure landed in
+``581ad90``). The other two test files
 (``tests/test_anchor_prg.py`` and
 ``tests/test_phi_alignment_probe.py``) were retroactively
 backfilled with the same patterns in commits
@@ -32,7 +34,7 @@ CLI, anything with closed-form formulas the test can hand-compute.
 ``arborist/qa/warrant_chain.py``), retrieval code, anything where
 inputs and outputs lack a hand-derivable mathematical relationship.
 The verifier-style tests in ``tests/test_warrant_resolver.py``
-(33 tests) and ``tests/test_warrant_chain.py`` (10 tests) are
+(<!--AUTOCOUNT:tests:tests/test_warrant_resolver.py-->23<!--/AUTOCOUNT--> tests) and ``tests/test_warrant_chain.py`` (<!--AUTOCOUNT:tests:tests/test_warrant_chain.py-->9<!--/AUTOCOUNT--> tests) are
 appropriately structured for their domain — descriptive
 function-name-per-input-shape works better than parametrized
 tables for parsers, and there's no formula to hand-check.
@@ -262,7 +264,7 @@ When opening a new ``bench/scripts/<X>.py`` or substrate primitive:
 9. [ ] Module-export shape test (asdict round-trip,
    JSON-serializability, version token presence)
 
-The 51 tests fox shipped for ``t3_bound_calculator`` are the
+The <!--AUTOCOUNT:tests:tests/test_t3_bound_calculator.py-->53<!--/AUTOCOUNT--> tests fox shipped for ``t3_bound_calculator`` are the
 exemplar; new modules should aim for similar coverage density on
 their own surface.
 

docs/seven-point-program.md

@@ -73,9 +73,10 @@ frame and discarding the others.
   clauses bind into the run-DAG `preflight` stage). Future:
   runtime-side polarity contract emission for multi-frame answers.
 - **Pinning tests**: `tests/test_quantifier_classifier.py`
-  (10-rung intensity ladder, 70 tests), `tests/test_metacognition.py`
-  (4 detectors + governance + audit-line tails, 68 tests),
-  `tests/test_dag.py::test_preflight_*` (DAG binding, 9 tests).
+  (10-rung intensity ladder, <!--AUTOCOUNT:tests:tests/test_quantifier_classifier.py-->70<!--/AUTOCOUNT--> tests), `tests/test_metacognition.py`
+  (4 detectors + governance + audit-line tails, <!--AUTOCOUNT:tests:tests/test_metacognition.py-->58<!--/AUTOCOUNT--> tests),
+  `tests/test_dag.py::test_preflight_*` (DAG binding; subset of
+  the `test_dag.py` suite tagged below).
   Future: per-shape frame-detector tests + multi-frame answer-shape
   live fixtures.
 - **Open ticket**:
@@ -107,7 +108,7 @@ cap / reminder / reject decisions hash-bound).
   preflight-extended shapes), `build_reject_run_dag` (3-stage),
   `preflight_node_hash` / `build_preflight_node_payload` (5-clause
   payload), `arborist/qa/evidence.py:evidence_map_root`.
-- **Pinning tests**: `tests/test_dag.py` (24 tests including
+- **Pinning tests**: `tests/test_dag.py` (<!--AUTOCOUNT:tests:tests/test_dag.py-->33<!--/AUTOCOUNT--> tests including
   preflight stage + reject path), `tests/test_evidence.py`.
 - **Open ticket**: [#000001 Retrieval-keywords audit gap](ticket-000001-retrieval-keywords-audit-gap.md)
   closed; future retrieval-side refinement still possible.

docs/soft-hash-channel-analysis.md

@@ -389,11 +389,14 @@ The reduction in §4 leaves three loose threads:
    ablation infrastructure at `bench/scripts/phi_alignment_probe.py`
    (`measure_alignment(W, hessian_eval, *, k_top, k_bot, epsilon)
    → AlignmentReport`; Lanczos top-k + bottom-k via
-   `scipy.sparse.linalg.eigsh`). 14 tests in
+   `scipy.sparse.linalg.eigsh`). <!--AUTOCOUNT:tests:tests/test_phi_alignment_probe.py-->23<!--/AUTOCOUNT--> tests in
    `tests/test_phi_alignment_probe.py` covering verdict
    classification (`STRUCTURAL_ALIGNMENT`, `NO_ALIGNMENT`,
-   `ANTI_ALIGNED`), boundary cases, and KAT regression. KAT
-   fixture at `bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl`
+   `ANTI_ALIGNED`), boundary cases, monotonicity, closure
+   (full-spectrum sum), parametrized invalid-input cones, and
+   KAT regression. KAT fixture (<!--AUTOCOUNT:fixture-rows:bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl-->30<!--/AUTOCOUNT--> vectors:
+   10 aligned + 10 uniform + 10 anti-aligned synthetic
+   checkpoints) at `bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl`
    pins synthetic-checkpoint verdicts so the probe is regression-
    guarded ahead of v7 deployment ramp-up. Lands under
    `bench/scripts/` (measurement tool, not a substrate primitive
@@ -414,11 +417,13 @@ The reduction in §4 leaves three loose threads:
    **Phase 1 landed 2026-05-10** under #000035: reference
    implementation at `arborist/substrate/anchor_prg.py`
    (HMAC-SHA-512 counter-mode KDF, pure stdlib — `hashlib` +
-   `hmac`, no third-party crypto dep). 20 tests in
+   `hmac`, no third-party crypto dep). <!--AUTOCOUNT:tests:tests/test_anchor_prg.py-->27<!--/AUTOCOUNT--> tests in
    `tests/test_anchor_prg.py` covering determinism, range
    invariants, chi² uniformity, dim_h boundary (1, 16384),
-   seed-bit-flip and hash-bit-flip avalanche, input validation,
-   module-export shape, and KAT regression. 10 pinned KAT
+   seed-bit-flip and hash-bit-flip avalanche, hand-computed
+   HMAC-SHA-512 first-block formula, prefix-extension closure,
+   parametrized invalid-input cones, input validation,
+   module-export shape, and KAT regression. <!--AUTOCOUNT:fixture-rows:bench/fixtures/phi-prg/known-answer-tests.jsonl-->10<!--/AUTOCOUNT--> pinned KAT
    vectors at `bench/fixtures/phi-prg/known-answer-tests.jsonl`
    covering block-boundary cases (dim_h=16 = one HMAC block;
    dim_h=17 = two blocks with truncation), one-bit-flip
@@ -460,7 +465,7 @@ The reduction in §4 leaves three loose threads:
 
 **Open questions:** §9.1 (Hessian alignment under `φ_linear`:
 synthetic-ablation infrastructure landed 2026-05-10 per #000034
-Phase 1a — probe + 14 tests + KAT fixture; parks on v7
+Phase 1a — probe + <!--AUTOCOUNT:tests:tests/test_phi_alignment_probe.py-->23<!--/AUTOCOUNT--> tests + KAT fixture; parks on v7
 deployment data per #000034 Phase 1b for the actual checkpoint
 measurement),
 §9.2 (PRG cryptographic strength for `φ_PRG`: decision pinned +

docs/tickets/ticket-000006-bench-emergent-findings.md

@@ -722,19 +722,19 @@ Two cryptographic-primitive Phase 1 deliverables landed
 2026-05-10 ahead of v7 plastic-training deployment:
 
 - **#000034 Phase 1a** (`1dfb8b9`): Hessian-alignment probe at
-  `bench/scripts/phi_alignment_probe.py` + 23 tests (was 14;
+  `bench/scripts/phi_alignment_probe.py` + <!--AUTOCOUNT:tests:tests/test_phi_alignment_probe.py-->23<!--/AUTOCOUNT--> tests (was 14;
   +9 from `a4b3056` 2026-05-10 pattern backfill applying the
   monotonicity / hand-formula / closure / parametrized-invalid
   patterns from `docs/calculator-test-patterns.md`) +
-  30-vector KAT fixture (10 aligned + 10 uniform + 10
+  <!--AUTOCOUNT:fixture-rows:bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl-->30<!--/AUTOCOUNT-->-vector KAT fixture (10 aligned + 10 uniform + 10
   anti-aligned synthetic checkpoints, deterministic-seeded) at
   `bench/fixtures/phi-alignment/synthetic-checkpoints.jsonl`.
 - **#000035 Phase 1** (earlier today): φ_PRG reference impl at
   `arborist/substrate/anchor_prg.py` (HMAC-SHA-512 counter-mode
-  KDF) + 27 tests (was 20; +7 from `de997f7` 2026-05-10 pattern
+  KDF) + <!--AUTOCOUNT:tests:tests/test_anchor_prg.py-->27<!--/AUTOCOUNT--> tests (was 20; +7 from `de997f7` 2026-05-10 pattern
   backfill — prefix-extension closure invariant, hand-computed
   HMAC-SHA-512 first-block formula, parametrized invalid-input
-  cones) + 10-vector KAT fixture at
+  cones) + <!--AUTOCOUNT:fixture-rows:bench/fixtures/phi-prg/known-answer-tests.jsonl-->10<!--/AUTOCOUNT-->-vector KAT fixture at
   `bench/fixtures/phi-prg/known-answer-tests.jsonl`.
 
 Both are KAT-pinned + bench-pinned regression artifacts that

docs/warrant-substrate-cookbook.md

@@ -459,7 +459,7 @@ than waiting for bench-time STRICT-rate drift to surface it.
 
 ### Citation-alias / term-alias mechanism (#000041 + #000042)
 
-- `tests/test_aliases.py` — 28 tests covering
+- `tests/test_aliases.py` — <!--AUTOCOUNT:tests:tests/test_aliases.py-->28<!--/AUTOCOUNT--> tests covering
   `add_citation_alias` / `add_term_alias` audit-fail-closed
   (empty `decision_by` raises ValueError), domain isolation
   (same term different domain stays distinct), lowercase
@@ -472,13 +472,13 @@ than waiting for bench-time STRICT-rate drift to surface it.
 
 ### Warrant-resolver chain (#000031 Phase 1+2+3)
 
-- `tests/test_warrant_resolver.py` — 23 tests covering citation
+- `tests/test_warrant_resolver.py` — <!--AUTOCOUNT:tests:tests/test_warrant_resolver.py-->23<!--/AUTOCOUNT--> tests covering citation
   parsing (3 patterns: "Title by Author", multi-author Oxford
   comma, semicolon-separated multi-cite); cascade builder
   (5-query variants: title-phrase, parenthetical-phrase,
   AND-top-5, OR-top-3, legacy); `via_citation_alias` floor
   propagation through ResolutionMatch (B-1 attribution fix).
-- `tests/test_warrant_chain.py` — 9 tests covering
+- `tests/test_warrant_chain.py` — <!--AUTOCOUNT:tests:tests/test_warrant_chain.py-->9<!--/AUTOCOUNT--> tests covering
   `warrant_chain_lookup` (process_id LIKE filter, missing-table
   fail-closed, +alias variant matching), `has_warrant_chain`
   empty-set short-circuit, verifier suppression of
@@ -487,7 +487,7 @@ than waiting for bench-time STRICT-rate drift to surface it.
 
 ### Textbook ingest license-discipline gate
 
-- `tests/test_textbooks_manifest.py` — 43 tests covering every
+- `tests/test_textbooks_manifest.py` — <!--AUTOCOUNT:tests:tests/test_textbooks_manifest.py-->43<!--/AUTOCOUNT--> tests covering every
   license token in `_ALLOWED_LICENSES` (parametrized 12-token
   sweep), placeholder rows allowed without URLs, disallowed
   license + emit URLs raises, CLI dispatch, return-code
@@ -505,49 +505,49 @@ than waiting for bench-time STRICT-rate drift to surface it.
 
 ### Substrate-paper-spec'd primitives (#000012 + #000018 + #000034)
 
-- `tests/test_anchor_prg.py` — 27 tests for φ_PRG HMAC-SHA-512
+- `tests/test_anchor_prg.py` — <!--AUTOCOUNT:tests:tests/test_anchor_prg.py-->27<!--/AUTOCOUNT--> tests for φ_PRG HMAC-SHA-512
   expansion (#000035 Phase 1). Covers KAT regression, hand-formula
   (first-block matches direct ``hmac.new``), prefix-extending
   closure invariant, output-length monotonicity per dim_h.
-- `tests/test_phi_alignment_probe.py` — **23 tests** (was 14;
+- `tests/test_phi_alignment_probe.py` — **<!--AUTOCOUNT:tests:tests/test_phi_alignment_probe.py-->23<!--/AUTOCOUNT--> tests** (was 14;
   +9 from `a4b3056` 2026-05-10 pattern backfill) for φ_linear
   Hessian-alignment probe (#000034 Phase 1a). Covers KAT
   regression, hand-formula (uniform baseline), monotonicity in
   W concentration + dim_h, closure (a_top + a_bot ≡
   full-spectrum on dense decomposition), Lanczos eigenvalue
   ordering invariant.
-- `tests/test_t3_bound_calculator.py` — **53 tests** (was 51;
+- `tests/test_t3_bound_calculator.py` — **<!--AUTOCOUNT:tests:tests/test_t3_bound_calculator.py-->53<!--/AUTOCOUNT--> tests** (was 51;
   +2 from `581ad90` 2026-05-10 KAT-fixture-gap closure) for the
   T3 per-window covert-channel bound calculator (#000036 §11);
   pins the closed-form B1/B2/B3 formulas, monotonicity in each
   input, recommendation-text mode transitions, and the §11
   worked-example bit-for-bit (with doc-calibration update
   surfaced through the test).
-- `tests/test_fork_score.py` — 18 tests for v8 ForkScore
+- `tests/test_fork_score.py` — <!--AUTOCOUNT:tests:tests/test_fork_score.py-->18<!--/AUTOCOUNT--> tests for v8 ForkScore
   (#000012 Phase 1a); pins SIGNAL_FLOOR (5pp) + HARD_REGRESSION_FLOOR
   (5pp), score = sum-of-breakdown closure, security_risk inert
   under default iota=0 (opt-in), NEG_INF_REGRESSION hard-reject.
-- `tests/test_substrate_fork_score.py` — 27 tests covering the
+- `tests/test_substrate_fork_score.py` — <!--AUTOCOUNT:tests:tests/test_substrate_fork_score.py-->27<!--/AUTOCOUNT--> tests covering the
   ``arborist substrate score`` CLI surface (renamed from
   ``test_v8_fork_score.py`` in `a4058a4` per the 2026-05-10
   v-prefix retirement). Adapter tests + 4 in-process CLI tests
   via ``build_parser()`` + 1 real subprocess invocation
   catching entry-point / sys.argv drift the in-process tests
   miss.
-- `tests/test_weights.py` — 16 tests for WeightSet defaults
+- `tests/test_weights.py` — <!--AUTOCOUNT:tests:tests/test_weights.py-->16<!--/AUTOCOUNT--> tests for WeightSet defaults
   (each weight value pinned to its docstring rationale; PR that
   flips alpha=1.0→0.5 fires this test), greek-letter and
   Python-safe key aliases, frozen-dataclass invariant.
-- `tests/test_pi_star_protocol_and_registry.py` — 21 tests for
+- `tests/test_pi_star_protocol_and_registry.py` — <!--AUTOCOUNT:tests:tests/test_pi_star_protocol_and_registry.py-->21<!--/AUTOCOUNT--> tests for
   `PiStar` Protocol contract + registry mutation discipline
   (no public unregister; `name@version` content-pinned).
 
 ### Q&A / verifier scaffolding
 
-- `tests/test_qa_progress.py` — 31 tests for the Progress
+- `tests/test_qa_progress.py` — <!--AUTOCOUNT:tests:tests/test_qa_progress.py-->31<!--/AUTOCOUNT--> tests for the Progress
   emitter (env / TTY / cli-override precedence; truthy/falsy
   spelling matrix; fail-closed on missing `.isatty`).
-- `tests/test_qa_prompts.py` — 20 tests pinning load-bearing
+- `tests/test_qa_prompts.py` — <!--AUTOCOUNT:tests:tests/test_qa_prompts.py-->20<!--/AUTOCOUNT--> tests pinning load-bearing
   system prompts (worked-example presence, two-pointer cap,
   pointer-mode no-quote-instruction discipline,
   JSON-mode first-char-`{` / last-char-`}` rule); a silent
@@ -556,7 +556,7 @@ than waiting for bench-time STRICT-rate drift to surface it.
 
 ### Concept-relations write-side
 
-- `tests/test_concepts_extract.py` — 20 tests for synonym /
+- `tests/test_concepts_extract.py` — <!--AUTOCOUNT:tests:tests/test_concepts_extract.py-->20<!--/AUTOCOUNT--> tests for synonym /
   IDF / FTS5-titles extractors (`_title_tokens` stopword
   + length-floor + dedupe; `EXTRACTORS` registry contract;
   `link_reciprocity_synonym` idempotency + self-overlap