arborist/docs/tickets/ticket-000044-autocount-doc-drift-discipline.md

Ticket #000044 — AUTOCOUNT doc-drift discipline

Status: closed · landed across fc5ba50 / 03c0f6a / 6c6defb / f5dbfab / 3b30126 2026-05-10 (mechanism + 4 metrics + 54 tags + cookbook + π* extensions). This ticket retroactively captures the design as the canonical record. Opened: 2026-05-10 Scope: Lock the AUTOCOUNT regression-test pattern that catches numeric-claim drift between docs/**/*.md prose and live state (test files, JSONL fixtures, SQLite row counts). Format, supported metrics, skip semantics, install discipline, and the empirical baseline that motivated it. Audience: future blackops shifts who write or refresh docs that name counts; fox; anyone adding a new metric. Hard constraint: the format must be invisible in rendered markdown (GitHub strips HTML comments). The harness must not fail when operator state (shards, qa.db) is absent — that would break fresh-checkout / CI runs. Closed-ticket point-in-time counts are deliberately NOT tagged.

---

1. Why this exists

On 2026-05-10, the doc-drift pattern recurred four times in one day:

Commit	What drifted
6cbbf95	cookbook test count claims (10 cells)
14bcb99	cookbook test/code ratio table
5c21e83	#000006 amend snapshot
30a9488	citation alias count + B-2 narrative

Each drift cost a Read + Edit + commit cycle (~5 min). Cumulative cost was 20+ minutes that day, with no guarantee the next drift gets caught before someone external reads it. The pattern was always the same shape: fox writes a snapshot at time T, more work lands by T+30min, snapshot is stale, someone notices later.

Five-step algorithm walk:

1. Make the requirement less dumb. The requirement "documented counts match live state" was authored by anyone who reads the doc — there's no single named owner. Closest thing: fox writes the prose; future readers expect numerical honesty. Honest claims need a mechanism.
2. Delete the part. Considered removing counts entirely from docs — see §5. Rejected because counts carry comparative value ("23 tests vs the prior 14, +9 from backfill").
3. Simplify and optimize. A regression test is the simplest loud-signal mechanism that doesn't auto-rewrite prose (preserving fox's "no scripted prose touching" instinct).
4. Accelerate cycle time. From 5min/drift to a test failure in next CI run (or immediately during pytest tests/test_doc_counts.py).
5. Automate. Last, not first. Hand-rolled (each refresh manual) → scripted (regression test fires + names the file + line + claimed-vs-live).

---

2. Format

<!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->

• metric — one of four documented metrics (§3).
• path — metric-specific target string.
• N — the claimed value (author-typed, machine-verified).

GitHub + most markdown renderers strip HTML comments, so the rendered output shows just N. Tags are invisible in published docs but make every numeric claim machine-checkable.

The number lives inside the tag delimiters — single source of truth. No risk of a "tag says 23 but prose says 25" desync, because the prose IS the inside-of-tag content.

---

3. Supported metrics

3.1 tests

<!--AUTOCOUNT:tests:tests/test_anchor_prg.py-->31<!--/AUTOCOUNT-->

Live value: pytest --collect-only -q <path> count, with @pytest.mark.parametrize expansion. The harness batches every tagged path into one subprocess call (~0.5s for 50+ tags).

3.2 fixture-rows

<!--AUTOCOUNT:fixture-rows:bench/fixtures/phi-prg/known-answer-tests.jsonl-->10<!--/AUTOCOUNT-->

Live value: non-blank, non-comment-prefixed line count in the JSONL file. Skips lines starting with # (rare in JSONL but defensive).

3.3 db-rows

<!--AUTOCOUNT:db-rows:citation_aliases-->74<!--/AUTOCOUNT-->
<!--AUTOCOUNT:db-rows:002.db:concept_relations-->1234<!--/AUTOCOUNT-->

Live value: SELECT COUNT(*) FROM <table> against ~/.arborist/shards/000.db (default) or ~/.arborist/shards/<shard> (explicit shard prefix). Table name validated as bare identifier before string-interpolating into the query. SQLite connection opens mode=ro URI.

3.4 db-where

<!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
<!--AUTOCOUNT:db-where:001.db:documents?source_type=wikipedia_xml-->867695<!--/AUTOCOUNT-->

Live value: SELECT COUNT(*) FROM <table> WHERE <column> = ? with <value> bound as a SQL parameter (not string-interpolated). Single-column equality predicate only — anything more complex is out of scope for v1 (write a custom assertion test instead).

---

4. Skip semantics (CI / fresh-checkout safety)

Operator state (~/.arborist/shards/*.db, qa.db) is not present in CI or fresh checkouts. The db-rows and db-where metrics skip with a logged note rather than failing when:

• The DB file is absent (_DB_MISSING)
• The table doesn't exist (_TABLE_MISSING)
• A sqlite error occurs (_DB_ERROR)

Skipped claims print to captured stdout (visible under pytest -s) with file + line + reason; the test still passes as long as no present-DB claim has drifted.

The tests and fixture-rows metrics fail loudly when their targets are missing — those targets live in the repo, not in operator state. A missing test file or fixture is always a real defect.

Smoke verified 2026-05-10 with HOME=/tmp/empty pytest tests/test_doc_counts.py: 3 db-rows claims gracefully skip with informative messages; suite passes.

---

5. What NOT to tag

Closed-ticket "N tests pass" snapshots. Tickets like #000031 §8, #000008 §12 record landing-time test counts as a frozen historical claim. Those rows record "code at the moment of ticket closure", not "code today". Tagging them would fire on every successive change to the codebase, defeating the design- log discipline that closed tickets stay archival.

Aggregate / suite-wide numbers. Counts like "the suite has 2300+ tests" in README are floors, not exact claims. Tagging them is brittle (test count grows monotonically) and the floor is structurally non-drifting — "2000+" stays accurate when the suite is 2350.

Historical-journey arcs. Statements like "count grew 40 → 54 → 74" carry trajectory information; tagging the trajectory endpoints would lose narrative shape. Tag the current-state claim adjacent to the narrative if needed; leave the journey prose untagged.

Counts that are derivations of other tagged counts. If docs say "92/92 (100%)" and both numerator and denominator are the same claim-pack count, tag once not twice.

---

6. Install discipline

When writing a new doc that names a count:

1. Author the prose with the count inline.
2. Wrap the count in the AUTOCOUNT tag pair on save.
3. Run pytest tests/test_doc_counts.py. ~3-5s. If it passes, ship.

When refreshing an existing count:

1. Note the file + line from the test's failure message.
2. Edit the number inside the tag (the tag itself stays).
3. Re-run the test.

When closing a ticket whose body has live-state counts:

1. Decide if the count is point-in-time (no tag — preserve as historical record) or load-bearing for future readers (tag).
2. Most ticket bodies are point-in-time. Keep the discipline by default: tag in cookbook + reference docs, NOT in tickets.

---

7. Future metrics (deferred)

The 4 metrics above cover today's drift surface. Patterns worth adding when a new drift class fires:

• file-lines — line count of a non-JSONL file (e.g. paper length: "834 lines" for v8 consensus paper at docs/_source/merkle-agi-v8-consensus.rst). Pattern surfaced but not currently drifting.
• gh-pr-comments-count — number of comments on a referenced GitHub PR. Only meaningful if docs cite external PR state. Speculative.
• module-loc — Python file line count. Lower priority than the others — tools like cloc already report this and tagging it doesn't catch a class of drift that's not already caught by the test count drift adjacent to it.
• commit-hash-exists — assert a referenced commit hash resolves in git. Catches stale commit references after force pushes or branch deletions. Speculative.

Add a metric by:

1. Adding a _live_<metric>(target: str) -> int helper in tests/test_doc_counts.py.
2. Adding the dispatch branch in test_doc_autocount_claims_match_live.
3. Adding the metric name to the known set in test_autocount_metric_names_are_documented (so undocumented metric typos fire loud).
4. Adding the metric to the module docstring.
5. Tagging at least one real claim with the new metric.

---

8. Empirical baseline

As of 2026-05-10 landing of 3b30126:

• 3 tests in tests/test_doc_counts.py: test_doc_autocount_claims_match_live, test_autocount_tags_are_well_formed, test_autocount_metric_names_are_documented.
• 58 active tagged claims across 8 doc files at #000044 landing (counted post-code-fence-strip — the parser ignores tags inside ``` blocks since those are illustrative examples, not live claims).
• Tags by metric: 46 tests + 6 fixture-rows + 3 db-rows + 3 db-where.
• Harness runtime: 2-4s. Single pytest --collect-only subprocess batches all tests claims.
• Drift catches since landing: 1 same-day catch (fox's uncommitted +6 prometheus + +3 prometheus_audit tests fired the harness within minutes during my cookbook edit, refresh was 60-second turnaround).

---

9. Scope boundaries — what this ticket does NOT do

• Does not auto-rewrite prose. The harness fires; the maintainer refreshes by hand. Auto-rewriting prose was a rejected option (Five-step §5 — automate last) because it touches load-bearing wording that humans need to author.
• Does not validate prose quality, just numeric claims. Stale phase descriptions ("Phase 1 awaiting fox approval" when Phase 1 has landed) are out of scope — different drift class, different mechanism needed.
• Does not scan source-code docstrings or CLI --help output. Limited to docs/**/*.md. Could extend; pattern hasn't manifested there.
• Does not lock numeric claims to a fixed value. The CLAIM is what's tagged, not a target. Updating the number is one-line.
• Does not introduce new dependencies. Pure stdlib (re, sqlite3, subprocess).

---

10. Sibling discipline — §-status section drift

AUTOCOUNT catches numeric drift. A parallel drift class showed up in the same 2026-05-10 audit wave: prose §-status sections inside ticket bodies (typically §7 Status or §11 Status) freeze at landing time while file headers update inline as new phases land. The §-section becomes a point-in-time snapshot that diverges from the load-bearing surface fox actually refreshes.

The 2026-05-10 audit caught this in 5 of 6 in-progress tickets:

Ticket	§-section	Drift caught
#000034	§7 lead	"Open · awaiting" / body said "Phase 1a landed"
#000035	§7 lead	clean (no drift)
#000036	§7 lead + closure(c)	"Awaits fox review" / file: "in flight with dav1d"
#000037	§20 + §17.1	"open · awaiting go/no-go" / file: Phase 0+1+2 landed
#000025	§11 lead	"Open · awaiting" / file: 1a + 1b.2 + 1c + 1d + 1e landed
#000012	§7 lead + Phase 1a tests	Phase 1a only / file: 1a + 1b landed; test file renamed

The discipline has two different rules depending on ticket state:

10.1 In-progress tickets

Body §-status sections must match the file header. Refresh in place when the file header changes. The header is the load-bearing surface (it's what TICKETS.md index rows mirror); body sections that lag are pure drift.

Walk pattern when amending an in-progress ticket:

1. Update file header status line.
2. Walk §-status body section. If it disagrees with the header, refresh in the same commit.
3. Walk TICKETS.md index row. If it's a condensed version of the header, refresh that too.

10.2 Closed tickets

Body §-status sections are archival. They record the state at landing time and stay frozen. Do NOT rewrite a closed ticket's §-status section when later phases land — even if the ticket is still being amended with B-1 / B-2 follow-ups under the same ticket number.

Subsequent phases on a closed ticket go in:

• The file header (consolidated across all phases)
• A NEW section appended below (e.g. "§8.b Subsequent phases" or "Phase B follow-ups"), not by rewriting §8

Rewriting §8 of #000031 with Phase 2 content would erase the Phase 1 landing record. That record is the design log; future readers tracing how the textbook substrate evolved need to see what landed at Phase 1 vs what was added later.

10.3 The rule in one line

In-progress §-status must match header. Closed §-status is archival and stays frozen.

This rule is NOT machine-checked. No regression test enforces it; the audit pattern is manual + periodic. Worth a sweep when:

• Any ticket flips from in-progress to closed (final §-section refresh before freezing)
• Any in-progress ticket gains a new phase landing (refresh § to match)
• Quarterly housekeeping pass on all in-progress tickets

Long-term, this could become a tagged claim (e.g. an AUTOCOUNT metric like ticket-header-status-matches-section that asserts §-status text matches header text up to whitespace normalization). Deferred until the audit pattern recurs enough to justify the surface.

---

11. References

• tests/test_doc_counts.py — the harness module (docstring is the canonical format reference).
• fc5ba50 — initial regression test + 29 tags.
• 03c0f6a — db-rows metric + 15 tag backfill (cookbook table
  • #000035).
• 6c6defb — db-where metric + prometheus controller tags (#000037).
• f5dbfab — calculator-test-patterns.md extension with the three π* discipline pins.
• 3b30126 — cookbook harvest section + #000037/#000025 status refresh (the most recent drift catch + refresh cycle).
• docs/calculator-test-patterns.md — sister doc with the test patterns this discipline complements.