# Ticket #000044 — AUTOCOUNT doc-drift discipline

**Status:** closed · landed across `fc5ba50` / `03c0f6a` / `6c6defb` / `f5dbfab` / `3b30126` 2026-05-10 (mechanism + 4 metrics + 54 tags + cookbook + π* extensions). This ticket retroactively captures the design as the canonical record.
**Opened:** 2026-05-10
**Scope:** Lock the AUTOCOUNT regression-test pattern that catches
numeric-claim drift between `docs/**/*.md` prose and live state
(test files, JSONL fixtures, SQLite row counts). Format,
supported metrics, skip semantics, install discipline, and the
empirical baseline that motivated it.
**Audience:** future blackops shifts who write or refresh docs
that name counts; fox; anyone adding a new metric.
**Hard constraint:** the format must be invisible in rendered
markdown (GitHub strips HTML comments). The harness must not
fail when operator state (shards, qa.db) is absent — that would
break fresh-checkout / CI runs. Closed-ticket point-in-time
counts are deliberately NOT tagged.

---

## 1. Why this exists

On 2026-05-10, the doc-drift pattern recurred four times in one
day:

| Commit    | What drifted                               |
|-----------|---------------------------------------------|
| `6cbbf95` | cookbook test count claims (10 cells)       |
| `14bcb99` | cookbook test/code ratio table              |
| `5c21e83` | #000006 amend snapshot                      |
| `30a9488` | citation alias count + B-2 narrative        |

Each drift cost a Read + Edit + commit cycle (~5 min).
Cumulative cost was 20+ minutes that day, with no guarantee the
next drift gets caught before someone external reads it. The
pattern was always the same shape: fox writes a snapshot at time
T, more work lands by T+30min, snapshot is stale, someone
notices later.

Five-step algorithm walk:

1. **Make the requirement less dumb.** The requirement
   "documented counts match live state" was authored by anyone
   who reads the doc — there's no single named owner. Closest
   thing: fox writes the prose; future readers expect numerical
   honesty. Honest claims need a mechanism.
2. **Delete the part.** Considered removing counts entirely from
   docs — see §5. Rejected because counts carry comparative
   value ("23 tests vs the prior 14, +9 from backfill").
3. **Simplify and optimize.** A regression test is the simplest
   loud-signal mechanism that doesn't auto-rewrite prose
   (preserving fox's "no scripted prose touching" instinct).
4. **Accelerate cycle time.** From 5min/drift to a test failure
   in next CI run (or immediately during `pytest tests/test_doc_counts.py`).
5. **Automate.** Last, not first. Hand-rolled (each refresh
   manual) → scripted (regression test fires + names the file +
   line + claimed-vs-live).

---

## 2. Format

```markdown
<!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->
```

- `metric` — one of four documented metrics (§3).
- `path` — metric-specific target string.
- `N` — the claimed value (author-typed, machine-verified).

GitHub + most markdown renderers strip HTML comments, so the
rendered output shows just `N`. Tags are invisible in published
docs but make every numeric claim machine-checkable.

The number lives inside the tag delimiters — single source of
truth. No risk of a "tag says 23 but prose says 25" desync,
because the prose IS the inside-of-tag content.

---

## 3. Supported metrics

### 3.1 `tests`

```markdown
<!--AUTOCOUNT:tests:tests/test_anchor_prg.py-->31<!--/AUTOCOUNT-->
```

Live value: `pytest --collect-only -q <path>` count, with
`@pytest.mark.parametrize` expansion. The harness batches every
tagged path into one subprocess call (~0.5s for 50+ tags).

### 3.2 `fixture-rows`

```markdown
<!--AUTOCOUNT:fixture-rows:bench/fixtures/phi-prg/known-answer-tests.jsonl-->10<!--/AUTOCOUNT-->
```

Live value: non-blank, non-comment-prefixed line count in the
JSONL file. Skips lines starting with `#` (rare in JSONL but
defensive).

### 3.3 `db-rows`

```markdown
<!--AUTOCOUNT:db-rows:citation_aliases-->74<!--/AUTOCOUNT-->
<!--AUTOCOUNT:db-rows:002.db:concept_relations-->1234<!--/AUTOCOUNT-->
```

Live value: `SELECT COUNT(*) FROM <table>` against
`~/.arborist/shards/000.db` (default) or
`~/.arborist/shards/<shard>` (explicit shard prefix). Table name
validated as bare identifier before string-interpolating into the
query. SQLite connection opens `mode=ro` URI.

### 3.4 `db-where`

```markdown
<!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
<!--AUTOCOUNT:db-where:001.db:documents?source_type=wikipedia_xml-->867695<!--/AUTOCOUNT-->
```

Live value: `SELECT COUNT(*) FROM <table> WHERE <column> = ?`
with `<value>` bound as a SQL parameter (not string-interpolated).
Single-column equality predicate only — anything more complex is
out of scope for v1 (write a custom assertion test instead).

---

## 4. Skip semantics (CI / fresh-checkout safety)

Operator state (`~/.arborist/shards/*.db`, `qa.db`) is not
present in CI or fresh checkouts. The `db-rows` and `db-where`
metrics skip with a logged note rather than failing when:

- The DB file is absent (`_DB_MISSING`)
- The table doesn't exist (`_TABLE_MISSING`)
- A sqlite error occurs (`_DB_ERROR`)

Skipped claims print to captured stdout (visible under `pytest -s`)
with file + line + reason; the test still passes as long as no
present-DB claim has drifted.

The `tests` and `fixture-rows` metrics fail loudly when their
targets are missing — those targets live in the repo, not in
operator state. A missing test file or fixture is always a real
defect.

Smoke verified 2026-05-10 with `HOME=/tmp/empty pytest
tests/test_doc_counts.py`: 3 db-rows claims gracefully skip
with informative messages; suite passes.

---

## 5. What NOT to tag

**Closed-ticket "N tests pass" snapshots.** Tickets like #000031
§8, #000008 §12 record landing-time test counts as a frozen
historical claim. Those rows record "code at the moment of
ticket closure", not "code today". Tagging them would fire on
every successive change to the codebase, defeating the design-
log discipline that closed tickets stay archival.

**Aggregate / suite-wide numbers.** Counts like "the suite has
2300+ tests" in README are floors, not exact claims. Tagging
them is brittle (test count grows monotonically) and the floor
is structurally non-drifting — "2000+" stays accurate when the
suite is 2350.

**Historical-journey arcs.** Statements like "count grew 40 →
54 → 74" carry trajectory information; tagging the trajectory
endpoints would lose narrative shape. Tag the current-state
claim adjacent to the narrative if needed; leave the journey
prose untagged.

**Counts that are derivations of other tagged counts.** If
docs say "92/92 (100%)" and both numerator and denominator are
the same claim-pack count, tag once not twice.

---

## 6. Install discipline

When writing a new doc that names a count:

1. Author the prose with the count inline.
2. Wrap the count in the AUTOCOUNT tag pair on save.
3. Run `pytest tests/test_doc_counts.py`. ~3-5s. If it passes,
   ship.

When refreshing an existing count:

1. Note the file + line from the test's failure message.
2. Edit the number inside the tag (the tag itself stays).
3. Re-run the test.

When closing a ticket whose body has live-state counts:

1. Decide if the count is point-in-time (no tag — preserve as
   historical record) or load-bearing for future readers (tag).
2. Most ticket bodies are point-in-time. Keep the discipline by
   default: tag in cookbook + reference docs, NOT in tickets.

---

## 7. Future metrics (deferred)

The 4 metrics above cover today's drift surface. Patterns worth
adding when a new drift class fires:

- **`file-lines`** — line count of a non-JSONL file (e.g.
  paper length: "834 lines" for v8 consensus paper at
  `docs/_source/merkle-agi-v8-consensus.rst`). Pattern surfaced
  but not currently drifting.
- **`gh-pr-comments-count`** — number of comments on a referenced
  GitHub PR. Only meaningful if docs cite external PR state.
  Speculative.
- **`module-loc`** — Python file line count. Lower priority
  than the others — tools like `cloc` already report this
  and tagging it doesn't catch a class of drift that's not
  already caught by the test count drift adjacent to it.
- **`commit-hash-exists`** — assert a referenced commit hash
  resolves in git. Catches stale commit references after force
  pushes or branch deletions. Speculative.

Add a metric by:

1. Adding a `_live_<metric>(target: str) -> int` helper in
   `tests/test_doc_counts.py`.
2. Adding the dispatch branch in
   `test_doc_autocount_claims_match_live`.
3. Adding the metric name to the `known` set in
   `test_autocount_metric_names_are_documented` (so
   undocumented metric typos fire loud).
4. Adding the metric to the module docstring.
5. Tagging at least one real claim with the new metric.

---

## 8. Empirical baseline

As of 2026-05-10 landing of `3b30126`:

- **<!--AUTOCOUNT:tests:tests/test_doc_counts.py-->3<!--/AUTOCOUNT--> tests** in
  `tests/test_doc_counts.py`:
  `test_doc_autocount_claims_match_live`,
  `test_autocount_tags_are_well_formed`,
  `test_autocount_metric_names_are_documented`.
- **58 active tagged claims** across 8 doc files at #000044
  landing (counted post-code-fence-strip — the parser ignores
  tags inside ```` ``` ```` blocks since those are illustrative
  examples, not live claims).
- Tags by metric: 46 tests + 6 fixture-rows + 3 db-rows + 3 db-where.
- Harness runtime: 2-4s. Single `pytest --collect-only` subprocess
  batches all `tests` claims.
- Drift catches since landing: 1 same-day catch (fox's
  uncommitted +6 prometheus + +3 prometheus_audit tests fired
  the harness within minutes during my cookbook edit, refresh
  was 60-second turnaround).

---

## 9. Scope boundaries — what this ticket does NOT do

- **Does not** auto-rewrite prose. The harness fires; the
  maintainer refreshes by hand. Auto-rewriting prose was a
  rejected option (Five-step §5 — automate last) because it
  touches load-bearing wording that humans need to author.
- **Does not** validate prose quality, just numeric claims.
  Stale phase descriptions ("Phase 1 awaiting fox approval"
  when Phase 1 has landed) are out of scope — different drift
  class, different mechanism needed.
- **Does not** scan source-code docstrings or CLI `--help`
  output. Limited to `docs/**/*.md`. Could extend; pattern
  hasn't manifested there.
- **Does not** lock numeric claims to a fixed value. The
  CLAIM is what's tagged, not a target. Updating the number
  is one-line.
- **Does not** introduce new dependencies. Pure stdlib
  (`re`, `sqlite3`, `subprocess`).

---

## 10. Sibling discipline — §-status section drift

AUTOCOUNT catches numeric drift. A parallel drift class showed
up in the same 2026-05-10 audit wave: prose §-status sections
inside ticket bodies (typically §7 Status or §11 Status) freeze
at landing time while file headers update inline as new phases
land. The §-section becomes a point-in-time snapshot that
diverges from the load-bearing surface fox actually refreshes.

The 2026-05-10 audit caught this in 5 of 6 in-progress tickets:

| Ticket   | §-section            | Drift caught                  |
|----------|----------------------|-------------------------------|
| #000034  | §7 lead              | "Open · awaiting" / body said "Phase 1a landed" |
| #000035  | §7 lead              | clean (no drift)              |
| #000036  | §7 lead + closure(c) | "Awaits fox review" / file: "in flight with dav1d" |
| #000037  | §20 + §17.1          | "open · awaiting go/no-go" / file: Phase 0+1+2 landed |
| #000025  | §11 lead             | "Open · awaiting" / file: 1a + 1b.2 + 1c + 1d + 1e landed |
| #000012  | §7 lead + Phase 1a tests | Phase 1a only / file: 1a + 1b landed; test file renamed |

The discipline has **two different rules** depending on ticket
state:

### 10.1 In-progress tickets

Body §-status sections must match the file header. Refresh in
place when the file header changes. The header is the
load-bearing surface (it's what TICKETS.md index rows mirror);
body sections that lag are pure drift.

Walk pattern when amending an in-progress ticket:

1. Update file header status line.
2. Walk §-status body section. If it disagrees with the header,
   refresh in the same commit.
3. Walk TICKETS.md index row. If it's a condensed version of
   the header, refresh that too.

### 10.2 Closed tickets

Body §-status sections are **archival**. They record the state
at landing time and stay frozen. Do NOT rewrite a closed
ticket's §-status section when later phases land — even if the
ticket is still being amended with B-1 / B-2 follow-ups under
the same ticket number.

Subsequent phases on a closed ticket go in:

- The file header (consolidated across all phases)
- A NEW section appended below (e.g. "§8.b Subsequent phases"
  or "Phase B follow-ups"), not by rewriting §8

Rewriting §8 of #000031 with Phase 2 content would erase the
Phase 1 landing record. That record is the design log; future
readers tracing how the textbook substrate evolved need to see
what landed at Phase 1 vs what was added later.

### 10.3 The rule in one line

**In-progress §-status must match header. Closed §-status is
archival and stays frozen.**

This rule is NOT machine-checked. No regression test enforces
it; the audit pattern is manual + periodic. Worth a sweep when:

- Any ticket flips from in-progress to closed (final §-section
  refresh before freezing)
- Any in-progress ticket gains a new phase landing (refresh §
  to match)
- Quarterly housekeeping pass on all in-progress tickets

Long-term, this could become a tagged claim (e.g. an AUTOCOUNT
metric like `ticket-header-status-matches-section` that
asserts §-status text matches header text up to whitespace
normalization). Deferred until the audit pattern recurs enough
to justify the surface.

---

## 11. References

- `tests/test_doc_counts.py` — the harness module (docstring is
  the canonical format reference).
- `fc5ba50` — initial regression test + 29 tags.
- `03c0f6a` — db-rows metric + 15 tag backfill (cookbook table
  + #000035).
- `6c6defb` — db-where metric + prometheus controller tags
  (#000037).
- `f5dbfab` — calculator-test-patterns.md extension with the
  three π* discipline pins.
- `3b30126` — cookbook harvest section + #000037/#000025
  status refresh (the most recent drift catch + refresh cycle).
- `docs/calculator-test-patterns.md` — sister doc with the test
  patterns this discipline complements.
