ticket #000044: AUTOCOUNT doc-drift discipline + code-fence parser fix
Lock the AUTOCOUNT regression-test pattern as the design log canonical record. Previously declined when surface was 1-metric + 29 tags; now mature enough (4 metrics + 58 tags + 1 same-day drift-catch since landing) to formalize. == Ticket content == 10 sections covering: 1. Why this exists — the 4-drift-day baseline (6cbbf95/14bcb99/5c21e83/30a9488) that motivated mechanization. Five-step walk through justifying each choice (Step 5 last). 2. Format — `<!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->`. 3. Four supported metrics with examples + skip semantics: `tests`, `fixture-rows`, `db-rows`, `db-where`. 4. Skip-on-absence — operator state (shards, qa.db) absence is a logged skip, not a fail. Smoke verified 2026-05-10 with HOME=/tmp/empty. 5. What NOT to tag — closed-ticket point-in-time snapshots, aggregate floors ("2000+"), historical journey arcs. 6. Install discipline at write time + at refresh time. 7. Future metrics deferred (file-lines, gh-pr-comments-count, module-loc, commit-hash-exists) with the "add a metric" recipe. 8. Empirical baseline at landing (3 test functions, 58 active tagged claims across 8 doc files, harness runtime 2-4s). 9. Scope boundaries — does NOT auto-rewrite, does NOT validate prose quality, does NOT scan docstrings, does NOT lock values, does NOT add deps. 10. References — every landing commit + sister doc. Closed at landing (status quo sincefc5ba502026-05-10 morning; this ticket is retroactive design log per the convention "every ticket flips to `closed · landed in commit <sha>` when the work ships"). == Code-fence parser fix == Adding the ticket itself surfaced an oversight: my AUTOCOUNT examples in §3.3 + §3.4 used literal tag pairs in ``` fenced code blocks. The parser was reading them as live claims and firing on the illustrative `db-rows:002.db:concept_relations` claim (compared 1234 vs live 72576 — both meaningless because it's an example). Fix: `_strip_fenced_code_blocks` substitutes the body of every triple-backtick block with newlines before regex scanning. Line numbers stay aligned (newline-preserving substitution); tags inside fences are skipped because their parent text no longer matches the regex. Both helper functions (`_iter_claims` and the well-formed-tags test) walk through the stripped text, so the strip discipline is consistent across all three test functions. == TICKETS.md index == Added #000044 row marked closed with the 5-commit landing trail. Bumped Next ID 000044 → 000045. == Verification == $ pytest tests/test_doc_counts.py 3 passed in 2.80s $ pytest tests/ -q 2337 passed, 37 skipped in 108.29s Hygiene: fox's in-flight changes to arborist/qa/runner.py + arborist/substrate/prometheus.py + tests/test_prometheus*.py left untouched in working tree.
This commit is contained in:
parent
3b301267ff
commit
8da29fde69
3 changed files with 334 additions and 4 deletions
|
|
@ -81,6 +81,7 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000044 | AUTOCOUNT doc-drift discipline | closed · landed across `fc5ba50` / `03c0f6a` / `6c6defb` / `f5dbfab` / `3b30126` 2026-05-10 (mechanism + 4 metrics + 54 tags across 7 doc files; harness catches drift at test time, refresh is 60-second turnaround) | 2026-05-10 | — |
|
||||
| #000043 | Empirical tightening of T3 bound constants (#000036 Phase 2) | open · awaiting go/no-go (parks until v7 deployment surfaces measurement data) | 2026-05-10 | — |
|
||||
| #000042 | Term-aliases table (vocabulary-mismatch bridge) | closed · 13 rows live across geometry + classical-physics + arithmetic domains by 2026-05-10 | 2026-05-09 | — |
|
||||
| #000041 | Citation-aliases table (PD substitutes for proprietary cites) | closed · 74 rows live as of 2026-05-10 (count grew 40 → 54 → 74; Goldstein/Newton, Mendelson/Enderton/Jech/Landau/Gödel→{Russell IMP, Russell PoM, De Morgan, Boole, Cantor, Peano, Dedekind, SF-LF}, Stanley/Brualdi/Knuth → Bogart+Levin+Keller-Trotter, Dummit-Foote/Barendregt/Böhm-Jacopini → Judson/PLFA/SF, Kolmogorov → Grinstead-Snell+Laplace) | 2026-05-09 | — |
|
||||
|
|
@ -127,4 +128,4 @@ Newest first. Update on every open/close.
|
|||
|
||||
## Next ID
|
||||
|
||||
`000044`
|
||||
`000045`
|
||||
|
|
|
|||
297
docs/tickets/ticket-000044-autocount-doc-drift-discipline.md
Normal file
297
docs/tickets/ticket-000044-autocount-doc-drift-discipline.md
Normal file
|
|
@ -0,0 +1,297 @@
|
|||
# Ticket #000044 — AUTOCOUNT doc-drift discipline
|
||||
|
||||
**Status:** closed · landed across `fc5ba50` / `03c0f6a` / `6c6defb` / `f5dbfab` / `3b30126` 2026-05-10 (mechanism + 4 metrics + 54 tags + cookbook + π* extensions). This ticket retroactively captures the design as the canonical record.
|
||||
**Opened:** 2026-05-10
|
||||
**Scope:** Lock the AUTOCOUNT regression-test pattern that catches
|
||||
numeric-claim drift between `docs/**/*.md` prose and live state
|
||||
(test files, JSONL fixtures, SQLite row counts). Format,
|
||||
supported metrics, skip semantics, install discipline, and the
|
||||
empirical baseline that motivated it.
|
||||
**Audience:** future blackops shifts who write or refresh docs
|
||||
that name counts; fox; anyone adding a new metric.
|
||||
**Hard constraint:** the format must be invisible in rendered
|
||||
markdown (GitHub strips HTML comments). The harness must not
|
||||
fail when operator state (shards, qa.db) is absent — that would
|
||||
break fresh-checkout / CI runs. Closed-ticket point-in-time
|
||||
counts are deliberately NOT tagged.
|
||||
|
||||
---
|
||||
|
||||
## 1. Why this exists
|
||||
|
||||
On 2026-05-10, the doc-drift pattern recurred four times in one
|
||||
day:
|
||||
|
||||
| Commit | What drifted |
|
||||
|-----------|---------------------------------------------|
|
||||
| `6cbbf95` | cookbook test count claims (10 cells) |
|
||||
| `14bcb99` | cookbook test/code ratio table |
|
||||
| `5c21e83` | #000006 amend snapshot |
|
||||
| `30a9488` | citation alias count + B-2 narrative |
|
||||
|
||||
Each drift cost a Read + Edit + commit cycle (~5 min).
|
||||
Cumulative cost was 20+ minutes that day, with no guarantee the
|
||||
next drift gets caught before someone external reads it. The
|
||||
pattern was always the same shape: fox writes a snapshot at time
|
||||
T, more work lands by T+30min, snapshot is stale, someone
|
||||
notices later.
|
||||
|
||||
Five-step algorithm walk:
|
||||
|
||||
1. **Make the requirement less dumb.** The requirement
|
||||
"documented counts match live state" was authored by anyone
|
||||
who reads the doc — there's no single named owner. Closest
|
||||
thing: fox writes the prose; future readers expect numerical
|
||||
honesty. Honest claims need a mechanism.
|
||||
2. **Delete the part.** Considered removing counts entirely from
|
||||
docs — see §5. Rejected because counts carry comparative
|
||||
value ("23 tests vs the prior 14, +9 from backfill").
|
||||
3. **Simplify and optimize.** A regression test is the simplest
|
||||
loud-signal mechanism that doesn't auto-rewrite prose
|
||||
(preserving fox's "no scripted prose touching" instinct).
|
||||
4. **Accelerate cycle time.** From 5min/drift to a test failure
|
||||
in next CI run (or immediately during `pytest tests/test_doc_counts.py`).
|
||||
5. **Automate.** Last, not first. Hand-rolled (each refresh
|
||||
manual) → scripted (regression test fires + names the file +
|
||||
line + claimed-vs-live).
|
||||
|
||||
---
|
||||
|
||||
## 2. Format
|
||||
|
||||
```markdown
|
||||
<!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->
|
||||
```
|
||||
|
||||
- `metric` — one of four documented metrics (§3).
|
||||
- `path` — metric-specific target string.
|
||||
- `N` — the claimed value (author-typed, machine-verified).
|
||||
|
||||
GitHub + most markdown renderers strip HTML comments, so the
|
||||
rendered output shows just `N`. Tags are invisible in published
|
||||
docs but make every numeric claim machine-checkable.
|
||||
|
||||
The number lives inside the tag delimiters — single source of
|
||||
truth. No risk of a "tag says 23 but prose says 25" desync,
|
||||
because the prose IS the inside-of-tag content.
|
||||
|
||||
---
|
||||
|
||||
## 3. Supported metrics
|
||||
|
||||
### 3.1 `tests`
|
||||
|
||||
```markdown
|
||||
<!--AUTOCOUNT:tests:tests/test_anchor_prg.py-->27<!--/AUTOCOUNT-->
|
||||
```
|
||||
|
||||
Live value: `pytest --collect-only -q <path>` count, with
|
||||
`@pytest.mark.parametrize` expansion. The harness batches every
|
||||
tagged path into one subprocess call (~0.5s for 50+ tags).
|
||||
|
||||
### 3.2 `fixture-rows`
|
||||
|
||||
```markdown
|
||||
<!--AUTOCOUNT:fixture-rows:bench/fixtures/phi-prg/known-answer-tests.jsonl-->10<!--/AUTOCOUNT-->
|
||||
```
|
||||
|
||||
Live value: non-blank, non-comment-prefixed line count in the
|
||||
JSONL file. Skips lines starting with `#` (rare in JSONL but
|
||||
defensive).
|
||||
|
||||
### 3.3 `db-rows`
|
||||
|
||||
```markdown
|
||||
<!--AUTOCOUNT:db-rows:citation_aliases-->74<!--/AUTOCOUNT-->
|
||||
<!--AUTOCOUNT:db-rows:002.db:concept_relations-->1234<!--/AUTOCOUNT-->
|
||||
```
|
||||
|
||||
Live value: `SELECT COUNT(*) FROM <table>` against
|
||||
`~/.arborist/shards/000.db` (default) or
|
||||
`~/.arborist/shards/<shard>` (explicit shard prefix). Table name
|
||||
validated as bare identifier before string-interpolating into the
|
||||
query. SQLite connection opens `mode=ro` URI.
|
||||
|
||||
### 3.4 `db-where`
|
||||
|
||||
```markdown
|
||||
<!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
|
||||
<!--AUTOCOUNT:db-where:001.db:documents?source_type=wikipedia_xml-->867695<!--/AUTOCOUNT-->
|
||||
```
|
||||
|
||||
Live value: `SELECT COUNT(*) FROM <table> WHERE <column> = ?`
|
||||
with `<value>` bound as a SQL parameter (not string-interpolated).
|
||||
Single-column equality predicate only — anything more complex is
|
||||
out of scope for v1 (write a custom assertion test instead).
|
||||
|
||||
---
|
||||
|
||||
## 4. Skip semantics (CI / fresh-checkout safety)
|
||||
|
||||
Operator state (`~/.arborist/shards/*.db`, `qa.db`) is not
|
||||
present in CI or fresh checkouts. The `db-rows` and `db-where`
|
||||
metrics skip with a logged note rather than failing when:
|
||||
|
||||
- The DB file is absent (`_DB_MISSING`)
|
||||
- The table doesn't exist (`_TABLE_MISSING`)
|
||||
- A sqlite error occurs (`_DB_ERROR`)
|
||||
|
||||
Skipped claims print to captured stdout (visible under `pytest -s`)
|
||||
with file + line + reason; the test still passes as long as no
|
||||
present-DB claim has drifted.
|
||||
|
||||
The `tests` and `fixture-rows` metrics fail loudly when their
|
||||
targets are missing — those targets live in the repo, not in
|
||||
operator state. A missing test file or fixture is always a real
|
||||
defect.
|
||||
|
||||
Smoke verified 2026-05-10 with `HOME=/tmp/empty pytest
|
||||
tests/test_doc_counts.py`: 3 db-rows claims gracefully skip
|
||||
with informative messages; suite passes.
|
||||
|
||||
---
|
||||
|
||||
## 5. What NOT to tag
|
||||
|
||||
**Closed-ticket "N tests pass" snapshots.** Tickets like #000031
|
||||
§8, #000008 §12 record landing-time test counts as a frozen
|
||||
historical claim. Those rows record "code at the moment of
|
||||
ticket closure", not "code today". Tagging them would fire on
|
||||
every successive change to the codebase, defeating the design-
|
||||
log discipline that closed tickets stay archival.
|
||||
|
||||
**Aggregate / suite-wide numbers.** Counts like "the suite has
|
||||
2300+ tests" in README are floors, not exact claims. Tagging
|
||||
them is brittle (test count grows monotonically) and the floor
|
||||
is structurally non-drifting — "2000+" stays accurate when the
|
||||
suite is 2350.
|
||||
|
||||
**Historical-journey arcs.** Statements like "count grew 40 →
|
||||
54 → 74" carry trajectory information; tagging the trajectory
|
||||
endpoints would lose narrative shape. Tag the current-state
|
||||
claim adjacent to the narrative if needed; leave the journey
|
||||
prose untagged.
|
||||
|
||||
**Counts that are derivations of other tagged counts.** If
|
||||
docs say "92/92 (100%)" and both numerator and denominator are
|
||||
the same claim-pack count, tag once not twice.
|
||||
|
||||
---
|
||||
|
||||
## 6. Install discipline
|
||||
|
||||
When writing a new doc that names a count:
|
||||
|
||||
1. Author the prose with the count inline.
|
||||
2. Wrap the count in the AUTOCOUNT tag pair on save.
|
||||
3. Run `pytest tests/test_doc_counts.py`. ~3-5s. If it passes,
|
||||
ship.
|
||||
|
||||
When refreshing an existing count:
|
||||
|
||||
1. Note the file + line from the test's failure message.
|
||||
2. Edit the number inside the tag (the tag itself stays).
|
||||
3. Re-run the test.
|
||||
|
||||
When closing a ticket whose body has live-state counts:
|
||||
|
||||
1. Decide if the count is point-in-time (no tag — preserve as
|
||||
historical record) or load-bearing for future readers (tag).
|
||||
2. Most ticket bodies are point-in-time. Keep the discipline by
|
||||
default: tag in cookbook + reference docs, NOT in tickets.
|
||||
|
||||
---
|
||||
|
||||
## 7. Future metrics (deferred)
|
||||
|
||||
The 4 metrics above cover today's drift surface. Patterns worth
|
||||
adding when a new drift class fires:
|
||||
|
||||
- **`file-lines`** — line count of a non-JSONL file (e.g.
|
||||
paper length: "834 lines" for v8 consensus paper at
|
||||
`docs/_source/merkle-agi-v8-consensus.rst`). Pattern surfaced
|
||||
but not currently drifting.
|
||||
- **`gh-pr-comments-count`** — number of comments on a referenced
|
||||
GitHub PR. Only meaningful if docs cite external PR state.
|
||||
Speculative.
|
||||
- **`module-loc`** — Python file line count. Lower priority
|
||||
than the others — tools like `cloc` already report this
|
||||
and tagging it doesn't catch a class of drift that's not
|
||||
already caught by the test count drift adjacent to it.
|
||||
- **`commit-hash-exists`** — assert a referenced commit hash
|
||||
resolves in git. Catches stale commit references after force
|
||||
pushes or branch deletions. Speculative.
|
||||
|
||||
Add a metric by:
|
||||
|
||||
1. Adding a `_live_<metric>(target: str) -> int` helper in
|
||||
`tests/test_doc_counts.py`.
|
||||
2. Adding the dispatch branch in
|
||||
`test_doc_autocount_claims_match_live`.
|
||||
3. Adding the metric name to the `known` set in
|
||||
`test_autocount_metric_names_are_documented` (so
|
||||
undocumented metric typos fire loud).
|
||||
4. Adding the metric to the module docstring.
|
||||
5. Tagging at least one real claim with the new metric.
|
||||
|
||||
---
|
||||
|
||||
## 8. Empirical baseline
|
||||
|
||||
As of 2026-05-10 landing of `3b30126`:
|
||||
|
||||
- **<!--AUTOCOUNT:tests:tests/test_doc_counts.py-->3<!--/AUTOCOUNT--> tests** in
|
||||
`tests/test_doc_counts.py`:
|
||||
`test_doc_autocount_claims_match_live`,
|
||||
`test_autocount_tags_are_well_formed`,
|
||||
`test_autocount_metric_names_are_documented`.
|
||||
- **58 active tagged claims** across 8 doc files at #000044
|
||||
landing (counted post-code-fence-strip — the parser ignores
|
||||
tags inside ```` ``` ```` blocks since those are illustrative
|
||||
examples, not live claims).
|
||||
- Tags by metric: 46 tests + 6 fixture-rows + 3 db-rows + 3 db-where.
|
||||
- Harness runtime: 2-4s. Single `pytest --collect-only` subprocess
|
||||
batches all `tests` claims.
|
||||
- Drift catches since landing: 1 same-day catch (fox's
|
||||
uncommitted +6 prometheus + +3 prometheus_audit tests fired
|
||||
the harness within minutes during my cookbook edit, refresh
|
||||
was 60-second turnaround).
|
||||
|
||||
---
|
||||
|
||||
## 9. Scope boundaries — what this ticket does NOT do
|
||||
|
||||
- **Does not** auto-rewrite prose. The harness fires; the
|
||||
maintainer refreshes by hand. Auto-rewriting prose was a
|
||||
rejected option (Five-step §5 — automate last) because it
|
||||
touches load-bearing wording that humans need to author.
|
||||
- **Does not** validate prose quality, just numeric claims.
|
||||
Stale phase descriptions ("Phase 1 awaiting fox approval"
|
||||
when Phase 1 has landed) are out of scope — different drift
|
||||
class, different mechanism needed.
|
||||
- **Does not** scan source-code docstrings or CLI `--help`
|
||||
output. Limited to `docs/**/*.md`. Could extend; pattern
|
||||
hasn't manifested there.
|
||||
- **Does not** lock numeric claims to a fixed value. The
|
||||
CLAIM is what's tagged, not a target. Updating the number
|
||||
is one-line.
|
||||
- **Does not** introduce new dependencies. Pure stdlib
|
||||
(`re`, `sqlite3`, `subprocess`).
|
||||
|
||||
---
|
||||
|
||||
## 10. References
|
||||
|
||||
- `tests/test_doc_counts.py` — the harness module (docstring is
|
||||
the canonical format reference).
|
||||
- `fc5ba50` — initial regression test + 29 tags.
|
||||
- `03c0f6a` — db-rows metric + 15 tag backfill (cookbook table
|
||||
+ #000035).
|
||||
- `6c6defb` — db-where metric + prometheus controller tags
|
||||
(#000037).
|
||||
- `f5dbfab` — calculator-test-patterns.md extension with the
|
||||
three π* discipline pins.
|
||||
- `3b30126` — cookbook harvest section + #000037/#000025
|
||||
status refresh (the most recent drift catch + refresh cycle).
|
||||
- `docs/calculator-test-patterns.md` — sister doc with the test
|
||||
patterns this discipline complements.
|
||||
Loading…
Add table
Add a link
Reference in a new issue