The three "92 claim_pack docs" tags were drifting against shard 000.db's
21 rows because the harness only counted one shard, but the doc prose
("#000031 closed at 92") meant the corpus total (21+16+38+17 across
genesis shards 000-003).
Two-line fix path: extend the harness to sum across all ???.db shards
via a `*:` prefix (e.g. `*:documents?source_type=claim_pack`), then
prefix the three drifted tags. Aligns the harness scope with the
semantic scope of the claim instead of forcing the claim to shrink to
one shard.
The `*:` glob:
- Matches `[0-9][0-9][0-9].db` basenames only (operator sidecars
qa.db / snapshots.db / selfmodel-chain.db skipped)
- Skips shards lacking the named table (schema-version tolerance)
- Returns _DB_MISSING when no genesis shard exists (CI / fresh-
checkout skip semantic preserved)
- Returns _TABLE_MISSING when no contributing shard has the table
Documented in ticket-000044 §3.4 + a third example showing the new
syntax. Diagnosis credit to a sub-agent investigation that confirmed
zero eviction/falsification audit events on claim_packs — the data is
intact; the harness was just single-shard.
Lock the AUTOCOUNT regression-test pattern as the design log
canonical record. Previously declined when surface was 1-metric
+ 29 tags; now mature enough (4 metrics + 58 tags + 1 same-day
drift-catch since landing) to formalize.
== Ticket content ==
10 sections covering:
1. Why this exists — the 4-drift-day baseline (6cbbf95 / 14bcb99 /
5c21e83 / 30a9488) that motivated mechanization. Five-step
walk through justifying each choice (Step 5 last).
2. Format — `<!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->`.
3. Four supported metrics with examples + skip semantics:
`tests`, `fixture-rows`, `db-rows`, `db-where`.
4. Skip-on-absence — operator state (shards, qa.db) absence is a
logged skip, not a fail. Smoke verified 2026-05-10 with
HOME=/tmp/empty.
5. What NOT to tag — closed-ticket point-in-time snapshots,
aggregate floors ("2000+"), historical journey arcs.
6. Install discipline at write time + at refresh time.
7. Future metrics deferred (file-lines, gh-pr-comments-count,
module-loc, commit-hash-exists) with the "add a metric"
recipe.
8. Empirical baseline at landing (3 test functions, 58 active
tagged claims across 8 doc files, harness runtime 2-4s).
9. Scope boundaries — does NOT auto-rewrite, does NOT validate
prose quality, does NOT scan docstrings, does NOT lock
values, does NOT add deps.
10. References — every landing commit + sister doc.
Closed at landing (status quo since fc5ba50 2026-05-10 morning;
this ticket is retroactive design log per the convention "every
ticket flips to `closed · landed in commit <sha>` when the work
ships").
== Code-fence parser fix ==
Adding the ticket itself surfaced an oversight: my AUTOCOUNT
examples in §3.3 + §3.4 used literal tag pairs in ``` fenced
code blocks. The parser was reading them as live claims and
firing on the illustrative `db-rows:002.db:concept_relations`
claim (compared 1234 vs live 72576 — both meaningless because
it's an example).
Fix: `_strip_fenced_code_blocks` substitutes the body of every
triple-backtick block with newlines before regex scanning. Line
numbers stay aligned (newline-preserving substitution); tags
inside fences are skipped because their parent text no longer
matches the regex.
Both helper functions (`_iter_claims` and the well-formed-tags
test) walk through the stripped text, so the strip discipline
is consistent across all three test functions.
== TICKETS.md index ==
Added #000044 row marked closed with the 5-commit landing trail.
Bumped Next ID 000044 → 000045.
== Verification ==
$ pytest tests/test_doc_counts.py
3 passed in 2.80s
$ pytest tests/ -q
2337 passed, 37 skipped in 108.29s
Hygiene: fox's in-flight changes to arborist/qa/runner.py +
arborist/substrate/prometheus.py + tests/test_prometheus*.py
left untouched in working tree.
Fan-out follow-up: extends AUTOCOUNT with a new metric for filtered
SQL-row claims, then tags fox's prometheus controller test surfaces
shipped this evening under #000037.
== Task 3: db-where metric ==
New metric ``db-where`` for tagging single-column equality
predicates. Target syntax::
<table>?<column>=<value>
<shard>:<table>?<column>=<value>
Resolves to ``SELECT COUNT(*) FROM <table> WHERE <column> = ?``
with ``<value>`` bound as a SQL parameter (no string
interpolation), so author typos or stray content can't escape
the predicate. Column + table names validated as bare
identifiers before string-interpolating into the query template;
sqlite3 connection opens with ``mode=ro`` URI flag.
Same skip-on-absence semantics as ``db-rows``: missing DB or
table yields a logged skip note, not a test failure. Sentinel
returns reuse the same _DB_MISSING / _TABLE_MISSING / _DB_ERROR
constants.
Smoke verified::
_live_db_where('documents?source_type=claim_pack') → 92
_live_db_where('documents?source_type=wikipedia_xml') → 866782
_live_db_where('001.db:documents?source_type=wikipedia_xml')
→ 867695
_live_db_where('documents') # malformed (no ?) → -4
Tagged claims using the new metric (cookbook):
- L36 ``92 records total`` for the claim-pack source
- L349 ``The 92 chains have three quality tiers``
Both resolve to ``documents WHERE source_type='claim_pack'``
in shard ``000.db`` — the live count of claim-pack records.
L5 ``18/92 → 92/92`` historical narrative left untagged
(expressing a journey arc, not current state).
== Task 2: tag prometheus controller test surfaces ==
fox shipped two test files this evening under #000037 that
weren't previously inventoried in any reference doc:
- ``tests/test_prometheus.py`` — 36 tests covering Phase 1
controller (commits ``f625cac`` + ``f9f5ae4``). Verifier-style
discipline (NOT calculator pattern — it's a pure-function
state-machine controller with no closed-form math).
- ``tests/test_prometheus_audit.py`` — 14 tests covering Phase 2
``controller_events`` sibling table (commit ``a786d6d``). Pins
no-chain-mutation invariant (advisory writes never enter
audit_events.event_hash preimage).
Added two paragraphs to cookbook §"Substrate-paper-spec'd
primitives" describing the test discipline + algorithmic surfaces
each pins. Renamed section header from
"(#000012 + #000018 + #000034)" to
"(#000012 + #000018 + #000034 + #000037)" to keep the
ticket-set roster current.
Two new rows in the test/code-density table:
| substrate/prometheus.py | 893 | 804 (36 tests) | 0.90 |
| substrate/prometheus_audit.py | 200 | 388 (14 tests) | 1.94 |
All four counts AUTOCOUNT-tagged (2 inline prose + 2 table rows
= 4 new tags). prometheus_audit's ratio of 1.94 is high because
the test file pins a lot of write-path invariants for what is
nominally a small (200 LOC) sibling-table module — appropriate
for foundation-level audit-discipline code.
== Task 4: test_full_suite_total_fixture_count flake — investigated ==
Earlier today's transient ``-x`` flake (1 failed, 1799 passed)
did NOT reproduce in current tree state (2328 passed, 37 skipped
under same flags). Root-cause investigation:
- ``_DEFAULT_FIXTURES`` is read-only at module scope; no test
mutates it.
- The test reads JSONL fixtures from ``bench/fixtures/`` via
``_run_one``; those files weren't being written by parallel
tests.
Hypothesis (not confirmed, since flake didn't repro): transient
filesystem state during heavy-parallel-commit window
(``f625cac`` / ``a786d6d`` / ``6142437`` / ``f9f5ae4`` all
landed in succession around 17:23-17:24 EDT 2026-05-10 while my
test run was in flight). No structural defect identified. If
flake recurs, capture stdout + filesystem state at failure time
to confirm.
== Task 1: Walked fox's 6 evening commits via Explore agent ==
Agent reported what shipped under #000037 + #000012 evening
push (Prometheus-Σ Phases 0/1/2/dry-run + v8 consensus paper).
Findings used to drive task 2 above. Notable design choices
worth surfacing as reference:
- prometheus.py is **pure function** (no DB / LLM / scheduler);
returns advisory ``ControllerDecision`` + optional proposal
records, never mutations. Verifier-style test discipline.
- ``controller_events`` sibling table never enters
``audit_events.event_hash`` preimage — audit chain unaffected.
- Phase 3 sleep-sweep scheduler **deferred** in favor of
read-only dry-run simulator surfacing 5 design constraints
(chunk_size = Hermes concurrency NOT candidate pool;
capital_cost flat=1.0 needs split; τ_qa per audit_mode;
Target B = 4.4% canonical-shape match; quarantined-row veto
exercises end-to-end). Calibration substrate for eventual
scheduler.
- Per-branch controller latency 12.5 µs at chunk_size=4 → not
the bottleneck; Hermes witness fan-out is.
v8 consensus paper at ``docs/_source/merkle-agi-v8-consensus.rst``
(834 lines; 11 parts) closes the loop from single-validator
Proof-of-Upgrade to multi-validator BFT selection. Phase 1c
(branch-set persistence) remains proposed-not-opened.
== Coverage ==
Total tags after this commit: 54 (was 49; +5)
Tags by metric:
tests: 45 (+4 new prometheus + table rows)
fixture-rows: 2
db-rows: 3
db-where: 2 (new metric, both 92 claim-pack)
Files with tags:
docs/warrant-substrate-cookbook.md 32 (+5)
docs/calculator-test-patterns.md 8
docs/soft-hash-channel-analysis.md 5
docs/tickets/ticket-000006-bench-emergent... 4
docs/seven-point-program.md 3
docs/tickets/ticket-000035-prg-choice-phi-prg.md 2
== Verification ==
$ pytest tests/test_doc_counts.py -v
3 passed in 2.60s
$ pytest tests/ -q
2328 passed, 37 skipped in 106.06s
Fan-out follow-up to ``fc5ba50``. Two thrusts in one commit since
they exercise the same surface:
== Task 3: extend AUTOCOUNT with db-rows metric ==
New metric ``db-rows`` for tagging live SQLite row counts (alias
tables, claim-pack records, etc — operator state that drifted on
``30a9488`` and earlier). Target syntax::
<!--AUTOCOUNT:db-rows:citation_aliases-->74<!--/AUTOCOUNT-->
<!--AUTOCOUNT:db-rows:002.db:concept_relations-->1234<!--/AUTOCOUNT-->
Default shard: ``~/.arborist/shards/000.db`` (where the alias
tables live per ``arborist.cli._aliases_db_path``). Operator state
is graceful-skip semantics: when DB or table is absent (CI, fresh
checkout, sibling repo), the claim is logged as skipped and the
test still passes. Drift only fires when the DB IS present and
the count diverged.
Sentinel returns:
- ``_DB_MISSING`` (-2): shards dir not present → skip
- ``_TABLE_MISSING`` (-3): DB present but table absent → skip
- ``_DB_ERROR`` (-4): malformed table name or sqlite error → skip
Table name validated against ``[A-Za-z_][A-Za-z0-9_]*`` regex
before string-interpolating into ``SELECT COUNT(*) FROM <table>``;
this is belt-and-suspenders since AUTOCOUNT tags are author-
controlled, but the dynamic SQL surface deserves a bouncer.
Smoke verified under HOME redirect to ``/tmp/<empty>``: 3 db-rows
claims gracefully skip with informative line-numbered messages,
suite still passes.
== Task 2: backfill 15 tags ==
Cookbook test/code-density table (lines 569-579, 10 rows) — every
``(N tests)`` cell now machine-checked:
| aliases.py | 512 | 469 (28 tests) | 0.92 |
→
| aliases.py | 512 | 469 (<!--AUTOCOUNT:tests:tests/test_aliases.py-->28<!--/AUTOCOUNT--> tests) | 0.92 |
Markdown renderers strip HTML comments — table cells display
``28 tests`` unchanged. The ``warrant_resolver.py`` row stays
untagged because its test count is split across two test files
(verifier + parser) and the cell encodes a combined "~430"
instead of one collected count.
Cookbook alias-count surfaces (3 db-rows tags):
- L364 ``citation_aliases (74 rows live as of 2026-05-10)``
- L437 ``#000041 — citation-aliases table + 74 live rows``
- L438 ``#000042 — term-aliases table + 13 live rows``
Ticket #000035 (in progress, line 274) — refresh ``20 tests``
→ ``27 tests`` for ``test_anchor_prg.py`` + tag. Same drift
pattern as ``5c21e83``: ticket prose was written before the
``de997f7`` 2026-05-10 pattern backfill that added 7 tests
(prefix-extension closure, hand-formula, parametrized
invalid-input cones). Also tagged ``L279``'s 10-vector KAT
fixture claim with ``fixture-rows`` metric.
== Closed-ticket counts deliberately not tagged ==
#000028, #000030, #000042, #000031, #000004, #000026, #000009,
#000032, #000008 all carry historical "N tests pass" snapshots
from their landing date. Those are point-in-time records, not
live claims — drifting from current state is BY DESIGN. Tagging
them would fire the test on every successive change to the
codebase. Closed tickets are the design log; we don't backfill
them.
== Coverage summary ==
Total tags after this commit: 44 (was 29; +15)
Tags by metric:
tests: 39
fixture-rows: 2
db-rows: 3
Files with tags:
docs/warrant-substrate-cookbook.md 27 (was 14)
docs/soft-hash-channel-analysis.md 5
docs/tickets/ticket-000006-bench-emergent... 4
docs/seven-point-program.md 3
docs/calculator-test-patterns.md 3
docs/tickets/ticket-000035-prg-choice-phi-prg.md 2 (new)
== Verification ==
$ .venv/bin/pytest tests/test_doc_counts.py -v
3 passed in 4.32s
$ .venv/bin/pytest -q
2276 passed, 54 skipped in 168.34s
$ HOME=/tmp/empty pytest tests/test_doc_counts.py -v -s
3 db-rows AUTOCOUNT claim(s) skipped:
docs/warrant-substrate-cookbook.md:364 db-rows:citation_aliases skipped — /tmp/empty/.arborist/shards not present (CI / fresh checkout)
docs/warrant-substrate-cookbook.md:437 db-rows:citation_aliases skipped — /tmp/empty/.arborist/shards not present (CI / fresh checkout)
docs/warrant-substrate-cookbook.md:438 db-rows:term_aliases skipped — /tmp/empty/.arborist/shards not present (CI / fresh checkout)
3 passed in 4.78s
No new dependencies. No schema changes.
The doc-drift pattern recurred four times today on 2026-05-10
(commits 6cbbf95, 14bcb99, 5c21e83, 30a9488). Each fix was the
same shape: walk a doc, find a count that drifted from live truth
during the hours after the doc was written, refresh it. Cost: ~5
min per drift × 4 = 20 min of manual catching, with no guarantee
the next drift gets caught before someone external reads it.
Per fox's selection: regression test that makes drift loud at
test time instead of relying on visual catching.
== Mechanism ==
`tests/test_doc_counts.py` scans `docs/**/*.md` for AUTOCOUNT
tags of the form:
<!--AUTOCOUNT:metric:path-->N<!--/AUTOCOUNT-->
Two metrics supported:
- `tests` — pytest collected count for path. Batches every
tagged path into one `pytest --collect-only` subprocess
(~0.5s total).
- `fixture-rows` — non-blank-non-comment line count in a JSONL
fixture.
GitHub and most markdown renderers strip HTML comments, so
readers see only `N`. The tags are invisible in rendered output
but make the claim machine-checkable. Three tests in the file:
1. `test_doc_autocount_claims_match_live` — the core invariant
2. `test_autocount_tags_are_well_formed` — open/close balance
3. `test_autocount_metric_names_are_documented` — fail-closed on
undocumented metrics (catches typos)
Failure message names the doc file, line number, and the
claimed-vs-live diff. Example:
`docs/foo.md:42 AUTOCOUNT(tests:tests/test_x.py) claims 23, live is 27`
== 29 tags installed across 5 docs ==
While installing tags I had to read the surrounding prose, which
surfaced six stale counts that had drifted same-day:
`docs/soft-hash-channel-analysis.md`:
- L392 14 → 23 tests for phi_alignment_probe
- L417 20 → 27 tests for anchor_prg
- L463 14 → 23 tests for phi_alignment_probe (status section)
`docs/seven-point-program.md`:
- L77 68 → 58 tests for metacognition (drift -10; the file
shed tests during a refactor and the doc didn't catch up)
- L78 9 tests for `test_dag.py::test_preflight_*` — removed
count entirely; pytest selector subsets aren't currently
supported by the AUTOCOUNT metric set (would need a
`tests-matching` metric; not worth the surface for one claim).
- L110 24 → 33 tests for test_dag.py
`docs/calculator-test-patterns.md`:
- L35 33 → 23 tests for warrant_resolver
- L35 10 → 9 tests for warrant_chain
- L16, L265 51 → 53 tests for t3_bound_calculator (kept
initial-shipment provenance in prose)
== Coverage installed ==
calculator-test-patterns.md 3 tagged claims
soft-hash-channel-analysis.md 5 tagged claims
warrant-substrate-cookbook.md 14 tagged claims
seven-point-program.md 3 tagged claims
tickets/ticket-000006-bench-... 4 tagged claims
---
29 tagged claims
Every count that drifted today is now tagged. Future drift
fires the regression test at the next pytest run instead of
waiting for human catching.
== Discipline pattern ==
Walk this pattern for any new doc that names a count:
1. Surround the number with the tag pair:
`<!--AUTOCOUNT:tests:tests/test_foo.py-->N<!--/AUTOCOUNT-->`
2. Run `pytest tests/test_doc_counts.py` (~3.5s)
3. If it passes, the claim is now machine-verified
Aim to tag counts on first authorship. Retrofitting is cheap
but only catches drift after the fact.
== Out of scope ==
Test counts inside source code (docstrings, CLI --help) are not
scanned — would expand the test surface significantly and the
drift pattern hasn't manifested there. Add `**/*.py` scope when
that pattern surfaces.
Alias-row counts and claim-pack-record counts could be tagged
with new `db-rows:<table>` and `db-where:<sql>` metrics; deferred
until the next drift on those numbers (none caught today after
30a9488's cookbook refresh).
== Verification ==
$ .venv/bin/pytest tests/test_doc_counts.py -v
3 passed in 3.89s
$ .venv/bin/pytest -q
2276 passed, 54 skipped in 153.21s
No new dependencies. No schema changes. No source-code changes.
