tests/doc_counts: extend with db-rows metric + backfill 15 tags (cookbook table + #000035)
Fan-out follow-up to ``fc5ba50``. Two thrusts in one commit since they exercise the same surface: == Task 3: extend AUTOCOUNT with db-rows metric == New metric ``db-rows`` for tagging live SQLite row counts (alias tables, claim-pack records, etc — operator state that drifted on ``30a9488`` and earlier). Target syntax:: <!--AUTOCOUNT:db-rows:citation_aliases-->74<!--/AUTOCOUNT--> <!--AUTOCOUNT:db-rows:002.db:concept_relations-->1234<!--/AUTOCOUNT--> Default shard: ``~/.arborist/shards/000.db`` (where the alias tables live per ``arborist.cli._aliases_db_path``). Operator state is graceful-skip semantics: when DB or table is absent (CI, fresh checkout, sibling repo), the claim is logged as skipped and the test still passes. Drift only fires when the DB IS present and the count diverged. Sentinel returns: - ``_DB_MISSING`` (-2): shards dir not present → skip - ``_TABLE_MISSING`` (-3): DB present but table absent → skip - ``_DB_ERROR`` (-4): malformed table name or sqlite error → skip Table name validated against ``[A-Za-z_][A-Za-z0-9_]*`` regex before string-interpolating into ``SELECT COUNT(*) FROM <table>``; this is belt-and-suspenders since AUTOCOUNT tags are author- controlled, but the dynamic SQL surface deserves a bouncer. Smoke verified under HOME redirect to ``/tmp/<empty>``: 3 db-rows claims gracefully skip with informative line-numbered messages, suite still passes. == Task 2: backfill 15 tags == Cookbook test/code-density table (lines 569-579, 10 rows) — every ``(N tests)`` cell now machine-checked: | aliases.py | 512 | 469 (28 tests) | 0.92 | → | aliases.py | 512 | 469 (<!--AUTOCOUNT:tests:tests/test_aliases.py-->28<!--/AUTOCOUNT--> tests) | 0.92 | Markdown renderers strip HTML comments — table cells display ``28 tests`` unchanged. The ``warrant_resolver.py`` row stays untagged because its test count is split across two test files (verifier + parser) and the cell encodes a combined "~430" instead of one collected count. Cookbook alias-count surfaces (3 db-rows tags): - L364 ``citation_aliases (74 rows live as of 2026-05-10)`` - L437 ``#000041 — citation-aliases table + 74 live rows`` - L438 ``#000042 — term-aliases table + 13 live rows`` Ticket #000035 (in progress, line 274) — refresh ``20 tests`` → ``27 tests`` for ``test_anchor_prg.py`` + tag. Same drift pattern as ``5c21e83``: ticket prose was written before the ``de997f7`` 2026-05-10 pattern backfill that added 7 tests (prefix-extension closure, hand-formula, parametrized invalid-input cones). Also tagged ``L279``'s 10-vector KAT fixture claim with ``fixture-rows`` metric. == Closed-ticket counts deliberately not tagged == #000028, #000030, #000042, #000031, #000004, #000026, #000009, #000032, #000008 all carry historical "N tests pass" snapshots from their landing date. Those are point-in-time records, not live claims — drifting from current state is BY DESIGN. Tagging them would fire the test on every successive change to the codebase. Closed tickets are the design log; we don't backfill them. == Coverage summary == Total tags after this commit: 44 (was 29; +15) Tags by metric: tests: 39 fixture-rows: 2 db-rows: 3 Files with tags: docs/warrant-substrate-cookbook.md 27 (was 14) docs/soft-hash-channel-analysis.md 5 docs/tickets/ticket-000006-bench-emergent... 4 docs/seven-point-program.md 3 docs/calculator-test-patterns.md 3 docs/tickets/ticket-000035-prg-choice-phi-prg.md 2 (new) == Verification == $ .venv/bin/pytest tests/test_doc_counts.py -v 3 passed in 4.32s $ .venv/bin/pytest -q 2276 passed, 54 skipped in 168.34s $ HOME=/tmp/empty pytest tests/test_doc_counts.py -v -s 3 db-rows AUTOCOUNT claim(s) skipped: docs/warrant-substrate-cookbook.md:364 db-rows:citation_aliases skipped — /tmp/empty/.arborist/shards not present (CI / fresh checkout) docs/warrant-substrate-cookbook.md:437 db-rows:citation_aliases skipped — /tmp/empty/.arborist/shards not present (CI / fresh checkout) docs/warrant-substrate-cookbook.md:438 db-rows:term_aliases skipped — /tmp/empty/.arborist/shards not present (CI / fresh checkout) 3 passed in 4.78s No new dependencies. No schema changes.
This commit is contained in:
parent
fc5ba507dc
commit
03c0f6a6d5
3 changed files with 126 additions and 25 deletions
|
|
@ -271,12 +271,16 @@ the moment v7 needs it.
|
|||
from any version number. Acceptance-criteria §5 item 1 reads
|
||||
through to "or equivalent location" so the move doesn't
|
||||
invalidate the original criterion.
|
||||
- ``tests/test_anchor_prg.py`` — 20 tests covering determinism,
|
||||
- ``tests/test_anchor_prg.py`` — <!--AUTOCOUNT:tests:tests/test_anchor_prg.py-->27<!--/AUTOCOUNT--> tests covering determinism,
|
||||
range invariants, chi² loose-uniformity sanity, dim_h boundary
|
||||
(1, 16384), seed-bit-flip avalanche, hash-bit-flip avalanche,
|
||||
input validation (short/long/non-bytes hashes; non-positive /
|
||||
non-int dim_h), module export shape, and KAT regression.
|
||||
- ``bench/fixtures/phi-prg/known-answer-tests.jsonl`` — 10 pinned
|
||||
hand-computed HMAC-SHA-512 first-block formula, prefix-extension
|
||||
closure, parametrized invalid-input cones, input validation
|
||||
(short/long/non-bytes hashes; non-positive / non-int dim_h),
|
||||
module export shape, and KAT regression. (Was 20; +7 from
|
||||
``de997f7`` 2026-05-10 pattern backfill per
|
||||
``docs/calculator-test-patterns.md``.)
|
||||
- ``bench/fixtures/phi-prg/known-answer-tests.jsonl`` — <!--AUTOCOUNT:fixture-rows:bench/fixtures/phi-prg/known-answer-tests.jsonl-->10<!--/AUTOCOUNT--> pinned
|
||||
KAT vectors covering the placeholder seed (smoke), block-boundary
|
||||
cases (dim_h=16 = exactly one HMAC-SHA-512 block; dim_h=17 = two
|
||||
blocks with truncation), seed/hash one-bit-flip variants, and a
|
||||
|
|
|
|||
|
|
@ -361,7 +361,7 @@ The 92 chains have three quality tiers:
|
|||
*proves* the axiom is a separate, weaker claim. Heuristic-grade.
|
||||
|
||||
The schema doesn't currently distinguish these tiers. Audit
|
||||
reviewers walk `citation_aliases` (74 rows live as of 2026-05-10) + read
|
||||
reviewers walk `citation_aliases` (<!--AUTOCOUNT:db-rows:citation_aliases-->74<!--/AUTOCOUNT--> rows live as of 2026-05-10) + read
|
||||
`decision_rationale` fields to grade each chain. If tier
|
||||
distinction becomes load-bearing, add a `tier ∈ {primary,
|
||||
substitute, soft}` column to `derivations`.
|
||||
|
|
@ -434,8 +434,8 @@ arborist warrant-resolve --use-aliases --write
|
|||
## References
|
||||
|
||||
- `#000031` — surface-ingest cited textbooks
|
||||
- `#000041` — citation-aliases table + 74 live rows
|
||||
- `#000042` — term-aliases table + 13 live rows (5 arithmetic + 4 classical-physics + 4 geometry)
|
||||
- `#000041` — citation-aliases table + <!--AUTOCOUNT:db-rows:citation_aliases-->74<!--/AUTOCOUNT--> live rows
|
||||
- `#000042` — term-aliases table + <!--AUTOCOUNT:db-rows:term_aliases-->13<!--/AUTOCOUNT--> live rows (5 arithmetic + 4 classical-physics + 4 geometry)
|
||||
- `#000038` — Phase 4 content acquisition (still-blocked
|
||||
proprietary candidates: Hilbert-Ackermann 1928, Mendelson,
|
||||
Enderton, Jech, Goldstein, Barendregt, Stanley, Brualdi, Knuth;
|
||||
|
|
@ -566,17 +566,17 @@ than waiting for bench-time STRICT-rate drift to surface it.
|
|||
|
||||
| domain | LOC tested | test LOC | test/code ratio |
|
||||
|---|---|---|---|
|
||||
| aliases.py | 512 | 469 (28 tests) | 0.92 |
|
||||
| aliases.py | 512 | 469 (<!--AUTOCOUNT:tests:tests/test_aliases.py-->28<!--/AUTOCOUNT--> tests) | 0.92 |
|
||||
| warrant_resolver.py | ~800 | ~430 (combined) | 0.54 |
|
||||
| warrant_chain.py | 89 | 320 (9 tests) | 3.6 |
|
||||
| t3_bound_calculator.py | 249 | 446 (53 tests) | 1.79 |
|
||||
| fork_score.py | 298 | 403 (18 tests) | 1.35 |
|
||||
| weights.py | 73 | 180 (16 tests) | 2.5 |
|
||||
| pi_star/protocol+registry | 124 | 280 (21 tests) | 2.3 |
|
||||
| qa/progress.py | 85 | 226 (31 tests) | 2.7 |
|
||||
| qa/prompts.py | 153 | 207 (20 tests) | 1.35 |
|
||||
| concepts/extract.py | 288 | 278 (20 tests) | 0.97 |
|
||||
| textbooks_manifest.py | 236 | 463 (43 tests) | 1.96 |
|
||||
| warrant_chain.py | 89 | 320 (<!--AUTOCOUNT:tests:tests/test_warrant_chain.py-->9<!--/AUTOCOUNT--> tests) | 3.6 |
|
||||
| t3_bound_calculator.py | 249 | 446 (<!--AUTOCOUNT:tests:tests/test_t3_bound_calculator.py-->53<!--/AUTOCOUNT--> tests) | 1.79 |
|
||||
| fork_score.py | 298 | 403 (<!--AUTOCOUNT:tests:tests/test_fork_score.py-->18<!--/AUTOCOUNT--> tests) | 1.35 |
|
||||
| weights.py | 73 | 180 (<!--AUTOCOUNT:tests:tests/test_weights.py-->16<!--/AUTOCOUNT--> tests) | 2.5 |
|
||||
| pi_star/protocol+registry | 124 | 280 (<!--AUTOCOUNT:tests:tests/test_pi_star_protocol_and_registry.py-->21<!--/AUTOCOUNT--> tests) | 2.3 |
|
||||
| qa/progress.py | 85 | 226 (<!--AUTOCOUNT:tests:tests/test_qa_progress.py-->31<!--/AUTOCOUNT--> tests) | 2.7 |
|
||||
| qa/prompts.py | 153 | 207 (<!--AUTOCOUNT:tests:tests/test_qa_prompts.py-->20<!--/AUTOCOUNT--> tests) | 1.35 |
|
||||
| concepts/extract.py | 288 | 278 (<!--AUTOCOUNT:tests:tests/test_concepts_extract.py-->20<!--/AUTOCOUNT--> tests) | 0.97 |
|
||||
| textbooks_manifest.py | 236 | 463 (<!--AUTOCOUNT:tests:tests/test_textbooks_manifest.py-->43<!--/AUTOCOUNT--> tests) | 1.96 |
|
||||
|
||||
Pattern: small contract-defining modules (weights, prompts,
|
||||
warrant_chain) get 1.4-3.6× test LOC because they're foundation;
|
||||
|
|
|
|||
|
|
@ -21,6 +21,12 @@ Where ``metric`` is one of:
|
|||
live value is ``pytest --collect-only`` count.
|
||||
- ``fixture-rows`` — ``path`` is a JSONL fixture; the live value
|
||||
is the non-blank-non-comment line count.
|
||||
- ``db-rows`` — ``path`` is either ``<table>`` (resolves against
|
||||
the default shard ``~/.arborist/shards/000.db``) or
|
||||
``<shard>:<table>`` (resolves against ``~/.arborist/shards/<shard>``).
|
||||
When the DB or table isn't present (CI or fresh checkout), the
|
||||
claim is skipped with a logged note rather than failing — claim-
|
||||
pack and alias counts are operator state, not source-state.
|
||||
|
||||
GitHub and most markdown renderers strip HTML comments, so readers
|
||||
see only ``N``. The tags are invisible in rendered output but
|
||||
|
|
@ -95,6 +101,62 @@ def _live_fixture_rows(path: str) -> int:
|
|||
)
|
||||
|
||||
|
||||
_DB_MISSING = -2
|
||||
_TABLE_MISSING = -3
|
||||
_DB_ERROR = -4
|
||||
|
||||
# Default location for arborist alias / claim-pack tables. Operator state,
|
||||
# not source state — absence is not a failure, it's a skip signal.
|
||||
_DEFAULT_SHARDS_DIR = Path.home() / ".arborist" / "shards"
|
||||
_DEFAULT_SHARD = _DEFAULT_SHARDS_DIR / "000.db"
|
||||
|
||||
|
||||
def _live_db_rows(target: str) -> int:
|
||||
"""Count rows in a SQLite table.
|
||||
|
||||
Target syntax::
|
||||
|
||||
<table> # ~/.arborist/shards/000.db (default)
|
||||
<shard>:<table> # ~/.arborist/shards/<shard>
|
||||
|
||||
Sentinel returns:
|
||||
_DB_MISSING (-2) DB file not present — caller marks skipped
|
||||
_TABLE_MISSING (-3) DB present but table absent — caller marks skipped
|
||||
_DB_ERROR (-4) sqlite3 error — caller marks skipped
|
||||
N >= 0 live row count
|
||||
"""
|
||||
import sqlite3
|
||||
|
||||
if ":" in target:
|
||||
shard_name, table = target.split(":", 1)
|
||||
db = _DEFAULT_SHARDS_DIR / shard_name
|
||||
else:
|
||||
db = _DEFAULT_SHARD
|
||||
table = target
|
||||
if not db.exists():
|
||||
return _DB_MISSING
|
||||
# Validate table name is a bare identifier — defends against the
|
||||
# dynamic SQL string interpolation below. Tags are author-controlled
|
||||
# but this is belt-and-suspenders.
|
||||
if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", table):
|
||||
return _DB_ERROR
|
||||
try:
|
||||
c = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||
try:
|
||||
row = c.execute(
|
||||
f"SELECT COUNT(*) FROM {table}" # noqa: S608 — validated above
|
||||
).fetchone()
|
||||
return int(row[0]) if row else 0
|
||||
finally:
|
||||
c.close()
|
||||
except sqlite3.OperationalError as exc:
|
||||
if "no such table" in str(exc).lower():
|
||||
return _TABLE_MISSING
|
||||
return _DB_ERROR
|
||||
except sqlite3.Error:
|
||||
return _DB_ERROR
|
||||
|
||||
|
||||
def _iter_claims() -> Iterator[tuple[Path, int, str, str, int]]:
|
||||
"""Yield (doc_path, lineno, metric, target_path, claimed_n)."""
|
||||
for md in sorted(DOCS_DIR.rglob("*.md")):
|
||||
|
|
@ -110,7 +172,7 @@ def _iter_claims() -> Iterator[tuple[Path, int, str, str, int]]:
|
|||
)
|
||||
|
||||
|
||||
def test_doc_autocount_claims_match_live() -> None:
|
||||
def test_doc_autocount_claims_match_live(capsys: pytest.CaptureFixture[str]) -> None:
|
||||
"""Every ``<!--AUTOCOUNT:-->`` claim in ``docs/`` matches live."""
|
||||
claims = list(_iter_claims())
|
||||
# Batch the pytest-collect call across every tests:* claim — one
|
||||
|
|
@ -119,25 +181,60 @@ def test_doc_autocount_claims_match_live() -> None:
|
|||
test_counts = _live_test_count_batch(test_paths)
|
||||
|
||||
drifts: list[str] = []
|
||||
skipped: list[str] = []
|
||||
for doc, lineno, metric, target, claimed in claims:
|
||||
rel = doc.relative_to(REPO_ROOT)
|
||||
if metric == "tests":
|
||||
live = test_counts.get(target, -1)
|
||||
if live < 0:
|
||||
drifts.append(
|
||||
f"{rel}:{lineno} AUTOCOUNT({metric}:{target}) target "
|
||||
f"missing or uncollectable"
|
||||
)
|
||||
continue
|
||||
elif metric == "fixture-rows":
|
||||
live = _live_fixture_rows(target)
|
||||
if live < 0:
|
||||
drifts.append(
|
||||
f"{rel}:{lineno} AUTOCOUNT({metric}:{target}) target "
|
||||
f"missing or uncollectable"
|
||||
)
|
||||
continue
|
||||
elif metric == "db-rows":
|
||||
live = _live_db_rows(target)
|
||||
if live == _DB_MISSING:
|
||||
skipped.append(
|
||||
f"{rel}:{lineno} db-rows:{target} skipped — "
|
||||
f"{_DEFAULT_SHARDS_DIR} not present (CI / fresh checkout)"
|
||||
)
|
||||
continue
|
||||
if live == _TABLE_MISSING:
|
||||
skipped.append(
|
||||
f"{rel}:{lineno} db-rows:{target} skipped — table not "
|
||||
f"present in shard"
|
||||
)
|
||||
continue
|
||||
if live == _DB_ERROR:
|
||||
skipped.append(
|
||||
f"{rel}:{lineno} db-rows:{target} skipped — sqlite "
|
||||
f"error or invalid table name"
|
||||
)
|
||||
continue
|
||||
else:
|
||||
drifts.append(f"{rel}:{lineno} unknown AUTOCOUNT metric {metric!r}")
|
||||
continue
|
||||
if live < 0:
|
||||
drifts.append(
|
||||
f"{rel}:{lineno} AUTOCOUNT({metric}:{target}) target "
|
||||
f"missing or uncollectable"
|
||||
)
|
||||
elif live != claimed:
|
||||
if live != claimed:
|
||||
drifts.append(
|
||||
f"{rel}:{lineno} AUTOCOUNT({metric}:{target}) claims "
|
||||
f"{claimed}, live is {live}"
|
||||
)
|
||||
if skipped:
|
||||
# Print to captured stdout — pytest -v shows it; the suite still
|
||||
# passes as long as no live count actually drifted.
|
||||
with capsys.disabled():
|
||||
print(f"\n{len(skipped)} db-rows AUTOCOUNT claim(s) skipped:")
|
||||
for s in skipped:
|
||||
print(f" {s}")
|
||||
assert not drifts, "Doc count drift detected:\n " + "\n ".join(drifts)
|
||||
|
||||
|
||||
|
|
@ -168,7 +265,7 @@ def test_autocount_tags_are_well_formed() -> None:
|
|||
|
||||
def test_autocount_metric_names_are_documented() -> None:
|
||||
"""Fail-closed if a doc uses an undocumented metric."""
|
||||
known = {"tests", "fixture-rows"}
|
||||
known = {"tests", "fixture-rows", "db-rows"}
|
||||
seen: set[str] = set()
|
||||
for _, _, metric, _, _ in _iter_claims():
|
||||
seen.add(metric)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue