test_doc_counts: AUTOCOUNT db-where supports *: glob for corpus-wide claims

The three "92 claim_pack docs" tags were drifting against shard 000.db's
21 rows because the harness only counted one shard, but the doc prose
("#000031 closed at 92") meant the corpus total (21+16+38+17 across
genesis shards 000-003).

Two-line fix path: extend the harness to sum across all ???.db shards
via a `*:` prefix (e.g. `*:documents?source_type=claim_pack`), then
prefix the three drifted tags. Aligns the harness scope with the
semantic scope of the claim instead of forcing the claim to shrink to
one shard.

The `*:` glob:
  - Matches `[0-9][0-9][0-9].db` basenames only (operator sidecars
    qa.db / snapshots.db / selfmodel-chain.db skipped)
  - Skips shards lacking the named table (schema-version tolerance)
  - Returns _DB_MISSING when no genesis shard exists (CI / fresh-
    checkout skip semantic preserved)
  - Returns _TABLE_MISSING when no contributing shard has the table

Documented in ticket-000044 §3.4 + a third example showing the new
syntax. Diagnosis credit to a sub-agent investigation that confirmed
zero eviction/falsification audit events on claim_packs — the data is
intact; the harness was just single-shard.
This commit is contained in:
russell@unturf.com 2026-05-31 11:37:39 -04:00
parent 18db3c2caf
commit d8469613ce
No known key found for this signature in database
4 changed files with 69 additions and 6 deletions

View file

@ -1030,7 +1030,7 @@ These are the right guardrails for the pure-function layer.
### 17.1 Warrant-promotion integration (#000031)
#000031 closed at <!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
#000031 closed at <!--AUTOCOUNT:db-where:*:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
atomic claim-pack records (started at 78, reached 92 by 2026-05-09
under Phase 2.5). All 92 records now resolve to at least one
textbook substrate via the 74 citation-aliases (#000041) + 13

View file

@ -115,8 +115,9 @@ query. SQLite connection opens `mode=ro` URI.
### 3.4 `db-where`
```markdown
<!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
<!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->21<!--/AUTOCOUNT-->
<!--AUTOCOUNT:db-where:001.db:documents?source_type=wikipedia_xml-->867695<!--/AUTOCOUNT-->
<!--AUTOCOUNT:db-where:*:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
```
Live value: `SELECT COUNT(*) FROM <table> WHERE <column> = ?`
@ -124,6 +125,16 @@ with `<value>` bound as a SQL parameter (not string-interpolated).
Single-column equality predicate only — anything more complex is
out of scope for v1 (write a custom assertion test instead).
The `*:` prefix sums across every `???.db` shard (basename glob
`[0-9][0-9][0-9].db`) in `~/.arborist/shards/`. Use this when the
claim is corpus-wide rather than scoped to one shard — e.g. "92
claim_pack docs across the corpus" (21+16+38+17 across genesis
shards 000-003). Operator sidecar dbs (qa.db, snapshots.db,
selfmodel-chain.db) are skipped. Shards lacking the named table
contribute zero (different schema versions don't fail the assert);
if NO shard has the table, the metric returns `_TABLE_MISSING`
(skipped, like the single-shard case).
---
## 4. Skip semantics (CI / fresh-checkout safety)

View file

@ -34,7 +34,7 @@ patterns that keep the chains audit-clean.
## Pillar-by-pillar substrate map
Eight pillars in the claim-pack source (`#000029`);
<!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
<!--AUTOCOUNT:db-where:*:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
records total. As of 2026-05-10, every record resolves to at
least one substrate in `~/.arborist/crawl/textbook_*.db`.
@ -346,7 +346,7 @@ Full cascade order, tightest → loosest:
## Honest tier breakdown
The <!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT--> chains have three quality tiers:
The <!--AUTOCOUNT:db-where:*:documents?source_type=claim_pack-->92<!--/AUTOCOUNT--> chains have three quality tiers:
- **Direct primary (~25 records)** — citation matches the literal
textbook ingested. Hilbert pillar IV (18), Goldstein → Newton

View file

@ -173,8 +173,14 @@ def _live_db_where(target: str) -> int:
Target syntax::
<table>?<column>=<value>
<shard>:<table>?<column>=<value>
<table>?<column>=<value> # default shard 000.db
<shard>:<table>?<column>=<value> # one named shard
*:<table>?<column>=<value> # sum across ALL ???.db shards
The ``*`` form is for claims that span the corpus (e.g. "92
claim_pack docs" is 21+16+38+17 across genesis shards 000-003).
Globs match basenames ``[0-9][0-9][0-9].db`` only — operator
sidecar dbs (qa.db, snapshots.db) aren't summed.
Same sentinel returns as ``_live_db_rows``. ``<value>`` is bound
as a SQL parameter (no string interpolation), so even author-
@ -195,6 +201,8 @@ def _live_db_where(target: str) -> int:
if ":" in head:
shard_name, table = head.split(":", 1)
if shard_name == "*":
return _live_db_where_sum_shards(table, column, value)
db = _DEFAULT_SHARDS_DIR / shard_name
else:
db = _DEFAULT_SHARD
@ -225,6 +233,50 @@ def _live_db_where(target: str) -> int:
return _DB_ERROR
def _live_db_where_sum_shards(table: str, column: str, value: str) -> int:
"""Sum rows matching ``<column>=<value>`` across all genesis shards
(``???.db`` basename pattern) in ``_DEFAULT_SHARDS_DIR``.
Skips shards where the table is absent (different schema versions);
returns _DB_MISSING when no shard matches the glob at all (operator
state, not source state). Identifier validation matches the single-
shard path.
"""
import sqlite3
if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", table):
return _DB_ERROR
shards = sorted(_DEFAULT_SHARDS_DIR.glob("[0-9][0-9][0-9].db"))
if not shards:
return _DB_MISSING
total = 0
saw_any_table = False
for db in shards:
try:
c = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
try:
row = c.execute(
f"SELECT COUNT(*) FROM {table} WHERE {column} = ?",
(value,),
).fetchone()
if row:
total += int(row[0])
saw_any_table = True
finally:
c.close()
except sqlite3.OperationalError as exc:
if "no such table" in str(exc).lower():
continue # shard predates the table — skip, don't fail
if "no such column" in str(exc).lower():
return _DB_ERROR
return _DB_ERROR
except sqlite3.Error:
return _DB_ERROR
if not saw_any_table:
return _TABLE_MISSING
return total
def _strip_fenced_code_blocks(text: str) -> str:
"""Replace contents of triple-backtick fenced code blocks with newlines.