test_doc_counts: AUTOCOUNT db-where supports *: glob for corpus-wide claims
The three "92 claim_pack docs" tags were drifting against shard 000.db's
21 rows because the harness only counted one shard, but the doc prose
("#000031 closed at 92") meant the corpus total (21+16+38+17 across
genesis shards 000-003).
Two-line fix path: extend the harness to sum across all ???.db shards
via a `*:` prefix (e.g. `*:documents?source_type=claim_pack`), then
prefix the three drifted tags. Aligns the harness scope with the
semantic scope of the claim instead of forcing the claim to shrink to
one shard.
The `*:` glob:
- Matches `[0-9][0-9][0-9].db` basenames only (operator sidecars
qa.db / snapshots.db / selfmodel-chain.db skipped)
- Skips shards lacking the named table (schema-version tolerance)
- Returns _DB_MISSING when no genesis shard exists (CI / fresh-
checkout skip semantic preserved)
- Returns _TABLE_MISSING when no contributing shard has the table
Documented in ticket-000044 §3.4 + a third example showing the new
syntax. Diagnosis credit to a sub-agent investigation that confirmed
zero eviction/falsification audit events on claim_packs — the data is
intact; the harness was just single-shard.
This commit is contained in:
parent
18db3c2caf
commit
d8469613ce
4 changed files with 69 additions and 6 deletions
|
|
@ -1030,7 +1030,7 @@ These are the right guardrails for the pure-function layer.
|
||||||
|
|
||||||
### 17.1 Warrant-promotion integration (#000031)
|
### 17.1 Warrant-promotion integration (#000031)
|
||||||
|
|
||||||
#000031 closed at <!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
|
#000031 closed at <!--AUTOCOUNT:db-where:*:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
|
||||||
atomic claim-pack records (started at 78, reached 92 by 2026-05-09
|
atomic claim-pack records (started at 78, reached 92 by 2026-05-09
|
||||||
under Phase 2.5). All 92 records now resolve to at least one
|
under Phase 2.5). All 92 records now resolve to at least one
|
||||||
textbook substrate via the 74 citation-aliases (#000041) + 13
|
textbook substrate via the 74 citation-aliases (#000041) + 13
|
||||||
|
|
|
||||||
|
|
@ -115,8 +115,9 @@ query. SQLite connection opens `mode=ro` URI.
|
||||||
### 3.4 `db-where`
|
### 3.4 `db-where`
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
<!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
|
<!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->21<!--/AUTOCOUNT-->
|
||||||
<!--AUTOCOUNT:db-where:001.db:documents?source_type=wikipedia_xml-->867695<!--/AUTOCOUNT-->
|
<!--AUTOCOUNT:db-where:001.db:documents?source_type=wikipedia_xml-->867695<!--/AUTOCOUNT-->
|
||||||
|
<!--AUTOCOUNT:db-where:*:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
|
||||||
```
|
```
|
||||||
|
|
||||||
Live value: `SELECT COUNT(*) FROM <table> WHERE <column> = ?`
|
Live value: `SELECT COUNT(*) FROM <table> WHERE <column> = ?`
|
||||||
|
|
@ -124,6 +125,16 @@ with `<value>` bound as a SQL parameter (not string-interpolated).
|
||||||
Single-column equality predicate only — anything more complex is
|
Single-column equality predicate only — anything more complex is
|
||||||
out of scope for v1 (write a custom assertion test instead).
|
out of scope for v1 (write a custom assertion test instead).
|
||||||
|
|
||||||
|
The `*:` prefix sums across every `???.db` shard (basename glob
|
||||||
|
`[0-9][0-9][0-9].db`) in `~/.arborist/shards/`. Use this when the
|
||||||
|
claim is corpus-wide rather than scoped to one shard — e.g. "92
|
||||||
|
claim_pack docs across the corpus" (21+16+38+17 across genesis
|
||||||
|
shards 000-003). Operator sidecar dbs (qa.db, snapshots.db,
|
||||||
|
selfmodel-chain.db) are skipped. Shards lacking the named table
|
||||||
|
contribute zero (different schema versions don't fail the assert);
|
||||||
|
if NO shard has the table, the metric returns `_TABLE_MISSING`
|
||||||
|
(skipped, like the single-shard case).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Skip semantics (CI / fresh-checkout safety)
|
## 4. Skip semantics (CI / fresh-checkout safety)
|
||||||
|
|
|
||||||
|
|
@ -34,7 +34,7 @@ patterns that keep the chains audit-clean.
|
||||||
## Pillar-by-pillar substrate map
|
## Pillar-by-pillar substrate map
|
||||||
|
|
||||||
Eight pillars in the claim-pack source (`#000029`);
|
Eight pillars in the claim-pack source (`#000029`);
|
||||||
<!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
|
<!--AUTOCOUNT:db-where:*:documents?source_type=claim_pack-->92<!--/AUTOCOUNT-->
|
||||||
records total. As of 2026-05-10, every record resolves to at
|
records total. As of 2026-05-10, every record resolves to at
|
||||||
least one substrate in `~/.arborist/crawl/textbook_*.db`.
|
least one substrate in `~/.arborist/crawl/textbook_*.db`.
|
||||||
|
|
||||||
|
|
@ -346,7 +346,7 @@ Full cascade order, tightest → loosest:
|
||||||
|
|
||||||
## Honest tier breakdown
|
## Honest tier breakdown
|
||||||
|
|
||||||
The <!--AUTOCOUNT:db-where:documents?source_type=claim_pack-->92<!--/AUTOCOUNT--> chains have three quality tiers:
|
The <!--AUTOCOUNT:db-where:*:documents?source_type=claim_pack-->92<!--/AUTOCOUNT--> chains have three quality tiers:
|
||||||
|
|
||||||
- **Direct primary (~25 records)** — citation matches the literal
|
- **Direct primary (~25 records)** — citation matches the literal
|
||||||
textbook ingested. Hilbert pillar IV (18), Goldstein → Newton
|
textbook ingested. Hilbert pillar IV (18), Goldstein → Newton
|
||||||
|
|
|
||||||
|
|
@ -173,8 +173,14 @@ def _live_db_where(target: str) -> int:
|
||||||
|
|
||||||
Target syntax::
|
Target syntax::
|
||||||
|
|
||||||
<table>?<column>=<value>
|
<table>?<column>=<value> # default shard 000.db
|
||||||
<shard>:<table>?<column>=<value>
|
<shard>:<table>?<column>=<value> # one named shard
|
||||||
|
*:<table>?<column>=<value> # sum across ALL ???.db shards
|
||||||
|
|
||||||
|
The ``*`` form is for claims that span the corpus (e.g. "92
|
||||||
|
claim_pack docs" is 21+16+38+17 across genesis shards 000-003).
|
||||||
|
Globs match basenames ``[0-9][0-9][0-9].db`` only — operator
|
||||||
|
sidecar dbs (qa.db, snapshots.db) aren't summed.
|
||||||
|
|
||||||
Same sentinel returns as ``_live_db_rows``. ``<value>`` is bound
|
Same sentinel returns as ``_live_db_rows``. ``<value>`` is bound
|
||||||
as a SQL parameter (no string interpolation), so even author-
|
as a SQL parameter (no string interpolation), so even author-
|
||||||
|
|
@ -195,6 +201,8 @@ def _live_db_where(target: str) -> int:
|
||||||
|
|
||||||
if ":" in head:
|
if ":" in head:
|
||||||
shard_name, table = head.split(":", 1)
|
shard_name, table = head.split(":", 1)
|
||||||
|
if shard_name == "*":
|
||||||
|
return _live_db_where_sum_shards(table, column, value)
|
||||||
db = _DEFAULT_SHARDS_DIR / shard_name
|
db = _DEFAULT_SHARDS_DIR / shard_name
|
||||||
else:
|
else:
|
||||||
db = _DEFAULT_SHARD
|
db = _DEFAULT_SHARD
|
||||||
|
|
@ -225,6 +233,50 @@ def _live_db_where(target: str) -> int:
|
||||||
return _DB_ERROR
|
return _DB_ERROR
|
||||||
|
|
||||||
|
|
||||||
|
def _live_db_where_sum_shards(table: str, column: str, value: str) -> int:
|
||||||
|
"""Sum rows matching ``<column>=<value>`` across all genesis shards
|
||||||
|
(``???.db`` basename pattern) in ``_DEFAULT_SHARDS_DIR``.
|
||||||
|
|
||||||
|
Skips shards where the table is absent (different schema versions);
|
||||||
|
returns _DB_MISSING when no shard matches the glob at all (operator
|
||||||
|
state, not source state). Identifier validation matches the single-
|
||||||
|
shard path.
|
||||||
|
"""
|
||||||
|
import sqlite3
|
||||||
|
|
||||||
|
if not re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*", table):
|
||||||
|
return _DB_ERROR
|
||||||
|
shards = sorted(_DEFAULT_SHARDS_DIR.glob("[0-9][0-9][0-9].db"))
|
||||||
|
if not shards:
|
||||||
|
return _DB_MISSING
|
||||||
|
total = 0
|
||||||
|
saw_any_table = False
|
||||||
|
for db in shards:
|
||||||
|
try:
|
||||||
|
c = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||||
|
try:
|
||||||
|
row = c.execute(
|
||||||
|
f"SELECT COUNT(*) FROM {table} WHERE {column} = ?",
|
||||||
|
(value,),
|
||||||
|
).fetchone()
|
||||||
|
if row:
|
||||||
|
total += int(row[0])
|
||||||
|
saw_any_table = True
|
||||||
|
finally:
|
||||||
|
c.close()
|
||||||
|
except sqlite3.OperationalError as exc:
|
||||||
|
if "no such table" in str(exc).lower():
|
||||||
|
continue # shard predates the table — skip, don't fail
|
||||||
|
if "no such column" in str(exc).lower():
|
||||||
|
return _DB_ERROR
|
||||||
|
return _DB_ERROR
|
||||||
|
except sqlite3.Error:
|
||||||
|
return _DB_ERROR
|
||||||
|
if not saw_any_table:
|
||||||
|
return _TABLE_MISSING
|
||||||
|
return total
|
||||||
|
|
||||||
|
|
||||||
def _strip_fenced_code_blocks(text: str) -> str:
|
def _strip_fenced_code_blocks(text: str) -> str:
|
||||||
"""Replace contents of triple-backtick fenced code blocks with newlines.
|
"""Replace contents of triple-backtick fenced code blocks with newlines.
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue