engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

store: per-process migration memoization (#000026 Phase 1)

Browse source 
`connect()` used to run executescript(SCHEMA_SQL) + 7 forward-
migration probes on every open. Profile of `who wrote virt-back?`
on 38 GB of real shards (warm cache) showed 588 connect() calls
per query, each running the full probe sequence — 10,623 total
SQLite executes. Migrations are forward-only and idempotent within
a code version, so once we've run them on a path in this process
there's no work to do on subsequent opens.

Cache shape: `set[str]` keyed by `str(Path(p).resolve())`.
Migration block runs once per (path, process); subsequent calls on
the same shard skip it entirely. Per-connection PRAGMAs
(foreign_keys=ON, synchronous=NORMAL, cache_size, temp_store,
mmap_size) still run every time — SQLite scopes foreign_keys
per-connection and our schema's FK CASCADE behavior depends on it.
That's why `PRAGMA foreign_keys = ON` moved out of the cached
SCHEMA_SQL block into the always-run pragma section.

Cache invalidation: explicit only.
`store.invalidate_migration_cache(path)` for callers who replace a
shard at the same path (snapshot-restore flows). `_clear_migration_
cache()` for tests. We don't auto-detect file replacement —
(dev, inode) is unreliable under tmpfs inode reuse, and (mtime,
size) drifts naturally as SQLite operates on the file (WAL
checkpoints, page growth). Path-only with explicit invalidation
is the honest contract.

Re-profile (same query, same shards, warm cache):

  metric                         before    after
  _migrate_*  (each function)       586        7   ← per shard
  executescript                     586        7
  SQLite executes                10,623    3,687  (-65%)
  wall (warm)                    14.5 s   13.4 s

The warm-cache wall delta is small because the probes were many-
but-cheap; residual cost lives in FTS5 search (6.7 s) and
synonym_expand (2.8 s, both separate concerns). The 65% execute
drop is the cold-cache win — each redundant executescript() had
been triggering disk reads at the 75 s scale the reviewer reported.

Tests (6, all green): first connect runs all 7 probes, second
connect runs zero, schema integrity preserved across re-opens,
explicit invalidation re-probes, distinct paths each get one probe,
clear-cache helper works.

Full suite: 1306 passed, 36 skipped. Found and fixed an FK CASCADE
regression mid-implementation: PRAGMA foreign_keys = ON was inside
SCHEMA_SQL, so memoization was silently turning it off on subsequent
opens. test_burn_doc.py caught it. Moved to the per-connection
pragma block.

Ticket #000026 status: Phase 1 landed; Phase 2 (baseline artifact)
and Phase 3 (warrant-quality finding) queued.
This commit is contained in:
russell@unturf.com 2026-05-08 12:48:48 -04:00
parent a81494a979
commit ec92ebc575
No known key found for this signature in database
4 changed files with 282 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/TICKETS.md
View file

@ -61,7 +61,7 @@ Newest first. Update on every open/close.
| ID | Title | Status | Opened | Directive |
|----------|------------------------------------------------|-----------------------|------------|-----------|
| #000026 | Real-shard workload baseline + search latency | open · awaiting go/no-go | 2026-05-08 | — |
| #000026 | Real-shard workload baseline + search latency | in progress · Phase 1 landed 2026-05-08 | 2026-05-08 | — |
| #000025 | 5F battery (Function · Finetuning · Falsification · Formulate · Feedback Loop) | in progress · Phase 1a landed 2026-05-08 | 2026-05-07 | — |
| #000024 | 5T Phase 1b + Dav1DPrometheus vocabulary alignment | closed · landed 2026-05-08 | 2026-05-07 | — |
| #000023 | 5S Phase 1b: Syllogism · Synthesis · Semiotics | closed · landed 2026-05-08 | 2026-05-07 | — |

38
docs/tickets/ticket-000026-real-shard-workload-baseline.md
View file

@ -1,6 +1,6 @@
# Ticket #000026 — Real-shard workload baseline + search latency
**Status:** open · awaiting go/no-go
**Status:** in progress · Phase 1 landed 2026-05-08
**Opened:** 2026-05-08
**Scope:** establish a reproducible baseline for arborist's behavior
on real shards (~38 GB Wikipedia + crawl). Capture latency, capital
@ -231,6 +231,36 @@ prose, not enforced.
## Status
Open · awaiting go/no-go. Profile data captured 2026-05-08
(commit `0a2e347`). Implementation can land Phase 1 in one
commit (~30 LOC + test); Phase 2 in a follow-up.
In progress.
**Phase 1 landed 2026-05-08.** Per-process migration memoization
in `arborist.store.connect()` keyed by resolved path. Migrations
+ `executescript(SCHEMA_SQL)` run once per (path, process); per-
connection PRAGMAs (`foreign_keys=ON`, `synchronous=NORMAL`,
`cache_size`, `temp_store`, `mmap_size`) still run every time —
SQLite scopes `foreign_keys` per-connection and our schema
relies on FK CASCADE.
Re-profile of `who wrote virt-back?` (commit post-Phase 1, warm
cache):
```
metric before after
_migrate_* (each function) 586 7 ← one per shard
executescript 586 7
SQLite executes 10,623 3,687 (-65%)
wall (warm) 14.5 s 13.4 s
```
Warm-cache wall delta is small because the migration probes were
many-but-cheap; the residual budget lives in FTS5 search (6.7 s,
separate concern) and `synonym_expand` (2.8 s, separate concern).
The 65% execute drop is the cold-cache win — each redundant
`executescript` had been triggering disk reads at 75 s scale per
the reviewer's report.
Phase 2 (baseline artifact) and Phase 3 (warrant-quality finding)
queued. Authorship warrant ladder remains explicitly out of scope
per the design choices section.
Phase 1 landed in commit `<filled on commit>`.