# Ticket #000026 — Real-shard workload baseline + search latency

**Status:** closed · Phase 1 + 2 + 3 landed 2026-05-09 (Phase 3 in `60b5748`)
**Opened:** 2026-05-08
**Scope:** establish a reproducible baseline for arborist's behavior
on real shards (~38 GB Wikipedia + crawl). Capture latency, capital
cost, memory writes, fork-score readiness. Use the baseline to
confirm or reject performance claims and as the parent state for
future mutation selection.
**Audience:** anyone who maintains arborist's QA, retrieval, or
selection layers; agent operators planning to use arborist at scale.
**Hard constraint:** baselines are reports, not gates. They must
NEVER block CI or PR merges; they're observation, not enforcement.

## Problem statement

The bench harness (5S/5T/5F/5R) gives reproducible signal on
synthetic fixtures. The π* registry, audit chain, and ForkScore land
without exercising real corpora. The `virt-back` query
(2026-05-08) demonstrated three things:

1. **Primary-source retrieval works** — primary article ranked #1
   from a single-domain crawl, claim-lattice pointers grounded the
   answer, noise sources stayed unused.
2. **Warrant quality is acceptable but not strongest** — the cited
   evidence was a copyright footer ("© Russell Ballestrini")
   rather than explicit author/maintainer metadata.
3. **Latency is broken, not slow** — total ~75s wall on the cold
   first run; 15s on a profiled re-run. Search dominated. The LLM
   was 2.7s of that 15s. Profile pinned the cause: **588
   `connect()` calls per query, each running 7 migration probes**
   on already-fully-migrated shards. 10,623 SQLite execute calls
   per query.

We need a real-shard baseline that captures these signals
explicitly, alongside artifacts (run-DAG roots, capital ledger
deltas, memory writes) that future mutation selection can compare
against.

## Profile findings (2026-05-08, virt-back query, warm cache)

```
total wall:                14.5 s
search (cumulative):        9.9 s
  fts5.search                 6.7 s   (7 calls)
  synonym_expand              3.0 s   (2 calls × 1.5 s each)
  _load_indices               2.6 s   (3 calls — overlaps above)
  _load_token_idf             0.3 s   (1 call)
LLM call (chat_completion): 2.7 s
SQLite executes:           10,623 calls / 7.07 s
  per-conn migration probes:    586 × 7 functions
  per-conn `executescript`:     586 × ~1.4 ms = 0.85 s
connections opened:          588   (one per route per shard ×
                                    rerank lookups)
```

The headline: **opening a SQLite connection on an
already-fully-migrated shard runs the same 7 forward-migration
probes every time**. With 6 shards × 4 routes (body/title/phrase/
core_keyword) × multiple reranker passes, that's 588 redundant
schema-probe sequences in a single query.

## Design choices (latency)

### Option A — Memoize migrations per (shard path, process)

Module-level `set` of (path, mtime) tuples. First `connect()` runs
the probes; subsequent calls on the same path skip them. Ditches
~12 s of redundant work in the worst case.

```
Pros:   smallest patch (~30 LOC); zero schema impact;
        fully transparent to callers.
Cons:   stale cache if a shard's schema is mutated externally
        between opens (rare; stat-based mtime check covers it).
```

### Option B — Connection pool per shard

Open one connection per shard at search start; share across all 4
routes. Eliminates 18 of the 24 search-route connections directly
(6 shards × 4 routes → 6 connections).

```
Pros:   addresses connection volume at the source.
Cons:   requires lifecycle care (close on errors, no leaks);
        bigger diff in `_search_corpus`.
```

### Option C — `--read-only` connect path

New `connect_readonly(path)` that skips the migration block
entirely. Search/rerank paths use it; ingest/audit-write paths use
the existing `connect()`.

```
Pros:   conceptually cleanest split — read paths don't pay for
        write-path schema evolution.
Cons:   two API surfaces to maintain; new callers must pick.
```

### Option D — Preflight migration once on shards-dir scan

`discover_shards()` already walks the shards directory; have it
migrate each shard exactly once at process start. After that all
`connect()` calls assume migrations are current.

```
Pros:   migration cost paid once per process even when
        `connect()` is called from multiple paths.
Cons:   semantics change — `connect()` no longer self-heals on
        first touch. New shard added mid-process is missed.
```

### Recommendation

**A first** (smallest, safest, biggest dollar-per-line ratio),
**B as a follow-up** if connection volume still hurts after A.
C/D are deeper changes that don't pay back unless A+B prove
insufficient.

## Design choices (warrant quality)

The `virt-back` answer cited the page footer for authorship. That's
acceptable but indistinguishable in the schema from a stronger
warrant. Reviewer (2026-05-08) proposed an authorship-warrant
ladder:

```
1. AUTHOR_PACKAGE_METADATA       (pyproject.toml, setup.py)
2. AUTHOR_REPOSITORY_OWNER       (git owner + README byline)
3. AUTHOR_PAGE_BYLINE            (article-byline metadata)
4. AUTHOR_PRIMARY_PAGE_TITLE     (project page on primary site)
5. AUTHOR_COPYRIGHT_FOOTER       (current `virt-back` warrant)
6. AUTHOR_SECONDARY_SOURCE       (third-party claim)
```

This is content-extraction layered on top of retrieval — orthogonal
to the latency fix and to the ForkScore selection layer. It belongs
in a separate ticket *after* the latency fix lands; until search
returns in <5 s on real shards, sharper warrant labels just decorate
a slow path.

**Out of scope for #000026.** Captured here as a finding for
follow-up (likely #000027 or sibling).

## Scope (this ticket)

### Phase 1 — latency fix

1. Implement Option A (per-process migration memoization).
2. Re-profile the `virt-back` query; confirm <5 s search target.
3. Spot-check 3 other real queries (1 wikipedia-only, 1 crawl-only,
   1 cross-shard) — confirm each lands <8 s wall.
4. Add regression test: `connect()` on an already-migrated shard
   runs zero migration probes after first touch.

### Phase 2 — baseline artifact

1. New script: `bench/scripts/real_shard_baseline.py`.
2. Runs a fixed set of ~10 queries against `~/.arborist/shards`.
3. Emits one JSON: `bench/results/real-shard-baseline.json`.
   Schema:
   ```json
   {
     "schema_version": "real-shard-baseline-v1",
     "commit_sha": "...",
     "shards_digest": "<sha256 of sorted shard fingerprints>",
     "test_count": 1300,
     "fixture_count": 660,
     "queries": [
       {
         "question": "who wrote virt-back?",
         "audit_mode": "STRICT",
         "verifier_method": "claim_lattice",
         "n_verified": 2,
         "n_quotes": 2,
         "wall_ms": 4231,
         "search_ms": 1820,
         "llm_ms": 2280,
         "primary_source_at_rank_1": true,
         "primary_source_used": true,
         "warrant_quality": "copyright_footer"
       },
       ...
     ],
     "capital_ledger_delta": {...},
     "memory_records_delta": {...},
     "fork_score_preview": {...}
   }
   ```
4. New Makefile target: `make bench-real-shard`.
5. New docs page: `docs/_source/real-shard-baseline.rst`.

### Phase 3 — fold latency findings into the report

The baseline JSON adds a `findings` section that captures the
warrant-quality observation, slowest query, etc. — operator-facing
prose, not enforced.

## Out of scope

- Authorship warrant ladder implementation (deferred to a follow-up
  ticket; this ticket only *records* the finding).
- CI integration. Baseline runs are slow + need a local shard set,
  so they live outside CI for now. CI re-enable is its own work
  (small, possibly direct-commit, not a ticket).
- Public docs refresh. Adjacent but separable; if the baseline
  exposes claims that the public docs misrepresent, document them
  in a follow-up.
- ForkScore live wiring. Phase 1a landed; live integration belongs
  in #000012 Phase 1b.

## Risks / things to watch

- **Migration memoization staleness.** If two arborist processes
  modify the same shard concurrently and only one runs migrations,
  the other's cached "already migrated" claim is wrong. Mitigation:
  cache key includes mtime + size; invalidate on stat change.
- **Per-conn PRAGMA cost.** The four PRAGMAs (synchronous, cache,
  temp_store, mmap) still run on every connect. They're cheap
  (~50µs each) — measure before optimizing further.
- **`synonym_expand` 1.5s cost.** Profile lists it at 3 s
  cumulative across 2 calls. If migration memoization doesn't drop
  this, it's its own subticket (concept_relations index missing or
  query plan suboptimal).
- **Real-shard baseline drift.** The shards on `neoblanka` are not
  reproducible from public sources alone (the `crawl_russell_*`
  shard is one operator's crawl). Document this in the baseline
  doc and treat the fingerprint as advisory, not pinning.

## Status

In progress.

**Phase 1 landed 2026-05-08.** Per-process migration memoization
in `arborist.store.connect()` keyed by resolved path. Migrations
+ `executescript(SCHEMA_SQL)` run once per (path, process); per-
connection PRAGMAs (`foreign_keys=ON`, `synchronous=NORMAL`,
`cache_size`, `temp_store`, `mmap_size`) still run every time —
SQLite scopes `foreign_keys` per-connection and our schema
relies on FK CASCADE.

Re-profile of `who wrote virt-back?` (commit post-Phase 1, warm
cache):

```
metric                       before    after
_migrate_*  (each function)     586        7   ← one per shard
executescript                   586        7
SQLite executes               10,623    3,687  (-65%)
wall (warm)                   14.5 s   13.4 s
```

Warm-cache wall delta is small because the migration probes were
many-but-cheap; the residual budget lives in FTS5 search (6.7 s,
separate concern) and `synonym_expand` (2.8 s, separate concern).
The 65% execute drop is the cold-cache win — each redundant
`executescript` had been triggering disk reads at 75 s scale per
the reviewer's report.

**Phase 2 landed 2026-05-08.** Baseline artifact:

- ``bench/scripts/real_shard_baseline.py`` — runs an 8-question
  fixture against a real shards directory, captures per-query
  wall/search/llm/verify timings, audit_mode, verifier_method,
  primary-source-used flag, and writes:
  - ``bench/results/real-shard-baseline.json`` — durable JSON
    with commit_sha, shards_fingerprint (sha256 of sorted
    name/size/mtime tuples), per-query rows, summary metrics
  - ``bench/results/real-shard-baseline.md`` — short markdown
    table (the thing humans read)
- ``bench/fixtures/real-shard-baseline-v1.jsonl`` — the question
  set: virt-back, France, Mac OS X, Linux, Microsoft, AMD/Intel
  (rivalry path), and two canonical-projection cases (math + logic).
- ``make bench-real-shard`` — Makefile target. Honors
  ``ARBORIST_SHARDS_DIR``; defaults to ``$HOME/.arborist/shards``.

First run (post-Phase-1 + lazy concepts; commit `e78814c` plus
unstaged refactor):

```
audit_mode      n
STRICT          4   (virt-back, France, Mac OS X, Linux)
HYBRID          2   (Microsoft founder, AMD/Intel)
CANONICAL       2   (0.1+0.2, A IMPL B)

wall median     4.1 s   (range 0.6 ms – 7.2 s)
primary used    4 / 8
search target   <5 s    ← met for 5 of 6 RAG queries
```

`who wrote virt-back?` lands at 6.3 s wall (vs reviewer's 75 s
report), audit STRICT, cited evidence still includes the copyright
footer (warrant-quality finding deferred to follow-up ticket).

**Phase 2 follow-up landed in same fan-out:** lazy concept-relations
loading. Profile of the same query showed `synonym_expand` 2.8 s ×
2 calls eating 4 s of search budget. Replaced the eager 290 K-row
full-table scan in `arborist.concepts.query` with targeted
WHERE token IN (qtokens) OR target IN (qtokens) loads, plus
process-wide per-token cache. Per-query SQL rows: ~290 K → ~300.
Wall delta on virt-back query: 13.4 s → 9.3 s (-30%) post-Phase-1.

Phase 3 (warrant-quality finding) is queued as a follow-up ticket
when fox is ready. Authorship warrant ladder remains out of scope
per the design choices section.

Phase 1 landed in commit `ec92ebc`.
Phase 2 landed in commit `8b1de20`.

**Phase 3 landed 2026-05-09.** Authorship warrant ladder as a
sidecar classifier in `arborist/qa/warrant_authorship.py`:

```
Tier 1  AUTHOR_PACKAGE_METADATA       (pyproject.toml authors,
                                       package.json author, etc.)
Tier 2  AUTHOR_REPOSITORY_OWNER       (github.com/<owner>/...)
Tier 3  AUTHOR_PAGE_BYLINE            (By NAME / Author: NAME /
                                       <meta name=author>)
Tier 4  AUTHOR_PRIMARY_PAGE_TITLE     (cited evidence is the
                                       entity's own primary page)
Tier 5  AUTHOR_COPYRIGHT_FOOTER       (© NAME / Copyright YYYY NAME
                                       — the current `virt-back`
                                       warrant)
Tier 6  AUTHOR_SECONDARY_SOURCE       (third-party claim)
NO_AUTHORSHIP_SIGNAL                  (question doesn't smell like
                                       an authorship inquiry —
                                       sidecar stays quiet)
```

Sidecar discipline: never enters proof path; never raises;
returns dict with `tier`, `tier_rank` (1=strongest, 99=quiet),
`signals` (every detector that fired), `candidate_names`,
`note`. Hint regex gates on "who wrote/created/authored/maintains/
owns/manages X" + "author/creator/maintainer/owner of Y" patterns
so non-authorship queries don't get false-tier-6 noise.

20 tests covering each tier (positive + boundary) + sidecar
contract (never raises on garbage input) + tier-ordering
(strongest signal wins when multiple fire). Authorship-warrant
ladder is now available; wiring it into the `arborist inspect`
sidecar output and the audit-line render-tail is a follow-up
(small, non-load-bearing).

Authorship warrant ladder landed in commit `60b5748`.
