crawl central-db + query auto-include + read-seam provenance

- make crawl-ingest writes to one central crawl db (CRAWL_DB, default
  ~/.arborist/crawl/web.db) instead of per-domain shards in the
  peer-shared main dir: keeps locally-crawled content out of peer
  sharing by default and a growing domain set under SQLite's 10-attach
  cap (Makefile, docs/crawler.md).

- arborist query auto-includes the local crawl db (query() gains
  extra_shards; CLI --include-shard / --no-crawl-db, default-on when
  web.db exists). Fix latent --db single-file query AttributeError
  (cli.py). Persist used / used_pointer_ids + retrieval_purity into
  merkle_proof so read-only consumers can see which chunks fed the
  answer (qa/query.py).

- arborist.read: read-only seam for dashboards / verifiers; on a
  multi-source context root surface the real primary source instead of
  the opaque corpus://multi-source sentinel (read.py). Backs the
  arborist-viz Merkle Command Center (#000069).

- tests for extra_shards, the CLI crawl-db resolver, and the read seam.

docs/TICKETS.md

@@ -111,6 +111,7 @@ Newest first. Update on every open/close.
 
 | ID       | Title                                          | Status                | Opened     | Directive |
 |----------|------------------------------------------------|-----------------------|------------|-----------|
+| #000069  | Arborist VIZ / Merkle Command Center (Pyramid + six.js + SSE browser dashboard) | **open · awaiting go/no-go · doc-only scaffold** (2026-05-27; filed from `/home/fox/Downloads/TICKET_0000VIZ_*`, stack corrected same day per fox). Configurable browser dashboard for inspecting arborist's content-addressed state: Merkle root explorer, proof verifier, claim warrant + graveyard, audit timeline, run-DAG replay, cache-key explainer, root diff, 3D Merkle lattice, optional circuit/activation traces. Read-only consumer; arborist proper stays source-of-truth, dashboard projects state. **Stack pinned to unturf-native** (fox 2026-05-27, supersedes proposal §3): **Pyramid + Jinja2 + SQLAlchemy** (matches `remarkbox` / `make_post_sell` / `unhomeschool.com` idiom), **SSE** (`text/event-stream` via Pyramid streaming response) for live audit/claim/falsifier patches, **vanilla JS + six.js** (fox's patched three.js fork at `git.unturf.com/gumyum/six.js` — three.js r175 + CWE-407 patches incl. ObjectBVH O(N)→O(log N); bundles vendored from `~/git/cupPCB/cdn/six/`; third-instance MOAD-0001 dogfood alongside `java-topology` + gumyum-engine) for 3D widgets and large-graph rendering, SQLite for dashboard metadata (no PostgreSQL/ClickHouse/Redis/NATS by default — promote on measured need), no React / no Next.js / no Node build step. Server-rendered SVG (or Graphviz `.dot` per existing `docs/diagrams/*.dot` pattern) replaces React Flow for run-DAG widgets. Browser-side proof verification dropped from v1 (server-side Pyramid view returns PASS/FAIL + receipt; reinstate phase-N only if third-party-verification use case surfaces). **Three filing-note gates before phase 0** (in ticket body): **F-1** sibling-repo home — implementation lives in a new `~/git/arborist-viz` (Pyramid Python, matches existing unturf apps), not in-tree; arborist's contribution is the read-API spec + view package + arborist library import via `arborist.embed`. **F-2** scope split — proposal carries 8 phases (§17 phases 0–8); recommended cut keeps phases 0–3 (schema + shell + proof/root widgets + claim/audit/run widgets) inside #000069, and spawns sibling tickets for SSE streaming (4), 3D six.js (5), massive-graph (6, only if measured need surfaces), circuit-tracing (7, gated on #000062), embeddable widgets (8) — Dav1d-audience rule. **F-3** upstream prereqs — phase 7 (circuit/activation) consumes **#000062 Mechanistic Witness**'s `MechanisticWitnessRoot`; phase 3's claim-graveyard widget projects **#000059**'s bounded-ingestion graveyard. Hard constraints: arborist soft-vs-hard discipline applies verbatim (attribution weights renderable but never `audit_mode`, never causal without intervention/ablation evidence); private-leaf default-deny (commitments + hashes + redacted maps only without explicit auth); every widget exposes its data query + source roots. Reserved scope: NOT a replacement for `arborist controller-events` / `arborist analyze` / `arborist inspect` CLI — those stay canonical inspector surfaces; VIZ is the projection layer. | 2026-05-27 | — |
 | #000068  | Verifier-blind missed-answer falsification guard | **in progress · Phase 1+2+3 landed 2026-05-27 · Phase 4 default flip NO-GO** (Phase 2 bench 2026-05-27 76q × n=3 claim_lattice Hermes-3-8B: 2/228 sidecar fires, both STRONG confidence, both the Ballestrini regression fixture, 100% precision, 0/226 false positives across non-Ballestrini runs. Phase 3 demote flag opt-in via `--demote-on-missed-answer` on `query`/`ask` — wires `EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL` for strong/medium confidence on lattice modes; lower rungs + non-lattice modes get `· missed-answer` tail tag. `answerability_demote_enabled` added to `_VERIFIER_POLICY_FIELDS` so flipping the flag partitions cache via verifier_policy_hash. Default OFF per Dav1d Phase 4 NO-GO — 100% precision at n=2 fires is too few samples to claim precision floor empirically; default flip blocks on wider bench + human spot-check. 47 tests (36 Phase 1 + 11 Phase 3) all passing. End-to-end verified live: 4/4 Hermes runs on Ballestrini with --demote-on-missed-answer rendered EVIDENCE-MISSED-PARTIAL.) Original opening 2026-05-27 (Dav1d de-novo review GO for Phase 1 with seven hardenings folded into spec — subject-token cue-stripping, answer-type alignment, confidence_class, candidate cap=10, precise offset_start/end/basis, cache-hit recompute-on-read, Phase 1 out of verifier_policy_hash). Original opening 2026-05-27; sibling to the user-payload-layout work shipped 2026-05-26, split out per the Dav1d-audience rule — `feedback_ticket_proliferation`). Surfaced by the Ballestrini case: evidence E2 literally contained the song names, Hermes-3-8B under `user_payload_layout=tail` said *"specific songs by her are not mentioned in the provided evidence blocks"*, verifier marked the run `EVIDENCE-WARRANTED` 2/2 because nothing positive was unsupported. **Verifier-blind false-negative class** — existing layered verifier (quote/span/entity/paraphrase + Rule 8 + Rule 9 + claim ceiling) guards unsupported *presence*, has no hook for unsupported *absence*. Layout fixes attention placement on the specific instance (n=3 bench 2026-05-27 confirms bookend/per_chunk recover Ballestrini); layout alone can't close the class — adversarial phrasing or bigger prompt resurfaces it under any layout. Proposed deterministic sidecar in `arborist/qa/inspect.py:diagnose_missed_answer`: three-clause conjunction — **(A)** answer matches denial pattern ("not mentioned", "not provided", "the evidence does not say", …, closed list versioned via `denial_patterns_version`); **(B)** question is extraction shape (reuse `arborist.qa.quantifier` classifier — `ALL`/`COMPREHENSIVE`/`OPEN_REQUEST` intensities, OR surface cues "songs by"/"works by"/"who wrote"/"list"/"name all"); **(C)** evidence contains candidate spans near subject tokens (reuse `entity_proximity_n`/`entity_proximity_window` from verify.py — quoted strings, title-case spans, comma-separated title lists within W chars of stemmed subject content tokens). All three must fire. Output: `result["answerability"]` with `missed_answer_candidate_spans` list (evidence_id + offset + text). **Hash discipline:** sidecar fields (`denial_patterns_version`, `extraction_cues_version`, `answerability_threshold`) fold into `governance_policy_hash` only; an optional `answerability_demote_enabled` flag (default OFF) wires `EVIDENCE-WARRANTED → EVIDENCE-MISSED-PARTIAL` in `_render_audit_label`, and IF on folds into BOTH `governance_policy_hash` AND `verifier_policy_hash` (changes rendered audit_mode, so verifier hash must move — the deliberate opt-in moves the verifier hash, sidecar-only stays out). No LLM-as-judge. Never writes `providence_cache`/`audit_events`. Never promotes claims. Pattern verbatim from `arborist.qa.inspect.diagnose_*` (deflection, coherence, title-relevance). Phases: 1 sidecar read-only, 2 bench + threshold tuning, 3 demote opt-in, 4 default decision (bench-gated). 5F-Falsification fixture: Ballestrini case already in `bench/qa_questions.txt` under "entity list". Full spec in `docs/tickets/ticket-000068-verifier-blind-missed-answer-guard.md`. | 2026-05-27 | D2 |
 | #000067  | M-aware cold-pack hydration (route incoming docs by content hash into M target shards) | **open · scaffold · prereq for #46 genesis test** (2026-05-26; surfaced while preparing the 3090 SPV-wallet validation). Today's `hydrate_from_metadata_pack` takes a single `conn` and writes every incoming row into one shard. With the corpus now in M=4 hash-routed topology (#000065), a fresh peer needs to land each document on `shard_for_document(document_root, M)` — same routing function as the producer. Without this, a fresh peer's `~/.arborist/shards/` is just one big single-shard DB and the M=4 ATTACH-and-route assumption #000065 was sized for doesn't hold consumer-side. Two coherent shapes: **(α) two-step kludge** — hydrate into single shard, then `arborist corpus reshard --to M` on the consumer. Works today (proven by the 2026-05-26 reshard executor) but doubles the wall time and treats packed shards as if they came from an arbitrary topology. **(β) direct M-aware hydrate** — extend `hydrate_from_metadata_pack` to accept `targets: list[sqlite3.Connection]` + `M: int` and route per-row at restore time (reusing `arborist.document.shard_for_document` + the table-routing rules in `arborist/migrate.py`). Manifest carries `corpus_shard_count` so the unpacker knows M from the pack itself. β is the right answer — α exists only as a fallback if 20-min-window pressure forces it. Sequence: (1) add `corpus_shard_count` to pack manifest (read from source meta during `dump_shard_metadata`); (2) `restore_shard_metadata_routed(targets, M, table_dir)` in `cold_pack_metadata.py` mirroring `_route_per_doc_table` from migrate.py; (3) `hydrate_from_metadata_pack` gains a `targets`/`shards_dir` param; (4) `arborist cold unpack --shards-dir DIR` initialises M target shards from the manifest's `corpus_shard_count` and routes; (5) regression test: pack 2 shards → hydrate into fresh 4 shards → assert every doc on its hash-routed target. Refactor opportunity: the routing rules (ROUTED_BY_DOCUMENT_ROOT, CONSOLIDATED_TABLES) currently live in migrate.py; this ticket can either duplicate them in cold_pack_metadata.py (fast) or factor into a shared `arborist/multi_shard.py` module (cleaner). The shared-module path is more honest given graft mode (#000066) wants the same primitives. Out of scope: graft / overlay mode (that's #000066 — overlays onto populated, this is hydrate-into-empty). | 2026-05-26 | — |
 | #000066  | Cold-pack overlay / graft mode (pack-as-package, witness-pattern audit chain) | **scaffold-only · awaiting go/no-go** (2026-05-26; surfaced while running #000065 reshard, fox extension: "we could envision a pack for wikipedia 2010, wikipedia current, etc"). Extend #000061 cold-pack hydration with a second mode: overlay an existing pack onto a populated shard set instead of hydrating into empty. Doc/chunk/edge/concept overlay is trivial (`INSERT OR IGNORE` on content-addressed PKs collapses dupes); FTS5 overlay is trivial (new chunks → new rowids → new FTS rows). The interesting part is the audit chain — can't naively append the pack's events because `prev_event_hash` linkage breaks across the join. Chosen approach: **graft receipt**. Append one new `event_type='graft'` event to the host chain carrying `(pack_hash, snapshot_root, corpus_name, event_count, first_event_hash, last_event_hash, manifest_root)`; the pack file itself becomes the durable witness for the absorbed events (anyone can re-fetch the pack, walk its internal chain, and verify it matches the receipt). Host chain stays linear; pack chain is a "witnessed subgraph." This is the same witness pattern Merkle-AGI v8/v9 is heading toward, but bought at near-zero schema cost. Rejected alternatives: re-chain everything (breaks external refs to old event_hashes — cache_keys anchoring to old `audit_event_hash`, snapshots, etc. — silently invalid); chain forest with new `chain_id` column (right answer when graft dominates the lifecycle, but premature now). **Pack-as-package extension** (fox 2026-05-26): each pack carries a `corpus_name` field in its manifest (`wikipedia-2010`, `wikipedia-current`, `arxiv-cs`, `textbooks-undergrad`, …) so operators pick which corpora to graft — `arborist cold graft wikipedia-current` becomes as natural as `apt install firefox`. Multiple packs of the same corpus name: most-recent `snapshot_root` wins; older packs stay in the bucket until GC. URI conflicts across corpora (e.g., `wikipedia.org/wiki/Foo` in both 2010 and current): different content → different `document_root` → both stored, `supersedes` edges per CLAUDE.md invariant. Providence-cache conflicts: same `cache_key` with different answer → existing v9.8 falsification framework handles it (`state='stale'` or `quarantined`). Mesh-peer-corpus-merge: each peer's pack is a graftable package; partition reconciliation becomes "exchange the packs you each carry, graft what you lack". The mesh-of-arborists semantic. Sequence: (1) `corpus_name` field in #000061 manifest format + alias index in bucket (`corpora/<name>/latest.json` pointer to active pack_hash); (2) `arborist cold graft <pack_hash>` / `arborist cold graft --corpus <name>` mode in evict.py — read pack, INSERT OR IGNORE per-table, emit graft receipt; (3) conflict-policy flag (`--on-uri-conflict {supersedes,skip,fail}`, default `supersedes`); (4) `arborist cold list-corpora` shows available packages in a bucket. Scaffold first, code only when (a) #000065 reshard lands and stabilises (b) a second corpus exists (the wikipedia-current snapshot, or first textbook bundle ready to graft onto wikipedia-2010 base) (c) at least two peers want to exchange. | 2026-05-26 | — |
@@ -182,4 +183,4 @@ Newest first. Update on every open/close.
 
 ## Next ID
 
-`000069`
+`000070`

docs/crawler.md

@@ -226,16 +226,29 @@ arborist crawler recrawl-check [--domain D] [--limit N]
 - `--author` — default author surname appended to titles for warrant
   resolution (only with `--ingest`).
 
-Make targets drive the textbook-crawl workflow; crawl shards land in
+Make targets drive both crawl workflows; crawl shards land in
 `~/.arborist/crawl/` (separate from the main `~/.arborist/shards` so
-SQLite's attached-DB limit isn't tripped):
+SQLite's 10-attached-DB limit isn't tripped, and so locally crawled
+content isn't shared as a peer by default):
 
 ```
-make crawl-textbooks            # BFS-crawl every manifest entry with a crawl_url
-make textbook ID=<id>           # ingest one textbook by id
-make crawl-textbooks-stats      # docs-per-shard summary
+make crawl-ingest URL=https://x.com   # general web crawl → ONE central
+                                      # db (CRAWL_DB, default web.db)
+make crawl-textbooks                  # BFS-crawl every manifest entry
+                                      # with a crawl_url (warrant substrate)
+make textbook ID=<id>                 # ingest one textbook by id
+make crawl-textbooks-stats            # docs-per-shard summary
 ```
 
+General web crawls (`make crawl-ingest`) all flow into a **single**
+central db rather than one-per-domain: content-addressing lets many
+domains coexist in one file (idempotent re-ingest, `supersedes` edges
+on change), and a single file always attaches under the 10-DB cap.
+Query it standalone with `arborist --db ~/.arborist/crawl/web.db query
+"…"`, or attach it alongside the main corpus when you want unified
+results. The per-host textbook crawls stay separate — they are warrant
+substrate, resolved through a different path.
+
 ## Source map
 
 | File | Role |