- make crawl-ingest writes to one central crawl db (CRAWL_DB, default
~/.arborist/crawl/web.db) instead of per-domain shards in the
peer-shared main dir: keeps locally-crawled content out of peer
sharing by default and a growing domain set under SQLite's 10-attach
cap (Makefile, docs/crawler.md).
- arborist query auto-includes the local crawl db (query() gains
extra_shards; CLI --include-shard / --no-crawl-db, default-on when
web.db exists). Fix latent --db single-file query AttributeError
(cli.py). Persist used / used_pointer_ids + retrieval_purity into
merkle_proof so read-only consumers can see which chunks fed the
answer (qa/query.py).
- arborist.read: read-only seam for dashboards / verifiers; on a
multi-source context root surface the real primary source instead of
the opaque corpus://multi-source sentinel (read.py). Backs the
arborist-viz Merkle Command Center (#000069).
- tests for extra_shards, the CLI crawl-db resolver, and the read seam.
New docs/crawler.md covering the crawl path we actually run: BFS
same-host discovery, robots/feed/sitemap handling, polite vs --fast,
the shared-session + no-HEAD + crawl-delay-fix speedups, and the
content-addressed payoff. Leads on the two store-derived diagnostics:
duplicate detection (group by document_root — body, not URI) and
partial-overlap (shared chunk leaves), plus orphan finding
(sitemap − BFS-reached) and the planned crawl-report webmaster tools.
Honest pros/cons: orphans invisible to crawl by design, single-host,
no JS execution, --fast is anti-social off your own turf.
