engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

crawl central-db + query auto-include + read-seam provenance

Browse source 
- make crawl-ingest writes to one central crawl db (CRAWL_DB, default
  ~/.arborist/crawl/web.db) instead of per-domain shards in the
  peer-shared main dir: keeps locally-crawled content out of peer
  sharing by default and a growing domain set under SQLite's 10-attach
  cap (Makefile, docs/crawler.md).

- arborist query auto-includes the local crawl db (query() gains
  extra_shards; CLI --include-shard / --no-crawl-db, default-on when
  web.db exists). Fix latent --db single-file query AttributeError
  (cli.py). Persist used / used_pointer_ids + retrieval_purity into
  merkle_proof so read-only consumers can see which chunks fed the
  answer (qa/query.py).

- arborist.read: read-only seam for dashboards / verifiers; on a
  multi-source context root surface the real primary source instead of
  the opaque corpus://multi-source sentinel (read.py). Backs the
  arborist-viz Merkle Command Center (#000069).

- tests for extra_shards, the CLI crawl-db resolver, and the read seam.
This commit is contained in:
russell@unturf.com 2026-05-29 13:45:47 -04:00
parent 6eada6dc89
commit 7f7eeefeb9
No known key found for this signature in database
10 changed files with 3460 additions and 20 deletions
Download patch file Download diff file Expand all files Collapse all files

24
Makefile
View file

@ -1331,18 +1331,20 @@ test-crawler: bootstrap-crawler ## run only the lifted crawler tests
# HEAD requests later. URL is required; DEPTH and MAX have safe defaults.
CRAWL_DEPTH ?= 2
CRAWL_MAX ?= 0
# Per-domain shard so `make query` (which reads $(SHARDS_DIR)) sees the
# crawled content. Shard filename derived from the seed URL's hostname:
# https://russell.ballestrini.net -> $(SHARDS_DIR)/crawl_russell_ballestrini_net.db
# Override with CRAWL_SHARD=... when you want a custom path.
crawl-ingest: bootstrap-crawler ## crawl URL=https://x.com [DEPTH=2 MAX=0 FAST=1] into $(SHARDS_DIR)/crawl_<domain>.db
@if [ -z "$(URL)" ]; then echo "usage: make crawl-ingest URL=https://example.com [DEPTH=2 MAX=0 FAST=1 CRAWL_SHARD=path]" >&2; exit 2; fi
@mkdir -p $(SHARDS_DIR)
# General web crawls land in ONE central crawl db ($(CRAWL_DB)), inside
# $(CRAWL_SHARDS_DIR) so they stay OUT of the peer-shared main $(SHARDS_DIR)
# by default and a growing set of crawled domains never trips SQLite's
# 10-attached-database limit. Content-addressing lets many domains share
# one file safely (idempotent re-ingest; `supersedes` edges on change).
# To query it: `--db $(CRAWL_DB)` standalone, or attach it alongside the
# main corpus (still just one extra file, under the 10-attach cap).
# Override the path with CRAWL_DB=... (or CRAWL_SHARD=... for a one-off).
CRAWL_DB ?= $(CRAWL_SHARDS_DIR)/web.db
crawl-ingest: bootstrap-crawler ## crawl URL=https://x.com [DEPTH=2 MAX=0 FAST=1 CRAWL_DB=path] into the central crawl db ($(CRAWL_DB))
@if [ -z "$(URL)" ]; then echo "usage: make crawl-ingest URL=https://example.com [DEPTH=2 MAX=0 FAST=1 CRAWL_DB=path]" >&2; exit 2; fi
@mkdir -p $(CRAWL_SHARDS_DIR)
@shard="$(CRAWL_SHARD)"; \
if [ -z "$$shard" ]; then \
domain=$$(echo "$(URL)" | sed -E 's,^https?://([^/]+).*$$,\1,' | tr '.' '_'); \
shard="$(SHARDS_DIR)/crawl_$${domain}.db"; \
fi; \
if [ -z "$$shard" ]; then shard="$(CRAWL_DB)"; fi; \
echo " shard: $$shard" >&2; \
$(ARBORIST) --db "$$shard" crawl --seed-url "$(URL)" --depth $(CRAWL_DEPTH) --max-pages $(CRAWL_MAX) $(if $(FAST),--fast,) --ingest