crawler: bridge to ingest + fast recrawl-check via conditional HEAD

Glues the verbatim crawler lift to aborist's existing ingest pipeline,
plus adds a freshness probe that asks "does this need recrawling?"
without paying for a body fetch when the answer is no.

Bridge (aborist/sources/crawler/bridge.py):
- crawl_seed(seed, depth, max_pages): BFS-discovers same-domain URLs
  via AsyncWebFetcher. Exact-hostname filter so subdomain crossover is
  off by default. Caps at max_pages. URL dedup, fragment-strip
  canonicalization. Sync wrapper around an async core so the CLI
  remains a one-liner.
- ingest_crawled(conn, urls): runs the URL list through the standard
  ingest_source() path so chunking_version / canonicalization_version
  / Merkle commitment all stay consistent. Captures ETag +
  Last-Modified per response and writes one row per document_root
  into the new document_http_meta table.

Schema (idempotent migration, mirrors _migrate_mesh_peer_chains):
  document_http_meta (
    document_root    TEXT PRIMARY KEY,
    etag             TEXT,
    last_modified    TEXT,
    last_fetched_at  INTEGER NOT NULL,
    last_status      INTEGER,
    last_checked_at  INTEGER,
    FK(document_root) -> documents(document_root) ON DELETE CASCADE
  )

Recrawl-check (recrawl_check):
- Sends conditional HEAD with If-None-Match + If-Modified-Since.
  304 -> fresh (no body transfer needed). 200 -> stale (server says
  the body changed). 404/410 -> gone. Other -> unreachable.
- Updates last_status + last_checked_at so consecutive runs visit
  oldest-first (NULLS FIRST in ORDER BY).
- Optional --domain filter so operators can probe one site at a time.

CLI:
  aborist crawl --seed-url URL [--depth 2] [--max-pages 20] [--ingest]
  aborist crawler recrawl-check [--domain DOMAIN] [--limit 100]

Makefile:
  crawl-ingest URL=https://example.com [DEPTH=2 MAX=20]
  recrawl-check [DOMAIN=example.com LIMIT=100]

Both targets gate on bootstrap-crawler so [crawler] extras land first.

Tests: 17 new in tests/crawler/test_bridge.py — same-domain filter,
BFS bounds (depth + max_pages), dedup, fragment normalization, ETag
capture, content-type filter, HEAD classification (304/200/404),
domain filter, last_checked_at update. All stub the network via a
fake httpx.Client; default suite still 273 passed, 1 skipped.

End-to-end test path:
  make bootstrap-crawler
  make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 MAX=20
  make query Q="who is russell ballestrini?"
  make recrawl-check DOMAIN=russell.ballestrini.net

Makefile

@@ -31,7 +31,8 @@ SEARCH_Q     ?= computer
         ingest-grok ingest-grok-media \
         ingest-self ingest-git ingest-hg \
         verify search stats test docs chain-check chain-check-shards \
-        falsify burn bootstrap-crawler test-crawler clean clean-db clean-data help
+        falsify burn bootstrap-crawler test-crawler crawl-ingest recrawl-check \
+        clean clean-db clean-data help
 
 all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
 
@@ -321,6 +322,22 @@ bootstrap-crawler: bootstrap ## install [crawler] extras into the venv
 test-crawler: bootstrap-crawler ## run only the lifted crawler tests
 	$(VENV)/bin/pytest -q tests/crawler
 
+# Bridge target: BFS a seed URL, ingest discovered pages into a shard,
+# capture ETag/Last-Modified for each so recrawl-check can do conditional
+# HEAD requests later. URL is required; DEPTH and MAX have safe defaults.
+CRAWL_DEPTH ?= 2
+CRAWL_MAX   ?= 20
+crawl-ingest: bootstrap-crawler ## crawl URL=https://x.com [DEPTH=2 MAX=20] and ingest into $(DB)
+	@if [ -z "$(URL)" ]; then echo "usage: make crawl-ingest URL=https://example.com [DEPTH=2 MAX=20]" >&2; exit 2; fi
+	$(ABORIST) --db $(DB) crawl --seed-url "$(URL)" --depth $(CRAWL_DEPTH) --max-pages $(CRAWL_MAX) --ingest
+
+# Fast freshness probe: conditional HEAD per doc, classify fresh/stale/gone.
+# Send only If-None-Match + If-Modified-Since headers — server returns 304
+# with no body when content unchanged.
+RECRAWL_LIMIT ?= 100
+recrawl-check: bootstrap-crawler ## conditional HEAD on ingested docs [DOMAIN=foo.com LIMIT=100]
+	$(ABORIST) --db $(DB) crawler recrawl-check $(if $(DOMAIN),--domain $(DOMAIN),) --limit $(RECRAWL_LIMIT)
+
 DOT_SRCS := $(wildcard docs/diagrams/*.dot)
 DOT_PNGS := $(DOT_SRCS:.dot=.png)