crawler: verbatim lift from agents.ai.unturf.com (off by default)

Lifts the async web fetcher into aborist as an opt-in source. The
implementation comes directly from ~/git/agents.ai.unturf.com/core
(rev 2026-04-28); aborist's adaptations are minimal and documented in
aborist/sources/crawler/__init__.py:

  core/async_web_fetcher.py  ->  aborist/sources/crawler/async_web_fetcher.py
  core/web_fetch.py          ->  aborist/sources/crawler/web_fetch.py

Two source-side changes during the lift:

1. Heavy deps (aiohttp, bs4, miniuri) wrapped in try/except so a bare
   `import aborist.sources.crawler` raises ImportError with the install
   hint instead of leaking AttributeErrors deep in user code.

2. Chat-bot fetch triggers (`has_fresh_fetch_trigger`,
   `has_web_fetch_trigger` from agents.ai.unturf.com/core/keywords)
   replaced with NotImplementedError stubs. Aborist has no chat
   surface — fetch intent is detected at the application layer. The
   two test classes that exercised these triggers are
   `@pytest.mark.skip`'d with the same rationale.

Not lifted: web_cache_manager.py — it backs page caching with
SQLAlchemy. Aborist has its own content-addressed cache via
providence_cache; no need to carry SQLAlchemy as a dep just for
crawled-page memoization.

Off by default:

- `[crawler]` extras section in pyproject.toml carries the heavy
  deps. `[dev]` pulls them in so the crawler tests can run.
- `make test` ignores tests/crawler/ entirely.
- `make bootstrap-crawler` installs the extras into the venv.
- `make test-crawler` runs only the lifted tests after extras land.

Tests: 74 passed, 9 skipped (the chat-bot trigger tests deliberately
dropped). Default `make test` stays at 273 passed, 1 skipped.

Makefile

@@ -31,7 +31,7 @@ SEARCH_Q     ?= computer
         ingest-grok ingest-grok-media \
         ingest-self ingest-git ingest-hg \
         verify search stats test docs chain-check chain-check-shards \
-        falsify burn clean clean-db clean-data help
+        falsify burn bootstrap-crawler test-crawler clean clean-db clean-data help
 
 all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
 
@@ -309,8 +309,17 @@ search: bootstrap ## keyword search; override SEARCH_Q (or pass Q=...)
 stats: bootstrap ## counts: documents, chunks, edges, audit chain
 	$(ABORIST) --db $(DB) stats
 
-test: bootstrap ## run pytest suite
-	$(VENV)/bin/pytest -q
+test: bootstrap ## run pytest suite (excludes opt-in crawler tests)
+	$(VENV)/bin/pytest -q --ignore=tests/crawler
+
+# Crawler tests are off-by-default — they hit the network in many cases
+# and require the heavy [crawler] extras (aiohttp, bs4, lxml, etc.).
+# Bootstrap installs the extras into the existing venv idempotently.
+bootstrap-crawler: bootstrap ## install [crawler] extras into the venv
+	$(PIP) install -e '.[crawler]'
+
+test-crawler: bootstrap-crawler ## run only the lifted crawler tests
+	$(VENV)/bin/pytest -q tests/crawler
 
 DOT_SRCS := $(wildcard docs/diagrams/*.dot)
 DOT_PNGS := $(DOT_SRCS:.dot=.png)