diff --git a/docs/crawler.md b/docs/crawler.md
new file mode 100644
index 0000000..7389edb
--- /dev/null
+++ b/docs/crawler.md
@@ -0,0 +1,246 @@
+# Web crawler — surface ingest from live sites
+
+Arborist's crawler turns a live website into surface-layer documents: it
+walks a site from a seed URL, fetches each page, and runs the HTML
+through the same normalize → chunk → Merkle → upsert pipeline as any
+other source. The output lands in a content-addressed store, which is
+what makes the crawler's two headline diagnostics — **duplicate
+detection** and **orphan finding** — nearly free.
+
+It pairs with the textbook-ingest workflow (`docs/warrant-substrate-cookbook.md`)
+and the retrieval pipeline (`arborist/qa/query.py`).
+
+> Status: discovery + ingest + recrawl-check ship today. The webmaster
+> report (`crawl-report`) described under [Diagnostics](#diagnostics-the-webmaster-tools-angle)
+> is designed-not-built — this doc pins the contract.
+
+## Default off
+
+The crawler touches the network only when you ask it to, and its heavy
+dependencies (`aiohttp`, `beautifulsoup4`, `lxml`, …) are an opt-in
+extra:
+
+```
+pip install 'arborist[crawler]'      # or: make bootstrap-crawler
+```
+
+The default test suite never imports the crawler module. Nothing in a
+normal lookup / ingest / Q&A path reaches it.
+
+## How discovery works
+
+The crawl driver is `arborist/sources/crawler/bridge.py`
+(`_crawl_seed_async`). It is a **breadth-first walk that stays on one
+hostname**:
+
+1. Start from `--seed-url`. Fetch it, extract its links.
+2. Keep links that are **same exact hostname** (`_same_domain`: no
+   subdomain crossover — `x.com` does not follow into `docs.x.com`),
+   `http(s)`, and not already seen.
+3. Enqueue the survivors at `depth + 1`. Repeat until the queue drains,
+   `--depth` is hit, or `--max-pages` is reached (`0` = no cap, depth is
+   the only bound).
+
+Discovery fetches each page only far enough to extract its links; the
+arborist-side ingest is a separate step (`--ingest`).
+
+### What it refuses to crawl
+
+- **`robots.txt` Disallow is always honored** — in both polite and fast
+  mode. A blocked URL is skipped, never fetched.
+- **Feeds and sitemaps are skipped at discovery** (`_looks_like_feed_url`):
+  `*.atom`, `*.rss`, `*.rdf`, `/feed/`, `/feeds/`, `sitemap.xml`,
+  `sitemap_index.xml`, `/sitemaps/*.xml`, and the WordPress legacy feed
+  names. They are crawl *infrastructure*, not knowledge — `ingest_crawled`
+  would discard them anyway, and a multi-MB `atom.xml` is a slow
+  straggler that gates everything fetched alongside it. A second filter
+  at ingest (URL pattern **and** response `Content-Type`) catches feeds
+  served at unconventional paths. `xhtml` is preserved — it *is* HTML.
+
+The crawler **never proactively probes** `/sitemap.xml` or guesses feed
+locations on the path we use. (The fetcher carries an unused
+sitemap-discovery `crawl()` strategy method from its upstream lift; the
+arborist bridge calls only `fetch_webpage`.)
+
+## Polite vs fast
+
+`--fast` flips the underlying `AsyncWebFetcher` into `fast_mode`. The one
+rule that matters: **fast mode drops the rate limit, never the access
+rules.**
+
+| | polite (default) | `--fast` |
+|---|---|---|
+| robots `Disallow` | honored | honored |
+| robots `crawl-delay` | honored; **2.0s** fallback when robots is silent/absent | **ignored** (0s) |
+| concurrency per wave | 1 (sequential) | `CPU × 3` |
+| connections | shared keepalive session | shared keepalive session |
+| per-page requests | one `GET` | one `GET` |
+| page `GET` timeout | 15s (robots 5s) | 15s (robots 5s) |
+
+Use `--fast` only where aggressive fetching is acceptable: your own
+sites, or dumps you have been authorized to mirror. On someone else's
+infrastructure, the polite default is the courteous neighbour.
+
+### Why fast is fast
+
+Each fetch wave runs up to `CPU × 3` requests concurrently through **one
+shared `aiohttp` session** with a keepalive connector, so connections
+are reused across the whole crawl instead of paying a fresh TCP+TLS per
+page. There is **no preflight `HEAD`** — `aiohttp` exposes response
+headers before the body is read, so the binary content-type guard skips
+images/video/audio without downloading them, and the second round trip
+the old `HEAD` cost is gone. Link extraction uses `lxml` (a C parser
+that releases the GIL), so parsing genuinely parallelises across a wave
+rather than serialising on the interpreter lock.
+
+Measured on a robots-less site (`russell.ballestrini.net`, 242 URIs),
+these together took a full `--fast` crawl from ~26s to ~5s. The largest
+single factor was making fast mode actually ignore the crawl-delay on a
+site whose `robots.txt` 404s — previously it fell back to the 2s default
+and every concurrent wave slept.
+
+## Ingest and recrawl
+
+With `--ingest`, each discovered page is fetched with `httpx`, parsed,
+and run through `ingest_source`, so crawled documents land in the same
+tables as Wikipedia or textbook sources. The crawler additionally
+records each document's `ETag` and `Last-Modified` in
+`document_http_meta`, so `arborist crawler recrawl-check` can later send
+conditional `HEAD` requests and classify each URL **fresh** (304) /
+**stale** (200) / **gone** (404) without re-downloading bodies.
+
+Re-ingest is idempotent: same content → same `document_root` → no-op.
+Same URI with changed content → a new document plus a `supersedes` edge
+(lossless history).
+
+## The content-addressed payoff
+
+This is where the crawler differs from an off-the-shelf spider. Every
+document's identity is its **Merkle root over canonicalized body text**
+(`ingest.py:_compute_artifacts`):
+
+```
+text   = canonicalize(content)          # body only — norm-v1
+leaves = [hash_leaf(chunk) for chunk in chunker.split(text)]
+document_root = MerkleTree.build(leaves).root
+```
+
+The **URI is not part of the root** — it is a separate `document_uri`
+column. Two consequences fall straight out of that:
+
+### Duplicate detection (free)
+
+Two different URLs whose bodies canonicalize to the same text produce
+the **same `document_root`**. So exact-and-near-duplicate pages are a
+`GROUP BY document_root HAVING COUNT(*) > 1` query — no re-crawl, no
+heuristics. Because the body is `canonicalize()`d before hashing,
+trivial whitespace/markup differences collapse too, which is the right
+behaviour for a "duplicate content" signal.
+
+Leaves are **per chunk**, so the same data also yields **partial
+overlap**: pages that share chunk-leaf hashes without being full dupes
+(boilerplate headers/footers, syndicated sections) cluster via a
+`chunks` join. No other crawler hands you "these N URLs are byte- or
+chunk-identical" for nothing — it is a property of the store's design.
+
+### Orphan finding
+
+Crawl gives the **link-reachable set** (what BFS discovered). A
+sitemap — fetched read-only, *once*, never ingested — gives the
+**declared set**. The difference is the orphans:
+
+```
+orphans = sitemap_urls − bfs_reachable_urls
+```
+
+These are pages a site lists but nothing links to. Surfacing them is a
+classic webmaster need and costs us a single extra fetch on top of a
+crawl we already ran.
+
+## Diagnostics: the webmaster-tools angle
+
+The store's by-products make a small "webmaster report" cheap. Planned
+shape: `arborist crawl-report --seed-url … [--sitemap …]`, off by
+default, read-only, never writing `providence_cache` or `audit_events`.
+
+| Signal | How it's derived | Extra cost |
+|---|---|---|
+| Orphan pages | `sitemap − BFS-reached` | one sitemap fetch |
+| Exact duplicate pages | group by `document_root` | free (store query) |
+| Partial-overlap clusters | shared chunk-leaf hashes | free (store query) |
+| Broken links / 404s | per-URL status during crawl | free |
+| Redirect chains | 301/308 hops observed | free |
+| Depth / crawl-budget map | BFS depth per URL | free |
+| Thin / untitled pages | chunk count + missing `<title>` | free |
+
+A likely v1 keeps it to **orphans + exact dupes + broken links**, with
+partial-overlap and the rest as follow-ons.
+
+## Pros and cons
+
+**Pros**
+
+- **Compliant by default.** robots `Disallow` always honored;
+  crawl-delay honored unless you explicitly opt into `--fast`.
+- **Fast when you own the target.** Concurrent waves, keepalive
+  sessions, no redundant `HEAD`, GIL-releasing parse.
+- **Duplicate and partial-overlap detection for free**, because identity
+  is content not URL.
+- **Idempotent, lossless re-ingest** with a conditional-`HEAD`
+  recrawl-check that avoids re-downloading unchanged pages.
+- **Same store as every other source** — crawled pages are queryable,
+  distillable, and Merkle-verifiable like any document.
+
+**Cons / trade-offs**
+
+- **Orphans are invisible to the crawl itself.** We follow links and do
+  not consume sitemaps for discovery, so a page reachable *only* via the
+  sitemap is never ingested. That is a deliberate default (crawl what is
+  linked) — orphan *finding* is a report you run on top, not a crawl
+  behaviour.
+- **Single hostname, exact match.** No subdomain crossover and no
+  cross-domain following by design; multi-host sites need multiple
+  seeds.
+- **No JavaScript execution.** Links and content rendered client-side
+  (SPA routes) are not seen — this is an HTML fetcher, not a headless
+  browser.
+- **`--fast` is anti-social off your own turf.** Ignoring crawl-delay
+  and fanning out `CPU × 3` is fine for sites you control and rude
+  elsewhere. The default is polite for a reason.
+- **Diverged from upstream.** Arborist's `AsyncWebFetcher` no longer
+  matches the `agents.ai.unturf.com/core` verbatim lift (shared session,
+  dropped `HEAD`, fast-mode crawl-delay fix). A candidate to upstream.
+
+## CLI and make targets
+
+```
+arborist crawl --seed-url URL [--depth N] [--max-pages N] [--ingest] [--fast] [--author NAME]
+arborist crawler recrawl-check [--domain D] [--limit N]
+```
+
+- `--depth` — max BFS depth (default 2).
+- `--max-pages` — cap discovery at N URLs (`0` = no cap, depth bounds it).
+- `--ingest` — ingest discovered pages; without it, the URL list is
+  printed only.
+- `--fast` — fast mode (see table). Disallow still honored.
+- `--author` — default author surname appended to titles for warrant
+  resolution (only with `--ingest`).
+
+Make targets drive the textbook-crawl workflow; crawl shards land in
+`~/.arborist/crawl/` (separate from the main `~/.arborist/shards` so
+SQLite's attached-DB limit isn't tripped):
+
+```
+make crawl-textbooks            # BFS-crawl every manifest entry with a crawl_url
+make textbook ID=<id>           # ingest one textbook by id
+make crawl-textbooks-stats      # docs-per-shard summary
+```
+
+## Source map
+
+| File | Role |
+|---|---|
+| `arborist/sources/crawler/bridge.py` | BFS driver, feed-skip, ingest, recrawl-check |
+| `arborist/sources/crawler/async_web_fetcher.py` | fetch + robots + crawl-delay + shared session |
+| `arborist/ingest.py` | `document_root` derivation (content, not URI) |
+| `arborist/merkle.py` | Merkle tree / proof conventions |