arborist/docs/crawler.md

Web crawler — surface ingest from live sites

Arborist's crawler turns a live website into surface-layer documents: it walks a site from a seed URL, fetches each page, and runs the HTML through the same normalize → chunk → Merkle → upsert pipeline as any other source. The output lands in a content-addressed store, which is what makes the crawler's two headline diagnostics — duplicate detection and orphan finding — nearly free.

It pairs with the textbook-ingest workflow (docs/warrant-substrate-cookbook.md) and the retrieval pipeline (arborist/qa/query.py).

Status: discovery + ingest + recrawl-check ship today. The webmaster report (crawl-report) described under Diagnostics is designed-not-built — this doc pins the contract.

Default off

The crawler touches the network only when you ask it to, and its heavy dependencies (aiohttp, beautifulsoup4, lxml, …) are an opt-in extra:

pip install 'arborist[crawler]'      # or: make bootstrap-crawler

The default test suite never imports the crawler module. Nothing in a normal lookup / ingest / Q&A path reaches it.

How discovery works

The crawl driver is arborist/sources/crawler/bridge.py (_crawl_seed_async). It is a breadth-first walk that stays on one hostname:

1. Start from --seed-url. Fetch it, extract its links.
2. Keep links that are same exact hostname (_same_domain: no subdomain crossover — x.com does not follow into docs.x.com), http(s), and not already seen.
3. Enqueue the survivors at depth + 1. Repeat until the queue drains, --depth is hit, or --max-pages is reached (0 = no cap, depth is the only bound).

Discovery fetches each page only far enough to extract its links; the arborist-side ingest is a separate step (--ingest).

What it refuses to crawl

• robots.txt Disallow is always honored — in both polite and fast mode. A blocked URL is skipped, never fetched.
• Feeds and sitemaps are skipped at discovery (_looks_like_feed_url): *.atom, *.rss, *.rdf, /feed/, /feeds/, sitemap.xml, sitemap_index.xml, /sitemaps/*.xml, and the WordPress legacy feed names. They are crawl infrastructure, not knowledge — ingest_crawled would discard them anyway, and a multi-MB atom.xml is a slow straggler that gates everything fetched alongside it. A second filter at ingest (URL pattern and response Content-Type) catches feeds served at unconventional paths. xhtml is preserved — it is HTML.

The crawler never proactively probes /sitemap.xml or guesses feed locations on the path we use. (The fetcher carries an unused sitemap-discovery crawl() strategy method from its upstream lift; the arborist bridge calls only fetch_webpage.)

Polite vs fast

--fast flips the underlying AsyncWebFetcher into fast_mode. The one rule that matters: fast mode drops the rate limit, never the access rules.

	polite (default)	--fast
robots Disallow	honored	honored
robots crawl-delay	honored; 2.0s fallback when robots is silent/absent	ignored (0s)
concurrency per wave	1 (sequential)	CPU × 3
connections	shared keepalive session	shared keepalive session
per-page requests	one GET	one GET
page GET timeout	15s (robots 5s)	15s (robots 5s)

Use --fast only where aggressive fetching is acceptable: your own sites, or dumps you have been authorized to mirror. On someone else's infrastructure, the polite default is the courteous neighbour.

Why fast is fast

Each fetch wave runs up to CPU × 3 requests concurrently through one shared aiohttp session with a keepalive connector, so connections are reused across the whole crawl instead of paying a fresh TCP+TLS per page. There is no preflight HEAD — aiohttp exposes response headers before the body is read, so the binary content-type guard skips images/video/audio without downloading them, and the second round trip the old HEAD cost is gone. Link extraction uses lxml (a C parser that releases the GIL), so parsing genuinely parallelises across a wave rather than serialising on the interpreter lock.

Measured on a robots-less site (russell.ballestrini.net, 242 URIs), these together took a full --fast crawl from ~26s to ~5s. The largest single factor was making fast mode actually ignore the crawl-delay on a site whose robots.txt 404s — previously it fell back to the 2s default and every concurrent wave slept.

Ingest and recrawl

With --ingest, each discovered page is fetched with httpx, parsed, and run through ingest_source, so crawled documents land in the same tables as Wikipedia or textbook sources. The crawler additionally records each document's ETag and Last-Modified in document_http_meta, so arborist crawler recrawl-check can later send conditional HEAD requests and classify each URL fresh (304) / stale (200) / gone (404) without re-downloading bodies.

Re-ingest is idempotent: same content → same document_root → no-op. Same URI with changed content → a new document plus a supersedes edge (lossless history).

The content-addressed payoff

This is where the crawler differs from an off-the-shelf spider. Every document's identity is its Merkle root over canonicalized body text (ingest.py:_compute_artifacts):

text   = canonicalize(content)          # body only — norm-v1
leaves = [hash_leaf(chunk) for chunk in chunker.split(text)]
document_root = MerkleTree.build(leaves).root

The URI is not part of the root — it is a separate document_uri column. Two consequences fall straight out of that:

Duplicate detection (free)

Two different URLs whose bodies canonicalize to the same text produce the same document_root. So exact-and-near-duplicate pages are a GROUP BY document_root HAVING COUNT(*) > 1 query — no re-crawl, no heuristics. Because the body is canonicalize()d before hashing, trivial whitespace/markup differences collapse too, which is the right behaviour for a "duplicate content" signal.

Leaves are per chunk, so the same data also yields partial overlap: pages that share chunk-leaf hashes without being full dupes (boilerplate headers/footers, syndicated sections) cluster via a chunks join. No other crawler hands you "these N URLs are byte- or chunk-identical" for nothing — it is a property of the store's design.

Orphan finding

Crawl gives the link-reachable set (what BFS discovered). A sitemap — fetched read-only, once, never ingested — gives the declared set. The difference is the orphans:

orphans = sitemap_urls − bfs_reachable_urls

These are pages a site lists but nothing links to. Surfacing them is a classic webmaster need and costs us a single extra fetch on top of a crawl we already ran.

Diagnostics: the webmaster-tools angle

The store's by-products make a small "webmaster report" cheap. Planned shape: arborist crawl-report --seed-url … [--sitemap …], off by default, read-only, never writing providence_cache or audit_events.

Signal	How it's derived	Extra cost
Orphan pages	sitemap − BFS-reached	one sitemap fetch
Exact duplicate pages	group by document_root	free (store query)
Partial-overlap clusters	shared chunk-leaf hashes	free (store query)
Broken links / 404s	per-URL status during crawl	free
Redirect chains	301/308 hops observed	free
Depth / crawl-budget map	BFS depth per URL	free
Thin / untitled pages	chunk count + missing <title>	free

A likely v1 keeps it to orphans + exact dupes + broken links, with partial-overlap and the rest as follow-ons.

Pros and cons

Pros

• Compliant by default. robots Disallow always honored; crawl-delay honored unless you explicitly opt into --fast.
• Fast when you own the target. Concurrent waves, keepalive sessions, no redundant HEAD, GIL-releasing parse.
• Duplicate and partial-overlap detection for free, because identity is content not URL.
• Idempotent, lossless re-ingest with a conditional-HEAD recrawl-check that avoids re-downloading unchanged pages.
• Same store as every other source — crawled pages are queryable, distillable, and Merkle-verifiable like any document.

Cons / trade-offs

• Orphans are invisible to the crawl itself. We follow links and do not consume sitemaps for discovery, so a page reachable only via the sitemap is never ingested. That is a deliberate default (crawl what is linked) — orphan finding is a report you run on top, not a crawl behaviour.
• Single hostname, exact match. No subdomain crossover and no cross-domain following by design; multi-host sites need multiple seeds.
• No JavaScript execution. Links and content rendered client-side (SPA routes) are not seen — this is an HTML fetcher, not a headless browser.
• --fast is anti-social off your own turf. Ignoring crawl-delay and fanning out CPU × 3 is fine for sites you control and rude elsewhere. The default is polite for a reason.
• Diverged from upstream. Arborist's AsyncWebFetcher no longer matches the agents.ai.unturf.com/core verbatim lift (shared session, dropped HEAD, fast-mode crawl-delay fix). A candidate to upstream.

CLI and make targets

arborist crawl --seed-url URL [--depth N] [--max-pages N] [--ingest] [--fast] [--author NAME]
arborist crawler recrawl-check [--domain D] [--limit N]

• --depth — max BFS depth (default 2).
• --max-pages — cap discovery at N URLs (0 = no cap, depth bounds it).
• --ingest — ingest discovered pages; without it, the URL list is printed only.
• --fast — fast mode (see table). Disallow still honored.
• --author — default author surname appended to titles for warrant resolution (only with --ingest).

Make targets drive both crawl workflows; crawl shards land in ~/.arborist/crawl/ (separate from the main ~/.arborist/shards so SQLite's 10-attached-DB limit isn't tripped, and so locally crawled content isn't shared as a peer by default):

make crawl-ingest URL=https://x.com   # general web crawl → ONE central
                                      # db (CRAWL_DB, default web.db)
make crawl-textbooks                  # BFS-crawl every manifest entry
                                      # with a crawl_url (warrant substrate)
make textbook ID=<id>                 # ingest one textbook by id
make crawl-textbooks-stats            # docs-per-shard summary

General web crawls (make crawl-ingest) all flow into a single central db rather than one-per-domain: content-addressing lets many domains coexist in one file (idempotent re-ingest, supersedes edges on change), and a single file always attaches under the 10-DB cap. Query it standalone with arborist --db ~/.arborist/crawl/web.db query "…", or attach it alongside the main corpus when you want unified results. The per-host textbook crawls stay separate — they are warrant substrate, resolved through a different path.

Source map

File	Role
arborist/sources/crawler/bridge.py	BFS driver, feed-skip, ingest, recrawl-check
arborist/sources/crawler/async_web_fetcher.py	fetch + robots + crawl-delay + shared session
arborist/ingest.py	document_root derivation (content, not URI)
arborist/merkle.py	Merkle tree / proof conventions