docs: web crawler guide — discovery, fast mode, dedupe, orphans
New docs/crawler.md covering the crawl path we actually run: BFS same-host discovery, robots/feed/sitemap handling, polite vs --fast, the shared-session + no-HEAD + crawl-delay-fix speedups, and the content-addressed payoff. Leads on the two store-derived diagnostics: duplicate detection (group by document_root — body, not URI) and partial-overlap (shared chunk leaves), plus orphan finding (sitemap − BFS-reached) and the planned crawl-report webmaster tools. Honest pros/cons: orphans invisible to crawl by design, single-host, no JS execution, --fast is anti-social off your own turf.
This commit is contained in:
parent
dee689cd91
commit
24c7596bc4
1 changed files with 246 additions and 0 deletions
246
docs/crawler.md
Normal file
246
docs/crawler.md
Normal file
|
|
@ -0,0 +1,246 @@
|
|||
# Web crawler — surface ingest from live sites
|
||||
|
||||
Arborist's crawler turns a live website into surface-layer documents: it
|
||||
walks a site from a seed URL, fetches each page, and runs the HTML
|
||||
through the same normalize → chunk → Merkle → upsert pipeline as any
|
||||
other source. The output lands in a content-addressed store, which is
|
||||
what makes the crawler's two headline diagnostics — **duplicate
|
||||
detection** and **orphan finding** — nearly free.
|
||||
|
||||
It pairs with the textbook-ingest workflow (`docs/warrant-substrate-cookbook.md`)
|
||||
and the retrieval pipeline (`arborist/qa/query.py`).
|
||||
|
||||
> Status: discovery + ingest + recrawl-check ship today. The webmaster
|
||||
> report (`crawl-report`) described under [Diagnostics](#diagnostics-the-webmaster-tools-angle)
|
||||
> is designed-not-built — this doc pins the contract.
|
||||
|
||||
## Default off
|
||||
|
||||
The crawler touches the network only when you ask it to, and its heavy
|
||||
dependencies (`aiohttp`, `beautifulsoup4`, `lxml`, …) are an opt-in
|
||||
extra:
|
||||
|
||||
```
|
||||
pip install 'arborist[crawler]' # or: make bootstrap-crawler
|
||||
```
|
||||
|
||||
The default test suite never imports the crawler module. Nothing in a
|
||||
normal lookup / ingest / Q&A path reaches it.
|
||||
|
||||
## How discovery works
|
||||
|
||||
The crawl driver is `arborist/sources/crawler/bridge.py`
|
||||
(`_crawl_seed_async`). It is a **breadth-first walk that stays on one
|
||||
hostname**:
|
||||
|
||||
1. Start from `--seed-url`. Fetch it, extract its links.
|
||||
2. Keep links that are **same exact hostname** (`_same_domain`: no
|
||||
subdomain crossover — `x.com` does not follow into `docs.x.com`),
|
||||
`http(s)`, and not already seen.
|
||||
3. Enqueue the survivors at `depth + 1`. Repeat until the queue drains,
|
||||
`--depth` is hit, or `--max-pages` is reached (`0` = no cap, depth is
|
||||
the only bound).
|
||||
|
||||
Discovery fetches each page only far enough to extract its links; the
|
||||
arborist-side ingest is a separate step (`--ingest`).
|
||||
|
||||
### What it refuses to crawl
|
||||
|
||||
- **`robots.txt` Disallow is always honored** — in both polite and fast
|
||||
mode. A blocked URL is skipped, never fetched.
|
||||
- **Feeds and sitemaps are skipped at discovery** (`_looks_like_feed_url`):
|
||||
`*.atom`, `*.rss`, `*.rdf`, `/feed/`, `/feeds/`, `sitemap.xml`,
|
||||
`sitemap_index.xml`, `/sitemaps/*.xml`, and the WordPress legacy feed
|
||||
names. They are crawl *infrastructure*, not knowledge — `ingest_crawled`
|
||||
would discard them anyway, and a multi-MB `atom.xml` is a slow
|
||||
straggler that gates everything fetched alongside it. A second filter
|
||||
at ingest (URL pattern **and** response `Content-Type`) catches feeds
|
||||
served at unconventional paths. `xhtml` is preserved — it *is* HTML.
|
||||
|
||||
The crawler **never proactively probes** `/sitemap.xml` or guesses feed
|
||||
locations on the path we use. (The fetcher carries an unused
|
||||
sitemap-discovery `crawl()` strategy method from its upstream lift; the
|
||||
arborist bridge calls only `fetch_webpage`.)
|
||||
|
||||
## Polite vs fast
|
||||
|
||||
`--fast` flips the underlying `AsyncWebFetcher` into `fast_mode`. The one
|
||||
rule that matters: **fast mode drops the rate limit, never the access
|
||||
rules.**
|
||||
|
||||
| | polite (default) | `--fast` |
|
||||
|---|---|---|
|
||||
| robots `Disallow` | honored | honored |
|
||||
| robots `crawl-delay` | honored; **2.0s** fallback when robots is silent/absent | **ignored** (0s) |
|
||||
| concurrency per wave | 1 (sequential) | `CPU × 3` |
|
||||
| connections | shared keepalive session | shared keepalive session |
|
||||
| per-page requests | one `GET` | one `GET` |
|
||||
| page `GET` timeout | 15s (robots 5s) | 15s (robots 5s) |
|
||||
|
||||
Use `--fast` only where aggressive fetching is acceptable: your own
|
||||
sites, or dumps you have been authorized to mirror. On someone else's
|
||||
infrastructure, the polite default is the courteous neighbour.
|
||||
|
||||
### Why fast is fast
|
||||
|
||||
Each fetch wave runs up to `CPU × 3` requests concurrently through **one
|
||||
shared `aiohttp` session** with a keepalive connector, so connections
|
||||
are reused across the whole crawl instead of paying a fresh TCP+TLS per
|
||||
page. There is **no preflight `HEAD`** — `aiohttp` exposes response
|
||||
headers before the body is read, so the binary content-type guard skips
|
||||
images/video/audio without downloading them, and the second round trip
|
||||
the old `HEAD` cost is gone. Link extraction uses `lxml` (a C parser
|
||||
that releases the GIL), so parsing genuinely parallelises across a wave
|
||||
rather than serialising on the interpreter lock.
|
||||
|
||||
Measured on a robots-less site (`russell.ballestrini.net`, 242 URIs),
|
||||
these together took a full `--fast` crawl from ~26s to ~5s. The largest
|
||||
single factor was making fast mode actually ignore the crawl-delay on a
|
||||
site whose `robots.txt` 404s — previously it fell back to the 2s default
|
||||
and every concurrent wave slept.
|
||||
|
||||
## Ingest and recrawl
|
||||
|
||||
With `--ingest`, each discovered page is fetched with `httpx`, parsed,
|
||||
and run through `ingest_source`, so crawled documents land in the same
|
||||
tables as Wikipedia or textbook sources. The crawler additionally
|
||||
records each document's `ETag` and `Last-Modified` in
|
||||
`document_http_meta`, so `arborist crawler recrawl-check` can later send
|
||||
conditional `HEAD` requests and classify each URL **fresh** (304) /
|
||||
**stale** (200) / **gone** (404) without re-downloading bodies.
|
||||
|
||||
Re-ingest is idempotent: same content → same `document_root` → no-op.
|
||||
Same URI with changed content → a new document plus a `supersedes` edge
|
||||
(lossless history).
|
||||
|
||||
## The content-addressed payoff
|
||||
|
||||
This is where the crawler differs from an off-the-shelf spider. Every
|
||||
document's identity is its **Merkle root over canonicalized body text**
|
||||
(`ingest.py:_compute_artifacts`):
|
||||
|
||||
```
|
||||
text = canonicalize(content) # body only — norm-v1
|
||||
leaves = [hash_leaf(chunk) for chunk in chunker.split(text)]
|
||||
document_root = MerkleTree.build(leaves).root
|
||||
```
|
||||
|
||||
The **URI is not part of the root** — it is a separate `document_uri`
|
||||
column. Two consequences fall straight out of that:
|
||||
|
||||
### Duplicate detection (free)
|
||||
|
||||
Two different URLs whose bodies canonicalize to the same text produce
|
||||
the **same `document_root`**. So exact-and-near-duplicate pages are a
|
||||
`GROUP BY document_root HAVING COUNT(*) > 1` query — no re-crawl, no
|
||||
heuristics. Because the body is `canonicalize()`d before hashing,
|
||||
trivial whitespace/markup differences collapse too, which is the right
|
||||
behaviour for a "duplicate content" signal.
|
||||
|
||||
Leaves are **per chunk**, so the same data also yields **partial
|
||||
overlap**: pages that share chunk-leaf hashes without being full dupes
|
||||
(boilerplate headers/footers, syndicated sections) cluster via a
|
||||
`chunks` join. No other crawler hands you "these N URLs are byte- or
|
||||
chunk-identical" for nothing — it is a property of the store's design.
|
||||
|
||||
### Orphan finding
|
||||
|
||||
Crawl gives the **link-reachable set** (what BFS discovered). A
|
||||
sitemap — fetched read-only, *once*, never ingested — gives the
|
||||
**declared set**. The difference is the orphans:
|
||||
|
||||
```
|
||||
orphans = sitemap_urls − bfs_reachable_urls
|
||||
```
|
||||
|
||||
These are pages a site lists but nothing links to. Surfacing them is a
|
||||
classic webmaster need and costs us a single extra fetch on top of a
|
||||
crawl we already ran.
|
||||
|
||||
## Diagnostics: the webmaster-tools angle
|
||||
|
||||
The store's by-products make a small "webmaster report" cheap. Planned
|
||||
shape: `arborist crawl-report --seed-url … [--sitemap …]`, off by
|
||||
default, read-only, never writing `providence_cache` or `audit_events`.
|
||||
|
||||
| Signal | How it's derived | Extra cost |
|
||||
|---|---|---|
|
||||
| Orphan pages | `sitemap − BFS-reached` | one sitemap fetch |
|
||||
| Exact duplicate pages | group by `document_root` | free (store query) |
|
||||
| Partial-overlap clusters | shared chunk-leaf hashes | free (store query) |
|
||||
| Broken links / 404s | per-URL status during crawl | free |
|
||||
| Redirect chains | 301/308 hops observed | free |
|
||||
| Depth / crawl-budget map | BFS depth per URL | free |
|
||||
| Thin / untitled pages | chunk count + missing `<title>` | free |
|
||||
|
||||
A likely v1 keeps it to **orphans + exact dupes + broken links**, with
|
||||
partial-overlap and the rest as follow-ons.
|
||||
|
||||
## Pros and cons
|
||||
|
||||
**Pros**
|
||||
|
||||
- **Compliant by default.** robots `Disallow` always honored;
|
||||
crawl-delay honored unless you explicitly opt into `--fast`.
|
||||
- **Fast when you own the target.** Concurrent waves, keepalive
|
||||
sessions, no redundant `HEAD`, GIL-releasing parse.
|
||||
- **Duplicate and partial-overlap detection for free**, because identity
|
||||
is content not URL.
|
||||
- **Idempotent, lossless re-ingest** with a conditional-`HEAD`
|
||||
recrawl-check that avoids re-downloading unchanged pages.
|
||||
- **Same store as every other source** — crawled pages are queryable,
|
||||
distillable, and Merkle-verifiable like any document.
|
||||
|
||||
**Cons / trade-offs**
|
||||
|
||||
- **Orphans are invisible to the crawl itself.** We follow links and do
|
||||
not consume sitemaps for discovery, so a page reachable *only* via the
|
||||
sitemap is never ingested. That is a deliberate default (crawl what is
|
||||
linked) — orphan *finding* is a report you run on top, not a crawl
|
||||
behaviour.
|
||||
- **Single hostname, exact match.** No subdomain crossover and no
|
||||
cross-domain following by design; multi-host sites need multiple
|
||||
seeds.
|
||||
- **No JavaScript execution.** Links and content rendered client-side
|
||||
(SPA routes) are not seen — this is an HTML fetcher, not a headless
|
||||
browser.
|
||||
- **`--fast` is anti-social off your own turf.** Ignoring crawl-delay
|
||||
and fanning out `CPU × 3` is fine for sites you control and rude
|
||||
elsewhere. The default is polite for a reason.
|
||||
- **Diverged from upstream.** Arborist's `AsyncWebFetcher` no longer
|
||||
matches the `agents.ai.unturf.com/core` verbatim lift (shared session,
|
||||
dropped `HEAD`, fast-mode crawl-delay fix). A candidate to upstream.
|
||||
|
||||
## CLI and make targets
|
||||
|
||||
```
|
||||
arborist crawl --seed-url URL [--depth N] [--max-pages N] [--ingest] [--fast] [--author NAME]
|
||||
arborist crawler recrawl-check [--domain D] [--limit N]
|
||||
```
|
||||
|
||||
- `--depth` — max BFS depth (default 2).
|
||||
- `--max-pages` — cap discovery at N URLs (`0` = no cap, depth bounds it).
|
||||
- `--ingest` — ingest discovered pages; without it, the URL list is
|
||||
printed only.
|
||||
- `--fast` — fast mode (see table). Disallow still honored.
|
||||
- `--author` — default author surname appended to titles for warrant
|
||||
resolution (only with `--ingest`).
|
||||
|
||||
Make targets drive the textbook-crawl workflow; crawl shards land in
|
||||
`~/.arborist/crawl/` (separate from the main `~/.arborist/shards` so
|
||||
SQLite's attached-DB limit isn't tripped):
|
||||
|
||||
```
|
||||
make crawl-textbooks # BFS-crawl every manifest entry with a crawl_url
|
||||
make textbook ID=<id> # ingest one textbook by id
|
||||
make crawl-textbooks-stats # docs-per-shard summary
|
||||
```
|
||||
|
||||
## Source map
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `arborist/sources/crawler/bridge.py` | BFS driver, feed-skip, ingest, recrawl-check |
|
||||
| `arborist/sources/crawler/async_web_fetcher.py` | fetch + robots + crawl-delay + shared session |
|
||||
| `arborist/ingest.py` | `document_root` derivation (content, not URI) |
|
||||
| `arborist/merkle.py` | Merkle tree / proof conventions |
|
||||
Loading…
Add table
Add a link
Reference in a new issue