The lifted AsyncWebFetcher already had a fast_mode constructor knob;
just unbroken plumbing was missing. fast_mode trades politeness for
throughput:
- timeouts: 5s vs 15-60s
- parallel page workers: cpu_count() * 3 vs 1
- ignores robots.txt 'crawl-delay' directive
- robots.txt 'Disallow' is STILL honored (only the delay drops)
Surface:
aborist crawl --seed-url ... --fast
make crawl-ingest URL=... FAST=1
Tests: two regressions pin that fast=True / fast=False each translate
to the right AsyncWebFetcher(fast_mode=...) construction. Capturing
fetcher fixture to avoid network. 21 bridge tests + 273 default
suite, all passing.
russell@unturf.com
dfe39a0bf4