russell@unturf.com
dfe39a0bf4
The lifted AsyncWebFetcher already had a fast_mode constructor knob; just unbroken plumbing was missing. fast_mode trades politeness for throughput: - timeouts: 5s vs 15-60s - parallel page workers: cpu_count() * 3 vs 1 - ignores robots.txt 'crawl-delay' directive - robots.txt 'Disallow' is STILL honored (only the delay drops) Surface: aborist crawl --seed-url ... --fast make crawl-ingest URL=... FAST=1 Tests: two regressions pin that fast=True / fast=False each translate to the right AsyncWebFetcher(fast_mode=...) construction. Capturing fetcher fixture to avoid network. 21 bridge tests + 273 default suite, all passing. |
||
|---|---|---|
| .. | ||
| test_async_web_fetcher.py | ||
| test_bridge.py | ||
| test_web_fetch.py | ||