engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

crawler: --fast flag exposes the verbatim AsyncWebFetcher fast_mode

Browse source 
The lifted AsyncWebFetcher already had a fast_mode constructor knob;
just unbroken plumbing was missing. fast_mode trades politeness for
throughput:

  - timeouts: 5s vs 15-60s
  - parallel page workers: cpu_count() * 3 vs 1
  - ignores robots.txt 'crawl-delay' directive
  - robots.txt 'Disallow' is STILL honored (only the delay drops)

Surface:

  aborist crawl --seed-url ... --fast
  make crawl-ingest URL=... FAST=1

Tests: two regressions pin that fast=True / fast=False each translate
to the right AsyncWebFetcher(fast_mode=...) construction. Capturing
fetcher fixture to avoid network. 21 bridge tests + 273 default
suite, all passing.
This commit is contained in:
russell@unturf.com 2026-04-28 21:13:58 -04:00
parent f04417161c
commit dfe39a0bf4
No known key found for this signature in database
4 changed files with 67 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

6
Makefile
View file

@ -327,9 +327,9 @@ test-crawler: bootstrap-crawler ## run only the lifted crawler tests
# HEAD requests later. URL is required; DEPTH and MAX have safe defaults.
CRAWL_DEPTH ?= 2
CRAWL_MAX ?= 0
crawl-ingest: bootstrap-crawler ## crawl URL=https://x.com [DEPTH=2 MAX=0] and ingest into $(DB) (MAX=0 = no cap, depth is only bound)
@if [ -z "$(URL)" ]; then echo "usage: make crawl-ingest URL=https://example.com [DEPTH=2 MAX=0]" >&2; exit 2; fi
$(ABORIST) --db $(DB) crawl --seed-url "$(URL)" --depth $(CRAWL_DEPTH) --max-pages $(CRAWL_MAX) --ingest
crawl-ingest: bootstrap-crawler ## crawl URL=https://x.com [DEPTH=2 MAX=0 FAST=1] and ingest into $(DB)
@if [ -z "$(URL)" ]; then echo "usage: make crawl-ingest URL=https://example.com [DEPTH=2 MAX=0 FAST=1]" >&2; exit 2; fi
$(ABORIST) --db $(DB) crawl --seed-url "$(URL)" --depth $(CRAWL_DEPTH) --max-pages $(CRAWL_MAX) $(if $(FAST),--fast,) --ingest
# Fast freshness probe: conditional HEAD per doc, classify fresh/stale/gone.
# Send only If-None-Match + If-Modified-Since headers — server returns 304