engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

crawler: --max-pages defaults to 0 (no cap); depth is the only bound

Browse source 
Per fox: 20 was a wrong default — typical sites have many more pages
within depth 2 than that, and the cap was silently truncating real
crawls. New behavior:

  aborist crawl --depth 2                  # no page cap
  aborist crawl --depth 2 --max-pages 50   # opt-in cap when needed
  make crawl-ingest URL=... DEPTH=2        # no page cap
  make crawl-ingest URL=... DEPTH=2 MAX=50 # explicit cap

Implementation: bridge's BFS loop treats max_pages=0 as "unbounded"
(while-condition becomes `queue and (max_pages == 0 or len(...) < max_pages)`).
Default flows from CLI argparse default=0 down to the bridge.

Tests: pinned max_pages=0 → no cap with a 30-page chain regression
test (19 passed in tests/crawler/test_bridge.py). Default suite still
273 passed, 1 skipped.
This commit is contained in:
russell@unturf.com 2026-04-28 20:59:22 -04:00
parent e2540eaaec
commit 552dc0def2
No known key found for this signature in database
4 changed files with 26 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

6
Makefile
View file

@ -326,9 +326,9 @@ test-crawler: bootstrap-crawler ## run only the lifted crawler tests
# capture ETag/Last-Modified for each so recrawl-check can do conditional
# HEAD requests later. URL is required; DEPTH and MAX have safe defaults.
CRAWL_DEPTH ?= 2
CRAWL_MAX ?= 20
crawl-ingest: bootstrap-crawler ## crawl URL=https://x.com [DEPTH=2 MAX=20] and ingest into $(DB)
@if [ -z "$(URL)" ]; then echo "usage: make crawl-ingest URL=https://example.com [DEPTH=2 MAX=20]" >&2; exit 2; fi
CRAWL_MAX ?= 0
crawl-ingest: bootstrap-crawler ## crawl URL=https://x.com [DEPTH=2 MAX=0] and ingest into $(DB) (MAX=0 = no cap, depth is only bound)
@if [ -z "$(URL)" ]; then echo "usage: make crawl-ingest URL=https://example.com [DEPTH=2 MAX=0]" >&2; exit 2; fi
$(ABORIST) --db $(DB) crawl --seed-url "$(URL)" --depth $(CRAWL_DEPTH) --max-pages $(CRAWL_MAX) --ingest
# Fast freshness probe: conditional HEAD per doc, classify fresh/stale/gone.