engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

crawler: --max-pages defaults to 0 (no cap); depth is the only bound

Browse source 
Per fox: 20 was a wrong default — typical sites have many more pages
within depth 2 than that, and the cap was silently truncating real
crawls. New behavior:

  aborist crawl --depth 2                  # no page cap
  aborist crawl --depth 2 --max-pages 50   # opt-in cap when needed
  make crawl-ingest URL=... DEPTH=2        # no page cap
  make crawl-ingest URL=... DEPTH=2 MAX=50 # explicit cap

Implementation: bridge's BFS loop treats max_pages=0 as "unbounded"
(while-condition becomes `queue and (max_pages == 0 or len(...) < max_pages)`).
Default flows from CLI argparse default=0 down to the bridge.

Tests: pinned max_pages=0 → no cap with a 30-page chain regression
test (19 passed in tests/crawler/test_bridge.py). Default suite still
273 passed, 1 skipped.
This commit is contained in:
russell@unturf.com 2026-04-28 20:59:22 -04:00
parent e2540eaaec
commit 552dc0def2
No known key found for this signature in database
4 changed files with 26 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

12
tests/crawler/test_bridge.py
View file

@ -102,6 +102,18 @@ def test_crawl_seed_respects_max_pages():
assert len(out) == 5
def test_crawl_seed_max_pages_zero_means_no_cap():
"""Default `--max-pages 0` lets depth be the only bound. Pin so a
future "let's set a sensible default" PR can't silently re-introduce
a cap."""
# Linear chain of 30 pages within depth 30; max_pages=0 should
# discover all of them.
page_links = {f"https://x.com/p{i}": [f"/p{i+1}"] for i in range(30)}
page_links["https://x.com/"] = ["/p0"]
out = _run_crawl(page_links, "https://x.com/", depth=30, max_pages=0)
assert len(out) >= 30 # seed + at least 30 chained pages
def test_crawl_seed_respects_max_depth():
page_links = {
"https://x.com/": ["/a"],