russell@unturf.com 552dc0def2 crawler: --max-pages defaults to 0 (no cap); depth is the only bound ... Per fox: 20 was a wrong default — typical sites have many more pages within depth 2 than that, and the cap was silently truncating real crawls. New behavior: aborist crawl --depth 2 # no page cap aborist crawl --depth 2 --max-pages 50 # opt-in cap when needed make crawl-ingest URL=... DEPTH=2 # no page cap make crawl-ingest URL=... DEPTH=2 MAX=50 # explicit cap Implementation: bridge's BFS loop treats max_pages=0 as "unbounded" (while-condition becomes `queue and (max_pages == 0 or len(...) < max_pages)`). Default flows from CLI argparse default=0 down to the bridge. Tests: pinned max_pages=0 → no cap with a 30-page chain regression test (19 passed in tests/crawler/test_bridge.py). Default suite still 273 passed, 1 skipped.		2026-04-28 20:59:22 -04:00
..
test_async_web_fetcher.py	crawler: verbatim lift from agents.ai.unturf.com (off by default)	2026-04-28 20:23:36 -04:00
test_bridge.py	crawler: --max-pages defaults to 0 (no cap); depth is the only bound	2026-04-28 20:59:22 -04:00
test_web_fetch.py	crawler: verbatim lift from agents.ai.unturf.com (off by default)	2026-04-28 20:23:36 -04:00