arborist/tests/crawler
russell@unturf.com e2540eaaec
crawler/bridge: pass extract_links=True so BFS actually expands
The verbatim AsyncWebFetcher.fetch_webpage defaults extract_links=False
and returns (html, []) when omitted. Bridge's BFS therefore terminated
at the seed — a real-world crawl of russell.ballestrini.net discovered
exactly 1 URL (the seed). One-line fix; the kwarg was always there in
the lifted API.

Added test_crawl_seed_passes_extract_links_true that records every
fetch_webpage call and asserts the kwarg is True. Pin so this can't
silently regress when bridge internals get reorganized.

Updated the stub fetcher signature in the existing tests to accept
the new kwarg without behavior change.
2026-04-28 20:56:14 -04:00
..
test_async_web_fetcher.py crawler: verbatim lift from agents.ai.unturf.com (off by default) 2026-04-28 20:23:36 -04:00
test_bridge.py crawler/bridge: pass extract_links=True so BFS actually expands 2026-04-28 20:56:14 -04:00
test_web_fetch.py crawler: verbatim lift from agents.ai.unturf.com (off by default) 2026-04-28 20:23:36 -04:00