russell@unturf.com
9e196bcd82
Two crawler-discovery changes surfaced while chasing fast-crawl wall time on russell.ballestrini.net: 1. Feed-skip in BFS discovery: the bridge fetched feed/sitemap URLs (a multi-MB atom.xml among them) only for ingest_crawled to discard them. Gate enqueue on the existing _looks_like_feed_url so we never fetch crawl-infrastructure URLs — less wasted work and one fewer slow wave straggler. 2. lxml link extraction, DRY'd: the three duplicated BeautifulSoup html.parser closures (fresh fetch + 2 cache paths) collapse into one module-level extract_page_links() backed by lxml.html (C parser, releases the GIL so to_thread actually parallelises) with a BS4 fallback for markup lxml rejects. Parse on a 24-page wave 3.5s->2.5s. Honest scope: neither moves full-crawl wall time much — measurement showed the dominant cost is the per-page HEAD+GET double round-trip on a per-call ClientSession, not parsing. These are correct-and-cleaner on their own; the wall-time lever (shared session + drop redundant HEAD) is a separate change. lxml extraction is regression-pinned against the BS4 fallback for parity. |
||
|---|---|---|
| .. | ||
| test_async_web_fetcher.py | ||
| test_bridge.py | ||
| test_web_fetch.py | ||