perf+cleanup: skip feeds in crawl discovery; lxml link extraction

Two crawler-discovery changes surfaced while chasing fast-crawl wall
time on russell.ballestrini.net:

1. Feed-skip in BFS discovery: the bridge fetched feed/sitemap URLs
   (a multi-MB atom.xml among them) only for ingest_crawled to discard
   them. Gate enqueue on the existing _looks_like_feed_url so we never
   fetch crawl-infrastructure URLs — less wasted work and one fewer
   slow wave straggler.

2. lxml link extraction, DRY'd: the three duplicated BeautifulSoup
   html.parser closures (fresh fetch + 2 cache paths) collapse into one
   module-level extract_page_links() backed by lxml.html (C parser,
   releases the GIL so to_thread actually parallelises) with a BS4
   fallback for markup lxml rejects. Parse on a 24-page wave 3.5s->2.5s.

Honest scope: neither moves full-crawl wall time much — measurement
showed the dominant cost is the per-page HEAD+GET double round-trip on
a per-call ClientSession, not parsing. These are correct-and-cleaner
on their own; the wall-time lever (shared session + drop redundant
HEAD) is a separate change. lxml extraction is regression-pinned
against the BS4 fallback for parity.

tests/crawler/test_bridge.py

@@ -138,6 +138,26 @@ def test_crawl_seed_drops_off_domain_links():
     assert "https://x.com/local" in out
 
 
+def test_crawl_seed_skips_feed_urls_during_discovery():
+    """Feed/sitemap links must not be crawled — they're discarded at
+    ingest and a big feed is a slow wave straggler. Pin that the BFS
+    drops them at enqueue (not just at ingest)."""
+    page_links = {
+        "https://x.com/": [
+            "/a",
+            "/feeds/all.atom.xml",  # big feed — must be skipped
+            "/sitemap.xml",
+            "/b.rss",
+        ],
+        "https://x.com/a": [],
+    }
+    out = _run_crawl(page_links, "https://x.com/", depth=2, max_pages=10)
+    assert "https://x.com/a" in out
+    assert "https://x.com/feeds/all.atom.xml" not in out
+    assert "https://x.com/sitemap.xml" not in out
+    assert "https://x.com/b.rss" not in out
+
+
 def test_crawl_seed_dedups():
     page_links = {
         "https://x.com/": ["/a", "/a", "/b"],