engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

crawler: filter bs4 XMLParsedAsHTMLWarning at subpackage boundary

Browse source 
A real BFS hits sitemap.xml / RSS feeds via the verbatim crawler's
generic page handler. bs4 warns "you're using an HTML parser on XML"
on each — benign (parsing still works) but spams stderr during a
deep crawl, drowning the progress heartbeat we just wired in.

Filter at aborist/sources/crawler/__init__.py so the lifted source
stays untouched. The filter applies whenever any caller imports from
this subpackage, including the CLI's `from ... bridge import ...`
(which runs __init__.py first per Python import semantics).
This commit is contained in:
russell@unturf.com 2026-04-28 21:07:27 -04:00
parent d2ceba201e
commit f04417161c
No known key found for this signature in database
1 changed files with 11 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

11
aborist/sources/crawler/__init__.py
View file

@ -39,6 +39,17 @@ honor ``robots.txt`` — that's what the underlying parser already does
when given a URL.
"""
# Suppress bs4's "you're parsing XML with the HTML parser" warning when the
# crawler hits a sitemap.xml / RSS feed via the generic page handler. Benign
# (parsing still works) but spams stderr during a deep BFS. Filtering here —
# at the aborist subpackage boundary — keeps the verbatim lift untouched.
import warnings as _warnings
try:
from bs4 import XMLParsedAsHTMLWarning as _XMLParsedAsHTMLWarning
_warnings.filterwarnings("ignore", category=_XMLParsedAsHTMLWarning)
except ImportError: # pragma: no cover
pass
from aborist.sources.crawler.async_web_fetcher import (
AsyncWebFetcher,
CrawlMode,