engineering/pig.py
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Fix: header/logo images were not downloaded due to soup mutation

Browse source 
The page_content extraction was decomposing nav/header/footer from the
soup BEFORE media extraction, causing header/logo images to be missed.

Now uses a fresh BeautifulSoup parse of the body for text extraction,
preserving the original soup for media extraction.
This commit is contained in:
Russell Ballestrini 2025-12-31 18:01:27 -05:00
parent 2e48e83c9f
commit 0a5762fbc4
1 changed files with 6 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

9
async_web_fetcher.py
View file

@ -471,13 +471,16 @@ def extract_media_from_html(html: str, base_url: str, mode: CrawlMode = CrawlMod
# Extract page content (body text) for full-text search
# This enables blog images to be searchable by post content
# IMPORTANT: Parse a fresh copy so we don't remove header/nav images from the soup
# that will be used for media extraction below
page_content = ''
body = soup.find('body')
if body:
# Remove script, style, nav, footer elements
for tag in body.find_all(['script', 'style', 'nav', 'footer', 'header', 'aside']):
body_soup = BeautifulSoup(str(body), 'html.parser')
# Remove script, style, nav, footer elements from the COPY only
for tag in body_soup.find_all(['script', 'style', 'nav', 'footer', 'header', 'aside']):
tag.decompose()
page_content = body.get_text(separator=' ', strip=True)[:10000] # Limit to 10k chars
page_content = body_soup.get_text(separator=' ', strip=True)[:10000] # Limit to 10k chars
def get_context_for_element(element) -> dict:
"""Extract contextual metadata from surrounding HTML elements."""