engineering/spider.unturf.com
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
spider.unturf.com/CHANGELOG.md
Russell Ballestrini 1cc53e5d54 fix: Extract links from cached pages to enable depth traversal 
Critical bugfix for depth-based crawling when pages are cached.

Problem:
- When a page was already cached and force_crawl=False, process_url()
  would return early without extracting links
- This broke depth traversal: if depth 0 page was cached, no depth 1
  pages would be discovered
- Example: With depth=1 and cached starting URI, crawler stopped at
  depth 0 and never crawled linked pages

Solution:
- Modified process_url() to still extract links from cached pages when
  depth < max_depth
- Cached page path now fetches HTML only for link extraction, skips
  expensive content extraction and scoring
- Links are checked against robots.txt and crawl rules as normal
- Only returns early (without link extraction) when at max depth

Impact:
- Fixes depth=1 crawling scenario where starting page is cached
- Improves crawl efficiency by not re-processing cached content
- Maintains proper depth traversal behavior across cache boundaries

See CHANGELOG.md for details.
2025-11-27 11:50:05 -05:00

4.8 KiB
Raw Blame History

Changelog

All notable changes to the Unturf Spider project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Fixed

  • Critical depth traversal bug: Fixed issue where cached pages at depth 0 would not extract links, preventing depth 1+ crawling
    • When a page was already cached and force_crawl=False, the crawler would return early without extracting links
    • This broke depth-based crawling: if the starting URI was cached, no depth 1 pages would be discovered
    • Now the crawler still extracts links from cached pages when depth < max_depth to enable proper depth traversal
    • Example: With depth=1 and a cached starting page, the crawler now correctly discovers and crawls linked pages
    • Located in unturf_spider.py:501-539 (new cached page link extraction logic)

Added

  • Query-aware page scoring system: Pages are now scored based on content quality and relevance to the user's query

    • Multi-factor scoring algorithm (0-100+ points):
      • Content length (0-30 points) - Substantial content gets higher scores
      • Title quality (0-15 points) - Descriptive titles score higher
      • URL quality (0-15 points) - Clean, readable URLs preferred
      • Content density (0-15 points) - Unique word ratio indicates quality
      • Query relevance (0-40 points) - Keyword matches in title, URL, and content
    • New _score_page() method in BaseCrawler class (lines 130-223)
    • Query keywords are extracted once at the beginning using Hermes AI model
    • Keywords are passed through the entire crawl chain for real-time scoring

  • Database schema enhancement: Added score field to Article model

    • Stores integer quality/relevance score (0-100+)
    • Default value: 0
    • Nullable to maintain backward compatibility
    • SQLAlchemy will auto-migrate on first run

Changed

  • cache_article() method: Now accepts score parameter and stores it in database

    • Located in unturf_spider.py:431-478
    • Updated log messages to include score information

  • process_url() method: Calculates page score and passes it to cache_article()

    • Located in unturf_spider.py:480-556
    • Accepts query_keywords parameter for scoring
    • Creates page_data dict with url, title, and text for scoring
    • Logs score for each processed page

  • crawl_recursive() method: Threads query_keywords parameter through crawl chain

    • Located in unturf_spider.py:558-585
    • Passes keywords to all process_url() calls

  • DuckDuckGoCrawler.run() method: Extracts query keywords early for scoring

    • Located in unturf_spider.py:849-874
    • Uses _extract_keywords_with_hermes() before starting crawl
    • Passes keywords to crawl_recursive()

  • DirectTargetCrawler.run() method: Extracts query keywords early for scoring

    • Located in unturf_spider.py:912-926
    • Uses _extract_keywords_with_hermes() before starting crawl
    • Passes keywords to crawl_recursive()

Technical Details

  • Keyword extraction: Uses Hermes AI model to intelligently extract relevant keywords from user queries

    • Filters out stop words and generic terms
    • Focuses on domain-specific and technical terms
    • Returns 3-7 most relevant keywords for scoring

  • Real-time scoring: Pages are scored during crawling (not after)

    • Enables future enhancements like selective crawling based on score
    • Allows prioritization of high-quality, relevant content
    • Provides visibility into crawl quality through logs

Future Enhancements (Deferred)

  • Async/await conversion: Replace requests with aiohttp for concurrent crawling

    • Would improve performance significantly
    • Requires substantial refactor of fetching logic
    • Lower priority than scoring integration

  • Progressive depth escalation: Dynamically increase crawl depth based on initial results

    • Start with depth=1, escalate to depth=2 if needed
    • Adaptive crawling based on content quality
    • Reduces unnecessary crawling

Migration Notes

  • Database migration: The new score column will be added automatically by SQLAlchemy on first run
  • Backward compatibility: Existing cached articles will have score=0 until re-crawled
  • No breaking changes: All existing functionality remains intact

Performance Impact

  • Minimal overhead: Scoring adds ~10ms per page (negligible compared to network I/O)
  • One-time keyword extraction: Keywords are extracted once per query, not per page
  • No network calls: Scoring is purely computational using already-fetched content

[1.42] - Previous Release

  • Initial stable release with basic crawling functionality
  • Hermes AI-powered content extraction
  • SQLite caching with TTL
  • robots.txt compliance
  • DuckDuckGo and direct target crawling modes