Critical bugfix for depth-based crawling when pages are cached.
Problem:
- When a page was already cached and force_crawl=False, process_url()
would return early without extracting links
- This broke depth traversal: if depth 0 page was cached, no depth 1
pages would be discovered
- Example: With depth=1 and cached starting URI, crawler stopped at
depth 0 and never crawled linked pages
Solution:
- Modified process_url() to still extract links from cached pages when
depth < max_depth
- Cached page path now fetches HTML only for link extraction, skips
expensive content extraction and scoring
- Links are checked against robots.txt and crawl rules as normal
- Only returns early (without link extraction) when at max depth
Impact:
- Fixes depth=1 crawling scenario where starting page is cached
- Improves crawl efficiency by not re-processing cached content
- Maintains proper depth traversal behavior across cache boundaries
See CHANGELOG.md for details.
Implement multi-factor page scoring to prioritize high-quality, relevant content during crawls.
Key changes:
- Add score field to Article model for storing quality/relevance scores (0-100+)
- Implement _score_page() method with 5 scoring factors:
* Content length (0-30 points)
* Title quality (0-15 points)
* URL quality (0-15 points)
* Content density (0-15 points)
* Query relevance (0-40 points)
- Thread query_keywords parameter through entire crawl chain
- Extract keywords early using Hermes AI for real-time scoring
- Update cache_article() to store scores in database
- Update process_url() to calculate and log page scores
- Update both DuckDuckGoCrawler and DirectTargetCrawler to extract keywords
Benefits:
- Pages are scored during crawling based on query relevance
- Enables future selective crawling based on score thresholds
- Provides visibility into crawl quality through detailed logs
- Minimal performance overhead (~10ms per page)
Future enhancements deferred:
- Async/await conversion with aiohttp
- Progressive depth escalation based on results
See CHANGELOG.md for detailed documentation.
