Implement multi-factor page scoring to prioritize high-quality, relevant content during crawls.
Key changes:
- Add score field to Article model for storing quality/relevance scores (0-100+)
- Implement _score_page() method with 5 scoring factors:
* Content length (0-30 points)
* Title quality (0-15 points)
* URL quality (0-15 points)
* Content density (0-15 points)
* Query relevance (0-40 points)
- Thread query_keywords parameter through entire crawl chain
- Extract keywords early using Hermes AI for real-time scoring
- Update cache_article() to store scores in database
- Update process_url() to calculate and log page scores
- Update both DuckDuckGoCrawler and DirectTargetCrawler to extract keywords
Benefits:
- Pages are scored during crawling based on query relevance
- Enables future selective crawling based on score thresholds
- Provides visibility into crawl quality through detailed logs
- Minimal performance overhead (~10ms per page)
Future enhancements deferred:
- Async/await conversion with aiohttp
- Progressive depth escalation based on results
See CHANGELOG.md for detailed documentation.
