Russell Ballestrini
f438c09171
Implement multi-factor page scoring to prioritize high-quality, relevant content during crawls. Key changes: - Add score field to Article model for storing quality/relevance scores (0-100+) - Implement _score_page() method with 5 scoring factors: * Content length (0-30 points) * Title quality (0-15 points) * URL quality (0-15 points) * Content density (0-15 points) * Query relevance (0-40 points) - Thread query_keywords parameter through entire crawl chain - Extract keywords early using Hermes AI for real-time scoring - Update cache_article() to store scores in database - Update process_url() to calculate and log page scores - Update both DuckDuckGoCrawler and DirectTargetCrawler to extract keywords Benefits: - Pages are scored during crawling based on query relevance - Enables future selective crawling based on score thresholds - Provides visibility into crawl quality through detailed logs - Minimal performance overhead (~10ms per page) Future enhancements deferred: - Async/await conversion with aiohttp - Progressive depth escalation based on results See CHANGELOG.md for detailed documentation.
36 lines
1.2 KiB
Text
36 lines
1.2 KiB
Text
# requirements.txt
|
|
# Specifies Python dependencies for unturf_spider.py.
|
|
# Install with: pip install -r requirements.txt
|
|
# Versions pinned for compatibility with Python 3.13.
|
|
|
|
# HTTP requests for web crawling and API calls
|
|
# for HTTP requests to fetch pages and call APIs (e.g., DuckDuckGo, Hermes).
|
|
requests
|
|
|
|
# HTML parsing for extracting content from web pages
|
|
# Used for parsing HTML with BeautifulSoup to extract article text.
|
|
beautifulsoup4
|
|
|
|
# SQL database management for caching articles
|
|
# Used for SQLite database operations to store and query articles.
|
|
sqlalchemy
|
|
|
|
# OpenAI client for interacting with Hermes API endpoints
|
|
# Used for API calls to Hermes endpoints (hermes.ai.unturf.com/v1, hermes2).
|
|
openai
|
|
|
|
# URL parsing and robots.txt handling
|
|
# Used for URL parsing and robots.txt via urllib.robotparser.
|
|
urllib3
|
|
|
|
# Text analysis for TF-IDF similarity search
|
|
# Used for TfidfVectorizer and cosine_similarity to rank articles.
|
|
scikit-learn
|
|
|
|
# Numerical computations for text analysis
|
|
# Used by scikit-learn for TF-IDF and cosine similarity calculations.
|
|
numpy
|
|
|
|
# SSL certificate verification for secure HTTP requests
|
|
# Used to ensure secure HTTPS connections with updated SSL certificates.
|
|
certifi
|