Russell Ballestrini
1cc53e5d54
Critical bugfix for depth-based crawling when pages are cached. Problem: - When a page was already cached and force_crawl=False, process_url() would return early without extracting links - This broke depth traversal: if depth 0 page was cached, no depth 1 pages would be discovered - Example: With depth=1 and cached starting URI, crawler stopped at depth 0 and never crawled linked pages Solution: - Modified process_url() to still extract links from cached pages when depth < max_depth - Cached page path now fetches HTML only for link extraction, skips expensive content extraction and scoring - Links are checked against robots.txt and crawl rules as normal - Only returns early (without link extraction) when at max depth Impact: - Fixes depth=1 crawling scenario where starting page is cached - Improves crawl efficiency by not re-processing cached content - Maintains proper depth traversal behavior across cache boundaries See CHANGELOG.md for details. |
||
|---|---|---|
| .gitignore | ||
| CHANGELOG.md | ||
| requirements.txt | ||
| unturf_spider.py | ||