engineering/spider.unturf.com
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Migrated from https://git2.unturf.com/engineering/unturf/spider.unturf.com.git
11 commits 1 branch 0 tags 60 KiB
Find a file
Russell Ballestrini 0589138396 Add detailed crawl diagnostic logging to upstream spider 
Ported from Discord bot's async web fetcher improvements.

Add comprehensive logging to understand crawl behavior:
- Log crawl parameters at start (max_depth, URLs, keywords)
- Debug log for crawl queue state during processing
- Detailed link extraction stats with skip reasons:
  - Total links found
  - Links added to crawl queue
  - Links skipped by robots.txt
  - Links skipped (wrong domain)
  - Links skipped (max depth reached)

Applied to both:
- Fresh page fetching and link extraction
- Cached page link extraction for depth traversal

This diagnostic logging helps identify why crawlers find fewer
pages than expected (e.g., robots.txt blocking, domain filtering,
depth limits).

No crawl logic changes - purely diagnostic visibility.
2025-11-27 12:46:34 -05:00
.gitignore Add chromadb so we can do RAG 2025-04-26 00:22:43 -04:00
CHANGELOG.md Add detailed crawl diagnostic logging to upstream spider 2025-11-27 12:46:34 -05:00
requirements.txt feat: Add query-aware page scoring system for intelligent crawling 2025-11-27 11:35:04 -05:00
unturf_spider.py Add detailed crawl diagnostic logging to upstream spider 2025-11-27 12:46:34 -05:00