- Hydra mode: RSS/Atom/Sitemap feed discovery and persistence
- Media provenance: first_seen_at, last_seen_at, discovered_at tracking
- Job purge: --purge-job CLI and DELETE API endpoint
- Updated feature grid on about page
- Added code examples for hydra and purge
Order of operations (critical for data integrity):
1. Get job info and collect hashes/URIs to delete
2. Delete MediaSource records (breaks references)
3. Delete Page records
4. Delete orphan Media records (not referenced by other jobs)
5. Delete orphan media files from vault
6. Delete screenshot files for deleted pages
7. Delete CrawlJob record
8. Delete state files
CLI: python neopig.py --purge-job JOB_ID
API: DELETE /api/crawl/jobs/{job_id}?purge=true (default)
UI: Delete button now purges by default
- Add last_seen_at column to Media table (updated on every encounter)
- first_seen_at tracks initial discovery
- last_seen_at tracks most recent sighting
- MediaSource.discovered_at already tracks per-page first link time
- Enables tracking media reuse across the web
- Store discovered RSS/Atom/Sitemap feed URLs in hydra-{domain}.json
- ALWAYS check known feeds on every crawl (self-healing)
- --hydra flag discovers NEW feeds, adds them to persistent list
- fetch_feeds() accepts feed_urls param for known feeds
- Tracks which feeds actually return items for persistence
- Hydra state file (hydra-{domain}.json) never rotates
- Tracks seen URLs with first_seen timestamp, title, published date
- _filter_new_hydra_urls() returns only NEW URLs not seen before
- Enables detecting new posts when sites publish via RSS/Atom/Sitemap
- --hydra CLI flag and UI checkbox (off by default)
- discover_feeds(): finds RSS/Atom links and common sitemap URLs
- parse_rss(), parse_atom(), parse_sitemap(): extract URLs from feeds
- fetch_feeds(): async method to fetch and parse all discovered feeds
- hydra_urls injected into crawl queue with high priority (score 100)
- Supports nested sitemaps (sitemap index files)
- Store depth, max_pages, fast, screenshots in crawl_jobs table
- Resume uses stored settings instead of hardcoded defaults
- Progress bar shows pages_crawled / total instead of animation
- fetch_with_depth now uses asyncio.gather for parallel page fetches
- Semaphore limits concurrency to 3x CPU count in fast mode, 1 otherwise
- Pre-filters candidates before batch fetch to optimize
- Logs parallel fetch info when using multiple workers
- Per-job log capture to data/logs/{job_id}.log
- /api/crawl/jobs/{id}/logs endpoint to fetch logs
- Console button on each job card in crawl UI
- Modal with auto-scrolling, color-coded log levels
- Logs refresh every 2s while viewing running jobs
The filename fallback lookup was matching external links like
https://pypi.org/project/virt-back/ because the filename 'virt-back'
matched a media file in the database.
Now:
1. Filename fallback only applies to URLs with media extensions
2. href rewriting only applies to exact URI matches or media file URLs
3. External links are preserved as-is
The page_content extraction was decomposing nav/header/footer from the
soup BEFORE media extraction, causing header/logo images to be missed.
Now uses a fresh BeautifulSoup parse of the body for text extraction,
preserving the original soup for media extraction.
