- Add /api/crawl/jobs/{job_id}/replay SSE endpoint for replaying crawls
- Replay streams media at original discovery pace with speed multiplier
- Add Replay button to crawl jobs UI (works even when crawling disabled)
- Track active replays to prevent double-clicking same job
- Live feed page handles ?replay=JOB_ID parameter for replay mode
- Add NEOPIG_NO_VCS=1 env var to skip VCS repository detection
- Fix hljs.highlightAll() call when highlight.js not loaded
- database.py: Add media_uri to get_page_media and search queries
- View gallery: Show filename under each thumbnail
- Search results: Show filename above hash
- Live feed: Show filename as card title
Order of operations (critical for data integrity):
1. Get job info and collect hashes/URIs to delete
2. Delete MediaSource records (breaks references)
3. Delete Page records
4. Delete orphan Media records (not referenced by other jobs)
5. Delete orphan media files from vault
6. Delete screenshot files for deleted pages
7. Delete CrawlJob record
8. Delete state files
CLI: python neopig.py --purge-job JOB_ID
API: DELETE /api/crawl/jobs/{job_id}?purge=true (default)
UI: Delete button now purges by default
- Add last_seen_at column to Media table (updated on every encounter)
- first_seen_at tracks initial discovery
- last_seen_at tracks most recent sighting
- MediaSource.discovered_at already tracks per-page first link time
- Enables tracking media reuse across the web
- Store depth, max_pages, fast, screenshots in crawl_jobs table
- Resume uses stored settings instead of hardcoded defaults
- Progress bar shows pages_crawled / total instead of animation
- README.md: Complete rewrite in About page style with origin story
- Documents all storage systems: filevault, ImageVault, domain_vault
- Full database schema and method reference
- Screenshot capture with all engines and config options
- HTML to Markdown converter with content detection
- CLI reference for neopig.py, archive.py, serp.py
- All API endpoints documented
- Add lookup_pages_by_uris() for batch page hash lookup
- Rewrite internal links (same domain) to /page/{uri_hash}
- Links to archived pages now stay within neopig
- Fix asyncio error: use create_task() directly instead of BackgroundTasks
- Default depth=9, max_pages=-1 for full site crawls
- Add update_crawl_job_stats() for live progress updates
- Poll jobs every 2s while running, show live stats
- Animated progress bar for running jobs
- Fix get_media_type_from_extension() None path crash
- ProcessPoolExecutor: 1 process per CPU core (bypasses GIL)
- ThreadPoolExecutor: 6 threads per process for I/O throughput
- Thread-local SQLite connections with WAL mode
- Manager().Value() for cross-process progress counter
- Real-time tqdm updates polling shared counter every 50ms
- ~19 pages/sec on 4-core system (4887 pages in 2 min)
- Bootstrap C program extracts serve.py and runs from tarball
- Archives now include neopig source files for self-contained crawling
- --upgrade-neopig flag to update neopig in existing archives
- html2md.py: smart HTML-to-markdown converter for forums/blogs/Q&A
- Fix vault path defaults (data/vault instead of vault)
- Streaming tar.gz creation without temp copies
- URL rewriting for local media references in archives
- backfill-markdown: regenerate markdown with absolute URLs for a domain
- Download button on /view page with slugified filename from alt/title
- html2text baseurl resolves relative URLs during crawl
- Fallback URL lookup by filename for older crawls
- Video hover-to-play on all video elements
- Avatar CSS float:left for text wrapping
- FileResponse inline disposition (view not download)
- Dynamic screenshot delay based on content length
