- Fix asyncio error: use create_task() directly instead of BackgroundTasks
- Default depth=9, max_pages=-1 for full site crawls
- Add update_crawl_job_stats() for live progress updates
- Poll jobs every 2s while running, show live stats
- Animated progress bar for running jobs
- Fix get_media_type_from_extension() None path crash
- Use lxml parser (2-5x faster than html.parser), fallback to html.parser
- Combine 20+ noise selectors into single query (1 DOM traversal vs 20)
- Add hint parameter to skip content detection for known types
- Added lxml to requirements.txt
- ProcessPoolExecutor: 1 process per CPU core (bypasses GIL)
- ThreadPoolExecutor: 6 threads per process for I/O throughput
- Thread-local SQLite connections with WAL mode
- Manager().Value() for cross-process progress counter
- Real-time tqdm updates polling shared counter every 50ms
- ~19 pages/sec on 4-core system (4887 pages in 2 min)
- Filter completed URIs at query level for faster backfill resume
- Split oversized screenshots (>65000px) into multiple JPEG chunks
- Progress bar shows total including skipped, advances on errors
- Suppress PIL DecompressionBombWarning for large screenshots
- Cleaner error logging (truncate verbose wkhtmltoimage output)
- Per-page capture logs moved to debug level
- Removed embedded serve.py (872 lines) from archive.py
- bootstrap.c extracts neopig/*.py to /tmp and runs serp.py
- serp.py: tarball mode serves media directly from tar.gz
- OffsetFile wrapper for reading .run files at correct offset
- ArchiveDB: simple SQLite wrapper for archive search (no async deps)
- Archives bundle all neopig source files for self-contained operation
- --upgrade-neopig flag with progress logging
- Bootstrap C program extracts serve.py and runs from tarball
- Archives now include neopig source files for self-contained crawling
- --upgrade-neopig flag to update neopig in existing archives
- html2md.py: smart HTML-to-markdown converter for forums/blogs/Q&A
- Fix vault path defaults (data/vault instead of vault)
- Streaming tar.gz creation without temp copies
- URL rewriting for local media references in archives
- backfill-markdown: regenerate markdown with absolute URLs for a domain
- Download button on /view page with slugified filename from alt/title
- html2text baseurl resolves relative URLs during crawl
- Fallback URL lookup by filename for older crawls
- Video hover-to-play on all video elements
- Avatar CSS float:left for text wrapping
- FileResponse inline disposition (view not download)
- Dynamic screenshot delay based on content length
- archive.py: New site archiver wrapper with embedded serve.py
- --serve flag starts SERP server alongside crawl for live viewing
- --fast mode for sites without robots.txt (no crawl delay)
- Symlink-based storage: domain views link to hash vault
- Tarball resolves symlinks to include only domain content
- Fixed bytes_downloaded/bytes_stored accounting for screenshots
- Data directory (data/) for databases and state files
- Standardized on - separator in filenames
New files:
- filevault.py: Hash-based file storage with use_pairs option (v1.1.0)
- async_filevault.py: Async wrapper using asyncio.to_thread()
- domain_vault.py: Triple vault system for web archival (HTML, Media, Linkpeek)
- screenshot.py: Async screenshot capture using uri2png
- tests/unit/: Comprehensive test suite (85 tests)
Sync-to-async conversions:
- storage.py: Wrap Path operations in asyncio.to_thread()
- domain_vault.py: Wrap exists(), mkdir(), rglob(), os.walk() in asyncio.to_thread()
- screenshot.py: Wrap read_bytes(), write_bytes(), unlink() in asyncio.to_thread()
All sync filesystem operations now run in thread pool to avoid blocking async loop.
