requirements.txt used >= floors and installed with no hashes, so every CI run and
install re-resolved to whatever PyPI served. A poisoned release of any dep
(fastapi, aiohttp, lxml, pillow...) would land unverified.
- requirements.lock: 42 pkgs pinned to exact versions + SHA256 (1258 hashes), uv
(targeted py3.11 to match CI image)
- Makefile install + CI: pip install --require-hashes -r requirements.lock
- requirements.txt stays the loose source; make pins-lock regenerates
Validated: lock installs under --require-hashes, 460 tests pass under resolved versions.
- Change default port from 8000 to 31337 across all files
- About page: and → &, beast → pig
- Search results: SVG placeholder for code files (shows extension)
- Download button: remove .bin fallback, use original extension
- archive.py: New site archiver wrapper with embedded serve.py
- --serve flag starts SERP server alongside crawl for live viewing
- --fast mode for sites without robots.txt (no crawl delay)
- Symlink-based storage: domain views link to hash vault
- Tarball resolves symlinks to include only domain content
- Fixed bytes_downloaded/bytes_stored accounting for screenshots
- Data directory (data/) for databases and state files
- Standardized on - separator in filenames
New files:
- filevault.py: Hash-based file storage with use_pairs option (v1.1.0)
- async_filevault.py: Async wrapper using asyncio.to_thread()
- domain_vault.py: Triple vault system for web archival (HTML, Media, Linkpeek)
- screenshot.py: Async screenshot capture using uri2png
- tests/unit/: Comprehensive test suite (85 tests)
Sync-to-async conversions:
- storage.py: Wrap Path operations in asyncio.to_thread()
- domain_vault.py: Wrap exists(), mkdir(), rglob(), os.walk() in asyncio.to_thread()
- screenshot.py: Wrap read_bytes(), write_bytes(), unlink() in asyncio.to_thread()
All sync filesystem operations now run in thread pool to avoid blocking async loop.
