Russell Ballestrini
21e344f675
- archive.py: New site archiver wrapper with embedded serve.py - --serve flag starts SERP server alongside crawl for live viewing - --fast mode for sites without robots.txt (no crawl delay) - Symlink-based storage: domain views link to hash vault - Tarball resolves symlinks to include only domain content - Fixed bytes_downloaded/bytes_stored accounting for screenshots - Data directory (data/) for databases and state files - Standardized on - separator in filenames
24 lines
306 B
Text
24 lines
306 B
Text
# Core crawler
|
|
aiohttp>=3.9.0
|
|
aiofiles>=23.0.0
|
|
beautifulsoup4>=4.12.0
|
|
html5lib>=1.1
|
|
miniuri>=0.1.0
|
|
|
|
# Database
|
|
aiosqlite>=0.19.0
|
|
|
|
# Storage
|
|
filevault>=0.1.0
|
|
|
|
# SERP server
|
|
fastapi>=0.104.0
|
|
uvicorn>=0.24.0
|
|
python-multipart>=0.0.6
|
|
|
|
# Progress bar
|
|
tqdm>=4.66.0
|
|
|
|
# Optional
|
|
pillow>=10.0.0
|
|
html2text>=2024.2.26
|