engineering/pig.py
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add archive.py with --serve flag for live crawl watching

Browse source 
- archive.py: New site archiver wrapper with embedded serve.py
- --serve flag starts SERP server alongside crawl for live viewing
- --fast mode for sites without robots.txt (no crawl delay)
- Symlink-based storage: domain views link to hash vault
- Tarball resolves symlinks to include only domain content
- Fixed bytes_downloaded/bytes_stored accounting for screenshots
- Data directory (data/) for databases and state files
- Standardized on - separator in filenames
This commit is contained in:
Russell Ballestrini 2025-12-29 11:23:03 -05:00
parent 444317a71b
commit 21e344f675
8 changed files with 1924 additions and 171 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored
View file

@ -23,3 +23,4 @@ vendor/
# OS
.DS_Store
Thumbs.db
data/