modified: serp.py
new file: templates/about.html.j2
new file: templates/base.html.j2
new file: templates/crawl.html.j2
new file: templates/import.html.j2
new file: templates/live.html.j2
new file: templates/partials/search_box.html.j2
new file: templates/phantom.html.j2
new file: templates/search.html.j2
new file: templates/view.html.j2
- Use lxml parser (2-5x faster than html.parser), fallback to html.parser
- Combine 20+ noise selectors into single query (1 DOM traversal vs 20)
- Add hint parameter to skip content detection for known types
- Added lxml to requirements.txt
- Bootstrap C program extracts serve.py and runs from tarball
- Archives now include neopig source files for self-contained crawling
- --upgrade-neopig flag to update neopig in existing archives
- html2md.py: smart HTML-to-markdown converter for forums/blogs/Q&A
- Fix vault path defaults (data/vault instead of vault)
- Streaming tar.gz creation without temp copies
- URL rewriting for local media references in archives
- backfill-markdown: regenerate markdown with absolute URLs for a domain
- Download button on /view page with slugified filename from alt/title
- html2text baseurl resolves relative URLs during crawl
- Fallback URL lookup by filename for older crawls
- Video hover-to-play on all video elements
- Avatar CSS float:left for text wrapping
- FileResponse inline disposition (view not download)
- Dynamic screenshot delay based on content length
- archive.py: New site archiver wrapper with embedded serve.py
- --serve flag starts SERP server alongside crawl for live viewing
- --fast mode for sites without robots.txt (no crawl delay)
- Symlink-based storage: domain views link to hash vault
- Tarball resolves symlinks to include only domain content
- Fixed bytes_downloaded/bytes_stored accounting for screenshots
- Data directory (data/) for databases and state files
- Standardized on - separator in filenames
