Commit graph

5 commits

Author SHA1 Message Date
416b3cb760 Add self-extracting archives with bundled neopig
- Bootstrap C program extracts serve.py and runs from tarball
- Archives now include neopig source files for self-contained crawling
- --upgrade-neopig flag to update neopig in existing archives
- html2md.py: smart HTML-to-markdown converter for forums/blogs/Q&A
- Fix vault path defaults (data/vault instead of vault)
- Streaming tar.gz creation without temp copies
- URL rewriting for local media references in archives
2025-12-30 06:55:20 -05:00
a1a58fbb50 Add backfill-markdown, download button, fix image hydration
- backfill-markdown: regenerate markdown with absolute URLs for a domain
- Download button on /view page with slugified filename from alt/title
- html2text baseurl resolves relative URLs during crawl
- Fallback URL lookup by filename for older crawls
- Video hover-to-play on all video elements
- Avatar CSS float:left for text wrapping
- FileResponse inline disposition (view not download)
- Dynamic screenshot delay based on content length
2025-12-29 15:05:39 -05:00
222c23bb14 modified: CLAUDE.md
modified:   archive.py
	modified:   database.py
	modified:   neopig.py
	modified:   screenshot.py
	modified:   serp.py
2025-12-29 12:13:33 -05:00
444317a71b modified: async_web_fetcher.py
modified:   database.py
	modified:   neopig.py
2025-12-29 05:08:31 -05:00
6dab26d6fa Initial neopig: async media crawler with MD5 deduplication
- Multi-target concurrent crawling support
- Content-addressable vault storage (MD5 hash)
- SQLite database with page context for searchability
- Live SERP with real-time polling feed (/live)
- URI naming convention (media_uri, page_uri)
- GIF and image object-fit: contain for proper display
2025-12-21 17:26:42 -05:00