- README.md: Complete rewrite in About page style with origin story
- Documents all storage systems: filevault, ImageVault, domain_vault
- Full database schema and method reference
- Screenshot capture with all engines and config options
- HTML to Markdown converter with content detection
- CLI reference for neopig.py, archive.py, serp.py
- All API endpoints documented
- Use lxml parser (2-5x faster than html.parser), fallback to html.parser
- Combine 20+ noise selectors into single query (1 DOM traversal vs 20)
- Add hint parameter to skip content detection for known types
- Added lxml to requirements.txt
- ProcessPoolExecutor: 1 process per CPU core (bypasses GIL)
- ThreadPoolExecutor: 6 threads per process for I/O throughput
- Thread-local SQLite connections with WAL mode
- Manager().Value() for cross-process progress counter
- Real-time tqdm updates polling shared counter every 50ms
- ~19 pages/sec on 4-core system (4887 pages in 2 min)
- Bootstrap C program extracts serve.py and runs from tarball
- Archives now include neopig source files for self-contained crawling
- --upgrade-neopig flag to update neopig in existing archives
- html2md.py: smart HTML-to-markdown converter for forums/blogs/Q&A
- Fix vault path defaults (data/vault instead of vault)
- Streaming tar.gz creation without temp copies
- URL rewriting for local media references in archives
