The filename fallback lookup was matching external links like
https://pypi.org/project/virt-back/ because the filename 'virt-back'
matched a media file in the database.
Now:
1. Filename fallback only applies to URLs with media extensions
2. href rewriting only applies to exact URI matches or media file URLs
3. External links are preserved as-is
The page_content extraction was decomposing nav/header/footer from the
soup BEFORE media extraction, causing header/logo images to be missed.
Now uses a fresh BeautifulSoup parse of the body for text extraction,
preserving the original soup for media extraction.
- README.md: Complete rewrite in About page style with origin story
- Documents all storage systems: filevault, ImageVault, domain_vault
- Full database schema and method reference
- Screenshot capture with all engines and config options
- HTML to Markdown converter with content detection
- CLI reference for neopig.py, archive.py, serp.py
- All API endpoints documented
- Default to 'all' mode on crawl page
- Add Fresh/Fast/Screenshots checkboxes
- Show job types (crawl vs backfill) with distinct labels
- Add Search/Live links per job for domain filtering
- Add Stop button to cancel running crawls
- Track active tasks for cancellation on shutdown
- Fix duplicate job creation (pass job_id to pig.crawl)
- Adaptive polling (3s running, 10s idle)
- Remove page column height constraints on search
- Update uncloseai prompt to use machine learning persona
- Add lookup_pages_by_uris() for batch page hash lookup
- Rewrite internal links (same domain) to /page/{uri_hash}
- Links to archived pages now stay within neopig
- Fix asyncio error: use create_task() directly instead of BackgroundTasks
- Default depth=9, max_pages=-1 for full site crawls
- Add update_crawl_job_stats() for live progress updates
- Poll jobs every 2s while running, show live stats
- Animated progress bar for running jobs
- Fix get_media_type_from_extension() None path crash
- Use lxml parser (2-5x faster than html.parser), fallback to html.parser
- Combine 20+ noise selectors into single query (1 DOM traversal vs 20)
- Add hint parameter to skip content detection for known types
- Added lxml to requirements.txt
- ProcessPoolExecutor: 1 process per CPU core (bypasses GIL)
- ThreadPoolExecutor: 6 threads per process for I/O throughput
- Thread-local SQLite connections with WAL mode
- Manager().Value() for cross-process progress counter
- Real-time tqdm updates polling shared counter every 50ms
- ~19 pages/sec on 4-core system (4887 pages in 2 min)
- Filter completed URIs at query level for faster backfill resume
- Split oversized screenshots (>65000px) into multiple JPEG chunks
- Progress bar shows total including skipped, advances on errors
- Suppress PIL DecompressionBombWarning for large screenshots
- Cleaner error logging (truncate verbose wkhtmltoimage output)
- Per-page capture logs moved to debug level
- Removed embedded serve.py (872 lines) from archive.py
- bootstrap.c extracts neopig/*.py to /tmp and runs serp.py
- serp.py: tarball mode serves media directly from tar.gz
- OffsetFile wrapper for reading .run files at correct offset
- ArchiveDB: simple SQLite wrapper for archive search (no async deps)
- Archives bundle all neopig source files for self-contained operation
- --upgrade-neopig flag with progress logging
- Bootstrap C program extracts serve.py and runs from tarball
- Archives now include neopig source files for self-contained crawling
- --upgrade-neopig flag to update neopig in existing archives
- html2md.py: smart HTML-to-markdown converter for forums/blogs/Q&A
- Fix vault path defaults (data/vault instead of vault)
- Streaming tar.gz creation without temp copies
- URL rewriting for local media references in archives
- backfill-markdown: regenerate markdown with absolute URLs for a domain
- Download button on /view page with slugified filename from alt/title
- html2text baseurl resolves relative URLs during crawl
- Fallback URL lookup by filename for older crawls
- Video hover-to-play on all video elements
- Avatar CSS float:left for text wrapping
- FileResponse inline disposition (view not download)
- Dynamic screenshot delay based on content length
- archive.py: New site archiver wrapper with embedded serve.py
- --serve flag starts SERP server alongside crawl for live viewing
- --fast mode for sites without robots.txt (no crawl delay)
- Symlink-based storage: domain views link to hash vault
- Tarball resolves symlinks to include only domain content
- Fixed bytes_downloaded/bytes_stored accounting for screenshots
- Data directory (data/) for databases and state files
- Standardized on - separator in filenames
