Commit graph

7 commits

Author SHA1 Message Date
a3c4c19e54 Fix CI to install from requirements.txt 2025-12-22 20:44:18 -05:00
1847a2fee6 Add comprehensive unit tests for async modules
Tests for:
- async_web_fetcher: URL handling, media extraction, crawl modes, scoring
- domain_vault: VaultManager, HTML/Media/Linkpeek vaults
- screenshot: ScreenshotCapture configuration and availability
- storage: ImageVault with MD5 deduplication

185 tests total, all passing.
2025-12-22 20:40:58 -05:00
54087c5bb8 Fix CI: use python3 instead of python 2025-12-22 20:24:28 -05:00
fc33032d74 Add build runner tag to CI 2025-12-22 20:20:37 -05:00
f8a1cf6730 Add GitLab CI to run tests on push 2025-12-22 20:18:43 -05:00
bc11c516b6 Add filevault system with async wrapper and convert sync ops to async
New files:
- filevault.py: Hash-based file storage with use_pairs option (v1.1.0)
- async_filevault.py: Async wrapper using asyncio.to_thread()
- domain_vault.py: Triple vault system for web archival (HTML, Media, Linkpeek)
- screenshot.py: Async screenshot capture using uri2png
- tests/unit/: Comprehensive test suite (85 tests)

Sync-to-async conversions:
- storage.py: Wrap Path operations in asyncio.to_thread()
- domain_vault.py: Wrap exists(), mkdir(), rglob(), os.walk() in asyncio.to_thread()
- screenshot.py: Wrap read_bytes(), write_bytes(), unlink() in asyncio.to_thread()

All sync filesystem operations now run in thread pool to avoid blocking async loop.
2025-12-22 20:18:20 -05:00
6dab26d6fa Initial neopig: async media crawler with MD5 deduplication
- Multi-target concurrent crawling support
- Content-addressable vault storage (MD5 hash)
- SQLite database with page context for searchability
- Live SERP with real-time polling feed (/live)
- URI naming convention (media_uri, page_uri)
- GIF and image object-fit: contain for proper display
2025-12-21 17:26:42 -05:00