Russell Ballestrini bc11c516b6 Add filevault system with async wrapper and convert sync ops to async

New files:
- filevault.py: Hash-based file storage with use_pairs option (v1.1.0)
- async_filevault.py: Async wrapper using asyncio.to_thread()
- domain_vault.py: Triple vault system for web archival (HTML, Media, Linkpeek)
- screenshot.py: Async screenshot capture using uri2png
- tests/unit/: Comprehensive test suite (85 tests)

Sync-to-async conversions:
- storage.py: Wrap Path operations in asyncio.to_thread()
- domain_vault.py: Wrap exists(), mkdir(), rglob(), os.walk() in asyncio.to_thread()
- screenshot.py: Wrap read_bytes(), write_bytes(), unlink() in asyncio.to_thread()

All sync filesystem operations now run in thread pool to avoid blocking async loop.

2025-12-22 20:18:20 -05:00

3.7 KiB

Raw Blame History

neopig

Neo Python Image Grabber

Based on pig.py by Russell Ballestrini

neopig is a full-domain async media crawler with:

MD5 deduplication - same content stored once, multiple sources tracked
SQLite metadata index - searchable page context, alt text, titles
Content-addressed vault - files stored by hash, not filename
Web SERP interface - search and browse crawled media
Multi-target crawling - crawl multiple domains concurrently
Resume support - restart crawls without re-downloading
Page screenshots - optional uri2png integration

Setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Usage

CLI Crawler

# Crawl single target for images
python neopig.py https://example.com --mode images

# Crawl multiple targets concurrently
python neopig.py https://site1.com https://site2.com https://site3.com --mode images

# Crawl with keywords for tagging
python neopig.py https://example.com -k "landscape" "nature" --mode images

# Crawl all media (images + videos + audio)
python neopig.py https://example.com --mode media

# Limit crawl depth and pages
python neopig.py https://example.com --depth 5 --max-pages 500 --mode images

# Index URLs without downloading
python neopig.py https://example.com --no-download --mode images

# Enable page screenshots (requires uri2png)
python neopig.py https://example.com --mode images --screenshot

# Screenshots with custom viewport
python neopig.py https://example.com --mode images --screenshot --screenshot-width 1920 --screenshot-height 1080

Options

positional arguments:
  targets               Target URI(s) to crawl

optional arguments:
  -h, --help            show this help message and exit
  -k, --keywords        Keywords to tag media with
  -m, --mode            Crawl mode: text, images, videos, media, all
  -d, --depth           Crawl depth (-1 = unlimited, default: -1)
  -p, --max-pages       Maximum pages to crawl (-1 = unlimited)
  --db                  Database path (default: neopig.db)
  --vault               Vault storage path (default: vault)
  --no-download         Don't download media, just index URLs
  -v, --verbose         Verbose output

screenshot options (all off by default, requires uri2png):
  --screenshot          Enable page screenshots
  --screenshot-width    Viewport width in pixels (default: 1280)
  --screenshot-height   Viewport height in pixels (default: 1024)
  --screenshot-delay    Delay after page load in ms (default: 1000)

Web SERP (Search Engine Results Page)

# Start the web interface
python serp.py --host 0.0.0.0 --port 8000

# With custom db/vault paths
python serp.py --port 8000 --db mydata.db --vault ./myfiles

Then visit:

http://localhost:8000/ - Search interface
http://localhost:8000/live - Live feed of recent media
http://localhost:8000/crawl - Start new crawls via web UI

Architecture

neopig.py           CLI crawler entry point
serp.py             FastAPI web interface (search, live feed, crawl UI)
async_web_fetcher.py  Async HTTP client with depth crawling
database.py         SQLite schema and queries
storage.py          Content-addressed vault (MD5 hash storage)

Database Schema

media - Deduplicated content (md5_hash, type, mime, analysis status)
media_sources - All contexts where media was found (page URL, alt text, title)
crawl_jobs - Crawl history and statistics

Output

Media is stored in a content-addressed vault:

vault/
  ab/
    abcd1234...5678.jpg
  cd/
    cdef5678...1234.png

Files are named by their MD5 hash, organized in 2-character prefix directories.

3.7 KiB Raw Blame History