neopig

Neo Python Image Grabber

Based on pig.py by Russell Ballestrini

neopig is a full-domain async media crawler with:

• MD5 deduplication - same content stored once, multiple sources tracked
• SQLite metadata index - searchable page context, alt text, titles
• Content-addressed vault - files stored by hash, not filename
• Web SERP interface - search and browse crawled media
• Multi-target crawling - crawl multiple domains concurrently
• Resume support - restart crawls without re-downloading
• Page screenshots - optional uri2png integration

Setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Usage

CLI Crawler

# Crawl single target for images
python neopig.py https://example.com --mode images

# Crawl multiple targets concurrently
python neopig.py https://site1.com https://site2.com https://site3.com --mode images

# Crawl with keywords for tagging
python neopig.py https://example.com -k "landscape" "nature" --mode images

# Crawl all media (images + videos + audio)
python neopig.py https://example.com --mode media

# Limit crawl depth and pages
python neopig.py https://example.com --depth 5 --max-pages 500 --mode images

# Index URLs without downloading
python neopig.py https://example.com --no-download --mode images

# Enable page screenshots (requires uri2png)
python neopig.py https://example.com --mode images --screenshot

# Screenshots with custom viewport
python neopig.py https://example.com --mode images --screenshot --screenshot-width 1920 --screenshot-height 1080

Options

positional arguments:
  targets               Target URI(s) to crawl

optional arguments:
  -h, --help            show this help message and exit
  -k, --keywords        Keywords to tag media with
  -m, --mode            Crawl mode: text, images, videos, media, all
  -d, --depth           Crawl depth (-1 = unlimited, default: -1)
  -p, --max-pages       Maximum pages to crawl (-1 = unlimited)
  --db                  Database path (default: neopig.db)
  --vault               Vault storage path (default: vault)
  --no-download         Don't download media, just index URLs
  -v, --verbose         Verbose output

screenshot options (all off by default, requires uri2png):
  --screenshot          Enable page screenshots
  --screenshot-width    Viewport width in pixels (default: 1280)
  --screenshot-height   Viewport height in pixels (default: 1024)
  --screenshot-delay    Delay after page load in ms (default: 1000)

Web SERP (Search Engine Results Page)

# Start the web interface
python serp.py --host 0.0.0.0 --port 8000

# With custom db/vault paths
python serp.py --port 8000 --db mydata.db --vault ./myfiles

Then visit:

• http://localhost:8000/ - Search interface
• http://localhost:8000/live - Live feed of recent media
• http://localhost:8000/crawl - Start new crawls via web UI

Architecture

neopig.py           CLI crawler entry point
serp.py             FastAPI web interface (search, live feed, crawl UI)
async_web_fetcher.py  Async HTTP client with depth crawling
database.py         SQLite schema and queries
storage.py          Content-addressed vault (MD5 hash storage)

Database Schema

• media - Deduplicated content (md5_hash, type, mime, analysis status)
• media_sources - All contexts where media was found (page URL, alt text, title)
• crawl_jobs - Crawl history and statistics

Output

Media is stored in a content-addressed vault:

vault/
  ab/
    abcd1234...5678.jpg
  cd/
    cdef5678...1234.png

Files are named by their MD5 hash, organized in 2-character prefix directories.