pig.py/CLAUDE.md

CLAUDE.md - neopig Project Reference

Commit Rules

NEVER attribute Claude in commit messages. No co-author tags, no "Generated with Claude" footers, no AI mentions. Keep commits clean and human-authored in appearance.

Quick Start

# Setup
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Run crawler
python neopig.py https://example.com --mode images

# Run SERP web interface
python serp.py --host 0.0.0.0 --port 8000

# Run tests
pytest tests/ -v

Project Overview

neopig (Neo Python Image Grabber) is a full-domain async media crawler with:

• MD5 content-addressable deduplication
• SQLite metadata indexing with full-text search
• Triple git-tracked vault system (HTML, media, screenshots)
• FastAPI SERP web interface for search/browse
• Optional page screenshot capture via uri2png

Based on pig.py by Russell Ballestrini.

Architecture

neopig.py             # Main CLI crawler entry point (NeoPig class)
archive.py            # Site archiver - creates distributable tar.gz packages
async_web_fetcher.py  # Async HTTP client, CrawlMode enum, robots.txt handling
database.py           # SQLite schema: crawl_jobs, media, media_sources tables
storage.py            # ImageVault - content-addressed storage by MD5 hash
domain_vault.py       # Triple filevault: DomainHtmlVault, DomainMediaVault, DomainLinkpeekVault
screenshot.py         # ScreenshotCapture wrapper for uri2png
serp.py               # FastAPI SERP server with search, live feed, crawl UI

Key Classes

NeoPig (neopig.py)

Main crawler orchestrator. Initializes database, vault, fetcher, and screenshot modules.

pig = NeoPig(db_path="neopig.db", vault_path="vault")
await pig.init()
stats = await pig.crawl(
    target_uri="https://example.com",
    keywords=["tag1", "tag2"],
    mode=CrawlMode.IMAGES,
    depth=-1,  # unlimited
    max_pages=-1,
)

CrawlMode (async_web_fetcher.py)

class CrawlMode(Enum):
    TEXT = "text"       # Extract text content
    IMAGES = "images"   # Images only
    VIDEOS = "videos"   # Videos only
    MEDIA = "media"     # All media (images + videos + audio)
    ALL = "all"         # Full domain slurp

Database Schema (database.py)

• crawl_jobs: id, target_uri, keywords (JSON), mode, status, started_at, completed_at, stats (JSON)
• media: md5_hash (PK), media_type, mime_type, file_size, keywords, alt_text, title, first_seen_at, analysis_status, analysis_result
• media_sources: Tracks all contexts where media was found (page_uri, page_title, page_description, page_content, alt_text, link_text, detail_page_uri, detail_title, detail_content, searchable_text)

Triple Vault System (domain_vault.py)

Three separate git-tracked vaults per domain with 9-layer deep hash paths:

1. HTML Vault (html_vault/): Stores page HTML

   • {path}/index.html.og - Original HTML
   • {path}/index.html - Rewritten with neopig media paths

2. Media Vault (media_vault/): Stores images/videos with git LFS

   • Files mirror original URL paths

3. Linkpeek Vault (linkpeek_vault/): Stores page screenshots with git LFS

   • One PNG per page, named after URL path

Environment: NEOPIG_VAULT_SALT - secret salt for domain hashing (privacy)

ImageVault (storage.py)

Content-addressed storage. Files stored by MD5 hash in 256-bucket directory structure:

vault/
  ab/abcd1234...5678.jpg
  cd/cdef5678...1234.png

Site Archiver (archive.py)

Archive sunset sites into distributable tar.gz packages:

# Basic usage - archive entire site
python archive.py https://discourse-urho3d.github.io/

# Custom output directory
python archive.py https://example.com -o ./archives/

# Limit depth and pages
python archive.py https://example.com --depth 5 --max-pages 500

# Disable screenshots (faster)
python archive.py https://example.com --no-screenshots

# Disable markdown conversion
python archive.py https://example.com --no-markdown

Output structure:

{domain}-{date}/
    index.html          # Archive index with sitemap
    html/               # Original HTML pages
    markdown/           # Converted markdown (optional)
    media/              # Images, videos, audio
    screenshots/        # Page screenshots (optional)
    archive.db          # SQLite FTS5 search database
    serve.py            # Embedded Pyramid search server
    requirements.txt    # Server dependencies (pyramid)
    metadata.json       # Crawl metadata and statistics

Embedded Search Server:

# Option 1: Extract and run
tar -xzf example.com-20251229.tar.gz
cd example.com-20251229
pip install -r requirements.txt
python serve.py
# Open http://localhost:6543

# Option 2: Serve directly from tar.gz (no extraction)
python serve.py example.com-20251229.tar.gz

Self-Extracting Executable (.run):

# Create self-extracting archive (builds bootstrap if needed)
make run TARBALL=example.com-20251229.tar.gz

# Run it (just needs python3 + pyramid on target)
./example.com-20251229.run
# Opens http://localhost:6543

The .run file is a single executable containing:

• C bootstrap (~14KB)
• Full tar.gz archive
• NEOPIG trailer with offset

When run, it extracts serve.py to /tmp and launches the search server.

Screenshot Engines

neopig supports multiple screenshot backends via uri2png. Auto-detects the lightest available:

Engine	Install	Speed	Notes
wkhtmltoimage	apt install wkhtmltopdf	Fast	Native Qt WebKit, no browser download
cutycapt	apt install cutycapt	Fast	Native Qt WebKit, no browser download
playwright-webkit	pip install playwright && playwright install webkit	Medium	Lighter than Chromium
playwright-chromium	pip install playwright && playwright install chromium	Slow	Most compatible, heaviest

# List available engines
python neopig.py --list-engines

# Use specific engine
python neopig.py https://example.com --screenshot --screenshot-engine wkhtmltoimage
python archive.py https://example.com --screenshot-engine cutycapt

# Auto-detect (default) - picks lightest available
python neopig.py https://example.com --screenshot

Recommendation: Install wkhtmltopdf for fast, lightweight screenshots without browser downloads.

CLI Usage (neopig.py)

# Single target
python neopig.py https://example.com --mode images

# Multiple targets concurrently
python neopig.py https://site1.com https://site2.com --mode media

# With keywords for tagging
python neopig.py https://example.com -k "tag1" "tag2" --mode images

# Limit depth and pages
python neopig.py https://example.com --depth 5 --max-pages 500

# Index only (no download)
python neopig.py https://example.com --no-download

# Enable screenshots (requires uri2png)
python neopig.py https://example.com --screenshot --screenshot-width 1920 --screenshot-height 1080

SERP API Endpoints (serp.py)

• GET / - Search UI
• GET /crawl - Crawler UI
• GET /live - Live feed (watch images appear)
• GET /view/{md5_hash} - Media detail page
• GET /media/{md5_hash} - Serve media file
• GET /api/stats - Database statistics
• GET /api/search?q=&type=&limit= - Search media
• GET /api/media/{md5_hash} - Media info JSON
• POST /api/crawl - Start crawl job
• GET /api/crawl/jobs - List crawl jobs
• GET /api/crawl/jobs/{id} - Get job status
• GET /health - Health check

Dependencies

Core:

• aiohttp, aiofiles - async HTTP/file operations
• beautifulsoup4, html5lib - HTML parsing
• aiosqlite - async SQLite
• miniuri - URI parsing

SERP:

• fastapi, uvicorn - web server
• python-multipart - form handling

Optional:

• filevault - content-addressed storage backend
• pillow - image processing
• uri2png - page screenshots (wkhtmltoimage, cutycapt, or playwright backends)

Makefile Targets

make install         # Create venv, install deps
make test            # Run pytest
make serp            # Start basic SERP server
make server          # Start SERP + screenshot server
make crawl ARGS="..."  # Run crawler with args
make vendor-install  # Install uri2png with playwright
make clean           # Remove venv and test artifacts

Testing

pytest tests/ -v --tb=short

Tests use pytest-asyncio. Config in tests/conftest.py and pytest.ini.

Key Patterns

Skeleton Key Approach

Media records track both:

• Embedding context: page_title, page_content from listing page
• Detail context: detail_title, detail_content from detail page (Pinterest-style galleries)

This enables finding images by ANY associated text.

MediaMetadata Accumulator (async_web_fetcher.py)

"Never clobber, always append" - collects ALL metadata:

• img.alt, img.title, a.title, a.text, figcaption, nearby headings, page title
• Produces combined searchable_text for full-text search

Deduplication

• Content: MD5 hash of file bytes
• Context: UNIQUE(md5_hash, media_uri, page_uri) - same content from different pages tracked separately

Robots.txt Compliance

AsyncWebFetcher respects robots.txt with configurable crawl delay (default 2s).

File Extensions

Images: .jpg .jpeg .png .gif .webp .svg .bmp .ico .tiff .avif Videos: .mp4 .webm .mov .avi .mkv .m4v .ogv .flv .wmv Audio: .mp3 .wav .ogg .m4a .flac .aac .wma