pig.py/CLAUDE.md

CLAUDE.md - neopig Project Reference

Commit Rules

NEVER attribute Claude in commit messages. No co-author tags, no "Generated with Claude" footers, no AI mentions. Keep commits clean and human-authored in appearance.

Naming Conventions

Always use uri, never url. This applies to variable names, function names, column names, and comments. URI is the correct term (Uniform Resource Identifier).

Quick Start

# Setup
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Run crawler
python neopig.py https://example.com --mode images

# Run SERP web interface
python serp.py --host 0.0.0.0 --port 8000

# Run tests
pytest tests/ -v

Project Overview

neopig (Neo Python Image Grabber) is a full-domain async media crawler with:

• MD5 content-addressable deduplication
• SQLite metadata indexing with full-text search
• Triple git-tracked vault system (HTML, media, screenshots)
• FastAPI SERP web interface for search/browse
• Optional page screenshot capture via uri2png

Based on pig.py by Russell Ballestrini.

Architecture

neopig.py             # Main CLI crawler entry point (NeoPig class)
archive.py            # Site archiver - creates distributable tar.gz packages
async_web_fetcher.py  # Async HTTP client, CrawlMode enum, robots.txt handling
database.py           # SQLite schema: crawl_jobs, media, media_sources tables
storage.py            # ImageVault - content-addressed storage by MD5 hash
domain_vault.py       # Triple filevault: DomainHtmlVault, DomainMediaVault, DomainLinkpeekVault
screenshot.py         # ScreenshotCapture wrapper for uri2png
serp.py               # FastAPI SERP server with search, live feed, crawl UI

Key Classes

NeoPig (neopig.py)

Main crawler orchestrator. Initializes database, vault, fetcher, and screenshot modules.

pig = NeoPig(db_path="neopig.db", vault_path="vault")
await pig.init()
stats = await pig.crawl(
    target_uri="https://example.com",
    keywords=["tag1", "tag2"],
    mode=CrawlMode.IMAGES,
    depth=-1,  # unlimited
    max_pages=-1,
)

CrawlMode (async_web_fetcher.py)

class CrawlMode(Enum):
    TEXT = "text"       # Extract text content
    IMAGES = "images"   # Images only
    VIDEOS = "videos"   # Videos only
    MEDIA = "media"     # All media (images + videos + audio)
    ALL = "all"         # Full domain slurp

Database Schema (database.py)

• crawl_jobs: id, target_uri, keywords (JSON), mode, status, started_at, completed_at, stats (JSON)
• media: md5_hash (PK), media_type, mime_type, file_size, keywords, alt_text, title, first_seen_at, analysis_status, analysis_result
• media_sources: Tracks all contexts where media was found (page_uri, page_title, page_description, page_content, alt_text, link_text, detail_page_uri, detail_title, detail_content, searchable_text)

Triple Vault System (domain_vault.py)

Three separate git-tracked vaults per domain with 9-layer deep hash paths:

1. HTML Vault (html_vault/): Stores page HTML

   • {path}/index.html.og - Original HTML
   • {path}/index.html - Rewritten with neopig media paths

2. Media Vault (media_vault/): Stores images/videos with git LFS

   • Files mirror original URL paths

3. Linkpeek Vault (linkpeek_vault/): Stores page screenshots with git LFS

   • One PNG per page, named after URL path

Environment: NEOPIG_VAULT_SALT - secret salt for domain hashing (privacy)

ImageVault (storage.py)

Content-addressed storage. Files stored by MD5 hash in 256-bucket directory structure:

vault/
  ab/abcd1234...5678.jpg
  cd/cdef5678...1234.png

Site Archiver (archive.py)

Archive sunset sites into distributable tar.gz packages:

# Basic usage - archive entire site
python archive.py https://discourse-urho3d.github.io/

# Custom output directory
python archive.py https://example.com -o ./archives/

# Limit depth and pages
python archive.py https://example.com --depth 5 --max-pages 500

# Disable screenshots (faster)
python archive.py https://example.com --no-screenshots

# Disable markdown conversion
python archive.py https://example.com --no-markdown

Output structure:

{domain}-{date}/
    index.html          # Archive index with sitemap
    html/               # Original HTML pages
    markdown/           # Converted markdown (optional)
    media/              # Images, videos, audio
    screenshots/        # Page screenshots (optional)
    archive.db          # SQLite FTS5 search database
    serve.py            # Embedded Pyramid search server
    requirements.txt    # Server dependencies (pyramid)
    metadata.json       # Crawl metadata and statistics

Embedded Search Server:

# Option 1: Extract and run
tar -xzf example.com-20251229.tar.gz
cd example.com-20251229
pip install -r requirements.txt
python serve.py
# Open http://localhost:6543

# Option 2: Serve directly from tar.gz (no extraction)
python serve.py example.com-20251229.tar.gz

Self-Extracting Executable (.run):

# Create self-extracting archive (builds bootstrap if needed)
make run TARBALL=example.com-20251229.tar.gz

# Run it (just needs python3 + pyramid on target)
./example.com-20251229.run
# Opens http://localhost:6543

The .run file is a single executable containing:

• C bootstrap (~14KB)
• Full tar.gz archive
• NEOPIG trailer with offset

When run, it extracts serve.py to /tmp and launches the search server.

Screenshot Engines

neopig supports multiple screenshot backends via uri2png. Auto-detects the lightest available:

Engine	Install	Speed	Notes
wkhtmltoimage	apt install wkhtmltopdf	Fast	Native Qt WebKit, no browser download
cutycapt	apt install cutycapt	Fast	Native Qt WebKit, no browser download
playwright-webkit	pip install playwright && playwright install webkit	Medium	Lighter than Chromium
playwright-chromium	pip install playwright && playwright install chromium	Slow	Most compatible, heaviest

# List available engines
python neopig.py --list-engines

# Use specific engine
python neopig.py https://example.com --screenshot --screenshot-engine wkhtmltoimage
python archive.py https://example.com --screenshot-engine cutycapt

# Auto-detect (default) - picks lightest available
python neopig.py https://example.com --screenshot

Recommendation: Install wkhtmltopdf for fast, lightweight screenshots without browser downloads.

CLI Usage (neopig.py)

# Single target
python neopig.py https://example.com --mode images

# Multiple targets concurrently
python neopig.py https://site1.com https://site2.com --mode media

# With keywords for tagging
python neopig.py https://example.com -k "tag1" "tag2" --mode images

# Limit depth and pages
python neopig.py https://example.com --depth 5 --max-pages 500

# Index only (no download)
python neopig.py https://example.com --no-download

# Enable screenshots (requires uri2png)
python neopig.py https://example.com --screenshot --screenshot-width 1920 --screenshot-height 1080

SERP API Endpoints (serp.py)

• GET / - Search UI
• GET /crawl - Crawler UI
• GET /live - Live feed (watch images appear)
• GET /view/{md5_hash} - Media detail page
• GET /media/{md5_hash} - Serve media file
• GET /api/stats - Database statistics
• GET /api/search?q=&type=&limit= - Search media
• GET /api/media/{md5_hash} - Media info JSON
• POST /api/crawl - Start crawl job
• GET /api/crawl/jobs - List crawl jobs
• GET /api/crawl/jobs/{id} - Get job status
• GET /health - Health check

Dependencies

Core:

• aiohttp, aiofiles - async HTTP/file operations
• beautifulsoup4, html5lib - HTML parsing
• aiosqlite - async SQLite
• miniuri - URI parsing

SERP:

• fastapi, uvicorn - web server
• python-multipart - form handling

Optional:

• filevault - content-addressed storage backend
• pillow - image processing
• uri2png - page screenshots (wkhtmltoimage, cutycapt, or playwright backends)

Makefile Targets

make install         # Create venv, install deps
make test            # Run pytest
make serp            # Start basic SERP server
make server          # Start SERP + screenshot server
make crawl ARGS="..."  # Run crawler with args
make vendor-install  # Install uri2png with playwright
make clean           # Remove venv and test artifacts

Testing

pytest tests/ -v --tb=short

Tests use pytest-asyncio. Config in tests/conftest.py and pytest.ini.

Key Patterns

Skeleton Key Approach

Media records track both:

• Embedding context: page_title, page_content from listing page
• Detail context: detail_title, detail_content from detail page (Pinterest-style galleries)

This enables finding images by ANY associated text.

MediaMetadata Accumulator (async_web_fetcher.py)

"Never clobber, always append" - collects ALL metadata:

• img.alt, img.title, a.title, a.text, figcaption, nearby headings, page title
• Produces combined searchable_text for full-text search

Deduplication

• Content: MD5 hash of file bytes
• Context: UNIQUE(md5_hash, media_uri, page_uri) - same content from different pages tracked separately

Robots.txt Compliance

AsyncWebFetcher respects robots.txt with configurable crawl delay (default 2s).

File Extensions

Images: .jpg .jpeg .png .gif .webp .svg .bmp .ico .tiff .avif Videos: .mp4 .webm .mov .avi .mkv .m4v .ogv .flv .wmv Audio: .mp3 .wav .ogg .m4a .flac .aac .wma