pig.py/CLAUDE.md

CLAUDE.md - neopig Project Reference

Commit Rules

NEVER attribute Claude in commit messages. No co-author tags, no "Generated with Claude" footers, no AI mentions. Keep commits clean and human-authored in appearance.

Naming Conventions

Always use uri, never url. This applies to variable names, function names, column names, and comments. URI is the correct term (Uniform Resource Identifier).

Database Rules

NEVER write raw SQL strings. Always use SQLAlchemy ORM with proper model queries. No text(), no f-strings with SQL, no string concatenation for queries. Only exception: comments explicitly stating raw SQL is allowed (e.g., for FTS5 virtual tables).

WE ARE THE CDN

NEVER use external CDNs for JavaScript, CSS, or any static assets. All dependencies must be vendored into static/vendor/. No cloudflare, jsdelivr, unpkg, or any external asset hosts. Self-host everything.

Current vendored assets:

• static/vendor/highlight.min.js - highlight.js syntax highlighter
• static/vendor/highlight-github-dark.min.css - GitHub Dark theme for highlight.js

Quick Start

# Setup
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Run crawler
python neopig.py https://example.com --mode images

# Run SERP web interface
python serp.py --host 0.0.0.0 --port 31337

# Run tests
pytest tests/ -v

Project Overview

neopig (Neo Python Image Grabber) is a full-domain async media crawler with:

• MD5 content-addressable deduplication
• SQLite metadata indexing with full-text search
• Triple git-tracked vault system (HTML, media, screenshots)
• FastAPI SERP web interface for search/browse
• Optional page screenshot capture via uri2png

Based on pig.py by Russell Ballestrini.

Source: git.unturf.com/engineering/unturf/pig.py (public domain)

Architecture

neopig.py             # Main CLI crawler entry point (NeoPig class)
archive.py            # Site archiver - creates distributable tar.gz packages
async_web_fetcher.py  # Async HTTP client, CrawlMode enum, robots.txt handling
database.py           # SQLite schema: crawl_jobs, media, media_sources tables
filevault.py          # FileVault 2.0 - content-addressed storage (Vault, AsyncVault)
domain_vault.py       # Triple vault: DomainHtmlVault, DomainMediaVault, DomainLinkpeekVault
repo.py               # VCS detection, cloning, symlink-based indexing (git/hg/svn)
screenshot.py         # ScreenshotCapture wrapper for uri2png
serp.py               # FastAPI SERP server with search, live feed, crawl UI

data/                 # Database and state files directory
  neopig.db           # Main SQLite database
vault/                # Content-addressed media storage (9-deep hex pairs)
repo_vault/           # VCS repos with symlinks to vault (git/hg/svn clones)

Key Classes

NeoPig (neopig.py)

Main crawler orchestrator. Initializes database, vault, fetcher, and screenshot modules.

pig = NeoPig(db_path="data/neopig.db", vault_path="vault")
await pig.init()
stats = await pig.crawl(
    target_uri="https://example.com",
    keywords=["tag1", "tag2"],
    mode=CrawlMode.IMAGES,
    depth=-1,  # 0=single page, 1=page+links, -1=unlimited
    max_pages=-1,
)

CrawlMode (async_web_fetcher.py)

class CrawlMode(Enum):
    TEXT = "text"       # Extract text content
    IMAGES = "images"   # Images only
    VIDEOS = "videos"   # Videos only
    MEDIA = "media"     # All media (images + videos + audio)
    ALL = "all"         # Full domain slurp

Database Schema (database.py)

• crawl_jobs: id, target_uri, keywords (JSON), mode, status, started_at, completed_at, stats (JSON)
• media: md5_hash (PK), media_type, mime_type, file_size, keywords, alt_text, title, first_seen_at, last_seen_at, score, analysis_status, analysis_result
• media_sources: Tracks all contexts where media was found (page_uri, page_title, page_description, page_content, alt_text, link_text, detail_page_uri, detail_title, detail_content, searchable_text, discovered_at)

Media Provenance Tracking:

• first_seen_at: When media was first discovered
• last_seen_at: Updated every time media is encountered (tracks activity)
• discovered_at (MediaSource): When each page first linked to that media (tracks reuse)

Triple Vault System (domain_vault.py)

Three separate git-tracked vaults per domain with 9-layer deep hash paths:

1. HTML Vault (html_vault/): Stores page HTML

   • {path}/index.html.og - Original HTML
   • {path}/index.html - Rewritten with neopig media paths

2. Media Vault (media_vault/): Stores images/videos with git LFS

   • Files mirror original URL paths

3. Linkpeek Vault (linkpeek_vault/): Stores page screenshots with git LFS

   • One PNG per page, named after URL path

Environment: NEOPIG_VAULT_SALT - secret salt for domain hashing (privacy)

FileVault 2.0.0 (filevault.py)

Content-addressed storage with 9-deep hex pair directory structure:

vault/
  ab/cd/ef/12/34/56/78/9a/bc/abcdef123456...5678.jpg

Features:

• Sync (Vault) and async (AsyncVault) implementations
• Content-addressable: store(hash, data, ext) / get(hash)
• Seed-based: create_filename(seed, ext) for deterministic paths
• Thread-safe file locking (fcntl)
• Optional in-memory existence cache

from filevault import AsyncVault, content_hash

vault = AsyncVault("vault", depth=9)
await vault.init()

# Store by content hash
data = b"image bytes"
h = content_hash(data)
path = await vault.store(h, data, ".jpg")

# Retrieve
data = await vault.get(h)

Site Archiver (archive.py)

Archive sunset sites into distributable tar.gz packages:

# Basic usage - archive entire site
python archive.py https://discourse-urho3d.github.io/

# Custom output directory
python archive.py https://example.com -o ./archives/

# Limit depth and pages
python archive.py https://example.com --depth 5 --max-pages 500

# Disable screenshots (faster)
python archive.py https://example.com --no-screenshots

# Disable markdown conversion
python archive.py https://example.com --no-markdown

Output structure:

{domain}-{date}/
    index.html          # Archive index with sitemap
    html/               # Original HTML pages
    markdown/           # Converted markdown (optional)
    media/              # Images, videos, audio
    screenshots/        # Page screenshots (optional)
    archive.db          # SQLite FTS5 search database
    neopig/             # Embedded neopig + serp.py server
    requirements.txt    # Server dependencies (fastapi, uvicorn)
    metadata.json       # Crawl metadata and statistics

Embedded Search Server:

# Option 1: Extract and run
tar -xzf example.com-20251229.tar.gz
cd example.com-20251229
pip install -r requirements.txt
python neopig/serp.py
# Open http://localhost:31337

# Option 2: Serve directly from tar.gz (no extraction)
python neopig/serp.py example.com-20251229.tar.gz

Self-Extracting Executable (.run):

# Create self-extracting archive (builds bootstrap if needed)
make run TARBALL=example.com-20251229.tar.gz

# Run it (just needs python3 + fastapi/uvicorn)
./example.com-20251229.run
# Opens http://localhost:31337

The .run file is a single executable containing:

• C bootstrap (~14KB)
• Full tar.gz archive
• NEOPIG trailer with offset

When run, it extracts serve.py to /tmp and launches the search server.

Screenshot Engines

neopig supports multiple screenshot backends via uri2png. Auto-detects the lightest available:

Engine	Install	Speed	Notes
wkhtmltoimage	apt install wkhtmltopdf	Fast	Native Qt WebKit, no browser download
cutycapt	apt install cutycapt	Fast	Native Qt WebKit, no browser download
playwright-webkit	pip install playwright && playwright install webkit	Medium	Lighter than Chromium
playwright-chromium	pip install playwright && playwright install chromium	Slow	Most compatible, heaviest

# List available engines
python neopig.py --list-engines

# Use specific engine
python neopig.py https://example.com --screenshot-engine wkhtmltoimage
python archive.py https://example.com --screenshot-engine cutycapt

# Disable screenshots
python neopig.py https://example.com --no-screenshot

Recommendation: Install wkhtmltopdf for fast, lightweight screenshots without browser downloads.

CLI Usage (neopig.py)

# Single target
python neopig.py https://example.com --mode images

# Multiple targets concurrently
python neopig.py https://site1.com https://site2.com --mode media

# With keywords for tagging
python neopig.py https://example.com -k "tag1" "tag2" --mode images

# Depth control: 0=single page, 1=page+links, -1=unlimited (default)
python neopig.py https://example.com --depth 0              # Single page only
python neopig.py https://example.com --depth 3 --max-pages 500  # 3 levels deep

# Index only (no download)
python neopig.py https://example.com --no-download

# Custom screenshot viewport (screenshots enabled by default)
python neopig.py https://example.com --screenshot-width 1920 --screenshot-height 1080

# Disable screenshots
python neopig.py https://example.com --no-screenshot

# Backfill markdown with absolute URLs (after crawl, fixes image hydration)
python neopig.py --backfill-markdown example.com --db data/neopig.db

# Hydra mode: discover and track RSS/Atom/Sitemap feeds
python neopig.py https://example.com --hydra

# Purge a job and ALL its data (media, screenshots, pages, state)
python neopig.py --purge-job 123

Depth values: 0=single page, 1=page+direct links, 2+=deeper traversal, -1=unlimited. Max depth is 18 (use -1 for truly unlimited).

Hydra Mode

Hydra mode discovers and persists RSS/Atom/Sitemap feed URLs for a domain. On subsequent crawls, these feeds are automatically checked for new content (self-healing).

# First crawl with --hydra: discovers feeds, persists them
python neopig.py https://example.com --hydra

# Subsequent crawls: automatically checks known feeds for new URLs
python neopig.py https://example.com

State file: data/state/hydra-{domain}.json

• feeds: Discovered feed URLs with discovery timestamp
• seen_urls: URLs already crawled (for delta detection)

VCS Mode (repo.py)

Like Hydra mode for feeds, VCS detection is a "smart source" that bypasses slow HTTP crawling. When neopig detects a git/hg/svn repository URL, it clones directly instead of scraping the web UI.

# Auto-detect and clone git repo
python neopig.py https://github.com/user/repo

# Mercurial repo
python neopig.py https://hg.mozilla.org/mozilla-central

# SSH URL
python neopig.py git@github.com:user/repo.git

# SVN (legacy)
python neopig.py https://svn.apache.org/repos/asf/project/trunk

Storage Strategy: Symlinks + Content-Addressed

repo_vault/
  github.com/
    user/
      repo/
        src/main.py -> ../../../vault/ab/cd/.../hash.py  (symlink)
        README.md -> ../../vault/12/34/.../hash.md
        .git/  (preserved for pull)

vault/
  ab/cd/.../hash.py  (actual content, deduplicated)

Supported VCS:

• git (GitHub, GitLab, Bitbucket, Codeberg, sr.ht, etc.)
• hg (Mercurial)
• svn (Subversion)
• fossil

Database columns (MediaSource):

• repo_uri: Clone URL
• repo_path: File path within repo
• commit_hash: Commit when indexed
• vcs_type: git/hg/svn/fossil

Job Management

# Purge a job completely (CLI)
python neopig.py --purge-job JOB_ID

# Purge via API (default behavior)
DELETE /api/crawl/jobs/{job_id}?purge=true

# Just delete job record (keep data)
DELETE /api/crawl/jobs/{job_id}?purge=false

Purge deletes in order: MediaSource → Pages → orphan Media → files → state files.

SERP API Endpoints (serp.py)

• GET / - Search UI
• GET /crawl - Crawler UI
• GET /live - Live feed (watch images appear)
• GET /about - About page
• GET /view/{md5_hash} - Media detail page
• GET /media/{md5_hash} - Serve media file
• GET /api/stats - Database statistics
• GET /api/search?q=&type=&limit= - Search media
• GET /api/media/{md5_hash} - Media info JSON
• POST /api/crawl - Start crawl job
• GET /api/crawl/jobs - List crawl jobs
• GET /api/crawl/jobs/{id} - Get job status
• DELETE /api/crawl/jobs/{id}?purge=true - Purge job and all data
• GET /health - Health check

Dependencies

Core:

• aiohttp, aiofiles - async HTTP/file operations
• beautifulsoup4, html5lib - HTML parsing
• aiosqlite - async SQLite
• miniuri - URI parsing

SERP:

• fastapi, uvicorn - web server
• python-multipart - form handling

Optional:

• filevault - content-addressed storage backend
• pillow - image processing
• uri2png - page screenshots (wkhtmltoimage, cutycapt, or playwright backends)

Makefile Targets

make install         # Create venv, install deps
make test            # Run pytest
make serp            # Start basic SERP server
make server          # Start SERP + screenshot server
make crawl ARGS="..."  # Run crawler with args
make vendor-install  # Install uri2png with playwright
make clean           # Remove venv and test artifacts

Testing

pytest tests/ -v --tb=short

Tests use pytest-asyncio. Config in tests/conftest.py and pytest.ini.

Key Patterns

Skeleton Key Approach

Media records track both:

• Embedding context: page_title, page_content from listing page
• Detail context: detail_title, detail_content from detail page (Pinterest-style galleries)

This enables finding images by ANY associated text.

MediaMetadata Accumulator (async_web_fetcher.py)

"Never clobber, always append" - collects ALL metadata:

• img.alt, img.title, a.title, a.text, figcaption, nearby headings, page title
• Produces combined searchable_text for full-text search

Deduplication

• Content: MD5 hash of file bytes
• Context: UNIQUE(md5_hash, media_uri, page_uri) - same content from different pages tracked separately

Robots.txt Compliance

AsyncWebFetcher respects robots.txt with configurable crawl delay (default 2s).

File Extensions

Images: .jpg .jpeg .png .gif .webp .svg .bmp .ico .tiff .avif Videos: .mp4 .webm .mov .avi .mkv .m4v .ogv .flv .wmv Audio: .mp3 .wav .ogg .m4a .flac .aac .wma