neopig

Neo Python Image Grabber — A full-domain async media crawler with content-addressable storage, full-text search, and page screenshot capture.

Based on pig.py by Russell Ballestrini

Source: git.unturf.com/engineering/unturf/pig.py (public domain)

---

The Origin Story

"pig.py is a very simple python command line tool to download all the images from a given uri." — Russell Ballestrini, August 22, 2011

Back in 2011, Russell Ballestrini created pig.py — a delightfully simple Python script that did one thing well: download all the images from a webpage. It was placed in the public domain, a gift to anyone who needed to grab images from the web.

The original was elegant in its simplicity:

python pig.py https://www.foxhop.net

That was it. Point it at a URL, and it would slurp down every image it could find. No configuration, no complexity — just a hungry little pig gobbling up pixels.

The Great Bitbucket Extinction

Then came the dark times. The original source lived at bitbucket.org/russellballestrini/pig — click it, we dare you. In 2020, Atlassian swallowed Bitbucket whole and spat out a glorious fountain of Mercurial repositories into the void. The original pig.py, nestled in its cozy hg repo, was atomized in the great purge.

But here's where it gets weird.

Russell once wrote that "programming is like alchemy — instead of exchanging matter, we programmers exchange time." Programs are golems, familiar spirits, magical servants performing repetitive tasks. "It is more accurate to group programs with technology than magic, but less fun."

And speaking of alchemy: Marathon Fusion discovered that tokamak breeding blankets — wrapped in Mercury-Lithium alloy, like pigs in a blanket — can transmute Mercury-198 into Gold-197 through chrysopoeia. Fast neutrons trigger (n, 2n) reactions; unstable mercury decays into stable gold within 64 hours. Two metric tons of gold per gigawatt. The alchemists' dream realized, wrapped in radioactive patience (17.7 years of cooling before you can touch your transmuted treasure).

A golden goose born from the ashes of deprecated version control. The old pig was archived, but a new creature stirred in the digital depths...

The Evolution to neopig

Like a phoenix rising from dead Bitbucket repos, neopig emerged — a chimera, a griffin, a more hungry and gluttonous beast than its predecessor ever dreamed of being. Where pig.py sipped politely from single pages, neopig devours entire domains.

---

Features

• Async Crawling — Full-domain recursive crawling with configurable depth, respecting robots.txt and crawl delays
• Content-Addressed Storage — MD5-based deduplication in a vault system. Same image from 100 pages? Stored once
• Full-Text Search — Every image indexed with its surrounding context: page title, alt text, captions, nearby headings
• Page Screenshots — Full-page captures via uri2png (wkhtmltoimage, Playwright, etc.)
• Markdown Conversion — Intelligent HTML-to-Markdown with forum post detection, noise removal, and content extraction
• Distributable Archives — Self-contained tar.gz packages with embedded search servers for offline browsing
• Multi-Target Crawling — Crawl multiple domains concurrently
• Resume Support — Restart crawls without re-downloading

---

Quick Start

# Setup
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Crawl a site
python neopig.py https://example.com --mode images

# Run SERP web interface
python serp.py --host 0.0.0.0 --port 31337

# Archive a site into a distributable package
python archive.py https://discourse-urho3d.github.io/

---

Architecture

neopig.py             Main CLI crawler (NeoPig class)
archive.py            Site archiver - creates tar.gz packages
async_web_fetcher.py  Async HTTP client, robots.txt, CrawlMode enum
database.py           SQLite schema: crawl_jobs, media, pages, FTS5
filevault.py          Vault + AsyncVault - content-addressed storage
domain_vault.py       Triple vault: HTML, Media, Linkpeek per domain
screenshot.py         ScreenshotCapture wrapper for uri2png
html2md.py            Smart HTML-to-Markdown converter
serp.py               FastAPI web interface (search, live feed, crawl UI)

data/                 Database and state files
  neopig.db           Main SQLite database
vault/                Content-addressed media storage (by MD5 hash)

---

The Skeleton Key Approach

neopig uses what we call the "skeleton key" approach to media indexing. For every image, we capture all the text that might help you find it later:

• Page context: Title, description, headings near the image
• Image attributes: Alt text, title, surrounding captions
• Link context: The text of links pointing to the image
• Detail pages: For gallery sites, we follow through to detail pages (Pinterest-style)

The result? You can search for "sunset over mountains" and find that image even if it was named IMG_4372.jpg with no alt text — because the page title mentioned it, or someone linked to it with descriptive text.

---

Storage System

filevault.py — Sync and Async Vaults

The foundation of neopig's storage is filevault.py, providing both synchronous (Vault) and asynchronous (AsyncVault) content-addressable storage.

Vault (Synchronous)

from filevault import Vault, content_hash

vault = Vault(
    vaultpath="vault",           # Base storage path
    depth=9,                     # Directory tree depth (9 = 18 hex chars)
    salt="neopig",               # Salt for seed-based hashing
    enable_memory_cache=False    # Optional existence cache
)
vault.init()

# Content-addressable storage
data = b"Hello, World!"
h = content_hash(data)           # MD5 hash
path = vault.store(h, data, ".txt")
retrieved = vault.get(h)

# Seed-based paths (deterministic from arbitrary key)
path = vault.create_filename("user:123", ext="json")
vault.write_json(path, {"key": "value"})
data = vault.read_json(path)

AsyncVault (Asynchronous)

from filevault import AsyncVault, content_hash

vault = AsyncVault(vaultpath="vault", depth=9)
await vault.init()

# Store media by MD5 hash
data = image_bytes
h = content_hash(data)
path = await vault.store(h, data, ".jpg")

# Check existence
exists = await vault.exists(h)

# Retrieve
data = await vault.get(h)

# Get path
path = await vault.get_path(h)

# Delete
deleted = await vault.delete(h)

# Statistics
stats = await vault.stats()  # {path, depth, count, total_size}

Key Methods (both Vault and AsyncVault):

Method	Description
init()	Create vault base directory
store(hash, data, ext)	Store content by MD5 hash
exists(hash)	Check if content exists
get(hash)	Retrieve bytes by hash
get_path(hash)	Get Path object for stored file
delete(hash)	Remove file by hash
create_filename(seed, ext)	Deterministic path from seed
write_json(path, data)	Atomic JSON write
read_json(path, default)	JSON read with fallback
write_text(path, content)	Text write with locking
read_text(path, default)	Text read with locking
stats()	Count files and total size

Utility Functions:

Function	Description
content_hash(data)	Generate MD5 hash of bytes
hash_to_path(h, depth, ext)	Convert hash to directory path

Directory Structure:

With depth=9, files are distributed across a 9-layer deep tree using hex pairs:

vault/ab/cd/ef/12/34/56/78/9a/bc/abcdef123456789abc....jpg

This spreads files across 256^9 possible directories, keeping the filesystem snappy even with hundreds of thousands of files.

---

domain_vault.py — Triple Vault System

Three separate git-tracked vaults per domain for full site archival. Unlike filevault.py (content-addressed by hash), domain vaults store files by URL path to mirror original site structure.

Key differences from filevault:

• URL-based paths — Files stored at paths matching original URLs
• Git versioning — Each domain is a git repo (SSH cloneable)
• Git LFS — Media and screenshots use LFS for large files
• Salted privacy — Domain directories use salted hashes (set NEOPIG_VAULT_SALT)

html_vault/{9-layers}/{salted_hash}/about/index.html      # Mirrors /about
media_vault/{9-layers}/{salted_hash}/images/logo.png      # Mirrors /images/logo.png
linkpeek_vault/{9-layers}/{salted_hash}/about/index.png   # Screenshot of /about

1. DomainHtmlVault

Stores page HTML with git versioning:

from domain_vault import VaultManager

manager = VaultManager(
    html_vault_base='vault/html_vault',
    media_vault_base='vault/media_vault',
    linkpeek_vault_base='vault/linkpeek_vault',
)

html_vault = manager.get_html_vault('example.com')
await html_vault.init()

# Archive a page (saves both original and rewritten versions)
is_changed, chash = await html_vault.archive_page(
    url='https://example.com/about',
    html=html_content,
    media_mappings={'https://cdn.example.com/logo.png': '/media/ab/cd/.../hash.png'}
)

# Retrieve page
html = await html_vault.get_page(url, original=False)  # Rewritten
original_html = await html_vault.get_page(url, original=True)  # Original

# Commit changes
commit_hash = await html_vault.finish_crawl(stats)

Files:

• {url_path}/index.html.og — Original HTML with original URIs
• {url_path}/index.html — Rewritten with neopig media paths
• crawl_log.json — Crawl history

2. DomainMediaVault

Stores images/videos with git LFS, mirroring original URL paths:

media_vault = manager.get_media_vault('example.com')
await media_vault.init()

is_new, chash, file_path = await media_vault.archive_media(
    url='https://example.com/images/photo.jpg',  # Stored at images/photo.jpg
    content=image_bytes,
    page_url='https://example.com/gallery'
)

content, metadata = await media_vault.get_media(url)

3. DomainLinkpeekVault

Stores page screenshots with git LFS. Each page gets one PNG at a path matching its URL:

linkpeek_vault = manager.get_linkpeek_vault('example.com')
await linkpeek_vault.init()

# https://example.com/about -> about/index.png
# https://example.com/blog/post.html -> blog/post.png
is_new, chash, file_path = await linkpeek_vault.archive_screenshot(
    url='https://example.com/about',
    screenshot_data=png_bytes
)

content, metadata = await linkpeek_vault.get_screenshot(url)

Screenshot path mapping:

URL	Screenshot Path
https://example.com/	index.png
https://example.com/about	about/index.png
https://example.com/blog/post.html	blog/post.png

Helper Functions:

Function	Description
get_vault_salt()	Get NEOPIG_VAULT_SALT from environment
domain_hash(domain, salted)	MD5 hash of domain (optionally salted for privacy)
get_filevault_path(base, domain)	Generate 9-layer deep path for domain
url_to_filepath(url)	Convert URL to filesystem path
extract_media_urls(html, base_url)	Extract all media URLs from HTML

---

Database

database.py — SQLAlchemy Models

Async SQLite with WAL mode for concurrent access.

Models

CrawlJob — Tracks crawl sessions:

class CrawlJob(Base):
    id: int                    # Primary key
    target_uri: str            # Starting URL
    keywords: str              # JSON array
    mode: str                  # 'images', 'videos', 'media', 'all'
    status: str                # 'running', 'completed', 'paused'
    started_at: str            # ISO timestamp
    completed_at: str          # ISO timestamp
    stats: str                 # JSON object with crawl statistics

Media — Deduplicated content:

class Media(Base):
    md5_hash: str              # Primary key (32 char hex)
    media_type: str            # 'image', 'video', 'audio', 'screenshot'
    mime_type: str             # e.g., 'image/jpeg'
    file_size: int             # Bytes
    score: int                 # Quality score (1=screenshot, 10=full-res)
    first_seen_at: str         # ISO timestamp
    upgraded_at: str           # When score was upgraded
    analysis_status: str       # 'pending', 'completed', 'failed'
    analysis_result: str       # JSON from AI analysis

MediaSource — Context where media was found:

class MediaSource(Base):
    id: int                    # Primary key
    md5_hash: str              # FK to Media
    media_uri: str             # Original media URL
    page_uri: str              # Page where found
    page_title: str            # Title of page
    page_description: str      # Meta description
    page_content: str          # Extracted text
    alt_text: str              # Image alt attribute
    link_text: str             # Link anchor text
    detail_page_uri: str       # For gallery detail pages
    detail_title: str          # Detail page title
    detail_content: str        # Detail page content
    searchable_text: str       # Combined metadata for FTS
    crawl_job_id: int          # FK to CrawlJob
    discovered_at: str         # ISO timestamp

Page — Full-text searchable pages:

class Page(Base):
    id: int                    # Primary key
    uri: str                   # Full URL (unique)
    uri_hash: str              # MD5 of URI for clean URLs
    path: str                  # URL path component
    title: str                 # Page title
    description: str           # Meta description
    keywords: str              # JSON array
    content: str               # Extracted text (100KB max)
    markdown: str              # Converted markdown
    raw_html: str              # Original HTML
    crawl_job_id: int          # FK to CrawlJob
    crawled_at: str            # ISO timestamp

Database Class Methods

db = Database(db_path="data/neopig.db")
await db.init()

Crawl Jobs:

Method	Description
create_crawl_job(uri, keywords, mode)	Create job, returns ID
update_crawl_job_stats(id, stats)	Update running job stats
complete_crawl_job(id, stats)	Mark complete
pause_crawl_job(id, stats)	Mark paused
delete_crawl_job(id)	Delete job
get_crawl_jobs(limit)	List recent jobs
get_crawl_job(id)	Get single job

Media Records:

Method	Description
create_media_record(hash, uri, page_uri, ...)	Insert media + source
add_media_source(hash, uri, page_uri, ...)	Add context for existing media
upgrade_media_score(hash, new_score)	Upgrade if new score higher
check_media_uri_exists(uri, page_uri)	Check if already crawled
get_media_by_hash(hash)	Get single media record
get_media_sources(hash)	Get all sources for media
get_recent_media(limit, type)	Recently discovered media
get_recently_upgraded(limit)	Media with upgraded scores
get_crawled_media_uris()	All URIs mapped to hashes
get_crawled_screenshot_uris()	Page URIs with screenshots
get_page_screenshot(page_uri)	Get screenshot hash for page
get_page_screenshots(page_uri)	All screenshot hashes (chunked)

Pages:

Method	Description
store_page(uri, title, content, ...)	Upsert page record
get_page_by_uri(uri)	Lookup by URL
get_page_by_hash(uri_hash)	Lookup by MD5 hash
lookup_pages_by_uris(uris)	Batch lookup
backfill_page_hashes()	Generate missing uri_hash values
get_pages_without_screenshots(domain)	Pages needing screenshots

Search:

Method	Description
search_pages(query, limit)	FTS5 search with LIKE fallback
search_media_advanced(q, type, limit)	Multi-field media search
get_media_by_keyword(keyword, limit)	Simple keyword search

Statistics:

Method	Description
get_stats()	Total media, by type, by analysis status
get_domains_with_pages()	Domains with archived pages

---

Screenshot Capture

screenshot.py — Multi-Engine Screenshots

Wraps uri2png for async-compatible full-page captures.

ScreenshotConfig

from screenshot import ScreenshotCapture, ScreenshotConfig

config = ScreenshotConfig(
    enabled=True,              # Enable/disable screenshots
    width=1024,                # Viewport width
    height=768,                # Viewport height
    delay=1000,                # ms after DOM load
    timeout=30000,             # ms total timeout
    full_page=True,            # Capture full scrollable page
    format='jpeg',             # 'jpeg' or 'png'
    quality=93,                # JPEG quality (1-100)
    engine=None,               # Auto-detect or specify
    user_agent=None,           # Custom user agent
)

ScreenshotCapture

capture = ScreenshotCapture(config)
await capture.initialize()

# Check availability
if await capture.is_available():
    result = await capture.capture('https://example.com')
    # result = {data, md5_hash, mime_type, size, engine, format}

# Or capture to file
success = await capture.capture_to_file(url, '/path/to/output.jpg')

# Cleanup
await capture.cleanup()

Key Methods:

Method	Description
initialize()	Initialize engine, returns success bool
is_available()	Check if screenshots work
get_engine_name()	Get active engine name
list_engines()	List all available engines
capture(uri, content_length)	Capture screenshot, returns dict or list for oversized
capture_to_file(uri, path)	Capture directly to file
calculate_delay_for_content(length)	Dynamic delay for long pages
cleanup()	Release engine resources

Supported Engines (preference order):

Engine	Install	Notes
cutycapt	apt install cutycapt	Native Qt, fast, full-page
wkhtmltoimage	apt install wkhtmltopdf	Native Qt, fast
playwright-webkit	pip install playwright && playwright install webkit	Lighter than Chromium
playwright-chromium	pip install playwright && playwright install chromium	Most compatible

Oversized Image Handling:

JPEG has a 65500px dimension limit. For tall pages, capture() returns a list of chunks:

result = await capture.capture('https://example.com/long-page')
if isinstance(result, list):
    for chunk in result:
        # chunk = {data, md5_hash, chunk, total_chunks, ...}

---

HTML to Markdown

html2md.py — Smart Converter

Detects content structure and generates clean markdown.

from html2md import html_to_markdown

markdown = html_to_markdown(
    html=html_content,
    base_url='https://example.com',
    hint='forum'  # Optional: 'forum', 'blog', 'qa', 'ecommerce'
)

SmartMarkdownConverter

from html2md import SmartMarkdownConverter

converter = SmartMarkdownConverter(base_url='https://example.com')
markdown = converter.convert(html, hint=None)

Detection Logic:

1. Forum — Discourse, phpBB, vBulletin, XenForo patterns
2. Q&A — Stack Exchange style questions/answers
3. Blog — Article tags, .post, .entry patterns
4. E-commerce — Product info, prices, ratings
5. Generic — Fallback clean conversion

Key Methods:

Method	Description
convert(html, hint)	Main conversion with auto-detection
_remove_noise(soup)	Strip nav, scripts, ads, etc.
_detect_and_extract(soup)	Identify content type
_extract_forum_posts(soup, posts)	Parse forum threads
_extract_qa(soup, questions, answers)	Parse Q&A pages
_extract_blog(soup, articles)	Parse blog posts
_extract_ecommerce(soup, products)	Parse product pages
_element_to_markdown(element)	Convert single element
_inline_content(element)	Handle inline formatting
_table_to_markdown(table)	Convert HTML tables
_resolve_url(url)	Make URLs absolute

Forum Post Extraction:

For Discourse and similar forums, extracts:

• Avatar URL
• Username
• Timestamp
• Post content
• Quoted content

---

CLI Reference

neopig.py

# Basic crawl
python neopig.py https://example.com --mode images

# Multiple targets
python neopig.py https://site1.com https://site2.com --mode media

# With keywords
python neopig.py https://example.com -k "landscape" "nature" --mode images

# Limit scope
python neopig.py https://example.com --depth 5 --max-pages 500

# Fast mode (no crawl delay)
python neopig.py https://example.com --fast

# Start fresh (ignore resume state)
python neopig.py https://example.com --fresh

# Custom screenshot viewport
python neopig.py https://example.com --screenshot-width 1920 --screenshot-height 1080

# Disable screenshots
python neopig.py https://example.com --no-screenshot

# Specific screenshot engine
python neopig.py https://example.com --screenshot-engine wkhtmltoimage

# List available engines
python neopig.py --list-engines

# Run with live SERP viewer
python neopig.py https://example.com --serve --port 31337

# Backfill markdown for existing pages
python neopig.py --backfill-markdown example.com --trim-wrapper

# Backfill screenshots
python neopig.py --backfill-screenshots example.com --fast

archive.py

# Archive entire site
python archive.py https://discourse-urho3d.github.io/

# Custom output
python archive.py https://example.com -o ./archives/

# Fast mode
python archive.py https://example.com --fast

# Limit scope
python archive.py https://example.com --depth 5 --max-pages 500

# Without screenshots
python archive.py https://example.com --no-screenshot

# Watch live while archiving
python archive.py https://example.com --serve --port 31337

# Upgrade neopig in existing archive
python archive.py --upgrade-neopig example-20251231.tar.gz

serp.py

# Start web interface
python serp.py --host 0.0.0.0 --port 31337

# Custom database/vault
python serp.py --db data/neopig.db --vault data/vault

---

API Endpoints (serp.py)

Web UI

Route	Description
GET /	Search interface
GET /crawl	Crawler UI
GET /live	Live feed (watch images appear)
GET /random	Random media or page
GET /phantom	Phantom site export
GET /about	About neopig
GET /view/{md5_hash}	Media detail page
GET /page/{uri_hash}	Page viewer

API

Route	Description
GET /api/stats	Database statistics
GET /api/search?q=&type=&limit=	Search media
GET /api/pages/search?q=	Search pages
GET /api/media/{md5_hash}	Media info JSON
POST /api/crawl	Start crawl job
GET /api/crawl/jobs	List crawl jobs
GET /api/crawl/jobs/{id}	Get job status
DELETE /api/crawl/jobs/{id}	Delete job
GET /health	Health check

Media Serving

Route	Description
GET /media/{md5_hash}	Serve media file
GET /media/{bucket}/{filename}	Direct vault path

---

Preserving the Web

The web is ephemeral. Sites go dark. Forums shut down. Communities scatter. neopig is built for preservation — capturing not just the media, but the context that gives it meaning.

When you archive a site with neopig, you get:

• Original HTML with rewritten media links pointing to your local vault
• Full-text searchable markdown versions of every page
• Screenshots showing exactly how pages looked
• A SQLite database you can query, backup, and migrate
• Self-extracting archives that work offline forever

---

The Name

neopig = Neo (new) + Python Image Grabber

A tip of the hat to the original pig.py, with a nod to the Matrix's Neo — seeing through the surface of the web to the underlying content within.

---

License

neopig is open source. The original pig.py was placed in the public domain by Russell Ballestrini. This project continues that tradition of building useful tools for the community.