pig.py/README.md

# neopig

**Neo Python Image Grabber** — A full-domain async media crawler with content-addressable storage, full-text search, and page screenshot capture.

Based on [pig.py](http://russell.ballestrini.net/python-image-grabber-pig-py/) by Russell Ballestrini

**Source:** [git.unturf.com/engineering/unturf/pig.py](https://git.unturf.com/engineering/unturf/pig.py) (public domain)

---

## The Origin Story

> "pig.py is a *very* simple python command line tool to download all the images from a given uri."
> — Russell Ballestrini, August 22, 2011

Back in 2011, Russell Ballestrini created pig.py — a delightfully simple Python script that did one thing well: download all the images from a webpage. It was placed in the public domain, a gift to anyone who needed to grab images from the web.

The original was elegant in its simplicity:

```bash
python pig.py https://www.foxhop.net
```

That was it. Point it at a URL, and it would slurp down every image it could find. No configuration, no complexity — just a hungry little pig gobbling up pixels.

### The Great Bitbucket Extinction

Then came the dark times. The original source lived at ~~[bitbucket.org/russellballestrini/pig](https://bitbucket.org/russellballestrini/pig)~~ — click it, we dare you. In 2020, Atlassian swallowed Bitbucket whole and spat out a glorious fountain of Mercurial repositories into the void. The original pig.py, nestled in its cozy hg repo, was atomized in the great purge.

But here's where it gets *weird*.

Russell once wrote that ["programming is like alchemy — instead of exchanging matter, we programmers exchange time."](https://russell.ballestrini.net/programming-is-like-alchemy/) Programs are golems, familiar spirits, magical servants performing repetitive tasks. *"It is more accurate to group programs with technology than magic, but less fun."*

And speaking of alchemy: [Marathon Fusion](https://phys.org/news/2025-07-marathon-fusion-mercury-gold-energy.html) discovered that tokamak breeding blankets — wrapped in Mercury-Lithium alloy, like pigs in a blanket — can transmute Mercury-198 into Gold-197 through chrysopoeia. Fast neutrons trigger (n, 2n) reactions; unstable mercury decays into stable gold within 64 hours. Two metric tons of gold per gigawatt. The alchemists' dream realized, wrapped in radioactive patience (17.7 years of cooling before you can touch your transmuted treasure).

A golden goose born from the ashes of deprecated version control. The old pig was archived, but a new creature stirred in the digital depths...

### The Evolution to neopig

Like a phoenix rising from dead Bitbucket repos, **neopig** emerged — a chimera, a griffin, a more hungry and gluttonous beast than its predecessor ever dreamed of being. Where pig.py sipped politely from single pages, neopig *devours entire domains*.

---

## Features

- **Async Crawling** — Full-domain recursive crawling with configurable depth, respecting robots.txt and crawl delays
- **Content-Addressed Storage** — MD5-based deduplication in a vault system. Same image from 100 pages? Stored once
- **Full-Text Search** — Every image indexed with its surrounding context: page title, alt text, captions, nearby headings
- **Page Screenshots** — Full-page captures via [uri2png](https://github.com/russellballestrini/uri2png) (wkhtmltoimage, Playwright, etc.)
- **Markdown Conversion** — Intelligent HTML-to-Markdown with forum post detection, noise removal, and content extraction
- **Distributable Archives** — Self-contained tar.gz packages with embedded search servers for offline browsing
- **Multi-Target Crawling** — Crawl multiple domains concurrently
- **Resume Support** — Restart crawls without re-downloading

---

## Quick Start

```bash
# Setup
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Crawl a site
python neopig.py https://example.com --mode images

# Run SERP web interface
python serp.py --host 0.0.0.0 --port 31337

# Archive a site into a distributable package
python archive.py https://discourse-urho3d.github.io/
```

---

## Architecture

```
neopig.py             Main CLI crawler (NeoPig class)
archive.py            Site archiver - creates tar.gz packages
async_web_fetcher.py  Async HTTP client, robots.txt, CrawlMode enum
database.py           SQLite schema: crawl_jobs, media, pages, FTS5
filevault.py          Vault + AsyncVault - content-addressed storage
domain_vault.py       Triple vault: HTML, Media, Linkpeek per domain
screenshot.py         ScreenshotCapture wrapper for uri2png
html2md.py            Smart HTML-to-Markdown converter
serp.py               FastAPI web interface (search, live feed, crawl UI)

data/                 Database and state files
  neopig.db           Main SQLite database
vault/                Content-addressed media storage (by MD5 hash)
```

---

## The Skeleton Key Approach

neopig uses what we call the "skeleton key" approach to media indexing. For every image, we capture *all* the text that might help you find it later:

- **Page context:** Title, description, headings near the image
- **Image attributes:** Alt text, title, surrounding captions
- **Link context:** The text of links pointing to the image
- **Detail pages:** For gallery sites, we follow through to detail pages (Pinterest-style)

The result? You can search for "sunset over mountains" and find that image even if it was named `IMG_4372.jpg` with no alt text — because the page title mentioned it, or someone linked to it with descriptive text.

---

## Storage System

### filevault.py — Sync and Async Vaults

The foundation of neopig's storage is `filevault.py`, providing both synchronous (`Vault`) and asynchronous (`AsyncVault`) content-addressable storage.

#### Vault (Synchronous)

```python
from filevault import Vault, content_hash

vault = Vault(
    vaultpath="vault",           # Base storage path
    depth=9,                     # Directory tree depth (9 = 18 hex chars)
    salt="neopig",               # Salt for seed-based hashing
    enable_memory_cache=False    # Optional existence cache
)
vault.init()

# Content-addressable storage
data = b"Hello, World!"
h = content_hash(data)           # MD5 hash
path = vault.store(h, data, ".txt")
retrieved = vault.get(h)

# Seed-based paths (deterministic from arbitrary key)
path = vault.create_filename("user:123", ext="json")
vault.write_json(path, {"key": "value"})
data = vault.read_json(path)
```

#### AsyncVault (Asynchronous)

```python
from filevault import AsyncVault, content_hash

vault = AsyncVault(vaultpath="vault", depth=9)
await vault.init()

# Store media by MD5 hash
data = image_bytes
h = content_hash(data)
path = await vault.store(h, data, ".jpg")

# Check existence
exists = await vault.exists(h)

# Retrieve
data = await vault.get(h)

# Get path
path = await vault.get_path(h)

# Delete
deleted = await vault.delete(h)

# Statistics
stats = await vault.stats()  # {path, depth, count, total_size}
```

**Key Methods (both Vault and AsyncVault):**

| Method | Description |
|--------|-------------|
| `init()` | Create vault base directory |
| `store(hash, data, ext)` | Store content by MD5 hash |
| `exists(hash)` | Check if content exists |
| `get(hash)` | Retrieve bytes by hash |
| `get_path(hash)` | Get Path object for stored file |
| `delete(hash)` | Remove file by hash |
| `create_filename(seed, ext)` | Deterministic path from seed |
| `write_json(path, data)` | Atomic JSON write |
| `read_json(path, default)` | JSON read with fallback |
| `write_text(path, content)` | Text write with locking |
| `read_text(path, default)` | Text read with locking |
| `stats()` | Count files and total size |

**Utility Functions:**

| Function | Description |
|----------|-------------|
| `content_hash(data)` | Generate MD5 hash of bytes |
| `hash_to_path(h, depth, ext)` | Convert hash to directory path |

**Directory Structure:**

With `depth=9`, files are distributed across a 9-layer deep tree using hex pairs:

```
vault/ab/cd/ef/12/34/56/78/9a/bc/abcdef123456789abc....jpg
```

This spreads files across 256^9 possible directories, keeping the filesystem snappy even with hundreds of thousands of files.

---

### domain_vault.py — Triple Vault System

Three separate git-tracked vaults per domain for full site archival. Unlike `filevault.py` (content-addressed by hash), domain vaults store files by **URL path** to mirror original site structure.

**Key differences from filevault:**
- **URL-based paths** — Files stored at paths matching original URLs
- **Git versioning** — Each domain is a git repo (SSH cloneable)
- **Git LFS** — Media and screenshots use LFS for large files
- **Salted privacy** — Domain directories use salted hashes (set `NEOPIG_VAULT_SALT`)

```
html_vault/{9-layers}/{salted_hash}/about/index.html      # Mirrors /about
media_vault/{9-layers}/{salted_hash}/images/logo.png      # Mirrors /images/logo.png
linkpeek_vault/{9-layers}/{salted_hash}/about/index.png   # Screenshot of /about
```

#### 1. DomainHtmlVault

Stores page HTML with git versioning:

```python
from domain_vault import VaultManager

manager = VaultManager(
    html_vault_base='vault/html_vault',
    media_vault_base='vault/media_vault',
    linkpeek_vault_base='vault/linkpeek_vault',
)

html_vault = manager.get_html_vault('example.com')
await html_vault.init()

# Archive a page (saves both original and rewritten versions)
is_changed, chash = await html_vault.archive_page(
    url='https://example.com/about',
    html=html_content,
    media_mappings={'https://cdn.example.com/logo.png': '/media/ab/cd/.../hash.png'}
)

# Retrieve page
html = await html_vault.get_page(url, original=False)  # Rewritten
original_html = await html_vault.get_page(url, original=True)  # Original

# Commit changes
commit_hash = await html_vault.finish_crawl(stats)
```

**Files:**
- `{url_path}/index.html.og` — Original HTML with original URIs
- `{url_path}/index.html` — Rewritten with neopig media paths
- `crawl_log.json` — Crawl history

#### 2. DomainMediaVault

Stores images/videos with git LFS, mirroring original URL paths:

```python
media_vault = manager.get_media_vault('example.com')
await media_vault.init()

is_new, chash, file_path = await media_vault.archive_media(
    url='https://example.com/images/photo.jpg',  # Stored at images/photo.jpg
    content=image_bytes,
    page_url='https://example.com/gallery'
)

content, metadata = await media_vault.get_media(url)
```

#### 3. DomainLinkpeekVault

Stores page screenshots with git LFS. Each page gets one PNG at a path matching its URL:

```python
linkpeek_vault = manager.get_linkpeek_vault('example.com')
await linkpeek_vault.init()

# https://example.com/about -> about/index.png
# https://example.com/blog/post.html -> blog/post.png
is_new, chash, file_path = await linkpeek_vault.archive_screenshot(
    url='https://example.com/about',
    screenshot_data=png_bytes
)

content, metadata = await linkpeek_vault.get_screenshot(url)
```

**Screenshot path mapping:**
| URL | Screenshot Path |
|-----|-----------------|
| `https://example.com/` | `index.png` |
| `https://example.com/about` | `about/index.png` |
| `https://example.com/blog/post.html` | `blog/post.png` |

**Helper Functions:**

| Function | Description |
|----------|-------------|
| `get_vault_salt()` | Get NEOPIG_VAULT_SALT from environment |
| `domain_hash(domain, salted)` | MD5 hash of domain (optionally salted for privacy) |
| `get_filevault_path(base, domain)` | Generate 9-layer deep path for domain |
| `url_to_filepath(url)` | Convert URL to filesystem path |
| `extract_media_urls(html, base_url)` | Extract all media URLs from HTML |

---

## Database

### database.py — SQLAlchemy Models

Async SQLite with WAL mode for concurrent access.

#### Models

**CrawlJob** — Tracks crawl sessions:
```python
class CrawlJob(Base):
    id: int                    # Primary key
    target_uri: str            # Starting URL
    keywords: str              # JSON array
    mode: str                  # 'images', 'videos', 'media', 'all'
    status: str                # 'running', 'completed', 'paused'
    started_at: str            # ISO timestamp
    completed_at: str          # ISO timestamp
    stats: str                 # JSON object with crawl statistics
```

**Media** — Deduplicated content:
```python
class Media(Base):
    md5_hash: str              # Primary key (32 char hex)
    media_type: str            # 'image', 'video', 'audio', 'screenshot'
    mime_type: str             # e.g., 'image/jpeg'
    file_size: int             # Bytes
    score: int                 # Quality score (1=screenshot, 10=full-res)
    first_seen_at: str         # ISO timestamp
    upgraded_at: str           # When score was upgraded
    analysis_status: str       # 'pending', 'completed', 'failed'
    analysis_result: str       # JSON from AI analysis
```

**MediaSource** — Context where media was found:
```python
class MediaSource(Base):
    id: int                    # Primary key
    md5_hash: str              # FK to Media
    media_uri: str             # Original media URL
    page_uri: str              # Page where found
    page_title: str            # Title of page
    page_description: str      # Meta description
    page_content: str          # Extracted text
    alt_text: str              # Image alt attribute
    link_text: str             # Link anchor text
    detail_page_uri: str       # For gallery detail pages
    detail_title: str          # Detail page title
    detail_content: str        # Detail page content
    searchable_text: str       # Combined metadata for FTS
    crawl_job_id: int          # FK to CrawlJob
    discovered_at: str         # ISO timestamp
```

**Page** — Full-text searchable pages:
```python
class Page(Base):
    id: int                    # Primary key
    uri: str                   # Full URL (unique)
    uri_hash: str              # MD5 of URI for clean URLs
    path: str                  # URL path component
    title: str                 # Page title
    description: str           # Meta description
    keywords: str              # JSON array
    content: str               # Extracted text (100KB max)
    markdown: str              # Converted markdown
    raw_html: str              # Original HTML
    crawl_job_id: int          # FK to CrawlJob
    crawled_at: str            # ISO timestamp
```

#### Database Class Methods

```python
db = Database(db_path="data/neopig.db")
await db.init()
```

**Crawl Jobs:**

| Method | Description |
|--------|-------------|
| `create_crawl_job(uri, keywords, mode)` | Create job, returns ID |
| `update_crawl_job_stats(id, stats)` | Update running job stats |
| `complete_crawl_job(id, stats)` | Mark complete |
| `pause_crawl_job(id, stats)` | Mark paused |
| `delete_crawl_job(id)` | Delete job |
| `get_crawl_jobs(limit)` | List recent jobs |
| `get_crawl_job(id)` | Get single job |

**Media Records:**

| Method | Description |
|--------|-------------|
| `create_media_record(hash, uri, page_uri, ...)` | Insert media + source |
| `add_media_source(hash, uri, page_uri, ...)` | Add context for existing media |
| `upgrade_media_score(hash, new_score)` | Upgrade if new score higher |
| `check_media_uri_exists(uri, page_uri)` | Check if already crawled |
| `get_media_by_hash(hash)` | Get single media record |
| `get_media_sources(hash)` | Get all sources for media |
| `get_recent_media(limit, type)` | Recently discovered media |
| `get_recently_upgraded(limit)` | Media with upgraded scores |
| `get_crawled_media_uris()` | All URIs mapped to hashes |
| `get_crawled_screenshot_uris()` | Page URIs with screenshots |
| `get_page_screenshot(page_uri)` | Get screenshot hash for page |
| `get_page_screenshots(page_uri)` | All screenshot hashes (chunked) |

**Pages:**

| Method | Description |
|--------|-------------|
| `store_page(uri, title, content, ...)` | Upsert page record |
| `get_page_by_uri(uri)` | Lookup by URL |
| `get_page_by_hash(uri_hash)` | Lookup by MD5 hash |
| `lookup_pages_by_uris(uris)` | Batch lookup |
| `backfill_page_hashes()` | Generate missing uri_hash values |
| `get_pages_without_screenshots(domain)` | Pages needing screenshots |

**Search:**

| Method | Description |
|--------|-------------|
| `search_pages(query, limit)` | FTS5 search with LIKE fallback |
| `search_media_advanced(q, type, limit)` | Multi-field media search |
| `get_media_by_keyword(keyword, limit)` | Simple keyword search |

**Statistics:**

| Method | Description |
|--------|-------------|
| `get_stats()` | Total media, by type, by analysis status |
| `get_domains_with_pages()` | Domains with archived pages |

---

## Screenshot Capture

### screenshot.py — Multi-Engine Screenshots

Wraps uri2png for async-compatible full-page captures.

#### ScreenshotConfig

```python
from screenshot import ScreenshotCapture, ScreenshotConfig

config = ScreenshotConfig(
    enabled=True,              # Enable/disable screenshots
    width=1024,                # Viewport width
    height=768,                # Viewport height
    delay=1000,                # ms after DOM load
    timeout=30000,             # ms total timeout
    full_page=True,            # Capture full scrollable page
    format='jpeg',             # 'jpeg' or 'png'
    quality=93,                # JPEG quality (1-100)
    engine=None,               # Auto-detect or specify
    user_agent=None,           # Custom user agent
)
```

#### ScreenshotCapture

```python
capture = ScreenshotCapture(config)
await capture.initialize()

# Check availability
if await capture.is_available():
    result = await capture.capture('https://example.com')
    # result = {data, md5_hash, mime_type, size, engine, format}

# Or capture to file
success = await capture.capture_to_file(url, '/path/to/output.jpg')

# Cleanup
await capture.cleanup()
```

**Key Methods:**

| Method | Description |
|--------|-------------|
| `initialize()` | Initialize engine, returns success bool |
| `is_available()` | Check if screenshots work |
| `get_engine_name()` | Get active engine name |
| `list_engines()` | List all available engines |
| `capture(uri, content_length)` | Capture screenshot, returns dict or list for oversized |
| `capture_to_file(uri, path)` | Capture directly to file |
| `calculate_delay_for_content(length)` | Dynamic delay for long pages |
| `cleanup()` | Release engine resources |

**Supported Engines (preference order):**

| Engine | Install | Notes |
|--------|---------|-------|
| cutycapt | `apt install cutycapt` | Native Qt, fast, full-page |
| wkhtmltoimage | `apt install wkhtmltopdf` | Native Qt, fast |
| playwright-webkit | `pip install playwright && playwright install webkit` | Lighter than Chromium |
| playwright-chromium | `pip install playwright && playwright install chromium` | Most compatible |

**Oversized Image Handling:**

JPEG has a 65500px dimension limit. For tall pages, `capture()` returns a list of chunks:

```python
result = await capture.capture('https://example.com/long-page')
if isinstance(result, list):
    for chunk in result:
        # chunk = {data, md5_hash, chunk, total_chunks, ...}
```

---

## HTML to Markdown

### html2md.py — Smart Converter

Detects content structure and generates clean markdown.

```python
from html2md import html_to_markdown

markdown = html_to_markdown(
    html=html_content,
    base_url='https://example.com',
    hint='forum'  # Optional: 'forum', 'blog', 'qa', 'ecommerce'
)
```

#### SmartMarkdownConverter

```python
from html2md import SmartMarkdownConverter

converter = SmartMarkdownConverter(base_url='https://example.com')
markdown = converter.convert(html, hint=None)
```

**Detection Logic:**

1. **Forum** — Discourse, phpBB, vBulletin, XenForo patterns
2. **Q&A** — Stack Exchange style questions/answers
3. **Blog** — Article tags, .post, .entry patterns
4. **E-commerce** — Product info, prices, ratings
5. **Generic** — Fallback clean conversion

**Key Methods:**

| Method | Description |
|--------|-------------|
| `convert(html, hint)` | Main conversion with auto-detection |
| `_remove_noise(soup)` | Strip nav, scripts, ads, etc. |
| `_detect_and_extract(soup)` | Identify content type |
| `_extract_forum_posts(soup, posts)` | Parse forum threads |
| `_extract_qa(soup, questions, answers)` | Parse Q&A pages |
| `_extract_blog(soup, articles)` | Parse blog posts |
| `_extract_ecommerce(soup, products)` | Parse product pages |
| `_element_to_markdown(element)` | Convert single element |
| `_inline_content(element)` | Handle inline formatting |
| `_table_to_markdown(table)` | Convert HTML tables |
| `_resolve_url(url)` | Make URLs absolute |

**Forum Post Extraction:**

For Discourse and similar forums, extracts:
- Avatar URL
- Username
- Timestamp
- Post content
- Quoted content

---

## CLI Reference

### neopig.py

```bash
# Basic crawl
python neopig.py https://example.com --mode images

# Multiple targets
python neopig.py https://site1.com https://site2.com --mode media

# With keywords
python neopig.py https://example.com -k "landscape" "nature" --mode images

# Limit scope
python neopig.py https://example.com --depth 5 --max-pages 500

# Fast mode (no crawl delay)
python neopig.py https://example.com --fast

# Start fresh (ignore resume state)
python neopig.py https://example.com --fresh

# Custom screenshot viewport
python neopig.py https://example.com --screenshot-width 1920 --screenshot-height 1080

# Disable screenshots
python neopig.py https://example.com --no-screenshot

# Specific screenshot engine
python neopig.py https://example.com --screenshot-engine wkhtmltoimage

# List available engines
python neopig.py --list-engines

# Run with live SERP viewer
python neopig.py https://example.com --serve --port 31337

# Backfill markdown for existing pages
python neopig.py --backfill-markdown example.com --trim-wrapper

# Backfill screenshots
python neopig.py --backfill-screenshots example.com --fast
```

### archive.py

```bash
# Archive entire site
python archive.py https://discourse-urho3d.github.io/

# Custom output
python archive.py https://example.com -o ./archives/

# Fast mode
python archive.py https://example.com --fast

# Limit scope
python archive.py https://example.com --depth 5 --max-pages 500

# Without screenshots
python archive.py https://example.com --no-screenshot

# Watch live while archiving
python archive.py https://example.com --serve --port 31337

# Upgrade neopig in existing archive
python archive.py --upgrade-neopig example-20251231.tar.gz
```

### serp.py

```bash
# Start web interface
python serp.py --host 0.0.0.0 --port 31337

# Custom database/vault
python serp.py --db data/neopig.db --vault data/vault
```

---

## API Endpoints (serp.py)

### Web UI

| Route | Description |
|-------|-------------|
| `GET /` | Search interface |
| `GET /crawl` | Crawler UI |
| `GET /live` | Live feed (watch images appear) |
| `GET /random` | Random media or page |
| `GET /phantom` | Phantom site export |
| `GET /about` | About neopig |
| `GET /view/{md5_hash}` | Media detail page |
| `GET /page/{uri_hash}` | Page viewer |

### API

| Route | Description |
|-------|-------------|
| `GET /api/stats` | Database statistics |
| `GET /api/search?q=&type=&limit=` | Search media |
| `GET /api/pages/search?q=` | Search pages |
| `GET /api/media/{md5_hash}` | Media info JSON |
| `POST /api/crawl` | Start crawl job |
| `GET /api/crawl/jobs` | List crawl jobs |
| `GET /api/crawl/jobs/{id}` | Get job status |
| `DELETE /api/crawl/jobs/{id}` | Delete job |
| `GET /health` | Health check |

### Media Serving

| Route | Description |
|-------|-------------|
| `GET /media/{md5_hash}` | Serve media file |
| `GET /media/{bucket}/{filename}` | Direct vault path |

---

## Preserving the Web

The web is ephemeral. Sites go dark. Forums shut down. Communities scatter. neopig is built for preservation — capturing not just the media, but the *context* that gives it meaning.

When you archive a site with neopig, you get:

- Original HTML with rewritten media links pointing to your local vault
- Full-text searchable markdown versions of every page
- Screenshots showing exactly how pages looked
- A SQLite database you can query, backup, and migrate
- Self-extracting archives that work offline forever

---

## The Name

**neopig** = **Neo** (new) + **P**ython **I**mage **G**rabber

A tip of the hat to the original pig.py, with a nod to the Matrix's Neo — seeing through the surface of the web to the underlying content within.

---

## License

neopig is open source. The original pig.py was placed in the public domain by Russell Ballestrini. This project continues that tradition of building useful tools for the community.