pig.py/CLAUDE.md

# CLAUDE.md - neopig Project Reference

## CRITICAL: Unsandbox Service Management

**NEVER use `--destroy` on production services.** Services contain persistent data (SQLite databases, vault files) that cannot be recovered.

For redeployments, ALWAYS use `--redeploy`:
```bash
# CORRECT: Redeploy with updated bootstrap (preserves data)
un service --redeploy SERVICE_ID --bootstrap /path/to/bootstrap.sh

# WRONG: This DESTROYS all data permanently
un service --destroy SERVICE_ID  # NEVER DO THIS ON PROD
```

**Current production service:** `unsb-service-*` running neopig.on.unsandbox.com

If a service shows "Instance is not running" or "unreachable", try:
1. `--redeploy` with bootstrap script (may restart our instance)
2. Wait and check `--logs` periodically
3. Ask our user before any destructive action

## Commit Rules

**NEVER attribute Claude in commit messages.** No co-author tags, no "Generated with Claude" footers, no machine learning mentions. Keep commits clean and human-authored in appearance.

## Naming Conventions

**Always use `uri`, never `url`.** This applies to variable names, function names, column names, and comments. URI is our correct term (Uniform Resource Identifier).

## Database Rules

**NEVER write raw SQL strings.** Always use SQLAlchemy ORM with proper model queries. No `text()`, no f-strings with SQL, no string concatenation for queries. Only exception: comments explicitly stating raw SQL is allowed (e.g., for FTS5 virtual tables).

## WE ARE OUR CDN

**NEVER use external CDNs for JavaScript, CSS, or any static assets.** All dependencies must be vendored into `static/vendor/`. No cloudflare, jsdelivr, unpkg, or any external asset hosts. Self-host everything.

Current vendored assets:
- `static/vendor/highlight.min.js` - highlight.js syntax highlighter
- `static/vendor/highlight-github-dark.min.css` - GitHub Dark theme for highlight.js

## Quick Start

```bash
# Setup
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Run crawler
python neopig.py https://example.com --mode images

# Run SERP web interface
python serp.py --host 0.0.0.0 --port 31337

# Run tests
pytest tests/ -v
```

## Project Overview

**neopig** (Neo Python Image Grabber) is a full-domain async media crawler with:
- MD5 content-addressable deduplication
- SQLite metadata indexing with full-text search
- Triple git-tracked vault system (HTML, media, screenshots)
- FastAPI SERP web interface for search/browse
- Optional page screenshot capture via uri2png

Based on [pig.py](http://russell.ballestrini.net/python-image-grabber-pig-py/) by Russell Ballestrini.

**Source:** [git.unturf.com/engineering/unturf/pig.py](https://git.unturf.com/engineering/unturf/pig.py) (public domain)

## Architecture

```
neopig.py             # Main CLI crawler entry point (NeoPig class)
archive.py            # Site archiver - creates distributable tar.gz packages
async_web_fetcher.py  # Async HTTP client, CrawlMode enum, robots.txt handling
database.py           # SQLite schema: crawl_jobs, media, media_sources tables
filevault.py          # FileVault 2.0 - content-addressed storage (Vault, AsyncVault)
domain_vault.py       # Triple vault: DomainHtmlVault, DomainMediaVault, DomainLinkpeekVault
repo.py               # VCS detection, cloning, symlink-based indexing (git/hg/svn)
screenshot.py         # ScreenshotCapture wrapper for uri2png
serp.py               # FastAPI SERP server with search, live feed, crawl UI

data/                 # Database and state files directory
  neopig.db           # Main SQLite database
vault/                # Content-addressed media storage (9-deep hex pairs)
repo_vault/           # VCS repos with symlinks to vault (git/hg/svn clones)
```

## Key Classes

### NeoPig (neopig.py)
Main crawler orchestrator. Initializes database, vault, fetcher, and screenshot modules.

```python
pig = NeoPig(db_path="data/neopig.db", vault_path="vault")
await pig.init()
stats = await pig.crawl(
    target_uri="https://example.com",
    keywords=["tag1", "tag2"],
    mode=CrawlMode.IMAGES,
    depth=-1,  # 0=single page, 1=page+links, -1=unlimited
    max_pages=-1,
)
```

### CrawlMode (async_web_fetcher.py)
```python
class CrawlMode(Enum):
    TEXT = "text"       # Extract text content
    IMAGES = "images"   # Images only
    VIDEOS = "videos"   # Videos only
    MEDIA = "media"     # All media (images + videos + audio)
    ALL = "all"         # Full domain slurp
```

### Database Schema (database.py)
- **crawl_jobs**: id, target_uri, keywords (JSON), mode, status, started_at, completed_at, stats (JSON)
- **media**: md5_hash (PK), media_type, mime_type, file_size, keywords, alt_text, title, first_seen_at, last_seen_at, score, analysis_status, analysis_result
- **media_sources**: Tracks all contexts where media was found (page_uri, page_title, page_description, page_content, alt_text, link_text, detail_page_uri, detail_title, detail_content, searchable_text, discovered_at)

**Media Provenance Tracking:**
- `first_seen_at`: When media was first discovered
- `last_seen_at`: Updated every time media is encountered (tracks activity)
- `discovered_at` (MediaSource): When each page first linked to that media (tracks reuse)

### Triple Vault System (domain_vault.py)

Three separate git-tracked vaults per domain with 9-layer deep hash paths:

1. **HTML Vault** (`html_vault/`): Stores page HTML
   - `{path}/index.html.og` - Original HTML
   - `{path}/index.html` - Rewritten with neopig media paths

2. **Media Vault** (`media_vault/`): Stores images/videos with git LFS
   - Files mirror original URL paths

3. **Linkpeek Vault** (`linkpeek_vault/`): Stores page screenshots with git LFS
   - One PNG per page, named after URL path

Environment: `NEOPIG_VAULT_SALT` - secret salt for domain hashing (privacy)

### FileVault 2.0.0 (filevault.py)
Content-addressed storage with 9-deep hex pair directory structure:
```
vault/
  ab/cd/ef/12/34/56/78/9a/bc/abcdef123456...5678.jpg
```

Features:
- Sync (`Vault`) and async (`AsyncVault`) implementations
- Content-addressable: `store(hash, data, ext)` / `get(hash)`
- Seed-based: `create_filename(seed, ext)` for deterministic paths
- Thread-safe file locking (fcntl)
- Optional in-memory existence cache

```python
from filevault import AsyncVault, content_hash

vault = AsyncVault("vault", depth=9)
await vault.init()

# Store by content hash
data = b"image bytes"
h = content_hash(data)
path = await vault.store(h, data, ".jpg")

# Retrieve
data = await vault.get(h)
```

## Site Archiver (archive.py)

Archive sunset sites into distributable tar.gz packages:

```bash
# Basic usage - archive entire site
python archive.py https://discourse-urho3d.github.io/

# Custom output directory
python archive.py https://example.com -o ./archives/

# Limit depth and pages
python archive.py https://example.com --depth 5 --max-pages 500

# Disable screenshots (faster)
python archive.py https://example.com --no-screenshots

# Disable markdown conversion
python archive.py https://example.com --no-markdown
```

Output structure:
```
{domain}-{date}/
    index.html          # Archive index with sitemap
    html/               # Original HTML pages
    markdown/           # Converted markdown (optional)
    media/              # Images, videos, audio
    screenshots/        # Page screenshots (optional)
    archive.db          # SQLite FTS5 search database
    neopig/             # Embedded neopig + serp.py server
    requirements.txt    # Server dependencies (fastapi, uvicorn)
    metadata.json       # Crawl metadata and statistics
```

**Embedded Search Server:**
```bash
# Option 1: Extract and run
tar -xzf example.com-20251229.tar.gz
cd example.com-20251229
pip install -r requirements.txt
python neopig/serp.py
# Open http://localhost:31337

# Option 2: Serve directly from tar.gz (no extraction)
python neopig/serp.py example.com-20251229.tar.gz
```

**Self-Extracting Executable (.run):**
```bash
# Create self-extracting archive (builds bootstrap if needed)
make run TARBALL=example.com-20251229.tar.gz

# Run it (just needs python3 + fastapi/uvicorn)
./example.com-20251229.run
# Opens http://localhost:31337
```

Our `.run` file is a single executable containing:
- C bootstrap (~14KB)
- Full tar.gz archive
- NEOPIG trailer with offset

When run, it extracts serve.py to /tmp and launches our search server.

## Screenshot Engines

neopig supports multiple screenshot backends via uri2png. Auto-detects our lightest available:

| Engine | Install | Speed | Notes |
|--------|---------|-------|-------|
| wkhtmltoimage | `apt install wkhtmltopdf` | Fast | Native Qt WebKit, no browser download |
| cutycapt | `apt install cutycapt` | Fast | Native Qt WebKit, no browser download |
| playwright-webkit | `pip install playwright && playwright install webkit` | Medium | Lighter than Chromium |
| playwright-chromium | `pip install playwright && playwright install chromium` | Slow | Most compatible, heaviest |

```bash
# List available engines
python neopig.py --list-engines

# Use specific engine
python neopig.py https://example.com --screenshot-engine wkhtmltoimage
python archive.py https://example.com --screenshot-engine cutycapt

# Disable screenshots
python neopig.py https://example.com --no-screenshot
```

**Recommendation:** Install `wkhtmltopdf` for fast, lightweight screenshots without browser downloads.

## CLI Usage (neopig.py)

```bash
# Single target
python neopig.py https://example.com --mode images

# Multiple targets concurrently
python neopig.py https://site1.com https://site2.com --mode media

# With keywords for tagging
python neopig.py https://example.com -k "tag1" "tag2" --mode images

# Depth control: 0=single page, 1=page+links, -1=unlimited (default)
python neopig.py https://example.com --depth 0              # Single page only
python neopig.py https://example.com --depth 3 --max-pages 500  # 3 levels deep

# Index only (no download)
python neopig.py https://example.com --no-download

# Custom screenshot viewport (screenshots enabled by default)
python neopig.py https://example.com --screenshot-width 1920 --screenshot-height 1080

# Disable screenshots
python neopig.py https://example.com --no-screenshot

# Backfill markdown with absolute URLs (after crawl, fixes image hydration)
python neopig.py --backfill-markdown example.com --db data/neopig.db

# Hydra mode: discover and track RSS/Atom/Sitemap feeds
python neopig.py https://example.com --hydra

# Purge a job and ALL its data (media, screenshots, pages, state)
python neopig.py --purge-job 123
```

**Depth values:** 0=single page, 1=page+direct links, 2+=deeper traversal, -1=unlimited. Max depth is 18 (use -1 for truly unlimited).

## Hydra Mode

Hydra mode discovers and persists RSS/Atom/Sitemap feed URLs for a domain. On subsequent crawls, these feeds are automatically checked for new content (self-healing).

```bash
# First crawl with --hydra: discovers feeds, persists them
python neopig.py https://example.com --hydra

# Subsequent crawls: automatically checks known feeds for new URLs
python neopig.py https://example.com
```

State file: `data/state/hydra-{domain}.json`
- `feeds`: Discovered feed URLs with discovery timestamp
- `seen_urls`: URLs already crawled (for delta detection)

## VCS Mode (repo.py)

Like Hydra mode for feeds, VCS detection is a "smart source" that bypasses slow HTTP crawling. When neopig detects a git/hg/svn repository URL, it clones directly instead of scraping our web UI.

```bash
# Auto-detect and clone git repo
python neopig.py https://github.com/user/repo

# Mercurial repo
python neopig.py https://hg.mozilla.org/mozilla-central

# SSH URL
python neopig.py git@github.com:user/repo.git

# SVN (legacy)
python neopig.py https://svn.apache.org/repos/asf/project/trunk
```

**Storage Strategy: Symlinks + Content-Addressed**

```
repo_vault/
  github.com/
    user/
      repo/
        src/main.py -> ../../../vault/ab/cd/.../hash.py  (symlink)
        README.md -> ../../vault/12/34/.../hash.md
        .git/  (preserved for pull)

vault/
  ab/cd/.../hash.py  (actual content, deduplicated)
```

**Supported VCS:**
- git (GitHub, GitLab, Bitbucket, Codeberg, sr.ht, etc.)
- hg (Mercurial)
- svn (Subversion)
- fossil

**Database columns** (MediaSource):
- `repo_uri`: Clone URL
- `repo_path`: File path within repo
- `commit_hash`: Commit when indexed
- `vcs_type`: git/hg/svn/fossil

## Job Management

```bash
# Purge a job completely (CLI)
python neopig.py --purge-job JOB_ID

# Purge via API (default behavior)
DELETE /api/crawl/jobs/{job_id}?purge=true

# Just delete job record (keep data)
DELETE /api/crawl/jobs/{job_id}?purge=false
```

Purge deletes in order: MediaSource → Pages → orphan Media → files → state files.

## SERP API Endpoints (serp.py)

- `GET /` - Search UI
- `GET /crawl` - Crawler UI
- `GET /live` - Live feed (watch images appear)
- `GET /about` - About page
- `GET /view/{md5_hash}` - Media detail page
- `GET /media/{md5_hash}` - Serve media file
- `GET /api/stats` - Database statistics
- `GET /api/search?q=&type=&limit=` - Search media
- `GET /api/media/{md5_hash}` - Media info JSON
- `POST /api/crawl` - Start crawl job
- `GET /api/crawl/jobs` - List crawl jobs
- `GET /api/crawl/jobs/{id}` - Get job status
- `DELETE /api/crawl/jobs/{id}?purge=true` - Purge job and all data
- `GET /health` - Health check

## Dependencies

Core:
- aiohttp, aiofiles - async HTTP/file operations
- beautifulsoup4, html5lib - HTML parsing
- aiosqlite - async SQLite
- miniuri - URI parsing

SERP:
- fastapi, uvicorn - web server
- python-multipart - form handling

Optional:
- filevault - content-addressed storage backend
- pillow - image processing
- uri2png - page screenshots (wkhtmltoimage, cutycapt, or playwright backends)

## Makefile Targets

```bash
make install         # Create venv, install deps
make test            # Run pytest
make serp            # Start basic SERP server
make server          # Start SERP + screenshot server
make crawl ARGS="..."  # Run crawler with args
make vendor-install  # Install uri2png with playwright
make clean           # Remove venv and test artifacts
```

## Testing

```bash
pytest tests/ -v --tb=short
```

Tests use pytest-asyncio. Config in `tests/conftest.py` and `pytest.ini`.

## Key Patterns

### Skeleton Key Approach
Media records track both:
- **Embedding context**: page_title, page_content from listing page
- **Detail context**: detail_title, detail_content from detail page (Pinterest-style galleries)

This enables finding images by ANY associated text.

### MediaMetadata Accumulator (async_web_fetcher.py)
"Never clobber, always append" - collects ALL metadata:
- img.alt, img.title, a.title, a.text, figcaption, nearby headings, page title
- Produces combined `searchable_text` for full-text search

### Deduplication
- Content: MD5 hash of file bytes
- Context: UNIQUE(md5_hash, media_uri, page_uri) - same content from different pages tracked separately

### Robots.txt Compliance
AsyncWebFetcher respects robots.txt with configurable crawl delay (default 2s).

## File Extensions

Images: .jpg .jpeg .png .gif .webp .svg .bmp .ico .tiff .avif
Videos: .mp4 .webm .mov .avi .mkv .m4v .ogv .flv .wmv
Audio: .mp3 .wav .ogg .m4a .flac .aac .wma

## Style

- Prefer "our" for shared things; "a" when something is one of many; avoid "the" — it implies fixed, singular ownership. Most teams and systems are fluid and ever-changing, like water.
- **Never use "AI" — always say "machine learning."** We grow machine learning, not "AI." This term is forbidden in all permacomputer discourse, marketing, & documentation.

## Prime Mission — Balance All Workstations

Every node is a workstation. Every edge is a queue. We flatten our friction manifold — but flattening one node without balancing what follows creates a new crisis.

**MOAD-0001 & MOAD-0005 are coupled.** Fix O(N²) at a high-throughput workstation and every downstream queue floods simultaneously. Solving one defect creates the other if we do not stage capacity first.

**3 drivers. 3 million people.** Fix the dispatch. Stage the drivers first.

- **Workaholic node** (high betweenness + high speedup): IS our bottleneck. Unblock without staging = collapse.
- **Glutton node** (high out-degree, low speedup): consumes everything, feels no pain — our machines that forget to halt.
- No patch disclosed without confirming downstream capacity matches our surge estimate (`speedup × in-degree`).
- Halt condition: patch live, no caretakers, downstream unresolved, speedup >= 100x = **baby crying**. Assign team first.

Full factory model & live DAG: `~/git/undefect.com/generate_dag.py`.
Shard source of truth: `~/git/unsandbox.com/blackops/BLACKOPS.md`.

### Eight Forms of Capital — Stewardship Check

Every feature, patch, & system decision touches at least one of our 8 capital queues (Roland & Landua, via unturf.com/eight-forms-of-capital/):

Living · Material · Financial · Intellectual · Experiential · Social · Cultural · Spiritual

Before shipping: does this drain a workaholic to feed a glutton? Does it route away from a food desert? Does it grow financial capital at the expense of living capital? If yes — stop. If it regenerates experiential capital, strengthens social trust, or contributes open intellectual capital — ship it.

Platform tax = O(N²) friction in our exchange layer. Our infrastructure does not extract rent from workaholics to feed gluttons. That is our obligation as permacomputer stewards. Full ledger: `~/git/unsandbox.com/blackops/BLACKOPS.md`.