Russell Ballestrini
bc11c516b6
New files: - filevault.py: Hash-based file storage with use_pairs option (v1.1.0) - async_filevault.py: Async wrapper using asyncio.to_thread() - domain_vault.py: Triple vault system for web archival (HTML, Media, Linkpeek) - screenshot.py: Async screenshot capture using uri2png - tests/unit/: Comprehensive test suite (85 tests) Sync-to-async conversions: - storage.py: Wrap Path operations in asyncio.to_thread() - domain_vault.py: Wrap exists(), mkdir(), rglob(), os.walk() in asyncio.to_thread() - screenshot.py: Wrap read_bytes(), write_bytes(), unlink() in asyncio.to_thread() All sync filesystem operations now run in thread pool to avoid blocking async loop.
120 lines
3.7 KiB
Markdown
120 lines
3.7 KiB
Markdown
# neopig
|
|
|
|
*Neo Python Image Grabber*
|
|
|
|
Based on [pig.py](http://russell.ballestrini.net/python-image-grabber-pig-py/) by Russell Ballestrini
|
|
|
|
neopig is a full-domain async media crawler with:
|
|
- **MD5 deduplication** - same content stored once, multiple sources tracked
|
|
- **SQLite metadata index** - searchable page context, alt text, titles
|
|
- **Content-addressed vault** - files stored by hash, not filename
|
|
- **Web SERP interface** - search and browse crawled media
|
|
- **Multi-target crawling** - crawl multiple domains concurrently
|
|
- **Resume support** - restart crawls without re-downloading
|
|
- **Page screenshots** - optional [uri2png](https://github.com/russellballestrini/uri2png) integration
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
python -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
## Usage
|
|
|
|
### CLI Crawler
|
|
|
|
```bash
|
|
# Crawl single target for images
|
|
python neopig.py https://example.com --mode images
|
|
|
|
# Crawl multiple targets concurrently
|
|
python neopig.py https://site1.com https://site2.com https://site3.com --mode images
|
|
|
|
# Crawl with keywords for tagging
|
|
python neopig.py https://example.com -k "landscape" "nature" --mode images
|
|
|
|
# Crawl all media (images + videos + audio)
|
|
python neopig.py https://example.com --mode media
|
|
|
|
# Limit crawl depth and pages
|
|
python neopig.py https://example.com --depth 5 --max-pages 500 --mode images
|
|
|
|
# Index URLs without downloading
|
|
python neopig.py https://example.com --no-download --mode images
|
|
|
|
# Enable page screenshots (requires uri2png)
|
|
python neopig.py https://example.com --mode images --screenshot
|
|
|
|
# Screenshots with custom viewport
|
|
python neopig.py https://example.com --mode images --screenshot --screenshot-width 1920 --screenshot-height 1080
|
|
```
|
|
|
|
### Options
|
|
|
|
```
|
|
positional arguments:
|
|
targets Target URI(s) to crawl
|
|
|
|
optional arguments:
|
|
-h, --help show this help message and exit
|
|
-k, --keywords Keywords to tag media with
|
|
-m, --mode Crawl mode: text, images, videos, media, all
|
|
-d, --depth Crawl depth (-1 = unlimited, default: -1)
|
|
-p, --max-pages Maximum pages to crawl (-1 = unlimited)
|
|
--db Database path (default: neopig.db)
|
|
--vault Vault storage path (default: vault)
|
|
--no-download Don't download media, just index URLs
|
|
-v, --verbose Verbose output
|
|
|
|
screenshot options (all off by default, requires uri2png):
|
|
--screenshot Enable page screenshots
|
|
--screenshot-width Viewport width in pixels (default: 1280)
|
|
--screenshot-height Viewport height in pixels (default: 1024)
|
|
--screenshot-delay Delay after page load in ms (default: 1000)
|
|
```
|
|
|
|
### Web SERP (Search Engine Results Page)
|
|
|
|
```bash
|
|
# Start the web interface
|
|
python serp.py --host 0.0.0.0 --port 8000
|
|
|
|
# With custom db/vault paths
|
|
python serp.py --port 8000 --db mydata.db --vault ./myfiles
|
|
```
|
|
|
|
Then visit:
|
|
- `http://localhost:8000/` - Search interface
|
|
- `http://localhost:8000/live` - Live feed of recent media
|
|
- `http://localhost:8000/crawl` - Start new crawls via web UI
|
|
|
|
## Architecture
|
|
|
|
```
|
|
neopig.py CLI crawler entry point
|
|
serp.py FastAPI web interface (search, live feed, crawl UI)
|
|
async_web_fetcher.py Async HTTP client with depth crawling
|
|
database.py SQLite schema and queries
|
|
storage.py Content-addressed vault (MD5 hash storage)
|
|
```
|
|
|
|
## Database Schema
|
|
|
|
- **media** - Deduplicated content (md5_hash, type, mime, analysis status)
|
|
- **media_sources** - All contexts where media was found (page URL, alt text, title)
|
|
- **crawl_jobs** - Crawl history and statistics
|
|
|
|
## Output
|
|
|
|
Media is stored in a content-addressed vault:
|
|
```
|
|
vault/
|
|
ab/
|
|
abcd1234...5678.jpg
|
|
cd/
|
|
cdef5678...1234.png
|
|
```
|
|
|
|
Files are named by their MD5 hash, organized in 2-character prefix directories.
|