Adds neopig/arborist_sink.py: when enabled, every crawled page neopig
stores is also committed to an arborist content-addressed store — a
Merkle document_root + append-only audit chain — alongside neopig's
existing md5/FileVault storage. Verifiable dedup + FTS5 + tamper-evident
provenance; replaces nothing.
Strictly opt-in: a hard no-op unless NEOPIG_ARBORIST_ENABLED is set AND
arborist is importable. neopig behaves byte-identically without it. This
keeps neopig public-domain by default — arborist is AGPL, pulled only
when an operator opts in. neopig never touches arborist's tables; all
writes go through arborist.embed (no-raw-SQL rule preserved). Sync
SQLite writes run off the event loop via to_thread, lock-serialized,
and failures are swallowed so the mirror can never break a crawl.
Wired into NeoPig.__init__ (self.arborist_sink) and the store_page hook
in the crawl path. Phase-0 scope: page text only; media-manifest edges
are a follow-on. Tests cover disabled-by-default, flag-without-arborist,
the page->Document mapping, and enabled end-to-end + content-idempotence.
- Add LICENSE file (public domain, permacomputer values)
- Add license headers to all 27 Python source files
- Restore full about_ch9_p2 with "terminated 🟣" content
- Translate about_ch9_p2 to all 26 languages
The permacomputer is community-owned infrastructure optimized around:
TRUTH, FREEDOM, HARMONY, LOVE
Setting Orange, 5th of Chaos, YOLD 3192 - Celebrate Mungday!
- Add Sign Maker comments throughout codebase
- Fix OVER_9000 = 9001 (it's OVER 9000, not exactly 9000)
- Change English flag from 🇺🇸 to 🏴 (St. George's Cross)
- Fnord. You didn't see this commit.
- Add /api/crawl/jobs/{job_id}/replay SSE endpoint for replaying crawls
- Replay streams media at original discovery pace with speed multiplier
- Add Replay button to crawl jobs UI (works even when crawling disabled)
- Track active replays to prevent double-clicking same job
- Live feed page handles ?replay=JOB_ID parameter for replay mode
- Add NEOPIG_NO_VCS=1 env var to skip VCS repository detection
- Fix hljs.highlightAll() call when highlight.js not loaded
- Change default port from 8000 to 31337 across all files
- About page: and → &, beast → pig
- Search results: SVG placeholder for code files (shows extension)
- Download button: remove .bin fallback, use original extension
Order of operations (critical for data integrity):
1. Get job info and collect hashes/URIs to delete
2. Delete MediaSource records (breaks references)
3. Delete Page records
4. Delete orphan Media records (not referenced by other jobs)
5. Delete orphan media files from vault
6. Delete screenshot files for deleted pages
7. Delete CrawlJob record
8. Delete state files
CLI: python neopig.py --purge-job JOB_ID
API: DELETE /api/crawl/jobs/{job_id}?purge=true (default)
UI: Delete button now purges by default
- Store discovered RSS/Atom/Sitemap feed URLs in hydra-{domain}.json
- ALWAYS check known feeds on every crawl (self-healing)
- --hydra flag discovers NEW feeds, adds them to persistent list
- fetch_feeds() accepts feed_urls param for known feeds
- Tracks which feeds actually return items for persistence
- Hydra state file (hydra-{domain}.json) never rotates
- Tracks seen URLs with first_seen timestamp, title, published date
- _filter_new_hydra_urls() returns only NEW URLs not seen before
- Enables detecting new posts when sites publish via RSS/Atom/Sitemap
- --hydra CLI flag and UI checkbox (off by default)
- discover_feeds(): finds RSS/Atom links and common sitemap URLs
- parse_rss(), parse_atom(), parse_sitemap(): extract URLs from feeds
- fetch_feeds(): async method to fetch and parse all discovered feeds
- hydra_urls injected into crawl queue with high priority (score 100)
- Supports nested sitemaps (sitemap index files)
- README.md: Complete rewrite in About page style with origin story
- Documents all storage systems: filevault, ImageVault, domain_vault
- Full database schema and method reference
- Screenshot capture with all engines and config options
- HTML to Markdown converter with content detection
- CLI reference for neopig.py, archive.py, serp.py
- All API endpoints documented
- ProcessPoolExecutor: 1 process per CPU core (bypasses GIL)
- ThreadPoolExecutor: 6 threads per process for I/O throughput
- Thread-local SQLite connections with WAL mode
- Manager().Value() for cross-process progress counter
- Real-time tqdm updates polling shared counter every 50ms
- ~19 pages/sec on 4-core system (4887 pages in 2 min)
- Filter completed URIs at query level for faster backfill resume
- Split oversized screenshots (>65000px) into multiple JPEG chunks
- Progress bar shows total including skipped, advances on errors
- Suppress PIL DecompressionBombWarning for large screenshots
- Cleaner error logging (truncate verbose wkhtmltoimage output)
- Per-page capture logs moved to debug level
- Bootstrap C program extracts serve.py and runs from tarball
- Archives now include neopig source files for self-contained crawling
- --upgrade-neopig flag to update neopig in existing archives
- html2md.py: smart HTML-to-markdown converter for forums/blogs/Q&A
- Fix vault path defaults (data/vault instead of vault)
- Streaming tar.gz creation without temp copies
- URL rewriting for local media references in archives
- backfill-markdown: regenerate markdown with absolute URLs for a domain
- Download button on /view page with slugified filename from alt/title
- html2text baseurl resolves relative URLs during crawl
- Fallback URL lookup by filename for older crawls
- Video hover-to-play on all video elements
- Avatar CSS float:left for text wrapping
- FileResponse inline disposition (view not download)
- Dynamic screenshot delay based on content length
- archive.py: New site archiver wrapper with embedded serve.py
- --serve flag starts SERP server alongside crawl for live viewing
- --fast mode for sites without robots.txt (no crawl delay)
- Symlink-based storage: domain views link to hash vault
- Tarball resolves symlinks to include only domain content
- Fixed bytes_downloaded/bytes_stored accounting for screenshots
- Data directory (data/) for databases and state files
- Standardized on - separator in filenames
New files:
- filevault.py: Hash-based file storage with use_pairs option (v1.1.0)
- async_filevault.py: Async wrapper using asyncio.to_thread()
- domain_vault.py: Triple vault system for web archival (HTML, Media, Linkpeek)
- screenshot.py: Async screenshot capture using uri2png
- tests/unit/: Comprehensive test suite (85 tests)
Sync-to-async conversions:
- storage.py: Wrap Path operations in asyncio.to_thread()
- domain_vault.py: Wrap exists(), mkdir(), rglob(), os.walk() in asyncio.to_thread()
- screenshot.py: Wrap read_bytes(), write_bytes(), unlink() in asyncio.to_thread()
All sync filesystem operations now run in thread pool to avoid blocking async loop.
