Adds neopig/arborist_sink.py: when enabled, every crawled page neopig
stores is also committed to an arborist content-addressed store — a
Merkle document_root + append-only audit chain — alongside neopig's
existing md5/FileVault storage. Verifiable dedup + FTS5 + tamper-evident
provenance; replaces nothing.
Strictly opt-in: a hard no-op unless NEOPIG_ARBORIST_ENABLED is set AND
arborist is importable. neopig behaves byte-identically without it. This
keeps neopig public-domain by default — arborist is AGPL, pulled only
when an operator opts in. neopig never touches arborist's tables; all
writes go through arborist.embed (no-raw-SQL rule preserved). Sync
SQLite writes run off the event loop via to_thread, lock-serialized,
and failures are swallowed so the mirror can never break a crawl.
Wired into NeoPig.__init__ (self.arborist_sink) and the store_page hook
in the crawl path. Phase-0 scope: page text only; media-manifest edges
are a follow-on. Tests cover disabled-by-default, flag-without-arborist,
the page->Document mapping, and enabled end-to-end + content-idempotence.
fast_mode is meant to ignore crawl-delay, but the per-domain delay was
only zeroed on the robots.txt-200 path. A site whose robots.txt 404s
(or errors) fell back to default_crawl_delay (2s), so under fast_mode
every concurrent fetch wave still slept ~2s — the dominant cost of a
wide crawl on a robots-less host. _enforce_crawl_delay now short-circuits
when fast_mode. Disallow unaffected; polite mode still enforces the delay.
Fixed upstream in agents.ai.unturf.com/core; this is the same one-line
fix in this copy of the verbatim-lift fetcher.
- base.css: Core layout, typography, container
- nav.css: Navigation bar styles
- forms.css: Form elements, buttons, inputs, checkboxes
- quests.css: CSS-only easter eggs
- results.css: Media grid and result card styles
Templates now link external CSS via extra_css_links block.
Page-specific styles remain inline (correct pattern).
Fixed OVER_9000 test (it's OVER 9000, so 9001).
Phase 1 of module restructuring as outlined in docs/REFACTOR.md:
New neopig/ package modules:
- live.py: Live media queue (get_live_queue, emit_live_media)
- state.py: AppendOnlyStateLog, state file helpers
- html_utils.py: trim_html_wrapper, extract_meta_from_html
- logging.py: TqdmLoggingHandler, job logging functions
- backfill/: markdown and screenshot backfill operations
Package features:
- Lazy import of NeoPig/main from neopig.py via __getattr__
- Full backwards compatibility with existing imports
- 42 new unit tests for extracted modules
Total: 458 tests passing
- Extract 1100+ lines of translations from serp.py to i18n.py
- Add Georgian (ka) as 27th supported language
- Add 74 about page body text keys (hero, chapters, features, footer)
- Update about.html.j2 to use translation placeholders
- Full Chinese translations for all body text
- English fallback for other 25 languages (can be translated later)
- serp.py reduced from 3291 to 2142 lines
- i18n.py now 3808 lines with 190 keys per language
New files:
- filevault.py: Hash-based file storage with use_pairs option (v1.1.0)
- async_filevault.py: Async wrapper using asyncio.to_thread()
- domain_vault.py: Triple vault system for web archival (HTML, Media, Linkpeek)
- screenshot.py: Async screenshot capture using uri2png
- tests/unit/: Comprehensive test suite (85 tests)
Sync-to-async conversions:
- storage.py: Wrap Path operations in asyncio.to_thread()
- domain_vault.py: Wrap exists(), mkdir(), rglob(), os.walk() in asyncio.to_thread()
- screenshot.py: Wrap read_bytes(), write_bytes(), unlink() in asyncio.to_thread()
All sync filesystem operations now run in thread pool to avoid blocking async loop.
