Adds neopig/arborist_sink.py: when enabled, every crawled page neopig
stores is also committed to an arborist content-addressed store — a
Merkle document_root + append-only audit chain — alongside neopig's
existing md5/FileVault storage. Verifiable dedup + FTS5 + tamper-evident
provenance; replaces nothing.
Strictly opt-in: a hard no-op unless NEOPIG_ARBORIST_ENABLED is set AND
arborist is importable. neopig behaves byte-identically without it. This
keeps neopig public-domain by default — arborist is AGPL, pulled only
when an operator opts in. neopig never touches arborist's tables; all
writes go through arborist.embed (no-raw-SQL rule preserved). Sync
SQLite writes run off the event loop via to_thread, lock-serialized,
and failures are swallowed so the mirror can never break a crawl.
Wired into NeoPig.__init__ (self.arborist_sink) and the store_page hook
in the crawl path. Phase-0 scope: page text only; media-manifest edges
are a follow-on. Tests cover disabled-by-default, flag-without-arborist,
the page->Document mapping, and enabled end-to-end + content-idempotence.
fast_mode is meant to ignore crawl-delay, but the per-domain delay was
only zeroed on the robots.txt-200 path. A site whose robots.txt 404s
(or errors) fell back to default_crawl_delay (2s), so under fast_mode
every concurrent fetch wave still slept ~2s — the dominant cost of a
wide crawl on a robots-less host. _enforce_crawl_delay now short-circuits
when fast_mode. Disallow unaffected; polite mode still enforces the delay.
Fixed upstream in agents.ai.unturf.com/core; this is the same one-line
fix in this copy of the verbatim-lift fetcher.
- Add LICENSE file (public domain, permacomputer values)
- Add license headers to all 27 Python source files
- Restore full about_ch9_p2 with "terminated 🟣" content
- Translate about_ch9_p2 to all 26 languages
The permacomputer is community-owned infrastructure optimized around:
TRUTH, FREEDOM, HARMONY, LOVE
- Add _initialized check to finish_crawl() and get_stats() in all 3 vault classes
- Prevents FileNotFoundError when git commands run on non-existent directories
- Update test_crawl.py to use pytest tmp_path fixture
- Add network marker to skip functional tests in CI with '-m "not network"'
Phase 1 of module restructuring as outlined in docs/REFACTOR.md:
New neopig/ package modules:
- live.py: Live media queue (get_live_queue, emit_live_media)
- state.py: AppendOnlyStateLog, state file helpers
- html_utils.py: trim_html_wrapper, extract_meta_from_html
- logging.py: TqdmLoggingHandler, job logging functions
- backfill/: markdown and screenshot backfill operations
Package features:
- Lazy import of NeoPig/main from neopig.py via __getattr__
- Full backwards compatibility with existing imports
- 42 new unit tests for extracted modules
Total: 458 tests passing
