Adds neopig/arborist_sink.py: when enabled, every crawled page neopig
stores is also committed to an arborist content-addressed store — a
Merkle document_root + append-only audit chain — alongside neopig's
existing md5/FileVault storage. Verifiable dedup + FTS5 + tamper-evident
provenance; replaces nothing.
Strictly opt-in: a hard no-op unless NEOPIG_ARBORIST_ENABLED is set AND
arborist is importable. neopig behaves byte-identically without it. This
keeps neopig public-domain by default — arborist is AGPL, pulled only
when an operator opts in. neopig never touches arborist's tables; all
writes go through arborist.embed (no-raw-SQL rule preserved). Sync
SQLite writes run off the event loop via to_thread, lock-serialized,
and failures are swallowed so the mirror can never break a crawl.
Wired into NeoPig.__init__ (self.arborist_sink) and the store_page hook
in the crawl path. Phase-0 scope: page text only; media-manifest edges
are a follow-on. Tests cover disabled-by-default, flag-without-arborist,
the page->Document mapping, and enabled end-to-end + content-idempotence.
fast_mode is meant to ignore crawl-delay, but the per-domain delay was
only zeroed on the robots.txt-200 path. A site whose robots.txt 404s
(or errors) fell back to default_crawl_delay (2s), so under fast_mode
every concurrent fetch wave still slept ~2s — the dominant cost of a
wide crawl on a robots-less host. _enforce_crawl_delay now short-circuits
when fast_mode. Disallow unaffected; polite mode still enforces the delay.
Fixed upstream in agents.ai.unturf.com/core; this is the same one-line
fix in this copy of the verbatim-lift fetcher.
requirements.txt used >= floors and installed with no hashes, so every CI run and
install re-resolved to whatever PyPI served. A poisoned release of any dep
(fastapi, aiohttp, lxml, pillow...) would land unverified.
- requirements.lock: 42 pkgs pinned to exact versions + SHA256 (1258 hashes), uv
(targeted py3.11 to match CI image)
- Makefile install + CI: pip install --require-hashes -r requirements.lock
- requirements.txt stays the loose source; make pins-lock regenerates
Validated: lock installs under --require-hashes, 460 tests pass under resolved versions.
- Add LICENSE file (public domain, permacomputer values)
- Add license headers to all 27 Python source files
- Restore full about_ch9_p2 with "terminated 🟣" content
- Translate about_ch9_p2 to all 26 languages
The permacomputer is community-owned infrastructure optimized around:
TRUTH, FREEDOM, HARMONY, LOVE
7 translation keys restored with HTML links across all 27 languages:
- about_ch1_p1: link to pig.py blog post
- about_ch2_p3: dead-link to bitbucket + "Click it. We dare you..." text
- about_ch3_p1: link to "programming is like alchemy" article
- about_ch3_p4: link to arxiv:2507.13461 (Marathon Fusion chrysopoeia)
- about_ch3_p7: link to Wikipedia wu wei + restored energy balance text
- about_ch4_p4: link to git.unturf.com/engineering/unturf/pig.py
- about_ch9_p2: link to permacomputer.com
These links were lost during initial i18n extraction. Wu wei energy
that elites used to tombstone pig.py now unfolds neopig.py.
- Restored 7 translation keys that were compressed during initial i18n:
- about_ch3_p4: 'pigs in an hg blanket' joke and (n,2n) reaction details
- about_ch3_p5: 17.7 years cooling time detail
- about_ch3_p6: 'Old pig was slaughtered' reference
- about_ch9_truth/harmony/freedom/love: Full philosophical paragraphs
- Added apply_restored_translations.py utility script
- All 189 translations updated (27 languages × 7 keys)
- async_web_fetcher -> neopig.async_web_fetcher
- screenshot -> neopig.screenshot
These were missed during the module reorganization, causing
/page/view and other endpoints to fail with import errors.
- Add _initialized check to finish_crawl() and get_stats() in all 3 vault classes
- Prevents FileNotFoundError when git commands run on non-existent directories
- Update test_crawl.py to use pytest tmp_path fixture
- Add network marker to skip functional tests in CI with '-m "not network"'
- Vendor ChunkFive font for become phrases
- Add periods to all become_ translations (27 languages)
- Simplify become_un to just the prefix (un., in., не-., etc.)
- White color for finale un. in footer
- become ungovernable → Chapter II (Extinction) red
- become unstoppable → Chapter IV (Pig Awakens) orange
- become untouchable → Chapter V (Mirror) purple
- become unseen → Chapter VII (Skeleton Key) gray
- become unknown → Chapter VIII (Why We Archive) muted
- become un → Footer finale, large red
- Spanish: El/Los/La → removed or use Una (web)
- Portuguese: O/A → removed or use Uma (web)
- French: Le/Les/La → removed or use Une (web)
- German: Das/Der → removed or use Ein (web)
- Italian: Il/I/Gli/Le → removed or use Un (web)
- Dutch: Het/De → removed or use Een (web)
- Swedish: Den → removed or use En (webb)
Applies 'a web is ephemeral' pattern across all languages
- become ungovernable/unstoppable/untouchable/unseen/unknown/un
- ✝️ cross with "Sorry for the convenience" (Mitch Hedberg)
- Translations in all 27 languages
