Adds neopig/arborist_sink.py: when enabled, every crawled page neopig
stores is also committed to an arborist content-addressed store — a
Merkle document_root + append-only audit chain — alongside neopig's
existing md5/FileVault storage. Verifiable dedup + FTS5 + tamper-evident
provenance; replaces nothing.
Strictly opt-in: a hard no-op unless NEOPIG_ARBORIST_ENABLED is set AND
arborist is importable. neopig behaves byte-identically without it. This
keeps neopig public-domain by default — arborist is AGPL, pulled only
when an operator opts in. neopig never touches arborist's tables; all
writes go through arborist.embed (no-raw-SQL rule preserved). Sync
SQLite writes run off the event loop via to_thread, lock-serialized,
and failures are swallowed so the mirror can never break a crawl.
Wired into NeoPig.__init__ (self.arborist_sink) and the store_page hook
in the crawl path. Phase-0 scope: page text only; media-manifest edges
are a follow-on. Tests cover disabled-by-default, flag-without-arborist,
the page->Document mapping, and enabled end-to-end + content-idempotence.
russell@unturf.com
ced92de67c