feat: optional arborist provenance sink (opt-in, default off)

Adds neopig/arborist_sink.py: when enabled, every crawled page neopig
stores is also committed to an arborist content-addressed store — a
Merkle document_root + append-only audit chain — alongside neopig's
existing md5/FileVault storage. Verifiable dedup + FTS5 + tamper-evident
provenance; replaces nothing.

Strictly opt-in: a hard no-op unless NEOPIG_ARBORIST_ENABLED is set AND
arborist is importable. neopig behaves byte-identically without it. This
keeps neopig public-domain by default — arborist is AGPL, pulled only
when an operator opts in. neopig never touches arborist's tables; all
writes go through arborist.embed (no-raw-SQL rule preserved). Sync
SQLite writes run off the event loop via to_thread, lock-serialized,
and failures are swallowed so the mirror can never break a crawl.

Wired into NeoPig.__init__ (self.arborist_sink) and the store_page hook
in the crawl path. Phase-0 scope: page text only; media-manifest edges
are a follow-on. Tests cover disabled-by-default, flag-without-arborist,
the page->Document mapping, and enabled end-to-end + content-idempotence.

neopig.py

@@ -66,6 +66,7 @@ from neopig.database import Database, SCORE_SCREENSHOT, SCORE_OG_IMAGE, SCORE_TH
 from neopig.screenshot import ScreenshotCapture, ScreenshotConfig
 from neopig.domain_vault import VaultManager, DomainHtmlVault, DomainMediaVault, DomainLinkpeekVault, extract_media_urls
 from neopig.repo import detect_vcs, clone_repo_async, pull_repo_async, get_repo_path, walk_files, get_commit_hash, get_file_language, is_binary_file
+from neopig.arborist_sink import ArboristSink
 from tqdm import tqdm
 
 logger = logging.getLogger(__name__)
@@ -355,6 +356,9 @@ class NeoPig:
         self.screenshot = ScreenshotCapture(screenshot_config or ScreenshotConfig())
         self.screenshot_config = screenshot_config or ScreenshotConfig()
         self.vault_path = vault_path
+        # Optional, opt-in arborist provenance mirror (default OFF; no-op
+        # unless NEOPIG_ARBORIST_ENABLED is set and arborist is installed).
+        self.arborist_sink = ArboristSink()
 
         # Track stats
         self.stats = {
@@ -818,6 +822,16 @@ class NeoPig:
             crawl_job_id=crawl_job_id,
         )
 
+        # Opt-in: also commit the page text to arborist for verifiable
+        # dedup + audit-chained provenance. No-op when the sink is off.
+        await self.arborist_sink.ingest_page(
+            uri=uri,
+            title=title,
+            content=content,
+            path=path,
+            crawl_job_id=crawl_job_id,
+        )
+
     async def _archive_media_to_vault(
         self,
         url: str,