arborist/docs/embedding.md

Embedding arborist in another Python app

arborist is usable as a library: another Python project can use it as a content-addressed, Merkle-committed, audit-chained store for documents it already has — without the CLI, a server, or any wire protocol. The supported surface is arborist.embed. Import from there, not from arborist's internal modules, so internal refactors don't break you.

This is how a downstream archival/crawler app (e.g. neopig) can gain verifiable dedup, FTS5 search, and an append-only audit chain over its archived page text while keeping its own storage for everything else.

The contract

1. You produce Documents. A Document is a uri + normalized text content + a source_type tag (+ optional title, edges, extra).
2. You hand them to ingest. arborist canonicalizes → chunks → Merkle- roots → FTS5-indexes → writes one audit event. Idempotent: same content → same document_root → no-op.
3. arborist owns its SQLite file. Every write goes through this API; you never touch arborist's tables. (That also means an ORM app with a "no raw SQL" rule stays clean — there's no SQL for you to write.)
4. Your own content-address key is orthogonal. If you already dedup by md5 (or anything), keep it — arborist's document_root is an additional hard-hash commitment, not a replacement. See the soft-hash vs hard-hash rule in the project CLAUDE.md.

arborist.embed is core (python + sqlite3); no [crawler]/[nli]/ [vec] extra is needed to embed.

Minimal use

from arborist.embed import open_store, ingest_documents, search, Document, Edge

conn = open_store("data/arborist.db")          # creates + migrates schema

ingest_documents(conn, [
    Document(
        uri="https://example.com/post",
        content="the page's extracted prose",
        source_type="neopig_html",
        title="A Post",
        edges=[Edge(edge_type="embeds_media", dst_uri="https://example.com/img.jpg")],
        extra={"crawl_job_id": 7, "md5": "..."},   # your provenance, carried along
    ),
])

for hit in search(conn, "extracted prose", limit=10):
    print(hit.document_uri)

conn.close()

For a stateful corpus, subclass Source (set source_type, implement iter_documents()) and call arborist.ingest.ingest_source — exactly how arborist's own sources/ are written. For full multi-route retrieval / RAG, use arborist.qa.query directly.

What you get

• Content dedup with proofs — identical content (across different URIs) collapses to one document_root; per-chunk leaves expose partial overlap. See docs/crawler.md.
• FTS5 search over chunked content.
• Append-only audit chain — every ingest is one hash-linked audit_events row; tamper becomes detectable.
• Mesh-ready — peers can re-derive and cross-verify roots.

Surface (arborist.embed)

Symbol	What
open_store(db_path)	open/create/migrate a store; returns a sqlite3.Connection you own
ingest_documents(conn, docs, *, source_type=…, **kw)	ingest an iterable of Documents; returns IngestStats
search(conn, query, *, limit=20)	FTS5 BM25 over chunk content; returns list[Hit]
Document, Edge, Source, Hit, IngestStats	re-exported types