# Embedding arborist in another Python app

arborist is usable as a library: another Python project can use it as a
content-addressed, Merkle-committed, audit-chained store for documents it
already has — without the CLI, a server, or any wire protocol. The
supported surface is **`arborist.embed`**. Import from there, not from
arborist's internal modules, so internal refactors don't break you.

This is how a downstream archival/crawler app (e.g. neopig) can gain
verifiable dedup, FTS5 search, and an append-only audit chain over its
archived page text while keeping its own storage for everything else.

## The contract

1. **You produce `Document`s.** A `Document` is a `uri` + normalized text
   `content` + a `source_type` tag (+ optional `title`, `edges`, `extra`).
2. **You hand them to ingest.** arborist canonicalizes → chunks → Merkle-
   roots → FTS5-indexes → writes one audit event. Idempotent: same content
   → same `document_root` → no-op.
3. **arborist owns its SQLite file.** Every write goes through this API;
   you never touch arborist's tables. (That also means an ORM app with a
   "no raw SQL" rule stays clean — there's no SQL for you to write.)
4. **Your own content-address key is orthogonal.** If you already dedup by
   md5 (or anything), keep it — arborist's `document_root` is an
   *additional* hard-hash commitment, not a replacement. See the
   soft-hash vs hard-hash rule in the project CLAUDE.md.

`arborist.embed` is **core** (python + sqlite3); no `[crawler]`/`[nli]`/
`[vec]` extra is needed to embed.

## Minimal use

```python
from arborist.embed import open_store, ingest_documents, search, Document, Edge

conn = open_store("data/arborist.db")          # creates + migrates schema

ingest_documents(conn, [
    Document(
        uri="https://example.com/post",
        content="the page's extracted prose",
        source_type="neopig_html",
        title="A Post",
        edges=[Edge(edge_type="embeds_media", dst_uri="https://example.com/img.jpg")],
        extra={"crawl_job_id": 7, "md5": "..."},   # your provenance, carried along
    ),
])

for hit in search(conn, "extracted prose", limit=10):
    print(hit.document_uri)

conn.close()
```

For a stateful corpus, subclass `Source` (set `source_type`, implement
`iter_documents()`) and call `arborist.ingest.ingest_source` — exactly how
arborist's own `sources/` are written. For full multi-route retrieval /
RAG, use `arborist.qa.query` directly.

## What you get

- **Content dedup with proofs** — identical content (across different URIs)
  collapses to one `document_root`; per-chunk leaves expose partial
  overlap. See `docs/crawler.md`.
- **FTS5 search** over chunked content.
- **Append-only audit chain** — every ingest is one hash-linked
  `audit_events` row; tamper becomes detectable.
- **Mesh-ready** — peers can re-derive and cross-verify roots.

## Surface (`arborist.embed`)

| Symbol | What |
|---|---|
| `open_store(db_path)` | open/create/migrate a store; returns a `sqlite3.Connection` you own |
| `ingest_documents(conn, docs, *, source_type=…, **kw)` | ingest an iterable of `Document`s; returns `IngestStats` |
| `search(conn, query, *, limit=20)` | FTS5 BM25 over chunk content; returns `list[Hit]` |
| `Document`, `Edge`, `Source`, `Hit`, `IngestStats` | re-exported types |