russell@unturf.com
a4e1dc9a10
A stable façade so another Python app can use arborist as a content-addressed / Merkle / audit-chained store without the CLI or a wire protocol. Import from arborist.embed, not internal modules, so refactors don't break embedders. Surface: open_store(path), ingest_documents(conn, docs), search(conn, q), plus re-exported Document/Edge/Source/Hit/IngestStats. Core only (python+sqlite3) — no extras. _IterableSource adapts a plain doc iterable into the Source contract. This is the seam for using arborist as neopig's optional provenance backend: neopig produces Documents from crawled pages, arborist gives content-dedup (document_root) + FTS5 + an append-only audit chain alongside neopig's existing md5/FileVault storage. Docs in docs/embedding.md. 6 tests pin open/ingest/dedup/idempotence/edges/search.
77 lines
3.3 KiB
Markdown
77 lines
3.3 KiB
Markdown
# Embedding arborist in another Python app
|
|
|
|
arborist is usable as a library: another Python project can use it as a
|
|
content-addressed, Merkle-committed, audit-chained store for documents it
|
|
already has — without the CLI, a server, or any wire protocol. The
|
|
supported surface is **`arborist.embed`**. Import from there, not from
|
|
arborist's internal modules, so internal refactors don't break you.
|
|
|
|
This is how a downstream archival/crawler app (e.g. neopig) can gain
|
|
verifiable dedup, FTS5 search, and an append-only audit chain over its
|
|
archived page text while keeping its own storage for everything else.
|
|
|
|
## The contract
|
|
|
|
1. **You produce `Document`s.** A `Document` is a `uri` + normalized text
|
|
`content` + a `source_type` tag (+ optional `title`, `edges`, `extra`).
|
|
2. **You hand them to ingest.** arborist canonicalizes → chunks → Merkle-
|
|
roots → FTS5-indexes → writes one audit event. Idempotent: same content
|
|
→ same `document_root` → no-op.
|
|
3. **arborist owns its SQLite file.** Every write goes through this API;
|
|
you never touch arborist's tables. (That also means an ORM app with a
|
|
"no raw SQL" rule stays clean — there's no SQL for you to write.)
|
|
4. **Your own content-address key is orthogonal.** If you already dedup by
|
|
md5 (or anything), keep it — arborist's `document_root` is an
|
|
*additional* hard-hash commitment, not a replacement. See the
|
|
soft-hash vs hard-hash rule in the project CLAUDE.md.
|
|
|
|
`arborist.embed` is **core** (python + sqlite3); no `[crawler]`/`[nli]`/
|
|
`[vec]` extra is needed to embed.
|
|
|
|
## Minimal use
|
|
|
|
```python
|
|
from arborist.embed import open_store, ingest_documents, search, Document, Edge
|
|
|
|
conn = open_store("data/arborist.db") # creates + migrates schema
|
|
|
|
ingest_documents(conn, [
|
|
Document(
|
|
uri="https://example.com/post",
|
|
content="the page's extracted prose",
|
|
source_type="neopig_html",
|
|
title="A Post",
|
|
edges=[Edge(edge_type="embeds_media", dst_uri="https://example.com/img.jpg")],
|
|
extra={"crawl_job_id": 7, "md5": "..."}, # your provenance, carried along
|
|
),
|
|
])
|
|
|
|
for hit in search(conn, "extracted prose", limit=10):
|
|
print(hit.document_uri)
|
|
|
|
conn.close()
|
|
```
|
|
|
|
For a stateful corpus, subclass `Source` (set `source_type`, implement
|
|
`iter_documents()`) and call `arborist.ingest.ingest_source` — exactly how
|
|
arborist's own `sources/` are written. For full multi-route retrieval /
|
|
RAG, use `arborist.qa.query` directly.
|
|
|
|
## What you get
|
|
|
|
- **Content dedup with proofs** — identical content (across different URIs)
|
|
collapses to one `document_root`; per-chunk leaves expose partial
|
|
overlap. See `docs/crawler.md`.
|
|
- **FTS5 search** over chunked content.
|
|
- **Append-only audit chain** — every ingest is one hash-linked
|
|
`audit_events` row; tamper becomes detectable.
|
|
- **Mesh-ready** — peers can re-derive and cross-verify roots.
|
|
|
|
## Surface (`arborist.embed`)
|
|
|
|
| Symbol | What |
|
|
|---|---|
|
|
| `open_store(db_path)` | open/create/migrate a store; returns a `sqlite3.Connection` you own |
|
|
| `ingest_documents(conn, docs, *, source_type=…, **kw)` | ingest an iterable of `Document`s; returns `IngestStats` |
|
|
| `search(conn, query, *, limit=20)` | FTS5 BM25 over chunk content; returns `list[Hit]` |
|
|
| `Document`, `Edge`, `Source`, `Hit`, `IngestStats` | re-exported types |
|