docs: 3 concepts diagrams + Python-library cookbook recipes
Address Grok's two minor-improvement flags on the docs.
New docs/diagrams/{three-layer-stack,cache-key-8dim,falsification-states}.{dot,svg,png}
embedded into docs/_source/concepts.rst — visual scaffolding for the
3-layer stack, 8-dim cache_key composition, and falsification state
machine (previously prose+tables only).
docs/_source/cookbook.rst gains a "Use arborist as a Python library"
section: open_store + ingest_documents, custom Source subclass,
audit-chain walk + verify, Merkle proof round-trip, programmatic
arborist.qa.query() with OpenAICompatibleClient + StubClient swap.
Every Python recipe smoke-tested against a scratch DB before publish.
make docs-api: 0 new warnings. make test: 2557 passed.
This commit is contained in:
parent
b6bb31a836
commit
06e6c7a918
11 changed files with 1045 additions and 0 deletions
|
|
@ -141,3 +141,236 @@ keywords without changing what the LLM sees as the question:
|
|||
|
||||
Provenance gap on this is tracked in
|
||||
:doc:`api/qa` (``arborist.qa.query``).
|
||||
|
||||
Use arborist as a Python library
|
||||
================================
|
||||
|
||||
Everything below uses the supported embedding surface,
|
||||
:mod:`arborist.embed`. Import from there, not from internal modules —
|
||||
internal refactors are free to move things around behind that seam.
|
||||
See :doc:`api/storage` for the full ``arborist.store`` reference and
|
||||
:doc:`api/substrate` for the Merkle primitives the recipes call into.
|
||||
|
||||
Open a store and ingest documents you already hold
|
||||
---------------------------------------------------
|
||||
|
||||
The minimum useful contact surface: pass in your own
|
||||
:class:`~arborist.document.Document` objects, get back content-addressed
|
||||
storage with an audit chain. Idempotent — re-running with the same
|
||||
``content`` yields the same ``document_root`` and skips the insert.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from pathlib import Path
|
||||
from arborist.embed import (
|
||||
open_store, ingest_documents, search, Document, Edge,
|
||||
)
|
||||
|
||||
conn = open_store(Path("data/arborist.db")) # creates + migrates
|
||||
|
||||
stats = ingest_documents(conn, [
|
||||
Document(
|
||||
uri="https://example.com/post-a",
|
||||
content="anarcho-capitalism describes a stateless society "
|
||||
"where private property and free markets coordinate "
|
||||
"without coercion.",
|
||||
source_type="my_app",
|
||||
title="Anarcho-Capitalism Primer",
|
||||
edges=[Edge(edge_type="references",
|
||||
dst_uri="https://example.com/post-b")],
|
||||
extra={"md5": "deadbeef"}, # your provenance, carried along
|
||||
),
|
||||
])
|
||||
print(stats) # IngestStats(seen=1, inserted=1, ...)
|
||||
|
||||
for hit in search(conn, "free markets", limit=5):
|
||||
print(hit.document_uri, round(hit.score, 3))
|
||||
|
||||
conn.close()
|
||||
|
||||
Define a custom :class:`~arborist.source.Source` for stateful corpora
|
||||
---------------------------------------------------------------------
|
||||
|
||||
When you have a corpus (a directory tree, a paginated API, a database
|
||||
table) it's cleaner to express it as a :class:`Source`. The ABC has one
|
||||
required method, :meth:`iter_documents`, which must be deterministic
|
||||
and idempotent. That's exactly the contract every built-in source under
|
||||
``arborist/sources/`` already implements.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from arborist.embed import Source, Document
|
||||
from arborist.ingest import ingest_source
|
||||
from arborist.embed import open_store
|
||||
|
||||
class TaggedDocs(Source):
|
||||
"""Ingest a list of (uri, body) pairs under a shared tag."""
|
||||
|
||||
source_type = "tagged_example"
|
||||
|
||||
def __init__(self, tag, items):
|
||||
self.tag = tag
|
||||
self._items = items
|
||||
|
||||
def iter_documents(self):
|
||||
for uri, body in self._items:
|
||||
yield Document(
|
||||
uri=uri,
|
||||
content=body,
|
||||
source_type=self.source_type,
|
||||
extra={"tag": self.tag},
|
||||
)
|
||||
|
||||
conn = open_store("data/arborist.db")
|
||||
stats = ingest_source(conn, TaggedDocs("research", [
|
||||
("https://example.com/c", "third doc body about content-addressing."),
|
||||
]))
|
||||
print(stats)
|
||||
conn.close()
|
||||
|
||||
Walk and verify the audit chain
|
||||
-------------------------------
|
||||
|
||||
Every state-changing op writes one row in ``audit_events`` with
|
||||
``event_hash = sha256(prev_event_hash || canonical(body))``. Verifying
|
||||
the chain is just re-running that hash for every row and checking the
|
||||
linkage. (``make chain-check-shards`` does this at scale; the recipe
|
||||
below is the same logic, inlined.)
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import hashlib
|
||||
from arborist.embed import open_store
|
||||
from arborist.store import latest_event_hash
|
||||
|
||||
conn = open_store("data/arborist.db")
|
||||
print("head:", latest_event_hash(conn))
|
||||
|
||||
prev = None
|
||||
bad = 0
|
||||
for seq, eh, ph, body in conn.execute(
|
||||
"SELECT seq, event_hash, prev_event_hash, body "
|
||||
"FROM audit_events ORDER BY seq"
|
||||
):
|
||||
h = hashlib.sha256()
|
||||
if ph is not None:
|
||||
h.update(bytes.fromhex(ph))
|
||||
h.update(body.encode("utf-8", errors="surrogatepass"))
|
||||
if h.hexdigest() != eh or (prev is not None and ph != prev):
|
||||
bad += 1
|
||||
prev = eh
|
||||
|
||||
print(f"chain breaks: {bad}") # 0 = intact
|
||||
conn.close()
|
||||
|
||||
Round-trip a Merkle inclusion proof
|
||||
-----------------------------------
|
||||
|
||||
The proof primitives from :mod:`arborist.merkle` are the Python port of
|
||||
``proxy.unturf.com``'s Go conventions. Use them to re-derive a
|
||||
``document_root`` from its leaves, build a proof for any chunk, and
|
||||
serialize the proof for over-the-wire delivery to another peer.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import json
|
||||
from arborist.embed import open_store
|
||||
from arborist.merkle import (
|
||||
MerkleTree, verify_proof, proof_to_dict, proof_from_dict,
|
||||
)
|
||||
|
||||
conn = open_store("data/arborist.db")
|
||||
|
||||
doc_root_hex, = conn.execute(
|
||||
"SELECT document_root FROM documents LIMIT 1"
|
||||
).fetchone()
|
||||
leaves = [
|
||||
bytes.fromhex(r[0]) for r in conn.execute(
|
||||
"SELECT leaf_hash FROM chunks WHERE document_root=? ORDER BY idx",
|
||||
(doc_root_hex,),
|
||||
)
|
||||
]
|
||||
|
||||
tree = MerkleTree.build(leaves)
|
||||
assert tree.root.hex() == doc_root_hex # bit-identical re-derivation
|
||||
|
||||
proof = tree.proof(0)
|
||||
assert verify_proof(proof) # local round-trip
|
||||
|
||||
blob = json.dumps(proof_to_dict(proof)) # serialize for wire
|
||||
proof_received = proof_from_dict(json.loads(blob))
|
||||
assert verify_proof(proof_received) # any peer can verify
|
||||
|
||||
conn.close()
|
||||
|
||||
Two peers that ingested the same source with the same chunker and
|
||||
canonicalization will compute byte-identical ``document_root`` hashes
|
||||
and accept each other's proofs — see :doc:`api/mesh` for the
|
||||
federation primitives built on top of this property.
|
||||
|
||||
Run a Q&A from Python and read the result programmatically
|
||||
-----------------------------------------------------------
|
||||
|
||||
The CLI (``arborist query`` / ``make query``) is one entry point to
|
||||
the multi-route retrieval + LLM + verifier pipeline. The same surface
|
||||
is callable directly from Python via :func:`arborist.qa.query.query` —
|
||||
useful when you want to drive a batch sweep, integrate into a notebook,
|
||||
or wrap the result in your own application logic. Cache lookups, the
|
||||
8-dim cache_key, the verifier, the run-DAG, and the falsification gate
|
||||
all behave identically to the CLI path.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from pathlib import Path
|
||||
from arborist.qa.query import query
|
||||
from arborist.qa.client import OpenAICompatibleClient
|
||||
|
||||
client = OpenAICompatibleClient(
|
||||
base_url="https://hermes.ai.unturf.com/v1", # any OpenAI-compat
|
||||
)
|
||||
|
||||
result = query(
|
||||
question="What is anarcho-capitalism?",
|
||||
qa_db=Path.home() / ".arborist" / "qa.db", # cache lives here
|
||||
chat_client=client,
|
||||
model_id="adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
|
||||
shards_dir=Path.home() / ".arborist" / "shards", # OR single_db=...
|
||||
)
|
||||
|
||||
print(result["status"]) # cache_hit | cache_miss_then_written
|
||||
print(result["audit_mode"]) # STRICT | HYBRID | UNGROUNDED
|
||||
print(result["cache_key"]) # 64-char hex
|
||||
print(result["answer_text"])
|
||||
for src in result["sources"]:
|
||||
print(" ->", src["document_uri"], src["document_root"][:12],
|
||||
"role:", src["source_role"])
|
||||
|
||||
The first call writes one providence-cache row + one audit event;
|
||||
the second call with the same question replays from cache in ~100 ms
|
||||
(``status == "cache_hit"``). To exercise the pipeline deterministically
|
||||
in tests, swap the live client for :class:`~arborist.qa.client.StubClient`:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from arborist.qa.client import StubClient
|
||||
|
||||
stub = StubClient(answer='Anarcho-capitalism is "a political '
|
||||
'philosophy that advocates the elimination '
|
||||
'of centralized state dictums".')
|
||||
result = query(
|
||||
question="What is anarcho-capitalism?",
|
||||
qa_db=Path("/tmp/qa.db"),
|
||||
chat_client=stub,
|
||||
model_id="stub/test",
|
||||
single_db=Path("/tmp/arborist.db"),
|
||||
)
|
||||
assert result["audit_mode"] in {"STRICT", "HYBRID", "UNGROUNDED"}
|
||||
|
||||
That stubbed shape is exactly how the test suite drives the runner —
|
||||
the verifier still runs lexically against the assembled context, so
|
||||
the audit_mode classification is real even though the model output
|
||||
is canned.
|
||||
|
||||
For the single-document path (``arborist ask`` / one ``document_root``
|
||||
in hand), use :func:`arborist.qa.runner.ask` instead — same return
|
||||
shape, same cache_key invariants, scoped to one document.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue