russell@unturf.com
a9ee859657
Both items surfaced by the 2026-05-07 arborist-vs-donto comparison report (/home/fox/Downloads/arborist_vs_donto.pdf). Ticket #000022 — adapter LossReport (PRD I9 analogue). Today wikitext to_base() and html_page _normalize_text drop <ref> tags, image/category wikilinks, HTML chrome, whitespace runs without recording any of it; only the canonicalization-version pin survives. Ticket proposes a sidecar adapter_loss_reports table (Option A) over Merkle-bound loss_root (B) or audit-chain entries (C), preserving arborist's "soft signals are sidecars" discipline. ~1.6% storage tax expected, matching concept_relations. Doc-only; no code in this commit. Federation doc-discoverability: PDF author concluded "federation exists in code but the public docs page returned 404" because the mesh page lives at /api/mesh.html and the concepts orientation never surfaces the topic. Adds a "Federation across peers" section to concepts.rst pointing at api/mesh, a :ref:\`federation\` anchor + context lead on api/mesh.rst, and a footer link in concepts.rst's "Where to go next." Sphinx build passes; api/mesh.html#federation deep link resolves.
195 lines
7.6 KiB
ReStructuredText
195 lines
7.6 KiB
ReStructuredText
Concepts
|
|
========
|
|
|
|
Arborist is a content-addressed document store that gives every cached
|
|
LLM answer a verifiable Merkle proof tying it back to its source
|
|
documents. This page is the orientation: what the system is, the
|
|
core abstractions you'll see in code and docs, and how they compose.
|
|
|
|
What arborist is
|
|
---------------
|
|
|
|
A reference implementation of two papers stacked:
|
|
|
|
1. **Merkle Providence Reverse RAG** (whitepaper, April 2026) — the
|
|
runtime: question → cache → retrieval → LLM → verifier → cache write.
|
|
2. **Merkle-AGI v9.8** — the admissibility ledger that makes cached
|
|
answers cryptographically verifiable across peers.
|
|
|
|
Three layers stacked on one SQLite file (per shard):
|
|
|
|
- **Surface** — ingested documents (Wikipedia, HTML pages, anything
|
|
with a URI). Chunked, Merkle-rooted, FTS5-indexed.
|
|
- **Core** — distilled documents Merkle-bound back to surfaces via
|
|
per-chunk inclusion proofs. Recursive — a depth-N core can compress
|
|
into a depth-N+1 core.
|
|
- **Providence cache** — Q&A records keyed on the v9.8 8-dimension
|
|
invariant; every record carries an audit_mode and a Merkle proof.
|
|
|
|
.. figure:: diagrams/arborist-modules.svg
|
|
:alt: Arborist module graph
|
|
:width: 100%
|
|
|
|
Top-level module graph. Substrate (merkle, document) at the bottom;
|
|
pipelines (ingest, distill, qa) in the middle; CLI on top.
|
|
|
|
The Merkle commitment
|
|
---------------------
|
|
|
|
Two peers that ingest the same source + run the same chunker +
|
|
canonicalization compute **bit-identical** ``document_root`` hashes.
|
|
That is the v9.8 admissibility property: identity by content, not by
|
|
location.
|
|
|
|
Arborist uses fox's existing Go Merkle conventions verbatim
|
|
(``proxy.unturf.com/pkg/verified/merkle.go``):
|
|
|
|
- **Leaf hash:** ``sha256(0x00 || canonical_chunk_bytes)``.
|
|
- **Internal hash:** ``sha256(0x03 || left || right)``. Non-commutative —
|
|
``H(L,R) ≠ H(R,L)``.
|
|
- **Odd layers self-duplicate** the trailing element. Not zero-pad.
|
|
- **Proof carries explicit IsLeft flag** per sibling. Not lexical sort.
|
|
|
|
See :doc:`api/substrate` for the Python port.
|
|
|
|
Federation across peers
|
|
-----------------------
|
|
|
|
Identity-by-content is the federation primitive. Two peers that
|
|
ingest the same source with the same chunker and canonicalization
|
|
compute the same ``document_root`` and the same ``cache_key``;
|
|
that property is what lets peers exchange and verify each other's
|
|
providence records without a central authority. The mesh layer
|
|
implements gossip-based sync, group key management, and per-peer
|
|
audit-chain reconciliation on top of that.
|
|
|
|
See :doc:`api/mesh` (also reachable as :ref:`federation`) for the
|
|
mesh wire format, member-set semantics, and deploy notes.
|
|
|
|
The 8-dim cache key
|
|
-------------------
|
|
|
|
Every providence record is keyed on:
|
|
|
|
.. code-block::
|
|
|
|
cache_key = sha256(
|
|
source_root | question_hash | model_profile_hash |
|
|
conversation_hash | governance_policy_hash |
|
|
schema_version | canonicalization_version | chunking_version
|
|
)
|
|
|
|
Bumping any of the eight dimensions invalidates prior records on
|
|
lookup. This is how the system stays honest across model changes,
|
|
schema migrations, prompt edits, etc. — old answers don't silently
|
|
serve under new conditions.
|
|
|
|
The audit chain
|
|
---------------
|
|
|
|
Every state-changing operation writes one row in ``audit_events``
|
|
with ``event_hash = sha256(prev_event_hash || canonical(body))``.
|
|
Linear chain per shard, verified by ``make chain-check-shards`` (any
|
|
break is the loudest possible signal).
|
|
|
|
Always write via ``arborist.store.append_audit`` — never insert into
|
|
``audit_events`` directly.
|
|
|
|
The trichotomy and the four-rung ladder
|
|
----------------------------------------
|
|
|
|
Every answer carries two stacked labels.
|
|
|
|
**Schema layer** — v9.8 trichotomy, persisted, drives cache lookups
|
|
and the audit chain:
|
|
|
|
============= =============================================================
|
|
``audit_mode`` meaning
|
|
============= =============================================================
|
|
STRICT every evidence unit verifies against context
|
|
HYBRID some claims source-grounded, some emerged from training
|
|
UNGROUNDED no evidence, or none verifies — purely emergent
|
|
============= =============================================================
|
|
|
|
**Display layer** — four-rung ladder for claim-lattice modes only;
|
|
renderer-only transformation, schema unchanged:
|
|
|
|
==================== ==========================================================
|
|
rung what's actually proved
|
|
==================== ==========================================================
|
|
EVIDENCE-WARRANTED pointer verified + warrant ran & passed + no soft demotes
|
|
ANCHOR-WARRANTED pointer-linked + warrant passed; soft-demote violations
|
|
POINTER-LINKED pointer/source/chunk verified, but warrant didn't apply
|
|
UNGROUNDED no verified pairs
|
|
==================== ==========================================================
|
|
|
|
The point of the display ladder: ``STRICT`` in claim-lattice mode
|
|
is *not* "the answer is correct" — it's "every pointer resolved to a
|
|
valid evidence object, source_role allowed, citation coverage passed."
|
|
The display label spells out the actual property so users don't read
|
|
``STRICT`` as full semantic entailment.
|
|
|
|
.. figure:: diagrams/verifier-ladder.svg
|
|
:alt: Verifier ladder
|
|
:width: 80%
|
|
|
|
How ``(audit_mode, violations)`` maps to a display rung at render time.
|
|
|
|
The verifier (lexical, layered)
|
|
-------------------------------
|
|
|
|
Five strategies run in sequence; first to find evidence classifies.
|
|
Each is **lexical** (substring or token coverage), never embeddings —
|
|
soft signals stay out of the proof path.
|
|
|
|
1. **quote** — model wraps claims in ``"..."``; verbatim substring tested
|
|
2. **span** — bullet/sentence units substring-tested
|
|
3. **entity** — multi-word proper nouns with proximity-cluster gating
|
|
4. **paraphrase** — ≥85% token coverage on prose-shaped spans
|
|
5. **claim_lattice** — pointer-line ``[E1,E2]`` or JSON; runs **seven
|
|
deterministic hard checks**:
|
|
parser succeeded, evidence_id resolves, source_role allowed, claim
|
|
text non-empty, citation coverage threshold, pointer count cap,
|
|
anchor-class warrant.
|
|
|
|
See :doc:`api/qa` for the implementation; the whitepaper §13.8
|
|
covers the layered design.
|
|
|
|
Falsification state
|
|
-------------------
|
|
|
|
Every providence record carries a ``falsification_state`` in
|
|
``{live, failed, stale, quarantined}``. Cache lookups filter on
|
|
``state='live'``. Drift detection (re-ingest produces a different
|
|
``document_root``) flips the record to ``stale``.
|
|
|
|
Two ways to remove a wrong answer:
|
|
|
|
- ``make falsify KEY=...`` — record stays in DB, state flips to
|
|
``failed``. Audit-preserving. Use when downstream consumers might
|
|
reference it.
|
|
- ``make burn KEY=...`` — actually deletes the row. Refuses if the
|
|
record has children unless ``FORCE=1``. Use during scratch corpus
|
|
building.
|
|
|
|
Sidecar diagnostics
|
|
-------------------
|
|
|
|
The verifier stays binary; soft signals proliferate as **sidecars** —
|
|
read-only diagnostic functions that pull the same source chunks the
|
|
verifier saw, classify spans, detect topic shifts, surface metaphor
|
|
framing, and **never write to providence_cache or extend the audit
|
|
chain**. That invariant is what keeps ``audit_mode`` a binary
|
|
classification rather than a soft score.
|
|
|
|
See :doc:`api/qa` (``arborist.qa.inspect``).
|
|
|
|
Where to go next
|
|
----------------
|
|
|
|
* :doc:`quickstart` — install and run the canonical paths
|
|
* :doc:`api/makefile` — every workflow as a ``make`` target
|
|
* :doc:`api/substrate` — Merkle tree + document primitives
|
|
* :doc:`api/qa` — Q&A pipeline, verifier, evidence map, run-DAG
|
|
* :doc:`api/mesh` — federation primitives across peers
|
|
* :doc:`license` — full AGPL + Permacomputer Preamble
|