russell@unturf.com
fc039555cc
aborist/qa/query.py exposes query() — the user-facing RAG flow:
1. FTS5 search across all shards (chunks_fts can't be UNION'd as a
view, so each shard's index is queried independently and merged
by score).
2. Top-K distinct documents are selected within a max-context-chars
budget (default 60 KB so a 768-token response fits Hermes-3's
82 K context window comfortably).
3. context_root = Merkle root over the sorted source document_roots.
That's the v9.8 'source' dimension for multi-source answers —
a verifier can recompute it from the listed source roots.
4. 8-dim cache_key over (context_root, question_hash, model_profile,
conversation, governance_policy, schema, canonicalization,
chunking). Hit returns STRICT immediately; miss calls Hermes and
persists.
CLI: aborist [--shards-dir DIR] query "<question>"
Default qa_db is <shards-dir>/qa.db (or ~/.aborist/qa.db). Uses the
same OpenAICompatibleClient/StubClient as `ask`. --dry-run skips the
LLM and returns context-only.
Search escape fix: the prior FTS5 escape ANDed every token including
stopwords + punctuation, so "What is anarcho-capitalism?" required
the doc to literally contain "what" + "is" + "anarcho-capitalism?" —
zero hits. New tokenizer drops stopwords + punctuation and ORs the
remaining content tokens; BM25 ranks the multi-token matches highest.
Live demo against the 122k-doc 4-shard cluster:
Q "What is anarcho-capitalism?" 6.1 s wall miss / 0.45 s cache hit
Q "Who was George Washington?" 10.3 s wall miss
Both answers cite the source URIs Hermes was given.
57 tests passing (4 new query tests covering search → context →
cache → audit chain).
|
||
|---|---|---|
| aborist | ||
| bench | ||
| tests | ||
| .gitignore | ||
| CLAUDE.md | ||
| LICENSE | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
aborist
An arborist for trees and forests of cross-linked information.
Aborist ingests documents into a content-addressed, Merkle-committed store. Every
document carries a URI for backtrack and cross-link. Search hooks return hits with
declared audit modes (STRICT / HYBRID / VISUAL) so callers never overclaim
what a result actually proves.
Design
Aborist implements the runtime spec from Merkle-AGI v9.8 / Merkle Providence
Reverse RAG. The 8-dim admissibility key (source_root, question_hash,
model_profile_hash, conversation_hash, governance_policy_hash,
schema_version, canonicalization_version, chunking_version) plus
falsification state ensures cached records are never reused under drift.
Two layered document kinds:
- surface — diverse ingested content (full chunks, FTS-indexed)
- core — distilled records (haiku/equation/snippet) Merkle-signed back to
source surface roots via
derivations
Reversible eviction via chunk tier ∈ {hot, warm, cold}. Cold = leaf hash
- URI only; rehydratable from URI, identity verifiable via leaf hash.
Quick start
make all # bootstrap + fetch + ingest + verify + stats
make search Q='…'
License
AGPL-3.0-only.