russell@unturf.com
5413b83d45
Two issues surfaced by "tell me about permacomputer ?":
1. Stopword leak. _FTS5_STOPWORDS got "tell" added earlier but the
parallel _TITLE_STOPWORDS in qa/query.py wasn't synced. Title-LIKE
search picked up "tell" and returned every Tell_* article in the
corpus (Tell_(poker), Tell-Tale_Heart, Tell_Township, Tell_City,
etc.) all uniformly scored at 70.0. Synced the lists — title
tokenizer now drops the same question-shaping verbs as FTS5.
2. Grok docs lived only in --db (default ~/.aborist/aborist.db),
not in --shards-dir. `make query` reads the shard cluster, so
Grok was invisible to it. Added two targets:
ingest-grok-attached conversations -> $(SHARDS_DIR)/grok.db
ingest-grok-media-attached media prompts -> $(SHARDS_DIR)/grok.db
Both use --resume so re-runs are cheap.
Demo: "tell me about permacomputer ?" now answers from the
"MOAD Cheat Sheet: Common Software Defects" Grok conversation —
fox's own prior chat surfaced as corpus.
Note: 000.db already had Grok docs from an earlier ingest variant.
The cross-shard UNION currently sees them twice (376 rows, 188
unique document_roots). Retrieval is unaffected because
_search_corpus dedupes by root, but stats counts are inflated.
Cleanup is destructive (drop one copy) — fox decides which.
79 tests passing.
|
||
|---|---|---|
| aborist | ||
| bench | ||
| tests | ||
| .gitignore | ||
| CLAUDE.md | ||
| LICENSE | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
aborist
An arborist for trees and forests of cross-linked information.
Aborist ingests documents into a content-addressed, Merkle-committed store. Every
document carries a URI for backtrack and cross-link. Search hooks return hits with
declared audit modes (STRICT / HYBRID / VISUAL) so callers never overclaim
what a result actually proves.
Design
Aborist implements the runtime spec from Merkle-AGI v9.8 / Merkle Providence
Reverse RAG. The 8-dim admissibility key (source_root, question_hash,
model_profile_hash, conversation_hash, governance_policy_hash,
schema_version, canonicalization_version, chunking_version) plus
falsification state ensures cached records are never reused under drift.
Two layered document kinds:
- surface — diverse ingested content (full chunks, FTS-indexed)
- core — distilled records (haiku/equation/snippet) Merkle-signed back to
source surface roots via
derivations
Reversible eviction via chunk tier ∈ {hot, warm, cold}. Cold = leaf hash
- URI only; rehydratable from URI, identity verifiable via leaf hash.
Quick start
make all # bootstrap + fetch + ingest + verify + stats
make search Q='…'
License
AGPL-3.0-only.