russell@unturf.com
13ae4cfb24
The query "what is the fastest AMD CPU?" was returning Pentium_4 as a top source, and Hermes hallucinated by conflating Intel content with AMD context. Two compounding causes: 1. FTS5 was ORing all content tokens, so any doc with "fastest" OR "CPU" matched. BM25 then ranked Pentium_4 highly because the article is long and discusses CPU clock speed in detail. 2. The reranker only boosted titles that matched — it never penalized titles that didn't match. Pentium_4 had no penalty despite being topically wrong. Two fixes: - FTS5 escape now ANDs content tokens by default. Stopword list expanded to include question-shaping verbs (tell/show/describe/ explain/find) so they don't sneak in as required tokens. If AND returns zero hits, falls back to OR — pragmatic for fuzzy queries the user phrases differently than the article wording. All-stopword inputs still produce a non-empty MATCH so FTS5 doesn't error. - _filter_by_title_relevance hard-drops hits whose title shares no token with the query, before assembling context. If filtering would leave fewer than 3 hits, falls back to the unfiltered set (corpus might genuinely not have a topical match — let the LLM say "I don't know" rather than nothing). Together: AND-mode requires every content token in the body, and title filter requires the doc to actually be about the topic. Pentium articles drop out of AMD queries; Tell_(poker) drops out of plot queries. 57 tests still passing. |
||
|---|---|---|
| aborist | ||
| bench | ||
| tests | ||
| .gitignore | ||
| CLAUDE.md | ||
| LICENSE | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
aborist
An arborist for trees and forests of cross-linked information.
Aborist ingests documents into a content-addressed, Merkle-committed store. Every
document carries a URI for backtrack and cross-link. Search hooks return hits with
declared audit modes (STRICT / HYBRID / VISUAL) so callers never overclaim
what a result actually proves.
Design
Aborist implements the runtime spec from Merkle-AGI v9.8 / Merkle Providence
Reverse RAG. The 8-dim admissibility key (source_root, question_hash,
model_profile_hash, conversation_hash, governance_policy_hash,
schema_version, canonicalization_version, chunking_version) plus
falsification state ensures cached records are never reused under drift.
Two layered document kinds:
- surface — diverse ingested content (full chunks, FTS-indexed)
- core — distilled records (haiku/equation/snippet) Merkle-signed back to
source surface roots via
derivations
Reversible eviction via chunk tier ∈ {hot, warm, cold}. Cold = leaf hash
- URI only; rehydratable from URI, identity verifiable via leaf hash.
Quick start
make all # bootstrap + fetch + ingest + verify + stats
make search Q='…'
License
AGPL-3.0-only.