russell@unturf.com
326badf6d8
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).
(1) README — label refresh:
- Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
- Verifier section spells out both layers (schema trichotomy +
display ladder), the seven hard checks of claim_lattice, and
the five anchor classes of warrant.
- Architecture tree updated: concepts/ package added, qa/
sub-modules expanded (warrant, evidence, parse_claims, dag),
verify.py described as quote/span/entity/paraphrase + claim_lattice.
- Concept overlay description updated for corpus-derived layer
(concept_relations table, link_reciprocity extractor, 1.6%
tax cite).
- Test count: 326+ → 641+.
(2) docs/diagrams/ — Graphviz dot sources:
- aborist-modules.dot — top-level package graph (substrate /
storage / sources / retrieval / qa / mesh / cli)
- query-pipeline.dot — question → cache → retrieval → LLM →
verify → render → cache write, with phase budgets
- ingest-pipeline.dot — source doc → canonicalize → chunk →
Merkle → upsert (+ optional distill)
- verifier-ladder.dot — (audit_mode, violations) → display rung
decision tree
Existing mesh-*.dot kept as-is. Makefile `make docs` target
extended to also emit .svg alongside the existing .png so the
diagrams render in markdown viewers.
(3) docs/modules/ — per-module reference pages:
- index.md (links to every diagram + every module page)
- merkle.md, document.md, store.md, ingest.md, evict.md,
sources.md, search.md, concepts.md, qa.md, distill.md,
wikitext.md
Each page is a one-screenful concise reference: what the
module is for, public API, key invariants, embedded diagrams
where useful, link to source. Mesh stays at the existing
docs/mesh.md + docs/mesh-deploy.md (already comprehensive).
Tests: 641 passed (no code change).
2.6 KiB
aborist.search
The retrieval primitive. Today's only backend is FTS5 over the
chunks table; the SearchBackend ABC is in place so additional
backends (BM25 over titles, embedding-based vector search) can be
added without touching the rest of the QA pipeline.
SearchBackend ABC
from aborist.search import SearchBackend, AuditMode, Hit
class SearchBackend(ABC):
@abstractmethod
def search(self, query: str, limit: int = 20) -> list[Hit]: ...
Each Hit carries (document_root, document_uri, chunk_idx, snippet, score, audit_mode, title). audit_mode is the sticky
provenance label that tracks how the chunk made it into the index;
FTS5 backend always sets UNGROUNDED (search itself doesn't verify
anything — that's the QA pipeline's job).
FTS5Backend
Wraps the contentless chunks_fts virtual table. Two-mode query:
- AND-mode (strict, primary): every content token must appear in the doc. Keeps unrelated docs out of the context window.
- OR-mode (fallback): when AND returns 0 hits, fall back to OR
but capped to top-5 longest tokens (proxy for rarity). Long
topical synonyms fed via
extra_or_tokensjoin the pool —neurotechnology(15 chars) outranksthoughts(8) by length and surfaces brain-tech titles for vocabulary-mismatch queries.
from aborist.search import FTS5Backend
backend = FTS5Backend(conn)
# Plain search
hits = backend.search("permacomputer", limit=32)
# Search with synonym pool injection (used by qa.query._search_corpus)
hits = backend.search(
long_query,
limit=32,
extra_or_tokens=synonym_expand(qtokens, shards_dir=shards_dir),
)
Stopword & stopword-cap discipline
_FTS5_STOPWORDS filters question words (what, tell, please)
- generic connectors (
one,some,another,without,soon,currently) before AND/OR construction. Two principles:
- Stay in sync with
_TITLE_STOPWORDSinqa/query.py. A token filtered at retrieval time but kept at title-relevance check (or vice versa) creates ranking incoherence. _OR_FALLBACK_MAX_TOKENS=5caps the OR-mode pool. Without this, a 19-token OR clause matches millions of docs and forces BM25 to rank them all — 13s/shard observed pre-cap. Now 0.25s/shard.
Snippet building
FTS5 contentless mode means SQLite's built-in snippet() and
highlight() return empty. Aborist builds snippets in Python by
joining chunks_fts.rowid = chunks.chunk_id, decompressing the
chunk content, and locating query tokens locally
(_build_snippet).