arborist/tests
russell@unturf.com fc039555cc
multi-source corpus query: pose a question, the tree pulls related cached docs
aborist/qa/query.py exposes query() — the user-facing RAG flow:

  1. FTS5 search across all shards (chunks_fts can't be UNION'd as a
     view, so each shard's index is queried independently and merged
     by score).
  2. Top-K distinct documents are selected within a max-context-chars
     budget (default 60 KB so a 768-token response fits Hermes-3's
     82 K context window comfortably).
  3. context_root = Merkle root over the sorted source document_roots.
     That's the v9.8 'source' dimension for multi-source answers —
     a verifier can recompute it from the listed source roots.
  4. 8-dim cache_key over (context_root, question_hash, model_profile,
     conversation, governance_policy, schema, canonicalization,
     chunking). Hit returns STRICT immediately; miss calls Hermes and
     persists.

CLI: aborist [--shards-dir DIR] query "<question>"
  Default qa_db is <shards-dir>/qa.db (or ~/.aborist/qa.db). Uses the
  same OpenAICompatibleClient/StubClient as `ask`. --dry-run skips the
  LLM and returns context-only.

Search escape fix: the prior FTS5 escape ANDed every token including
stopwords + punctuation, so "What is anarcho-capitalism?" required
the doc to literally contain "what" + "is" + "anarcho-capitalism?" —
zero hits. New tokenizer drops stopwords + punctuation and ORs the
remaining content tokens; BM25 ranks the multi-token matches highest.

Live demo against the 122k-doc 4-shard cluster:
  Q "What is anarcho-capitalism?"  6.1 s wall miss / 0.45 s cache hit
  Q "Who was George Washington?"   10.3 s wall miss
Both answers cite the source URIs Hermes was given.

57 tests passing (4 new query tests covering search → context →
cache → audit chain).
2026-04-27 11:43:57 -04:00
..
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_distill.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_distill_recursive.py recursive distillation: core -> depth+1 core 2026-04-27 07:54:30 -04:00
test_evict.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_html_source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_ingest.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
test_qa.py add Q&A layer: v9.8 providence_cache writes with Merkle-bound proofs 2026-04-27 08:01:23 -04:00
test_query.py multi-source corpus query: pose a question, the tree pulls related cached docs 2026-04-27 11:43:57 -04:00
test_resume.py resumable ingest + per-shard audit chain integrity 2026-04-27 11:29:27 -04:00
test_tfidf.py add TF-IDF keyword distiller 2026-04-27 07:55:53 -04:00
test_versioned_ingest.py versioned re-ingest: same URI + changed content writes a 'supersedes' edge 2026-04-27 07:56:49 -04:00
test_wikipedia_old.py prepare full Wikipedia 2003-05-16 ingest: cur + old (revisions) 2026-04-27 08:10:42 -04:00