- make crawl-ingest writes to one central crawl db (CRAWL_DB, default
~/.arborist/crawl/web.db) instead of per-domain shards in the
peer-shared main dir: keeps locally-crawled content out of peer
sharing by default and a growing domain set under SQLite's 10-attach
cap (Makefile, docs/crawler.md).
- arborist query auto-includes the local crawl db (query() gains
extra_shards; CLI --include-shard / --no-crawl-db, default-on when
web.db exists). Fix latent --db single-file query AttributeError
(cli.py). Persist used / used_pointer_ids + retrieval_purity into
merkle_proof so read-only consumers can see which chunks fed the
answer (qa/query.py).
- arborist.read: read-only seam for dashboards / verifiers; on a
multi-source context root surface the real primary source instead of
the opaque corpus://multi-source sentinel (read.py). Backs the
arborist-viz Merkle Command Center (#000069).
- tests for extra_shards, the CLI crawl-db resolver, and the read seam.
Closes the FTS5 hyphen-tokenization asymmetry: `bi-polar is rare?`
retrieved only the Bi-Polar album/disambiguation cluster while the
medical-condition cluster (Bipolar disorder, Bipolar I/II disorder,
etc.) sat in the same shards untouched. `unicode61` splits hyphens
at index AND query time; `Bi-Polar Blues` indexes as [bi, polar, ...]
while `Bipolar disorder` indexes as [bipolar] — non-overlapping
token sets that never met.
Fix is query-layer only — no canonicalization_version bump, no
re-index, existing cache_keys stay valid:
- _hyphen_fold_variants(s): emit joined-no-hyphen variants for
every hyphenated run.
- _title_query_tokens(s): additively merges variants symmetrically
(queries AND titles when called on either).
- _filter_by_title_relevance: accept-path 5 — title stem-overlap
with hyphen-fold anchors passes the breadth gate. Rescues
`Bipolar disorder` (1-of-N qtoken match) without disrupting
non-hyphen queries (anchors empty → zero side effect).
- DEFAULT_QUERY_POLICY / DEFAULT_POLICY: hyphen_fold_v1: True
marker folds into governance_policy_hash; new records
cache-split cleanly from pre-fold records.
Live verification on /home/fox/.aborist/shards: same query now
retrieves `Bipolar disorder` (#5) and `Bipolar` disambiguation
(#7); model cites both, answer reads "Bi-polar disorder is not
rare; it affects approximately 2.8% of the U.S. population".
EVIDENCE-WARRANTED 2/2, properly grounded.
Tests: 4 new (3 unit, 1 integration with regression-pinned
Bipolar-disorder retrieval). Full suite 760 passed, 34 skipped.
Also: CLAUDE.md gains a close-when-complete hint for tickets — an
open ticket whose code already shipped is a stale map.
Regression fox surfaced 2026-05-02:
Q: "what technology are currently or soon available which may
enable one person to reconstruct and understand some or a
portion of another persons thoughts or ideas without
speaking or sign language."
→ sqlite3.OperationalError: Expression tree is too large
(maximum depth 1000)
Root cause: the v1 of _search_titles (commit 0052845) chained N
``CASE WHEN ... THEN 1 ELSE 0 END + ...`` expressions for the
title_score column. Each CASE WHEN is multiple tree nodes; +-
chained N times exceeded SQLite's default 1000-depth bound on
question texts with ~30+ content tokens.
Fix: simplified SQL — OR-chain WHERE + ORDER BY LENGTH(title) ASC
+ LIMIT bumped 4x to compensate for the lost smart sorting. The
caller's post-filter (word-boundary stem-aware token-set
intersect) does the actual title-relevance ranking; SQL just
needs to surface enough candidates for the post-filter to grade.
Also: cap the OR-chain at MAX_TITLE_LIKE_TOKENS=24 so pathological
200-token queries don't cascade SQL expression growth even
defensively. Beyond ~24 tokens the post-filter is doing all the
work; extra LIKEs just inflate candidate sets without signal.
5 new tests in tests/test_query.py covering the regression at
unit / integration / functional layers:
- test_unit_search_titles_handles_long_question_without_crash
50-token query through _search_titles directly. Pre-fix raised
sqlite3.OperationalError; post-fix returns row list.
- test_unit_search_titles_handles_zero_tokens
Defensive: empty token list → empty result, no SQL executed.
- test_unit_search_titles_caps_or_chain_at_max_tokens
200-token pathological query — bounded by MAX_TITLE_LIKE_TOKENS,
doesn't crash.
- test_integration_query_completes_on_long_question
End-to-end query() with StubClient + long question completes
without the SQLite error. Pre-fix raised before reaching the
LLM call.
- test_functional_long_question_returns_sources
The neurotech doc (richest body match) appears in top-K despite
the long-question retrieval path.
Test_query.py: 35 → 40 passing. Full suite (excluding parallel
test_concepts churn from concepts/query.py rewrite): no other
regressions.
Two follow-ups to the phrase-pattern retrieval fix (commit 1b8677d)
covering items 6 and 10-11 of fox's 2026-05-01 architectural review:
(1) Non-regression tests for the phrase route:
- test_phrase_route_skipped_when_question_shorter_than_min_n
pins the structural false-positive guard: the n=5/n=6 minimum
means a 4-token literal-geography query lacks enough tokens to
trigger the route at all.
- test_phrase_route_does_not_hijack_literal_geography_query
end-to-end: a 4-token "oceania east asia geography" query on
a synthetic 2-doc corpus surfaces only the geography-stub doc;
the orwell-stub doc (whose body has the diagnostic 5-gram) is
correctly NOT pulled in by the phrase route on a literal query.
(2) docs/ticket-000002-reference-frame-polarity-contract.md
Captures fox's Module L proposal verbatim as Appendix A and
extracts the implementation sketch into the standard ticket
body (problem statement, abstraction, CTI interpretation, three
pieces of code to write, test list, scope boundaries).
The phrase route closed the RETRIEVAL side of reference-frame
failure. Module L addresses the ANSWER side: today's substrate
answers Orwell queries as "the text does not directly state..."
when it should produce multi-frame answers distinguishing
Party propaganda from fictional-actual continuity. Forecast
cost ~3-4 hours; risk medium (prompt augmentation interaction
with claim_lattice prompt).
Module M = ticket #000001 (route provenance binding); not
duplicated. Module N (FP guards) partially landed via the
tests above; remaining tests folded into ticket #000002's
test list. Module H (relation warrant lite) lacks scope
detail; deferred without a ticket.
(3) docs/TICKETS.md updated: index gains #000002 row, Next ID
bumped to 000003.
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
Three layers, 14 new tests, full suite 611 passed (was 597):
UNIT — tests/test_query.py
test_query_returns_prompt_chars_breakdown
Asserts the result dict's prompt_chars carries exactly the five
expected keys and messages_total equals sum of message contents
the StubClient saw.
test_query_answer_chars_matches_answer_text
answer_chars == len(answer_text) — drift check.
test_query_cache_hit_also_returns_capacity_metrics
Cache-hit path populates prompt_chars + answer_chars (operators
inspecting cached records still want the breakdown).
test_query_evidence_chars_grows_with_topk
Sanity: more sources / larger budget → more evidence chars
(the metric tracks actual context build, not a stale constant).
INTEGRATION — tests/test_query.py (retrieval_keywords)
test_retrieval_keywords_does_not_alter_question_to_llm
Keywords don't appear in the LLM-facing question segment;
system prompt unchanged across runs. Pins the substrate
contract: keywords are FTS5/title-filter-only.
test_retrieval_keywords_changes_retrieved_sources
Different keyword sets surface different docs (the actual
user-visible behavior).
UNIT — tests/test_cli_render.py
test_render_shows_capacity_line_when_prompt_chars_present
Capacity line appears with messages_total + breakdown when
prompt_chars is in the result dict.
test_render_omits_capacity_line_on_legacy_results_without_prompt_chars
Backwards-compat: legacy results render cleanly without the
capacity line — no KeyError, no '0 chars' noise.
test_render_capacity_thousand_separators
61,550 not 61550 — operator legibility on daily renders.
UNIT/INTEGRATION — tests/test_bench_qa_sweep.py (NEW FILE)
Imports bench/qa_sweep.py via importlib.util so the module's
not in the Python path doesn't matter. Five tests:
- _summarize counts verdicts by mode
- deflections counted only on STRICT/HYBRID rows (not UNGROUNDED)
- rendered markdown has the headline summary + size buckets
- size buckets correctly stratify strict-rate by prompt_chars_total
- empty buckets are skipped (no '0 runs' noise)
FUNCTIONAL — live verification (no automated test, manual)
`make query Q="what is the capital of france?" BURN=1` confirmed
in commit f927298 to render the capacity one-liner under the
source list. Documented in that commit's body.
Also corrected the docstring on query()'s `retrieval_keywords` to
reflect that keywords don't enter cache_key DIRECTLY but do change
context_root + conversation_hash via source selection — so the same
question with different keywords lands under different cache_keys
(legitimately, since the LLM saw different contexts).
Each query/ask call now emits a 7-stage Merkle-DAG fingerprint stored
alongside the providence record. F from the toy-Hermes design pass.
Stages, in order:
question hash of question_hash (the 8-dim cache_key dim)
retrieval hash of sources summary (document_roots + roles +
scores + chunk_idx) — captures which docs ranked
context context_root (the source-Merkle for the assembly)
prompt conversation_hash
answer sha256(answer_text)
verify hash of verdict (audit_mode, verifier_method,
n_quotes, n_verified, claim_statuses)
final_label hash of (audit_mode, verifier_method, lookup_path)
run_dag_root = MerkleTree over those stage hashes (aborist conventions:
non-commutative HashCombine 0x03, leaf prefix 0x00, self-dup odd rule).
run_dag_blob = canonical JSON of {root, nodes} so an auditor can
recompute & verify (`verify_run_dag(blob)` returns True/False).
The DAG is NOT in cache_key. cache_key inputs determine the answer; the
answer determines the DAG — folding it back would create a cycle.
Instead it rides alongside as a per-record computation fingerprint.
Distinct from the linear `audit_events` chain (which tracks DB-wide
state changes); this is per-run computation provenance.
Schema: ALTER TABLE providence_cache ADD COLUMN run_dag_root TEXT;
ADD COLUMN run_dag_blob TEXT;
Idempotent migration in `_migrate_audit_mode`. Both rebuild templates
(VISUAL→UNGROUNDED dance, paraphrase verifier_method dance) updated to
include the new columns. Legacy records pre-2026-04-30 carry NULL.
Result dict gains `run_dag_root` so callers can verify without a DB
round-trip.
Tests:
- test_dag.py (9 tests): determinism, reactivity to each stage's input,
fixed stage order, round-trip verify, tamper-detection, JSON-string
acceptance.
- test_query.py: persistence on record + result, verify_run_dag round-
trip on the persisted blob.
473 tests pass (DAG +9, query +1, integration unchanged).
Two enhancements from the toy-Hermes design pass (2026-04-30):
A. Synthetic-elision-inside-quote diagnosis (sidecar only).
Distinct from interior_elision (model dropped a `(...)` aside source
carries) — synthetic_elision is the model writing literal `[...]`
between fragments of a `"..."` span, signaling self-elision while
claiming verbatim citation. The verifier still rejects (binary
discipline holds), but `aborist inspect` now reports
`diagnosis: synthetic_elision_inside_quote` with prefix/suffix
presence flags so an operator can judge whether the elided middle
was benign. Probe runs first in the classify-span chain (more
specific than trailing_artifact / interior_elision / paraphrase).
Catches the Brachiosaurus case: `"The film centers on the fictional
Isla Nublar [...] Universal Studios..."` — both halves are in source,
but the literal `[...]` isn't, so substring match correctly fails &
the sidecar tells the operator why.
C. Source-role classification + role-weighted context budget.
`_classify_source_role(title, qtokens_stem)` tags each top-K hit:
primary_answer_source 2.0× cap strong title-stem overlap
secondary_context_source 1.0× cap "list of", "characters",
"franchise", "history of"
noisy_background_source 0.5× cap "score", "music",
"video game", "merchandise"
sequel_background_source 0.5× cap "lost world", roman numerals
background_source 1.0× cap default
Order matters: noisy/sequel/secondary markers fire before the
primary check so peripheral pages with strong title overlap (e.g.
`Jurassic Park (film score)` shares 3 stems with the JP-film query)
don't claim a primary slot.
Cap loop now applies role weight on top of the baseline
`max_context_chars / top_k`. Total context still bounded by the
running `char_budget` — weights just shift how the budget gets
divided so primary pages get more text & noisy pages less, fixing
the case where a `(film score)` page consumed a primary slot.
`source_role` is persisted on `_Hit` and surfaces on
`merkle_proof.sources[*].source_role` in the providence record so
inspect & audits can see which slot each source occupied.
Tests:
- inspect: synthetic_elision_caught (Brachiosaurus regression),
synthetic_elision_does_not_fire_when_source_has_brackets (false-
positive guard).
- query: role classifier matrix (primary / secondary / noisy / sequel /
background) on JP-film-style titles, role persistence in sources list.
455 tests pass (sidecar +2, query +2).
Per fox: while iterating on retrieval/verifier knobs, a flag on the
query itself is more useful than a separate verb. One step reset:
change a knob, re-query, see fresh result.
Surface:
aborist query --burn ...
make query Q="..." BURN=1
make query-dry Q="..." BURN=1 (works on dry-run too)
Behavior in `aborist.qa.query.query()`:
- New `burn_existing: bool = False` parameter on query().
- After computing primary cache_key (per the active dedup mode) but
BEFORE the cache lookup, if burn_existing: DELETE the live row that
matches the primary cache_key & write one providence_burn audit
event with reason "query --burn (test-ergonomic mid-query bust)".
- The lookup then misses → fresh inference runs. Result reports
`burned_existing: 0|1` so the caller sees whether anything got
busted.
- The equivalence-class fallback cache_key is deliberately NOT
touched: prior alt-mode records stay as historic witnesses.
Tests (3): cache populates → cache hits → BURN=1 forces fresh
inference; --burn writes a providence_burn audit event; first-time
query with --burn is a clean no-op (burned_existing=0). 356 passed,
1 skipped.
Complementary to `aborist burn-kindergarten` (mass reset) — this is
the surgical version. Both are local-only by design; for cross-peer
invalidation use `make falsify` and let mesh sync broadcast.
Fox 2026-04-29: 'who is supermans girlfriend?' returned 7-of-8 unrelated
`Girlfriends`-titled articles (TV show, movies, songs); only 1 actual
Superman-related doc made it. Two coupled defects:
1. Title-overlap accept fired on ANY single-token match. A 2-token
query was admitting docs that shared only ONE qtoken. `Girlfriends`
passed because its title matched "girlfriend" even though no
"superman" anywhere in the doc.
2. `_body_density_passes` required at least HALF the qtokens — too
lenient for the 2-token case (1 of 2 = 50% = pass).
3. Possessive plural mismatch: question_hash strips apostrophes so
"superman's" → "supermans", but the corpus has bare "Superman".
`body.count("supermans")` returns 0 even on the canonical doc.
Three coordinated fixes in `aborist/qa/query.py`:
- New `_stem_token_for_match(t)` helper: strips a trailing `s` for
tokens > 4 chars (so `supermans` → `superman`, `girlfriends` →
`girlfriend`). Conservative: skips short tokens & double-s endings
to avoid `class` / `boss` / `pass` corruption.
- `_body_density_passes` now uses `_body_count_with_stem` (literal-
first, stem-fallback) AND a tightened breadth threshold:
≤ 2 tokens require ALL of them
3+ tokens require N - 1 (allow one weak signal token to miss)
- `_filter_by_title_relevance` mirrors the same breadth threshold for
title-overlap. Synonym fallback (any-match against synonym_expand)
is preserved ONLY for 1-token queries — otherwise a stray synonym
hit (e.g. AMD synonym matching an Intel-only title) over-recalls.
End-to-end on the actual corpus (3.4M articles, 8 shards):
before: 'supermans girlfriend' → 7 unrelated `Girlfriends` titles, 1 Superman doc
after: Lois_Lane in top-K (the canonical answer), other Superman-related
docs alongside, Girlfriends-only-titled docs filtered out
Tests:
- test_query_filter_requires_breadth_for_multi_token_queries — synthetic
3-doc corpus pinning the new behavior (Lois Lane keeps; girlfriends-tv
& superman-music both drop).
- test_query_filter_one_token_query_still_synonym_expands — pin the
1-token loose path is preserved (no over-tightening).
349 passed, 1 skipped.
Burned the two stale UNGROUNDED girlfriend-query cache records so a
fresh `make query` exercises the new filter end-to-end.
Different agents have different value functions on the same providence
records. Today aborist bakes one canonicalization policy into
governance_policy_hash and calls it universal. This adds two knobs that
let agents express their preference without breaking provenance.
- aborist/qa/keys.py: `canonical_question(q, mode=...)` and
`question_hash(q, mode=...)` accept "strict" (NFC + ws-collapse only;
every variant gets its own hash) or "equivalence_class" (default —
additionally lowercase + trailing-punct strip + article strip).
New constants: QUESTION_DEDUP_MODES, FIDELITY_MODES, defaults.
- aborist/qa/query.py + aborist/qa/runner.py: each call computes the
primary cache_key under policy["question_dedup"]. New `fidelity`
parameter:
"strict" only primary cache_key checked
"equivalence_class" primary first; if miss AND alternate mode
produces a different cache_key, try alternate
Cross-silo fallback works because _ckey_for_mode rewrites
policy["question_dedup"] to the alternate mode before computing
governance_policy_hash — so the fallback ckey matches what an agent
under that mode would have written. Lookups can find each other's
records when fidelity permits. Result dict gains `lookup_path` ∈
{"strict", "equivalence_class", "strict_fallback",
"equivalence_class_fallback", "miss"}.
- aborist/cli.py: `--question-dedup` and `--fidelity` flags on the
`query` subcommand. Human render annotates cache_hit lines that
came from a fallback ckey ("cached via equivalence_class_fallback").
- tests/test_query.py: three new regressions
* strict-policy + strict-fidelity: variants get distinct cache_keys
* cross-silo fallback: strict-policy reads eq-class-policy records
via fidelity=equivalence_class
* strict-fidelity refuses fallback (audit-grade): cache miss even
when the alternate silo has a hit
- CLAUDE.md: dedup-mode and fidelity convention bullets updated.
This is V1 + V2 of the substrate move sketched in conversation: write
policy determines which silo a record lives in; read fidelity determines
how loosely an agent walks across silos. Records stay exactly-keyed
(provenance hard); routing is per-agent (preference soft). Same
verifier_no_diagnostics discipline holds — no soft signal enters the
hard chain.
442 tests pass.
Fox caught on 2026-04-29: 'who is batman?' and 'who is the batman?' produced
different cache_keys despite question_hash collapsing both variants. The 8-dim
cache_key has TWO seams that touch question text — question_hash AND
conversation_hash (which hashes the messages list with the literal question
in the user turn). Only the former canonicalized; the latter took bytes as
written, so each variant got its own chash and missed cache.
Fix: decouple the two forms.
- aborist/qa/keys.py: extract canonical_question(text) → str; question_hash
composes _sha256(canonical_question(t)) so the equivalence class is
defined in one place.
- aborist/qa/query.py + aborist/qa/runner.py: build TWO message lists per
call. `messages` carries the verbatim question (Hermes sees the user's
natural phrasing — no grammar drift). `canonical_messages` substitutes
the canonical form, and conversation_hash hashes that. The LLM still
gets 'Who Is THE Batman?'; the cache_key collapses to 'who is batman'.
- tests/test_query.py: regression — three variants (with/without `?`,
with/without `the`) hit the same cache_key. First call populates,
later variants cache_hit. Captures stub messages to confirm the LLM
saw the verbatim question, not the canonical form.
437 tests pass.
Fox 2026-04-29: querying "who is batman" returned only ONE source
(List_of_Batman_comics — an 80 KB+ bibliography) despite top_k=8 &
the actual bio article being in the corpus. Greedy fill: hit #1
consumed the entire 60 KB budget, every subsequent doc dropped with
char_budget <= 0.
Fix in aborist/qa/query.py:
per_source_cap = max(1, max_context_chars // max(1, top_k))
for h in hits[:top_k]:
text = _load_doc_text(...)
if len(text) > per_source_cap:
text = text[:per_source_cap] # NEW: per-source cap first
if len(text) > char_budget:
text = text[:char_budget]
...
Each top_k hit gets at most max_context_chars/top_k chars (default
60K/8 = 7.5K each — plenty for a chunk or two of prose). Total
context ≤ max_context_chars by construction. top_k=1 preserves the
legacy behavior (single source can use the full budget).
End-to-end effect on Batman: the bio (Wikipedia/Batman article) lands
in context alongside List_of_Batman_comics; the model can paraphrase-
verify against the actual character introduction text instead of
fabricating from training.
Tests: 2 regressions in tests/test_query.py — multi-source delivery
when hit #1 is huge, and top_k=1 single-source still allowed full
budget. 337 passed, 1 skipped.
Burned the two stale Batman cache records (chain extended) so a
fresh `make query Q="who is batman?"` exercises the new path.
Adds aborist/qa/verify.py — three-strategy verifier (explicit quotes,
bullet/sentence spans, multi-word proper nouns) that lexically checks
every claim against retrieved context under norm-v1 + lowercase. The
strategy that fires is recorded as verifier_method for diagnostics.
Entity strategy gates classification via an entity_policy
(strict/hybrid/proximity/drop) so a single proper-noun match no longer
overclaims STRICT — proximity (default) requires a cluster of 3+
verified entities within 300 chars.
Wires into ask() and query(). System prompts now require verbatim
quoted spans for every factual claim, restated via a user-turn
grounding_reminder one message before sources arrive (recent
user-turn instructions outweigh decayed system-turn rules under long
context in Hermes).
providence_cache gains 5 columns (audit_mode, n_quotes, n_verified,
unverified_quotes, verifier_method) with CHECK constraints. A
connect-time _migrate_audit_mode() ALTERs legacy DBs idempotently.
Cache hits return the persisted audit_mode rather than asserting
STRICT unconditionally.
CLI:
emergent list VISUAL/HYBRID records; --aggregate ranks
unverified quotes by frequency (corpus-growth signal).
reclassify re-run the verifier against live providence records;
cold-source records skipped; --dry-run reports
transitions without writing; each change writes one
'providence_reclassify' audit event.
aborist/qa/query.py exposes query() — the user-facing RAG flow:
1. FTS5 search across all shards (chunks_fts can't be UNION'd as a
view, so each shard's index is queried independently and merged
by score).
2. Top-K distinct documents are selected within a max-context-chars
budget (default 60 KB so a 768-token response fits Hermes-3's
82 K context window comfortably).
3. context_root = Merkle root over the sorted source document_roots.
That's the v9.8 'source' dimension for multi-source answers —
a verifier can recompute it from the listed source roots.
4. 8-dim cache_key over (context_root, question_hash, model_profile,
conversation, governance_policy, schema, canonicalization,
chunking). Hit returns STRICT immediately; miss calls Hermes and
persists.
CLI: aborist [--shards-dir DIR] query "<question>"
Default qa_db is <shards-dir>/qa.db (or ~/.aborist/qa.db). Uses the
same OpenAICompatibleClient/StubClient as `ask`. --dry-run skips the
LLM and returns context-only.
Search escape fix: the prior FTS5 escape ANDed every token including
stopwords + punctuation, so "What is anarcho-capitalism?" required
the doc to literally contain "what" + "is" + "anarcho-capitalism?" —
zero hits. New tokenizer drops stopwords + punctuation and ORs the
remaining content tokens; BM25 ranks the multi-token matches highest.
Live demo against the 122k-doc 4-shard cluster:
Q "What is anarcho-capitalism?" 6.1 s wall miss / 0.45 s cache hit
Q "Who was George Washington?" 10.3 s wall miss
Both answers cite the source URIs Hermes was given.
57 tests passing (4 new query tests covering search → context →
cache → audit chain).
