arborist/aborist
russell@unturf.com faed58dd0a
qa(retrieval): fix _search_titles SQLite tree-depth overflow on long queries
Regression fox surfaced 2026-05-02:
  Q: "what technology are currently or soon available which may
      enable one person to reconstruct and understand some or a
      portion of another persons thoughts or ideas without
      speaking or sign language."
  → sqlite3.OperationalError: Expression tree is too large
    (maximum depth 1000)

Root cause: the v1 of _search_titles (commit 0052845) chained N
``CASE WHEN ... THEN 1 ELSE 0 END + ...`` expressions for the
title_score column. Each CASE WHEN is multiple tree nodes; +-
chained N times exceeded SQLite's default 1000-depth bound on
question texts with ~30+ content tokens.

Fix: simplified SQL — OR-chain WHERE + ORDER BY LENGTH(title) ASC
+ LIMIT bumped 4x to compensate for the lost smart sorting. The
caller's post-filter (word-boundary stem-aware token-set
intersect) does the actual title-relevance ranking; SQL just
needs to surface enough candidates for the post-filter to grade.

Also: cap the OR-chain at MAX_TITLE_LIKE_TOKENS=24 so pathological
200-token queries don't cascade SQL expression growth even
defensively. Beyond ~24 tokens the post-filter is doing all the
work; extra LIKEs just inflate candidate sets without signal.

5 new tests in tests/test_query.py covering the regression at
unit / integration / functional layers:

- test_unit_search_titles_handles_long_question_without_crash
  50-token query through _search_titles directly. Pre-fix raised
  sqlite3.OperationalError; post-fix returns row list.
- test_unit_search_titles_handles_zero_tokens
  Defensive: empty token list → empty result, no SQL executed.
- test_unit_search_titles_caps_or_chain_at_max_tokens
  200-token pathological query — bounded by MAX_TITLE_LIKE_TOKENS,
  doesn't crash.
- test_integration_query_completes_on_long_question
  End-to-end query() with StubClient + long question completes
  without the SQLite error. Pre-fix raised before reaching the
  LLM call.
- test_functional_long_question_returns_sources
  The neurotech doc (richest body match) appears in top-K despite
  the long-question retrieval path.

Test_query.py: 35 → 40 passing. Full suite (excluding parallel
test_concepts churn from concepts/query.py rewrite): no other
regressions.
2026-05-01 22:19:20 -04:00
..
concepts qa(concepts): corpus-derived concept_relations table replaces frozensets 2026-05-01 21:14:53 -04:00
distill storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
mesh mesh: per-peer audit chain-of-claims tracking on the wire 2026-04-28 19:41:25 -04:00
qa qa(retrieval): fix _search_titles SQLite tree-depth overflow on long queries 2026-05-01 22:19:20 -04:00
search qa: strip 'all/there/know/everything' from retrieval queries; bench gains deflection rate 2026-04-30 20:15:25 -04:00
sources qa: self-reference thought chains — STRICT-as-fact substrate 2026-05-01 10:16:47 -04:00
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
cli.py qa(verify): Rule 8 — title-relevance promoted to hard verifier check 2026-05-01 20:59:14 -04:00
compress.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
document.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
evict.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
ingest.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
progress.py progress reporter + structured benchmark 2026-04-27 11:37:20 -04:00
snapshot.py add corpus-level snapshots: single-hash identity for the forest 2026-04-27 21:29:10 -04:00
source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
store.py qa(concepts): corpus-derived concept_relations table replaces frozensets 2026-05-01 21:14:53 -04:00
wikitext.py verify: strip wikitext from context before substring matching 2026-04-28 15:48:07 -04:00