arborist/aborist
russell@unturf.com 2d6a86b991
qa: --retrieval-keywords flag for explicit retrieval augmentation
Empirically observed 2026-05-01: long discursive questions like
'what technology is currently or soon available which may enable
one person to reconstruct another person's thoughts...' under-
retrieve because their content tokens get diluted by template
phrasing. AND-mode FTS5 returns zero hits; OR-fallback ranks
unrelated articles by raw BM25.

Fox's discovery: appending domain keywords ('transcranial knowledge
acquisition') to the question lifts the verdict from HYBRID 6/10
to STRICT 1/1 by narrowing OR-mode retrieval to the topical
article (Neurotechnology). The keywords act as an operator hint
about what the question is really asking.

This commit exposes that pattern as a first-class flag without
polluting the question text:

  --retrieval-keywords TEXT   on `aborist query`
  K="..."                     on `make query`

Plumbing: the keywords are concatenated with the question for
FTS5 search and title-filter token construction only. The LLM
still sees the original question; the verifier still checks against
the original question; cache_key still computes from the original
question. Keywords are session-only — successive calls with
different keywords on the same question can cache-hit each other.
Pair with BURN=1 for fresh inference per call.

Live verification:

  make query Q="...thoughts...without speaking or sign language."
       K="transcranial knowledge acquisition" BURN=1
  -> STRICT 2/2 via claim_lattice, 37.5s, Neurotechnology article
     cited for fMRI and DBS claims.

Without K: HYBRID 6/10 with Videoconferencing/Telepathy as top
sources — model hallucinated structure across irrelevant chunks.
2026-05-01 10:30:13 -04:00
..
distill storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
mesh mesh: per-peer audit chain-of-claims tracking on the wire 2026-04-28 19:41:25 -04:00
qa qa: --retrieval-keywords flag for explicit retrieval augmentation 2026-05-01 10:30:13 -04:00
search qa: strip 'all/there/know/everything' from retrieval queries; bench gains deflection rate 2026-04-30 20:15:25 -04:00
sources qa: self-reference thought chains — STRICT-as-fact substrate 2026-05-01 10:16:47 -04:00
__init__.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
cli.py qa: --retrieval-keywords flag for explicit retrieval augmentation 2026-05-01 10:30:13 -04:00
compress.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
document.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
evict.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
ingest.py storage cheats + TF-IDF retrieval fix 2026-04-27 17:24:51 -04:00
merkle.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
progress.py progress reporter + structured benchmark 2026-04-27 11:37:20 -04:00
snapshot.py add corpus-level snapshots: single-hash identity for the forest 2026-04-27 21:29:10 -04:00
source.py phase 0 explore: aborist core + sources + distill + evict 2026-04-27 07:53:18 -04:00
store.py qa: G0 claim-lattice-pointer answer mode — CTI quote-by-pointer 2026-04-29 21:19:19 -04:00
wikitext.py verify: strip wikitext from context before substring matching 2026-04-28 15:48:07 -04:00