three relevance signals: title + TF-IDF cores + body density

The "permacomputer" query fell into title-relevance's empty-fallback
because "permacomputer" appears nowhere in any title. Fallback was
top-1 by raw BM25, getting one tangential source. Same problem for
any user-coined term: title-LIKE alone can't find it.

_filter_by_title_relevance now grows three accept paths:

1. Title-token overlap (existing).
2. TF-IDF core keyword overlap. A doc's distinctive low-frequency
   terms get distilled into its tfidf-keywords-v1 core. A query for a
   rare term ("permacomputer", "unturf") hits docs whose core lists
   it as a keyword even when no title does.
3. Body density. Docs that mention a query token >= 3 times pass
   regardless of title or core. Cheap proxy for "actually about it."

Empty-all-three fallback widens from top-1 to top-N (default 5) — the
LLM gets enough context to honestly say "I don't know" rather than
fabricating from a single tangential source.

Implementation:
- _docs_with_core_keyword_match: SQL join chunks -> documents -> derivations
  finds source docs whose TF-IDF core content matches a token. Folded
  into the per-shard search loop alongside FTS5 + title-LIKE.
- _body_density_passes: counts case-insensitive occurrences in
  concatenated chunk content. Threshold 3.
- _search_corpus stashes core_match_roots + root_to_shard on the
  returned hits list as side-channel info for _rerank.

Makefile: distill-shards-tfidf-parallel for the per-shard TF-IDF run.

Demo on the live cluster (after TF-IDF distill of 95k cur surface docs):

  Q "tell me about permacomputer ?"
    sources: MOAD Cheat Sheet + Unturf AI Infrastructure Expansion
    answer: synthesized from both, bounds coverage honestly

  Q "what is unturf about ?"
    sources: Unturf Trademark + Unturf AI Infrastructure Expansion
    answer: "grassroots, community-driven initiative ... ethical AI
             hardware, software, and networks"

Both queries now find user-coined terms via TF-IDF cores even though
the terms never appear in any title.

79 tests passing.
This commit is contained in:
russell@unturf.com 2026-04-27 13:39:29 -04:00
parent 5413b83d45
commit 0192255f7d
No known key found for this signature in database
2 changed files with 172 additions and 17 deletions

View file

@ -151,6 +151,14 @@ distill-shards-parallel: bootstrap ## one distill process per shard (parallel)
$(ABORIST) --db $$shard distill --process first-sentence-v1 --kind surface & \
done; wait
# TF-IDF cores serve as enriched titles for retrieval — distinctive
# low-frequency terms that surface in body text get promoted into
# something queryable without a real title match.
distill-shards-tfidf-parallel: bootstrap ## TF-IDF cores per shard, in parallel
@for shard in $(SHARDS_DIR)/*.db; do \
$(ABORIST) --db $$shard distill --process tfidf-keywords-v1 --kind surface & \
done; wait
ingest: ingest-cur ## default ingest = cur (use ingest-old or *-parallel for full)
# Grok account-export ETL.