three relevance signals: title + TF-IDF cores + body density
The "permacomputer" query fell into title-relevance's empty-fallback
because "permacomputer" appears nowhere in any title. Fallback was
top-1 by raw BM25, getting one tangential source. Same problem for
any user-coined term: title-LIKE alone can't find it.
_filter_by_title_relevance now grows three accept paths:
1. Title-token overlap (existing).
2. TF-IDF core keyword overlap. A doc's distinctive low-frequency
terms get distilled into its tfidf-keywords-v1 core. A query for a
rare term ("permacomputer", "unturf") hits docs whose core lists
it as a keyword even when no title does.
3. Body density. Docs that mention a query token >= 3 times pass
regardless of title or core. Cheap proxy for "actually about it."
Empty-all-three fallback widens from top-1 to top-N (default 5) — the
LLM gets enough context to honestly say "I don't know" rather than
fabricating from a single tangential source.
Implementation:
- _docs_with_core_keyword_match: SQL join chunks -> documents -> derivations
finds source docs whose TF-IDF core content matches a token. Folded
into the per-shard search loop alongside FTS5 + title-LIKE.
- _body_density_passes: counts case-insensitive occurrences in
concatenated chunk content. Threshold 3.
- _search_corpus stashes core_match_roots + root_to_shard on the
returned hits list as side-channel info for _rerank.
Makefile: distill-shards-tfidf-parallel for the per-shard TF-IDF run.
Demo on the live cluster (after TF-IDF distill of 95k cur surface docs):
Q "tell me about permacomputer ?"
sources: MOAD Cheat Sheet + Unturf AI Infrastructure Expansion
answer: synthesized from both, bounds coverage honestly
Q "what is unturf about ?"
sources: Unturf Trademark + Unturf AI Infrastructure Expansion
answer: "grassroots, community-driven initiative ... ethical AI
hardware, software, and networks"
Both queries now find user-coined terms via TF-IDF cores even though
the terms never appear in any title.
79 tests passing.
This commit is contained in:
parent
5413b83d45
commit
0192255f7d
2 changed files with 172 additions and 17 deletions
8
Makefile
8
Makefile
|
|
@ -151,6 +151,14 @@ distill-shards-parallel: bootstrap ## one distill process per shard (parallel)
|
|||
$(ABORIST) --db $$shard distill --process first-sentence-v1 --kind surface & \
|
||||
done; wait
|
||||
|
||||
# TF-IDF cores serve as enriched titles for retrieval — distinctive
|
||||
# low-frequency terms that surface in body text get promoted into
|
||||
# something queryable without a real title match.
|
||||
distill-shards-tfidf-parallel: bootstrap ## TF-IDF cores per shard, in parallel
|
||||
@for shard in $(SHARDS_DIR)/*.db; do \
|
||||
$(ABORIST) --db $$shard distill --process tfidf-keywords-v1 --kind surface & \
|
||||
done; wait
|
||||
|
||||
ingest: ingest-cur ## default ingest = cur (use ingest-old or *-parallel for full)
|
||||
|
||||
# Grok account-export ETL.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue