engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

sync title-LIKE stopwords + Grok shard target

Browse source 
Two issues surfaced by "tell me about permacomputer ?":

1. Stopword leak. _FTS5_STOPWORDS got "tell" added earlier but the
   parallel _TITLE_STOPWORDS in qa/query.py wasn't synced. Title-LIKE
   search picked up "tell" and returned every Tell_* article in the
   corpus (Tell_(poker), Tell-Tale_Heart, Tell_Township, Tell_City,
   etc.) all uniformly scored at 70.0. Synced the lists — title
   tokenizer now drops the same question-shaping verbs as FTS5.

2. Grok docs lived only in --db (default ~/.aborist/aborist.db),
   not in --shards-dir. `make query` reads the shard cluster, so
   Grok was invisible to it. Added two targets:
     ingest-grok-attached         conversations -> $(SHARDS_DIR)/grok.db
     ingest-grok-media-attached   media prompts -> $(SHARDS_DIR)/grok.db
   Both use --resume so re-runs are cheap.

Demo: "tell me about permacomputer ?" now answers from the
"MOAD Cheat Sheet: Common Software Defects" Grok conversation —
fox's own prior chat surfaced as corpus.

Note: 000.db already had Grok docs from an earlier ingest variant.
The cross-shard UNION currently sees them twice (376 rows, 188
unique document_roots). Retrieval is unaffected because
_search_corpus dedupes by root, but stats counts are inflated.
Cleanup is destructive (drop one copy) — fox decides which.

79 tests passing.
This commit is contained in:
russell@unturf.com 2026-04-27 13:33:16 -04:00
parent 787c243179
commit 5413b83d45
No known key found for this signature in database
2 changed files with 33 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

24
Makefile
View file

@ -26,6 +26,7 @@ VERIFY_N ?= 10
SEARCH_Q ?= computer
.PHONY: all bootstrap fetch fetch-cur fetch-old ingest ingest-cur ingest-old \
ingest-grok ingest-grok-media \
verify search stats test clean clean-db clean-data help
all: bootstrap fetch-cur ingest-cur verify stats ## bootstrap → fetch cur → ingest cur → verify → stats
@ -152,6 +153,29 @@ distill-shards-parallel: bootstrap ## one distill process per shard (parallel)
ingest: ingest-cur ## default ingest = cur (use ingest-old or *-parallel for full)
# Grok account-export ETL.
# Point GROK_EXPORT at the directory xAI delivered (the one containing
# `ttl/30d/export_data/<user-id>/prod-grok-backend.json`). The source class
# auto-walks down to find the JSON.
GROK_EXPORT ?= $(HOME)/Downloads/ab8ef1f0-0d08-4f87-89c2-d4509e18115b
ingest-grok: bootstrap ## ingest Grok conversations into $(DB) (single-DB mode)
$(ABORIST) --db $(DB) ingest --source grok_export --path $(GROK_EXPORT)
ingest-grok-media: bootstrap ## ingest Grok media-generation prompts into $(DB)
$(ABORIST) --db $(DB) ingest --source grok_media --path $(GROK_EXPORT)
# Grok lives in its own shard inside the attach-forever cluster so cross-
# shard queries (`make query`) see it alongside the Wikipedia 2003 corpus.
# Single shard (rank 0/1) — Grok conversations are private and small.
GROK_SHARD := $(SHARDS_DIR)/grok.db
ingest-grok-attached: bootstrap ## ingest Grok conversations into $(GROK_SHARD)
@mkdir -p $(SHARDS_DIR)
$(ABORIST) --db $(GROK_SHARD) ingest --source grok_export --path $(GROK_EXPORT) --resume
ingest-grok-media-attached: bootstrap ## ingest Grok media prompts into $(GROK_SHARD)
@mkdir -p $(SHARDS_DIR)
$(ABORIST) --db $(GROK_SHARD) ingest --source grok_media --path $(GROK_EXPORT) --resume
verify: bootstrap ## round-trip Merkle proofs for VERIFY_N random documents
$(ABORIST) --db $(DB) verify -n $(VERIFY_N)