engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

cli: arborist corpus-query — local shards through the unified Corpus pipeline

Browse source 
Parallel to `arborist cloud query`: same corpus_query.run_query
orchestrator, only the Corpus adapter differs (SqliteShardCorpus
instead of SidecarBucketCorpus). Proves the protocol works against
both backends with identical pipeline code:

  arborist query "Q"          legacy 2000-line query() (untouched)
  arborist cloud query "Q"    corpus_query.run_query + SidecarBucketCorpus
  arborist corpus-query "Q"   corpus_query.run_query + SqliteShardCorpus  ← NEW

Same render layer (_render_cloud_query_human), same audit_mode +
sources + capacity + timings shape across both new paths.

Caveat: single-DB mode only for now. SqliteShardCorpus.fts_body uses
`chunks_fts JOIN chunks ON rowid`, which doesn't bridge per-shard
rowid namespaces under connect_query() ATTACH-and-UNION. Multi-shard
search needs a per-shard query + merge — the next protocol method
(fts_body_per_shard) to add. Until then, --db <path> or auto-picks
the first non-system .db under --shards-dir.

Live demo (homer + virt-back queries):
    make corpus-query Q="who developed virt-back?" LLM=qwen
    → EVIDENCE-WARRANTED · via claim_lattice  1/1  2.29s
       (search 0.01s · llm 2.26s · verify 0.01s · total 2.29s)
       Same E1 citation + render as `make cloud-query` on the same
       question; only "0 HTTP requests" footer betrays local vs cloud.

Makefile: `make corpus-query Q="..." [LLM=qwen]`.
This commit is contained in:
russell@unturf.com 2026-05-31 07:54:17 -04:00
parent c3529ad571
commit 96417cd958
No known key found for this signature in database
2 changed files with 141 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

10
Makefile
View file

@ -1629,6 +1629,16 @@ cloud-query: bootstrap ## bucket-direct end-to-end via manifest [Q="..." JSON=1
$(if $(LLM_MODEL),--model $(LLM_MODEL),) \
$(if $(JSON),--json,)
corpus-query: bootstrap ## LOCAL shards via the same Corpus pipeline cloud-query uses [Q="..." JSON=1 LLM=qwen|hermes TOP_K=N MAX_CONTEXT=N]
@test -n "$(Q)" || { echo 'usage: make corpus-query Q="your question" [JSON=1] [LLM=qwen|hermes] [TOP_K=4] [MAX_CONTEXT=24000]'; exit 2; }
$(if $(LLM_ENDPOINT),@echo "# llm: $(LLM_ENDPOINT) / $(LLM_MODEL)" >&2,)
@$(ARBORIST) --shards-dir $(SHARDS_DIR) corpus-query '$(Q)' \
--top-k $(or $(TOP_K),4) \
--max-context-chars $(or $(MAX_CONTEXT),24000) \
$(if $(LLM_ENDPOINT),--endpoint $(LLM_ENDPOINT),) \
$(if $(LLM_MODEL),--model $(LLM_MODEL),) \
$(if $(JSON),--json,)
# ---------------------------------------------------------------------------
# Head-to-head: local query (BURN=1) vs cloud-query, same question, same data.
# ---------------------------------------------------------------------------