engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

arborist query: parallel per-shard FTS5 + make-query honors LLM=qwen

Browse source 
Two perf fixes after the snapshot_root() removal in 9f4152e
unblocked the real bottlenecks on multi-token natural-language
queries against the 5-shard genesis corpus.

1. MultiShardSqliteCorpus.fts_body + _fanout now parallel
============================================================
Previously sequential — 5 shards × ~6s per shard on a query like
"when did aliens film come out?" = 25s wall time per fts_body call.
ThreadPoolExecutor over the per-shard sqlite3 connections drops
that to roughly max(per_shard) instead of sum.

Connections are now opened with `check_same_thread=False`
(`arborist.store.connect()` gains a kwarg, default True preserves
the existing behavior; MultiShardSqliteCorpus passes False so
read-only fan-out works across threads). Read-only FTS5 queries
serialize under SQLite's internal locks; the per-Connection thread
check is what was blocking cross-thread use.

Measured (in-process, 5 shards, "when did aliens film come out?"):
  before:  25.5 s sequential fan-out
  after:    7.6 s parallel fan-out (warm)
            12.3 s parallel fan-out (cold)

End-to-end `arborist query` on the same fixture:
  before (post-snapshot_root fix):  65-110 s
  after:                            14-18 s (cache hit)
                                    16-18 s (cache miss + persist)
  legacy reference (--legacy):       10-13 s

Default path is now within 30-40% of legacy on this workload (down
from the 5-6× slowdown fox saw before). Headroom remains because
providence_query still re-runs retrieval inside run_query on miss
(redundant ~7 s); a future refactor can thread precomputed_hits
through.

2. Makefile: `make query LLM=qwen` actually routes to Qwen
============================================================
The `ifeq ($(LLM),qwen)` block setting LLM_ENDPOINT + LLM_MODEL
lived AT LINE 1633, AFTER the `query:` target at line 191.
Make evaluates top-down, so by the time `query:` ran the LLM
variables weren't set — and the recipe didn't reference them
anyway. Result: `make query LLM=qwen Q="..."` silently used
Hermes, the CLI default.

Fix: moved the ifeq block to line 192 (just above the `query:`
target so both `query:` AND the later `cloud-query:` see the
same definitions), and added
`$(if $(LLM_ENDPOINT),--endpoint $(LLM_ENDPOINT) --model $(LLM_MODEL),)`
to the query recipe. cloud-query unchanged.

Smoke:
  $ LLM=qwen Q="..." make query
  # llm:    https://qwen.ai.unturf.com/v1 / Qwen3.6-27B-UD-Q4_K_XL.gguf
  .venv/bin/arborist ... query ... --endpoint https://qwen.ai.unturf.com/v1
                                   --model Qwen3.6-27B-UD-Q4_K_XL.gguf ...

176 tests pass.
This commit is contained in:
russell@unturf.com 2026-06-01 12:07:48 -04:00
parent 9f4152e136
commit ec371d189a
No known key found for this signature in database
3 changed files with 95 additions and 41 deletions
Download patch file Download diff file Expand all files Collapse all files

19
Makefile
View file

@ -188,11 +188,24 @@ ANSWER_MODE ?= claim_lattice
# Override session-wide: LAYOUT_DEFAULT=bookend make query Q="..."
LAYOUT_DEFAULT ?= tail
LAYOUT ?= $(LAYOUT_DEFAULT)
query: bootstrap ## ask the corpus a question [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N K="extra retrieval keywords" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote LAYOUT=tail|bookend|per_chunk BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1]; JSON by default
# LLM endpoint toggle (shared with cloud-query — same ifeq lives down
# below; this one runs first because `query:` is defined ABOVE
# cloud-query and Make evaluates top-down). `make query LLM=qwen
# Q="..."` swaps to Qwen3.6-27B on uncloseai. LLM unset / LLM=hermes
# leaves --endpoint/--model unset so the CLI falls back to its
# built-in default (Hermes-3-8B on ai.unturf.com).
ifeq ($(LLM),qwen)
LLM_ENDPOINT ?= https://qwen.ai.unturf.com/v1
LLM_MODEL ?= Qwen3.6-27B-UD-Q4_K_XL.gguf
endif
query: bootstrap ## ask the corpus a question [JSON=1 BURN=1 LLM=qwen|hermes REPAIR=1 REPROMPTS=N K="extra retrieval keywords" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote LAYOUT=tail|bookend|per_chunk BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1]; JSON by default
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \
echo "usage: make query Q=\"your question\" [JSON=1 BURN=1 REPAIR=1 REPROMPTS=N K=\"extra retrieval keywords\" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote LAYOUT=tail|bookend|per_chunk BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1]"; exit 2; \
echo "usage: make query Q=\"your question\" [JSON=1 BURN=1 LLM=qwen|hermes REPAIR=1 REPROMPTS=N K=\"extra retrieval keywords\" ANSWER_MODE=claim_lattice|claim_lattice_pointer|quote LAYOUT=tail|bookend|per_chunk BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 WITNESS=1 XLANG=1 XLANG_MT=1]"; exit 2; \
fi
$(ARBORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(JSON),--json,) $(if $(BURN),--burn,) $(if $(REPAIR),--repair,) $(if $(REPROMPTS),--repair-reprompts $(REPROMPTS),) $(if $(ANSWER_MODE),--answer-mode $(ANSWER_MODE),) --user-payload-layout $(LAYOUT) $(if $(K),--retrieval-keywords "$(K)",) $(if $(BROAD),--apply-quantifier-caps,) $(if $(REJECT_BROAD),--reject-broad,) $(if $(ALLOW_BROAD),--allow-broad,) $(if $(WITNESS),--witness,) $(if $(XLANG),--crosslang-guard,) $(if $(XLANG_MT),--crosslang-translate,) "$(Q)"
$(if $(LLM_ENDPOINT),@echo "# llm: $(LLM_ENDPOINT) / $(LLM_MODEL)" >&2,)
$(ARBORIST) --shards-dir $(SHARDS_DIR) query --top-k $(QUERY_TOP_K) $(if $(LLM_ENDPOINT),--endpoint $(LLM_ENDPOINT) --model $(LLM_MODEL),) $(if $(JSON),--json,) $(if $(BURN),--burn,) $(if $(REPAIR),--repair,) $(if $(REPROMPTS),--repair-reprompts $(REPROMPTS),) $(if $(ANSWER_MODE),--answer-mode $(ANSWER_MODE),) --user-payload-layout $(LAYOUT) $(if $(K),--retrieval-keywords "$(K)",) $(if $(BROAD),--apply-quantifier-caps,) $(if $(REJECT_BROAD),--reject-broad,) $(if $(ALLOW_BROAD),--allow-broad,) $(if $(WITNESS),--witness,) $(if $(XLANG),--crosslang-guard,) $(if $(XLANG_MT),--crosslang-translate,) "$(Q)"
query-dry: bootstrap ## like 'make query' but skip the LLM call (dry-run) [JSON=1 BURN=1 ANSWER_MODE=... BROAD=1 REJECT_BROAD=1 ALLOW_BROAD=1 XLANG=1 XLANG_MT=1]
@if [ -z "$$Q" ] && [ -z "$(Q)" ]; then \