engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

cloud ask: unified multi-shard via bucket manifest + read-ahead tuning

Browse source 
`BUCKET_URL` (one env var) → client GETs `clones/manifest.json` →
opens HttpRangeVFS per listed shard → FTS5 across all shards in
parallel (ThreadPoolExecutor; per-thread apsw.Connection) → merge by
BM25 score → pull chunks from the owning shard → LLM + verify.
No per-query --shard-url, no path proliferation.

Two manifests published on s3://arborist/clones/:
  manifest.json       — default: virtback only (2.5MB, ~5s/query)
  manifest-full.json  — opt-in: all 5 shards (35GB, prohibitive over
                        WAN due to FTS5 b-tree walk pattern; needs
                        smaller shards or co-located query proxy)

HttpRangeVFS read-ahead tuned from per-page (4KB) to 64KB block-aligned
cache. Each cache miss fetches one 64KB block; subsequent reads within
the block are local-fast. Lower miss count, similar bytes-on-wire
(64KB amortizes well over typical 4-16 page b-tree clusters; larger
read-ahead like 4MB over-fetches on random FTS5 reads).

Sample run (default manifest):
    make cloud-ask Q="who developed virt-back?"
    → EVIDENCE-WARRANTED · via claim_lattice  1/1  4.71s  (bucket-direct)
       21 HTTP requests · 1344 KB

ACL: genesis full-bench shards flipped to public-read (CC-BY-SA
Wikipedia content). Reachable now if you want to play with the slow
multi-shard path; not in the default manifest because chat latency
matters more than coverage breadth.
This commit is contained in:
russell@unturf.com 2026-05-30 13:33:58 -04:00
parent 87f7d920e7
commit 331e748bcb
No known key found for this signature in database
4 changed files with 272 additions and 27 deletions
Download patch file Download diff file Expand all files Collapse all files

14
Makefile
View file

@ -1552,6 +1552,12 @@ wallet-ask: bootstrap ## query the wallet server with your own question [Q="..."
SHARD_URL ?= https://nyc3.digitaloceanspaces.com/arborist/clones/virtback/000.db
BLOB_BASE ?=
# BUCKET_URL is the root the multi-shard cloud-ask points at. Client GETs
# $BUCKET_URL/clones/manifest.json and queries every listed shard. One
# config var → entire corpus. Default = our DO Spaces public bucket with
# genesis Wikipedia + russell.ballestrini.net.
BUCKET_URL ?= https://nyc3.digitaloceanspaces.com/arborist
bootstrap-bucket: bootstrap ## install apsw (bucket-direct extra)
$(PIP) install 'apsw>=3.45'
@ -1572,10 +1578,10 @@ cloud-fetch-chunk: bootstrap ## fetch + hash-verify one chunk from a bucket [LEA
@test -n "$(BLOB_BASE)" || { echo 'BLOB_BASE required (per-chunk blobs/<hash> base URL)'; exit 2; }
@$(ARBORIST) cloud fetch-chunk "$(LEAF_HASH)" --blob-base "$(BLOB_BASE)"
cloud-ask: bootstrap ## bucket-direct end-to-end: FTS → LLM → verify [Q="..." JSON=1 SHARD_URL=... TOP_K=N MAX_CONTEXT=N CACHE_MB=N]
@test -n "$(Q)" || { echo 'usage: make cloud-ask Q="your question" [JSON=1] [SHARD_URL=...] [TOP_K=4] [MAX_CONTEXT=24000] [CACHE_MB=64]'; exit 2; }
@echo "# shard: $(SHARD_URL)" >&2
@$(ARBORIST) cloud ask '$(Q)' --shard-url "$(SHARD_URL)" \
cloud-ask: bootstrap ## bucket-direct end-to-end via manifest [Q="..." JSON=1 BUCKET_URL=... TOP_K=N MAX_CONTEXT=N CACHE_MB=N]
@test -n "$(Q)" || { echo 'usage: make cloud-ask Q="your question" [JSON=1] [BUCKET_URL=...] [TOP_K=4] [MAX_CONTEXT=24000] [CACHE_MB=64]'; exit 2; }
@echo "# bucket: $(BUCKET_URL)" >&2
@$(ARBORIST) cloud ask '$(Q)' --bucket-url "$(BUCKET_URL)" \
--top-k $(or $(TOP_K),4) \
--max-context-chars $(or $(MAX_CONTEXT),24000) \
--cache-mb $(or $(CACHE_MB),64) \