engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

wallet/sidecar: HTTP-optimized inverted-index sidecar (v2 + BM25)

Browse source 
The bucket-direct FTS5 path runs at WAN-RTT × b-tree-page-count
latency: 4+ minutes per query on a 9 GB shard. 64 KB pages and
read-ahead don't help — FTS5 working set on a large corpus exceeds
any affordable HTTP cache. Need a different data structure.

Sidecar = custom binary inverted index:
  - one ~500 MB file per shard, ONE HTTP RANGE GET to download
  - sorted term dictionary + concatenated posting lists +
    doc table (root, uri, title, length)
  - varint deltas on doc_ids, varint tf per posting
  - BM25 scoring with idf, tf, doc length normalization
  - sub-ms query latency after a one-time load

Format (v2):
  HEADER (60 B)         magic + dict_count + docs_count + offsets
  DICT (~30 % of file)  sorted terms with (df, posting_off, posting_len)
  POSTINGS (~55 %)      per-term: varint num_docs + (delta, tf) pairs
  DOCS (~15 %)          (root_32, uri_len, uri, title_len, title, doc_len)
  DOC_OFFSETS (~1 %)    u64 per doc — random-access into DOCS

Producer: `arborist sidecar build --shard X.db --out X.bin`
  tokenizes chunk text (matches `_to_fts5` sanitizer for build/query
  parity), tracks per-doc tf, builds inverted index, writes file.
Consumer: `arborist sidecar search "..." --sidecar X.bin`
  loads file once, binary-searches dict, decodes posting lists on
  demand, returns BM25-ranked SidecarHits.

Bench (genesis shard 002, 7.6 GB → sidecar 542 MB, v1 presence-only):
  Q: "elixir"                3 ms — top hits all elixir articles
  Q: "barack obama"        2.8 ms — both terms present
  Q: "anarchism"           193 ms — slow only on huge posting lists
v2 BM25 small-corpus sanity:
  Q: "who developed virt-back?"  →  virt-back article #1 (score 9.65)

Makefile + CLI:
  make sidecar-build SHARD=... OUT=...
  make sidecar-search Q="..." SIDECAR=...
This commit is contained in:
russell@unturf.com 2026-05-30 15:38:19 -04:00
parent 331e748bcb
commit 349dd0fd48
No known key found for this signature in database
3 changed files with 623 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

10
Makefile
View file

@ -1573,6 +1573,16 @@ cloud-snapshot-root: bootstrap ## compute snapshot_root of the bucket-resident s
@echo "# shard: $(SHARD_URL)" >&2
@$(ARBORIST) cloud snapshot-root --shard-url "$(SHARD_URL)" --cache-mb $(or $(CACHE_MB),32)
sidecar-build: bootstrap ## build HTTP-optimized inverted-index sidecar [SHARD=path OUT=path]
@test -n "$(SHARD)" || { echo 'usage: make sidecar-build SHARD=/path/to/shard.db OUT=/path/to/sidecar.bin'; exit 2; }
@test -n "$(OUT)" || { echo 'OUT required'; exit 2; }
@$(ARBORIST) sidecar build --shard "$(SHARD)" --out "$(OUT)"
sidecar-search: bootstrap ## query a local sidecar.bin [Q="..." SIDECAR=/path/sidecar.bin LIMIT=N MODE=or|and]
@test -n "$(Q)" || { echo 'usage: make sidecar-search Q="your question" SIDECAR=/path/sidecar.bin [LIMIT=N] [MODE=or|and]'; exit 2; }
@test -n "$(SIDECAR)" || { echo 'SIDECAR=/path/sidecar.bin required'; exit 2; }
@$(ARBORIST) sidecar search '$(Q)' --sidecar "$(SIDECAR)" --limit $(or $(LIMIT),8) --mode $(or $(MODE),or)
cloud-fetch-chunk: bootstrap ## fetch + hash-verify one chunk from a bucket [LEAF_HASH=hex BLOB_BASE=https://...]
@test -n "$(LEAF_HASH)" || { echo 'usage: make cloud-fetch-chunk LEAF_HASH=hex BLOB_BASE=https://.../blobs'; exit 2; }
@test -n "$(BLOB_BASE)" || { echo 'BLOB_BASE required (per-chunk blobs/<hash> base URL)'; exit 2; }