wallet/sidecar: CLI-callable build runner for parallel batch builds
This commit is contained in:
parent
f669901f30
commit
b932784154
2 changed files with 38 additions and 5 deletions
12
Makefile
12
Makefile
|
|
@ -1552,11 +1552,13 @@ wallet-ask: bootstrap ## query the wallet server with your own question [Q="..."
|
|||
SHARD_URL ?= https://nyc3.digitaloceanspaces.com/arborist/clones/virtback/000.db
|
||||
BLOB_BASE ?=
|
||||
|
||||
# BUCKET_URL is the root the multi-shard cloud-ask points at. Client GETs
|
||||
# $BUCKET_URL/clones/manifest.json and queries every listed shard. One
|
||||
# config var → entire corpus. Default = our DO Spaces public bucket with
|
||||
# genesis Wikipedia + russell.ballestrini.net.
|
||||
BUCKET_URL ?= https://nyc3.digitaloceanspaces.com/arborist
|
||||
# BUCKET_URL is the manifest the multi-shard cloud-ask points at. Client
|
||||
# GETs $BUCKET_URL (treated as a manifest URL if it ends in .json, else
|
||||
# appends `clones/manifest.json`) and queries every listed shard.
|
||||
# Default = sidecar-enabled manifest on DO Spaces: per-shard inverted-
|
||||
# index sidecar.bin where available (sub-second FTS) + bucket-direct
|
||||
# FTS5 fallback for shards still being indexed.
|
||||
BUCKET_URL ?= https://nyc3.digitaloceanspaces.com/arborist/clones/manifest-sidecar.json
|
||||
|
||||
bootstrap-bucket: bootstrap ## install apsw (bucket-direct extra)
|
||||
$(PIP) install 'apsw>=3.45'
|
||||
|
|
|
|||
31
arborist/wallet/_sidecar_build_runner.py
Normal file
31
arborist/wallet/_sidecar_build_runner.py
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
"""CLI-callable sidecar build runner.
|
||||
|
||||
Tiny shim so shell scripts can do
|
||||
python3 -m arborist.wallet._sidecar_build_runner SRC DST
|
||||
without inlining multiline Python that gets eaten by shell quoting.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
import time
|
||||
|
||||
from arborist.wallet.sidecar import build_sidecar
|
||||
|
||||
|
||||
def main(src: str, dst: str) -> int:
|
||||
t0 = time.time()
|
||||
stats = build_sidecar(src, dst)
|
||||
elapsed = time.time() - t0
|
||||
print(
|
||||
f"build_secs={elapsed:.1f} "
|
||||
f"file_size_mb={stats['file_bytes']/1024**2:.1f} "
|
||||
f"terms={stats['terms']} docs={stats['docs']}"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) != 3:
|
||||
print("usage: python -m arborist.wallet._sidecar_build_runner SRC DST", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
sys.exit(main(sys.argv[1], sys.argv[2]))
|
||||
Loading…
Add table
Add a link
Reference in a new issue