engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: correct storage figures in pagers — actual per-shard target is ~10 GB, live deployment is 4 shards × ~9.6 GB ≈ 38 GB (was wrongly stated as ~2 GB)

Browse source 
Shard capacity convention is documented in arborist/search/fts5.py:113 ("~50ms cold per token on a 10GB shard"). The earlier "~2 GB for a Wikipedia-sized corpus" line in both pagers was a fabricated figure off by 20×. Replaced with the real numbers: per-shard ~10 GB design target, live deployment of four shards totalling ~38 GB and holding 3.5 M documents / 6.2 M chunks.
This commit is contained in:
russell@unturf.com 2026-05-14 11:16:20 -04:00
parent 1035d256fe
commit 3a9389f24b
No known key found for this signature in database
2 changed files with 11 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

11
docs/_source/arborist-one-pager.rst
View file

@ -51,10 +51,13 @@ policy hit the same record. Reproducible. Replayable. Shareable.
cited claim resolves to a chain of evidence ending at a public-domain
or open-licensed source.
**What it costs.** Python 3.12. SQLite, one file (~2 GB for a
Wikipedia-sized corpus). No GPU for the proof path. Use any
OpenAI-compatible inference endpoint; the free reference endpoint is
`hermes.ai.unturf.com <https://hermes.ai.unturf.com>`_. Source:
**What it costs.** Python 3.12. SQLite, one file per shard, each
designed around ~10 GB. A Wikipedia-class corpus lands across a
handful of shards (the live deployment is four shards, ~38 GB on
disk, holding 3.5 M documents / 6.2 M chunks). No GPU for the proof
path. Use any OpenAI-compatible inference endpoint; the free
reference endpoint is `hermes.ai.unturf.com
<https://hermes.ai.unturf.com>`_. Source:
`git.unturf.com/engineering/unturf/arborist
<https://git.unturf.com/engineering/unturf/arborist>`_. Full
whitepaper: `unfirehose.com/merkle-providence-reverse-rag.html

5
docs/_source/arborist-two-pager.rst
View file

@ -115,7 +115,10 @@ What it costs
================== ============================================================
Language Python 3.12
Storage SQLite — one file, ~2 GB for a Wikipedia-sized corpus
Storage SQLite — one file per shard, designed around ~10 GB each.
A Wikipedia-class corpus lands across a handful of shards
(the live deployment is 4 × ~9.6 GB ≈ 38 GB, holding
3.5 M documents and 6.2 M chunks).
Compute No GPU required for the proof path. Inference runs wherever
you want — locally, or via an OpenAI-compatible endpoint.
Optional Cross-encoder rerankers (80–560 MB) for harder relevance