From 3a9389f24b68ed499ded8285708f76a3b0a8a3c2 Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Thu, 14 May 2026 11:16:20 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20correct=20storage=20figures=20in=20page?= =?UTF-8?q?rs=20=E2=80=94=20actual=20per-shard=20target=20is=20~10=20GB,?= =?UTF-8?q?=20live=20deployment=20is=204=20shards=20=C3=97=20~9.6=20GB=20?= =?UTF-8?q?=E2=89=88=2038=20GB=20(was=20wrongly=20stated=20as=20~2=20GB)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Shard capacity convention is documented in arborist/search/fts5.py:113 ("~50ms cold per token on a 10GB shard"). The earlier "~2 GB for a Wikipedia-sized corpus" line in both pagers was a fabricated figure off by 20×. Replaced with the real numbers: per-shard ~10 GB design target, live deployment of four shards totalling ~38 GB and holding 3.5 M documents / 6.2 M chunks. --- docs/_source/arborist-one-pager.rst | 11 +++++++---- docs/_source/arborist-two-pager.rst | 5 ++++- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/_source/arborist-one-pager.rst b/docs/_source/arborist-one-pager.rst index 48a9eec..549be03 100644 --- a/docs/_source/arborist-one-pager.rst +++ b/docs/_source/arborist-one-pager.rst @@ -51,10 +51,13 @@ policy hit the same record. Reproducible. Replayable. Shareable. cited claim resolves to a chain of evidence ending at a public-domain or open-licensed source. -**What it costs.** Python 3.12. SQLite, one file (~2 GB for a -Wikipedia-sized corpus). No GPU for the proof path. Use any -OpenAI-compatible inference endpoint; the free reference endpoint is -`hermes.ai.unturf.com `_. Source: +**What it costs.** Python 3.12. SQLite, one file per shard, each +designed around ~10 GB. A Wikipedia-class corpus lands across a +handful of shards (the live deployment is four shards, ~38 GB on +disk, holding 3.5 M documents / 6.2 M chunks). No GPU for the proof +path. Use any OpenAI-compatible inference endpoint; the free +reference endpoint is `hermes.ai.unturf.com +`_. Source: `git.unturf.com/engineering/unturf/arborist `_. Full whitepaper: `unfirehose.com/merkle-providence-reverse-rag.html diff --git a/docs/_source/arborist-two-pager.rst b/docs/_source/arborist-two-pager.rst index 6a0ed36..a047719 100644 --- a/docs/_source/arborist-two-pager.rst +++ b/docs/_source/arborist-two-pager.rst @@ -115,7 +115,10 @@ What it costs ================== ============================================================ Language Python 3.12 -Storage SQLite — one file, ~2 GB for a Wikipedia-sized corpus +Storage SQLite — one file per shard, designed around ~10 GB each. + A Wikipedia-class corpus lands across a handful of shards + (the live deployment is 4 × ~9.6 GB ≈ 38 GB, holding + 3.5 M documents and 6.2 M chunks). Compute No GPU required for the proof path. Inference runs wherever you want — locally, or via an OpenAI-compatible endpoint. Optional Cross-encoder rerankers (80–560 MB) for harder relevance