bench-maxing.md gains a 'Bench harness — the speed playbook'
section capturing the 2026-05-02 speed wins as durable doctrine:
- Sample-level shuffled scheduling vs cell-grouped (+58%
throughput, true i.i.d. variance for n>=3)
- Persistent httpx client (TLS handshake amortization)
- Concurrency tuning (vLLM peak at c=3-4, brutal past c=4)
- Per-mode max_context_chars from bench feedback (the bench
is the substrate's voice; let it drive policy)
- --seed for reproducibility
- --resume for stop/start-able bench
- Smoke fixture for inner-loop iteration
- pytest -n auto (3.6× speedup on test suite)
README:
- Updated whitepaper section refs (§6/7/8/9/13) — old refs
pointed at §13.4.11/13.8/13.9 which no longer exist after
the whitepaper rewrite landed.
- 'Tests' section renamed 'Tests & bench' with make targets
for bench-qa, bench-qa-smoke, test-live. Resume + concurrency
semantics surfaced.
The link was still pointing at the old pre-Providence slug
(/downloads/merkle-reverse-rag-whitepaper.pdf). Canonical URL is
/merkle-providence-reverse-rag-whitepaper.pdf at the static root,
where `make publish-and-commit` keeps it fresh on every whitepaper
edit. Same fix as the unsandbox.com Phoenix templates (commit
0d1a69c) — last remaining stale reference.
Per fox: README was stale on most of what shipped over the past
sessions. Sweep:
- New Quickstart section at the top (replaces "What this gets you")
with two end-to-end paths: Wikipedia 2003 dump and crawling a live
site (russell.ballestrini.net as the worked example). Each is
five-ish commands and ends with a real `make query`.
- New "Data: live websites (the crawler)" section between git/hg
ingest and the OpenAI-export placeholder. Documents:
- bootstrap-crawler / crawl-ingest with all knobs (URL, DEPTH,
MAX, FAST, CRAWL_SHARD)
- per-domain shard naming so make query auto-discovers the data
- feed/sitemap skip at ingest (discovery infrastructure, not
knowledge)
- ETag/Last-Modified capture + recrawl-check freshness probe
- provenance pointer to ~/git/agents.ai.unturf.com
- Asking-the-corpus section gains JSON=1, default human render note,
question_hash equivalence (trailing punctuation stripped), and the
full query path now reflects wikitext-strip-before-LLM step.
- New "Verifying answers (audit modes)" section explains the
trichotomy + four-strategy verifier (quote/span/entity/paraphrase),
trailing-citation strip, and the corpus-growth signal flow
(emergent → reclassify).
- New "Marking and burning records" section covers falsify vs burn
(kindergarten rule, KIND= for documents and cores), chain-check.
- Mesh section: dropped "on the roadmap" claim — wire layer +
serve/sync/pull verbs are shipped. Pointers to docs/mesh.md and
docs/mesh-deploy.md.
- Architecture diagram updated to current tree (wikitext.py,
sources/crawler/, qa/inspect.py, qa/verify.py, mesh/, vcs.py).
- Whitepaper pointer updated to canonical rst path; mesh docs added.
- Test counts: 326+ default suite, separate test-crawler target,
default suite never hits the network.
Phase 1 of the federation/gossip layer fox sketched as the natural
extension of v9.8 admissibility's content-addressed identity. Two
peers ingesting the same dump already compute identical document_roots
and identical 8-dim cache_keys; the mesh layer is the wire-and-trust
plumbing that lets them dedup answers, exchange Merkle proofs, and
cleanly distrust an evicted member without a hard fork.
Cryptography (cryptography lib, audited):
Ed25519 — every membership mutation + (future) gossip envelope
is signed by the actor's pubkey.
X25519 ECDH — wraps each epoch's symmetric mesh secret to every
current member's DH pubkey via HKDF-derived AEAD key.
ChaCha20-P1305— AEAD for envelope payloads + per-member secret wrap.
State machine:
mesh_identity — singleton; this peer's keys + group name
mesh_roster — per-epoch (member_id, sign_pub, dh_pub, role)
mesh_epochs — epoch_id -> {started_at, started_event_hash,
secret_envelope JSON, reason}
meta:mesh.enabled flag — off by default; gates everything
Eviction works by rotating to a new epoch whose envelope omits the
kicked member. Their prior signatures stay verifiable (the older
roster row is retained), but any gossip from epoch+1 onward is
opaque to them — the secret was never shared with their pubkey.
Authority gate: only roster members with role='admin' can add or
kick. Self-kick is rejected explicitly. The last admin can't be
kicked. Schedule-rotate (refresh secret, no roster change) is open
to any current member as a session-hygiene op.
Audit-chain integration: every mesh state mutation writes an audit
event (mesh_init, mesh_enable/disable, mesh_epoch_rotate). The
epoch's started_event_hash backfills into mesh_epochs after the
audit row commits, giving each epoch a tamper-evident pin into the
ledger.
CLI subcommands: mesh init, mesh status, mesh enable, mesh disable,
mesh members, mesh add, mesh kick, mesh rotate. All read-only or
local-state-only — no network code paths in this commit.
The HTTP gossip wire (`mesh sync`, `mesh serve`) is the next phase.
Schema, cryptography, and roster state machine are all in place to
support it without further migration.
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`,
yielding one Document per text file. URI shape `git://<repo>/file/<path>`
intentionally omits the commit hash — re-ingest after new commits produces
fresh document_roots that aborist's prior-doc detection chains via
`supersedes` edges, so the audit trail and Merkle tree grow as the repo
grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files
filtered out by default. Commit hash + timestamp + subject ride along in
`extra` (informational only; not part of the Merkle commitment).
MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes
semantics, same shape.
Makefile adds:
make ingest-self (this repo -> aborist-self.db)
make ingest-git GIT_REPO=/path/to/repo (arbitrary git clone)
make ingest-hg HG_REPO=/path/to/repo (mercurial)
Each lands in its own shard file alongside existing shards/grok.db,
keeping per-shard write paths independent of the wikipedia 4-way ingest's
WAL writer lock.
Adds a new section after the 2003 Wikipedia / distillation block that
covers Phase IV (2006+) XML dumps: fetch-xml / ingest-xml-attached /
ingest-abstract targets, the WP_XML_YEAR/MONTH/DATE knobs for picking
any dated snapshot in dumps.wikimedia.org/archive, plus a pointer at
the abstract.xml feed as a cheap-coverage alternative to the full
pages-articles dump (~5-10 GB instead of ~95 GB on disk for enwiki
2010-11). Existing 2003 section retitled to flag the format split.
Replace the boilerplate AGPL-3.0 header in LICENSE with the exact
license clause from the Merkle Providence Reverse RAG whitepaper
(April 2026, Russell Ballestrini / David Wong / Riley Morgan):
License: AGPL-3.0-only · This algorithm, its implementation, &
all associated code carry the GNU Affero General Public License
v3.0 (only). You may use, modify, & distribute under those terms.
No proprietary relicensing exists.
README's License section quotes the same clause and links the
whitepaper as the canonical source.
Source: https://unfirehose.com/downloads/merkle-reverse-rag-whitepaper.pdf
Adds a "Get the source" section with the GitLab SSH + HTTPS clone
URLs, then platform-specific prerequisite blocks:
- macOS: Xcode CLT for git/make, Homebrew for python@3.12
- Ubuntu/Debian: apt install python3/python3-venv/build-essential
/curl/bzip2; 22.04 has 3.10, 24.04 has 3.12
- Windows: WSL2 Ubuntu-24.04 is the supported path. Native cmd
+ Git Bash mostly works for Python but several make targets call
bash idioms. `wsl --install -d Ubuntu-24.04` from an admin shell.
- OpenBSD: pkg_add git python-3.12 gmake curl. Default `make` is
BSD make; the project uses GNU-make features so substitute
`gmake` everywhere.
Bootstrap step (make bootstrap) is unchanged — it stays at the end
of the Setup section once the platform prereqs are in place.
Replaces the stub README with a working user guide:
- 30-second quickstart that lands a queryable corpus in ~3 min
- Setup section (Python 3.10+, make bootstrap creates the venv +
installs the package in editable mode)
- Wikipedia 2003-05-16 dataset:
fetch-cur / fetch-old / fetch what each downloads & where
ingest-cur-attached SHARDS=N sharded ingest, the canonical
path (no WAL contention)
ingest-cur-parallel / ingest-cur single-DB variants
--resume rsync-style restart
- Distillation (first-sentence + TF-IDF cores) and what they buy
- Personal Grok export ingest (point GROK_EXPORT at the xAI bundle)
- OpenAI export placeholder section (one new Source subclass; the
pattern matches sources/grok.py)
- Asking the corpus: query path explained briefly, 8-dim cache key,
per-phase timings, LLM endpoint env vars
- Falsification: how to mark a wrong cached answer stale
- Inspection: stats-shards / analyze-shards / verify-shards / activity
- Architecture tree (one screen)
- Pointer to source papers (whitepaper + v7 substrate)
Reflects the current shipped feature set including the three
retrieval relevance signals (title + TF-IDF cores + body density),
concept overlay, sharded distill, falsification, activity timeline,
and per-phase timings.
