Three workstreams, full suite 2482 passed, experimental paths default-OFF.
#000055 — Windows quickstart without make
tasks.py (pure-stdlib runner) + make.bat shim + .gitattributes;
README Windows section rewritten. Quickstart needs only Python
3.10+ (no make/bzip2/curl/bash). Mirrors the Makefile quickstart
subset; drift-pinned by tests/test_tasks_runner.py.
#000001 §7 Phase 0 — deterministic cross-language guard
arborist/qa/crosslang.py: non-English signal (¿/¡/non-ASCII) + an
es function-word stoppack. Fail-closed to UNGROUNDED before
retrieval/LLM (mirrors the quantifier reject-DAG) when no content
token survives, else strips es stopwords from the retrieval query
only. English path byte-identical by construction. Default OFF
(crosslang_guard_enabled). Measured: the anarcocapitalismo field
case 10.4s -> 1.6s.
#000056 — Operation Sandwich (cross-language grounding)
arborist/qa/mt/: opus-mt es/fr/ru<->en, lazy per-pair memoised
singleton (fixes the 88%-engine-error concurrency defect),
manifest-pinned, [mt] extra; entity_mask wrapper. Sandwich =
translate query in (retrieval + LLM prompt) -> English answer ->
UNTOUCHED verifier grounds English-vs-English -> translate the
verified answer out as display-only (banner-labelled, zero
grounding). question_hash + verifier_policy_hash invariant; MT
engine identity binds into RetrievalPlan, not governance. CLI
--crosslang-translate / make XLANG_MT=1. Default OFF; entity_mask
default OFF (measured net-negative at bench scale). Fan-out bench
(bench/*.py): Spanish ~0% -> 71% grounded vs the real no-support
baseline; the round-trip predictor was tried and refuted; the
entity-mask lever failed at scale (corpus-title anchoring untried).
CLAUDE.md: cross-language bright-line convention + module map.
Pre-existing modified diagram files are intentionally excluded.
Today's autonomous-quality run pushed test count from 1655
to 2093 (≈+438 across both shifts). README.md still cited the
older "641+" milestone from earlier in the project. Updated to
the current ~2000+ number with a parallel-execution latency
note.
Per shared-things convention: prefer 'our' for community-owned things;
'the' implies fixed singular ownership. The permacomputer is collective
infrastructure, so 'our' fits better than 'the'.
Updated three places (single canonical preamble text):
- LICENSE (Permacomputer Preamble section, our text — not the AGPL)
- README.md License section
- docs/_source/conf.py rst_epilog (per-page footer on RTD)
Mirrors the per-page footer that ships in the Sphinx docs (rst_epilog
in docs/_source/conf.py). Single source of canonical preamble text;
README and RTD pages stay aligned.
Whitepaper link already present — kept as the verbatim attribution.
549 → 379 lines. Removed sections that don't help onboarding:
- OpenAI export placeholder (10 lines): unimplemented feature, misleads
new users browsing source list
- Marking and burning records (21 lines): operational detail, not
first-run material
- Mesh / federation full subcommand reference (27 lines): off-by-default
feature; replaced with one-line note + link to docs/mesh.md
- Architecture full source tree (76 lines): now generated from docstrings
via Sphinx (make docs-api); replaced with one paragraph pointing to RTD
- Sidecar diagnostics + metaphor cue dictionary (39 lines): too deep for
README; lives in code docstrings + qa.inspect module
Also fixed broken references to deleted docs (concept-relations-design.md,
mesh-deploy.md, modules.md) — text rewritten to be self-contained or
point to whitepaper.
Onboarding flow preserved: title, quickstart, setup, basic query,
verifier ladder summary, license.
Three changes that shape the same lever:
(1) The metaphor-cue wordlist now unions /usr/share/dict/words +
/usr/share/dict/american-english + /usr/share/dict/british-english.
The Debian split made the prior 'just symlink to american-english'
miss British spellings (colour, organisation, realise) which
silently became false negatives on British-speaker questions.
Union: 102,485 → 104,305 entries on this machine. ~1,820 added
British-specific entries.
(2) Supplemental dictionary support: operators can layer
domain-specific vocabulary into the morphological substrate.
Two paths:
- Env var: ABORIST_METAPHOR_DICTS=/path/a:/path/b
- Programmatic: register_metaphor_dictionary(path)
Each supplemental dict is one word per line. The cue suffix
tests (-ly stem, -ing stem, -est stem) then resolve domain
stems automatically — adding 'aerodynamic' to a custom dict
makes 'aerodynamically' classify as adverbial without code
changes.
Use case: 'a tree with its own vocabulary' — an aviation
forest, a medical corpus, a legal-domain shard each carries
jargon the standard wordlist doesn't cover. Register once,
suffix tests pick up domain stems forever.
(3) README gains a 'Sidecar diagnostics' section with a table of
the three sidecars (deflection, title-relevance, metaphor-
deflection) plus a 'Metaphor-deflection cue dictionary' subsection
explaining the derivation rule, the load order, and the per-
forest vocabulary configurability. Architecturally documents
why the rule is *derived* from the union (Phase-2 lesson) and
not hand-curated.
3 new tests in tests/test_inspect.py:
- register_metaphor_dictionary unions a custom path's words
- ABORIST_METAPHOR_DICTS env var supplements with two paths
- re-registering same path is idempotent
763/34 tests pass.
12 per-module files + index = 13 files of 50-185 lines each =
1,064 lines of API reference scattered across a directory.
Each per-module file had real meat (API examples, ASCII tree,
conventions) but the cognitive cost of 'which file is this in?'
outweighed the navigation benefit.
Built via concatenation + patch-fix:
- cat index.md + per-module files in topological order
- rewrite ../diagrams/ -> diagrams/ (relative to docs/modules.md)
- rewrite ../../aborist/ -> ../aborist/
- rewrite ../TICKETS.md -> TICKETS.md, ../mesh.md -> mesh.md, etc.
- inter-module links (./<name>.md, <name>.md) -> #<name>-py anchors
- demote per-module H1 -> H2, H2 -> H3, etc., so the wrapper H1
is the only top-level heading
- de-dup the index.md's (now-H2) 'Aborist module reference'
header against the wrapper, replace with 'Diagrams index'
- inject explicit <a id="<name>-py"></a> anchors after each
module's H2 so the TOC links resolve regardless of GitHub's
auto-slug rules
- polish TOC link text: '[merkle.md](#merkle-py)' -> '[↓](#...)'
(the '.md' suffix made no sense once it's an in-doc anchor)
References updated:
- README.md (×2)
Net: 1,124 single-file lines vs 1,064 across 13 files. Slightly
larger because of the patch-fix scaffolding (anchors + section
markers), but one Cmd-F covers everything.
751/34 tests still pass.
bench-maxing.md gains a 'Bench harness — the speed playbook'
section capturing the 2026-05-02 speed wins as durable doctrine:
- Sample-level shuffled scheduling vs cell-grouped (+58%
throughput, true i.i.d. variance for n>=3)
- Persistent httpx client (TLS handshake amortization)
- Concurrency tuning (vLLM peak at c=3-4, brutal past c=4)
- Per-mode max_context_chars from bench feedback (the bench
is the substrate's voice; let it drive policy)
- --seed for reproducibility
- --resume for stop/start-able bench
- Smoke fixture for inner-loop iteration
- pytest -n auto (3.6× speedup on test suite)
README:
- Updated whitepaper section refs (§6/7/8/9/13) — old refs
pointed at §13.4.11/13.8/13.9 which no longer exist after
the whitepaper rewrite landed.
- 'Tests' section renamed 'Tests & bench' with make targets
for bench-qa, bench-qa-smoke, test-live. Resume + concurrency
semantics surfaced.
The link was still pointing at the old pre-Providence slug
(/downloads/merkle-reverse-rag-whitepaper.pdf). Canonical URL is
/merkle-providence-reverse-rag-whitepaper.pdf at the static root,
where `make publish-and-commit` keeps it fresh on every whitepaper
edit. Same fix as the unsandbox.com Phoenix templates (commit
0d1a69c) — last remaining stale reference.
Per fox: README was stale on most of what shipped over the past
sessions. Sweep:
- New Quickstart section at the top (replaces "What this gets you")
with two end-to-end paths: Wikipedia 2003 dump and crawling a live
site (russell.ballestrini.net as the worked example). Each is
five-ish commands and ends with a real `make query`.
- New "Data: live websites (the crawler)" section between git/hg
ingest and the OpenAI-export placeholder. Documents:
- bootstrap-crawler / crawl-ingest with all knobs (URL, DEPTH,
MAX, FAST, CRAWL_SHARD)
- per-domain shard naming so make query auto-discovers the data
- feed/sitemap skip at ingest (discovery infrastructure, not
knowledge)
- ETag/Last-Modified capture + recrawl-check freshness probe
- provenance pointer to ~/git/agents.ai.unturf.com
- Asking-the-corpus section gains JSON=1, default human render note,
question_hash equivalence (trailing punctuation stripped), and the
full query path now reflects wikitext-strip-before-LLM step.
- New "Verifying answers (audit modes)" section explains the
trichotomy + four-strategy verifier (quote/span/entity/paraphrase),
trailing-citation strip, and the corpus-growth signal flow
(emergent → reclassify).
- New "Marking and burning records" section covers falsify vs burn
(kindergarten rule, KIND= for documents and cores), chain-check.
- Mesh section: dropped "on the roadmap" claim — wire layer +
serve/sync/pull verbs are shipped. Pointers to docs/mesh.md and
docs/mesh-deploy.md.
- Architecture diagram updated to current tree (wikitext.py,
sources/crawler/, qa/inspect.py, qa/verify.py, mesh/, vcs.py).
- Whitepaper pointer updated to canonical rst path; mesh docs added.
- Test counts: 326+ default suite, separate test-crawler target,
default suite never hits the network.
Phase 1 of the federation/gossip layer fox sketched as the natural
extension of v9.8 admissibility's content-addressed identity. Two
peers ingesting the same dump already compute identical document_roots
and identical 8-dim cache_keys; the mesh layer is the wire-and-trust
plumbing that lets them dedup answers, exchange Merkle proofs, and
cleanly distrust an evicted member without a hard fork.
Cryptography (cryptography lib, audited):
Ed25519 — every membership mutation + (future) gossip envelope
is signed by the actor's pubkey.
X25519 ECDH — wraps each epoch's symmetric mesh secret to every
current member's DH pubkey via HKDF-derived AEAD key.
ChaCha20-P1305— AEAD for envelope payloads + per-member secret wrap.
State machine:
mesh_identity — singleton; this peer's keys + group name
mesh_roster — per-epoch (member_id, sign_pub, dh_pub, role)
mesh_epochs — epoch_id -> {started_at, started_event_hash,
secret_envelope JSON, reason}
meta:mesh.enabled flag — off by default; gates everything
Eviction works by rotating to a new epoch whose envelope omits the
kicked member. Their prior signatures stay verifiable (the older
roster row is retained), but any gossip from epoch+1 onward is
opaque to them — the secret was never shared with their pubkey.
Authority gate: only roster members with role='admin' can add or
kick. Self-kick is rejected explicitly. The last admin can't be
kicked. Schedule-rotate (refresh secret, no roster change) is open
to any current member as a session-hygiene op.
Audit-chain integration: every mesh state mutation writes an audit
event (mesh_init, mesh_enable/disable, mesh_epoch_rotate). The
epoch's started_event_hash backfills into mesh_epochs after the
audit row commits, giving each epoch a tamper-evident pin into the
ledger.
CLI subcommands: mesh init, mesh status, mesh enable, mesh disable,
mesh members, mesh add, mesh kick, mesh rotate. All read-only or
local-state-only — no network code paths in this commit.
The HTTP gossip wire (`mesh sync`, `mesh serve`) is the next phase.
Schema, cryptography, and roster state machine are all in place to
support it without further migration.
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`,
yielding one Document per text file. URI shape `git://<repo>/file/<path>`
intentionally omits the commit hash — re-ingest after new commits produces
fresh document_roots that aborist's prior-doc detection chains via
`supersedes` edges, so the audit trail and Merkle tree grow as the repo
grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files
filtered out by default. Commit hash + timestamp + subject ride along in
`extra` (informational only; not part of the Merkle commitment).
MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes
semantics, same shape.
Makefile adds:
make ingest-self (this repo -> aborist-self.db)
make ingest-git GIT_REPO=/path/to/repo (arbitrary git clone)
make ingest-hg HG_REPO=/path/to/repo (mercurial)
Each lands in its own shard file alongside existing shards/grok.db,
keeping per-shard write paths independent of the wikipedia 4-way ingest's
WAL writer lock.
Adds a new section after the 2003 Wikipedia / distillation block that
covers Phase IV (2006+) XML dumps: fetch-xml / ingest-xml-attached /
ingest-abstract targets, the WP_XML_YEAR/MONTH/DATE knobs for picking
any dated snapshot in dumps.wikimedia.org/archive, plus a pointer at
the abstract.xml feed as a cheap-coverage alternative to the full
pages-articles dump (~5-10 GB instead of ~95 GB on disk for enwiki
2010-11). Existing 2003 section retitled to flag the format split.
Replace the boilerplate AGPL-3.0 header in LICENSE with the exact
license clause from the Merkle Providence Reverse RAG whitepaper
(April 2026, Russell Ballestrini / David Wong / Riley Morgan):
License: AGPL-3.0-only · This algorithm, its implementation, &
all associated code carry the GNU Affero General Public License
v3.0 (only). You may use, modify, & distribute under those terms.
No proprietary relicensing exists.
README's License section quotes the same clause and links the
whitepaper as the canonical source.
Source: https://unfirehose.com/downloads/merkle-reverse-rag-whitepaper.pdf
Adds a "Get the source" section with the GitLab SSH + HTTPS clone
URLs, then platform-specific prerequisite blocks:
- macOS: Xcode CLT for git/make, Homebrew for python@3.12
- Ubuntu/Debian: apt install python3/python3-venv/build-essential
/curl/bzip2; 22.04 has 3.10, 24.04 has 3.12
- Windows: WSL2 Ubuntu-24.04 is the supported path. Native cmd
+ Git Bash mostly works for Python but several make targets call
bash idioms. `wsl --install -d Ubuntu-24.04` from an admin shell.
- OpenBSD: pkg_add git python-3.12 gmake curl. Default `make` is
BSD make; the project uses GNU-make features so substitute
`gmake` everywhere.
Bootstrap step (make bootstrap) is unchanged — it stays at the end
of the Setup section once the platform prereqs are in place.
Replaces the stub README with a working user guide:
- 30-second quickstart that lands a queryable corpus in ~3 min
- Setup section (Python 3.10+, make bootstrap creates the venv +
installs the package in editable mode)
- Wikipedia 2003-05-16 dataset:
fetch-cur / fetch-old / fetch what each downloads & where
ingest-cur-attached SHARDS=N sharded ingest, the canonical
path (no WAL contention)
ingest-cur-parallel / ingest-cur single-DB variants
--resume rsync-style restart
- Distillation (first-sentence + TF-IDF cores) and what they buy
- Personal Grok export ingest (point GROK_EXPORT at the xAI bundle)
- OpenAI export placeholder section (one new Source subclass; the
pattern matches sources/grok.py)
- Asking the corpus: query path explained briefly, 8-dim cache key,
per-phase timings, LLM endpoint env vars
- Falsification: how to mark a wrong cached answer stale
- Inspection: stats-shards / analyze-shards / verify-shards / activity
- Architecture tree (one screen)
- Pointer to source papers (whitepaper + v7 substrate)
Reflects the current shipped feature set including the three
retrieval relevance signals (title + TF-IDF cores + body density),
concept overlay, sharded distill, falsification, activity timeline,
and per-phase timings.
