Burn deletes a providence_cache row outright; falsify (the existing
verb) flips state and keeps the row. Burn is for early/scratch use
during a tree's genesis when downstream consumers haven't ingested
the answer yet.
Children gate: refuses if any falsifications reference the cache_key.
Caller can override with --force; the audit event records forced=True
plus the child count at burn time so the chain can reconstruct what
was orphaned. Refusal is non-destructive AND skips the audit event so
scripts can retry without polluting history.
Always writes a 'providence_burn' audit event regardless of force.
Body fields: cache_key, burned_audit_mode, burned_n_verified,
burned_state, question_text, reason, by_actor, child_falsifications_at_burn,
forced. The chain-break query (`make chain-check`) shows 0 after burn —
the new event extends the prior chain cleanly.
Surface:
aborist burn --cache-key <hex> [--reason '...'] [--force]
make burn KEY=<hex> REASON='...' [FORCE=1]
Tests (7): happy-path delete + audit event, refusal on children,
--force override, unknown-key not-found, chain-integrity preserved,
CLI happy path, CLI non-zero exit on refusal.
- aborist/qa/verify.py: three strategies tried in sequence. quote uses
sequential pairing (1st & 2nd `"`, 3rd & 4th, ...) which eliminates
the phantom inter-pair captures regex pairing produced on adjacent
quote pairs like `"title" prose "quote"`. span checks bullet/sentence
lines verbatim. entity matches multi-word proper nouns; gated by
policy ∈ {strict, hybrid, drop, proximity}. Default proximity
promotes to STRICT only when N=3 verified entities cluster within
W=300 chars in source — separates structural grounding (cast list,
infobox) from incidental mention (scattered plot summary).
- aborist/store.py: schema CHECK now `('STRICT','HYBRID','UNGROUNDED')`.
Migration helper rebuilds legacy `('STRICT','HYBRID','VISUAL')`
tables via temp-table copy, translating VISUAL → UNGROUNDED in the
SELECT. Idempotent — DDL inspection skips the rebuild on already-
migrated DBs.
- aborist/cli.py: new `aborist reclassify` re-runs the verifier
against existing live records under the current entity policy; no
LLM calls. --compare runs all four policies side-by-side, --dry-run
reports transitions without writing. Cold-source records skipped.
One providence_reclassify audit event per changed row.
- AuditMode.UNGROUNDED replaces VISUAL across search backend, FTS5,
test fixtures, CLAUDE.md. The substrate (Merkle-AGI) name was
about FOR-style visualization; in the RAG layer the semantic is
"no recoverable grounding," so the label now says that.
DEFAULT_QUERY_POLICY gains entity_policy + entity_proximity_n +
entity_proximity_window so any tuning folds into governance_policy_hash
and invalidates cache cleanly.
Implements the protocol contract pinned in docs/mesh.md. Each peer
runs a stdlib ThreadingHTTPServer; clients send Ed25519-signed
envelopes carrying one of:
ANNOUNCE_ROOT document_root + source_uri + versions
ANNOUNCE_DERIVATION core_root <- surface_roots (proof-blob hash)
ANNOUNCE_PROVIDENCE cache_key + audit_mode + answer_hash
ANNOUNCE_FALSIFICATION cache_key + reason
REQUEST_BODY/DELIVER_BODY pull-on-miss + Merkle-root verify
WireEnvelope canonicalizes via sorted-key separator-tight JSON; the
canonical bytes are what get signed. Receivers verify the signature
against sender_id's sign_pub looked up in mesh_roster at the
envelope's epoch_id — non-members of that epoch produce no valid
signature, so they're silently rejected (401, no audit event).
Each accepted ANNOUNCE_* writes one 'mesh_received' event into the
local audit chain whose body is the full signed envelope (the
remote's signature stays attached for non-repudiation). The
receiver's chain stays internally consistent because we append in
receive-order.
REQUEST_BODY/DELIVER_BODY: the responder ships the document text
plus per-chunk leaf hashes; the client re-derives the Merkle root
from those leaves and refuses to return any body whose leaves
don't reconstruct the requested root.
Tests: 17 unit (envelope canonicalization, sig verify against
roster, type validation, tampered-body rejection, audit-event
write) + 5 end-to-end (two real HTTP peers on ephemeral ports
exchanging announces and bodies). 22/22 green.
Deferred to later commits, per docs/mesh.md:
- CLI verbs (mesh serve, mesh sync) — pending downstream merge
- per-peer chain-of-claims tracking (catchup, replay, fork detection)
- optional AEAD encryption of message bodies (contract says optional)
Adds aborist/wikitext.py with to_base() — deterministic wikitext →
prose conversion via mwparserfromhell. Drops <ref>...</ref>,
[[File:...]], [[Image:...]], [[Category:...]] entirely; resolves
piped wikilinks to display text; collapses templates, formatting,
and HTML markup. extract_wikilinks() preserves the link graph for
the definition-cloud artifact (recoverable from any page on demand
without re-parsing wikitext at query time).
Wires to_base() into verify_quotes(). Without the strip, the
verifier compared the model's clean prose against [[Cloud Strife]],
[[Shinra Electric Power Company|Shinra]], etc. and falsely flagged
real source quotes as VISUAL. Concrete case from a make-query run:
'Cloud Strife, an unsociable mercenary who claims to be a former
1st Class member of Shinra's SOLDIER unit;' is verbatim in the
Final_Fantasy_VII article wikitext (modulo markup). With the strip
that quote now verifies; the genuine model hallucinations in the
same answer still flag honestly. Side-effect: 43% smaller context
size on average so less LLM token waste.
BASE_VERSION = 'wikitext-base-v1' is the algorithm pin. Bump when
the strip rules change. Soft-imported in verify.py so environments
without mwparserfromhell installed degrade gracefully (no strip,
same behavior as before this commit).
Build:
- pyproject.toml: new [wikitext] extras (mwparserfromhell>=0.6),
pulled in by [dev]
- Makefile: chain-check / chain-check-shards targets — fast
audit-chain integrity probe, counts dangling prev_event_hash
references; 0 = chain intact
Tests:
- tests/test_wikitext.py: 31 tests (rules, idempotence, real-corpus
fixture)
- tests/test_verify.py: 2 regression tests pinning the FF7 flip
(Cloud Strife quote: VISUAL → STRICT after strip; genuine
hallucination: stays VISUAL)
- tests/fixtures/ff7_characters_chunk0.wikitext: real chunk from a
shard, used to validate the strip on actual Wikipedia content
Adds aborist/qa/verify.py — three-strategy verifier (explicit quotes,
bullet/sentence spans, multi-word proper nouns) that lexically checks
every claim against retrieved context under norm-v1 + lowercase. The
strategy that fires is recorded as verifier_method for diagnostics.
Entity strategy gates classification via an entity_policy
(strict/hybrid/proximity/drop) so a single proper-noun match no longer
overclaims STRICT — proximity (default) requires a cluster of 3+
verified entities within 300 chars.
Wires into ask() and query(). System prompts now require verbatim
quoted spans for every factual claim, restated via a user-turn
grounding_reminder one message before sources arrive (recent
user-turn instructions outweigh decayed system-turn rules under long
context in Hermes).
providence_cache gains 5 columns (audit_mode, n_quotes, n_verified,
unverified_quotes, verifier_method) with CHECK constraints. A
connect-time _migrate_audit_mode() ALTERs legacy DBs idempotently.
Cache hits return the persisted audit_mode rather than asserting
STRICT unconditionally.
CLI:
emergent list VISUAL/HYBRID records; --aggregate ranks
unverified quotes by frequency (corpus-growth signal).
reclassify re-run the verifier against live providence records;
cold-source records skipped; --dry-run reports
transitions without writing; each change writes one
'providence_reclassify' audit event.
Adds docs/mesh.md and five graphviz diagrams covering the federation
layer: identity stack, epoch state machine, per-member secret envelope,
gossip wire contract, and operator decision tree. Pins the protocol
contract for the upcoming HTTP wire (mesh sync, mesh serve).
Makefile gets a 'docs' target with pattern rule so PNG renders are
incremental from .dot sources.
A snapshot_root is MerkleTree.build([sorted DISTINCT document_roots]).root
— one 32-byte hash naming the entire content-addressed forest at a
point in time. Two peers that ingested the same dump compute bit-
identical snapshot_roots, so cross-machine "are we synced?" becomes
an O(1) hash comparison; the same root doubles as the TF-IDF
scope_root we sketched in the mesh design (snapshot_root *is* a
scope), and pins Q&A answers to a verifiable corpus state.
Schema: one new table, additive over existing data.
snapshots(snapshot_root PK, taken_at, audit_event_hash, doc_count,
parent_snapshot, reason)
+ idx_snapshots_taken_at
API:
compute_snapshot_root(conn, *, document_roots=None) -> (root, count)
create_snapshot(conn, *, reason, parent_snapshot=None) -> dict
verify_snapshot(conn, snapshot_root) -> dict (matches: bool)
diff_against_current(conn, snapshot_root) -> dict (added/removed/unchanged)
list_snapshots(conn, *, limit) -> list[dict]
CLI:
aborist snapshot create [--reason "..."] [--parent <hex>]
aborist snapshot list [--limit N]
aborist snapshot verify <snapshot_root>
aborist snapshot diff <snapshot_root>
Cross-shard: with --shards-dir + --db, the snapshot is computed over
the cluster-wide UNION view but persisted into args.db (a dedicated
snapshots store, conventionally ~/.aborist/shards/snapshots.db).
parent_snapshot auto-links to the latest prior snapshot in the
writer DB, giving a chain for free.
Each snapshot creation writes an audit_event of type 'snapshot_create'
with subject_root=snapshot_root, so the corpus's pinned states are
themselves tamper-evidently logged.
Defect caught + fixed during live test on the 2010 enwiki ingest:
verify and diff disagreed on the same connection because compute used
a list (with cross-shard duplicate document_roots) while diff used a
set. Two shards can land identical document_roots when canonicalize()
maps two structurally-similar pages to the same byte stream — rare
but real. Switched the underlying SQL to SELECT DISTINCT so a
membership snapshot is always order- AND multiplicity-independent.
Live test: 2010-11 enwiki corpus snapshotted at
43797e46605de08dbab06cdcaf5be7ad78243b193c56f8580200dee6bcc7e1b9
doc_count: 3,468,134 (after dedup)
verify + diff round-trip both report identical against current state.
11 new tests covering empty corpus, single-doc degenerate, order
independence, drift detection, audit-chain pinning, parent auto-link,
and idempotent creation on unchanged corpus. 136 tests + 1 skipped.
After the title-search baseline drop, FTS5 BM25 still surfaced short
stub articles ahead of long topical ones for queries with generic
tokens. Intel_4040 (1974 4-bit microcontroller, 1-paragraph stub
mentioning "intel"/"fastest"/"cpu" once each) outscored Intel_Core_2
(real 2010-era CPU article, hundreds of mentions of every query
token) because BM25 normalizes by document length and short docs
that hit every term get an outsized boost.
Add `_rerank_by_body_coverage` between the relevance filter and the
title-overlap rerank. For each surviving candidate (typically 24-32
post-dedup), fetch the full body and sum sqrt(count(t)) over query
tokens, then add `coverage * 0.6` to the score.
Sqrt scaling is intentional: linear sums let enumerative list pages
(e.g. List_of_Intel_microprocessors with 1000+ "intel" mentions)
runaway-dominate, while a hard cap saturates and erases the signal
between short-stub and long-topical articles. Sqrt grows fast enough
to differentiate (5 mentions -> 2.2; 200 mentions -> 14.1) but
sublinearly enough to keep enumerations from overwhelming the merge.
Verified on the 2010-11 enwiki corpus (no distill cores yet):
before: Intel_4040 #1 (43.5), Intel_Core_i7 #6 (39.0)
after: Intel_Core_2 #1 (51.9), Intel_Atom #2 (50.5),
Intel_4040 #3 (49.6)
Body fetch cost is bounded by `over_fetch` (32 candidates per shard,
24-50 unique post-dedup). Each fetch concatenates already-decompressed
chunks via `_load_doc_text` which we use elsewhere in the query path,
so this adds at most one extra DB round-trip per surviving hit.
125 tests still passing.
Two structural defects in the corpus retrieval path were causing
the Intel-CPU query (and any query whose terms hit many article
titles generically) to surface 80486-era legacy articles instead
of contemporary topic articles:
1. _search_titles entries arrived with a 50-point baseline score
(50 + 10*overlap) plus _rerank_by_title doubled the overlap
bonus on top. A single-token title match against a generic
word like "intel" landed at score 70-80 standalone, swamping
FTS5 BM25 hits that scored 5-15. Result: every Intel_*-prefixed
article in the corpus rose to the top of every "intel" query,
regardless of whether the article was actually about a fast CPU.
Drop the 50-point baseline. Title overlap now scores `10*overlap`
from this path, with `_rerank_by_title` adding another `10*overlap`
for any hit (FTS5 or title) whose title overlaps query tokens.
Net effect: title-only-overlap=1 hits score ~10-20, FTS5 hits with
title-overlap score ~25-45, FTS5-only-no-title-overlap score ~5-25.
Body relevance now beats single-token title coincidence.
2. _body_density_passes counted ANY query token's mentions toward
`min_mentions`, so Intel_8086 (with 30 "Intel" mentions and zero
"fastest" or "cpu" mentions) passed the gate for a query of
{intel, fastest, cpu}. Tighten to require breadth: at least
half the query tokens must each appear at least once, AND total
mentions must clear min_mentions. For 1-token queries this is
identical to the prior behavior.
Verified end-to-end on the fresh 2010-11 enwiki corpus (3.47M docs,
no distill cores yet). Pre-fix: top-K all Intel_8086/80286/80386/etc.,
Hermes refused to identify a fastest CPU. Post-fix: Intel Core i7
in top-K, Hermes correctly identifies Core i7-980X Extreme Edition
Gulftown (Intel's 2010 flagship — 32nm, 6 cores, 3.33 GHz, 12 MB L3).
Same 2003-corpus AMD-query result preserved (already had Athlon /
Athlon_XP_3200+ in top-K via FTS5 BM25 + multi-token TF-IDF cores).
125 tests + 1 skipped, no regressions.
Phase 1 of the federation/gossip layer fox sketched as the natural
extension of v9.8 admissibility's content-addressed identity. Two
peers ingesting the same dump already compute identical document_roots
and identical 8-dim cache_keys; the mesh layer is the wire-and-trust
plumbing that lets them dedup answers, exchange Merkle proofs, and
cleanly distrust an evicted member without a hard fork.
Cryptography (cryptography lib, audited):
Ed25519 — every membership mutation + (future) gossip envelope
is signed by the actor's pubkey.
X25519 ECDH — wraps each epoch's symmetric mesh secret to every
current member's DH pubkey via HKDF-derived AEAD key.
ChaCha20-P1305— AEAD for envelope payloads + per-member secret wrap.
State machine:
mesh_identity — singleton; this peer's keys + group name
mesh_roster — per-epoch (member_id, sign_pub, dh_pub, role)
mesh_epochs — epoch_id -> {started_at, started_event_hash,
secret_envelope JSON, reason}
meta:mesh.enabled flag — off by default; gates everything
Eviction works by rotating to a new epoch whose envelope omits the
kicked member. Their prior signatures stay verifiable (the older
roster row is retained), but any gossip from epoch+1 onward is
opaque to them — the secret was never shared with their pubkey.
Authority gate: only roster members with role='admin' can add or
kick. Self-kick is rejected explicitly. The last admin can't be
kicked. Schedule-rotate (refresh secret, no roster change) is open
to any current member as a session-hygiene op.
Audit-chain integration: every mesh state mutation writes an audit
event (mesh_init, mesh_enable/disable, mesh_epoch_rotate). The
epoch's started_event_hash backfills into mesh_epochs after the
audit row commits, giving each epoch a tamper-evident pin into the
ledger.
CLI subcommands: mesh init, mesh status, mesh enable, mesh disable,
mesh members, mesh add, mesh kick, mesh rotate. All read-only or
local-state-only — no network code paths in this commit.
The HTTP gossip wire (`mesh sync`, `mesh serve`) is the next phase.
Schema, cryptography, and roster state machine are all in place to
support it without further migration.
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`,
yielding one Document per text file. URI shape `git://<repo>/file/<path>`
intentionally omits the commit hash — re-ingest after new commits produces
fresh document_roots that aborist's prior-doc detection chains via
`supersedes` edges, so the audit trail and Merkle tree grow as the repo
grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files
filtered out by default. Commit hash + timestamp + subject ride along in
`extra` (informational only; not part of the Merkle commitment).
MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes
semantics, same shape.
Makefile adds:
make ingest-self (this repo -> aborist-self.db)
make ingest-git GIT_REPO=/path/to/repo (arbitrary git clone)
make ingest-hg HG_REPO=/path/to/repo (mercurial)
Each lands in its own shard file alongside existing shards/grok.db,
keeping per-shard write paths independent of the wikipedia 4-way ingest's
WAL writer lock.
Adds a new section after the 2003 Wikipedia / distillation block that
covers Phase IV (2006+) XML dumps: fetch-xml / ingest-xml-attached /
ingest-abstract targets, the WP_XML_YEAR/MONTH/DATE knobs for picking
any dated snapshot in dumps.wikimedia.org/archive, plus a pointer at
the abstract.xml feed as a cheap-coverage alternative to the full
pages-articles dump (~5-10 GB instead of ~95 GB on disk for enwiki
2010-11). Existing 2003 section retitled to flag the format split.
Three cheats stack to drop on-disk store from ~21 KB to ~6.7 KB per doc on
the 2003 enwiki cur corpus (-67% measured, apples-to-apples reingest with
identical document/edge counts; Merkle proofs round-trip 30/30):
1. zstd-compressed chunks.content (level 3). Magic-byte detection on read
means legacy plaintext rows pass through unchanged. Cores stay plaintext
so qa.query._docs_with_core_keyword_match's SQL LOWER+LIKE keeps working.
2. edges WITHOUT ROWID. The composite PK (src_root, edge_type, dst_root,
dst_uri, anchor) covers every column, so a default rowid-based table
near-doubles row data in the PK index. WITHOUT ROWID makes the table
itself the B-tree. Drops idx_edges_dst_uri too — the only query that
filters on dst_uri alone is gravity_top_inbound, a one-shot analytic.
3. contentless FTS5 (content='', contentless_delete=1) eliminates the
28 MB / 1000 docs of duplicated chunk text the old chunks_fts stored.
chunks gets an explicit chunk_id INTEGER PRIMARY KEY so the FTS5
rowid maps back to chunks.chunk_id at search time. Snippets are
built in Python (search/fts5.py:_build_snippet) since SQL snippet()
returns empty in contentless mode.
TF-IDF retrieval also fixed: the prior LIKE '%intel%' substring match
let "intelligence", "intellectual", "intellivision" drown real hits like
Pentium_4 (whose TF-IDF core has "intel" as an exact keyword). Now uses
word-boundary `LIKE '%, intel, %'` patterns plus a match_count over the
distinct query tokens — multi-token coverage outranks single-token title
boosts. Pentium_4 surfaces #1 for "what is the fastest intel CPU?" with
the canonical 2003 answer (Pentium 4 3.20 GHz) instead of an empty
"insufficient sources" reply.
Schema-level changes affect new DBs only; existing v9.8 DBs keep
working at the old layout. Cross-shard UNION views explicitly list the
intersection of columns so a mixed cluster (legacy + new schema shards
in one --shards-dir) still unions cleanly.
Drops in for any dated snapshot in dumps.wikimedia.org/archive — pages-articles.xml.bz2
and pages-meta-current.xml.bz2 (cur snapshot, single-revision-per-page) plus
pages-meta-history.xml.bz2 (multi_revision=True). abstract.xml feed yields
pre-distilled summaries at ~1/100th the chunk volume. Streams .bz2/.gz directly
via iterparse with bounded memory; same shard / resume contract as the SQL
source. Falls back to title-prefix namespace filtering when older export
schemas omit the per-page <ns> element (e.g. enwiki 20101011).
Makefile defaults target enwiki 20101011 (6.2 GB). Override with WP_XML_YEAR /
WP_XML_MONTH / WP_XML_DATE / WP_XML_LANG to fetch any other archived snapshot.
Replace the boilerplate AGPL-3.0 header in LICENSE with the exact
license clause from the Merkle Providence Reverse RAG whitepaper
(April 2026, Russell Ballestrini / David Wong / Riley Morgan):
License: AGPL-3.0-only · This algorithm, its implementation, &
all associated code carry the GNU Affero General Public License
v3.0 (only). You may use, modify, & distribute under those terms.
No proprietary relicensing exists.
README's License section quotes the same clause and links the
whitepaper as the canonical source.
Source: https://unfirehose.com/downloads/merkle-reverse-rag-whitepaper.pdf
Adds a "Get the source" section with the GitLab SSH + HTTPS clone
URLs, then platform-specific prerequisite blocks:
- macOS: Xcode CLT for git/make, Homebrew for python@3.12
- Ubuntu/Debian: apt install python3/python3-venv/build-essential
/curl/bzip2; 22.04 has 3.10, 24.04 has 3.12
- Windows: WSL2 Ubuntu-24.04 is the supported path. Native cmd
+ Git Bash mostly works for Python but several make targets call
bash idioms. `wsl --install -d Ubuntu-24.04` from an admin shell.
- OpenBSD: pkg_add git python-3.12 gmake curl. Default `make` is
BSD make; the project uses GNU-make features so substitute
`gmake` everywhere.
Bootstrap step (make bootstrap) is unchanged — it stays at the end
of the Setup section once the platform prereqs are in place.
aborist/sources/grok.py exposes two Source subclasses for ingesting
xAI's user data export:
GrokExportSource walks ttl/<period>/export_data/<user-id>/ and
reads prod-grok-backend.json. Yields one Document
per conversation with:
- URI: grok://conversation/<conversation-id>
- title: the conversation's auto-generated title
- content: full message text in turn order
source_type='grok_export'.
GrokMediaPostsSource same export, but yields per media-generation
post (image/video prompts) under URI
grok://media/<post-id>.
source_type='grok_media'.
Both auto-walk down from the export root so callers can pass the
top-level directory xAI delivered (e.g., ~/Downloads/<user-uuid>/).
Wired into the CLI: aborist ingest --source {grok_export,grok_media}
accepts --path <export-root>. tests/test_grok_source.py covers the
walk + parse + Document shape with a fabricated mini-export fixture.
Once ingested, conversations become normal queryable docs in the
shard cluster — your prior chats become memory the corpus can
consult during RAG.
79 tests passing.
Replaces the stub README with a working user guide:
- 30-second quickstart that lands a queryable corpus in ~3 min
- Setup section (Python 3.10+, make bootstrap creates the venv +
installs the package in editable mode)
- Wikipedia 2003-05-16 dataset:
fetch-cur / fetch-old / fetch what each downloads & where
ingest-cur-attached SHARDS=N sharded ingest, the canonical
path (no WAL contention)
ingest-cur-parallel / ingest-cur single-DB variants
--resume rsync-style restart
- Distillation (first-sentence + TF-IDF cores) and what they buy
- Personal Grok export ingest (point GROK_EXPORT at the xAI bundle)
- OpenAI export placeholder section (one new Source subclass; the
pattern matches sources/grok.py)
- Asking the corpus: query path explained briefly, 8-dim cache key,
per-phase timings, LLM endpoint env vars
- Falsification: how to mark a wrong cached answer stale
- Inspection: stats-shards / analyze-shards / verify-shards / activity
- Architecture tree (one screen)
- Pointer to source papers (whitepaper + v7 substrate)
Reflects the current shipped feature set including the three
retrieval relevance signals (title + TF-IDF cores + body density),
concept overlay, sharded distill, falsification, activity timeline,
and per-phase timings.
The "permacomputer" query fell into title-relevance's empty-fallback
because "permacomputer" appears nowhere in any title. Fallback was
top-1 by raw BM25, getting one tangential source. Same problem for
any user-coined term: title-LIKE alone can't find it.
_filter_by_title_relevance now grows three accept paths:
1. Title-token overlap (existing).
2. TF-IDF core keyword overlap. A doc's distinctive low-frequency
terms get distilled into its tfidf-keywords-v1 core. A query for a
rare term ("permacomputer", "unturf") hits docs whose core lists
it as a keyword even when no title does.
3. Body density. Docs that mention a query token >= 3 times pass
regardless of title or core. Cheap proxy for "actually about it."
Empty-all-three fallback widens from top-1 to top-N (default 5) — the
LLM gets enough context to honestly say "I don't know" rather than
fabricating from a single tangential source.
Implementation:
- _docs_with_core_keyword_match: SQL join chunks -> documents -> derivations
finds source docs whose TF-IDF core content matches a token. Folded
into the per-shard search loop alongside FTS5 + title-LIKE.
- _body_density_passes: counts case-insensitive occurrences in
concatenated chunk content. Threshold 3.
- _search_corpus stashes core_match_roots + root_to_shard on the
returned hits list as side-channel info for _rerank.
Makefile: distill-shards-tfidf-parallel for the per-shard TF-IDF run.
Demo on the live cluster (after TF-IDF distill of 95k cur surface docs):
Q "tell me about permacomputer ?"
sources: MOAD Cheat Sheet + Unturf AI Infrastructure Expansion
answer: synthesized from both, bounds coverage honestly
Q "what is unturf about ?"
sources: Unturf Trademark + Unturf AI Infrastructure Expansion
answer: "grassroots, community-driven initiative ... ethical AI
hardware, software, and networks"
Both queries now find user-coined terms via TF-IDF cores even though
the terms never appear in any title.
79 tests passing.
Two issues surfaced by "tell me about permacomputer ?":
1. Stopword leak. _FTS5_STOPWORDS got "tell" added earlier but the
parallel _TITLE_STOPWORDS in qa/query.py wasn't synced. Title-LIKE
search picked up "tell" and returned every Tell_* article in the
corpus (Tell_(poker), Tell-Tale_Heart, Tell_Township, Tell_City,
etc.) all uniformly scored at 70.0. Synced the lists — title
tokenizer now drops the same question-shaping verbs as FTS5.
2. Grok docs lived only in --db (default ~/.aborist/aborist.db),
not in --shards-dir. `make query` reads the shard cluster, so
Grok was invisible to it. Added two targets:
ingest-grok-attached conversations -> $(SHARDS_DIR)/grok.db
ingest-grok-media-attached media prompts -> $(SHARDS_DIR)/grok.db
Both use --resume so re-runs are cheap.
Demo: "tell me about permacomputer ?" now answers from the
"MOAD Cheat Sheet: Common Software Defects" Grok conversation —
fox's own prior chat surfaced as corpus.
Note: 000.db already had Grok docs from an earlier ingest variant.
The cross-shard UNION currently sees them twice (376 rows, 188
unique document_roots). Retrieval is unaffected because
_search_corpus dedupes by root, but stats counts are inflated.
Cleanup is destructive (drop one copy) — fox decides which.
79 tests passing.
Even with rivalry exclusion, sources like 'CPU_monthly' (a roundup
discussing both AMD and Intel CPUs in the same article) sneak through
because their title matches "CPU" without naming a competitor. Hermes
then conflated facts: "the fastest AMD CPU is the 3.2 GHz Pentium 4."
Tightened the DEFAULT_QUERY_POLICY system prompt with four explicit
rules:
1. A single source may discuss multiple competitors. Attribute facts
only to the entity the source explicitly names.
2. If the question asks about ONE entity (e.g., "fastest AMD CPU"),
do not include competitor facts as if they were facts about the
asked entity.
3. AMD PR-rating numbers (3200+, 2500+) are NOT clock speeds in MHz.
Quote the source's own wording rather than interpreting model
numbers.
4. Say "I don't know based on the provided sources" when the sources
don't cover the answer. Don't speculate.
Demo on the same AMD query (post-distill, cores now in retrieval):
before: "the fastest AMD CPU is the 3.2 GHz Pentium 4 from May 2003"
after: "the fastest AMD CPU mentioned is the Athlon XP 3200+ which
was released in May 2003. The sources do not contain
information about CPUs released after 2003."
Sources include AMD, AMD_5x86, AMD_Duron + their cores, CPU_design,
CPU_monthly. Hermes correctly attributes Athlon XP 3200+ to AMD and
honestly bounds the corpus's coverage at 2003.
Bumping the system prompt invalidates the governance_policy_hash and
therefore every prior cache_key — by design, per v9.8 admissibility
(prompt change = different governance = fresh inference).
66 tests still passing.
aborist [--shards-dir DIR] providence --falsify CACHE_KEY \
--state {failed,stale,quarantined} --reason "X" --by-actor NAME
Searches every shard for the cache_key (it lives in exactly one),
flips falsification_state on the providence_cache row, appends a row
to the falsifications log, and writes a 'falsify' audit event so the
chain captures the act with from_state/to_state/reason/by_actor.
Future cache lookups skip records whose state != 'live', so a
falsified answer never gets reused. The record itself is preserved
(history matters); only its admissibility changes.
Makefile target:
make falsify KEY=<hex> REASON='why' [STATE=failed]
Demo on the live shard cluster: marked the AMD/Pentium-4 conflation
answer as failed, verified state flipped + audit row recorded +
falsifications log captured fox as actor + the Barton-clock-vs-PR
explanation as reason. Re-asking the same question generated a fresh
cache_key (corpus state shifted between attempts) so a new LLM call
ran — exactly the v9.8 admissibility behavior: each (8-dim cache_key)
is independently live/failed.
66 tests still passing.
New CLI: aborist [--shards-dir DIR] activity [--limit N] [--since-seconds S]
[--preview-chars N]
Aggregates across all shards into a single JSON document with:
shards the SQLite files queried
corpus counts: documents_total/surface/core,
providence_total/live/stale/failed,
audit_events_total
recent_qa providence_cache rows (cache_key, question, answer
preview, hit_count, state, last_hit_at) newest first
recent_ingests audit events of type=ingest with body's document_uri,
source_type, chunks, supersedes link
recent_derives audit events of type=derive (distillation results,
compression_ratio, compression_depth)
recent_falsifications records marked failed/stale/quarantined,
with reason and by_actor
--limit caps each list independently. --since-seconds filters to recent
events only. --preview-chars trims long answers.
Designed for an agent to read before deciding the next action — what
was just asked, what just got integrated, what's the current state.
Makefile target: make activity ACTIVITY_LIMIT=N
66 tests still passing.
Cache speedup was previously only visible by running 'time' externally.
Now every query/ask result includes a timings dict so the breakdown
shows up in the JSON itself:
search_ms FTS5 + title-LIKE across shards
context_ms load doc text into context window
cache_lookup_ms primary-key hit on providence_cache
llm_ms Hermes-3 round-trip (null on cache hit)
persist_ms insert record + chain audit event (null on cache hit)
total_ms end-to-end wall
Demo against the live shard cluster:
Q "What is photosynthesis?" (cache miss):
search 84.5 context 0.9 cache 0.1 llm 3983.3 persist 36.4
total 4106.1 ms
Q "What is photosynthesis?" (cache hit, same key):
search 78.5 context 0.9 cache 0.1
llm null persist null
total 85.7 ms → 48x speedup
The cache lookup itself is 0.1 ms (PK on cache_key). The 85 ms hit
baseline is the cost of re-computing the 8-dim cache_key — context_root
depends on retrieval results, so search runs even on hits. A secondary
question-only cache could collapse this if needed, but the v9.8
admissibility model wants context_root in the key.
Same shape for ask (per-doc Q&A).
66 tests still passing.
A small knowledge-graph layer that does two distinct jobs:
1. SYNONYM_GROUPS broaden retrieval. A query mentioning "Athlon"
now also matches AMD-titled docs because Athlon IS an AMD product.
Groups currently cover AMD-family, Intel-family, HTTP family,
FTP, Mac, Windows, Linux. Easy to extend.
2. RIVALRIES narrow retrieval. The pair (AMD-group, Intel-group)
means: if the query mentions one side and not the other, drop
docs whose titles contain the OTHER side's tokens. So a "fastest
AMD CPU" question never gets Pentium_4 in the context — even if
FTS5 BM25 ranks it high — because Pentium is in the Intel group
and Intel isn't in the query.
COMPARE_WORDS ("vs", "versus", "compare", "between", ...) suppress
the exclusion. "compare AMD vs Intel" keeps both sides. "what is
the fastest AMD CPU?" does not.
Demos against the 128k Wikipedia 2003-05-16 cur shards:
Q "fastest AMD CPU" → 8 sources, all AMD/CPU titled, NO Intel
Answer: "Athlon XP 3200+" (real AMD chip,
grounded in the AMD article)
Q "compare AMD vs Intel"
→ 8 sources, mix of Intel_8028x, Intel_8048x,
AMD_Duron — both sides preserved
Q "Athlon processor" → AMD, AMD_Duron, AMD_5x86, Athlon all
surfaced via synonym expansion
Phase 1 implementation hand-curates the groups; Phase 2 idea is to
derive them from Wikipedia's link/category graph (dense bidirectional
clusters → synonym groups; same-category-without-cross-links →
rivalry candidates).
66 tests passing.
The "what is HTTP?" query returned only Main_Page/Temp because BM25
over chunk content rated URL-heavy list pages (List_of_planned_cities,
Mary_Kay_Letourneau) above the actual HTTP protocol article. Many
http:// links in body trumped the topic article that names "HTTP" once
in title and a few times in body.
Same shape on AMD/Stoicism/etc. when the topic article is short or
when other articles incidentally repeat the query token in body text.
Fix: parallel SQL `LOWER(title) LIKE` search per shard alongside FTS5.
Title hits get a 50-baseline + 10-per-overlap score (well above body
BM25 scores), so an article whose title literally is the topic rises
to the top regardless of body term frequency.
After:
Q "what is HTTP?" → HTTP, Apache_HTTP_Server, HTTP_cookie
Hermes pulls the protocol definition
directly from the HTTP article body.
Q "fastest AMD CPU?" → AMD, AMD_Duron, CPU_design, ...
Answer: Athlon 1400 MHz on Thunderbird,
grounded in the AMD article.
Q "Athlon processor" → Athlon ranks in top sources, answer
correctly attributes to AMD.
The title-relevance filter still applies post-merge: hits without
title-token overlap are dropped if any title-matchers exist. So
poison sources (Pentium_4 in AMD queries) stay out of the context
window.
57 tests passing.
The query "what is the fastest AMD CPU?" was returning Pentium_4 as
a top source, and Hermes hallucinated by conflating Intel content
with AMD context. Two compounding causes:
1. FTS5 was ORing all content tokens, so any doc with "fastest" OR
"CPU" matched. BM25 then ranked Pentium_4 highly because the
article is long and discusses CPU clock speed in detail.
2. The reranker only boosted titles that matched — it never
penalized titles that didn't match. Pentium_4 had no penalty
despite being topically wrong.
Two fixes:
- FTS5 escape now ANDs content tokens by default. Stopword list
expanded to include question-shaping verbs (tell/show/describe/
explain/find) so they don't sneak in as required tokens. If AND
returns zero hits, falls back to OR — pragmatic for fuzzy queries
the user phrases differently than the article wording. All-stopword
inputs still produce a non-empty MATCH so FTS5 doesn't error.
- _filter_by_title_relevance hard-drops hits whose title shares no
token with the query, before assembling context. If filtering
would leave fewer than 3 hits, falls back to the unfiltered set
(corpus might genuinely not have a topical match — let the LLM
say "I don't know" rather than nothing).
Together: AND-mode requires every content token in the body, and
title filter requires the doc to actually be about the topic. Pentium
articles drop out of AMD queries; Tell_(poker) drops out of plot
queries.
57 tests still passing.
The optimized parser silently dropped articles whose body contained an
HTML decimal entity (&#NNNNN;) — affecting roughly any Wikipedia article
referencing a Japanese/Korean/Chinese interwiki link. ~6,000 docs missing
from the 128k cur ingest. The HTTP article was one casualty.
Root cause: when the scanner ran out of pending data while inside a
quoted string, flush_complete() returned with in_string=True. The outer
read-loop trimmed pending and called flush_complete() again, which RESET
i=0 and searched for the next "'" from the start of the buffer. The
first quote found was the title-field open quote of the article we were
already deep inside; bs=0 (no preceding backslash) so the parser
incorrectly flipped in_string=False and treated the next ';' (typically
inside an &#NNNNN; entity) as a statement terminator.
Fix: persist scan_pos across read() chunks. After a trim, remap
scan_pos = scan_pos - yield_from. The next call resumes scanning at the
correct position rather than restarting at 0.
Verified: optimized parser now yields identical statements to the
canonical char-by-char reference on the first 60 MB of the dump. HTTP
article correctly emitted at doc #7722 with full 2375-char body.
Cost: ~50% slower on a 5000-doc benchmark (11.8s vs broken 7.8s) due
to the correct cross-chunk state tracking. Worth the trade — a corpus
that randomly drops articles is worse than a slightly slower complete
one. Still ~3x faster than the original char-by-char parser.
The existing 122k cur shard set is missing the same ~6k articles. To
recover: rm -rf ~/.aborist/shards && make ingest-cur-attached SHARDS=4
takes ~3-4 min wall.
57 tests still passing.
Two issues surfaced by the user query
"tell me about back to the future plot?":
1. Mojibake in titles like 'La_Jet\\ufffde'.
The 2003 MediaWiki dumps mix Latin-1 raw bytes (e.g., 0xE9 for é)
with HTML entities. Opening the bz2 as utf-8 with errors="replace"
substitutes U+FFFD for any high byte. Latin-1 is lossless on every
byte and gives the right code point for the raw-byte cases. New
default for WikipediaSqlDump; modern dumps can pass encoding="utf-8"
explicitly.
The existing 122k cur shards keep their slightly-mangled titles
on a handful of articles — but a re-ingest from scratch (now ~2.5
min thanks to the parallel attached path) recovers them, and
--resume protects against starting over from zero.
2. Tell_(poker) outranking Back_to_the_Future for "tell me about
back to the future plot?" — BM25 on body text alone favors
short docs with rare body tokens. The title is a much stronger
topical signal.
New _rerank_by_title in qa/query.py: hits whose title contains
query tokens get +8 per match. After rerank on this query:
1. Back_to_the_Future 31.74 (was 15.74, was rank 3)
2. Back_to_the_Future_Part_III 27.83
3. Back_to_the_Future_Part_II 27.63
4. Tell_(poker) 23.41 (was 15.41, was rank 6)
5. Gunpowder_Plot 19.57
Hermes now produces a real plot summary because the canonical
Back_to_the_Future article is the lead source.
57 tests still passing.
Per CLAUDE.md "every workflow lives behind a make target," `aborist`
isn't on PATH outside the venv. Add `make query Q="<question>"` and
`make query-dry Q="..."` so the user-facing entry point is a make
invocation, not a venv path.
make query Q="What is anarcho-capitalism?"
make query-dry Q="..." # skip the LLM, return context only
Override QUERY_TOP_K=N to widen/narrow the source set.
aborist/qa/query.py exposes query() — the user-facing RAG flow:
1. FTS5 search across all shards (chunks_fts can't be UNION'd as a
view, so each shard's index is queried independently and merged
by score).
2. Top-K distinct documents are selected within a max-context-chars
budget (default 60 KB so a 768-token response fits Hermes-3's
82 K context window comfortably).
3. context_root = Merkle root over the sorted source document_roots.
That's the v9.8 'source' dimension for multi-source answers —
a verifier can recompute it from the listed source roots.
4. 8-dim cache_key over (context_root, question_hash, model_profile,
conversation, governance_policy, schema, canonicalization,
chunking). Hit returns STRICT immediately; miss calls Hermes and
persists.
CLI: aborist [--shards-dir DIR] query "<question>"
Default qa_db is <shards-dir>/qa.db (or ~/.aborist/qa.db). Uses the
same OpenAICompatibleClient/StubClient as `ask`. --dry-run skips the
LLM and returns context-only.
Search escape fix: the prior FTS5 escape ANDed every token including
stopwords + punctuation, so "What is anarcho-capitalism?" required
the doc to literally contain "what" + "is" + "anarcho-capitalism?" —
zero hits. New tokenizer drops stopwords + punctuation and ORs the
remaining content tokens; BM25 ranks the multi-token matches highest.
Live demo against the 122k-doc 4-shard cluster:
Q "What is anarcho-capitalism?" 6.1 s wall miss / 0.45 s cache hit
Q "Who was George Washington?" 10.3 s wall miss
Both answers cite the source URIs Hermes was given.
57 tests passing (4 new query tests covering search → context →
cache → audit chain).
aborist/progress.py — stdlib-only rate-limited stderr reporter.
Periodic lines (default every 2s) showing:
elapsed | seen | inserted | docs/s now | docs/s avg | percent | ETA
Wired into ingest_source via a `progress` parameter; CLI default-on
with --quiet to suppress and --total-estimate N to enable percent/ETA.
Demo on a 5k-doc ingest with --progress-interval 0.8:
[ 1s] | 200 seen | 200 new | 193 now | ( 193 avg) docs/s | 4.0% | ETA 24s
[ 7s] | 3,200 seen | 3,200 new | 458 now | ( 424 avg) docs/s | 64.0% | ETA 3s
[ 12s] | 5,001 seen | 5,000 new | 392 now | ( 413 avg) docs/s | 100.0% | ETA 0s
bench/run.sh + `make bench` — reproducible 5000-doc workload through
three configs:
serial single process, single SQLite
parallel-shared N shards, one shared SQLite (WAL serialized)
attached N shards, per-shard SQLite (true parallel writes)
Output is a one-shot table plus CSV at /tmp/aborist-bench/results.csv
so the ratchet is visible as we keep optimizing. Override via
BENCH_DOCS=N and SHARDS=N.
Latest baseline (this commit, on this machine):
config wall_s docs docs/s
serial 11.74 5000 425.7
parallel-shared 10.39 5000 481.1
attached 7.99 5000 625.5
53 tests still passing.
Resume (rsync-style)
Each shard DB gains a `meta` table. After every successful batch
flush, ingest_source persists `source_high_water:<source_type>` ->
the largest row id seen (cur_id or old_id). On --resume, the source
reads it back and skips rows whose id is <= the mark, so an ingest
killed at any point can be re-run cheaply: already-cached docs are
fast-forwarded past without re-hashing or DB writes.
WikipediaSqlDump now exposes `start_id` (skip threshold) and
`last_id` (running max). cur_id is surfaced in Document.extra
alongside old_id so both tables behave the same.
CLI: aborist ingest --resume
Demo on shared DB (cur):
Round 1: --limit 3000 --resume high_water = 5714
Round 2: --limit 5000 --resume skips 1..5714, picks up at 5715
Round 3: --limit 100 --resume skips 1..15362, ingests 100 more
Always idempotent on re-run; no dups, no missing docs.
Per-shard audit chain integrity for sharded analyze
Cross-shard analyze previously reported nonsense breaks counts —
each shard owns its own audit chain (genesis -> ... -> latest), and
the UNION view interleaves them so cross-shard transitions look
like break events. The fix: when --shards-dir is set, open each
shard's DB directly and run _check_audit_chain on it, then
aggregate.
Output now reads:
"audit_chain": {
"events": <total>,
"breaks": 0,
"shards": [{"shard": "000.db", "events": N, "breaks": 0}, ...]
}
53 tests passing. Tests cover: high-water write, skip-on-resume,
idempotency across two resume runs, kill-and-resume continuity.
aborist --shards-dir DIR distill iterates over each shard's DB and
runs distill_existing on it. Cores live in the same shard as their
source surface so the per-shard derivation chain stays
self-contained.
Per-shard processing also enables trivial parallelism via the
distill-shards-parallel Makefile target — one process per shard, no
DB contention because each shard is its own file.
End-to-end demo on 4-way 10k cur ingest:
ingest: 17.6 s wall (4 parallel)
distill: 6.2 s sequential per-shard (9185 cores from 10000 surface)
verify: 30/30 random Merkle proofs across shards
Aggregated counters cover all shards plus per-shard breakdown in the
output JSON so the user sees both the union and the parts.
Note: cross-shard analyze reports an inflated audit_chain.breaks count
because each shard owns its own chain genesis -> ... -> latest, but the
UNION view sees them interleaved. Fix coming next.
Each ingest shard owns its own SQLite file. N processes write in
true parallel (no WAL writer-lock contention). Cross-shard reads
attach all shards as UNION ALL views over the standard tables, so
existing query code (`SELECT * FROM documents`) works unchanged.
CLI:
aborist ingest --shards-dir DIR --shard 0/4 # write to DIR/000.db
aborist --shards-dir DIR stats # read across all shards
aborist --shards-dir DIR analyze # ditto
aborist --shards-dir DIR verify -n 20 # Merkle round-trips
# on a sample drawn
# from all shards
Makefile:
SHARDS_DIR=~/.aborist/shards
ingest-cur-attached SHARDS=4 ingest-old-attached SHARDS=4
stats-shards cross-shard stats
Implementation lives in store.connect_query():
- opens a fresh in-memory connection
- ATTACH DATABASE per shard file under aliases sh000, sh001, ...
- CREATE TEMP VIEW <table> AS UNION ALL SELECT * FROM <alias>.<table>
for the eight shardable tables (documents, chunks, merkle_nodes,
edges, derivations, providence_cache, audit_events, falsifications)
- exposes a _shards temp table for tools that want the shard list
Benchmark (20k cur, 4 cores):
Phase 1a serial 55.3 s
Phase 1b shared-WAL 43.1 s
Phase 2 attached-forever 40.1 s (no WAL contention)
The 7% gap vs Phase 1b is bz2-decompression duplication: each shard
decodes the full dump independently. Architectural wins are bigger
than that number — write isolation, per-shard crash recovery,
audit chain stays per-shard checkable, shards can migrate to other
disks, new shards are pure append (never rewrite existing files).
Merge cost remains: zero. `ls $SHARDS_DIR/*.db` is the registry.
Workers each parse the full SQL stream but only emit rows whose index
satisfies (idx % total == rank). N processes cooperate against a
single shared SQLite file; WAL+synchronous=NORMAL handles writer
contention via BEGIN IMMEDIATE serialization.
Implementation chose OS-level processes over in-process mp.Queue: the
gain ceiling is identical (writer is the bottleneck either way) and
the surface area is one new flag plus stride-filter logic instead of
a queue-based pipeline with picklable workers.
Makefile gains:
ingest-cur-parallel SHARDS=4
ingest-old-parallel SHARDS=4
Benchmark on 20k cur articles:
serial 55.3 s 361 docs/s 1.0x CPU usage
4-way parallel 43.1 s 464 docs/s ~3.0x CPU usage 1.28x wall
Both runs landed 19,999 unique documents — correctness verified.
The write-lock is now the bottleneck; further gains require Phase 2
attach-forever sharding (each shard owns its own SQLite file, queries
fan out via ATTACH DATABASE, zero merge cost).
Previous batching opened one transaction per N=200 docs but still ran
~200 execute()/executemany() calls per doc inside it. Profile after
the parser rewrite showed the prior-URI SELECT alone was 648 ms /
2000 docs.
Refactor accumulates ALL inserts across the batch into per-table
mega-lists, then flushes via a single executemany per table:
- documents one executemany for the whole batch
- chunks / chunks_fts one executemany each
- merkle_nodes one executemany
- edges one executemany (collected across all docs)
- audit_events one executemany; chain hashes computed in pure
Python via store.chain_audit_events (no DB
round-trip per event)
Prior-URI resolution becomes one IN-clause SELECT for all distinct
URIs in the batch, plus an in-Python dict carrying within-batch chain
state so two same-URI revisions in the same batch correctly form
v_old <- v_mid <- v_new supersedes edges.
Within-batch document_root collisions (two source rows with identical
content) are deduped via inserted_this_batch set — required because
the mega-INSERT would otherwise hit the documents PK constraint.
5000 wikipedia_old revisions: 12.8 s -> 7.8 s (~64% faster). 49 tests
still passing.
The stub from `git init` predated everything. Replace with a concise
working map: what aborist IS now, where the source papers live, what
the schema invariants are, what conventions never to silently change
(merkle ordering, chunker default = chunking_version bump = mass cache
falsification), where the live endpoints are, and the parser hot-path
gotchas.
Adds: source-paper inventory, architecture tree, schema invariants
(8-dim cache key, audit chain integrity, surface/core kinds, hot/warm/
cold tiers), Merkle conventions, live endpoint URLs, hot-path
performance notes, orientation protocol that runs `make test`,
`stats`, and `analyze`.
Preserves: blackops identity rules, no Co-Authored-By, fail-closed,
PYTHONUNBUFFERED for long-running processes.
Profile of the prior implementation (cProfile on 2000-doc ingest):
total 23.6s
parser 20.1s (85% of time, 27.7M list.append calls)
DB writes 2.9s
Three changes compound:
1. PRAGMA pack in store.connect():
synchronous=NORMAL skip per-commit fsync (durable under WAL replay)
cache_size=-65536 64 MB page cache
temp_store=MEMORY temp tables in RAM
mmap_size=256 MB page-cache reads bypass read() syscall
2. Batched transactions in ingest_source and distill_existing:
200 docs/tx default, --batch-size CLI flag.
One BEGIN/COMMIT per batch instead of per doc.
Audit chain SELECT-then-INSERT still chains correctly because
SQLite reads its own uncommitted writes within a transaction.
Idempotent on collision (INSERT OR IGNORE / pre-existence check)
so partial-batch re-runs are safe.
3. Parser rewrite (the actual bottleneck): replaced char-by-char
list-append loops with str.find / string slicing.
_decode_mysql_string fast-path: if no '\\\\' in input, return as-is.
_split_values_tuples uses _find_string_end (str.find with
backslash-parity check) instead of per-char scanning.
_iter_insert_statements buffered scan (str.find for ';' and "'")
crossing read-chunk boundaries via in_string flag.
~13x fewer Python function calls, semantics identical.
_upsert_edges now batches via executemany.
After: 2000 docs in 6.0s; 5000 old revisions in 12.8s wall (was 32.3s).
Throughput 85 -> 333 docs/sec. 49 tests still passing.
Cheaper to fail = cheaper to experiment.
analyze emits a compression-spectrum report:
compression_depth_histogram surface vs depth=1, 2, ...
distillers per-process compression ratio mean/min/max
documents_by_source_kind source/kind crosstab
chunks_by_tier hot/warm/cold distribution
audit_chain event count + integrity (breaks must be 0)
gravity_top_inbound the 'gravity wells' of the forest
Default LLM endpoint moves from uncloseai.com (marketing site, no /v1)
to hermes.ai.unturf.com/v1 — the actual reachable Hermes-3 inference
endpoint serving adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic on vLLM
with an 82K context window. uncloseai.com still works as a brand
override via ABORIST_LLM_ENDPOINT.
The 2003-05-16 archive ships three files:
20030516_cur_tablesql.bz2 82 MB current snapshot (single revision/page)
old_tablesqlbz2.1 640 MiB \
old_tablesqlbz2.2 252 MiB / split halves of old (full revision
history). Concatenate before bzcat.
Generalize the parser:
WikipediaSqlDump(table='cur'|'old') — shared statement parser, single
column-position contract for the
first 4 fields (id/ns/title/text)
WikipediaCurDump — back-compat wrapper, table='cur'
WikipediaOldDump — new, table='old'; old has no is_redirect, every
revision is real
Old rows surface old_id and old_timestamp via Document.extra so a
downstream pass can sort revisions chronologically before re-ingesting
through the supersedes-edge path.
Makefile gains:
fetch-cur / fetch-old / fetch (both)
ingest-cur / ingest-old / ingest (cur default)
WP_OLD target concatenates the two split parts
CLI ingest --source now accepts wikipedia_cur or wikipedia_old.
Smoke (real dump): 5 revisions of "AtlasShrugged/Companies" yielded
correctly with old_id=2..10, timestamps from January 2002.
aborist/qa/ implements the cache-first answer flow from the providence
whitepaper, scaled up to v9.8's full 8-dim admissibility invariant.
cache_key = SHA-256 of:
source_root | question_hash | model_profile_hash | conversation_hash
| governance_policy_hash | schema_version | canonicalization_version
| chunking_version
Any drift in any dimension yields a distinct cache_key — prior records
cannot serve. Falsification states (failed/stale/quarantined) gate
every cache hit.
- qa/keys.py — pure hash functions, deterministic & testable
- qa/client.py — ChatClient Protocol + StubClient + OpenAI-compatible
HTTP client (vllm/llama.cpp/uncloseai compatible)
- qa/runner.py — ask(): lookup -> hit (no LLM call, hit_count++)
OR miss (call client, write record, audit event,
proof binds answer to source root)
- cli.py — `aborist ask` and `aborist providence` subcommands
- pyproject.toml — httpx promoted from extras to core (used by both
html and qa); selectolax stays in [html] extras
Smoke (StubClient, no network): cache miss writes record with chunk_0
Merkle proof reconstructing source_root; cache hit returns same record
without calling client; 1085 audit events chained 0 breaks across
ingest/derive/evict/rehydrate/providence_write.
When a re-ingest produces a different document_root for an already-seen
URI, both versions now coexist in the store. A 'supersedes' edge from
the new doc to the prior one keeps the lineage addressable, and the
ingest audit body records the supersedes link.
Idempotent re-ingest is unchanged: identical bytes -> identical root ->
no new doc, no supersedes edge.
Three-version chains test that v3->v2 and v2->v1 edges form a walkable
history. The old version stays queryable (its chunks may later be
evicted to cold for storage savings while remaining provable).
Pure-Python (stdlib only). Uses chunk-level corpus baseline so terms
concentrated in fewer chunks outrank common ones. Default top-K = 16.
Output cores are comma-separated keyword lists — extreme compression
toward the tweet/haiku end of the planet metaphor. Same source can
now carry both a first-sentence-v1 core AND a tfidf-keywords-v1 core,
each derived independently and Merkle-signed back to the same surface.
The 'contributing_chunk_indices' for TF-IDF is every chunk that
contains at least one of the top-K keywords — proof binding remains
honest and cryptographically tight.
Distillers now scan kind='surface' OR kind='core'. Cores generate
deeper cores with compression_depth incremented. Each round of
recursion tightens the planet toward its center.
Audit body for derive events now records src_kind and the resulting
compression_depth so the chain reflects the layer transition.
Smoke against the existing 503 surface + 478 depth=1 core corpus
produced N deeper cores with proofs still binding back to their
depth=1 sources.
