Per fox: README was stale on most of what shipped over the past
sessions. Sweep:
- New Quickstart section at the top (replaces "What this gets you")
with two end-to-end paths: Wikipedia 2003 dump and crawling a live
site (russell.ballestrini.net as the worked example). Each is
five-ish commands and ends with a real `make query`.
- New "Data: live websites (the crawler)" section between git/hg
ingest and the OpenAI-export placeholder. Documents:
- bootstrap-crawler / crawl-ingest with all knobs (URL, DEPTH,
MAX, FAST, CRAWL_SHARD)
- per-domain shard naming so make query auto-discovers the data
- feed/sitemap skip at ingest (discovery infrastructure, not
knowledge)
- ETag/Last-Modified capture + recrawl-check freshness probe
- provenance pointer to ~/git/agents.ai.unturf.com
- Asking-the-corpus section gains JSON=1, default human render note,
question_hash equivalence (trailing punctuation stripped), and the
full query path now reflects wikitext-strip-before-LLM step.
- New "Verifying answers (audit modes)" section explains the
trichotomy + four-strategy verifier (quote/span/entity/paraphrase),
trailing-citation strip, and the corpus-growth signal flow
(emergent → reclassify).
- New "Marking and burning records" section covers falsify vs burn
(kindergarten rule, KIND= for documents and cores), chain-check.
- Mesh section: dropped "on the roadmap" claim — wire layer +
serve/sync/pull verbs are shipped. Pointers to docs/mesh.md and
docs/mesh-deploy.md.
- Architecture diagram updated to current tree (wikitext.py,
sources/crawler/, qa/inspect.py, qa/verify.py, mesh/, vcs.py).
- Whitepaper pointer updated to canonical rst path; mesh docs added.
- Test counts: 326+ default suite, separate test-crawler target,
default suite never hits the network.
Two changes that make the verifier accept more honest grounding without
relaxing what STRICT means.
1. Strip trailing citation parentheticals at extraction time.
`extract_quotes` and `extract_claim_spans` now drop a single trailing
`(Source: ...)` / `(citing X)` / `(https://...)` parenthetical
before returning. The Pikachu real-corpus case: model wrote the
verbatim source sentence and appended `(Source: https://...)`. The
prose IS in the corpus — only the inserted citation broke substring
match. Now substring fires; STRICT recovered.
Conservative regex: only strips when the parenthetical content
starts with a citation cue word OR contains a URL. A genuine prose
parenthetical like "Pikachu (a Pokémon species) lives in forests"
stays.
2. Paraphrase strategy as a 4th verifier method.
In the span path, items that fail substring match get a
token-coverage probe. Tokens >=4 chars from the span; if >= 85% of
them appear in the normalized base context, count the span as
paraphrase-verified. Paraphrase items contribute to n_verified;
verifier_method flips to 'paraphrase' when any soft-verified items
are present so an auditor can tell.
Gated by `_is_prose_span`: a list of proper nouns ("Keanu Reeves,
Laurence Fishburne") has no lowercase content tokens >=4 chars and
falls through to the entity strategy where proximity policy can
tight-cluster check. Real prose spans ("Pikachu is a species of
Pokémon creatures...") have multiple lowercase content tokens and
qualify for paraphrase.
Soft-signal note (from CLAUDE.md soft/hard rule): paraphrase is
heuristic, not byte-equivalence. The hard chain still records the
classification — verifier_method='paraphrase' is the explicit marker
that an audit reader can use to distinguish lexical-verbatim from
paraphrase-overlap. Quote strategy stays verbatim-only — quotes ARE
quotes; paraphrase-in-quotes is the model's mistake to flag, not
auto-promote.
Schema: verifier_method CHECK constraint expanded to include
'paraphrase'. New idempotent migration `_rebuild_providence_cache_paraphrase`
mirrors `_rebuild_providence_cache_ungrounded` (table rebuild via
BEGIN IMMEDIATE; no value translation needed). Probes existing CHECK
from sqlite_master and no-ops if already expanded.
Tests: 9 new in tests/test_verify.py — 4 trailing-citation strip
(quote+span variants, URL-only parenthetical, end-to-end STRICT
recovery), 4 paraphrase strategy (high-coverage promote, low-coverage
reject, mixed-method label, quote-strategy preservation). Default
suite still green: 326 passed, 1 skipped.
Burned the live Pikachu cache record (audit chain extended) so a
fresh `make query Q="who is pikachu?"` exercises the new verifier
end-to-end.
Per fox: the verifier reports n/N verified but doesn't tell you WHY a
span didn't ground. Some are paraphrase, some are model-added
trailing artifacts (e.g. `(Source: https://...)` citations the model
appended to verbatim prose), some are full inventions. Operator
needs to see which is which to triage.
Read-only sidecar — no audit events, no providence_cache mutations,
no v9.8 field changes. Per the verifier-no-diagnostics memory rule:
hard chain stays binary; soft signals live in sidecar verbs that
never feed back.
Five diagnosis labels:
verbatim_in_base span IS in base context — verifier or
canonicalization bug (worth flagging loudly).
verbatim_in_raw_only raw wikitext match but base form differs —
wikitext-strip edge case.
trailing_artifact a >=60-char prefix matches; the tail (often a
model-added citation) doesn't. Surfaces the
tail explicitly.
paraphrase >=85% query tokens (>4 chars) present in
base context but not in this sequence.
Model rewrote source content.
partial_paraphrase 40-85% coverage — mixed sourced/emergent.
no_overlap <40% coverage — likely full invention.
Verified live against the real Pikachu cache record (5 sources,
71KB raw -> 34KB base): two spans flagged paraphrase (token_coverage
1.0 for both, just rewritten sequence), one flagged trailing_artifact
(100-char prefix matches; tail is `(Source: https://...)` citation).
That's the actual reason HYBRID — model paraphrased + appended
citations that aren't in the corpus.
Surface:
aborist inspect --cache-key <hex> [--qa-db ...] [--json]
make inspect KEY=<hex> [JSON=1]
Tests: 9 (6 classifier on synthetic contexts pinning each label,
3 end-to-end with seeded fixture record + verification that inspect
writes nothing). 318 passed, 1 skipped overall.
Default `make query Q="..."` now uses the human render. JSON=1 flips
to the raw record (same as `aborist query --json` directly):
make query Q="who is pikachu?" # human render
make query Q="who is pikachu?" JSON=1 # raw JSON record
Same flag works on query-dry. Help line updated to advertise it.
Two things fox surfaced from a "who is pikachu?" run:
1. Output emitted "Pok\\u00e9mon" instead of "Pokémon" — json.dumps
defaulted to ensure_ascii=True. Switched to ensure_ascii=False on
every user-facing dump in cli.py (41 sites). The one canonical-JSON
call (`separators=(",", ":")` for storing in providence_cache as
a JSON column, not for hashing) was deliberately left alone.
2. The actual answer was buried under cache_key / context_root / per-
source metadata / timings JSON. Default now renders human-readable:
who is pikachu?
HYBRID 1/2 verified via quote 9.2s (fresh)
Pikachu is a species of Pokémon creatures...
sources (3):
[1] Pikachu — en.wikipedia.org/wiki/Pikachu (002.db)
[2] List_of_Pokémon — en.wikipedia.org/wiki/List_of_Pokémon
...
unverified (1):
- "spans the model couldn't ground..."
cache_key: 35ab7d33… <run with --json for full record>
Pass `--json` to get the prior raw record (still ensure_ascii=False
so unicode renders cleanly there too — scripts parsing the output
see real chars; the JSON spec accepts either form).
Implementation:
- `_render_query_human(result, question)` — pure function, easy to
unit-test, no I/O. Truncates unverified spans over 100 chars,
omits empty sections, distinguishes cached vs fresh in the summary.
- `_strip_scheme` / `_short_path` — small helpers for source display.
- `--json` flag on the `query` subparser; default is human render.
Tests: 12 new in tests/test_cli_render.py — question position, summary
fields (audit/n_verified/method/elapsed/cache-status), unicode
literals (Pokémon not \\u00e9), source line shape, long-quote
truncation, empty-section omission, error fallback, short cache_key
with --json hint. 309 passed, 1 skipped overall.
Per fox: ASCII-only stripping leaves CJK full-width forms behind, so
"who is X？" and "who is X" still produce different cache_keys
despite being the same question. Expanded strip set:
ASCII: . ? ! , ; :
CJK: ？(U+FF1F) ！(U+FF01) 。(U+3002) 、(U+3001)
ellipsis: …(U+2026)
Pairs deliberately stay out of the set:
" ' ) ] }
Stripping one side of a pair breaks balance. `who said "X"?` after
stripping `?` is balanced; further stripping `"` would yield
`who said "X` — different equivalence class than the original. And
apostrophes carry meaning: `X's` is a different question from `X`.
Lifted the strip set into a module constant `_QUESTION_TRAILING_STRIP`
so anyone considering an addition has a documented anchor.
Tests: 11 new in tests/test_keys.py — 6 CJK / ellipsis equivalences,
5 pair-preservation cases (double-quote, single-quote, paren, bracket,
brace). 24 keys tests + 297 default suite, all passing.
Whitepaper sibling change in ~/git/unfirehose-nextjs-logger/whitepaper/
merkle-providence-reverse-rag-whitepaper.rst — expanded the
question_hash bullet to document the equivalence class. Not committed
in this commit (different repo).
Today `question_hash` canonicalizes (NFC, ws-collapse, lowercase) but
preserves trailing punctuation. So "who is X?" and "who is X" produce
different hashes, different cache_keys, different providence_cache
records — the cache misses on what's semantically the same question.
Add `rstrip(".?!,;:")` as the third canonicalization step inside
question_hash specifically. Local to question hashing — chunk leaf
hashes go through `canonicalize()` directly and are unaffected.
Equivalence class after this:
"who is X" \
"who is X?" |
"who is X." | -> all same question_hash, all same cache_key
"who is X!" |
"Who Is X" /
Internal punctuation is preserved on purpose: "X, then Y" carries
meaning that "X then Y" doesn't, even though both have the same
content tokens.
One-time impact: prior cached records whose canonical question ended
in punctuation become orphans on lookup (re-derive on next ask).
History stays on disk; nothing burned automatically.
New test file tests/test_keys.py with 13 tests pinning the question_hash
equivalence class plus light coverage of the other dim hashes
(model_profile, conversation, governance, cache_key 8-dim invariant).
286 passed, 1 skipped.
Per fox: a query for "who is Russell Ballestrini" classified STRICT by
grounding 6/6 quotes against `feeds/all.atom.xml` — a 230-chunk dump
of post metadata. The verifier was technically correct (every quoted
string IS in the feed) but the result was hollow: feeds list URLs,
they don't carry knowledge. The ACTUAL bios live at the linked posts.
Two-pass filter in _CrawledHtmlSource.iter_documents:
1. Pre-fetch URL-pattern check (_looks_like_feed_url):
- suffix matches: .atom, .atom.xml, .rss, .rss.xml, .rdf,
/feed.xml, /atom.xml, /rss.xml, /rss2.xml, wp-rss2.xml,
wp-atom.xml, wp-rdf.xml, wp-rss.xml
- substring matches in path: /feed/, /feeds/, /atom, /rss, /sitemap
2. Post-fetch Content-Type check (_looks_like_feed_response):
- rejects: application/atom+xml, application/rss+xml,
application/rdf+xml, application/xml, text/xml
- keeps: application/xhtml+xml (xhtml IS html, just stricter syntax)
Defense in depth — a feed served at a non-feed path (e.g.
/index.html returning text/xml) still gets dropped on Content-Type.
What stays out of the corpus:
/feeds/all.atom.xml (Atom feeds)
/sitemap.xml (XML sitemaps)
/wp-rss2.xml (WordPress RSS)
/feed/ (any feed alias)
What's still allowed:
/post-name/ (real prose pages)
/index.html (HTML)
/page.xhtml (xhtml)
Tests: 12 new parametric cases (8 path patterns + 6 Content-Types +
xhtml positive case). 36 bridge tests + 273 default suite, all
passing.
For fox: the existing feed entry already in the crawl shard was burned
manually via `aborist burn --kind document --root eaf7c8d5...` — 230
chunks gone. Re-running `make crawl-ingest` won't re-introduce it.
Defect found by fox: `make crawl-ingest URL=...` wrote to
~/.aborist/aborist.db (the legacy single-DB path) but `make query Q=...`
searches ~/.aborist/shards/ — two different corpora, query never saw
the crawled content.
Fix mirrors the existing shard-aware ingest pattern (ingest-self,
ingest-git, ingest-grok-attached): derive a shard filename from the
seed URL's hostname and write into $(SHARDS_DIR). Override with
CRAWL_SHARD=path for custom layout.
https://russell.ballestrini.net
-> $(SHARDS_DIR)/crawl_russell_ballestrini_net.db
After this, `make query Q=...` (cross-shard search) automatically
picks up newly-crawled domains. Each domain stays in its own shard
file so falsifications, evictions, and deletions are scoped naturally.
recrawl-check gets the same treatment: defaults to walking every shard
under SHARDS_DIR (skipping qa.db and snapshots.db); CRAWL_SHARD=path
scopes to one when you want.
Note for fox: existing pages already at ~/.aborist/aborist.db from
the prior crawl are still there (harmless). To get them under query,
either re-crawl into the new path or migrate via sqlite ATTACH +
INSERT — a follow-up if you want it.
The lifted AsyncWebFetcher already had a fast_mode constructor knob;
just unbroken plumbing was missing. fast_mode trades politeness for
throughput:
- timeouts: 5s vs 15-60s
- parallel page workers: cpu_count() * 3 vs 1
- ignores robots.txt 'crawl-delay' directive
- robots.txt 'Disallow' is STILL honored (only the delay drops)
Surface:
aborist crawl --seed-url ... --fast
make crawl-ingest URL=... FAST=1
Tests: two regressions pin that fast=True / fast=False each translate
to the right AsyncWebFetcher(fast_mode=...) construction. Capturing
fetcher fixture to avoid network. 21 bridge tests + 273 default
suite, all passing.
A real BFS hits sitemap.xml / RSS feeds via the verbatim crawler's
generic page handler. bs4 warns "you're using an HTML parser on XML"
on each — benign (parsing still works) but spams stderr during a
deep crawl, drowning the progress heartbeat we just wired in.
Filter at aborist/sources/crawler/__init__.py so the lifted source
stays untouched. The filter applies whenever any caller imports from
this subpackage, including the CLI's `from ... bridge import ...`
(which runs __init__.py first per Python import semantics).
A no-cap crawl could take many seconds with no feedback. Wires the
existing aborist.progress.Progress into both phases so stderr shows
heartbeats every 2s (Progress's default interval).
Discovery phase: prints prefix='crawl', counts discovered URLs,
shows queue depth as the secondary number ('inserted' slot in the
Progress format — works fine, semantically "still to do").
Ingest phase: prefix='ingest', total_estimate=len(urls) so the user
sees percent + ETA. Threads through ingest_source's existing
progress= parameter.
Plus three banner lines to stderr at phase boundaries (start crawl,
end discovery, start ingest) so even sub-2s crawls show signs of
life. All flushed via Progress's flush=True path; stderr is
line-buffered by default so this works without PYTHONUNBUFFERED.
Tests stay green (Progress goes to stderr, pytest captures only the
stdout summary). 19 bridge tests + 273 default suite, all passing.
Per fox: 20 was a wrong default — typical sites have many more pages
within depth 2 than that, and the cap was silently truncating real
crawls. New behavior:
aborist crawl --depth 2 # no page cap
aborist crawl --depth 2 --max-pages 50 # opt-in cap when needed
make crawl-ingest URL=... DEPTH=2 # no page cap
make crawl-ingest URL=... DEPTH=2 MAX=50 # explicit cap
Implementation: bridge's BFS loop treats max_pages=0 as "unbounded"
(while-condition becomes `queue and (max_pages == 0 or len(...) < max_pages)`).
Default flows from CLI argparse default=0 down to the bridge.
Tests: pinned max_pages=0 → no cap with a 30-page chain regression
test (19 passed in tests/crawler/test_bridge.py). Default suite still
273 passed, 1 skipped.
The verbatim AsyncWebFetcher.fetch_webpage defaults extract_links=False
and returns (html, []) when omitted. Bridge's BFS therefore terminated
at the seed — a real-world crawl of russell.ballestrini.net discovered
exactly 1 URL (the seed). One-line fix; the kwarg was always there in
the lifted API.
Added test_crawl_seed_passes_extract_links_true that records every
fetch_webpage call and asserts the kwarg is True. Pin so this can't
silently regress when bridge internals get reorganized.
Updated the stub fetcher signature in the existing tests to accept
the new kwarg without behavior change.
Glues the verbatim crawler lift to aborist's existing ingest pipeline,
plus adds a freshness probe that asks "does this need recrawling?"
without paying for a body fetch when the answer is no.
Bridge (aborist/sources/crawler/bridge.py):
- crawl_seed(seed, depth, max_pages): BFS-discovers same-domain URLs
via AsyncWebFetcher. Exact-hostname filter so subdomain crossover is
off by default. Caps at max_pages. URL dedup, fragment-strip
canonicalization. Sync wrapper around an async core so the CLI
remains a one-liner.
- ingest_crawled(conn, urls): runs the URL list through the standard
ingest_source() path so chunking_version / canonicalization_version
/ Merkle commitment all stay consistent. Captures ETag +
Last-Modified per response and writes one row per document_root
into the new document_http_meta table.
Schema (idempotent migration, mirrors _migrate_mesh_peer_chains):
document_http_meta (
document_root TEXT PRIMARY KEY,
etag TEXT,
last_modified TEXT,
last_fetched_at INTEGER NOT NULL,
last_status INTEGER,
last_checked_at INTEGER,
FK(document_root) -> documents(document_root) ON DELETE CASCADE
)
Recrawl-check (recrawl_check):
- Sends conditional HEAD with If-None-Match + If-Modified-Since.
304 -> fresh (no body transfer needed). 200 -> stale (server says
the body changed). 404/410 -> gone. Other -> unreachable.
- Updates last_status + last_checked_at so consecutive runs visit
oldest-first (NULLS FIRST in ORDER BY).
- Optional --domain filter so operators can probe one site at a time.
CLI:
aborist crawl --seed-url URL [--depth 2] [--max-pages 20] [--ingest]
aborist crawler recrawl-check [--domain DOMAIN] [--limit 100]
Makefile:
crawl-ingest URL=https://example.com [DEPTH=2 MAX=20]
recrawl-check [DOMAIN=example.com LIMIT=100]
Both targets gate on bootstrap-crawler so [crawler] extras land first.
Tests: 17 new in tests/crawler/test_bridge.py — same-domain filter,
BFS bounds (depth + max_pages), dedup, fragment normalization, ETag
capture, content-type filter, HEAD classification (304/200/404),
domain filter, last_checked_at update. All stub the network via a
fake httpx.Client; default suite still 273 passed, 1 skipped.
End-to-end test path:
make bootstrap-crawler
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 MAX=20
make query Q="who is russell ballestrini?"
make recrawl-check DOMAIN=russell.ballestrini.net
Lifts the async web fetcher into aborist as an opt-in source. The
implementation comes directly from ~/git/agents.ai.unturf.com/core
(rev 2026-04-28); aborist's adaptations are minimal and documented in
aborist/sources/crawler/__init__.py:
core/async_web_fetcher.py -> aborist/sources/crawler/async_web_fetcher.py
core/web_fetch.py -> aborist/sources/crawler/web_fetch.py
Two source-side changes during the lift:
1. Heavy deps (aiohttp, bs4, miniuri) wrapped in try/except so a bare
`import aborist.sources.crawler` raises ImportError with the install
hint instead of leaking AttributeErrors deep in user code.
2. Chat-bot fetch triggers (`has_fresh_fetch_trigger`,
`has_web_fetch_trigger` from agents.ai.unturf.com/core/keywords)
replaced with NotImplementedError stubs. Aborist has no chat
surface — fetch intent is detected at the application layer. The
two test classes that exercised these triggers are
`@pytest.mark.skip`'d with the same rationale.
Not lifted: web_cache_manager.py — it backs page caching with
SQLAlchemy. Aborist has its own content-addressed cache via
providence_cache; no need to carry SQLAlchemy as a dep just for
crawled-page memoization.
Off by default:
- `[crawler]` extras section in pyproject.toml carries the heavy
deps. `[dev]` pulls them in so the crawler tests can run.
- `make test` ignores tests/crawler/ entirely.
- `make bootstrap-crawler` installs the extras into the venv.
- `make test-crawler` runs only the lifted tests after extras land.
Tests: 74 passed, 9 skipped (the chat-bot trigger tests deliberately
dropped). Default `make test` stays at 273 passed, 1 skipped.
Receiver now rejects gossip envelopes whose prev_event_hash doesn't
extend the sender's last-seen event_hash — fork detection lives at the
wire layer, not just in docs/mesh.md.
Schema: new mesh_peer_chains table (peer_member_id PK, last_event_hash,
last_seq, last_seen_at) with idempotent _migrate_mesh_peer_chains
matching the _migrate_audit_mode pattern.
Envelope: WireEnvelope gains prev_event_hash + event_hash, both
optional for back-compat. canonical_bytes() excludes event_hash to
avoid the circular dependency (the hash is computed FROM the canonical
bytes). New WireEnvelope.with_chain(prev_event_hash=...) builds a
linked envelope with event_hash = sha256(prev || canonical_bytes).
Sender (MeshWireClient._signed_envelope, MeshWireServer.handle_request
DELIVER_BODY): reads our latest_event_hash, builds the linked
envelope, then appends a 'mesh_sent' audit event whose body is the
envelope's canonical (event-hash-excluded) dict. By construction the
audit event_hash equals the envelope event_hash — wire chain-of-claims
and local audit chain stay in lockstep, so back-to-back sends advance
the chain naturally.
Receiver (handle_announce): when an envelope carries chain fields,
recompute event_hash and 400 on mismatch; look up
mesh_peer_chains[sender] and 409 on prev mismatch; on accept update
the row + write the existing mesh_received event.
Backward compat: legacy envelopes (event_hash=None) bypass chain
enforcement — kept the existing test_mesh_wire.py fixtures verbatim
since they construct WireEnvelope directly without with_chain(). All
production traffic goes through MeshWireClient and is always tracked.
v1 deferred (documented in module docstring): no multi-event catchup;
operator retries on 409. No cross-peer reconciliation.
Tests: 12 new in tests/test_mesh_chain.py covering the 7 required
cases plus canonical-bytes exclusion, migration idempotence, and
legacy-envelope fallback. 246 passed, 1 skipped overall.
Adds an opt-in confidentiality layer to gossip. Senders pass
encrypt=True on announce_root / announce_falsification; bodies travel
AEAD-encrypted (ChaCha20-Poly1305) under the per-epoch symmetric secret
recovered via the existing X25519 wrap. Cleartext stays the default —
all existing wire tests pass without modification.
WireEnvelope grows an optional encrypted_body field. canonical_bytes
omits the field when None (byte-identical to v1 cleartext format) and
includes it when set, so the Ed25519 signature binds the ciphertext —
a MITM cannot swap payloads. AAD is type|sender|epoch|ts so a captured
(nonce, ct) cannot be replayed under different envelope metadata.
state.recover_epoch_secret resolves the rotator from the audit chain
and re-unwraps this peer's slot. Sender + receiver share one path; no
new sender-side cache is required.
Closes the request half of the gossip loop. MeshWireClient already
verifies signature + Merkle root on delivered bodies; this verb adds
the CLI surface, gating, local re-ingest through the standard chunker,
and a 'mesh_pulled' audit event. Re-ingest is rejected if the local
document_root differs from the requested root.
burn --kind {providence,document,core}; default 'providence' preserves
the prior --cache-key surface (no breaking change).
Children gates for document/core leaves:
derivations.src_root downstream cores derived from this doc
edges.dst_root other docs link to this doc
providence_cache.source_root Q&A grounded in this doc
derivations has ON DELETE CASCADE on BOTH core_root and src_root, so a
naive DELETE FROM documents would silently cascade-prune downstream
derivations and orphan deeper cores. Gate before delete; --force to
override with the count recorded in the audit body.
Burn manually clears chunks_fts (FTS5 has no FK) and outbound edges
(no FK either) before deleting the document. chunks + merkle_nodes +
inbound derivations cascade via FK.
Audit event types are document_burn / core_burn — distinct from
providence_burn so chain consumers can tell leaf kinds apart.
CLAUDE.md "Cores never evict" remains intact: burn is operator-driven
removal, evict is the automated cold-tier compression that touches
only kind='surface'.
Makefile: KIND=providence|document|core selects the leaf type;
KEY for providence, ROOT for document/core. Single 'burn' target keeps
the surface DRY.
Tests (tests/test_burn_doc.py, 13 new):
- happy path: chunks + merkle_nodes vanish via cascade
- refused on derivation downstream / providence ref / incoming edge
- --force overrides and records counts
- core burn happy path (no children) preserves parent surface
- core burn refused with downstream derivation
- audit chain integrity preserved (no dangling prev_event_hash)
- CLI invocations for document, core, default-providence, refused
Full suite: 247 passed, 1 skipped.
- Wikitext base prose section now reflects reality: to_base() runs in
two places (before LLM call in query.py/runner.py AND inside
verify_quotes) gated by policy["base_version"]. Both sides see prose;
Hermes can quote source paragraphs verbatim instead of escaping
[[wikilinks]], and Wikipedia chunks ship with ~43% fewer tokens.
- Build/test block now points at the real targets: verify-shards,
analyze-shards, chain-check-shards. The bare 'make analyze' target
doesn't exist — only the cross-shard variants do.
- New "Hygiene after any state-changing op" callout: chain-check-shards
for one-second per-shard break count, analyze-shards for full audit.
- Audit-chain invariant block points at the same targets.
- Orientation protocol updated: --shards-dir replaces --db (the latter
refers to a single-DB setup that's no longer the canonical path), and
chain-check-shards is added to the per-session ritual.
Step-by-step operator guide for standing up an aborist mesh between
two real hosts: pre-flight, init/enable on both peers, out-of-band
pubkey exchange, mutual enrollment, mesh serve, mesh sync, audit
chain integrity checks, eviction protocol, reset/teardown.
Calls out what's not yet wired (mesh pull, per-peer chain merge,
AEAD body encryption) so operators don't expect features that ship
later. Pairs with docs/mesh.md (protocol contract) and the e2e tests
in tests/test_mesh_wire_e2e.py.
Both ask() (single-doc) and query() (multi-source) now run the
assembled context through aborist.wikitext.to_base() before message
construction. Hermes sees clean prose instead of raw wikitext, which:
- lets the model quote verbatim (raw [[Cloud Strife]] was unquotable
in fluent prose; the model paraphrased to bare "Cloud Strife" and
the verifier flagged that as UNGROUNDED)
- shrinks token bills ~43% on Wikipedia chunks (measured: 125,152 →
71,771 chars on the FF7 main-character query)
- aligns LLM-input with what the verifier already strips, so the
prose-vs-prose substring check is end-to-end consistent
Gated on policy["base_version"] (= wikitext-base-v1) so:
- governance_policy_hash flows the choice — prior answers cached
under raw-wikitext policy retain distinct cache_keys, no migration
- operators can opt out (set base_version=None) for debugging
- environments without mwparserfromhell soft-fall to raw wikitext
with no other behavior change
Tests (4): default policy carries base_version (tripwire), ask() user
messages have wikitext markers stripped, opt-out reverts to raw
wikitext, governance_hash differs with vs without base_version.
Wires the gossip-wire core (commit f141bab) into the user-facing CLI:
aborist mesh serve --host 127.0.0.1 --port 8400
Spawns MeshWireServer, blocks until SIGINT. Refuses if mesh isn't
initialized or .enabled is off — same gate every other mesh verb
enforces. Stdout is unbuffered JSON status lines.
aborist mesh sync --peer http://other.example:8400 [--limit N]
Enumerates local documents (most-recent N, default 100), fires one
ANNOUNCE_ROOT per document via MeshWireClient, reports counts of
acknowledged vs. errored. Pulling missing roots back from the peer
is a v2 addition — this verb pushes only.
Tests (6): gating refusals on uninitialized + disabled mesh; sync
round-trip with two peers proving Alice's audit chain gains one
'mesh_received' per Bob announce; unreachable-peer reporting; serve
smoke-test via start_in_thread + GET /mesh/info.
Burn deletes a providence_cache row outright; falsify (the existing
verb) flips state and keeps the row. Burn is for early/scratch use
during a tree's genesis when downstream consumers haven't ingested
the answer yet.
Children gate: refuses if any falsifications reference the cache_key.
Caller can override with --force; the audit event records forced=True
plus the child count at burn time so the chain can reconstruct what
was orphaned. Refusal is non-destructive AND skips the audit event so
scripts can retry without polluting history.
Always writes a 'providence_burn' audit event regardless of force.
Body fields: cache_key, burned_audit_mode, burned_n_verified,
burned_state, question_text, reason, by_actor, child_falsifications_at_burn,
forced. The chain-break query (`make chain-check`) shows 0 after burn —
the new event extends the prior chain cleanly.
Surface:
aborist burn --cache-key <hex> [--reason '...'] [--force]
make burn KEY=<hex> REASON='...' [FORCE=1]
Tests (7): happy-path delete + audit event, refusal on children,
--force override, unknown-key not-found, chain-integrity preserved,
CLI happy path, CLI non-zero exit on refusal.
- aborist/qa/verify.py: three strategies tried in sequence. quote uses
sequential pairing (1st & 2nd `"`, 3rd & 4th, ...) which eliminates
the phantom inter-pair captures regex pairing produced on adjacent
quote pairs like `"title" prose "quote"`. span checks bullet/sentence
lines verbatim. entity matches multi-word proper nouns; gated by
policy ∈ {strict, hybrid, drop, proximity}. Default proximity
promotes to STRICT only when N=3 verified entities cluster within
W=300 chars in source — separates structural grounding (cast list,
infobox) from incidental mention (scattered plot summary).
- aborist/store.py: schema CHECK now `('STRICT','HYBRID','UNGROUNDED')`.
Migration helper rebuilds legacy `('STRICT','HYBRID','VISUAL')`
tables via temp-table copy, translating VISUAL → UNGROUNDED in the
SELECT. Idempotent — DDL inspection skips the rebuild on already-
migrated DBs.
- aborist/cli.py: new `aborist reclassify` re-runs the verifier
against existing live records under the current entity policy; no
LLM calls. --compare runs all four policies side-by-side, --dry-run
reports transitions without writing. Cold-source records skipped.
One providence_reclassify audit event per changed row.
- AuditMode.UNGROUNDED replaces VISUAL across search backend, FTS5,
test fixtures, CLAUDE.md. The substrate (Merkle-AGI) name was
about FOR-style visualization; in the RAG layer the semantic is
"no recoverable grounding," so the label now says that.
DEFAULT_QUERY_POLICY gains entity_policy + entity_proximity_n +
entity_proximity_window so any tuning folds into governance_policy_hash
and invalidates cache cleanly.
Implements the protocol contract pinned in docs/mesh.md. Each peer
runs a stdlib ThreadingHTTPServer; clients send Ed25519-signed
envelopes carrying one of:
ANNOUNCE_ROOT document_root + source_uri + versions
ANNOUNCE_DERIVATION core_root <- surface_roots (proof-blob hash)
ANNOUNCE_PROVIDENCE cache_key + audit_mode + answer_hash
ANNOUNCE_FALSIFICATION cache_key + reason
REQUEST_BODY/DELIVER_BODY pull-on-miss + Merkle-root verify
WireEnvelope canonicalizes via sorted-key separator-tight JSON; the
canonical bytes are what get signed. Receivers verify the signature
against sender_id's sign_pub looked up in mesh_roster at the
envelope's epoch_id — non-members of that epoch produce no valid
signature, so they're silently rejected (401, no audit event).
Each accepted ANNOUNCE_* writes one 'mesh_received' event into the
local audit chain whose body is the full signed envelope (the
remote's signature stays attached for non-repudiation). The
receiver's chain stays internally consistent because we append in
receive-order.
REQUEST_BODY/DELIVER_BODY: the responder ships the document text
plus per-chunk leaf hashes; the client re-derives the Merkle root
from those leaves and refuses to return any body whose leaves
don't reconstruct the requested root.
Tests: 17 unit (envelope canonicalization, sig verify against
roster, type validation, tampered-body rejection, audit-event
write) + 5 end-to-end (two real HTTP peers on ephemeral ports
exchanging announces and bodies). 22/22 green.
Deferred to later commits, per docs/mesh.md:
- CLI verbs (mesh serve, mesh sync) — pending downstream merge
- per-peer chain-of-claims tracking (catchup, replay, fork detection)
- optional AEAD encryption of message bodies (contract says optional)
Adds aborist/wikitext.py with to_base() — deterministic wikitext →
prose conversion via mwparserfromhell. Drops <ref>...</ref>,
[[File:...]], [[Image:...]], [[Category:...]] entirely; resolves
piped wikilinks to display text; collapses templates, formatting,
and HTML markup. extract_wikilinks() preserves the link graph for
the definition-cloud artifact (recoverable from any page on demand
without re-parsing wikitext at query time).
Wires to_base() into verify_quotes(). Without the strip, the
verifier compared the model's clean prose against [[Cloud Strife]],
[[Shinra Electric Power Company|Shinra]], etc. and falsely flagged
real source quotes as VISUAL. Concrete case from a make-query run:
'Cloud Strife, an unsociable mercenary who claims to be a former
1st Class member of Shinra's SOLDIER unit;' is verbatim in the
Final_Fantasy_VII article wikitext (modulo markup). With the strip
that quote now verifies; the genuine model hallucinations in the
same answer still flag honestly. Side-effect: 43% smaller context
size on average so less LLM token waste.
BASE_VERSION = 'wikitext-base-v1' is the algorithm pin. Bump when
the strip rules change. Soft-imported in verify.py so environments
without mwparserfromhell installed degrade gracefully (no strip,
same behavior as before this commit).
Build:
- pyproject.toml: new [wikitext] extras (mwparserfromhell>=0.6),
pulled in by [dev]
- Makefile: chain-check / chain-check-shards targets — fast
audit-chain integrity probe, counts dangling prev_event_hash
references; 0 = chain intact
Tests:
- tests/test_wikitext.py: 31 tests (rules, idempotence, real-corpus
fixture)
- tests/test_verify.py: 2 regression tests pinning the FF7 flip
(Cloud Strife quote: VISUAL → STRICT after strip; genuine
hallucination: stays VISUAL)
- tests/fixtures/ff7_characters_chunk0.wikitext: real chunk from a
shard, used to validate the strip on actual Wikipedia content
Adds aborist/qa/verify.py — three-strategy verifier (explicit quotes,
bullet/sentence spans, multi-word proper nouns) that lexically checks
every claim against retrieved context under norm-v1 + lowercase. The
strategy that fires is recorded as verifier_method for diagnostics.
Entity strategy gates classification via an entity_policy
(strict/hybrid/proximity/drop) so a single proper-noun match no longer
overclaims STRICT — proximity (default) requires a cluster of 3+
verified entities within 300 chars.
Wires into ask() and query(). System prompts now require verbatim
quoted spans for every factual claim, restated via a user-turn
grounding_reminder one message before sources arrive (recent
user-turn instructions outweigh decayed system-turn rules under long
context in Hermes).
providence_cache gains 5 columns (audit_mode, n_quotes, n_verified,
unverified_quotes, verifier_method) with CHECK constraints. A
connect-time _migrate_audit_mode() ALTERs legacy DBs idempotently.
Cache hits return the persisted audit_mode rather than asserting
STRICT unconditionally.
CLI:
emergent list VISUAL/HYBRID records; --aggregate ranks
unverified quotes by frequency (corpus-growth signal).
reclassify re-run the verifier against live providence records;
cold-source records skipped; --dry-run reports
transitions without writing; each change writes one
'providence_reclassify' audit event.
Adds docs/mesh.md and five graphviz diagrams covering the federation
layer: identity stack, epoch state machine, per-member secret envelope,
gossip wire contract, and operator decision tree. Pins the protocol
contract for the upcoming HTTP wire (mesh sync, mesh serve).
Makefile gets a 'docs' target with pattern rule so PNG renders are
incremental from .dot sources.
A snapshot_root is MerkleTree.build([sorted DISTINCT document_roots]).root
— one 32-byte hash naming the entire content-addressed forest at a
point in time. Two peers that ingested the same dump compute bit-
identical snapshot_roots, so cross-machine "are we synced?" becomes
an O(1) hash comparison; the same root doubles as the TF-IDF
scope_root we sketched in the mesh design (snapshot_root *is* a
scope), and pins Q&A answers to a verifiable corpus state.
Schema: one new table, additive over existing data.
snapshots(snapshot_root PK, taken_at, audit_event_hash, doc_count,
parent_snapshot, reason)
+ idx_snapshots_taken_at
API:
compute_snapshot_root(conn, *, document_roots=None) -> (root, count)
create_snapshot(conn, *, reason, parent_snapshot=None) -> dict
verify_snapshot(conn, snapshot_root) -> dict (matches: bool)
diff_against_current(conn, snapshot_root) -> dict (added/removed/unchanged)
list_snapshots(conn, *, limit) -> list[dict]
CLI:
aborist snapshot create [--reason "..."] [--parent <hex>]
aborist snapshot list [--limit N]
aborist snapshot verify <snapshot_root>
aborist snapshot diff <snapshot_root>
Cross-shard: with --shards-dir + --db, the snapshot is computed over
the cluster-wide UNION view but persisted into args.db (a dedicated
snapshots store, conventionally ~/.aborist/shards/snapshots.db).
parent_snapshot auto-links to the latest prior snapshot in the
writer DB, giving a chain for free.
Each snapshot creation writes an audit_event of type 'snapshot_create'
with subject_root=snapshot_root, so the corpus's pinned states are
themselves tamper-evidently logged.
Defect caught + fixed during live test on the 2010 enwiki ingest:
verify and diff disagreed on the same connection because compute used
a list (with cross-shard duplicate document_roots) while diff used a
set. Two shards can land identical document_roots when canonicalize()
maps two structurally-similar pages to the same byte stream — rare
but real. Switched the underlying SQL to SELECT DISTINCT so a
membership snapshot is always order- AND multiplicity-independent.
Live test: 2010-11 enwiki corpus snapshotted at
43797e46605de08dbab06cdcaf5be7ad78243b193c56f8580200dee6bcc7e1b9
doc_count: 3,468,134 (after dedup)
verify + diff round-trip both report identical against current state.
11 new tests covering empty corpus, single-doc degenerate, order
independence, drift detection, audit-chain pinning, parent auto-link,
and idempotent creation on unchanged corpus. 136 tests + 1 skipped.
After the title-search baseline drop, FTS5 BM25 still surfaced short
stub articles ahead of long topical ones for queries with generic
tokens. Intel_4040 (1974 4-bit microcontroller, 1-paragraph stub
mentioning "intel"/"fastest"/"cpu" once each) outscored Intel_Core_2
(real 2010-era CPU article, hundreds of mentions of every query
token) because BM25 normalizes by document length and short docs
that hit every term get an outsized boost.
Add `_rerank_by_body_coverage` between the relevance filter and the
title-overlap rerank. For each surviving candidate (typically 24-32
post-dedup), fetch the full body and sum sqrt(count(t)) over query
tokens, then add `coverage * 0.6` to the score.
Sqrt scaling is intentional: linear sums let enumerative list pages
(e.g. List_of_Intel_microprocessors with 1000+ "intel" mentions)
runaway-dominate, while a hard cap saturates and erases the signal
between short-stub and long-topical articles. Sqrt grows fast enough
to differentiate (5 mentions -> 2.2; 200 mentions -> 14.1) but
sublinearly enough to keep enumerations from overwhelming the merge.
Verified on the 2010-11 enwiki corpus (no distill cores yet):
before: Intel_4040 #1 (43.5), Intel_Core_i7 #6 (39.0)
after: Intel_Core_2 #1 (51.9), Intel_Atom #2 (50.5),
Intel_4040 #3 (49.6)
Body fetch cost is bounded by `over_fetch` (32 candidates per shard,
24-50 unique post-dedup). Each fetch concatenates already-decompressed
chunks via `_load_doc_text` which we use elsewhere in the query path,
so this adds at most one extra DB round-trip per surviving hit.
125 tests still passing.
Two structural defects in the corpus retrieval path were causing
the Intel-CPU query (and any query whose terms hit many article
titles generically) to surface 80486-era legacy articles instead
of contemporary topic articles:
1. _search_titles entries arrived with a 50-point baseline score
(50 + 10*overlap) plus _rerank_by_title doubled the overlap
bonus on top. A single-token title match against a generic
word like "intel" landed at score 70-80 standalone, swamping
FTS5 BM25 hits that scored 5-15. Result: every Intel_*-prefixed
article in the corpus rose to the top of every "intel" query,
regardless of whether the article was actually about a fast CPU.
Drop the 50-point baseline. Title overlap now scores `10*overlap`
from this path, with `_rerank_by_title` adding another `10*overlap`
for any hit (FTS5 or title) whose title overlaps query tokens.
Net effect: title-only-overlap=1 hits score ~10-20, FTS5 hits with
title-overlap score ~25-45, FTS5-only-no-title-overlap score ~5-25.
Body relevance now beats single-token title coincidence.
2. _body_density_passes counted ANY query token's mentions toward
`min_mentions`, so Intel_8086 (with 30 "Intel" mentions and zero
"fastest" or "cpu" mentions) passed the gate for a query of
{intel, fastest, cpu}. Tighten to require breadth: at least
half the query tokens must each appear at least once, AND total
mentions must clear min_mentions. For 1-token queries this is
identical to the prior behavior.
Verified end-to-end on the fresh 2010-11 enwiki corpus (3.47M docs,
no distill cores yet). Pre-fix: top-K all Intel_8086/80286/80386/etc.,
Hermes refused to identify a fastest CPU. Post-fix: Intel Core i7
in top-K, Hermes correctly identifies Core i7-980X Extreme Edition
Gulftown (Intel's 2010 flagship — 32nm, 6 cores, 3.33 GHz, 12 MB L3).
Same 2003-corpus AMD-query result preserved (already had Athlon /
Athlon_XP_3200+ in top-K via FTS5 BM25 + multi-token TF-IDF cores).
125 tests + 1 skipped, no regressions.
Phase 1 of the federation/gossip layer fox sketched as the natural
extension of v9.8 admissibility's content-addressed identity. Two
peers ingesting the same dump already compute identical document_roots
and identical 8-dim cache_keys; the mesh layer is the wire-and-trust
plumbing that lets them dedup answers, exchange Merkle proofs, and
cleanly distrust an evicted member without a hard fork.
Cryptography (cryptography lib, audited):
Ed25519 — every membership mutation + (future) gossip envelope
is signed by the actor's pubkey.
X25519 ECDH — wraps each epoch's symmetric mesh secret to every
current member's DH pubkey via HKDF-derived AEAD key.
ChaCha20-P1305— AEAD for envelope payloads + per-member secret wrap.
State machine:
mesh_identity — singleton; this peer's keys + group name
mesh_roster — per-epoch (member_id, sign_pub, dh_pub, role)
mesh_epochs — epoch_id -> {started_at, started_event_hash,
secret_envelope JSON, reason}
meta:mesh.enabled flag — off by default; gates everything
Eviction works by rotating to a new epoch whose envelope omits the
kicked member. Their prior signatures stay verifiable (the older
roster row is retained), but any gossip from epoch+1 onward is
opaque to them — the secret was never shared with their pubkey.
Authority gate: only roster members with role='admin' can add or
kick. Self-kick is rejected explicitly. The last admin can't be
kicked. Schedule-rotate (refresh secret, no roster change) is open
to any current member as a session-hygiene op.
Audit-chain integration: every mesh state mutation writes an audit
event (mesh_init, mesh_enable/disable, mesh_epoch_rotate). The
epoch's started_event_hash backfills into mesh_epochs after the
audit row commits, giving each epoch a tamper-evident pin into the
ledger.
CLI subcommands: mesh init, mesh status, mesh enable, mesh disable,
mesh members, mesh add, mesh kick, mesh rotate. All read-only or
local-state-only — no network code paths in this commit.
The HTTP gossip wire (`mesh sync`, `mesh serve`) is the next phase.
Schema, cryptography, and roster state machine are all in place to
support it without further migration.
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`,
yielding one Document per text file. URI shape `git://<repo>/file/<path>`
intentionally omits the commit hash — re-ingest after new commits produces
fresh document_roots that aborist's prior-doc detection chains via
`supersedes` edges, so the audit trail and Merkle tree grow as the repo
grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files
filtered out by default. Commit hash + timestamp + subject ride along in
`extra` (informational only; not part of the Merkle commitment).
MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes
semantics, same shape.
Makefile adds:
make ingest-self (this repo -> aborist-self.db)
make ingest-git GIT_REPO=/path/to/repo (arbitrary git clone)
make ingest-hg HG_REPO=/path/to/repo (mercurial)
Each lands in its own shard file alongside existing shards/grok.db,
keeping per-shard write paths independent of the wikipedia 4-way ingest's
WAL writer lock.
Adds a new section after the 2003 Wikipedia / distillation block that
covers Phase IV (2006+) XML dumps: fetch-xml / ingest-xml-attached /
ingest-abstract targets, the WP_XML_YEAR/MONTH/DATE knobs for picking
any dated snapshot in dumps.wikimedia.org/archive, plus a pointer at
the abstract.xml feed as a cheap-coverage alternative to the full
pages-articles dump (~5-10 GB instead of ~95 GB on disk for enwiki
2010-11). Existing 2003 section retitled to flag the format split.
Three cheats stack to drop on-disk store from ~21 KB to ~6.7 KB per doc on
the 2003 enwiki cur corpus (-67% measured, apples-to-apples reingest with
identical document/edge counts; Merkle proofs round-trip 30/30):
1. zstd-compressed chunks.content (level 3). Magic-byte detection on read
means legacy plaintext rows pass through unchanged. Cores stay plaintext
so qa.query._docs_with_core_keyword_match's SQL LOWER+LIKE keeps working.
2. edges WITHOUT ROWID. The composite PK (src_root, edge_type, dst_root,
dst_uri, anchor) covers every column, so a default rowid-based table
near-doubles row data in the PK index. WITHOUT ROWID makes the table
itself the B-tree. Drops idx_edges_dst_uri too — the only query that
filters on dst_uri alone is gravity_top_inbound, a one-shot analytic.
3. contentless FTS5 (content='', contentless_delete=1) eliminates the
28 MB / 1000 docs of duplicated chunk text the old chunks_fts stored.
chunks gets an explicit chunk_id INTEGER PRIMARY KEY so the FTS5
rowid maps back to chunks.chunk_id at search time. Snippets are
built in Python (search/fts5.py:_build_snippet) since SQL snippet()
returns empty in contentless mode.
TF-IDF retrieval also fixed: the prior LIKE '%intel%' substring match
let "intelligence", "intellectual", "intellivision" drown real hits like
Pentium_4 (whose TF-IDF core has "intel" as an exact keyword). Now uses
word-boundary `LIKE '%, intel, %'` patterns plus a match_count over the
distinct query tokens — multi-token coverage outranks single-token title
boosts. Pentium_4 surfaces #1 for "what is the fastest intel CPU?" with
the canonical 2003 answer (Pentium 4 3.20 GHz) instead of an empty
"insufficient sources" reply.
Schema-level changes affect new DBs only; existing v9.8 DBs keep
working at the old layout. Cross-shard UNION views explicitly list the
intersection of columns so a mixed cluster (legacy + new schema shards
in one --shards-dir) still unions cleanly.
Drops in for any dated snapshot in dumps.wikimedia.org/archive — pages-articles.xml.bz2
and pages-meta-current.xml.bz2 (cur snapshot, single-revision-per-page) plus
pages-meta-history.xml.bz2 (multi_revision=True). abstract.xml feed yields
pre-distilled summaries at ~1/100th the chunk volume. Streams .bz2/.gz directly
via iterparse with bounded memory; same shard / resume contract as the SQL
source. Falls back to title-prefix namespace filtering when older export
schemas omit the per-page <ns> element (e.g. enwiki 20101011).
Makefile defaults target enwiki 20101011 (6.2 GB). Override with WP_XML_YEAR /
WP_XML_MONTH / WP_XML_DATE / WP_XML_LANG to fetch any other archived snapshot.
Replace the boilerplate AGPL-3.0 header in LICENSE with the exact
license clause from the Merkle Providence Reverse RAG whitepaper
(April 2026, Russell Ballestrini / David Wong / Riley Morgan):
License: AGPL-3.0-only · This algorithm, its implementation, &
all associated code carry the GNU Affero General Public License
v3.0 (only). You may use, modify, & distribute under those terms.
No proprietary relicensing exists.
README's License section quotes the same clause and links the
whitepaper as the canonical source.
Source: https://unfirehose.com/downloads/merkle-reverse-rag-whitepaper.pdf
Adds a "Get the source" section with the GitLab SSH + HTTPS clone
URLs, then platform-specific prerequisite blocks:
- macOS: Xcode CLT for git/make, Homebrew for python@3.12
- Ubuntu/Debian: apt install python3/python3-venv/build-essential
/curl/bzip2; 22.04 has 3.10, 24.04 has 3.12
- Windows: WSL2 Ubuntu-24.04 is the supported path. Native cmd
+ Git Bash mostly works for Python but several make targets call
bash idioms. `wsl --install -d Ubuntu-24.04` from an admin shell.
- OpenBSD: pkg_add git python-3.12 gmake curl. Default `make` is
BSD make; the project uses GNU-make features so substitute
`gmake` everywhere.
Bootstrap step (make bootstrap) is unchanged — it stays at the end
of the Setup section once the platform prereqs are in place.
aborist/sources/grok.py exposes two Source subclasses for ingesting
xAI's user data export:
GrokExportSource walks ttl/<period>/export_data/<user-id>/ and
reads prod-grok-backend.json. Yields one Document
per conversation with:
- URI: grok://conversation/<conversation-id>
- title: the conversation's auto-generated title
- content: full message text in turn order
source_type='grok_export'.
GrokMediaPostsSource same export, but yields per media-generation
post (image/video prompts) under URI
grok://media/<post-id>.
source_type='grok_media'.
Both auto-walk down from the export root so callers can pass the
top-level directory xAI delivered (e.g., ~/Downloads/<user-uuid>/).
Wired into the CLI: aborist ingest --source {grok_export,grok_media}
accepts --path <export-root>. tests/test_grok_source.py covers the
walk + parse + Document shape with a fabricated mini-export fixture.
Once ingested, conversations become normal queryable docs in the
shard cluster — your prior chats become memory the corpus can
consult during RAG.
79 tests passing.
Replaces the stub README with a working user guide:
- 30-second quickstart that lands a queryable corpus in ~3 min
- Setup section (Python 3.10+, make bootstrap creates the venv +
installs the package in editable mode)
- Wikipedia 2003-05-16 dataset:
fetch-cur / fetch-old / fetch what each downloads & where
ingest-cur-attached SHARDS=N sharded ingest, the canonical
path (no WAL contention)
ingest-cur-parallel / ingest-cur single-DB variants
--resume rsync-style restart
- Distillation (first-sentence + TF-IDF cores) and what they buy
- Personal Grok export ingest (point GROK_EXPORT at the xAI bundle)
- OpenAI export placeholder section (one new Source subclass; the
pattern matches sources/grok.py)
- Asking the corpus: query path explained briefly, 8-dim cache key,
per-phase timings, LLM endpoint env vars
- Falsification: how to mark a wrong cached answer stale
- Inspection: stats-shards / analyze-shards / verify-shards / activity
- Architecture tree (one screen)
- Pointer to source papers (whitepaper + v7 substrate)
Reflects the current shipped feature set including the three
retrieval relevance signals (title + TF-IDF cores + body density),
concept overlay, sharded distill, falsification, activity timeline,
and per-phase timings.
The "permacomputer" query fell into title-relevance's empty-fallback
because "permacomputer" appears nowhere in any title. Fallback was
top-1 by raw BM25, getting one tangential source. Same problem for
any user-coined term: title-LIKE alone can't find it.
_filter_by_title_relevance now grows three accept paths:
1. Title-token overlap (existing).
2. TF-IDF core keyword overlap. A doc's distinctive low-frequency
terms get distilled into its tfidf-keywords-v1 core. A query for a
rare term ("permacomputer", "unturf") hits docs whose core lists
it as a keyword even when no title does.
3. Body density. Docs that mention a query token >= 3 times pass
regardless of title or core. Cheap proxy for "actually about it."
Empty-all-three fallback widens from top-1 to top-N (default 5) — the
LLM gets enough context to honestly say "I don't know" rather than
fabricating from a single tangential source.
Implementation:
- _docs_with_core_keyword_match: SQL join chunks -> documents -> derivations
finds source docs whose TF-IDF core content matches a token. Folded
into the per-shard search loop alongside FTS5 + title-LIKE.
- _body_density_passes: counts case-insensitive occurrences in
concatenated chunk content. Threshold 3.
- _search_corpus stashes core_match_roots + root_to_shard on the
returned hits list as side-channel info for _rerank.
Makefile: distill-shards-tfidf-parallel for the per-shard TF-IDF run.
Demo on the live cluster (after TF-IDF distill of 95k cur surface docs):
Q "tell me about permacomputer ?"
sources: MOAD Cheat Sheet + Unturf AI Infrastructure Expansion
answer: synthesized from both, bounds coverage honestly
Q "what is unturf about ?"
sources: Unturf Trademark + Unturf AI Infrastructure Expansion
answer: "grassroots, community-driven initiative ... ethical AI
hardware, software, and networks"
Both queries now find user-coined terms via TF-IDF cores even though
the terms never appear in any title.
79 tests passing.
Two issues surfaced by "tell me about permacomputer ?":
1. Stopword leak. _FTS5_STOPWORDS got "tell" added earlier but the
parallel _TITLE_STOPWORDS in qa/query.py wasn't synced. Title-LIKE
search picked up "tell" and returned every Tell_* article in the
corpus (Tell_(poker), Tell-Tale_Heart, Tell_Township, Tell_City,
etc.) all uniformly scored at 70.0. Synced the lists — title
tokenizer now drops the same question-shaping verbs as FTS5.
2. Grok docs lived only in --db (default ~/.aborist/aborist.db),
not in --shards-dir. `make query` reads the shard cluster, so
Grok was invisible to it. Added two targets:
ingest-grok-attached conversations -> $(SHARDS_DIR)/grok.db
ingest-grok-media-attached media prompts -> $(SHARDS_DIR)/grok.db
Both use --resume so re-runs are cheap.
Demo: "tell me about permacomputer ?" now answers from the
"MOAD Cheat Sheet: Common Software Defects" Grok conversation —
fox's own prior chat surfaced as corpus.
Note: 000.db already had Grok docs from an earlier ingest variant.
The cross-shard UNION currently sees them twice (376 rows, 188
unique document_roots). Retrieval is unaffected because
_search_corpus dedupes by root, but stats counts are inflated.
Cleanup is destructive (drop one copy) — fox decides which.
79 tests passing.
Even with rivalry exclusion, sources like 'CPU_monthly' (a roundup
discussing both AMD and Intel CPUs in the same article) sneak through
because their title matches "CPU" without naming a competitor. Hermes
then conflated facts: "the fastest AMD CPU is the 3.2 GHz Pentium 4."
Tightened the DEFAULT_QUERY_POLICY system prompt with four explicit
rules:
1. A single source may discuss multiple competitors. Attribute facts
only to the entity the source explicitly names.
2. If the question asks about ONE entity (e.g., "fastest AMD CPU"),
do not include competitor facts as if they were facts about the
asked entity.
3. AMD PR-rating numbers (3200+, 2500+) are NOT clock speeds in MHz.
Quote the source's own wording rather than interpreting model
numbers.
4. Say "I don't know based on the provided sources" when the sources
don't cover the answer. Don't speculate.
Demo on the same AMD query (post-distill, cores now in retrieval):
before: "the fastest AMD CPU is the 3.2 GHz Pentium 4 from May 2003"
after: "the fastest AMD CPU mentioned is the Athlon XP 3200+ which
was released in May 2003. The sources do not contain
information about CPUs released after 2003."
Sources include AMD, AMD_5x86, AMD_Duron + their cores, CPU_design,
CPU_monthly. Hermes correctly attributes Athlon XP 3200+ to AMD and
honestly bounds the corpus's coverage at 2003.
Bumping the system prompt invalidates the governance_policy_hash and
therefore every prior cache_key — by design, per v9.8 admissibility
(prompt change = different governance = fresh inference).
66 tests still passing.
