aborist/sources/grok.py exposes two Source subclasses for ingesting
xAI's user data export:
GrokExportSource walks ttl/<period>/export_data/<user-id>/ and
reads prod-grok-backend.json. Yields one Document
per conversation with:
- URI: grok://conversation/<conversation-id>
- title: the conversation's auto-generated title
- content: full message text in turn order
source_type='grok_export'.
GrokMediaPostsSource same export, but yields per media-generation
post (image/video prompts) under URI
grok://media/<post-id>.
source_type='grok_media'.
Both auto-walk down from the export root so callers can pass the
top-level directory xAI delivered (e.g., ~/Downloads/<user-uuid>/).
Wired into the CLI: aborist ingest --source {grok_export,grok_media}
accepts --path <export-root>. tests/test_grok_source.py covers the
walk + parse + Document shape with a fabricated mini-export fixture.
Once ingested, conversations become normal queryable docs in the
shard cluster — your prior chats become memory the corpus can
consult during RAG.
79 tests passing.
A small knowledge-graph layer that does two distinct jobs:
1. SYNONYM_GROUPS broaden retrieval. A query mentioning "Athlon"
now also matches AMD-titled docs because Athlon IS an AMD product.
Groups currently cover AMD-family, Intel-family, HTTP family,
FTP, Mac, Windows, Linux. Easy to extend.
2. RIVALRIES narrow retrieval. The pair (AMD-group, Intel-group)
means: if the query mentions one side and not the other, drop
docs whose titles contain the OTHER side's tokens. So a "fastest
AMD CPU" question never gets Pentium_4 in the context — even if
FTS5 BM25 ranks it high — because Pentium is in the Intel group
and Intel isn't in the query.
COMPARE_WORDS ("vs", "versus", "compare", "between", ...) suppress
the exclusion. "compare AMD vs Intel" keeps both sides. "what is
the fastest AMD CPU?" does not.
Demos against the 128k Wikipedia 2003-05-16 cur shards:
Q "fastest AMD CPU" → 8 sources, all AMD/CPU titled, NO Intel
Answer: "Athlon XP 3200+" (real AMD chip,
grounded in the AMD article)
Q "compare AMD vs Intel"
→ 8 sources, mix of Intel_8028x, Intel_8048x,
AMD_Duron — both sides preserved
Q "Athlon processor" → AMD, AMD_Duron, AMD_5x86, Athlon all
surfaced via synonym expansion
Phase 1 implementation hand-curates the groups; Phase 2 idea is to
derive them from Wikipedia's link/category graph (dense bidirectional
clusters → synonym groups; same-category-without-cross-links →
rivalry candidates).
66 tests passing.
aborist/qa/query.py exposes query() — the user-facing RAG flow:
1. FTS5 search across all shards (chunks_fts can't be UNION'd as a
view, so each shard's index is queried independently and merged
by score).
2. Top-K distinct documents are selected within a max-context-chars
budget (default 60 KB so a 768-token response fits Hermes-3's
82 K context window comfortably).
3. context_root = Merkle root over the sorted source document_roots.
That's the v9.8 'source' dimension for multi-source answers —
a verifier can recompute it from the listed source roots.
4. 8-dim cache_key over (context_root, question_hash, model_profile,
conversation, governance_policy, schema, canonicalization,
chunking). Hit returns STRICT immediately; miss calls Hermes and
persists.
CLI: aborist [--shards-dir DIR] query "<question>"
Default qa_db is <shards-dir>/qa.db (or ~/.aborist/qa.db). Uses the
same OpenAICompatibleClient/StubClient as `ask`. --dry-run skips the
LLM and returns context-only.
Search escape fix: the prior FTS5 escape ANDed every token including
stopwords + punctuation, so "What is anarcho-capitalism?" required
the doc to literally contain "what" + "is" + "anarcho-capitalism?" —
zero hits. New tokenizer drops stopwords + punctuation and ORs the
remaining content tokens; BM25 ranks the multi-token matches highest.
Live demo against the 122k-doc 4-shard cluster:
Q "What is anarcho-capitalism?" 6.1 s wall miss / 0.45 s cache hit
Q "Who was George Washington?" 10.3 s wall miss
Both answers cite the source URIs Hermes was given.
57 tests passing (4 new query tests covering search → context →
cache → audit chain).
Resume (rsync-style)
Each shard DB gains a `meta` table. After every successful batch
flush, ingest_source persists `source_high_water:<source_type>` ->
the largest row id seen (cur_id or old_id). On --resume, the source
reads it back and skips rows whose id is <= the mark, so an ingest
killed at any point can be re-run cheaply: already-cached docs are
fast-forwarded past without re-hashing or DB writes.
WikipediaSqlDump now exposes `start_id` (skip threshold) and
`last_id` (running max). cur_id is surfaced in Document.extra
alongside old_id so both tables behave the same.
CLI: aborist ingest --resume
Demo on shared DB (cur):
Round 1: --limit 3000 --resume high_water = 5714
Round 2: --limit 5000 --resume skips 1..5714, picks up at 5715
Round 3: --limit 100 --resume skips 1..15362, ingests 100 more
Always idempotent on re-run; no dups, no missing docs.
Per-shard audit chain integrity for sharded analyze
Cross-shard analyze previously reported nonsense breaks counts —
each shard owns its own audit chain (genesis -> ... -> latest), and
the UNION view interleaves them so cross-shard transitions look
like break events. The fix: when --shards-dir is set, open each
shard's DB directly and run _check_audit_chain on it, then
aggregate.
Output now reads:
"audit_chain": {
"events": <total>,
"breaks": 0,
"shards": [{"shard": "000.db", "events": N, "breaks": 0}, ...]
}
53 tests passing. Tests cover: high-water write, skip-on-resume,
idempotency across two resume runs, kill-and-resume continuity.
The 2003-05-16 archive ships three files:
20030516_cur_tablesql.bz2 82 MB current snapshot (single revision/page)
old_tablesqlbz2.1 640 MiB \
old_tablesqlbz2.2 252 MiB / split halves of old (full revision
history). Concatenate before bzcat.
Generalize the parser:
WikipediaSqlDump(table='cur'|'old') — shared statement parser, single
column-position contract for the
first 4 fields (id/ns/title/text)
WikipediaCurDump — back-compat wrapper, table='cur'
WikipediaOldDump — new, table='old'; old has no is_redirect, every
revision is real
Old rows surface old_id and old_timestamp via Document.extra so a
downstream pass can sort revisions chronologically before re-ingesting
through the supersedes-edge path.
Makefile gains:
fetch-cur / fetch-old / fetch (both)
ingest-cur / ingest-old / ingest (cur default)
WP_OLD target concatenates the two split parts
CLI ingest --source now accepts wikipedia_cur or wikipedia_old.
Smoke (real dump): 5 revisions of "AtlasShrugged/Companies" yielded
correctly with old_id=2..10, timestamps from January 2002.
aborist/qa/ implements the cache-first answer flow from the providence
whitepaper, scaled up to v9.8's full 8-dim admissibility invariant.
cache_key = SHA-256 of:
source_root | question_hash | model_profile_hash | conversation_hash
| governance_policy_hash | schema_version | canonicalization_version
| chunking_version
Any drift in any dimension yields a distinct cache_key — prior records
cannot serve. Falsification states (failed/stale/quarantined) gate
every cache hit.
- qa/keys.py — pure hash functions, deterministic & testable
- qa/client.py — ChatClient Protocol + StubClient + OpenAI-compatible
HTTP client (vllm/llama.cpp/uncloseai compatible)
- qa/runner.py — ask(): lookup -> hit (no LLM call, hit_count++)
OR miss (call client, write record, audit event,
proof binds answer to source root)
- cli.py — `aborist ask` and `aborist providence` subcommands
- pyproject.toml — httpx promoted from extras to core (used by both
html and qa); selectolax stays in [html] extras
Smoke (StubClient, no network): cache miss writes record with chunk_0
Merkle proof reconstructing source_root; cache hit returns same record
without calling client; 1085 audit events chained 0 breaks across
ingest/derive/evict/rehydrate/providence_write.
When a re-ingest produces a different document_root for an already-seen
URI, both versions now coexist in the store. A 'supersedes' edge from
the new doc to the prior one keeps the lineage addressable, and the
ingest audit body records the supersedes link.
Idempotent re-ingest is unchanged: identical bytes -> identical root ->
no new doc, no supersedes edge.
Three-version chains test that v3->v2 and v2->v1 edges form a walkable
history. The old version stays queryable (its chunks may later be
evicted to cold for storage savings while remaining provable).
Pure-Python (stdlib only). Uses chunk-level corpus baseline so terms
concentrated in fewer chunks outrank common ones. Default top-K = 16.
Output cores are comma-separated keyword lists — extreme compression
toward the tweet/haiku end of the planet metaphor. Same source can
now carry both a first-sentence-v1 core AND a tfidf-keywords-v1 core,
each derived independently and Merkle-signed back to the same surface.
The 'contributing_chunk_indices' for TF-IDF is every chunk that
contains at least one of the top-K keywords — proof binding remains
honest and cryptographically tight.
Distillers now scan kind='surface' OR kind='core'. Cores generate
deeper cores with compression_depth incremented. Each round of
recursion tightens the planet toward its center.
Audit body for derive events now records src_kind and the resulting
compression_depth so the chain reflects the layer transition.
Smoke against the existing 503 surface + 478 depth=1 core corpus
produced N deeper cores with proofs still binding back to their
depth=1 sources.
