aborist/sources/grok.py exposes two Source subclasses for ingesting
xAI's user data export:
GrokExportSource walks ttl/<period>/export_data/<user-id>/ and
reads prod-grok-backend.json. Yields one Document
per conversation with:
- URI: grok://conversation/<conversation-id>
- title: the conversation's auto-generated title
- content: full message text in turn order
source_type='grok_export'.
GrokMediaPostsSource same export, but yields per media-generation
post (image/video prompts) under URI
grok://media/<post-id>.
source_type='grok_media'.
Both auto-walk down from the export root so callers can pass the
top-level directory xAI delivered (e.g., ~/Downloads/<user-uuid>/).
Wired into the CLI: aborist ingest --source {grok_export,grok_media}
accepts --path <export-root>. tests/test_grok_source.py covers the
walk + parse + Document shape with a fabricated mini-export fixture.
Once ingested, conversations become normal queryable docs in the
shard cluster — your prior chats become memory the corpus can
consult during RAG.
79 tests passing.
aborist [--shards-dir DIR] providence --falsify CACHE_KEY \
--state {failed,stale,quarantined} --reason "X" --by-actor NAME
Searches every shard for the cache_key (it lives in exactly one),
flips falsification_state on the providence_cache row, appends a row
to the falsifications log, and writes a 'falsify' audit event so the
chain captures the act with from_state/to_state/reason/by_actor.
Future cache lookups skip records whose state != 'live', so a
falsified answer never gets reused. The record itself is preserved
(history matters); only its admissibility changes.
Makefile target:
make falsify KEY=<hex> REASON='why' [STATE=failed]
Demo on the live shard cluster: marked the AMD/Pentium-4 conflation
answer as failed, verified state flipped + audit row recorded +
falsifications log captured fox as actor + the Barton-clock-vs-PR
explanation as reason. Re-asking the same question generated a fresh
cache_key (corpus state shifted between attempts) so a new LLM call
ran — exactly the v9.8 admissibility behavior: each (8-dim cache_key)
is independently live/failed.
66 tests still passing.
New CLI: aborist [--shards-dir DIR] activity [--limit N] [--since-seconds S]
[--preview-chars N]
Aggregates across all shards into a single JSON document with:
shards the SQLite files queried
corpus counts: documents_total/surface/core,
providence_total/live/stale/failed,
audit_events_total
recent_qa providence_cache rows (cache_key, question, answer
preview, hit_count, state, last_hit_at) newest first
recent_ingests audit events of type=ingest with body's document_uri,
source_type, chunks, supersedes link
recent_derives audit events of type=derive (distillation results,
compression_ratio, compression_depth)
recent_falsifications records marked failed/stale/quarantined,
with reason and by_actor
--limit caps each list independently. --since-seconds filters to recent
events only. --preview-chars trims long answers.
Designed for an agent to read before deciding the next action — what
was just asked, what just got integrated, what's the current state.
Makefile target: make activity ACTIVITY_LIMIT=N
66 tests still passing.
aborist/qa/query.py exposes query() — the user-facing RAG flow:
1. FTS5 search across all shards (chunks_fts can't be UNION'd as a
view, so each shard's index is queried independently and merged
by score).
2. Top-K distinct documents are selected within a max-context-chars
budget (default 60 KB so a 768-token response fits Hermes-3's
82 K context window comfortably).
3. context_root = Merkle root over the sorted source document_roots.
That's the v9.8 'source' dimension for multi-source answers —
a verifier can recompute it from the listed source roots.
4. 8-dim cache_key over (context_root, question_hash, model_profile,
conversation, governance_policy, schema, canonicalization,
chunking). Hit returns STRICT immediately; miss calls Hermes and
persists.
CLI: aborist [--shards-dir DIR] query "<question>"
Default qa_db is <shards-dir>/qa.db (or ~/.aborist/qa.db). Uses the
same OpenAICompatibleClient/StubClient as `ask`. --dry-run skips the
LLM and returns context-only.
Search escape fix: the prior FTS5 escape ANDed every token including
stopwords + punctuation, so "What is anarcho-capitalism?" required
the doc to literally contain "what" + "is" + "anarcho-capitalism?" —
zero hits. New tokenizer drops stopwords + punctuation and ORs the
remaining content tokens; BM25 ranks the multi-token matches highest.
Live demo against the 122k-doc 4-shard cluster:
Q "What is anarcho-capitalism?" 6.1 s wall miss / 0.45 s cache hit
Q "Who was George Washington?" 10.3 s wall miss
Both answers cite the source URIs Hermes was given.
57 tests passing (4 new query tests covering search → context →
cache → audit chain).
aborist/progress.py — stdlib-only rate-limited stderr reporter.
Periodic lines (default every 2s) showing:
elapsed | seen | inserted | docs/s now | docs/s avg | percent | ETA
Wired into ingest_source via a `progress` parameter; CLI default-on
with --quiet to suppress and --total-estimate N to enable percent/ETA.
Demo on a 5k-doc ingest with --progress-interval 0.8:
[ 1s] | 200 seen | 200 new | 193 now | ( 193 avg) docs/s | 4.0% | ETA 24s
[ 7s] | 3,200 seen | 3,200 new | 458 now | ( 424 avg) docs/s | 64.0% | ETA 3s
[ 12s] | 5,001 seen | 5,000 new | 392 now | ( 413 avg) docs/s | 100.0% | ETA 0s
bench/run.sh + `make bench` — reproducible 5000-doc workload through
three configs:
serial single process, single SQLite
parallel-shared N shards, one shared SQLite (WAL serialized)
attached N shards, per-shard SQLite (true parallel writes)
Output is a one-shot table plus CSV at /tmp/aborist-bench/results.csv
so the ratchet is visible as we keep optimizing. Override via
BENCH_DOCS=N and SHARDS=N.
Latest baseline (this commit, on this machine):
config wall_s docs docs/s
serial 11.74 5000 425.7
parallel-shared 10.39 5000 481.1
attached 7.99 5000 625.5
53 tests still passing.
Resume (rsync-style)
Each shard DB gains a `meta` table. After every successful batch
flush, ingest_source persists `source_high_water:<source_type>` ->
the largest row id seen (cur_id or old_id). On --resume, the source
reads it back and skips rows whose id is <= the mark, so an ingest
killed at any point can be re-run cheaply: already-cached docs are
fast-forwarded past without re-hashing or DB writes.
WikipediaSqlDump now exposes `start_id` (skip threshold) and
`last_id` (running max). cur_id is surfaced in Document.extra
alongside old_id so both tables behave the same.
CLI: aborist ingest --resume
Demo on shared DB (cur):
Round 1: --limit 3000 --resume high_water = 5714
Round 2: --limit 5000 --resume skips 1..5714, picks up at 5715
Round 3: --limit 100 --resume skips 1..15362, ingests 100 more
Always idempotent on re-run; no dups, no missing docs.
Per-shard audit chain integrity for sharded analyze
Cross-shard analyze previously reported nonsense breaks counts —
each shard owns its own audit chain (genesis -> ... -> latest), and
the UNION view interleaves them so cross-shard transitions look
like break events. The fix: when --shards-dir is set, open each
shard's DB directly and run _check_audit_chain on it, then
aggregate.
Output now reads:
"audit_chain": {
"events": <total>,
"breaks": 0,
"shards": [{"shard": "000.db", "events": N, "breaks": 0}, ...]
}
53 tests passing. Tests cover: high-water write, skip-on-resume,
idempotency across two resume runs, kill-and-resume continuity.
aborist --shards-dir DIR distill iterates over each shard's DB and
runs distill_existing on it. Cores live in the same shard as their
source surface so the per-shard derivation chain stays
self-contained.
Per-shard processing also enables trivial parallelism via the
distill-shards-parallel Makefile target — one process per shard, no
DB contention because each shard is its own file.
End-to-end demo on 4-way 10k cur ingest:
ingest: 17.6 s wall (4 parallel)
distill: 6.2 s sequential per-shard (9185 cores from 10000 surface)
verify: 30/30 random Merkle proofs across shards
Aggregated counters cover all shards plus per-shard breakdown in the
output JSON so the user sees both the union and the parts.
Note: cross-shard analyze reports an inflated audit_chain.breaks count
because each shard owns its own chain genesis -> ... -> latest, but the
UNION view sees them interleaved. Fix coming next.
Each ingest shard owns its own SQLite file. N processes write in
true parallel (no WAL writer-lock contention). Cross-shard reads
attach all shards as UNION ALL views over the standard tables, so
existing query code (`SELECT * FROM documents`) works unchanged.
CLI:
aborist ingest --shards-dir DIR --shard 0/4 # write to DIR/000.db
aborist --shards-dir DIR stats # read across all shards
aborist --shards-dir DIR analyze # ditto
aborist --shards-dir DIR verify -n 20 # Merkle round-trips
# on a sample drawn
# from all shards
Makefile:
SHARDS_DIR=~/.aborist/shards
ingest-cur-attached SHARDS=4 ingest-old-attached SHARDS=4
stats-shards cross-shard stats
Implementation lives in store.connect_query():
- opens a fresh in-memory connection
- ATTACH DATABASE per shard file under aliases sh000, sh001, ...
- CREATE TEMP VIEW <table> AS UNION ALL SELECT * FROM <alias>.<table>
for the eight shardable tables (documents, chunks, merkle_nodes,
edges, derivations, providence_cache, audit_events, falsifications)
- exposes a _shards temp table for tools that want the shard list
Benchmark (20k cur, 4 cores):
Phase 1a serial 55.3 s
Phase 1b shared-WAL 43.1 s
Phase 2 attached-forever 40.1 s (no WAL contention)
The 7% gap vs Phase 1b is bz2-decompression duplication: each shard
decodes the full dump independently. Architectural wins are bigger
than that number — write isolation, per-shard crash recovery,
audit chain stays per-shard checkable, shards can migrate to other
disks, new shards are pure append (never rewrite existing files).
Merge cost remains: zero. `ls $SHARDS_DIR/*.db` is the registry.
Workers each parse the full SQL stream but only emit rows whose index
satisfies (idx % total == rank). N processes cooperate against a
single shared SQLite file; WAL+synchronous=NORMAL handles writer
contention via BEGIN IMMEDIATE serialization.
Implementation chose OS-level processes over in-process mp.Queue: the
gain ceiling is identical (writer is the bottleneck either way) and
the surface area is one new flag plus stride-filter logic instead of
a queue-based pipeline with picklable workers.
Makefile gains:
ingest-cur-parallel SHARDS=4
ingest-old-parallel SHARDS=4
Benchmark on 20k cur articles:
serial 55.3 s 361 docs/s 1.0x CPU usage
4-way parallel 43.1 s 464 docs/s ~3.0x CPU usage 1.28x wall
Both runs landed 19,999 unique documents — correctness verified.
The write-lock is now the bottleneck; further gains require Phase 2
attach-forever sharding (each shard owns its own SQLite file, queries
fan out via ATTACH DATABASE, zero merge cost).
Profile of the prior implementation (cProfile on 2000-doc ingest):
total 23.6s
parser 20.1s (85% of time, 27.7M list.append calls)
DB writes 2.9s
Three changes compound:
1. PRAGMA pack in store.connect():
synchronous=NORMAL skip per-commit fsync (durable under WAL replay)
cache_size=-65536 64 MB page cache
temp_store=MEMORY temp tables in RAM
mmap_size=256 MB page-cache reads bypass read() syscall
2. Batched transactions in ingest_source and distill_existing:
200 docs/tx default, --batch-size CLI flag.
One BEGIN/COMMIT per batch instead of per doc.
Audit chain SELECT-then-INSERT still chains correctly because
SQLite reads its own uncommitted writes within a transaction.
Idempotent on collision (INSERT OR IGNORE / pre-existence check)
so partial-batch re-runs are safe.
3. Parser rewrite (the actual bottleneck): replaced char-by-char
list-append loops with str.find / string slicing.
_decode_mysql_string fast-path: if no '\\\\' in input, return as-is.
_split_values_tuples uses _find_string_end (str.find with
backslash-parity check) instead of per-char scanning.
_iter_insert_statements buffered scan (str.find for ';' and "'")
crossing read-chunk boundaries via in_string flag.
~13x fewer Python function calls, semantics identical.
_upsert_edges now batches via executemany.
After: 2000 docs in 6.0s; 5000 old revisions in 12.8s wall (was 32.3s).
Throughput 85 -> 333 docs/sec. 49 tests still passing.
Cheaper to fail = cheaper to experiment.
analyze emits a compression-spectrum report:
compression_depth_histogram surface vs depth=1, 2, ...
distillers per-process compression ratio mean/min/max
documents_by_source_kind source/kind crosstab
chunks_by_tier hot/warm/cold distribution
audit_chain event count + integrity (breaks must be 0)
gravity_top_inbound the 'gravity wells' of the forest
Default LLM endpoint moves from uncloseai.com (marketing site, no /v1)
to hermes.ai.unturf.com/v1 — the actual reachable Hermes-3 inference
endpoint serving adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic on vLLM
with an 82K context window. uncloseai.com still works as a brand
override via ABORIST_LLM_ENDPOINT.
The 2003-05-16 archive ships three files:
20030516_cur_tablesql.bz2 82 MB current snapshot (single revision/page)
old_tablesqlbz2.1 640 MiB \
old_tablesqlbz2.2 252 MiB / split halves of old (full revision
history). Concatenate before bzcat.
Generalize the parser:
WikipediaSqlDump(table='cur'|'old') — shared statement parser, single
column-position contract for the
first 4 fields (id/ns/title/text)
WikipediaCurDump — back-compat wrapper, table='cur'
WikipediaOldDump — new, table='old'; old has no is_redirect, every
revision is real
Old rows surface old_id and old_timestamp via Document.extra so a
downstream pass can sort revisions chronologically before re-ingesting
through the supersedes-edge path.
Makefile gains:
fetch-cur / fetch-old / fetch (both)
ingest-cur / ingest-old / ingest (cur default)
WP_OLD target concatenates the two split parts
CLI ingest --source now accepts wikipedia_cur or wikipedia_old.
Smoke (real dump): 5 revisions of "AtlasShrugged/Companies" yielded
correctly with old_id=2..10, timestamps from January 2002.
aborist/qa/ implements the cache-first answer flow from the providence
whitepaper, scaled up to v9.8's full 8-dim admissibility invariant.
cache_key = SHA-256 of:
source_root | question_hash | model_profile_hash | conversation_hash
| governance_policy_hash | schema_version | canonicalization_version
| chunking_version
Any drift in any dimension yields a distinct cache_key — prior records
cannot serve. Falsification states (failed/stale/quarantined) gate
every cache hit.
- qa/keys.py — pure hash functions, deterministic & testable
- qa/client.py — ChatClient Protocol + StubClient + OpenAI-compatible
HTTP client (vllm/llama.cpp/uncloseai compatible)
- qa/runner.py — ask(): lookup -> hit (no LLM call, hit_count++)
OR miss (call client, write record, audit event,
proof binds answer to source root)
- cli.py — `aborist ask` and `aborist providence` subcommands
- pyproject.toml — httpx promoted from extras to core (used by both
html and qa); selectolax stays in [html] extras
Smoke (StubClient, no network): cache miss writes record with chunk_0
Merkle proof reconstructing source_root; cache hit returns same record
without calling client; 1085 audit events chained 0 breaks across
ingest/derive/evict/rehydrate/providence_write.
Distillers now scan kind='surface' OR kind='core'. Cores generate
deeper cores with compression_depth incremented. Each round of
recursion tightens the planet toward its center.
Audit body for derive events now records src_kind and the resulting
compression_depth so the chain reflects the layer transition.
Smoke against the existing 503 surface + 478 depth=1 core corpus
produced N deeper cores with proofs still binding back to their
depth=1 sources.
