README: full setup + dataset + grok + openai-placeholder walkthrough
Replaces the stub README with a working user guide:
- 30-second quickstart that lands a queryable corpus in ~3 min
- Setup section (Python 3.10+, make bootstrap creates the venv +
installs the package in editable mode)
- Wikipedia 2003-05-16 dataset:
fetch-cur / fetch-old / fetch what each downloads & where
ingest-cur-attached SHARDS=N sharded ingest, the canonical
path (no WAL contention)
ingest-cur-parallel / ingest-cur single-DB variants
--resume rsync-style restart
- Distillation (first-sentence + TF-IDF cores) and what they buy
- Personal Grok export ingest (point GROK_EXPORT at the xAI bundle)
- OpenAI export placeholder section (one new Source subclass; the
pattern matches sources/grok.py)
- Asking the corpus: query path explained briefly, 8-dim cache key,
per-phase timings, LLM endpoint env vars
- Falsification: how to mark a wrong cached answer stale
- Inspection: stats-shards / analyze-shards / verify-shards / activity
- Architecture tree (one screen)
- Pointer to source papers (whitepaper + v7 substrate)
Reflects the current shipped feature set including the three
retrieval relevance signals (title + TF-IDF cores + body density),
concept overlay, sharded distill, falsification, activity timeline,
and per-phase timings.
This commit is contained in:
parent
0192255f7d
commit
24d8204c08
1 changed files with 180 additions and 20 deletions
200
README.md
200
README.md
|
|
@ -2,32 +2,192 @@
|
|||
|
||||
An arborist for trees and forests of cross-linked information.
|
||||
|
||||
Aborist ingests documents into a content-addressed, Merkle-committed store. Every
|
||||
document carries a URI for backtrack and cross-link. Search hooks return hits with
|
||||
declared audit modes (`STRICT` / `HYBRID` / `VISUAL`) so callers never overclaim
|
||||
what a result actually proves.
|
||||
Aborist ingests documents into a content-addressed, Merkle-committed SQLite store, distills them into recursive cores, and answers questions over the resulting corpus via an OpenAI-compatible LLM. Every cached answer carries a verifiable Merkle proof tying it back to its source documents — Merkle-AGI v9.8 / Merkle Providence Reverse RAG, runnable end-to-end.
|
||||
|
||||
## Design
|
||||
## What this gets you
|
||||
|
||||
Aborist implements the runtime spec from Merkle-AGI v9.8 / Merkle Providence
|
||||
Reverse RAG. The 8-dim admissibility key (`source_root`, `question_hash`,
|
||||
`model_profile_hash`, `conversation_hash`, `governance_policy_hash`,
|
||||
`schema_version`, `canonicalization_version`, `chunking_version`) plus
|
||||
falsification state ensures cached records are never reused under drift.
|
||||
```
|
||||
make bootstrap # one-time: venv + deps
|
||||
make fetch-cur # download Wikipedia 2003-05-16 (~82 MB)
|
||||
make ingest-cur-attached # ~3 min — 128k articles, 4 parallel shards
|
||||
make distill-shards-parallel
|
||||
make distill-shards-tfidf-parallel
|
||||
make query Q="What is anarcho-capitalism?"
|
||||
```
|
||||
|
||||
Two layered document kinds:
|
||||
- **surface** — diverse ingested content (full chunks, FTS-indexed)
|
||||
- **core** — distilled records (haiku/equation/snippet) Merkle-signed back to
|
||||
source surface roots via `derivations`
|
||||
After the last command, a Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer with a cryptographic proof of its sources. Repeat the same question and a STRICT-mode cache hit replays in ~100 ms.
|
||||
|
||||
Reversible eviction via chunk `tier` ∈ {`hot`, `warm`, `cold`}. Cold = leaf hash
|
||||
+ URI only; rehydratable from URI, identity verifiable via leaf hash.
|
||||
## Setup
|
||||
|
||||
## Quick start
|
||||
Requirements: Python 3.10+, `bzip2`, `curl`, `make`. SQLite 3.35+ (CPython's bundled sqlite3 is fine).
|
||||
|
||||
make all # bootstrap + fetch + ingest + verify + stats
|
||||
make search Q='…'
|
||||
```
|
||||
make bootstrap
|
||||
```
|
||||
|
||||
Creates `.venv/`, installs the package in editable mode with the `[dev,html]` extras, and exposes `aborist` at `.venv/bin/aborist`. No system-wide install. Re-running `make bootstrap` is a no-op if the venv is up to date.
|
||||
|
||||
After bootstrap, every workflow lives behind a `make` target. Run `make help` to list them.
|
||||
|
||||
## Data: Wikipedia 2003-05-16
|
||||
|
||||
The 2003 dataset lives at <https://dumps.wikimedia.org/archive/2003/2003-05-16/en/> as three files:
|
||||
|
||||
| file | size | what |
|
||||
|---|---|---|
|
||||
| `20030516_cur_tablesql.bz2` | 82 MB | one row per article: snapshot of every Wikipedia page on 2003-05-16 |
|
||||
| `old_tablesqlbz2.1` | 640 MiB | first half of the revision-history table (split bzip2 stream) |
|
||||
| `old_tablesqlbz2.2` | 252 MiB | second half — `cat` them together to decompress |
|
||||
|
||||
```
|
||||
make fetch-cur # snapshot only (~82 MB on disk)
|
||||
make fetch-old # full history (~1.4 GB on disk after concat)
|
||||
make fetch # both
|
||||
```
|
||||
|
||||
Files land in `data/`. Re-running is idempotent (curl skips if already present).
|
||||
|
||||
### Sharded ingest (the canonical path)
|
||||
|
||||
Per-shard SQLite files, no write-lock contention. Each shard process owns its own DB; cross-shard reads attach all shards as `UNION ALL` views.
|
||||
|
||||
```
|
||||
make ingest-cur-attached SHARDS=4 # cur snapshot, 4 parallel shards (~3 min)
|
||||
make ingest-old-attached SHARDS=4 # full history, ~30–40 min
|
||||
```
|
||||
|
||||
Shards land in `~/.aborist/shards/`. Override with `SHARDS_DIR=/path/to/somewhere`.
|
||||
|
||||
### Resumable
|
||||
|
||||
Add `--resume` (or just re-run the make target — `--resume` is the default for attached ingests). Each shard tracks its own high-water mark in `meta`; an interrupted ingest picks up where it left off without re-hashing already-cached docs.
|
||||
|
||||
### Single-DB ingest (simpler, smaller)
|
||||
|
||||
For experiments under a few thousand docs, a single SQLite file is fine:
|
||||
|
||||
```
|
||||
make ingest-cur INGEST_LIMIT=1000 # one DB at $(DB), default ~/.aborist/aborist.db
|
||||
make ingest-cur-parallel SHARDS=4 # 4 processes, one shared DB (WAL serialized)
|
||||
```
|
||||
|
||||
### Distillation (cores feed retrieval)
|
||||
|
||||
Two distillers ship: `first-sentence-v1` (one sentence per chunk) and `tfidf-keywords-v1` (top-K distinctive terms per doc). Cores are Merkle-signed back to their source docs and serve as enriched titles for retrieval.
|
||||
|
||||
```
|
||||
make distill-shards-parallel # first-sentence cores, per-shard
|
||||
make distill-shards-tfidf-parallel # TF-IDF cores, per-shard
|
||||
```
|
||||
|
||||
Run both — they generate independent cores per source. TF-IDF cores let neologisms (like a personal term that never appears in any title) match retrievals via keyword overlap.
|
||||
|
||||
## Data: personal Grok export
|
||||
|
||||
If you have an xAI data-export bundle, point `GROK_EXPORT` at its root directory (the one containing `ttl/30d/export_data/<user-id>/`):
|
||||
|
||||
```
|
||||
export GROK_EXPORT=$HOME/Downloads/<your-user-uuid>
|
||||
make ingest-grok-attached # conversations -> $(SHARDS_DIR)/grok.db
|
||||
make ingest-grok-media-attached # media prompts -> $(SHARDS_DIR)/grok.db
|
||||
```
|
||||
|
||||
Both walk the export tree, find `prod-grok-backend.json`, and yield one Document per conversation (or media-generation post). Conversation titles, full message text, and turn ordering are preserved. Each becomes a normal queryable doc in the cluster — your prior chats become memory the corpus can consult.
|
||||
|
||||
`--resume` is the default for these targets. Re-run any time to pick up new exports.
|
||||
|
||||
## Data: OpenAI / ChatGPT export (planned)
|
||||
|
||||
Not implemented yet. The shape will be one new `Source` subclass at `aborist/sources/openai.py` plus a Makefile target. The OpenAI ChatGPT data export is a `.zip` containing `conversations.json` with the `mapping`/`messages` tree shape. Adding it follows the same pattern as `aborist/sources/grok.py` — see that file as the template.
|
||||
|
||||
```
|
||||
# (placeholder)
|
||||
make ingest-openai-attached # OPENAI_EXPORT=$HOME/Downloads/<chatgpt-export>
|
||||
```
|
||||
|
||||
When this lands, conversations from both Grok and OpenAI will sit in the same shard cluster; queries fan out across all of them.
|
||||
|
||||
## Asking the corpus
|
||||
|
||||
```
|
||||
make query Q="What is the philosophy of stoicism?"
|
||||
make query Q="tell me about permacomputer ?"
|
||||
make query QUERY_TOP_K=12 Q="…" # widen the source set
|
||||
make query-dry Q="…" # assemble context but skip the LLM call
|
||||
```
|
||||
|
||||
The query path:
|
||||
|
||||
1. **Search** — FTS5 (body) + SQL `LIKE` (title) + `JOIN` over derivations (TF-IDF core keywords) across every shard. Three accept paths to the relevance filter.
|
||||
2. **Concept overlay** — synonym groups (`Athlon` ↔ `AMD`) widen retrieval; rivalry pairs (AMD ↔ Intel) narrow it unless the query has comparative phrasing ("compare X vs Y").
|
||||
3. **Context assembly** — top-K sources concatenated up to a 60 KB budget, fed to Hermes-3 with strict attribution rules in the system prompt.
|
||||
4. **Cache** — the v9.8 8-dim cache_key (`source_root | question_hash | model_profile | conversation | governance_policy | schema | canonicalization | chunking`) keys the answer in `qa.db`. Cache hits replay STRICT-mode in ~100 ms. Per-phase timings are returned in every result.
|
||||
|
||||
LLM endpoint defaults to `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B, 82K context, no auth). Override:
|
||||
|
||||
```
|
||||
export ABORIST_LLM_ENDPOINT="https://your-vllm.example/v1"
|
||||
export ABORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
|
||||
export ABORIST_LLM_API_KEY="..."
|
||||
```
|
||||
|
||||
## Mark a wrong answer
|
||||
|
||||
If the LLM produced something incorrect, mark its cache record stale so the next ask re-runs inference fresh:
|
||||
|
||||
```
|
||||
make falsify KEY=<cache_key> REASON="why it was wrong"
|
||||
```
|
||||
|
||||
The original record stays in the database (history matters). A `falsifications` row + `falsify` audit event record the act. Future lookups skip records whose `falsification_state != 'live'`.
|
||||
|
||||
## Inspecting
|
||||
|
||||
```
|
||||
make stats-shards # totals across shards
|
||||
make analyze-shards # compression spectrum, depth histogram, audit chain integrity
|
||||
make verify-shards # round-trip Merkle proofs on a random sample
|
||||
make activity # recent Q&A + ingests + derives + falsifications (agent timeline)
|
||||
make activity ACTIVITY_LIMIT=20
|
||||
```
|
||||
|
||||
`activity` is JSON; pipe through `jq` to drill in.
|
||||
|
||||
## Architecture (one screenful)
|
||||
|
||||
```
|
||||
aborist/
|
||||
├── merkle.py Python port of proxy.unturf.com/pkg/verified/merkle.go
|
||||
│ conventions: non-commutative HashCombine (0x03 prefix),
|
||||
│ explicit IsLeft per sibling, self-duplicate odd elements.
|
||||
├── store.py SQLite v9.8 schema (8-dim cache key, falsification state,
|
||||
│ audit chain, surface/core kinds, hot/warm/cold tier).
|
||||
├── ingest.py normalize → chunk → merkle → upsert. Bulk-batched writer.
|
||||
├── document.py Document, Edge, Chunker (tok-512-v1 default).
|
||||
├── source.py Source ABC: iter_documents() -> Iterator[Document].
|
||||
├── sources/
|
||||
│ ├── wikipedia.py cur + old MediaWiki SQL dumps (bz2-streamed).
|
||||
│ ├── html_page.py URL list + selectolax + httpx (robots-aware).
|
||||
│ └── grok.py xAI data export (conversations + media prompts).
|
||||
├── distill/ Distiller ABC + first_sentence + tfidf + recursion runner.
|
||||
├── search/ FTS5 backend + SearchBackend ABC + AuditMode.
|
||||
├── qa/ ask (single-doc) + query (multi-source RAG) + concepts overlay.
|
||||
├── evict.py hot ↔ cold tier transitions; rehydrate via source pipeline.
|
||||
└── cli.py argparse entrypoint (the make targets call into here).
|
||||
```
|
||||
|
||||
Source papers (read first if confused):
|
||||
|
||||
- `~/Downloads/merkle-providence-reverse-rag-whitepaper.pdf` — public spec, AGPL-3.0
|
||||
- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV/canonical encoding, theorems T1–T5)
|
||||
|
||||
## Tests
|
||||
|
||||
```
|
||||
make test # 79 tests, all stdlib + pytest
|
||||
make bench # ETL throughput across configs (serial / shared-WAL / attached)
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
AGPL-3.0-only.
|
||||
AGPL-3.0-only. Whitepaper, code, and derived knowledge are public-domain inheritable per the unturf.com permacomputer charter.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue