diff --git a/Makefile b/Makefile index 6db6a79..71d5d0e 100644 --- a/Makefile +++ b/Makefile @@ -421,11 +421,15 @@ recrawl-check: bootstrap-crawler ## conditional HEAD per ingested doc [DOMAIN=x. DOT_SRCS := $(wildcard docs/diagrams/*.dot) DOT_PNGS := $(DOT_SRCS:.dot=.png) +DOT_SVGS := $(DOT_SRCS:.dot=.svg) docs/diagrams/%.png: docs/diagrams/%.dot dot -Tpng $< -o $@ -docs: $(DOT_PNGS) ## render docs/diagrams/*.dot -> .png via graphviz +docs/diagrams/%.svg: docs/diagrams/%.dot + dot -Tsvg $< -o $@ + +docs: $(DOT_PNGS) $(DOT_SVGS) ## render docs/diagrams/*.dot -> .png + .svg via graphviz # Reproducible micro-benchmark over a fixed slice of cur. Lets you compare # ETL throughput across configs and catches regressions on optimization diff --git a/README.md b/README.md index 26cd958..ce4d186 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ make distill-shards-tfidf-parallel # core → keyword sets for retrieval make query Q="What is anarcho-capitalism?" ``` -A Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer plus a verifier audit_mode (STRICT / HYBRID / UNGROUNDED). Repeat the same question and a cache hit replays in ~100 ms. +A Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer plus a verifier label that names what the lexical verifier could confirm. The current four-rung ladder for claim-lattice modes is `POINTER-LINKED` → `ANCHOR-WARRANTED` → `EVIDENCE-WARRANTED` → `UNGROUNDED` (with `-PARTIAL` suffix on HYBRID). Repeat the same question and a cache hit replays in ~100 ms. ### B. Crawl any live website and query it @@ -282,10 +282,10 @@ Default render is human-readable: question, audit-mode summary line, answer, sou The query path: 1. **Search** — FTS5 (body) + SQL `LIKE` (title) + `JOIN` over derivations (TF-IDF core keywords) across every shard. Three accept paths to the relevance filter. -2. **Concept overlay** — synonym groups (`Athlon` ↔ `AMD`) widen retrieval; rivalry pairs (AMD ↔ Intel) narrow it unless the query has comparative phrasing ("compare X vs Y"). +2. **Concept overlay** — per-shard `concept_relations` SQLite table (corpus-derived, not hand-curated). Synonyms widen retrieval; rivalries narrow it unless the query uses comparative phrasing ("compare X vs Y"). Built-in extractor `link_reciprocity_synonym` reads the existing `edges` table for reciprocal A↔B link pairs and emits synonym edges between their title-tokens — works for Wikipedia (See-also bidirectional), HTML site internal links, or any corpus with bidirectional document links. See `docs/concept-relations-design.md` for the storage tradeoff (1.6% tax measured on 6 GB Wikipedia). 3. **Context assembly** — top-K sources concatenated up to a 60 KB budget. Wikitext is stripped to plain prose via `aborist.wikitext.to_base()` (the corpus stores raw `[[wikilinks]]` so the link graph is recoverable on demand; the LLM and verifier both see clean prose). 4. **LLM** — Hermes-3 with strict attribution rules in the system prompt + a user-turn grounding reminder. -5. **Verifier** — every claim is classified into the v9.8 trichotomy STRICT / HYBRID / UNGROUNDED via a layered lexical check (see below). +5. **Verifier** — every claim runs through a layered lexical check; the result rolls up into the v9.8 trichotomy (`audit_mode` ∈ STRICT / HYBRID / UNGROUNDED) at the schema layer AND a four-rung display ladder at render time (POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → UNGROUNDED). See below. 6. **Cache** — the v9.8 8-dim cache_key (`source_root | question_hash | model_profile | conversation | governance_policy | schema | canonicalization | chunking`) keys the answer in `qa.db`. Cache hits replay in ~100 ms. Per-phase timings in every result. LLM endpoint defaults to `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B, 82K context, no auth). Override: @@ -296,9 +296,11 @@ export ABORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct" export ABORIST_LLM_API_KEY="..." ``` -## Verifying answers (audit modes) +## Verifying answers (audit modes & label ladder) -Every answer carries an `audit_mode` set by `aborist/qa/verify.py`: +Two layers of labels stack on every answer: + +**Schema layer — v9.8 trichotomy** (`audit_mode` column, persisted, drives cache lookups & audit chain): | mode | meaning | |---|---| @@ -306,7 +308,18 @@ Every answer carries an `audit_mode` set by `aborist/qa/verify.py`: | **HYBRID** | some claims source-grounded, some emerged from training | | **UNGROUNDED** | no evidence, or none verifies — purely emergent | -Four strategies run in sequence; first to find evidence classifies. Each is lexical (substring or token-coverage), never embeddings — soft signals stay out of the proof path. +**Display layer — four-rung ladder** (claim-lattice modes only; renderer-only transformation, schema unchanged): + +| rung | what's actually proved | +|---|---| +| **EVIDENCE-WARRANTED** | pointer verified + warrant ran & passed + no soft demotes | +| **ANCHOR-WARRANTED** | pointer-linked + warrant passed; soft-demote violations present | +| **POINTER-LINKED** | pointer/source/chunk verified, but warrant either didn't apply or failed for some claim | +| **UNGROUNDED** | no verified pairs | + +HYBRID gets a `-PARTIAL` suffix on whichever rung applies. The point of the display ladder: `STRICT in claim_lattice mode` is NOT "the answer is correct" — it's "every pointer resolved to a valid evidence object whose source_role is allowed AND citation-coverage passed." The display label spells out the actual property so users don't read STRICT as full semantic entailment. + +**Verifier strategies** run in sequence; first to find evidence classifies. Each is lexical (substring or token-coverage), never embeddings — soft signals stay out of the proof path. | # | strategy | what triggers it | |---|---|---| @@ -314,6 +327,9 @@ Four strategies run in sequence; first to find evidence classifies. Each is lexi | 2 | `span` | bullet/sentence units substring-tested as fallback | | 3 | `entity` | multi-word proper nouns tested with proximity-cluster gating | | 4 | `paraphrase` | inside the span path: ≥85% token coverage on prose-shaped spans | +| 5 | `claim_lattice` | model emits `claim text. [E1,E2]` pointer-line OR `{"claims":[{"text":..., "evidence_ids":[...]}]}` JSON; verifier resolves pointers to runtime-built evidence objects & runs seven hard checks | + +The `claim_lattice` path runs **seven deterministic hard checks**: parser succeeded, evidence_id resolves, source_role allowed, claim text non-empty, citation coverage threshold, pointer count cap, anchor-class warrant. Anchor-class warrant composes five lexical anchor classes — proper-noun, date, entity-list, count (with digit↔word equivalence), and why-cause — each gated on either question shape, claim content, or both. See `docs/concept-relations-design.md` and the whitepaper §13.9 for the architecture. Trailing `(Source: https://...)` parentheticals the model appends to verbatim source sentences are stripped before substring testing, so verbatim-with-citation no longer flags HYBRID. @@ -396,6 +412,13 @@ make activity ACTIVITY_LIMIT=20 ## Architecture (one screenful) +![Aborist module graph](docs/diagrams/aborist-modules.svg) + +For per-module reference docs see [`docs/modules/`](docs/modules/index.md). +Full pipeline diagrams live in [`docs/diagrams/`](docs/diagrams/) — query, +ingest, verifier-ladder, plus the mesh sub-diagrams. Render any change +with `make docs`. + ``` aborist/ ├── merkle.py Python port of proxy.unturf.com/pkg/verified/merkle.go @@ -403,7 +426,8 @@ aborist/ │ IsLeft per sibling, self-duplicate odd elements). ├── store.py SQLite v9.8 schema (8-dim cache key, falsification │ state, audit chain, surface/core kinds, hot/warm/ -│ cold tier, document_http_meta, mesh_*). +│ cold tier, document_http_meta, concept_relations, +│ mesh_*). ├── ingest.py normalize → chunk → merkle → upsert. Bulk-batched. ├── document.py Document, Edge, Chunker (tok-512-v1 default). ├── source.py Source ABC: iter_documents() -> Iterator[Document]. @@ -417,14 +441,36 @@ aborist/ │ └── crawler/ verbatim AsyncWebFetcher lift + ingest bridge. ├── distill/ Distiller ABC + first_sentence + tfidf + runner. ├── search/ FTS5 backend + SearchBackend ABC + AuditMode. +├── concepts/ corpus-derived synonym & rivalry layer. +│ ├── store.py append-only concept_relations CRUD + idempotent +│ │ INSERT OR IGNORE on UNIQUE re-derivation key. +│ ├── query.py cross-shard synonym_expand & rivalry_excluded; +│ │ per-process LRU keyed on shard mtime; manual / +│ │ derived index split with per-token degree caps. +│ ├── extract.py pluggable extractor registry; built-in +│ │ `link_reciprocity_synonym` reads existing edges. +│ └── seed.py one-shot migration of legacy frozensets to +│ manual_legacy rows (clique edges per group). ├── qa/ │ ├── client.py ChatClient + StubClient + OpenAICompatibleClient. │ ├── keys.py 8-dim cache_key + question_hash (CJK-aware strip). -│ ├── verify.py 4-strategy lexical verifier (quote/span/entity/ -│ │ paraphrase). Citation-strip + wikitext-strip. +│ ├── verify.py quote/span/entity/paraphrase + claim_lattice +│ │ verifier (seven deterministic hard checks incl. +│ │ anchor-class warrant). Citation-strip + +│ │ wikitext-strip. +│ ├── warrant.py five anchor classes — proper-noun, date, +│ │ entity-list, count (digit↔word equivalence), +│ │ cause — each gated on question shape and/or +│ │ claim content. Lexical only; in the proof path. +│ ├── evidence.py EvidenceObject + spotlight excerpt (density +│ │ rank picks the load-bearing slice). +│ ├── parse_claims.py pointer-line parser (`claim. [E1,E2]`). +│ ├── concepts.py backwards-compat shim — delegates to aborist.concepts. +│ ├── dag.py per-run Merkle-DAG (7-stage quote / 9-stage CTI). │ ├── inspect.py sidecar diagnostic — read-only span classifier. │ ├── runner.py ask(): single-doc Q&A + cache + verify. -│ └── query.py query(): multi-source RAG + concept overlay. +│ └── query.py query(): multi-source RAG + concept overlay +│ (FTS5 AND→OR-with-synonyms fallback). ├── mesh/ off-by-default federation: identity (Ed25519/X25519), │ per-epoch roster, AEAD-wrapped epoch secrets, HTTP │ gossip wire, per-peer chain-of-claims tracking. @@ -432,16 +478,19 @@ aborist/ └── cli.py argparse entrypoint (the make targets call into here). ``` +For diagrams of how these modules wire together, see `docs/diagrams/` (rendered SVGs of the module graph, query pipeline, ingest pipeline, mesh wire, & verifier ladder). For per-module reference docs see `docs/modules/` — every top-level package has its own page. + Source papers (read first if confused): -- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — canonical whitepaper (rst, builds the PDF). Section 13.8 covers the verifier in depth. +- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — canonical whitepaper (rst, builds the PDF). §13.8 covers the verifier; §13.9 covers claim-lattice / CTI; §13.4.11 covers the corpus-derived concept layer. - `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV/canonical encoding, theorems T1–T5). +- `docs/concept-relations-design.md` — synonym & rivalry layer architecture + 1.6% storage tradeoff rationale. - `docs/mesh.md` + `docs/mesh-deploy.md` — protocol contract + two-host runbook for federation. ## Tests ``` -make test # 326+ tests, default suite (stdlib + pytest, no network) +make test # 641+ tests, default suite (stdlib + pytest, no network) make test-crawler # opt-in: tests for the verbatim crawler lift make bench # ETL throughput across configs (serial / shared-WAL / attached) ``` diff --git a/docs/diagrams/aborist-modules.dot b/docs/diagrams/aborist-modules.dot new file mode 100644 index 0000000..c649ac3 --- /dev/null +++ b/docs/diagrams/aborist-modules.dot @@ -0,0 +1,127 @@ +// Aborist module graph — top-level packages & their dependencies. +// +// Arrows point in the import direction (A → B means A imports from B). +// Cluster boxes group modules with shared concerns (storage, retrieval, +// federation, etc.). +// +// Render: dot -Tsvg aborist-modules.dot -o aborist-modules.svg + +digraph aborist_modules { + rankdir=LR + node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10] + edge [fontname="Helvetica", fontsize=9] + bgcolor="white" + + // ---- Substrate (pure crypto / serialization, no SQL) ---- + subgraph cluster_substrate { + label="substrate" + style="rounded,dashed" + color="#999999" + merkle [label="merkle.py\n(Python port of Go merkle.go)", fillcolor="#fff7e6"] + document [label="document.py\nDocument · Edge · Chunker", fillcolor="#fff7e6"] + wikitext [label="wikitext.py\nto_base() prose-strip", fillcolor="#fff7e6"] + } + + // ---- Storage layer ---- + subgraph cluster_storage { + label="storage" + style="rounded,dashed" + color="#999999" + store [label="store.py\nv9.8 SQLite schema +\naudit chain helpers", fillcolor="#e6f0ff"] + ingest [label="ingest.py\nnormalize → chunk → merkle → upsert", fillcolor="#e6f0ff"] + evict [label="evict.py\nhot ↔ cold tier", fillcolor="#e6f0ff"] + } + + // ---- Sources (corpus producers) ---- + subgraph cluster_sources { + label="sources/" + style="rounded,dashed" + color="#999999" + wikipedia [label="wikipedia.py\ncur + old SQL dumps", fillcolor="#e7ffe7"] + wiki_xml [label="wikipedia_xml.py\nphase IV iterparse", fillcolor="#e7ffe7"] + html_page [label="html_page.py\nselectolax + httpx", fillcolor="#e7ffe7"] + crawler [label="crawler/\nasync HTML BFS", fillcolor="#e7ffe7"] + grok [label="grok.py\nxAI export", fillcolor="#e7ffe7"] + vcs [label="vcs.py\ngit + Mercurial", fillcolor="#e7ffe7"] + } + + // ---- Retrieval & verifier ---- + subgraph cluster_retrieval { + label="retrieval & verifier" + style="rounded,dashed" + color="#999999" + search [label="search/\nFTS5 backend + ABC", fillcolor="#ffe6f0"] + concepts [label="concepts/\nsynonym & rivalry overlay\n(per-shard concept_relations)", fillcolor="#ffe6f0"] + qa_query [label="qa/query.py\nmulti-source RAG", fillcolor="#ffe6f0"] + qa_verify [label="qa/verify.py\nquote/span/entity/paraphrase\n+ claim_lattice (7 hard checks)", fillcolor="#ffe6f0"] + qa_warrant[label="qa/warrant.py\n5 anchor classes\n(proper-noun · date · count\n · entity-list · cause)", fillcolor="#ffe6f0"] + qa_evidence[label="qa/evidence.py\nEvidenceObject + spotlight", fillcolor="#ffe6f0"] + qa_runner [label="qa/runner.py\nask(): single-doc Q&A", fillcolor="#ffe6f0"] + qa_dag [label="qa/dag.py\nper-run Merkle DAG\n(7 / 9 stages)", fillcolor="#ffe6f0"] + qa_keys [label="qa/keys.py\n8-dim cache_key + question_hash", fillcolor="#ffe6f0"] + qa_inspect[label="qa/inspect.py\nsidecar diagnostic\n(read-only)", fillcolor="#ffe6f0"] + } + + // ---- Distill (surface → core) ---- + distill [label="distill/\nDistiller ABC + tfidf + first_sentence", fillcolor="#fff0e0"] + + // ---- Federation (off by default) ---- + mesh [label="mesh/\nfederation: identity, roster,\nAEAD epoch secret, gossip wire", fillcolor="#f0e6ff"] + + // ---- Entry point ---- + cli [label="cli.py\nargparse entrypoint\n(make targets call into here)", fillcolor="#dddddd", shape=note] + + // Substrate dependencies + document -> merkle + store -> merkle + store -> document + ingest -> store + ingest -> document + + // Sources -> ingest path + wikipedia -> ingest + wiki_xml -> ingest + html_page -> ingest [label="parse_html →\nedges rows"] + crawler -> html_page + grok -> ingest + vcs -> ingest + + // Retrieval pipeline + qa_query -> search + qa_query -> concepts [label="synonym_expand /\nrivalry_excluded"] + qa_query -> qa_keys + qa_query -> qa_verify + qa_query -> qa_evidence + qa_query -> qa_dag + qa_query -> wikitext [label="to_base()"] + qa_verify -> qa_warrant + qa_verify -> qa_evidence + qa_runner -> qa_query + qa_inspect -> qa_query [style=dashed, label="read-only sidecar"] + + // Concepts reads existing edges + concepts -> store [label="reads edges,\nwrites concept_relations"] + + // Search reads chunks + search -> store + + // Distill consumes store, writes derivations + distill -> store + distill -> document + + // Eviction + evict -> store + + // Mesh + mesh -> store + + // CLI dispatches everywhere + cli -> ingest + cli -> qa_runner + cli -> qa_query + cli -> qa_inspect + cli -> distill + cli -> evict + cli -> mesh + cli -> concepts +} diff --git a/docs/diagrams/aborist-modules.png b/docs/diagrams/aborist-modules.png new file mode 100644 index 0000000..1e756cf Binary files /dev/null and b/docs/diagrams/aborist-modules.png differ diff --git a/docs/diagrams/aborist-modules.svg b/docs/diagrams/aborist-modules.svg new file mode 100644 index 0000000..f89b006 --- /dev/null +++ b/docs/diagrams/aborist-modules.svg @@ -0,0 +1,443 @@ + + + + + + +aborist_modules + + +cluster_substrate + +substrate + + +cluster_storage + +storage + + +cluster_sources + +sources/ + + +cluster_retrieval + +retrieval & verifier + + + +merkle + +merkle.py +(Python port of Go merkle.go) + + + +document + +document.py +Document · Edge · Chunker + + + +document->merkle + + + + + +wikitext + +wikitext.py +to_base() prose-strip + + + +store + +store.py +v9.8 SQLite schema + +audit chain helpers + + + +store->merkle + + + + + +store->document + + + + + +ingest + +ingest.py +normalize → chunk → merkle → upsert + + + +ingest->document + + + + + +ingest->store + + + + + +evict + +evict.py +hot ↔ cold tier + + + +evict->store + + + + + +wikipedia + +wikipedia.py +cur + old SQL dumps + + + +wikipedia->ingest + + + + + +wiki_xml + +wikipedia_xml.py +phase IV iterparse + + + +wiki_xml->ingest + + + + + +html_page + +html_page.py +selectolax + httpx + + + +html_page->ingest + + +parse_html → +edges rows + + + +crawler + +crawler/ +async HTML BFS + + + +crawler->html_page + + + + + +grok + +grok.py +xAI export + + + +grok->ingest + + + + + +vcs + +vcs.py +git + Mercurial + + + +vcs->ingest + + + + + +search + +search/ +FTS5 backend + ABC + + + +search->store + + + + + +concepts + +concepts/ +synonym & rivalry overlay +(per-shard concept_relations) + + + +concepts->store + + +reads edges, +writes concept_relations + + + +qa_query + +qa/query.py +multi-source RAG + + + +qa_query->wikitext + + +to_base() + + + +qa_query->search + + + + + +qa_query->concepts + + +synonym_expand / +rivalry_excluded + + + +qa_verify + +qa/verify.py +quote/span/entity/paraphrase ++ claim_lattice (7 hard checks) + + + +qa_query->qa_verify + + + + + +qa_evidence + +qa/evidence.py +EvidenceObject + spotlight + + + +qa_query->qa_evidence + + + + + +qa_dag + +qa/dag.py +per-run Merkle DAG +(7 / 9 stages) + + + +qa_query->qa_dag + + + + + +qa_keys + +qa/keys.py +8-dim cache_key + question_hash + + + +qa_query->qa_keys + + + + + +qa_warrant + +qa/warrant.py +5 anchor classes +(proper-noun · date · count + · entity-list · cause) + + + +qa_verify->qa_warrant + + + + + +qa_verify->qa_evidence + + + + + +qa_runner + +qa/runner.py +ask(): single-doc Q&A + + + +qa_runner->qa_query + + + + + +qa_inspect + +qa/inspect.py +sidecar diagnostic +(read-only) + + + +qa_inspect->qa_query + + +read-only sidecar + + + +distill + +distill/ +Distiller ABC + tfidf + first_sentence + + + +distill->document + + + + + +distill->store + + + + + +mesh + +mesh/ +federation: identity, roster, +AEAD epoch secret, gossip wire + + + +mesh->store + + + + + +cli + + + +cli.py +argparse entrypoint +(make targets call into here) + + + +cli->ingest + + + + + +cli->evict + + + + + +cli->concepts + + + + + +cli->qa_query + + + + + +cli->qa_runner + + + + + +cli->qa_inspect + + + + + +cli->distill + + + + + +cli->mesh + + + + + diff --git a/docs/diagrams/ingest-pipeline.dot b/docs/diagrams/ingest-pipeline.dot new file mode 100644 index 0000000..373f2c5 --- /dev/null +++ b/docs/diagrams/ingest-pipeline.dot @@ -0,0 +1,55 @@ +// Ingest pipeline: how a source document becomes Merkle-committed +// content-addressed storage in a v9.8 shard. +// +// Render: dot -Tsvg ingest-pipeline.dot -o ingest-pipeline.svg + +digraph ingest_pipeline { + rankdir=TB + node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10] + edge [fontname="Helvetica", fontsize=9] + bgcolor="white" + + source [label="SOURCE\nWikipedia SQL dump · XML page · HTML URL ·\nGrok export · git/hg repo · crawled site", fillcolor="#fff7e6", shape=note] + + iter_documents [label="Source.iter_documents()\nABC method per source kind\nyields Document(uri, raw_content, kind, ...)", fillcolor="#e7ffe7"] + + parse [label="source-specific parse\n• wikipedia.py: SQL row → Document\n• html_page.py: selectolax → text + edges\n• vcs.py: HEAD walk + supersedes chain\n• grok.py: conversation → Document(s)", fillcolor="#e7ffe7"] + + canonicalize [label="canonicalize(text)\nNFC + ws-collapse + strip ends\n(pinned by canonicalization_version)", fillcolor="#fff0e0"] + + chunker [label="Chunker.chunk(text)\ntok-512-v1 default\n(pinned by chunking_version)", fillcolor="#fff0e0"] + + leaf_hash [label="per-chunk leaf hash\nsha256(0x00 || canonical_chunk_bytes)\nleaves are content-addressed", fillcolor="#fff0e0"] + + merkle_tree [label="MerkleTree.build(leaves)\nnon-commutative HashCombine 0x03\nodd-element rule = self-duplicate\n→ document_root (32 bytes)", fillcolor="#fff0e0"] + + upsert [label="store.upsert_document(doc)\n• documents row keyed on document_root\n• chunks rows with leaf_hash + tier\n• merkle_nodes for proof reconstruction\n• edges per outbound link\n• audit_event chained", fillcolor="#e6f0ff"] + + chunks_fts [label="chunks_fts INSERT\nFTS5 contentless index\nrowid = chunks.chunk_id", fillcolor="#e6f0ff"] + + supersedes [label="supersedes edge\nif same uri but different content,\nlink new → old (lossless history)", fillcolor="#e6f0ff"] + + audit [label="audit_events INSERT\nevent_hash = sha256(prev_hash || canonical(body))\nlinear chain — 0 breaks invariant", fillcolor="#e6f0ff"] + + // Optional: distill cores AFTER ingest + distill [label="distill/runner.py\n(optional, separate phase)\n• first_sentence\n• tfidf keyword core\n• per-contrib-chunk Merkle proof", fillcolor="#fff0e0", style="rounded,filled,dashed"] + + derivations [label="derivations row\ncore_root ← src_root\nproof_blob = inclusion proofs\nfor every contributing chunk", fillcolor="#fff0e0", style="rounded,filled,dashed"] + + end [label="SHARD READY\ndocuments + chunks + edges + merkle_nodes\n+ audit_events + (concept_relations\nback-derived later)", fillcolor="#d4edda", shape=note] + + source -> iter_documents + iter_documents -> parse + parse -> canonicalize + canonicalize -> chunker + chunker -> leaf_hash + leaf_hash -> merkle_tree + merkle_tree -> upsert + upsert -> chunks_fts + upsert -> supersedes + upsert -> audit + upsert -> end + upsert -> distill [style=dashed] + distill -> derivations [style=dashed] + derivations -> end [style=dashed] +} diff --git a/docs/diagrams/ingest-pipeline.png b/docs/diagrams/ingest-pipeline.png new file mode 100644 index 0000000..03d82d0 Binary files /dev/null and b/docs/diagrams/ingest-pipeline.png differ diff --git a/docs/diagrams/ingest-pipeline.svg b/docs/diagrams/ingest-pipeline.svg new file mode 100644 index 0000000..3fb484e --- /dev/null +++ b/docs/diagrams/ingest-pipeline.svg @@ -0,0 +1,223 @@ + + + + + + +ingest_pipeline + + + +source + + + +SOURCE +Wikipedia SQL dump · XML page · HTML URL · +Grok export · git/hg repo · crawled site + + + +iter_documents + +Source.iter_documents() +ABC method per source kind +yields Document(uri, raw_content, kind, ...) + + + +source->iter_documents + + + + + +parse + +source-specific parse +• wikipedia.py: SQL row → Document +• html_page.py: selectolax → text + edges +• vcs.py: HEAD walk + supersedes chain +• grok.py: conversation → Document(s) + + + +iter_documents->parse + + + + + +canonicalize + +canonicalize(text) +NFC + ws-collapse + strip ends +(pinned by canonicalization_version) + + + +parse->canonicalize + + + + + +chunker + +Chunker.chunk(text) +tok-512-v1 default +(pinned by chunking_version) + + + +canonicalize->chunker + + + + + +leaf_hash + +per-chunk leaf hash +sha256(0x00 || canonical_chunk_bytes) +leaves are content-addressed + + + +chunker->leaf_hash + + + + + +merkle_tree + +MerkleTree.build(leaves) +non-commutative HashCombine 0x03 +odd-element rule = self-duplicate +→ document_root (32 bytes) + + + +leaf_hash->merkle_tree + + + + + +upsert + +store.upsert_document(doc) +• documents row keyed on document_root +• chunks rows with leaf_hash + tier +• merkle_nodes for proof reconstruction +• edges per outbound link +• audit_event chained + + + +merkle_tree->upsert + + + + + +chunks_fts + +chunks_fts INSERT +FTS5 contentless index +rowid = chunks.chunk_id + + + +upsert->chunks_fts + + + + + +supersedes + +supersedes edge +if same uri but different content, +link new → old (lossless history) + + + +upsert->supersedes + + + + + +audit + +audit_events INSERT +event_hash = sha256(prev_hash || canonical(body)) +linear chain — 0 breaks invariant + + + +upsert->audit + + + + + +distill + +distill/runner.py +(optional, separate phase) +• first_sentence +• tfidf keyword core +• per-contrib-chunk Merkle proof + + + +upsert->distill + + + + + +end + + + +SHARD READY +documents + chunks + edges + merkle_nodes ++ audit_events + (concept_relations +back-derived later) + + + +upsert->end + + + + + +derivations + +derivations row +core_root ← src_root +proof_blob = inclusion proofs +for every contributing chunk + + + +distill->derivations + + + + + +derivations->end + + + + + diff --git a/docs/diagrams/mesh-data-flow.svg b/docs/diagrams/mesh-data-flow.svg new file mode 100644 index 0000000..b825a79 --- /dev/null +++ b/docs/diagrams/mesh-data-flow.svg @@ -0,0 +1,224 @@ + + + + + + +mesh_data_flow + + +cluster_alice + +peer: alice (admin) + + +cluster_bob + +peer: bob (member) + + +cluster_wire + +gossip messages (Ed25519-signed, optional AEAD) + + + +a_docs + +documents +(surface + core) + + + +m_root + +ANNOUNCE_ROOT +document_root, source_uri, +chunking_version, schema_version + + + +a_docs->m_root + + + + + +m_resp + +DELIVER_BODY +bytes + Merkle proof of inclusion +recipient verifies before insert + + + +a_docs->m_resp + + + + + +a_deriv + +derivations +(surface -> core proof_blob) + + + +m_der + +ANNOUNCE_DERIVATION +core_root <- surface_root(s), +distiller_id, proof_blob hash + + + +a_deriv->m_der + + + + + +a_prov + +providence_cache +(8-dim cache_key, audit_mode) + + + +m_prov + +ANNOUNCE_PROVIDENCE +cache_key, audit_mode, +answer_hash + + + +a_prov->m_prov + + + + + +a_falsif + +falsifications +('this answer is wrong') + + + +m_fals + +ANNOUNCE_FALSIFICATION +cache_key, reason, signed by witness + + + +a_falsif->m_fals + + + + + +a_audit + +audit_events +(append-only chain) + + + +b_audit + +audit_events + + + +a_audit->b_audit + + +audit chain merge +(prev_event_hash gates inserts) + + + +b_docs + +documents +(surface + core) + + + +m_pull + +REQUEST_BODY +document_root or chunk_root +(only on local miss) + + + +b_docs->m_pull + + +miss on root + + + +b_deriv + +derivations + + + +b_prov + +providence_cache + + + +b_falsif + +falsifications + + + +m_root->b_docs + + + + + +m_der->b_deriv + + + + + +m_prov->b_prov + + + + + +m_fals->b_falsif + + + + + +m_pull->a_docs + + + + + +m_resp->b_docs + + +verify proof, +then ingest + + + diff --git a/docs/diagrams/mesh-epoch-lifecycle.svg b/docs/diagrams/mesh-epoch-lifecycle.svg new file mode 100644 index 0000000..b3eb620 --- /dev/null +++ b/docs/diagrams/mesh-epoch-lifecycle.svg @@ -0,0 +1,126 @@ + + + + + + +mesh_epoch_lifecycle + + +cluster_ops + +Operator commands + + + +none + +no mesh state +(meta.mesh.enabled = 0) + + + +e0 + +epoch 0 +founder = sole admin +reason: 'genesis' + + + +none->e0 + + +aborist mesh init --group <name> + + + +e1 + +epoch N +roster + envelope +signed by actor + + + +e0->e1 + + +add | kick | rotate + + + +eN + +epoch N+1 +fresh secret +fresh envelope + + + +e1->eN + + +add | kick | rotate + + + +historic + + + +every prior epoch +stays on disk +(roster + signatures verifiable) + + + +e1->historic + + + + +eN->eN + + +rotate (no roster change) + + + +eN->historic + + + + +op_add + + + +aborist mesh add +--member-id bob ... +admin only + + + +op_kick + + + +aborist mesh kick +--member-id bob ... +admin only + + + +op_rotate + + + +aborist mesh rotate +any current member + + + diff --git a/docs/diagrams/mesh-group-decisions.svg b/docs/diagrams/mesh-group-decisions.svg new file mode 100644 index 0000000..7a425a9 --- /dev/null +++ b/docs/diagrams/mesh-group-decisions.svg @@ -0,0 +1,206 @@ + + + + + + +mesh_group_decisions + + + +start + +group steward has a question + + + +q_first + +is mesh +initialized +yet? + + + +start->q_first + + + + + +q_role + +is the question +about people +or secrets? + + + +q_first->q_role + + +yes + + + +a_init + +aborist mesh init --group <name> +then mesh enable +(creates epoch 0; founder = sole admin) + + + +q_first->a_init + + +no + + + +q_who + +add, remove, +or refresh? + + + +q_role->q_who + + +people + + + +a_rotate + +aborist mesh rotate +--reason '...' +(any current member) + + + +q_role->a_rotate + + +just rotate the secret + + + +q_admin + +are you +admin in +current epoch? + + + +q_who->q_admin + + +add or kick + + + +q_who->a_rotate + + +refresh secret only + + + +a_add + +aborist mesh add +--member-id <id> +--sign-pub <hex> +--dh-pub <hex> +[--role admin] + + + +q_admin->a_add + + +yes (admin) -> add + + + +a_kick + +aborist mesh kick +--member-id <id> +--reason '...' + + + +q_admin->a_kick + + +yes (admin) -> kick + + + +g_admin + +permission denied +(only admins of the +current epoch may add or kick) + + + +q_admin->g_admin + + +no + + + +g_add + + + +new member can decrypt +epoch+1 forward; +cannot reach prior epochs + + + +a_add->g_add + + + + +g_kick + + + +kicked member's prior +signatures stay valid forever; +opaque to gossip from epoch+1 on + + + +a_kick->g_kick + + + + +g_rot + + + +fresh secret on the same roster; +use on suspected secret leak +or scheduled hygiene + + + +a_rotate->g_rot + + + + diff --git a/docs/diagrams/mesh-identity-stack.svg b/docs/diagrams/mesh-identity-stack.svg new file mode 100644 index 0000000..b6c956a --- /dev/null +++ b/docs/diagrams/mesh-identity-stack.svg @@ -0,0 +1,150 @@ + + + + + + +mesh_identity_stack + + +cluster_peer + +One peer (e.g. alice) + + +cluster_group + +Per-epoch group state + + + +sign_priv + +Ed25519 priv +(sign_priv, 32B) + + + +sign_pub + +Ed25519 pub +(sign_pub, 32B) +shared with group + + + +sign_priv->sign_pub + + +public_key() + + + +audit + + + +audit_events +(prev_event_hash, event_hash) +append-only Merkle chain + + + +sign_priv->audit + + +signs each +membership op + + + +roster + +mesh_roster +(epoch, member_id, sign_pub, dh_pub, role) + + + +sign_pub->roster + + +enrolled at + + + +dh_priv + +X25519 priv +(dh_priv, 32B) + + + +dh_pub + +X25519 pub +(dh_pub, 32B) +shared with group + + + +dh_priv->dh_pub + + +public_key() + + + +secret + +symmetric epoch secret +32 random bytes +never stored in clear + + + +dh_priv->secret + + +ECDH-unwraps +self's envelope entry + + + +dh_pub->roster + + +enrolled at + + + +member_id + +member_id +8 hex chars + + + +member_id->roster + + +keys by + + + +epoch + +mesh_epochs +(epoch_id, secret_envelope, started_event_hash) + + + +epoch->secret + + +wrapped per member via ECDH + + + diff --git a/docs/diagrams/mesh-secret-envelope.svg b/docs/diagrams/mesh-secret-envelope.svg new file mode 100644 index 0000000..f4297e6 --- /dev/null +++ b/docs/diagrams/mesh-secret-envelope.svg @@ -0,0 +1,212 @@ + + + + + + +mesh_secret_envelope + + +cluster_wrap + +rotate_epoch() — done by actor (admin) + + +cluster_unwrap + +Each peer unwraps only their own slot + + + +secret + + +epoch_N_secret +32 random bytes + + + +wrap_a + +ChaCha20-Poly1305 +(shared_a, nonce_a, secret, + aad=alice.member_id) + + + +secret->wrap_a + + + + + +wrap_b + +ChaCha20-Poly1305 +(shared_b, nonce_b, secret, + aad=bob.member_id) + + + +secret->wrap_b + + + + + +wrap_c + +ChaCha20-Poly1305 +(shared_c, nonce_c, secret, + aad=carol.member_id) + + + +secret->wrap_c + + + + + +ecdh_a + +ECDH(rotator_dh_priv, +      alice_dh_pub) +  -> shared_a (HKDF) + + + +ecdh_a->wrap_a + + +key + + + +ecdh_b + +ECDH(rotator_dh_priv, +      bob_dh_pub) +  -> shared_b (HKDF) + + + +ecdh_b->wrap_b + + +key + + + +ecdh_c + +ECDH(rotator_dh_priv, +      carol_dh_pub) +  -> shared_c (HKDF) + + + +ecdh_c->wrap_c + + +key + + + +envelope + + + +mesh_epochs.secret_envelope (JSON) +{ +  alice: {nonce_b64, ct_b64}, +  bob:   {nonce_b64, ct_b64}, +  carol: {nonce_b64, ct_b64} +} + + + +wrap_a->envelope + + +alice slot + + + +wrap_b->envelope + + +bob slot + + + +wrap_c->envelope + + +carol slot + + + +u_alice + +alice: +ECDH(alice_dh_priv, +     rotator_dh_pub) +then AEAD-decrypt slot.alice + + + +envelope->u_alice + + + + + +u_bob + +bob: +ECDH(bob_dh_priv, +     rotator_dh_pub) +then AEAD-decrypt slot.bob + + + +envelope->u_bob + + + + + +u_carol + +carol: +ECDH(carol_dh_priv, +     rotator_dh_pub) +then AEAD-decrypt slot.carol + + + +envelope->u_carol + + + + + +evicted + +evicted dave +no slot in envelope +ValueError on unwrap + + + +envelope->evicted + + +dave NOT included + + + diff --git a/docs/diagrams/query-pipeline.dot b/docs/diagrams/query-pipeline.dot new file mode 100644 index 0000000..00d01e1 --- /dev/null +++ b/docs/diagrams/query-pipeline.dot @@ -0,0 +1,70 @@ +// Query pipeline: how a question becomes a verified answer. +// +// Per-stage timing budget at the top of each box (Hermes ~6-8s +// dominates; everything else target sub-second per phase). +// +// Render: dot -Tsvg query-pipeline.dot -o query-pipeline.svg + +digraph query_pipeline { + rankdir=TB + node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10] + edge [fontname="Helvetica", fontsize=9] + bgcolor="white" + + question [label="QUESTION\n\"what tech can reconstruct\nanother person's thoughts?\"", fillcolor="#fff7e6", shape=note] + + // Cache lookup first (free hit if previously asked) + cache_check [label="cache lookup\nproperence_cache by 8-dim cache_key\n• equivalence_class fallback\n• ~0.3ms warm", fillcolor="#e6f0ff"] + cache_hit [label="CACHE HIT\nreplay answer + Merkle proof\n~100ms total", fillcolor="#d4edda", shape=note] + + // Retrieval + fts5 [label="FTS5 BM25 search\n• AND-mode first (strict)\n• OR-mode fallback (top-5 longest +\n synonym pool)\n• ~0.25s/shard", fillcolor="#ffe6f0"] + title_like [label="title-LIKE backup\n(skipped if >5 tokens)\n• ~0.5s/shard for short queries", fillcolor="#ffe6f0"] + phrase_route [label="phrase-pattern route\n4-token sliding-window match\n(closes allusion gap)", fillcolor="#ffe6f0"] + concepts_lookup [label="concepts/ overlay\n• synonym_expand(qtokens)\n• rivalry_excluded()\n• per-process cache, ~1ms warm", fillcolor="#ffe6f0"] + + // Filter + filter [label="title-relevance filter\n4 accept paths:\n1. title-token overlap\n2. TF-IDF core match\n3. body density\n4. phrase match", fillcolor="#ffe6f0"] + rerank [label="rerank stack\n• body coverage (sqrt)\n• title token boost\n• source role boost\n• title purity", fillcolor="#ffe6f0"] + + // Evidence map (claim_lattice modes) + evidence_map [label="evidence map\n(claim_lattice modes)\nEvidenceObject per chunk\nrun-stable evidence_id (sha256)\nprompt-facing pointer_id (E1, E2…)", fillcolor="#fff0e0"] + + // Context assembly + context [label="context assembly\n• per-source cap\n• 60KB total budget\n• wikitext.to_base() prose-strip", fillcolor="#fff0e0"] + + // LLM + llm [label="LLM call (Hermes-3 / vLLM)\n6–8s — DOMINANT cost\nclaim_lattice_pointer or\nclaim_lattice (JSON)", fillcolor="#ffd4d4"] + + // Verification + verify [label="verify_claim_lattice\n7 hard checks:\n1. parser succeeded\n2. evidence_id resolves\n3. source_role allowed\n4. claim text non-empty\n5. citation coverage\n6. pointer count cap\n7. anchor-class warrant", fillcolor="#e7ffe7"] + warrant [label="warrant_check (5 anchor classes)\n• proper-noun (relation-gated)\n• date (year + month)\n• count (digit↔word)\n• entity-list\n• cause (why-shape)", fillcolor="#e7ffe7"] + + // Render + render [label="render\n• spotlight excerpt (density rank)\n• [E5 | Title | chunk_prefix: \"...\"]\n• four-rung label ladder", fillcolor="#fff7e6"] + + // Persistence + cache_write [label="providence_cache write\n• 8-dim cache_key\n• run_dag_root + run_dag_blob\n• audit_event chained\n• context_root committed", fillcolor="#e6f0ff"] + + answer [label="ANSWER\nlabel: POINTER-LINKED-PARTIAL ·\nvia claim_lattice · 2/2 · 11.6s", fillcolor="#d4edda", shape=note] + + // Edges + question -> cache_check + cache_check -> cache_hit [label="match"] + cache_check -> concepts_lookup [label="miss"] + concepts_lookup -> fts5 + concepts_lookup -> title_like + concepts_lookup -> phrase_route + fts5 -> filter + title_like -> filter + phrase_route -> filter + filter -> rerank + rerank -> evidence_map + evidence_map -> context + context -> llm + llm -> verify + verify -> warrant + warrant -> render + render -> cache_write + cache_write -> answer +} diff --git a/docs/diagrams/query-pipeline.png b/docs/diagrams/query-pipeline.png new file mode 100644 index 0000000..5a2d6b3 Binary files /dev/null and b/docs/diagrams/query-pipeline.png differ diff --git a/docs/diagrams/query-pipeline.svg b/docs/diagrams/query-pipeline.svg new file mode 100644 index 0000000..3f10eaf --- /dev/null +++ b/docs/diagrams/query-pipeline.svg @@ -0,0 +1,290 @@ + + + + + + +query_pipeline + + + +question + + + +QUESTION +"what tech can reconstruct +another person's thoughts?" + + + +cache_check + +cache lookup +properence_cache by 8-dim cache_key +• equivalence_class fallback +• ~0.3ms warm + + + +question->cache_check + + + + + +cache_hit + + + +CACHE HIT +replay answer + Merkle proof +~100ms total + + + +cache_check->cache_hit + + +match + + + +concepts_lookup + +concepts/ overlay +• synonym_expand(qtokens) +• rivalry_excluded() +• per-process cache, ~1ms warm + + + +cache_check->concepts_lookup + + +miss + + + +fts5 + +FTS5 BM25 search +• AND-mode first (strict) +• OR-mode fallback (top-5 longest + +  synonym pool) +• ~0.25s/shard + + + +filter + +title-relevance filter +4 accept paths: +1. title-token overlap +2. TF-IDF core match +3. body density +4. phrase match + + + +fts5->filter + + + + + +title_like + +title-LIKE backup +(skipped if >5 tokens) +• ~0.5s/shard for short queries + + + +title_like->filter + + + + + +phrase_route + +phrase-pattern route +4-token sliding-window match +(closes allusion gap) + + + +phrase_route->filter + + + + + +concepts_lookup->fts5 + + + + + +concepts_lookup->title_like + + + + + +concepts_lookup->phrase_route + + + + + +rerank + +rerank stack +• body coverage (sqrt) +• title token boost +• source role boost +• title purity + + + +filter->rerank + + + + + +evidence_map + +evidence map +(claim_lattice modes) +EvidenceObject per chunk +run-stable evidence_id (sha256) +prompt-facing pointer_id (E1, E2…) + + + +rerank->evidence_map + + + + + +context + +context assembly +• per-source cap +• 60KB total budget +• wikitext.to_base() prose-strip + + + +evidence_map->context + + + + + +llm + +LLM call (Hermes-3 / vLLM) +6–8s — DOMINANT cost +claim_lattice_pointer or +claim_lattice (JSON) + + + +context->llm + + + + + +verify + +verify_claim_lattice +7 hard checks: +1. parser succeeded +2. evidence_id resolves +3. source_role allowed +4. claim text non-empty +5. citation coverage +6. pointer count cap +7. anchor-class warrant + + + +llm->verify + + + + + +warrant + +warrant_check (5 anchor classes) +• proper-noun (relation-gated) +• date (year + month) +• count (digit↔word) +• entity-list +• cause (why-shape) + + + +verify->warrant + + + + + +render + +render +• spotlight excerpt (density rank) +• [E5 | Title | chunk_prefix: "..."] +• four-rung label ladder + + + +warrant->render + + + + + +cache_write + +providence_cache write +• 8-dim cache_key +• run_dag_root + run_dag_blob +• audit_event chained +• context_root committed + + + +render->cache_write + + + + + +answer + + + +ANSWER +label: POINTER-LINKED-PARTIAL · +via claim_lattice · 2/2 · 11.6s + + + +cache_write->answer + + + + + diff --git a/docs/diagrams/verifier-ladder.dot b/docs/diagrams/verifier-ladder.dot new file mode 100644 index 0000000..2d9a42d --- /dev/null +++ b/docs/diagrams/verifier-ladder.dot @@ -0,0 +1,49 @@ +// Verifier label ladder: how (audit_mode, violations) → display rung. +// +// Schema column `audit_mode` keeps the v9.8 trichotomy +// (STRICT / HYBRID / UNGROUNDED). The display ladder is a renderer- +// only transformation that names what's actually proved at each rung. +// +// Render: dot -Tsvg verifier-ladder.dot -o verifier-ladder.svg + +digraph verifier_ladder { + rankdir=TB + node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10] + edge [fontname="Helvetica", fontsize=9] + bgcolor="white" + + // Inputs + inputs [label="VERIFIER OUTPUT\naudit_mode ∈ {STRICT, HYBRID, UNGROUNDED}\nviolations: list[dict]", fillcolor="#fff7e6", shape=note] + + // Decision tree + is_ungrounded [label="audit_mode == UNGROUNDED?", fillcolor="#dddddd", shape=diamond] + has_warrant_miss [label="WARRANT_MISSING or\nTITLE_MISMATCH in violations?", fillcolor="#dddddd", shape=diamond] + has_soft_demote [label="any soft-demote violation?\n(LAZY_ANCHOR_DEMOTED,\nPOINTER_OVERFLOW_TRIMMED,\nTOO_MANY_CLAIMS, …)", fillcolor="#dddddd", shape=diamond] + is_hybrid [label="audit_mode == HYBRID?", fillcolor="#dddddd", shape=diamond] + + // Rungs (top to bottom = strongest to weakest) + evidence_warranted [label="EVIDENCE-WARRANTED\npointer verified +\nwarrant ran & passed +\nNO soft demotes", fillcolor="#28a745", fontcolor="white"] + anchor_warranted [label="ANCHOR-WARRANTED\npointer-linked +\nwarrant passed where it ran;\nsoft demotes present", fillcolor="#7cc66e"] + pointer_linked [label="POINTER-LINKED\npointer/source/chunk verified;\nwarrant didn't apply OR failed\nfor at least one claim", fillcolor="#ffc107"] + ungrounded [label="UNGROUNDED\nno verified pairs\nat all", fillcolor="#dc3545", fontcolor="white"] + + // Suffix annotation + partial [label="HYBRID → add `-PARTIAL` suffix\non whichever rung applies", fillcolor="#fff0e0", shape=note] + + // Decision flow + inputs -> is_ungrounded + is_ungrounded -> ungrounded [label="yes"] + is_ungrounded -> has_warrant_miss [label="no"] + has_warrant_miss -> pointer_linked [label="yes"] + has_warrant_miss -> has_soft_demote [label="no"] + has_soft_demote -> anchor_warranted [label="yes"] + has_soft_demote -> evidence_warranted [label="no"] + + // HYBRID adds suffix to all 3 non-UNGROUNDED rungs + evidence_warranted -> is_hybrid [style=dotted] + anchor_warranted -> is_hybrid [style=dotted] + pointer_linked -> is_hybrid [style=dotted] + is_hybrid -> partial [label="yes" style=dotted] + + {rank=same; evidence_warranted; anchor_warranted; pointer_linked; ungrounded} +} diff --git a/docs/diagrams/verifier-ladder.png b/docs/diagrams/verifier-ladder.png new file mode 100644 index 0000000..0f51ffc Binary files /dev/null and b/docs/diagrams/verifier-ladder.png differ diff --git a/docs/diagrams/verifier-ladder.svg b/docs/diagrams/verifier-ladder.svg new file mode 100644 index 0000000..c2ed7f9 --- /dev/null +++ b/docs/diagrams/verifier-ladder.svg @@ -0,0 +1,168 @@ + + + + + + +verifier_ladder + + + +inputs + + + +VERIFIER OUTPUT +audit_mode ∈ {STRICT, HYBRID, UNGROUNDED} +violations: list[dict] + + + +is_ungrounded + +audit_mode == UNGROUNDED? + + + +inputs->is_ungrounded + + + + + +has_warrant_miss + +WARRANT_MISSING or +TITLE_MISMATCH in violations? + + + +is_ungrounded->has_warrant_miss + + +no + + + +ungrounded + +UNGROUNDED +no verified pairs +at all + + + +is_ungrounded->ungrounded + + +yes + + + +has_soft_demote + +any soft-demote violation? +(LAZY_ANCHOR_DEMOTED, +POINTER_OVERFLOW_TRIMMED, +TOO_MANY_CLAIMS, …) + + + +has_warrant_miss->has_soft_demote + + +no + + + +pointer_linked + +POINTER-LINKED +pointer/source/chunk verified; +warrant didn't apply OR failed +for at least one claim + + + +has_warrant_miss->pointer_linked + + +yes + + + +evidence_warranted + +EVIDENCE-WARRANTED +pointer verified + +warrant ran & passed + +NO soft demotes + + + +has_soft_demote->evidence_warranted + + +no + + + +anchor_warranted + +ANCHOR-WARRANTED +pointer-linked + +warrant passed where it ran; +soft demotes present + + + +has_soft_demote->anchor_warranted + + +yes + + + +is_hybrid + +audit_mode == HYBRID? + + + +partial + + + +HYBRID → add `-PARTIAL` suffix +on whichever rung applies + + + +is_hybrid->partial + + +yes + + + +evidence_warranted->is_hybrid + + + + + +anchor_warranted->is_hybrid + + + + + +pointer_linked->is_hybrid + + + + + diff --git a/docs/modules/concepts.md b/docs/modules/concepts.md new file mode 100644 index 0000000..9838714 --- /dev/null +++ b/docs/modules/concepts.md @@ -0,0 +1,179 @@ +# `aborist.concepts` + +Per-shard `concept_relations` SQLite table — the corpus-derived +synonym, rivalry, antonym & category overlay that replaces the +hand-curated frozensets that lived in `aborist/qa/concepts.py` +through April 2026. + +**Full architecture rationale lives in +[`../concept-relations-design.md`](../concept-relations-design.md)** +including the 1.6% storage-tax measurement and the three-compactions- +considered-and-rejected analysis. This page is the API reference. + +## Sub-modules + +### `concepts.store` — append-only CRUD + +```python +from aborist.concepts.store import ( + add_concept_relation, # idempotent INSERT OR IGNORE + concept_relations_for_token, # read all relations for a token + purge_by_evidence_kind, # the only DELETE path + list_evidence_kinds, # diagnostic + RELATION_KINDS, # ('synonym','antonym','rivalry','category') +) +``` + +UNIQUE on `(source_root, relation_kind, token, target, evidence_kind)` +makes re-derivation idempotent. `purge_by_evidence_kind` lets an +operator revoke a single extractor's output without touching manual +or other-extractor rows. + +[`aborist/concepts/store.py`](../../aborist/concepts/store.py) + +### `concepts.query` — cross-shard lookup + +```python +from aborist.concepts import ( + synonym_expand, # query tokens → expanded set + rivalry_excluded, # query tokens → tokens to drop from results + has_compare_phrasing, # bool — does the query say "vs", "compare", etc. + invalidate_cache, # drop the per-process LRU +) + +expanded = synonym_expand({"thoughts"}, shards_dir=shards_dir) +# → {"thoughts", "telepathy", "neurotechnology", "mind", "cognition", ...} +``` + +Two synonym indices are loaded: + +- **`manual_index`** — `manual_legacy` + `manual` rows. Curated; + always expanded regardless of per-token degree. Captures the + brain-tech / AMD-family / Mac / Linux / etc. seed groups. +- **`derived_index`** — `link_reciprocity` & other corpus-derived + edges. Subject to **`MAX_NEIGHBORS_PER_TOKEN=8`** cap because the + Wikipedia link graph carries topic-adjacency noise on generic + tokens (person, thoughts, language). + +Overall **`MAX_TOTAL_TOKENS=50`** cap on expanded set bounds the +SQL clause count downstream so retrieval stays sub-second. + +A per-process LRU keyed on shard mtime avoids re-loading the index +on every query (290k rows across 4 shards loads in ~1.8s cold). + +[`aborist/concepts/query.py`](../../aborist/concepts/query.py) + +### `concepts.extract` — pluggable extractor framework + +```python +from aborist.concepts.extract import ( + EXTRACTORS, # registry: evidence_kind → callable + link_reciprocity_synonym, # built-in extractor +) + +# Run an extractor against a shard: +result = link_reciprocity_synonym(conn, derived_from="backfill@2026-05-01") +# → {"reciprocal_pairs": N, "synonyms_inserted": M, "synonyms_skipped": K} +``` + +Each extractor walks the shard's existing rows (`documents`, +`chunks`, `edges`, `derivations`) — **no new crawler needed** — & +emits concept relations under a stable `evidence_kind` string that +supports targeted purge. + +The built-in `link_reciprocity_synonym` reads the existing `edges` +table for reciprocal A↔B link pairs and emits a synonym edge between +every (title-token-of-A, title-token-of-B) pair. Works for +Wikipedia (See-also bidirectional), HTML site internal links +(russell.ballestrini.net pattern), or any document graph with +bidirectional links. Title-tokens are filtered to ≥4 chars + +stopword-stripped. + +To add a new extractor: + +1. Implement `(conn, *, derived_from) -> dict[str, int]` that calls + `add_concept_relation` for each finding. +2. Pick a stable `evidence_kind` string. +3. Register in `EXTRACTORS`. + +[`aborist/concepts/extract.py`](../../aborist/concepts/extract.py) + +### `concepts.seed` — legacy frozenset migration + +One-shot migration of the 8 hand-curated frozenset groups +(AMD-family, Intel-family, HTTP, FTP, Mac, Windows, Linux, +brain-tech) to `evidence_kind='manual_legacy'` rows. + +Writes **clique edges** within each group — every `(a, b)` pair — +so any member retrieves every other member (preserves the legacy +frozenset semantic where lookups didn't depend on which token in +the group was the anchor). + +```python +from aborist.concepts.seed import seed_legacy_concepts +result = seed_legacy_concepts(conn) +# → {"synonyms_inserted": N, "rivalries_inserted": M, "skipped": K} +``` + +Idempotent — re-running adds nothing if all the rows already exist. + +[`aborist/concepts/seed.py`](../../aborist/concepts/seed.py) + +## Data model + +```sql +CREATE TABLE concept_relations ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + source_root TEXT NOT NULL, + relation_kind TEXT NOT NULL CHECK (relation_kind IN + ('synonym','antonym','rivalry','category')), + token TEXT NOT NULL, + target TEXT NOT NULL, + evidence_kind TEXT NOT NULL, + confidence REAL NOT NULL DEFAULT 1.0, + derived_at INTEGER NOT NULL, + derived_from TEXT, + UNIQUE (source_root, relation_kind, token, target, evidence_kind) +); +``` + +**Append-only by construction.** Re-running an extractor adds +nothing if every relation already exists. No `UPDATE` path; only +`add_concept_relation` (insert) and `purge_by_evidence_kind` +(targeted delete). + +**Per-shard storage.** Concept relations live in the shard whose +document derived them. Mesh sync moves shards between peers; +relations come along. + +**Orthogonal to Merkle.** Writes to `concept_relations` NEVER +affect `document_root`, `chunk_root`, or `cache_key`. Backfilling +relations is safe across the entire corpus without invalidating +any cached answer or breaking any audit chain. + +## Storage cost — measured + +Backfill on 4 wiki shards (3.47M docs, 10.75M resolved edges): + +| Shard | Reciprocal pairs | Synonyms | Storage | +|---|---:|---:|---:| +| 000.db | 13,562 | 71,288 | 23.52 MB | +| 001.db | 14,078 | 73,351 | 24.20 MB | +| 002.db | 13,708 | 72,576 | 23.92 MB | +| 003.db | 13,800 | 72,633 | 23.94 MB | +| **total** | **55,148** | **289,848** | **95.58 MB** | + +**1.6% storage tax** on the 6 GB corpus. Backfill takes ~4 min +wall-clock total. Cost is paid once at backfill; every retrieval- +time lookup benefits. + +## Diagrams + +![module graph](../diagrams/aborist-modules.svg) +![query pipeline](../diagrams/query-pipeline.svg) + +## Source + +- [`aborist/concepts/`](../../aborist/concepts/) — package +- [`docs/concept-relations-design.md`](../concept-relations-design.md) — full design doc +- Whitepaper §13.4.11 — public-facing summary diff --git a/docs/modules/distill.md b/docs/modules/distill.md new file mode 100644 index 0000000..774a124 --- /dev/null +++ b/docs/modules/distill.md @@ -0,0 +1,76 @@ +# `aborist.distill` + +Surface → core distillation. Takes a set of surface documents +(the original ingest layer) and produces "core" documents — shorter, +more focused, Merkle-bound back to their contributing surface +chunks via per-chunk inclusion proofs. + +The "trees and forests of cross-linked information" tagline aborist +takes its name from comes from this layer: planet-toward-center +compression where each layer of cores derives from the previous, +recursively. + +## Layered design + +``` +distill/ +├── base.py Distiller ABC + DistillationResult dataclass +├── first_sentence.py no-ML stub: take the first sentence of each doc +├── tfidf.py pure-Python TF-IDF top-keyword extraction +└── runner.py batched distillation + per-contrib-chunk proofs +``` + +## `Distiller` ABC + +```python +from aborist.distill import Distiller, DistillationResult + +class Distiller(ABC): + @abstractmethod + def distill(self, docs: list[Document]) -> DistillationResult: ... +``` + +Each `DistillationResult` carries the new core's content + +references to every contributing surface chunk by `(document_root, +chunk_root)`. The runner writes one `derivations` row per core, +with `proof_blob = json.dumps(per_chunk_inclusion_proofs)`. + +## Built-in distillers + +### `FirstSentenceDistiller` (no-ML stub) + +Take the first sentence of each input doc, concatenate. Used as a +sanity-check for the pipeline + a baseline for measuring the +benefit of richer distillers. + +### `TfidfKeywordDistiller` + +Pure-Python TF-IDF. Computes term frequencies across the input +doc set + inverse document frequencies; emits the top-K terms per +doc as the core's content. The "permacomputer" neologism case +(every Grok conversation has the word, no Wikipedia article does) +is the canonical TF-IDF win — surfaces the topic that title-search +can't catch. + +## Why distill + +Three use cases: + +1. **Retrieval signal.** Cores feed the third accept path in + `_filter_by_title_relevance` — `core_match_roots` (TF-IDF top- + keywords contain a query token). Closes the gap for neologisms + that never make Wikipedia titles but ARE distinctive. + +2. **Hot/cold tier discipline.** `evict_to_cold` only touches + `kind='surface'` — cores never evict. Distilling surface to + cores then evicting surfaces gives a "long tail keeps small + cache" pattern with full provenance preserved. + +3. **Recursive abstraction.** Cores can themselves be distilled + into shorter cores. Each generation Merkle-binds back to the + previous via `derivations.proof_blob` — the audit chain stays + intact across an arbitrary distillation depth. + +## Source + +[`aborist/distill/`](../../aborist/distill/) diff --git a/docs/modules/document.md b/docs/modules/document.md new file mode 100644 index 0000000..db0e76c --- /dev/null +++ b/docs/modules/document.md @@ -0,0 +1,68 @@ +# `aborist.document` + +The data structures every source produces and every storage layer +consumes. Three core types: `Document`, `Edge`, `Chunker`. + +## `Document` + +A single ingest unit: a Wikipedia article, an HTML page, a Grok +conversation, a git commit message, etc. Carries both the raw +content AND the version tags that determine its identity: + +```python +@dataclass(frozen=True) +class Document: + document_uri: str # canonical URI (or stable surrogate for non-URI sources) + raw_content: str # source-of-truth bytes pre-canonicalization + kind: str # 'surface' / 'core' / 'visual' / etc. + chunking_version: str # e.g. 'tok-512-v1' — pinned by the Chunker + canonicalization_version: str # e.g. 'norm-v1' — pinned by canonicalize() + schema_version: str # e.g. 'v9.8.0' — store schema generation + title: str | None = None + edges: list[Edge] = () # outbound link graph + metadata: dict = ... # source-specific opaque payload +``` + +`document_root` is computed at ingest time as the Merkle root over +the canonicalized chunks. Two peers ingesting the same source + +running the same `chunking_version` + `canonicalization_version` +get bit-identical `document_root`s — the v9.8 admissibility property. + +## `Edge` + +One outbound link. `aborist/sources/wikipedia.py` emits one Edge +per `[[wikilink]]`; `aborist/sources/html_page.py` emits one per +``. The link graph IS the corpus topology — `concepts/extract.py` +later reads `edges` rows to derive synonym relations from +reciprocal links (no separate crawler needed). + +```python +@dataclass(frozen=True) +class Edge: + src_root: str # source document_root + dst_uri: str # always present + dst_root: str # '' (unresolved) until the dst doc is also ingested + edge_type: str # 'wikilink' / 'href' / 'citation' / 'derived_from' / ... + anchor: str # chunk index or fragment, '' if N/A +``` + +## `Chunker` + +ABC with one method `chunk(text: str) -> list[str]`. Default impl +is `TokenChunker` (`name='tok-512-v1'`) — splits on token-rough +windows so the resulting chunks are predictable for downstream FTS5 +indexing & for the LLM context budget. + +**Changing the chunker bumps `chunking_version` AND stales every +prior cache record** (chunking is one of the 8 cache_key dimensions). +Don't redefine `tok-512-v1`; add a new chunker as a new `name` +instead. + +## Diagrams + +![module graph](../diagrams/aborist-modules.svg) +![ingest pipeline](../diagrams/ingest-pipeline.svg) + +## Source + +[`aborist/document.py`](../../aborist/document.py) diff --git a/docs/modules/evict.md b/docs/modules/evict.md new file mode 100644 index 0000000..c5c596e --- /dev/null +++ b/docs/modules/evict.md @@ -0,0 +1,50 @@ +# `aborist.evict` + +Hot ↔ cold tier transitions. The corpus is large (3.47M Wikipedia +docs); not every chunk fits in working memory. `evict.py` is the +mechanism that moves rarely-touched chunks to a cold tier (still +indexed, just stored separately) and rehydrates them on demand +from the original source. + +## API surface + +```python +from aborist.evict import evict_to_cold, rehydrate + +# Move chunks unused for >threshold days to cold tier +evict_to_cold(conn, max_age_days=90, max_evictions=10000) + +# Pull a cold chunk back to hot from its original source +rehydrate(conn, document_root="abc123...") +``` + +## Invariant: cores never evict + +`evict_to_cold` filters `WHERE kind='surface'`. Cores are always +hot — they're the long-tail-friendly compression layer that justifies +evicting their underlying surfaces. Evicting cores would defeat the +purpose. + +## v9.8 falsification on drift + +When `rehydrate()` re-fetches a document and the recomputed +`document_root` differs from the stored one, the source has +changed since ingest (Wikipedia article was edited, HTML page was +republished, etc.). The cache record's `falsification_state` flips +from `live` to `stale` — every providence record keyed on that +`source_root` is no longer admissible to lookups. + +This is the **drift-detection-as-falsification** discipline: cache +hits don't blindly trust historical answers; they trust answers +that the SAME source still grounds. + +## Tier values + +`chunks.tier ∈ {'hot', 'cold'}`. Hot chunks live in `chunks.content`; +cold chunks live with NULL `content` and a `cold_uri` pointing at +the source. The QA pipeline's chunk-fetch path checks `tier`; on +'cold', it triggers `rehydrate` before continuing. + +## Source + +[`aborist/evict.py`](../../aborist/evict.py) diff --git a/docs/modules/index.md b/docs/modules/index.md new file mode 100644 index 0000000..f12c26a --- /dev/null +++ b/docs/modules/index.md @@ -0,0 +1,97 @@ +# Aborist module reference + +One page per top-level package + one diagram per major data flow. +This index is the entry point — every module page links back here +and to the relevant diagrams. + +## Diagrams + +| Diagram | What it shows | File | +|---|---|---| +| **Module graph** | Top-level packages & how they import each other | [`aborist-modules.svg`](../diagrams/aborist-modules.svg) ([dot](../diagrams/aborist-modules.dot)) | +| **Query pipeline** | Question → cache → retrieval → LLM → verify → render | [`query-pipeline.svg`](../diagrams/query-pipeline.svg) ([dot](../diagrams/query-pipeline.dot)) | +| **Ingest pipeline** | Source document → Merkle-committed shard | [`ingest-pipeline.svg`](../diagrams/ingest-pipeline.svg) ([dot](../diagrams/ingest-pipeline.dot)) | +| **Verifier ladder** | (audit_mode, violations) → display rung | [`verifier-ladder.svg`](../diagrams/verifier-ladder.svg) ([dot](../diagrams/verifier-ladder.dot)) | +| **Mesh data flow** | Federation: roster, gossip, AEAD envelope | [`mesh-data-flow.svg`](../diagrams/mesh-data-flow.svg) | +| **Mesh epoch lifecycle** | Epoch advance via add/kick/rotate | [`mesh-epoch-lifecycle.svg`](../diagrams/mesh-epoch-lifecycle.svg) | +| **Mesh identity stack** | Ed25519 sign + X25519 DH key derivation | [`mesh-identity-stack.svg`](../diagrams/mesh-identity-stack.svg) | +| **Mesh secret envelope** | AEAD-wrapped epoch secret per peer | [`mesh-secret-envelope.svg`](../diagrams/mesh-secret-envelope.svg) | +| **Mesh group decisions** | Membership change voting & quorum | [`mesh-group-decisions.svg`](../diagrams/mesh-group-decisions.svg) | + +Render diagrams locally: + +``` +make docs # runs `dot -Tsvg` and `-Tpng` on every docs/diagrams/*.dot +``` + +## Substrate (no SQL, pure data structures) + +| Module | One-line role | Doc | +|---|---|---| +| [`merkle.py`](../../aborist/merkle.py) | Merkle tree + proof — Python port of `proxy.unturf.com/pkg/verified/merkle.go` | [merkle.md](./merkle.md) | +| [`document.py`](../../aborist/document.py) | `Document`, `Edge`, `Chunker` (default `tok-512-v1`) | [document.md](./document.md) | +| [`wikitext.py`](../../aborist/wikitext.py) | `to_base()` — wikitext → plain prose, BASE_VERSION-pinned | [wikitext.md](./wikitext.md) | + +## Storage + +| Module | One-line role | Doc | +|---|---|---| +| [`store.py`](../../aborist/store.py) | v9.8 SQLite schema + audit chain helpers | [store.md](./store.md) | +| [`ingest.py`](../../aborist/ingest.py) | normalize → chunk → merkle → upsert (bulk-batched) | [ingest.md](./ingest.md) | +| [`evict.py`](../../aborist/evict.py) | hot ↔ cold tier transitions; rehydrate via source | [evict.md](./evict.md) | + +## Sources (corpus producers) + +| Module | One-line role | Doc | +|---|---|---| +| [`sources/wikipedia.py`](../../aborist/sources/wikipedia.py) | Wikipedia 2003 cur + old SQL dumps (bz2-streamed) | [sources.md](./sources.md) | +| [`sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py) | Phase IV XML dumps (iterparse, page + history) | [sources.md](./sources.md) | +| [`sources/html_page.py`](../../aborist/sources/html_page.py) | URL list + selectolax + httpx (robots-aware) | [sources.md](./sources.md) | +| [`sources/crawler/`](../../aborist/sources/crawler/) | verbatim AsyncWebFetcher lift + ingest bridge | [sources.md](./sources.md) | +| [`sources/grok.py`](../../aborist/sources/grok.py) | xAI data export (conversations + media prompts) | [sources.md](./sources.md) | +| [`sources/vcs.py`](../../aborist/sources/vcs.py) | git + Mercurial repos (HEAD walk, supersedes chain) | [sources.md](./sources.md) | + +## Search & retrieval + +| Module | One-line role | Doc | +|---|---|---| +| [`search/`](../../aborist/search/) | FTS5 backend + `SearchBackend` ABC + `AuditMode` enum | [search.md](./search.md) | +| [`concepts/`](../../aborist/concepts/) | Per-shard `concept_relations` synonym/rivalry overlay | [concepts.md](./concepts.md) | + +## Q&A pipeline + +| Module | One-line role | Doc | +|---|---|---| +| [`qa/keys.py`](../../aborist/qa/keys.py) | 8-dim cache_key + `question_hash` | [qa.md](./qa.md) | +| [`qa/client.py`](../../aborist/qa/client.py) | `ChatClient` + `StubClient` + `OpenAICompatibleClient` | [qa.md](./qa.md) | +| [`qa/runner.py`](../../aborist/qa/runner.py) | `ask()`: single-doc Q&A + cache + verify | [qa.md](./qa.md) | +| [`qa/query.py`](../../aborist/qa/query.py) | `query()`: multi-source RAG + concept overlay | [qa.md](./qa.md) | +| [`qa/verify.py`](../../aborist/qa/verify.py) | quote/span/entity/paraphrase + claim_lattice (7 hard checks) | [qa.md](./qa.md) | +| [`qa/warrant.py`](../../aborist/qa/warrant.py) | 5 anchor classes (proper-noun · date · count · entity-list · cause) | [qa.md](./qa.md) | +| [`qa/evidence.py`](../../aborist/qa/evidence.py) | `EvidenceObject` + spotlight excerpt (density rank) | [qa.md](./qa.md) | +| [`qa/parse_claims.py`](../../aborist/qa/parse_claims.py) | pointer-line parser (`claim. [E1,E2]`) | [qa.md](./qa.md) | +| [`qa/dag.py`](../../aborist/qa/dag.py) | per-run Merkle-DAG (7-stage quote / 9-stage CTI) | [qa.md](./qa.md) | +| [`qa/inspect.py`](../../aborist/qa/inspect.py) | sidecar diagnostic (read-only span classifier) | [qa.md](./qa.md) | + +## Distillation + +| Module | One-line role | Doc | +|---|---|---| +| [`distill/`](../../aborist/distill/) | `Distiller` ABC + `first_sentence` + `tfidf` + runner | [distill.md](./distill.md) | + +## Federation (off by default) + +| Module | One-line role | Doc | +|---|---|---| +| [`mesh/`](../../aborist/mesh/) | identity (Ed25519/X25519), per-epoch roster, AEAD envelope, gossip wire | [../mesh.md](../mesh.md), [../mesh-deploy.md](../mesh-deploy.md) | + +## Entry point + +| Module | One-line role | Doc | +|---|---|---| +| [`cli.py`](../../aborist/cli.py) | `argparse` entrypoint — every `make` target dispatches here | run `aborist --help` or any `make help` target | + +## Tickets, design docs, journals + +See [`../TICKETS.md`](../TICKETS.md) for the ticket index and the +list of design-reference docs that aren't tickets. diff --git a/docs/modules/ingest.md b/docs/modules/ingest.md new file mode 100644 index 0000000..916edb6 --- /dev/null +++ b/docs/modules/ingest.md @@ -0,0 +1,64 @@ +# `aborist.ingest` + +The bulk-batched pipeline that turns documents from a source into +Merkle-committed shard storage. Source-agnostic: anything that +implements `Source.iter_documents()` flows through here. + +![ingest pipeline](../diagrams/ingest-pipeline.svg) + +## Public API + +```python +from aborist.ingest import ingest_source +from aborist.sources.wikipedia import WikipediaSqlDump + +source = WikipediaSqlDump("/path/to/cur.sql.bz2") +ingest_source( + source, + db_path=Path("~/.aborist/shards/000.db"), + batch_size=200, # docs per transaction + progress_every=1000, +) +``` + +## What happens per document + +1. **Canonicalize** — `canonicalize(text)`: NFC + ws-collapse + + strip ends. Pinned by `canonicalization_version='norm-v1'`. +2. **Chunk** — `Chunker.chunk(canonical_text)` → list of token- + bounded substrings. Default `tok-512-v1` chunker. +3. **Hash leaves** — `sha256(0x00 || canonical_chunk_bytes)` per + chunk. +4. **Merkle tree** — `MerkleTree.build(leaves).root` → + `document_root`. Two peers running the same chunker on the same + canonicalized content compute bit-identical roots. +5. **Upsert** — `documents` row keyed on `document_root` (idempotent + re-ingest), `chunks` rows with leaf hashes, `merkle_nodes` for + proof reconstruction, `edges` per outbound link. +6. **FTS5** — `chunks_fts` insert with rowid = `chunks.chunk_id` + so the search-time JOIN lines up. +7. **Audit event** — one row per ingest batch in `audit_events`, + chained on `prev_event_hash`. + +## Batching discipline + +Default `batch_size=200`: balances Python GIL overhead vs SQLite +transaction commit cost. Lower it (e.g. 50) only to bound peak +memory on a low-RAM host. Higher (e.g. 1000) for ETL throughput on +SSD storage when memory isn't tight. + +`progress_every` prints a stderr line every N docs so long ingests +are observable. Use `PYTHONUNBUFFERED=1` for tail-able output. + +## Resumability + +Idempotent re-ingest: same content + same chunker + same canonicalize += same `document_root` = no-op insert. So a crashed ingest can be +restarted from the source's beginning without duplicating rows. + +Different content at the same URI gets a new `document_root` AND a +`supersedes` edge linking new → old (lossless history). + +## Source + +[`aborist/ingest.py`](../../aborist/ingest.py) diff --git a/docs/modules/merkle.md b/docs/modules/merkle.md new file mode 100644 index 0000000..28edaaa --- /dev/null +++ b/docs/modules/merkle.md @@ -0,0 +1,63 @@ +# `aborist.merkle` + +Pure Merkle tree + proof primitives. Python port of +`proxy.unturf.com/pkg/verified/merkle.go` — convention-identical. +Used everywhere a content-addressable handle is needed: per-chunk +leaves, document_root, evidence_map_root, run_dag_root, snapshots. + +## Conventions (do not silently change) + +These match the Go reference & are load-bearing for cross-language +verification (Go peer ↔ Python peer compute bit-identical roots): + +- **Leaf hash:** `sha256(0x00 || canonical_chunk_bytes)`. The `0x00` + prefix domain-separates leaves from internal nodes. +- **Internal hash:** `sha256(0x03 || left || right)`. The `0x03` + prefix is the **non-commutative** combine — `H(L,R) ≠ H(R,L)`. + Order matters. +- **Odd-element rule:** when a level has an odd count, the last + leaf is **self-duplicated** before pairing. NOT zero-padded. +- **Proof path:** each step carries an explicit `is_left: bool` + alongside the sibling hash so a verifier knows which side to put + the sibling on. Never sort siblings lexically — the order tells + the verifier the tree topology. + +## API surface + +```python +from aborist.merkle import MerkleTree, MerkleProof + +tree = MerkleTree.build([b"chunk_0_bytes", b"chunk_1_bytes", ...]) +tree.root # bytes(32) — sha256 of the whole tree +tree.leaves # list[bytes(32)] — leaf hashes in input order + +proof = tree.proof_for(leaf_index=2) +proof.siblings # list[(sibling_hash, is_left)] +proof.verify(leaf_hash=tree.leaves[2], root=tree.root) # bool +``` + +## When to read the source + +- Adding a new content-addressable artifact (cores, evidence maps, + snapshots, run-DAGs all touch this). +- Cross-language verification debugging (Go peer says one root, + Python peer says another — the difference is always in canonical + encoding, ordering, or one of the three prefix bytes above). +- Performance work — the Python build is ~3× slower than the Go + reference; if it ever shows up in profiling, that's the file. + +## Diagrams + +The module graph shows what depends on `merkle.py` (a lot — it's +substrate): + +![module graph](../diagrams/aborist-modules.svg) + +The ingest pipeline shows where leaf & root hashes get computed: + +![ingest pipeline](../diagrams/ingest-pipeline.svg) + +## Source + +[`aborist/merkle.py`](../../aborist/merkle.py) · +Reference: [`proxy.unturf.com/pkg/verified/merkle.go`](https://git.unturf.com/engineering/unturf/proxy.unturf.com/-/blob/main/pkg/verified/merkle.go) diff --git a/docs/modules/qa.md b/docs/modules/qa.md new file mode 100644 index 0000000..129ad63 --- /dev/null +++ b/docs/modules/qa.md @@ -0,0 +1,185 @@ +# `aborist.qa` + +The Q&A pipeline. Question → cache → retrieval → LLM → verify → +render → cache write. Lives in 9 sub-modules; this page is the +map. + +![query pipeline](../diagrams/query-pipeline.svg) + +## Sub-modules + +### `qa.client` — LLM transport + +`ChatClient` ABC with three concrete implementations: + +- `StubClient` — deterministic test fixture; returns canned + responses keyed on the input. Used in unit tests to avoid network. +- `OpenAICompatibleClient` — talks to any OpenAI-shape `/v1/chat/completions` + endpoint (Hermes-3 on vLLM by default). Includes HTTP retry layer + (3× exponential backoff on 5xx). +- (Future) `AnthropicClient` — Claude API direct. + +[`aborist/qa/client.py`](../../aborist/qa/client.py) + +### `qa.keys` — the 8-dim cache_key + +``` +cache_key = sha256( + source_root | question_hash | model_profile_hash | + conversation_hash | governance_policy_hash | + schema_version | canonicalization_version | chunking_version +) +``` + +Two question-hash modes (`strict` vs `equivalence_class`) live here. +Bumping any of these eight dimensions invalidates prior records on +lookup. The `verifier_policy_hash` (v9.9 9th dim) is also +implemented here. + +[`aborist/qa/keys.py`](../../aborist/qa/keys.py) + +### `qa.runner` — `ask()` for single-doc Q&A + +The simplest entry point. Take one document, ask one question, get +back an answer + audit_mode + cache record. Used by the CLI for +focused queries against one URI. + +[`aborist/qa/runner.py`](../../aborist/qa/runner.py) + +### `qa.query` — `query()` for multi-source RAG + +The main retrieval entry point. Walks shards, runs FTS5 BM25 with +AND→OR fallback (with synonym-pool injection in OR mode), filters +by title relevance with 4 accept paths, reranks by body coverage + +title boost + source role + title purity, assembles a 60 KB context +budget, calls the LLM, runs the verifier, persists to +`providence_cache`. + +[`aborist/qa/query.py`](../../aborist/qa/query.py) + +### `qa.verify` — the layered verifier + +Five strategies run in sequence; first to find evidence classifies: +1. `quote` — `"..."`-wrapped claims tested verbatim +2. `span` — bullet/sentence units substring-tested +3. `entity` — multi-word proper nouns with proximity gating +4. `paraphrase` — token coverage on prose-shaped spans (≥85%) +5. `claim_lattice` — pointer-line `[E1,E2]` or JSON; runs **seven + deterministic hard checks**: + 1. parser succeeded + 2. evidence_id resolves + 3. source_role allowed + 4. claim text non-empty + 5. citation coverage threshold + 6. pointer count cap (trim-and-verify) + 7. anchor-class warrant (see `qa.warrant`) + +The classifier output rolls up into the v9.8 trichotomy +`audit_mode ∈ {STRICT, HYBRID, UNGROUNDED}`. Display layer (in +`cli.py`) maps `(audit_mode, violations) → four-rung ladder`. + +![verifier ladder](../diagrams/verifier-ladder.svg) + +[`aborist/qa/verify.py`](../../aborist/qa/verify.py) + +### `qa.warrant` — anchor-class warrant + +Five lexical anchor classes the verifier composes: + +- **Proper-noun** — relation-question shape; at least one + Title-Case anchor must appear in some cited span +- **Date** — claim has a 4-digit year + month name; ALL components + required in some cited span +- **Entity-list** — entity-list-shape question; ≥1 named entity + must anchor (demote-don't-reject) +- **Count** — count-shape question; count token must appear in + word OR digit form (digit↔word equivalence) +- **Cause** — why-shape question; ≥1 cause anchor (proper noun OR + ≥5-char common noun outside stopword pool) + +The warrant layer earns proof-path entry by staying **lexical** — +no NLI, no embeddings. Substring tests over already-canonicalized +spans. See `docs/concept-relations-design.md` (sibling section) +for the relationship to retrieval-time synonym expansion. + +[`aborist/qa/warrant.py`](../../aborist/qa/warrant.py) + +### `qa.evidence` — EvidenceObject + spotlight + +Builds the runtime evidence map for claim-lattice modes. Each +chunk becomes one `EvidenceObject` carrying TWO ids: + +- `pointer_id` — short prompt-facing tag (`E1`, `E2`, …) +- `evidence_id` — content-addressed `E########` (sha256-derived) + +The model sees only `pointer_id`s in the prompt; the runtime maps +to `evidence_id` for the cache & run-DAG (run-stable identity). + +The spotlight excerpt picks the load-bearing slice via **density +rank** — find ALL match positions for ALL claim content tokens, +pick the position with maximum distinct-token cluster within +±half-window. Replaces the older first-match-of-longest-token +approach which lost the load-bearing slice on noisy chunks. + +[`aborist/qa/evidence.py`](../../aborist/qa/evidence.py) + +### `qa.parse_claims` — pointer-line parser + +Walks lines of the model output, pulls every `[E\d+]` and +`[E\d+,E\d+,…]` bracket payload, returns +`(claim_text, pointer_ids[])` per line. Lines without a tag get +`parse_status='NO_EVIDENCE_POINTER'` & count toward the denominator +so unsourced prose can't smuggle past the verifier. + +[`aborist/qa/parse_claims.py`](../../aborist/qa/parse_claims.py) + +### `qa.dag` — per-run Merkle DAG + +Commits each provenance step independently as a stage hash. Two +shapes: + +- **7-stage (quote mode):** question / retrieval / context / + prompt / answer / verify / final_label +- **9-stage (claim-lattice / CTI):** question / retrieval / + evidence_map / prompt / raw_answer / parsed_claim_lattice / + verify / render / final_label + +The `run_dag_root` is persisted alongside every providence record; +`run_dag_blob` carries the full `{root, nodes}` JSON so an auditor +can recompute & verify any step. + +[`aborist/qa/dag.py`](../../aborist/qa/dag.py) + +### `qa.inspect` — read-only sidecar + +Pulls source chunks for a given cache_key & classifies each +unverified span: `verbatim_in_base` / `verbatim_in_raw_only` / +`trailing_artifact` / `paraphrase` / `partial_paraphrase` / +`no_overlap`. Also includes the deflection-detection sidecar +(subject-anchor heuristic for adversarial-premise topic shift). + +**Sidecars never write to `providence_cache` or `audit_events`** — +they're diagnostic only. That invariant is what keeps `audit_mode` +a binary classification rather than a soft score. + +[`aborist/qa/inspect.py`](../../aborist/qa/inspect.py) + +### `qa.concepts` — backwards-compat shim + +Delegates to `aborist.concepts` (the corpus-derived synonym/rivalry +layer). Pre-2026-05-01 the data lived as hand-curated frozensets in +this file; now it's a per-shard SQLite table. The shim preserves +the legacy public API (`synonym_expand`, `rivalry_excluded`, +`has_compare_phrasing`) so call sites in `qa/query.py` didn't have +to change. + +[`aborist/qa/concepts.py`](../../aborist/qa/concepts.py) → +[`aborist/concepts/`](../../aborist/concepts/) + +## Source papers + +- Whitepaper §13.8 covers the layered verifier in depth +- Whitepaper §13.9 covers claim-lattice / CTI mode +- `docs/cti-architecture.md` is the architecture reference +- `docs/seven-point-program.md` enumerates the seven hard checks +- `docs/concept-relations-design.md` covers the synonym layer diff --git a/docs/modules/search.md b/docs/modules/search.md new file mode 100644 index 0000000..4dfe5c9 --- /dev/null +++ b/docs/modules/search.md @@ -0,0 +1,75 @@ +# `aborist.search` + +The retrieval primitive. Today's only backend is FTS5 over the +chunks table; the `SearchBackend` ABC is in place so additional +backends (BM25 over titles, embedding-based vector search) can be +added without touching the rest of the QA pipeline. + +## `SearchBackend` ABC + +```python +from aborist.search import SearchBackend, AuditMode, Hit + +class SearchBackend(ABC): + @abstractmethod + def search(self, query: str, limit: int = 20) -> list[Hit]: ... +``` + +Each `Hit` carries `(document_root, document_uri, chunk_idx, +snippet, score, audit_mode, title)`. `audit_mode` is the sticky +provenance label that tracks how the chunk made it into the index; +FTS5 backend always sets `UNGROUNDED` (search itself doesn't verify +anything — that's the QA pipeline's job). + +## `FTS5Backend` + +Wraps the contentless `chunks_fts` virtual table. Two-mode query: + +- **AND-mode (strict, primary):** every content token must appear + in the doc. Keeps unrelated docs out of the context window. +- **OR-mode (fallback):** when AND returns 0 hits, fall back to OR + but **capped to top-5 longest tokens** (proxy for rarity). Long + topical synonyms fed via `extra_or_tokens` join the pool — + `neurotechnology` (15 chars) outranks `thoughts` (8) by length + and surfaces brain-tech titles for vocabulary-mismatch queries. + +```python +from aborist.search import FTS5Backend +backend = FTS5Backend(conn) + +# Plain search +hits = backend.search("permacomputer", limit=32) + +# Search with synonym pool injection (used by qa.query._search_corpus) +hits = backend.search( + long_query, + limit=32, + extra_or_tokens=synonym_expand(qtokens, shards_dir=shards_dir), +) +``` + +## Stopword & stopword-cap discipline + +`_FTS5_STOPWORDS` filters question words (`what`, `tell`, `please`) ++ generic connectors (`one`, `some`, `another`, `without`, `soon`, +`currently`) before AND/OR construction. Two principles: + +- **Stay in sync with `_TITLE_STOPWORDS`** in `qa/query.py`. A token + filtered at retrieval time but kept at title-relevance check (or + vice versa) creates ranking incoherence. +- **`_OR_FALLBACK_MAX_TOKENS=5`** caps the OR-mode pool. Without + this, a 19-token OR clause matches millions of docs and forces + BM25 to rank them all — 13s/shard observed pre-cap. Now 0.25s/shard. + +## Snippet building + +FTS5 contentless mode means SQLite's built-in `snippet()` and +`highlight()` return empty. Aborist builds snippets in Python by +joining `chunks_fts.rowid = chunks.chunk_id`, decompressing the +chunk content, and locating query tokens locally +(`_build_snippet`). + +## Source + +[`aborist/search/fts5.py`](../../aborist/search/fts5.py) · +[`aborist/search/__init__.py`](../../aborist/search/__init__.py) diff --git a/docs/modules/sources.md b/docs/modules/sources.md new file mode 100644 index 0000000..e8ff2c4 --- /dev/null +++ b/docs/modules/sources.md @@ -0,0 +1,80 @@ +# `aborist.sources` + +Corpus producers. Each is a `Source` ABC implementation that yields +`Document` instances; the standard `ingest.ingest_source(source, db)` +pipeline takes them from there. + +The `Source` ABC lives in [`aborist/source.py`](../../aborist/source.py): + +```python +class Source(ABC): + @abstractmethod + def iter_documents(self) -> Iterator[Document]: ... +``` + +## Built-in sources + +### `wikipedia.py` — Phase III SQL dumps (the canonical bootstrap) + +Streams the Wikipedia 2003 `cur` (current revisions) and `old` +(revision history) SQL dumps. Hand-rolled escape-aware parser +(no `sqlite3` import — the dump is MySQL syntax). 4× speedup vs +char-by-char loops via `str.find` + slicing. cProfile any change. + +Default Wikipedia 2003-05-16 dump source: +`https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`. +robots.txt returned 404 → no rules. + +[`aborist/sources/wikipedia.py`](../../aborist/sources/wikipedia.py) + +### `wikipedia_xml.py` — Phase IV XML dumps + +Modern Wikipedia dump format (`enwiki-YYYYMMDD-pages-articles.xml.bz2`, +`enwiki-YYYYMMDD-pages-meta-history*.xml.bz2`). Uses `xml.etree.ElementTree.iterparse` +to stream-parse without loading the whole tree. + +[`aborist/sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py) + +### `html_page.py` — single-URL or URL-list HTML ingest + +Robots-aware (`urllib.robotparser`). Uses `selectolax` for fast +HTML parsing (CSS-selector based; ~10× faster than `lxml`). Pulls +the main body text + every `` as an `Edge` row. + +The `edges` rows are what the corpus-derived synonym extractor +later reads — no separate crawler needed for site-internal link +graphs. + +Optional dep: `pip install '.[html]'` for `selectolax` + `httpx`. + +[`aborist/sources/html_page.py`](../../aborist/sources/html_page.py) + +### `crawler/` — async BFS web crawl + +Verbatim lift of an `AsyncWebFetcher` implementation + an `ingest` +bridge. BFS-discovers same-domain URLs from a seed, respecting +`robots.txt` + crawl delays. Captures ETag + Last-Modified per URL +into `document_http_meta` so a future recrawl can send conditional +HEAD requests. + +[`aborist/sources/crawler/bridge.py`](../../aborist/sources/crawler/bridge.py) + +### `grok.py` — xAI Grok export + +Reads the `xAI-conversations.json` data export shape. Each +conversation becomes one `Document`; media prompts are kept +inline. + +[`aborist/sources/grok.py`](../../aborist/sources/grok.py) + +### `vcs.py` — git + Mercurial repositories + +HEAD walk. Each commit becomes a `Document` (commit message + diff +stat). The supersedes chain captures commit ancestry as edges. + +[`aborist/sources/vcs.py`](../../aborist/sources/vcs.py) + +## Source + +[`aborist/sources/`](../../aborist/sources/) · +[`aborist/source.py`](../../aborist/source.py) (ABC) diff --git a/docs/modules/store.md b/docs/modules/store.md new file mode 100644 index 0000000..188a2f7 --- /dev/null +++ b/docs/modules/store.md @@ -0,0 +1,81 @@ +# `aborist.store` + +The v9.8 SQLite schema, the audit chain, and the cross-shard +read-only view. Every table that holds runtime state lives here. + +## Schema overview (per shard) + +``` +documents – one row per source document, keyed on document_root +chunks – per-document chunk content + tier (hot/cold) +chunks_fts – FTS5 contentless index, rowid = chunks.chunk_id +merkle_nodes – internal-node hashes for proof reconstruction +edges – src_root → dst_root link graph (wikilink, href, …) +derivations – core_root ← src_root with proof_blob (Merkle) +providence_cache – Q&A records keyed on the v9.8 8-dim cache_key +audit_events – linear chain; event_hash = sha256(prev || canonical(body)) +falsifications – record_id → state transition + reason + actor +snapshots – named corpus roots (one hash names a forest) +document_http_meta – ETag + Last-Modified for crawler conditional fetches +concept_relations – per-shard synonym/rivalry/category/antonym overlay +mesh_* – federation tables (off by default) +``` + +## v9.8 invariants (do not break) + +- **Aborist is a v9.8 store.** Every providence record carries the + full **8-dim cache_key**: `source_root | question_hash | + model_profile_hash | conversation_hash | governance_policy_hash | + schema_version | canonicalization_version | chunking_version`. + Bumping any one invalidates prior records on lookup. +- **`falsification_state ∈ {live, failed, stale, quarantined}`.** + Cache lookups must filter on `state='live'`. Drift detection + flips to `stale`. +- **Audit chain.** Every state-changing op writes one row in + `audit_events` with `event_hash = sha256(prev_event_hash || + canonical(body))`. Chain integrity is verified in + `make analyze-shards`. **Never insert into `audit_events` + directly — use `aborist.store.append_audit`.** +- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`. +- **Idempotent re-ingest.** Same content → same `document_root` → + no-op insert. Same URI + different content → new doc + `supersedes` + edge linking new → old (lossless history). + +## API surface + +```python +from aborist.store import ( + connect, # writable connection to a single shard + connect_query, # read-only UNION view across all shards + discover_shards, # list *.db files in a shards_dir + transaction, # BEGIN IMMEDIATE / COMMIT / ROLLBACK context manager + get_meta, set_meta, + append_audit, # the ONLY way to write audit_events +) +``` + +Cross-shard reads use `connect_query(shards_dir=...)` which ATTACHes +every `*.db` and creates UNION views over the shardable tables +(`documents`, `chunks`, `merkle_nodes`, `edges`, `derivations`, +`providence_cache`, `audit_events`, `falsifications`, +`concept_relations`). + +## Performance pragmas + +`connect()` applies these per-connection: +- `journal_mode=WAL` (set in SCHEMA_SQL once at first ingest) +- `synchronous=NORMAL` (skip per-commit fsync; safe under WAL) +- `cache_size=-65536` (64 MB page cache) +- `temp_store=MEMORY` (no /tmp churn for temp tables) + +Don't downgrade to `synchronous=FULL` without a measured reason — +costs ~5× throughput. + +## Diagrams + +![module graph](../diagrams/aborist-modules.svg) +![ingest pipeline](../diagrams/ingest-pipeline.svg) + +## Source + +[`aborist/store.py`](../../aborist/store.py) diff --git a/docs/modules/wikitext.md b/docs/modules/wikitext.md new file mode 100644 index 0000000..f4f5e3c --- /dev/null +++ b/docs/modules/wikitext.md @@ -0,0 +1,46 @@ +# `aborist.wikitext` + +A single function: `to_base(raw)`. Converts MediaWiki wikitext to +plain prose deterministically. + +```python +from aborist.wikitext import to_base, BASE_VERSION + +prose = to_base("[[The Beatles]] are an [[English rock band]] from [[Liverpool]].") +# → "The Beatles are an English rock band from Liverpool." +``` + +## Why it exists + +The corpus stores raw wikitext (so the link graph is recoverable +on demand) but the LLM and verifier both want plain prose. Reasons: + +1. **Token efficiency.** Wikipedia chunks ship to Hermes with ~43% + fewer tokens after wikitext-strip — bigger context window for + the same chars budget. +2. **Verbatim citation.** The model can quote source paragraphs + verbatim instead of escaping `[[wikilinks]]`. The verifier's + substring test then matches cleanly. +3. **Pinned identity.** `BASE_VERSION='wikitext-base-v1'` lives in + `policy["base_version"]`, which folds into + `governance_policy_hash`. Bumping `BASE_VERSION` invalidates + every prior cache record on next lookup — same discipline as + `chunking_version` and `canonicalization_version`. + +## Hot-path discipline + +`to_base()` runs on the assembled context **before** the LLM call +in `aborist/qa/runner.py` and `aborist/qa/query.py`, **and again +inside `verify_quotes`** so the verifier compares like-against-like. +Both sides see prose. + +## Optional dependency + +Backed by `mwparserfromhell`. Install via `pip install '.[wikitext]'` +to enable. Without the dep, `_wikitext_to_base = None` and +`policy["base_version"] = None` — graceful fallback leaves raw +wikitext in both context and verifier (works, just less efficient). + +## Source + +[`aborist/wikitext.py`](../../aborist/wikitext.py)