engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: README label refresh + per-module reference + Graphviz diagrams

Browse source 
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).

(1) README — label refresh:
    - Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
      four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
      EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
    - Verifier section spells out both layers (schema trichotomy +
      display ladder), the seven hard checks of claim_lattice, and
      the five anchor classes of warrant.
    - Architecture tree updated: concepts/ package added, qa/
      sub-modules expanded (warrant, evidence, parse_claims, dag),
      verify.py described as quote/span/entity/paraphrase + claim_lattice.
    - Concept overlay description updated for corpus-derived layer
      (concept_relations table, link_reciprocity extractor, 1.6%
      tax cite).
    - Test count: 326+ → 641+.

(2) docs/diagrams/ — Graphviz dot sources:
    - aborist-modules.dot — top-level package graph (substrate /
      storage / sources / retrieval / qa / mesh / cli)
    - query-pipeline.dot — question → cache → retrieval → LLM →
      verify → render → cache write, with phase budgets
    - ingest-pipeline.dot — source doc → canonicalize → chunk →
      Merkle → upsert (+ optional distill)
    - verifier-ladder.dot — (audit_mode, violations) → display rung
      decision tree
    Existing mesh-*.dot kept as-is. Makefile `make docs` target
    extended to also emit .svg alongside the existing .png so the
    diagrams render in markdown viewers.

(3) docs/modules/ — per-module reference pages:
    - index.md (links to every diagram + every module page)
    - merkle.md, document.md, store.md, ingest.md, evict.md,
      sources.md, search.md, concepts.md, qa.md, distill.md,
      wikitext.md
    Each page is a one-screenful concise reference: what the
    module is for, public API, key invariants, embedded diagrams
    where useful, link to source. Mesh stays at the existing
    docs/mesh.md + docs/mesh-deploy.md (already comprehensive).

Tests: 641 passed (no code change).
This commit is contained in:
russell@unturf.com 2026-05-01 23:19:01 -04:00
parent 2f7132b9ec
commit 326badf6d8
No known key found for this signature in database
31 changed files with 3473 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

73
README.md
View file

@ -19,7 +19,7 @@ make distill-shards-tfidf-parallel # core → keyword sets for retrieval
make query Q="What is anarcho-capitalism?"
```
A Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer plus a verifier audit_mode (STRICT / HYBRID / UNGROUNDED). Repeat the same question and a cache hit replays in ~100 ms.
A Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer plus a verifier label that names what the lexical verifier could confirm. The current four-rung ladder for claim-lattice modes is `POINTER-LINKED` → `ANCHOR-WARRANTED` → `EVIDENCE-WARRANTED` → `UNGROUNDED` (with `-PARTIAL` suffix on HYBRID). Repeat the same question and a cache hit replays in ~100 ms.
### B. Crawl any live website and query it
@ -282,10 +282,10 @@ Default render is human-readable: question, audit-mode summary line, answer, sou
The query path:
1. **Search** — FTS5 (body) + SQL `LIKE` (title) + `JOIN` over derivations (TF-IDF core keywords) across every shard. Three accept paths to the relevance filter.
2. **Concept overlay** — synonym groups (`Athlon` ↔ `AMD`) widen retrieval; rivalry pairs (AMD ↔ Intel) narrow it unless the query has comparative phrasing ("compare X vs Y").
2. **Concept overlay** — per-shard `concept_relations` SQLite table (corpus-derived, not hand-curated). Synonyms widen retrieval; rivalries narrow it unless the query uses comparative phrasing ("compare X vs Y"). Built-in extractor `link_reciprocity_synonym` reads the existing `edges` table for reciprocal A↔B link pairs and emits synonym edges between their title-tokens — works for Wikipedia (See-also bidirectional), HTML site internal links, or any corpus with bidirectional document links. See `docs/concept-relations-design.md` for the storage tradeoff (1.6% tax measured on 6 GB Wikipedia).
3. **Context assembly** — top-K sources concatenated up to a 60 KB budget. Wikitext is stripped to plain prose via `aborist.wikitext.to_base()` (the corpus stores raw `[[wikilinks]]` so the link graph is recoverable on demand; the LLM and verifier both see clean prose).
4. **LLM** — Hermes-3 with strict attribution rules in the system prompt + a user-turn grounding reminder.
5. **Verifier** — every claim is classified into the v9.8 trichotomy STRICT / HYBRID / UNGROUNDED via a layered lexical check (see below).
5. **Verifier** — every claim runs through a layered lexical check; the result rolls up into the v9.8 trichotomy (`audit_mode` ∈ STRICT / HYBRID / UNGROUNDED) at the schema layer AND a four-rung display ladder at render time (POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → UNGROUNDED). See below.
6. **Cache** — the v9.8 8-dim cache_key (`source_root | question_hash | model_profile | conversation | governance_policy | schema | canonicalization | chunking`) keys the answer in `qa.db`. Cache hits replay in ~100 ms. Per-phase timings in every result.
LLM endpoint defaults to `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B, 82K context, no auth). Override:
@ -296,9 +296,11 @@ export ABORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
export ABORIST_LLM_API_KEY="..."
```
## Verifying answers (audit modes)
## Verifying answers (audit modes & label ladder)
Every answer carries an `audit_mode` set by `aborist/qa/verify.py`:
Two layers of labels stack on every answer:
**Schema layer — v9.8 trichotomy** (`audit_mode` column, persisted, drives cache lookups & audit chain):
| mode | meaning |
|---|---|
@ -306,7 +308,18 @@ Every answer carries an `audit_mode` set by `aborist/qa/verify.py`:
| **HYBRID** | some claims source-grounded, some emerged from training |
| **UNGROUNDED** | no evidence, or none verifies — purely emergent |
Four strategies run in sequence; first to find evidence classifies. Each is lexical (substring or token-coverage), never embeddings — soft signals stay out of the proof path.
**Display layer — four-rung ladder** (claim-lattice modes only; renderer-only transformation, schema unchanged):
| rung | what's actually proved |
|---|---|
| **EVIDENCE-WARRANTED** | pointer verified + warrant ran & passed + no soft demotes |
| **ANCHOR-WARRANTED** | pointer-linked + warrant passed; soft-demote violations present |
| **POINTER-LINKED** | pointer/source/chunk verified, but warrant either didn't apply or failed for some claim |
| **UNGROUNDED** | no verified pairs |
HYBRID gets a `-PARTIAL` suffix on whichever rung applies. The point of the display ladder: `STRICT in claim_lattice mode` is NOT "the answer is correct" — it's "every pointer resolved to a valid evidence object whose source_role is allowed AND citation-coverage passed." The display label spells out the actual property so users don't read STRICT as full semantic entailment.
**Verifier strategies** run in sequence; first to find evidence classifies. Each is lexical (substring or token-coverage), never embeddings — soft signals stay out of the proof path.
| # | strategy | what triggers it |
|---|---|---|
@ -314,6 +327,9 @@ Four strategies run in sequence; first to find evidence classifies. Each is lexi
| 2 | `span` | bullet/sentence units substring-tested as fallback |
| 3 | `entity` | multi-word proper nouns tested with proximity-cluster gating |
| 4 | `paraphrase` | inside the span path: ≥85% token coverage on prose-shaped spans |
| 5 | `claim_lattice` | model emits `claim text. [E1,E2]` pointer-line OR `{"claims":[{"text":..., "evidence_ids":[...]}]}` JSON; verifier resolves pointers to runtime-built evidence objects & runs seven hard checks |
The `claim_lattice` path runs **seven deterministic hard checks**: parser succeeded, evidence_id resolves, source_role allowed, claim text non-empty, citation coverage threshold, pointer count cap, anchor-class warrant. Anchor-class warrant composes five lexical anchor classes — proper-noun, date, entity-list, count (with digit↔word equivalence), and why-cause — each gated on either question shape, claim content, or both. See `docs/concept-relations-design.md` and the whitepaper §13.9 for the architecture.
Trailing `(Source: https://...)` parentheticals the model appends to verbatim source sentences are stripped before substring testing, so verbatim-with-citation no longer flags HYBRID.
@ -396,6 +412,13 @@ make activity ACTIVITY_LIMIT=20
## Architecture (one screenful)
![Aborist module graph](docs/diagrams/aborist-modules.svg)
For per-module reference docs see [`docs/modules/`](docs/modules/index.md).
Full pipeline diagrams live in [`docs/diagrams/`](docs/diagrams/) — query,
ingest, verifier-ladder, plus the mesh sub-diagrams. Render any change
with `make docs`.
```
aborist/
├── merkle.py Python port of proxy.unturf.com/pkg/verified/merkle.go
@ -403,7 +426,8 @@ aborist/
│ IsLeft per sibling, self-duplicate odd elements).
├── store.py SQLite v9.8 schema (8-dim cache key, falsification
│ state, audit chain, surface/core kinds, hot/warm/
│ cold tier, document_http_meta, mesh_*).
│ cold tier, document_http_meta, concept_relations,
│ mesh_*).
├── ingest.py normalize → chunk → merkle → upsert. Bulk-batched.
├── document.py Document, Edge, Chunker (tok-512-v1 default).
├── source.py Source ABC: iter_documents() -> Iterator[Document].
@ -417,14 +441,36 @@ aborist/
│ └── crawler/ verbatim AsyncWebFetcher lift + ingest bridge.
├── distill/ Distiller ABC + first_sentence + tfidf + runner.
├── search/ FTS5 backend + SearchBackend ABC + AuditMode.
├── concepts/ corpus-derived synonym & rivalry layer.
│ ├── store.py append-only concept_relations CRUD + idempotent
│ │ INSERT OR IGNORE on UNIQUE re-derivation key.
│ ├── query.py cross-shard synonym_expand & rivalry_excluded;
│ │ per-process LRU keyed on shard mtime; manual /
│ │ derived index split with per-token degree caps.
│ ├── extract.py pluggable extractor registry; built-in
│ │ `link_reciprocity_synonym` reads existing edges.
│ └── seed.py one-shot migration of legacy frozensets to
│ manual_legacy rows (clique edges per group).
├── qa/
│ ├── client.py ChatClient + StubClient + OpenAICompatibleClient.
│ ├── keys.py 8-dim cache_key + question_hash (CJK-aware strip).
│ ├── verify.py 4-strategy lexical verifier (quote/span/entity/
│ │ paraphrase). Citation-strip + wikitext-strip.
│ ├── verify.py quote/span/entity/paraphrase + claim_lattice
│ │ verifier (seven deterministic hard checks incl.
│ │ anchor-class warrant). Citation-strip +
│ │ wikitext-strip.
│ ├── warrant.py five anchor classes — proper-noun, date,
│ │ entity-list, count (digit↔word equivalence),
│ │ cause — each gated on question shape and/or
│ │ claim content. Lexical only; in the proof path.
│ ├── evidence.py EvidenceObject + spotlight excerpt (density
│ │ rank picks the load-bearing slice).
│ ├── parse_claims.py pointer-line parser (`claim. [E1,E2]`).
│ ├── concepts.py backwards-compat shim — delegates to aborist.concepts.
│ ├── dag.py per-run Merkle-DAG (7-stage quote / 9-stage CTI).
│ ├── inspect.py sidecar diagnostic — read-only span classifier.
│ ├── runner.py ask(): single-doc Q&A + cache + verify.
│ └── query.py query(): multi-source RAG + concept overlay.
│ └── query.py query(): multi-source RAG + concept overlay
│ (FTS5 AND→OR-with-synonyms fallback).
├── mesh/ off-by-default federation: identity (Ed25519/X25519),
│ per-epoch roster, AEAD-wrapped epoch secrets, HTTP
│ gossip wire, per-peer chain-of-claims tracking.
@ -432,16 +478,19 @@ aborist/
└── cli.py argparse entrypoint (the make targets call into here).
```
For diagrams of how these modules wire together, see `docs/diagrams/` (rendered SVGs of the module graph, query pipeline, ingest pipeline, mesh wire, & verifier ladder). For per-module reference docs see `docs/modules/` — every top-level package has its own page.
Source papers (read first if confused):
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — canonical whitepaper (rst, builds the PDF). Section 13.8 covers the verifier in depth.
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — canonical whitepaper (rst, builds the PDF). §13.8 covers the verifier; §13.9 covers claim-lattice / CTI; §13.4.11 covers the corpus-derived concept layer.
- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV/canonical encoding, theorems T1–T5).
- `docs/concept-relations-design.md` — synonym & rivalry layer architecture + 1.6% storage tradeoff rationale.
- `docs/mesh.md` + `docs/mesh-deploy.md` — protocol contract + two-host runbook for federation.
## Tests
```
make test # 326+ tests, default suite (stdlib + pytest, no network)
make test # 641+ tests, default suite (stdlib + pytest, no network)
make test-crawler # opt-in: tests for the verbatim crawler lift
make bench # ETL throughput across configs (serial / shared-WAL / attached)
```