docs: trim README — remove sections now in Sphinx/docs (31% reduction)
549 → 379 lines. Removed sections that don't help onboarding: - OpenAI export placeholder (10 lines): unimplemented feature, misleads new users browsing source list - Marking and burning records (21 lines): operational detail, not first-run material - Mesh / federation full subcommand reference (27 lines): off-by-default feature; replaced with one-line note + link to docs/mesh.md - Architecture full source tree (76 lines): now generated from docstrings via Sphinx (make docs-api); replaced with one paragraph pointing to RTD - Sidecar diagnostics + metaphor cue dictionary (39 lines): too deep for README; lives in code docstrings + qa.inspect module Also fixed broken references to deleted docs (concept-relations-design.md, mesh-deploy.md, modules.md) — text rewritten to be self-contained or point to whitepaper. Onboarding flow preserved: title, quickstart, setup, basic query, verifier ladder summary, license.
This commit is contained in:
parent
8e9e993571
commit
306b7f3701
1 changed files with 8 additions and 178 deletions
186
README.md
186
README.md
|
|
@ -256,17 +256,6 @@ Conditional `If-None-Match` / `If-Modified-Since` HEAD requests classify each in
|
|||
|
||||
The crawler is a verbatim lift from `~/git/agents.ai.unturf.com/core/` (provenance documented in `aborist/sources/crawler/__init__.py`); aborist-side changes drop the chat-bot fetch triggers and skip `web_cache_manager.py` (aborist has its own content-addressed cache). Run `make test-crawler` for the lift's own tests.
|
||||
|
||||
## Data: OpenAI / ChatGPT export (planned)
|
||||
|
||||
Not implemented yet. The shape will be one new `Source` subclass at `aborist/sources/openai.py` plus a Makefile target. The OpenAI ChatGPT data export is a `.zip` containing `conversations.json` with the `mapping`/`messages` tree shape. Adding it follows the same pattern as `aborist/sources/grok.py` — see that file as the template.
|
||||
|
||||
```
|
||||
# (placeholder)
|
||||
make ingest-openai-attached # OPENAI_EXPORT=$HOME/Downloads/<chatgpt-export>
|
||||
```
|
||||
|
||||
When this lands, conversations from both Grok and OpenAI will sit in the same shard cluster; queries fan out across all of them.
|
||||
|
||||
## Asking the corpus
|
||||
|
||||
```
|
||||
|
|
@ -282,7 +271,7 @@ Default render is human-readable: question, audit-mode summary line, answer, sou
|
|||
The query path:
|
||||
|
||||
1. **Search** — FTS5 (body) + SQL `LIKE` (title) + `JOIN` over derivations (TF-IDF core keywords) across every shard. Three accept paths to the relevance filter.
|
||||
2. **Concept overlay** — per-shard `concept_relations` SQLite table (corpus-derived, not hand-curated). Synonyms widen retrieval; rivalries narrow it unless the query uses comparative phrasing ("compare X vs Y"). Built-in extractor `link_reciprocity_synonym` reads the existing `edges` table for reciprocal A↔B link pairs and emits synonym edges between their title-tokens — works for Wikipedia (See-also bidirectional), HTML site internal links, or any corpus with bidirectional document links. See `docs/concept-relations-design.md` for the storage tradeoff (1.6% tax measured on 6 GB Wikipedia).
|
||||
2. **Concept overlay** — per-shard `concept_relations` SQLite table (corpus-derived, not hand-curated). Synonyms widen retrieval; rivalries narrow it unless the query uses comparative phrasing ("compare X vs Y"). Built-in extractor `link_reciprocity_synonym` reads the existing `edges` table for reciprocal A↔B link pairs and emits synonym edges between their title-tokens. ~1.6% storage tax measured on 6 GB Wikipedia.
|
||||
3. **Context assembly** — top-K sources concatenated up to a 60 KB budget. Wikitext is stripped to plain prose via `aborist.wikitext.to_base()` (the corpus stores raw `[[wikilinks]]` so the link graph is recoverable on demand; the LLM and verifier both see clean prose).
|
||||
4. **LLM** — Hermes-3 with strict attribution rules in the system prompt + a user-turn grounding reminder.
|
||||
5. **Verifier** — every claim runs through a layered lexical check; the result rolls up into the v9.8 trichotomy (`audit_mode` ∈ STRICT / HYBRID / UNGROUNDED) at the schema layer AND a four-rung display ladder at render time (POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → UNGROUNDED). See below.
|
||||
|
|
@ -329,7 +318,7 @@ HYBRID gets a `-PARTIAL` suffix on whichever rung applies. The point of the disp
|
|||
| 4 | `paraphrase` | inside the span path: ≥85% token coverage on prose-shaped spans |
|
||||
| 5 | `claim_lattice` | model emits `claim text. [E1,E2]` pointer-line OR `{"claims":[{"text":..., "evidence_ids":[...]}]}` JSON; verifier resolves pointers to runtime-built evidence objects & runs seven hard checks |
|
||||
|
||||
The `claim_lattice` path runs **seven deterministic hard checks**: parser succeeded, evidence_id resolves, source_role allowed, claim text non-empty, citation coverage threshold, pointer count cap, anchor-class warrant. Anchor-class warrant composes five lexical anchor classes — proper-noun, date, entity-list, count (with digit↔word equivalence), and why-cause — each gated on either question shape, claim content, or both. See `docs/concept-relations-design.md` and the whitepaper §13.9 for the architecture.
|
||||
The `claim_lattice` path runs **seven deterministic hard checks**: parser succeeded, evidence_id resolves, source_role allowed, claim text non-empty, citation coverage threshold, pointer count cap, anchor-class warrant. Anchor-class warrant composes five lexical anchor classes — proper-noun, date, entity-list, count (with digit↔word equivalence), and why-cause — each gated on either question shape, claim content, or both. See whitepaper §13.9 for the architecture.
|
||||
|
||||
Trailing `(Source: https://...)` parentheticals the model appends to verbatim source sentences are stripped before substring testing, so verbatim-with-citation no longer flags HYBRID.
|
||||
|
||||
|
|
@ -343,60 +332,9 @@ make inspect KEY=<cache_key>
|
|||
|
||||
Read-only **sidecar** diagnostic: pulls source chunks, classifies each unverified span as `verbatim_in_base` / `trailing_artifact` / `paraphrase` / `partial_paraphrase` / `no_overlap`. Writes nothing — sidecars never enter the v9.8 hard chain (that invariant is what keeps `audit_mode` a binary classification rather than a soft score).
|
||||
|
||||
## Marking and burning records
|
||||
|
||||
Two ways to remove a cached answer, depending on whether it has children:
|
||||
|
||||
```
|
||||
make falsify KEY=<cache_key> REASON="why it was wrong"
|
||||
```
|
||||
|
||||
The record stays in the database; `falsification_state` flips to `failed` and a `falsifications` row + `falsify` audit event record the act. Future lookups skip records whose `falsification_state != 'live'`.
|
||||
|
||||
```
|
||||
make burn KEY=<cache_key> REASON="..." # providence leaves
|
||||
make burn KIND=document ROOT=<hex> REASON="..." # surface documents
|
||||
make burn KIND=core ROOT=<hex> REASON="..." # distilled cores
|
||||
```
|
||||
|
||||
`burn` actually deletes the row. Refuses if the leaf has children (falsifications referencing the cache_key, derivations using the document, or other records pointing at it) — pass `FORCE=1` to override. Always writes an audit event. Use during early/scratch corpus building; falsify is the audit-preserving alternative once downstream consumers exist.
|
||||
|
||||
```
|
||||
make chain-check-shards # 0 chain breaks across every shard's audit_events
|
||||
```
|
||||
|
||||
## Mesh / federation (off by default)
|
||||
|
||||
Two machines that ingest the same dump compute *bit-identical* `document_root`s — that's the v9.8 admissibility property. The mesh layer is the wire-and-trust scaffolding that lets peers gossip those identities (plus derivations, falsifications, cross-witnesses) and dedup-by-content across instances.
|
||||
|
||||
It ships **off by default**. No code path touches the network unless the `mesh.enabled` flag is set. Initialization flow:
|
||||
|
||||
```
|
||||
aborist mesh init --group myteam # mint Ed25519 + X25519 keys; create epoch 0
|
||||
aborist mesh status # always-safe inspection; shows enabled/false until you flip it
|
||||
aborist mesh enable # flip the gating flag on
|
||||
```
|
||||
|
||||
Membership is per-epoch. Adding a member, kicking a member, or rotating the secret each bumps the epoch and writes an audit event:
|
||||
|
||||
```
|
||||
aborist mesh add --member-id bob --sign-pub <hex> --dh-pub <hex>
|
||||
aborist mesh kick --member-id bob --reason "..." # admin-only; bumps epoch, omits bob from new envelope
|
||||
aborist mesh rotate --reason "..." # refresh secret, same roster
|
||||
aborist mesh members # list current epoch's roster
|
||||
```
|
||||
|
||||
The kicked member's prior signatures stay verifiable forever (their roster row at older epochs is preserved on disk). They have no entry in the new epoch's secret envelope, so any AEAD-protected gossip from epoch+1 onward is opaque to them — that is the eviction guarantee.
|
||||
|
||||
Once two peers have enrolled each other, the HTTP gossip wire is live:
|
||||
|
||||
```
|
||||
aborist mesh serve --host 0.0.0.0 --port 8400 # blocks until SIGINT
|
||||
aborist mesh sync --peer http://other.example.com:8400 # announce local document_roots
|
||||
aborist mesh pull --peer http://... --root <hex> # pull a body, verify Merkle
|
||||
```
|
||||
|
||||
Every envelope is Ed25519-signed; bodies can opt into AEAD encryption under the per-epoch shared secret (`encrypt=True` on the client API). Receivers track each sender's chain-of-claims and reject envelopes whose `prev_event_hash` doesn't extend the last known event for that peer — fork detection at the wire layer. Two-host setup runbook: `docs/mesh-deploy.md`. Protocol contract + diagrams: `docs/mesh.md`.
|
||||
Optional gossip layer for peer-to-peer corpus sync. Default off — no code path touches the network unless `mesh.enabled` is set. See [`docs/mesh.md`](docs/mesh.md) for protocol contract and the `aborist mesh` CLI subcommands.
|
||||
|
||||
## Inspecting
|
||||
|
||||
|
|
@ -410,82 +348,14 @@ make activity ACTIVITY_LIMIT=20
|
|||
|
||||
`activity` is JSON; pipe through `jq` to drill in.
|
||||
|
||||
## Architecture (one screenful)
|
||||
## Architecture
|
||||
|
||||
|
||||
|
||||
For per-module reference docs see [`docs/modules.md`](docs/modules.md).
|
||||
Full pipeline diagrams live in [`docs/diagrams/`](docs/diagrams/) — query,
|
||||
ingest, verifier-ladder, plus the mesh sub-diagrams. Render any change
|
||||
with `make docs`.
|
||||
|
||||
```
|
||||
aborist/
|
||||
├── merkle.py Python port of proxy.unturf.com/pkg/verified/merkle.go
|
||||
│ (non-commutative HashCombine 0x03 prefix, explicit
|
||||
│ IsLeft per sibling, self-duplicate odd elements).
|
||||
├── store.py SQLite v9.8 schema (8-dim cache key, falsification
|
||||
│ state, audit chain, surface/core kinds, hot/warm/
|
||||
│ cold tier, document_http_meta, concept_relations,
|
||||
│ mesh_*).
|
||||
├── ingest.py normalize → chunk → merkle → upsert. Bulk-batched.
|
||||
├── document.py Document, Edge, Chunker (tok-512-v1 default).
|
||||
├── source.py Source ABC: iter_documents() -> Iterator[Document].
|
||||
├── wikitext.py to_base(): wikitext → plain prose, BASE_VERSION-pinned.
|
||||
├── sources/
|
||||
│ ├── wikipedia.py cur + old MediaWiki SQL dumps (bz2-streamed).
|
||||
│ ├── wikipedia_xml.py Phase IV XML dumps (iterparse, page + history).
|
||||
│ ├── html_page.py URL list + selectolax + httpx (robots-aware).
|
||||
│ ├── grok.py xAI data export (conversations + media prompts).
|
||||
│ ├── vcs.py git + Mercurial repos (HEAD walk, supersedes chain).
|
||||
│ └── crawler/ verbatim AsyncWebFetcher lift + ingest bridge.
|
||||
├── distill/ Distiller ABC + first_sentence + tfidf + runner.
|
||||
├── search/ FTS5 backend + SearchBackend ABC + AuditMode.
|
||||
├── concepts/ corpus-derived synonym & rivalry layer.
|
||||
│ ├── store.py append-only concept_relations CRUD + idempotent
|
||||
│ │ INSERT OR IGNORE on UNIQUE re-derivation key.
|
||||
│ ├── query.py cross-shard synonym_expand & rivalry_excluded;
|
||||
│ │ per-process LRU keyed on shard mtime; manual /
|
||||
│ │ derived index split with per-token degree caps.
|
||||
│ ├── extract.py pluggable extractor registry; built-in
|
||||
│ │ `link_reciprocity_synonym` reads existing edges.
|
||||
│ └── seed.py one-shot migration of legacy frozensets to
|
||||
│ manual_legacy rows (clique edges per group).
|
||||
├── qa/
|
||||
│ ├── client.py ChatClient + StubClient + OpenAICompatibleClient.
|
||||
│ ├── keys.py 8-dim cache_key + question_hash (CJK-aware strip).
|
||||
│ ├── verify.py quote/span/entity/paraphrase + claim_lattice
|
||||
│ │ verifier (seven deterministic hard checks incl.
|
||||
│ │ anchor-class warrant). Citation-strip +
|
||||
│ │ wikitext-strip.
|
||||
│ ├── warrant.py five anchor classes — proper-noun, date,
|
||||
│ │ entity-list, count (digit↔word equivalence),
|
||||
│ │ cause — each gated on question shape and/or
|
||||
│ │ claim content. Lexical only; in the proof path.
|
||||
│ ├── evidence.py EvidenceObject + spotlight excerpt (density
|
||||
│ │ rank picks the load-bearing slice).
|
||||
│ ├── parse_claims.py pointer-line parser (`claim. [E1,E2]`).
|
||||
│ ├── concepts.py backwards-compat shim — delegates to aborist.concepts.
|
||||
│ ├── dag.py per-run Merkle-DAG (7-stage quote / 9-stage CTI).
|
||||
│ ├── inspect.py sidecar diagnostic — read-only span classifier.
|
||||
│ ├── runner.py ask(): single-doc Q&A + cache + verify.
|
||||
│ └── query.py query(): multi-source RAG + concept overlay
|
||||
│ (FTS5 AND→OR-with-synonyms fallback).
|
||||
├── mesh/ off-by-default federation: identity (Ed25519/X25519),
|
||||
│ per-epoch roster, AEAD-wrapped epoch secrets, HTTP
|
||||
│ gossip wire, per-peer chain-of-claims tracking.
|
||||
├── evict.py hot ↔ cold tier transitions; rehydrate via source.
|
||||
└── cli.py argparse entrypoint (the make targets call into here).
|
||||
```
|
||||
|
||||
For diagrams of how these modules wire together, see `docs/diagrams/` (rendered SVGs of the module graph, query pipeline, ingest pipeline, mesh wire, & verifier ladder). For per-module reference docs see `docs/modules.md` — every top-level package has its own page.
|
||||
|
||||
Source papers (read first if confused):
|
||||
|
||||
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — canonical whitepaper (rst, builds the PDF). §6 Aborist architecture, §7 layered verifier, §8 claim-lattice / CTI, §9 four-rung ladder, §13 decisions & constraints.
|
||||
- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV/canonical encoding, theorems T1–T5).
|
||||
- `docs/concept-relations-design.md` — synonym & rivalry layer architecture + 1.6% storage tradeoff rationale.
|
||||
- `docs/mesh.md` + `docs/mesh-deploy.md` — protocol contract + two-host runbook for federation.
|
||||
Generated API reference (every module, class, and function from docstrings):
|
||||
build with `make docs-api` (output at `docs/_source/_build/html/`) or browse
|
||||
the published Read the Docs site. Pipeline diagrams live in
|
||||
[`docs/diagrams/`](docs/diagrams/); render with `make docs`.
|
||||
|
||||
## Tests & bench
|
||||
|
||||
|
|
@ -502,46 +372,6 @@ The default suite never hits the network. The crawler suite is gated behind `mak
|
|||
|
||||
`make bench-qa` writes JSONL + markdown into `bench/qa_results/<utc-stamp>.{jsonl,md}` (gitignored); design-log entries live in `docs/qa-modes-bench-<date>.md`. The bench is stop/start-able via `--resume <jsonl-path>` (same `--seed` required for shuffled-task-order alignment). `BENCH_QA_CONCURRENCY=N` Makefile variable overrides the default `4`. See `docs/bench-maxing.md` for the full speed playbook.
|
||||
|
||||
## Sidecar diagnostics
|
||||
|
||||
The verifier stays binary; soft signals proliferate as **sidecars** — read-only diagnostic functions that pull the same source chunks the verifier saw, classify spans / detect topic shifts / surface metaphor framing, and never write to `providence_cache` or extend the audit chain. Three live today:
|
||||
|
||||
| sidecar | flags | typical use |
|
||||
|---------|-------|-------------|
|
||||
| `diagnose_deflection(question, answer)` | model deflected — answer omits the question's subject anchor | adversarial-premise questions where the model answers an adjacent grounded question |
|
||||
| `diagnose_title_relevance(claim, cited_titles)` | claim's stems share zero overlap with cited source titles | retrieval-driven hallucinations where a chunk's source is structurally unrelated to the claim |
|
||||
| `diagnose_metaphor_deflection(question, answer)` | question carried strong poetic framing the answer ignored | metaphorical questions answered with literal taxonomic / definitional facts |
|
||||
|
||||
Run the per-record diagnostic via `aborist inspect --cache-key <hex>` (CLI: `make inspect KEY=...`). Sidecars run on demand, observe state, report — they participate in no proof path.
|
||||
|
||||
### Metaphor-deflection cue dictionary
|
||||
|
||||
`diagnose_metaphor_deflection` derives its poetic-cue vocabulary from a configurable union of wordlists. The default load order:
|
||||
|
||||
1. `/usr/share/dict/words` (OS default)
|
||||
2. `/usr/share/dict/american-english` (Debian split, if present)
|
||||
3. `/usr/share/dict/british-english` (Debian split, if present)
|
||||
4. Paths in `ABORIST_METAPHOR_DICTS` (colon-separated env var)
|
||||
5. Paths registered via `aborist.qa.inspect.register_metaphor_dictionary(path)`
|
||||
|
||||
A cue is a token that survives one of four morphological tests against the union — closed-class preposition (`amidst, despite, beneath, ...`), `-ly` adverb whose stem is in the union, `-ing` participle whose verb stem is in the union (with consonant-de-doubling for `running → run`), or `-est` superlative whose adjective stem is in the union (with i↔y swap for `rockiest → rocky`). The rule is *derived*, not hand-curated — same Phase-2 lesson the `concept_relations` synonym layer learned. Adding `butterfly` to a hand exception list scales poorly; trusting `butterf` not in `/usr/share/dict/words` to filter `butterfly` scales by definition.
|
||||
|
||||
**A forest with its own vocabulary** — domain-specific jargon (aviation, medical, legal, dynastic) registers its own wordlist once and the suffix tests pick up domain-specific adverbs / participles automatically:
|
||||
|
||||
```sh
|
||||
# Per-process: env var
|
||||
export ABORIST_METAPHOR_DICTS=/path/to/aviation-terms.txt:/path/to/medical-terms.txt
|
||||
.venv/bin/aborist inspect --cache-key <hex>
|
||||
```
|
||||
|
||||
```python
|
||||
# Programmatic: register once, applies to all subsequent diagnoses
|
||||
from aborist.qa.inspect import register_metaphor_dictionary
|
||||
register_metaphor_dictionary("/path/to/aviation-terms.txt")
|
||||
```
|
||||
|
||||
Each supplemental dict is one word per line; blank lines and case-mixed entries are tolerated. The cue suffix tests then resolve domain stems (`aerodynamic → aerodynamically`, `widgetize → widgetizing`) without code changes.
|
||||
|
||||
## License
|
||||
|
||||
License: AGPL-3.0-only · This algorithm, its implementation, & all associated code carry the GNU Affero General Public License v3.0 (only). You may use, modify, & distribute under those terms. No proprietary relicensing exists.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue