docs: README label refresh + per-module reference + Graphviz diagrams
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).
(1) README — label refresh:
- Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
- Verifier section spells out both layers (schema trichotomy +
display ladder), the seven hard checks of claim_lattice, and
the five anchor classes of warrant.
- Architecture tree updated: concepts/ package added, qa/
sub-modules expanded (warrant, evidence, parse_claims, dag),
verify.py described as quote/span/entity/paraphrase + claim_lattice.
- Concept overlay description updated for corpus-derived layer
(concept_relations table, link_reciprocity extractor, 1.6%
tax cite).
- Test count: 326+ → 641+.
(2) docs/diagrams/ — Graphviz dot sources:
- aborist-modules.dot — top-level package graph (substrate /
storage / sources / retrieval / qa / mesh / cli)
- query-pipeline.dot — question → cache → retrieval → LLM →
verify → render → cache write, with phase budgets
- ingest-pipeline.dot — source doc → canonicalize → chunk →
Merkle → upsert (+ optional distill)
- verifier-ladder.dot — (audit_mode, violations) → display rung
decision tree
Existing mesh-*.dot kept as-is. Makefile `make docs` target
extended to also emit .svg alongside the existing .png so the
diagrams render in markdown viewers.
(3) docs/modules/ — per-module reference pages:
- index.md (links to every diagram + every module page)
- merkle.md, document.md, store.md, ingest.md, evict.md,
sources.md, search.md, concepts.md, qa.md, distill.md,
wikitext.md
Each page is a one-screenful concise reference: what the
module is for, public API, key invariants, embedded diagrams
where useful, link to source. Mesh stays at the existing
docs/mesh.md + docs/mesh-deploy.md (already comprehensive).
Tests: 641 passed (no code change).
6
Makefile
|
|
@ -421,11 +421,15 @@ recrawl-check: bootstrap-crawler ## conditional HEAD per ingested doc [DOMAIN=x.
|
|||
|
||||
DOT_SRCS := $(wildcard docs/diagrams/*.dot)
|
||||
DOT_PNGS := $(DOT_SRCS:.dot=.png)
|
||||
DOT_SVGS := $(DOT_SRCS:.dot=.svg)
|
||||
|
||||
docs/diagrams/%.png: docs/diagrams/%.dot
|
||||
dot -Tpng $< -o $@
|
||||
|
||||
docs: $(DOT_PNGS) ## render docs/diagrams/*.dot -> .png via graphviz
|
||||
docs/diagrams/%.svg: docs/diagrams/%.dot
|
||||
dot -Tsvg $< -o $@
|
||||
|
||||
docs: $(DOT_PNGS) $(DOT_SVGS) ## render docs/diagrams/*.dot -> .png + .svg via graphviz
|
||||
|
||||
# Reproducible micro-benchmark over a fixed slice of cur. Lets you compare
|
||||
# ETL throughput across configs and catches regressions on optimization
|
||||
|
|
|
|||
73
README.md
|
|
@ -19,7 +19,7 @@ make distill-shards-tfidf-parallel # core → keyword sets for retrieval
|
|||
make query Q="What is anarcho-capitalism?"
|
||||
```
|
||||
|
||||
A Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer plus a verifier audit_mode (STRICT / HYBRID / UNGROUNDED). Repeat the same question and a cache hit replays in ~100 ms.
|
||||
A Hermes-3 inference runs against the local corpus, picks 4–8 source articles by Merkle root, and returns an answer plus a verifier label that names what the lexical verifier could confirm. The current four-rung ladder for claim-lattice modes is `POINTER-LINKED` → `ANCHOR-WARRANTED` → `EVIDENCE-WARRANTED` → `UNGROUNDED` (with `-PARTIAL` suffix on HYBRID). Repeat the same question and a cache hit replays in ~100 ms.
|
||||
|
||||
### B. Crawl any live website and query it
|
||||
|
||||
|
|
@ -282,10 +282,10 @@ Default render is human-readable: question, audit-mode summary line, answer, sou
|
|||
The query path:
|
||||
|
||||
1. **Search** — FTS5 (body) + SQL `LIKE` (title) + `JOIN` over derivations (TF-IDF core keywords) across every shard. Three accept paths to the relevance filter.
|
||||
2. **Concept overlay** — synonym groups (`Athlon` ↔ `AMD`) widen retrieval; rivalry pairs (AMD ↔ Intel) narrow it unless the query has comparative phrasing ("compare X vs Y").
|
||||
2. **Concept overlay** — per-shard `concept_relations` SQLite table (corpus-derived, not hand-curated). Synonyms widen retrieval; rivalries narrow it unless the query uses comparative phrasing ("compare X vs Y"). Built-in extractor `link_reciprocity_synonym` reads the existing `edges` table for reciprocal A↔B link pairs and emits synonym edges between their title-tokens — works for Wikipedia (See-also bidirectional), HTML site internal links, or any corpus with bidirectional document links. See `docs/concept-relations-design.md` for the storage tradeoff (1.6% tax measured on 6 GB Wikipedia).
|
||||
3. **Context assembly** — top-K sources concatenated up to a 60 KB budget. Wikitext is stripped to plain prose via `aborist.wikitext.to_base()` (the corpus stores raw `[[wikilinks]]` so the link graph is recoverable on demand; the LLM and verifier both see clean prose).
|
||||
4. **LLM** — Hermes-3 with strict attribution rules in the system prompt + a user-turn grounding reminder.
|
||||
5. **Verifier** — every claim is classified into the v9.8 trichotomy STRICT / HYBRID / UNGROUNDED via a layered lexical check (see below).
|
||||
5. **Verifier** — every claim runs through a layered lexical check; the result rolls up into the v9.8 trichotomy (`audit_mode` ∈ STRICT / HYBRID / UNGROUNDED) at the schema layer AND a four-rung display ladder at render time (POINTER-LINKED → ANCHOR-WARRANTED → EVIDENCE-WARRANTED → UNGROUNDED). See below.
|
||||
6. **Cache** — the v9.8 8-dim cache_key (`source_root | question_hash | model_profile | conversation | governance_policy | schema | canonicalization | chunking`) keys the answer in `qa.db`. Cache hits replay in ~100 ms. Per-phase timings in every result.
|
||||
|
||||
LLM endpoint defaults to `https://hermes.ai.unturf.com/v1` (Hermes-3 Llama-3.1-8B, 82K context, no auth). Override:
|
||||
|
|
@ -296,9 +296,11 @@ export ABORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
|
|||
export ABORIST_LLM_API_KEY="..."
|
||||
```
|
||||
|
||||
## Verifying answers (audit modes)
|
||||
## Verifying answers (audit modes & label ladder)
|
||||
|
||||
Every answer carries an `audit_mode` set by `aborist/qa/verify.py`:
|
||||
Two layers of labels stack on every answer:
|
||||
|
||||
**Schema layer — v9.8 trichotomy** (`audit_mode` column, persisted, drives cache lookups & audit chain):
|
||||
|
||||
| mode | meaning |
|
||||
|---|---|
|
||||
|
|
@ -306,7 +308,18 @@ Every answer carries an `audit_mode` set by `aborist/qa/verify.py`:
|
|||
| **HYBRID** | some claims source-grounded, some emerged from training |
|
||||
| **UNGROUNDED** | no evidence, or none verifies — purely emergent |
|
||||
|
||||
Four strategies run in sequence; first to find evidence classifies. Each is lexical (substring or token-coverage), never embeddings — soft signals stay out of the proof path.
|
||||
**Display layer — four-rung ladder** (claim-lattice modes only; renderer-only transformation, schema unchanged):
|
||||
|
||||
| rung | what's actually proved |
|
||||
|---|---|
|
||||
| **EVIDENCE-WARRANTED** | pointer verified + warrant ran & passed + no soft demotes |
|
||||
| **ANCHOR-WARRANTED** | pointer-linked + warrant passed; soft-demote violations present |
|
||||
| **POINTER-LINKED** | pointer/source/chunk verified, but warrant either didn't apply or failed for some claim |
|
||||
| **UNGROUNDED** | no verified pairs |
|
||||
|
||||
HYBRID gets a `-PARTIAL` suffix on whichever rung applies. The point of the display ladder: `STRICT in claim_lattice mode` is NOT "the answer is correct" — it's "every pointer resolved to a valid evidence object whose source_role is allowed AND citation-coverage passed." The display label spells out the actual property so users don't read STRICT as full semantic entailment.
|
||||
|
||||
**Verifier strategies** run in sequence; first to find evidence classifies. Each is lexical (substring or token-coverage), never embeddings — soft signals stay out of the proof path.
|
||||
|
||||
| # | strategy | what triggers it |
|
||||
|---|---|---|
|
||||
|
|
@ -314,6 +327,9 @@ Four strategies run in sequence; first to find evidence classifies. Each is lexi
|
|||
| 2 | `span` | bullet/sentence units substring-tested as fallback |
|
||||
| 3 | `entity` | multi-word proper nouns tested with proximity-cluster gating |
|
||||
| 4 | `paraphrase` | inside the span path: ≥85% token coverage on prose-shaped spans |
|
||||
| 5 | `claim_lattice` | model emits `claim text. [E1,E2]` pointer-line OR `{"claims":[{"text":..., "evidence_ids":[...]}]}` JSON; verifier resolves pointers to runtime-built evidence objects & runs seven hard checks |
|
||||
|
||||
The `claim_lattice` path runs **seven deterministic hard checks**: parser succeeded, evidence_id resolves, source_role allowed, claim text non-empty, citation coverage threshold, pointer count cap, anchor-class warrant. Anchor-class warrant composes five lexical anchor classes — proper-noun, date, entity-list, count (with digit↔word equivalence), and why-cause — each gated on either question shape, claim content, or both. See `docs/concept-relations-design.md` and the whitepaper §13.9 for the architecture.
|
||||
|
||||
Trailing `(Source: https://...)` parentheticals the model appends to verbatim source sentences are stripped before substring testing, so verbatim-with-citation no longer flags HYBRID.
|
||||
|
||||
|
|
@ -396,6 +412,13 @@ make activity ACTIVITY_LIMIT=20
|
|||
|
||||
## Architecture (one screenful)
|
||||
|
||||

|
||||
|
||||
For per-module reference docs see [`docs/modules/`](docs/modules/index.md).
|
||||
Full pipeline diagrams live in [`docs/diagrams/`](docs/diagrams/) — query,
|
||||
ingest, verifier-ladder, plus the mesh sub-diagrams. Render any change
|
||||
with `make docs`.
|
||||
|
||||
```
|
||||
aborist/
|
||||
├── merkle.py Python port of proxy.unturf.com/pkg/verified/merkle.go
|
||||
|
|
@ -403,7 +426,8 @@ aborist/
|
|||
│ IsLeft per sibling, self-duplicate odd elements).
|
||||
├── store.py SQLite v9.8 schema (8-dim cache key, falsification
|
||||
│ state, audit chain, surface/core kinds, hot/warm/
|
||||
│ cold tier, document_http_meta, mesh_*).
|
||||
│ cold tier, document_http_meta, concept_relations,
|
||||
│ mesh_*).
|
||||
├── ingest.py normalize → chunk → merkle → upsert. Bulk-batched.
|
||||
├── document.py Document, Edge, Chunker (tok-512-v1 default).
|
||||
├── source.py Source ABC: iter_documents() -> Iterator[Document].
|
||||
|
|
@ -417,14 +441,36 @@ aborist/
|
|||
│ └── crawler/ verbatim AsyncWebFetcher lift + ingest bridge.
|
||||
├── distill/ Distiller ABC + first_sentence + tfidf + runner.
|
||||
├── search/ FTS5 backend + SearchBackend ABC + AuditMode.
|
||||
├── concepts/ corpus-derived synonym & rivalry layer.
|
||||
│ ├── store.py append-only concept_relations CRUD + idempotent
|
||||
│ │ INSERT OR IGNORE on UNIQUE re-derivation key.
|
||||
│ ├── query.py cross-shard synonym_expand & rivalry_excluded;
|
||||
│ │ per-process LRU keyed on shard mtime; manual /
|
||||
│ │ derived index split with per-token degree caps.
|
||||
│ ├── extract.py pluggable extractor registry; built-in
|
||||
│ │ `link_reciprocity_synonym` reads existing edges.
|
||||
│ └── seed.py one-shot migration of legacy frozensets to
|
||||
│ manual_legacy rows (clique edges per group).
|
||||
├── qa/
|
||||
│ ├── client.py ChatClient + StubClient + OpenAICompatibleClient.
|
||||
│ ├── keys.py 8-dim cache_key + question_hash (CJK-aware strip).
|
||||
│ ├── verify.py 4-strategy lexical verifier (quote/span/entity/
|
||||
│ │ paraphrase). Citation-strip + wikitext-strip.
|
||||
│ ├── verify.py quote/span/entity/paraphrase + claim_lattice
|
||||
│ │ verifier (seven deterministic hard checks incl.
|
||||
│ │ anchor-class warrant). Citation-strip +
|
||||
│ │ wikitext-strip.
|
||||
│ ├── warrant.py five anchor classes — proper-noun, date,
|
||||
│ │ entity-list, count (digit↔word equivalence),
|
||||
│ │ cause — each gated on question shape and/or
|
||||
│ │ claim content. Lexical only; in the proof path.
|
||||
│ ├── evidence.py EvidenceObject + spotlight excerpt (density
|
||||
│ │ rank picks the load-bearing slice).
|
||||
│ ├── parse_claims.py pointer-line parser (`claim. [E1,E2]`).
|
||||
│ ├── concepts.py backwards-compat shim — delegates to aborist.concepts.
|
||||
│ ├── dag.py per-run Merkle-DAG (7-stage quote / 9-stage CTI).
|
||||
│ ├── inspect.py sidecar diagnostic — read-only span classifier.
|
||||
│ ├── runner.py ask(): single-doc Q&A + cache + verify.
|
||||
│ └── query.py query(): multi-source RAG + concept overlay.
|
||||
│ └── query.py query(): multi-source RAG + concept overlay
|
||||
│ (FTS5 AND→OR-with-synonyms fallback).
|
||||
├── mesh/ off-by-default federation: identity (Ed25519/X25519),
|
||||
│ per-epoch roster, AEAD-wrapped epoch secrets, HTTP
|
||||
│ gossip wire, per-peer chain-of-claims tracking.
|
||||
|
|
@ -432,16 +478,19 @@ aborist/
|
|||
└── cli.py argparse entrypoint (the make targets call into here).
|
||||
```
|
||||
|
||||
For diagrams of how these modules wire together, see `docs/diagrams/` (rendered SVGs of the module graph, query pipeline, ingest pipeline, mesh wire, & verifier ladder). For per-module reference docs see `docs/modules/` — every top-level package has its own page.
|
||||
|
||||
Source papers (read first if confused):
|
||||
|
||||
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — canonical whitepaper (rst, builds the PDF). Section 13.8 covers the verifier in depth.
|
||||
- `~/git/unfirehose-nextjs-logger/whitepaper/merkle-providence-reverse-rag-whitepaper.rst` — canonical whitepaper (rst, builds the PDF). §13.8 covers the verifier; §13.9 covers claim-lattice / CTI; §13.4.11 covers the corpus-derived concept layer.
|
||||
- `~/Downloads/merkle-agi-dag_v7.txt` — formal substrate (TLV/canonical encoding, theorems T1–T5).
|
||||
- `docs/concept-relations-design.md` — synonym & rivalry layer architecture + 1.6% storage tradeoff rationale.
|
||||
- `docs/mesh.md` + `docs/mesh-deploy.md` — protocol contract + two-host runbook for federation.
|
||||
|
||||
## Tests
|
||||
|
||||
```
|
||||
make test # 326+ tests, default suite (stdlib + pytest, no network)
|
||||
make test # 641+ tests, default suite (stdlib + pytest, no network)
|
||||
make test-crawler # opt-in: tests for the verbatim crawler lift
|
||||
make bench # ETL throughput across configs (serial / shared-WAL / attached)
|
||||
```
|
||||
|
|
|
|||
127
docs/diagrams/aborist-modules.dot
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
// Aborist module graph — top-level packages & their dependencies.
|
||||
//
|
||||
// Arrows point in the import direction (A → B means A imports from B).
|
||||
// Cluster boxes group modules with shared concerns (storage, retrieval,
|
||||
// federation, etc.).
|
||||
//
|
||||
// Render: dot -Tsvg aborist-modules.dot -o aborist-modules.svg
|
||||
|
||||
digraph aborist_modules {
|
||||
rankdir=LR
|
||||
node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10]
|
||||
edge [fontname="Helvetica", fontsize=9]
|
||||
bgcolor="white"
|
||||
|
||||
// ---- Substrate (pure crypto / serialization, no SQL) ----
|
||||
subgraph cluster_substrate {
|
||||
label="substrate"
|
||||
style="rounded,dashed"
|
||||
color="#999999"
|
||||
merkle [label="merkle.py\n(Python port of Go merkle.go)", fillcolor="#fff7e6"]
|
||||
document [label="document.py\nDocument · Edge · Chunker", fillcolor="#fff7e6"]
|
||||
wikitext [label="wikitext.py\nto_base() prose-strip", fillcolor="#fff7e6"]
|
||||
}
|
||||
|
||||
// ---- Storage layer ----
|
||||
subgraph cluster_storage {
|
||||
label="storage"
|
||||
style="rounded,dashed"
|
||||
color="#999999"
|
||||
store [label="store.py\nv9.8 SQLite schema +\naudit chain helpers", fillcolor="#e6f0ff"]
|
||||
ingest [label="ingest.py\nnormalize → chunk → merkle → upsert", fillcolor="#e6f0ff"]
|
||||
evict [label="evict.py\nhot ↔ cold tier", fillcolor="#e6f0ff"]
|
||||
}
|
||||
|
||||
// ---- Sources (corpus producers) ----
|
||||
subgraph cluster_sources {
|
||||
label="sources/"
|
||||
style="rounded,dashed"
|
||||
color="#999999"
|
||||
wikipedia [label="wikipedia.py\ncur + old SQL dumps", fillcolor="#e7ffe7"]
|
||||
wiki_xml [label="wikipedia_xml.py\nphase IV iterparse", fillcolor="#e7ffe7"]
|
||||
html_page [label="html_page.py\nselectolax + httpx", fillcolor="#e7ffe7"]
|
||||
crawler [label="crawler/\nasync HTML BFS", fillcolor="#e7ffe7"]
|
||||
grok [label="grok.py\nxAI export", fillcolor="#e7ffe7"]
|
||||
vcs [label="vcs.py\ngit + Mercurial", fillcolor="#e7ffe7"]
|
||||
}
|
||||
|
||||
// ---- Retrieval & verifier ----
|
||||
subgraph cluster_retrieval {
|
||||
label="retrieval & verifier"
|
||||
style="rounded,dashed"
|
||||
color="#999999"
|
||||
search [label="search/\nFTS5 backend + ABC", fillcolor="#ffe6f0"]
|
||||
concepts [label="concepts/\nsynonym & rivalry overlay\n(per-shard concept_relations)", fillcolor="#ffe6f0"]
|
||||
qa_query [label="qa/query.py\nmulti-source RAG", fillcolor="#ffe6f0"]
|
||||
qa_verify [label="qa/verify.py\nquote/span/entity/paraphrase\n+ claim_lattice (7 hard checks)", fillcolor="#ffe6f0"]
|
||||
qa_warrant[label="qa/warrant.py\n5 anchor classes\n(proper-noun · date · count\n · entity-list · cause)", fillcolor="#ffe6f0"]
|
||||
qa_evidence[label="qa/evidence.py\nEvidenceObject + spotlight", fillcolor="#ffe6f0"]
|
||||
qa_runner [label="qa/runner.py\nask(): single-doc Q&A", fillcolor="#ffe6f0"]
|
||||
qa_dag [label="qa/dag.py\nper-run Merkle DAG\n(7 / 9 stages)", fillcolor="#ffe6f0"]
|
||||
qa_keys [label="qa/keys.py\n8-dim cache_key + question_hash", fillcolor="#ffe6f0"]
|
||||
qa_inspect[label="qa/inspect.py\nsidecar diagnostic\n(read-only)", fillcolor="#ffe6f0"]
|
||||
}
|
||||
|
||||
// ---- Distill (surface → core) ----
|
||||
distill [label="distill/\nDistiller ABC + tfidf + first_sentence", fillcolor="#fff0e0"]
|
||||
|
||||
// ---- Federation (off by default) ----
|
||||
mesh [label="mesh/\nfederation: identity, roster,\nAEAD epoch secret, gossip wire", fillcolor="#f0e6ff"]
|
||||
|
||||
// ---- Entry point ----
|
||||
cli [label="cli.py\nargparse entrypoint\n(make targets call into here)", fillcolor="#dddddd", shape=note]
|
||||
|
||||
// Substrate dependencies
|
||||
document -> merkle
|
||||
store -> merkle
|
||||
store -> document
|
||||
ingest -> store
|
||||
ingest -> document
|
||||
|
||||
// Sources -> ingest path
|
||||
wikipedia -> ingest
|
||||
wiki_xml -> ingest
|
||||
html_page -> ingest [label="parse_html →\nedges rows"]
|
||||
crawler -> html_page
|
||||
grok -> ingest
|
||||
vcs -> ingest
|
||||
|
||||
// Retrieval pipeline
|
||||
qa_query -> search
|
||||
qa_query -> concepts [label="synonym_expand /\nrivalry_excluded"]
|
||||
qa_query -> qa_keys
|
||||
qa_query -> qa_verify
|
||||
qa_query -> qa_evidence
|
||||
qa_query -> qa_dag
|
||||
qa_query -> wikitext [label="to_base()"]
|
||||
qa_verify -> qa_warrant
|
||||
qa_verify -> qa_evidence
|
||||
qa_runner -> qa_query
|
||||
qa_inspect -> qa_query [style=dashed, label="read-only sidecar"]
|
||||
|
||||
// Concepts reads existing edges
|
||||
concepts -> store [label="reads edges,\nwrites concept_relations"]
|
||||
|
||||
// Search reads chunks
|
||||
search -> store
|
||||
|
||||
// Distill consumes store, writes derivations
|
||||
distill -> store
|
||||
distill -> document
|
||||
|
||||
// Eviction
|
||||
evict -> store
|
||||
|
||||
// Mesh
|
||||
mesh -> store
|
||||
|
||||
// CLI dispatches everywhere
|
||||
cli -> ingest
|
||||
cli -> qa_runner
|
||||
cli -> qa_query
|
||||
cli -> qa_inspect
|
||||
cli -> distill
|
||||
cli -> evict
|
||||
cli -> mesh
|
||||
cli -> concepts
|
||||
}
|
||||
BIN
docs/diagrams/aborist-modules.png
Normal file
|
After Width: | Height: | Size: 309 KiB |
443
docs/diagrams/aborist-modules.svg
Normal file
|
|
@ -0,0 +1,443 @@
|
|||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<!-- Generated by graphviz version 2.43.0 (0)
|
||||
-->
|
||||
<!-- Title: aborist_modules Pages: 1 -->
|
||||
<svg width="1613pt" height="953pt"
|
||||
viewBox="0.00 0.00 1613.00 952.66" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 948.66)">
|
||||
<title>aborist_modules</title>
|
||||
<polygon fill="white" stroke="transparent" points="-4,4 -4,-948.66 1609,-948.66 1609,4 -4,4"/>
|
||||
<g id="clust1" class="cluster">
|
||||
<title>cluster_substrate</title>
|
||||
<path fill="white" stroke="#999999" stroke-dasharray="5,2" d="M1245,-506.67C1245,-506.67 1585,-506.67 1585,-506.67 1591,-506.67 1597,-512.67 1597,-518.67 1597,-518.67 1597,-623.67 1597,-623.67 1597,-629.67 1591,-635.67 1585,-635.67 1585,-635.67 1245,-635.67 1245,-635.67 1239,-635.67 1233,-629.67 1233,-623.67 1233,-623.67 1233,-518.67 1233,-518.67 1233,-512.67 1239,-506.67 1245,-506.67"/>
|
||||
<text text-anchor="middle" x="1415" y="-620.47" font-family="Times,serif" font-size="14.00">substrate</text>
|
||||
</g>
|
||||
<g id="clust2" class="cluster">
|
||||
<title>cluster_storage</title>
|
||||
<path fill="white" stroke="#999999" stroke-dasharray="5,2" d="M710,-506.67C710,-506.67 1188,-506.67 1188,-506.67 1194,-506.67 1200,-512.67 1200,-518.67 1200,-518.67 1200,-623.67 1200,-623.67 1200,-629.67 1194,-635.67 1188,-635.67 1188,-635.67 710,-635.67 710,-635.67 704,-635.67 698,-629.67 698,-623.67 698,-623.67 698,-518.67 698,-518.67 698,-512.67 704,-506.67 710,-506.67"/>
|
||||
<text text-anchor="middle" x="949" y="-620.47" font-family="Times,serif" font-size="14.00">storage</text>
|
||||
</g>
|
||||
<g id="clust3" class="cluster">
|
||||
<title>cluster_sources</title>
|
||||
<path fill="white" stroke="#999999" stroke-dasharray="5,2" d="M221.5,-586.67C221.5,-586.67 580,-586.67 580,-586.67 586,-586.67 592,-592.67 592,-598.67 592,-598.67 592,-865.67 592,-865.67 592,-871.67 586,-877.67 580,-877.67 580,-877.67 221.5,-877.67 221.5,-877.67 215.5,-877.67 209.5,-871.67 209.5,-865.67 209.5,-865.67 209.5,-598.67 209.5,-598.67 209.5,-592.67 215.5,-586.67 221.5,-586.67"/>
|
||||
<text text-anchor="middle" x="400.75" y="-862.47" font-family="Times,serif" font-size="14.00">sources/</text>
|
||||
</g>
|
||||
<g id="clust4" class="cluster">
|
||||
<title>cluster_retrieval</title>
|
||||
<path fill="white" stroke="#999999" stroke-dasharray="5,2" d="M220,-121.67C220,-121.67 1200,-121.67 1200,-121.67 1206,-121.67 1212,-127.67 1212,-133.67 1212,-133.67 1212,-438.67 1212,-438.67 1212,-444.67 1206,-450.67 1200,-450.67 1200,-450.67 220,-450.67 220,-450.67 214,-450.67 208,-444.67 208,-438.67 208,-438.67 208,-133.67 208,-133.67 208,-127.67 214,-121.67 220,-121.67"/>
|
||||
<text text-anchor="middle" x="710" y="-435.47" font-family="Times,serif" font-size="14.00">retrieval & verifier</text>
|
||||
</g>
|
||||
<!-- merkle -->
|
||||
<g id="node1" class="node">
|
||||
<title>merkle</title>
|
||||
<path fill="#fff7e6" stroke="black" d="M1577,-577.67C1577,-577.67 1439,-577.67 1439,-577.67 1433,-577.67 1427,-571.67 1427,-565.67 1427,-565.67 1427,-553.67 1427,-553.67 1427,-547.67 1433,-541.67 1439,-541.67 1439,-541.67 1577,-541.67 1577,-541.67 1583,-541.67 1589,-547.67 1589,-553.67 1589,-553.67 1589,-565.67 1589,-565.67 1589,-571.67 1583,-577.67 1577,-577.67"/>
|
||||
<text text-anchor="middle" x="1508" y="-562.67" font-family="Helvetica,sans-Serif" font-size="10.00">merkle.py</text>
|
||||
<text text-anchor="middle" x="1508" y="-551.67" font-family="Helvetica,sans-Serif" font-size="10.00">(Python port of Go merkle.go)</text>
|
||||
</g>
|
||||
<!-- document -->
|
||||
<g id="node2" class="node">
|
||||
<title>document</title>
|
||||
<path fill="#fff7e6" stroke="black" d="M1378,-604.67C1378,-604.67 1253,-604.67 1253,-604.67 1247,-604.67 1241,-598.67 1241,-592.67 1241,-592.67 1241,-580.67 1241,-580.67 1241,-574.67 1247,-568.67 1253,-568.67 1253,-568.67 1378,-568.67 1378,-568.67 1384,-568.67 1390,-574.67 1390,-580.67 1390,-580.67 1390,-592.67 1390,-592.67 1390,-598.67 1384,-604.67 1378,-604.67"/>
|
||||
<text text-anchor="middle" x="1315.5" y="-589.67" font-family="Helvetica,sans-Serif" font-size="10.00">document.py</text>
|
||||
<text text-anchor="middle" x="1315.5" y="-578.67" font-family="Helvetica,sans-Serif" font-size="10.00">Document · Edge · Chunker</text>
|
||||
</g>
|
||||
<!-- document->merkle -->
|
||||
<g id="edge1" class="edge">
|
||||
<title>document->merkle</title>
|
||||
<path fill="none" stroke="black" d="M1390.15,-576.23C1398.9,-574.99 1407.91,-573.72 1416.85,-572.45"/>
|
||||
<polygon fill="black" stroke="black" points="1417.5,-575.89 1426.91,-571.02 1416.51,-568.96 1417.5,-575.89"/>
|
||||
</g>
|
||||
<!-- wikitext -->
|
||||
<g id="node3" class="node">
|
||||
<title>wikitext</title>
|
||||
<path fill="#fff7e6" stroke="black" d="M1362.5,-550.67C1362.5,-550.67 1268.5,-550.67 1268.5,-550.67 1262.5,-550.67 1256.5,-544.67 1256.5,-538.67 1256.5,-538.67 1256.5,-526.67 1256.5,-526.67 1256.5,-520.67 1262.5,-514.67 1268.5,-514.67 1268.5,-514.67 1362.5,-514.67 1362.5,-514.67 1368.5,-514.67 1374.5,-520.67 1374.5,-526.67 1374.5,-526.67 1374.5,-538.67 1374.5,-538.67 1374.5,-544.67 1368.5,-550.67 1362.5,-550.67"/>
|
||||
<text text-anchor="middle" x="1315.5" y="-535.67" font-family="Helvetica,sans-Serif" font-size="10.00">wikitext.py</text>
|
||||
<text text-anchor="middle" x="1315.5" y="-524.67" font-family="Helvetica,sans-Serif" font-size="10.00">to_base() prose-strip</text>
|
||||
</g>
|
||||
<!-- store -->
|
||||
<g id="node4" class="node">
|
||||
<title>store</title>
|
||||
<path fill="#e6f0ff" stroke="black" d="M1180,-556.17C1180,-556.17 1079,-556.17 1079,-556.17 1073,-556.17 1067,-550.17 1067,-544.17 1067,-544.17 1067,-527.17 1067,-527.17 1067,-521.17 1073,-515.17 1079,-515.17 1079,-515.17 1180,-515.17 1180,-515.17 1186,-515.17 1192,-521.17 1192,-527.17 1192,-527.17 1192,-544.17 1192,-544.17 1192,-550.17 1186,-556.17 1180,-556.17"/>
|
||||
<text text-anchor="middle" x="1129.5" y="-544.17" font-family="Helvetica,sans-Serif" font-size="10.00">store.py</text>
|
||||
<text text-anchor="middle" x="1129.5" y="-533.17" font-family="Helvetica,sans-Serif" font-size="10.00">v9.8 SQLite schema +</text>
|
||||
<text text-anchor="middle" x="1129.5" y="-522.17" font-family="Helvetica,sans-Serif" font-size="10.00">audit chain helpers</text>
|
||||
</g>
|
||||
<!-- store->merkle -->
|
||||
<g id="edge2" class="edge">
|
||||
<title>store->merkle</title>
|
||||
<path fill="none" stroke="black" d="M1190.03,-515.15C1203.96,-511.22 1218.87,-507.7 1233,-505.67 1302.07,-495.77 1321.85,-490.7 1390,-505.67 1417.34,-511.68 1445.91,-524.71 1467.97,-536.47"/>
|
||||
<polygon fill="black" stroke="black" points="1466.55,-539.68 1477,-541.41 1469.9,-533.54 1466.55,-539.68"/>
|
||||
</g>
|
||||
<!-- store->document -->
|
||||
<g id="edge3" class="edge">
|
||||
<title>store->document</title>
|
||||
<path fill="none" stroke="black" d="M1192.03,-552.73C1207.25,-556.95 1223.73,-561.51 1239.52,-565.89"/>
|
||||
<polygon fill="black" stroke="black" points="1238.68,-569.29 1249.25,-568.59 1240.55,-562.54 1238.68,-569.29"/>
|
||||
</g>
|
||||
<!-- ingest -->
|
||||
<g id="node5" class="node">
|
||||
<title>ingest</title>
|
||||
<path fill="#e6f0ff" stroke="black" d="M897,-604.67C897,-604.67 718,-604.67 718,-604.67 712,-604.67 706,-598.67 706,-592.67 706,-592.67 706,-580.67 706,-580.67 706,-574.67 712,-568.67 718,-568.67 718,-568.67 897,-568.67 897,-568.67 903,-568.67 909,-574.67 909,-580.67 909,-580.67 909,-592.67 909,-592.67 909,-598.67 903,-604.67 897,-604.67"/>
|
||||
<text text-anchor="middle" x="807.5" y="-589.67" font-family="Helvetica,sans-Serif" font-size="10.00">ingest.py</text>
|
||||
<text text-anchor="middle" x="807.5" y="-578.67" font-family="Helvetica,sans-Serif" font-size="10.00">normalize → chunk → merkle → upsert</text>
|
||||
</g>
|
||||
<!-- ingest->document -->
|
||||
<g id="edge5" class="edge">
|
||||
<title>ingest->document</title>
|
||||
<path fill="none" stroke="black" d="M909.07,-586.67C1003.2,-586.67 1142.19,-586.67 1230.52,-586.67"/>
|
||||
<polygon fill="black" stroke="black" points="1230.77,-590.17 1240.77,-586.67 1230.77,-583.17 1230.77,-590.17"/>
|
||||
</g>
|
||||
<!-- ingest->store -->
|
||||
<g id="edge4" class="edge">
|
||||
<title>ingest->store</title>
|
||||
<path fill="none" stroke="black" d="M909.31,-570.61C957.24,-562.97 1013.42,-554.01 1056.9,-547.08"/>
|
||||
<polygon fill="black" stroke="black" points="1057.5,-550.53 1066.82,-545.5 1056.39,-543.62 1057.5,-550.53"/>
|
||||
</g>
|
||||
<!-- evict -->
|
||||
<g id="node6" class="node">
|
||||
<title>evict</title>
|
||||
<path fill="#e6f0ff" stroke="black" d="M838.5,-550.67C838.5,-550.67 776.5,-550.67 776.5,-550.67 770.5,-550.67 764.5,-544.67 764.5,-538.67 764.5,-538.67 764.5,-526.67 764.5,-526.67 764.5,-520.67 770.5,-514.67 776.5,-514.67 776.5,-514.67 838.5,-514.67 838.5,-514.67 844.5,-514.67 850.5,-520.67 850.5,-526.67 850.5,-526.67 850.5,-538.67 850.5,-538.67 850.5,-544.67 844.5,-550.67 838.5,-550.67"/>
|
||||
<text text-anchor="middle" x="807.5" y="-535.67" font-family="Helvetica,sans-Serif" font-size="10.00">evict.py</text>
|
||||
<text text-anchor="middle" x="807.5" y="-524.67" font-family="Helvetica,sans-Serif" font-size="10.00">hot ↔ cold tier</text>
|
||||
</g>
|
||||
<!-- evict->store -->
|
||||
<g id="edge27" class="edge">
|
||||
<title>evict->store</title>
|
||||
<path fill="none" stroke="black" d="M850.66,-533.07C902.99,-533.56 993.06,-534.4 1056.43,-535"/>
|
||||
<polygon fill="black" stroke="black" points="1056.72,-538.5 1066.75,-535.09 1056.79,-531.5 1056.72,-538.5"/>
|
||||
</g>
|
||||
<!-- wikipedia -->
|
||||
<g id="node7" class="node">
|
||||
<title>wikipedia</title>
|
||||
<path fill="#e7ffe7" stroke="black" d="M324.5,-792.67C324.5,-792.67 229.5,-792.67 229.5,-792.67 223.5,-792.67 217.5,-786.67 217.5,-780.67 217.5,-780.67 217.5,-768.67 217.5,-768.67 217.5,-762.67 223.5,-756.67 229.5,-756.67 229.5,-756.67 324.5,-756.67 324.5,-756.67 330.5,-756.67 336.5,-762.67 336.5,-768.67 336.5,-768.67 336.5,-780.67 336.5,-780.67 336.5,-786.67 330.5,-792.67 324.5,-792.67"/>
|
||||
<text text-anchor="middle" x="277" y="-777.67" font-family="Helvetica,sans-Serif" font-size="10.00">wikipedia.py</text>
|
||||
<text text-anchor="middle" x="277" y="-766.67" font-family="Helvetica,sans-Serif" font-size="10.00">cur + old SQL dumps</text>
|
||||
</g>
|
||||
<!-- wikipedia->ingest -->
|
||||
<g id="edge6" class="edge">
|
||||
<title>wikipedia->ingest</title>
|
||||
<path fill="none" stroke="black" d="M336.82,-788.51C426.22,-805.68 595.67,-822.41 688,-729.67 702.98,-714.63 685.5,-651.83 698,-634.67 705.68,-624.12 716.21,-615.86 727.7,-609.39"/>
|
||||
<polygon fill="black" stroke="black" points="729.35,-612.47 736.64,-604.79 726.15,-606.25 729.35,-612.47"/>
|
||||
</g>
|
||||
<!-- wiki_xml -->
|
||||
<g id="node8" class="node">
|
||||
<title>wiki_xml</title>
|
||||
<path fill="#e7ffe7" stroke="black" d="M317.5,-684.67C317.5,-684.67 236.5,-684.67 236.5,-684.67 230.5,-684.67 224.5,-678.67 224.5,-672.67 224.5,-672.67 224.5,-660.67 224.5,-660.67 224.5,-654.67 230.5,-648.67 236.5,-648.67 236.5,-648.67 317.5,-648.67 317.5,-648.67 323.5,-648.67 329.5,-654.67 329.5,-660.67 329.5,-660.67 329.5,-672.67 329.5,-672.67 329.5,-678.67 323.5,-684.67 317.5,-684.67"/>
|
||||
<text text-anchor="middle" x="277" y="-669.67" font-family="Helvetica,sans-Serif" font-size="10.00">wikipedia_xml.py</text>
|
||||
<text text-anchor="middle" x="277" y="-658.67" font-family="Helvetica,sans-Serif" font-size="10.00">phase IV iterparse</text>
|
||||
</g>
|
||||
<!-- wiki_xml->ingest -->
|
||||
<g id="edge7" class="edge">
|
||||
<title>wiki_xml->ingest</title>
|
||||
<path fill="none" stroke="black" d="M329.54,-659.8C407.32,-649.31 559.3,-628.33 688,-607.67 690.66,-607.24 693.36,-606.81 696.08,-606.36"/>
|
||||
<polygon fill="black" stroke="black" points="696.72,-609.8 706.01,-604.71 695.57,-602.9 696.72,-609.8"/>
|
||||
</g>
|
||||
<!-- html_page -->
|
||||
<g id="node9" class="node">
|
||||
<title>html_page</title>
|
||||
<path fill="#e7ffe7" stroke="black" d="M572,-725.67C572,-725.67 489,-725.67 489,-725.67 483,-725.67 477,-719.67 477,-713.67 477,-713.67 477,-701.67 477,-701.67 477,-695.67 483,-689.67 489,-689.67 489,-689.67 572,-689.67 572,-689.67 578,-689.67 584,-695.67 584,-701.67 584,-701.67 584,-713.67 584,-713.67 584,-719.67 578,-725.67 572,-725.67"/>
|
||||
<text text-anchor="middle" x="530.5" y="-710.67" font-family="Helvetica,sans-Serif" font-size="10.00">html_page.py</text>
|
||||
<text text-anchor="middle" x="530.5" y="-699.67" font-family="Helvetica,sans-Serif" font-size="10.00">selectolax + httpx</text>
|
||||
</g>
|
||||
<!-- html_page->ingest -->
|
||||
<g id="edge8" class="edge">
|
||||
<title>html_page->ingest</title>
|
||||
<path fill="none" stroke="black" d="M584.01,-708.28C617.86,-705.86 660.53,-697.09 688,-670.67 699.97,-659.16 687.14,-647.24 698,-634.67 707.16,-624.07 719.1,-615.72 731.71,-609.17"/>
|
||||
<polygon fill="black" stroke="black" points="733.49,-612.2 740.98,-604.7 730.45,-605.89 733.49,-612.2"/>
|
||||
<text text-anchor="middle" x="645" y="-718.47" font-family="Helvetica,sans-Serif" font-size="9.00">parse_html →</text>
|
||||
<text text-anchor="middle" x="645" y="-708.47" font-family="Helvetica,sans-Serif" font-size="9.00">edges rows</text>
|
||||
</g>
|
||||
<!-- crawler -->
|
||||
<g id="node10" class="node">
|
||||
<title>crawler</title>
|
||||
<path fill="#e7ffe7" stroke="black" d="M313.5,-738.67C313.5,-738.67 240.5,-738.67 240.5,-738.67 234.5,-738.67 228.5,-732.67 228.5,-726.67 228.5,-726.67 228.5,-714.67 228.5,-714.67 228.5,-708.67 234.5,-702.67 240.5,-702.67 240.5,-702.67 313.5,-702.67 313.5,-702.67 319.5,-702.67 325.5,-708.67 325.5,-714.67 325.5,-714.67 325.5,-726.67 325.5,-726.67 325.5,-732.67 319.5,-738.67 313.5,-738.67"/>
|
||||
<text text-anchor="middle" x="277" y="-723.67" font-family="Helvetica,sans-Serif" font-size="10.00">crawler/</text>
|
||||
<text text-anchor="middle" x="277" y="-712.67" font-family="Helvetica,sans-Serif" font-size="10.00">async HTML BFS</text>
|
||||
</g>
|
||||
<!-- crawler->html_page -->
|
||||
<g id="edge9" class="edge">
|
||||
<title>crawler->html_page</title>
|
||||
<path fill="none" stroke="black" d="M325.6,-718.21C365.55,-716.15 422.93,-713.18 466.65,-710.92"/>
|
||||
<polygon fill="black" stroke="black" points="467.07,-714.4 476.88,-710.39 466.71,-707.41 467.07,-714.4"/>
|
||||
</g>
|
||||
<!-- grok -->
|
||||
<g id="node11" class="node">
|
||||
<title>grok</title>
|
||||
<path fill="#e7ffe7" stroke="black" d="M298.5,-630.67C298.5,-630.67 255.5,-630.67 255.5,-630.67 249.5,-630.67 243.5,-624.67 243.5,-618.67 243.5,-618.67 243.5,-606.67 243.5,-606.67 243.5,-600.67 249.5,-594.67 255.5,-594.67 255.5,-594.67 298.5,-594.67 298.5,-594.67 304.5,-594.67 310.5,-600.67 310.5,-606.67 310.5,-606.67 310.5,-618.67 310.5,-618.67 310.5,-624.67 304.5,-630.67 298.5,-630.67"/>
|
||||
<text text-anchor="middle" x="277" y="-615.67" font-family="Helvetica,sans-Serif" font-size="10.00">grok.py</text>
|
||||
<text text-anchor="middle" x="277" y="-604.67" font-family="Helvetica,sans-Serif" font-size="10.00">xAI export</text>
|
||||
</g>
|
||||
<!-- grok->ingest -->
|
||||
<g id="edge10" class="edge">
|
||||
<title>grok->ingest</title>
|
||||
<path fill="none" stroke="black" d="M310.66,-611.07C385.85,-607.37 574.05,-598.11 695.75,-592.12"/>
|
||||
<polygon fill="black" stroke="black" points="696.04,-595.61 705.86,-591.62 695.7,-588.62 696.04,-595.61"/>
|
||||
</g>
|
||||
<!-- vcs -->
|
||||
<g id="node12" class="node">
|
||||
<title>vcs</title>
|
||||
<path fill="#e7ffe7" stroke="black" d="M309.5,-846.67C309.5,-846.67 244.5,-846.67 244.5,-846.67 238.5,-846.67 232.5,-840.67 232.5,-834.67 232.5,-834.67 232.5,-822.67 232.5,-822.67 232.5,-816.67 238.5,-810.67 244.5,-810.67 244.5,-810.67 309.5,-810.67 309.5,-810.67 315.5,-810.67 321.5,-816.67 321.5,-822.67 321.5,-822.67 321.5,-834.67 321.5,-834.67 321.5,-840.67 315.5,-846.67 309.5,-846.67"/>
|
||||
<text text-anchor="middle" x="277" y="-831.67" font-family="Helvetica,sans-Serif" font-size="10.00">vcs.py</text>
|
||||
<text text-anchor="middle" x="277" y="-820.67" font-family="Helvetica,sans-Serif" font-size="10.00">git + Mercurial</text>
|
||||
</g>
|
||||
<!-- vcs->ingest -->
|
||||
<g id="edge11" class="edge">
|
||||
<title>vcs->ingest</title>
|
||||
<path fill="none" stroke="black" d="M321.57,-838.93C408.53,-856.48 602.61,-880.42 688,-768.67 706.13,-744.94 680.79,-659.07 698,-634.67 705.4,-624.18 715.63,-615.95 726.87,-609.51"/>
|
||||
<polygon fill="black" stroke="black" points="728.82,-612.44 736.07,-604.71 725.59,-606.23 728.82,-612.44"/>
|
||||
</g>
|
||||
<!-- search -->
|
||||
<g id="node13" class="node">
|
||||
<title>search</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M854.5,-245.67C854.5,-245.67 760.5,-245.67 760.5,-245.67 754.5,-245.67 748.5,-239.67 748.5,-233.67 748.5,-233.67 748.5,-221.67 748.5,-221.67 748.5,-215.67 754.5,-209.67 760.5,-209.67 760.5,-209.67 854.5,-209.67 854.5,-209.67 860.5,-209.67 866.5,-215.67 866.5,-221.67 866.5,-221.67 866.5,-233.67 866.5,-233.67 866.5,-239.67 860.5,-245.67 854.5,-245.67"/>
|
||||
<text text-anchor="middle" x="807.5" y="-230.67" font-family="Helvetica,sans-Serif" font-size="10.00">search/</text>
|
||||
<text text-anchor="middle" x="807.5" y="-219.67" font-family="Helvetica,sans-Serif" font-size="10.00">FTS5 backend + ABC</text>
|
||||
</g>
|
||||
<!-- search->store -->
|
||||
<g id="edge24" class="edge">
|
||||
<title>search->store</title>
|
||||
<path fill="none" stroke="black" d="M866.62,-236.81C881.21,-240.8 896.31,-246.51 909,-254.67 984.15,-303 991.42,-332.83 1037,-409.67 1047.99,-428.2 1043.58,-436.41 1055,-454.67 1067.09,-474.01 1084,-493.02 1098.56,-507.77"/>
|
||||
<polygon fill="black" stroke="black" points="1096.36,-510.51 1105.92,-515.07 1101.29,-505.54 1096.36,-510.51"/>
|
||||
</g>
|
||||
<!-- concepts -->
|
||||
<g id="node14" class="node">
|
||||
<title>concepts</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M875.5,-419.17C875.5,-419.17 739.5,-419.17 739.5,-419.17 733.5,-419.17 727.5,-413.17 727.5,-407.17 727.5,-407.17 727.5,-390.17 727.5,-390.17 727.5,-384.17 733.5,-378.17 739.5,-378.17 739.5,-378.17 875.5,-378.17 875.5,-378.17 881.5,-378.17 887.5,-384.17 887.5,-390.17 887.5,-390.17 887.5,-407.17 887.5,-407.17 887.5,-413.17 881.5,-419.17 875.5,-419.17"/>
|
||||
<text text-anchor="middle" x="807.5" y="-407.17" font-family="Helvetica,sans-Serif" font-size="10.00">concepts/</text>
|
||||
<text text-anchor="middle" x="807.5" y="-396.17" font-family="Helvetica,sans-Serif" font-size="10.00">synonym & rivalry overlay</text>
|
||||
<text text-anchor="middle" x="807.5" y="-385.17" font-family="Helvetica,sans-Serif" font-size="10.00">(per-shard concept_relations)</text>
|
||||
</g>
|
||||
<!-- concepts->store -->
|
||||
<g id="edge23" class="edge">
|
||||
<title>concepts->store</title>
|
||||
<path fill="none" stroke="black" d="M887.73,-405.83C933.45,-412.47 990.84,-425.23 1037,-449.67 1063.92,-463.93 1088.72,-488.25 1105.69,-507.4"/>
|
||||
<polygon fill="black" stroke="black" points="1103.05,-509.7 1112.24,-514.97 1108.35,-505.12 1103.05,-509.7"/>
|
||||
<text text-anchor="middle" x="982" y="-462.47" font-family="Helvetica,sans-Serif" font-size="9.00">reads edges,</text>
|
||||
<text text-anchor="middle" x="982" y="-452.47" font-family="Helvetica,sans-Serif" font-size="9.00">writes concept_relations</text>
|
||||
</g>
|
||||
<!-- qa_query -->
|
||||
<g id="node15" class="node">
|
||||
<title>qa_query</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M569.5,-302.67C569.5,-302.67 491.5,-302.67 491.5,-302.67 485.5,-302.67 479.5,-296.67 479.5,-290.67 479.5,-290.67 479.5,-278.67 479.5,-278.67 479.5,-272.67 485.5,-266.67 491.5,-266.67 491.5,-266.67 569.5,-266.67 569.5,-266.67 575.5,-266.67 581.5,-272.67 581.5,-278.67 581.5,-278.67 581.5,-290.67 581.5,-290.67 581.5,-296.67 575.5,-302.67 569.5,-302.67"/>
|
||||
<text text-anchor="middle" x="530.5" y="-287.67" font-family="Helvetica,sans-Serif" font-size="10.00">qa/query.py</text>
|
||||
<text text-anchor="middle" x="530.5" y="-276.67" font-family="Helvetica,sans-Serif" font-size="10.00">multi-source RAG</text>
|
||||
</g>
|
||||
<!-- qa_query->wikitext -->
|
||||
<g id="edge18" class="edge">
|
||||
<title>qa_query->wikitext</title>
|
||||
<path fill="none" stroke="black" d="M535.39,-266.28C542.84,-235.71 562.03,-174.56 602,-141.67 811.45,30.71 1005.11,54.79 1212,-120.67 1271.51,-171.14 1302.69,-419.93 1311.7,-504.49"/>
|
||||
<polygon fill="black" stroke="black" points="1308.24,-505.05 1312.76,-514.63 1315.2,-504.32 1308.24,-505.05"/>
|
||||
<text text-anchor="middle" x="982" y="-20.47" font-family="Helvetica,sans-Serif" font-size="9.00">to_base()</text>
|
||||
</g>
|
||||
<!-- qa_query->search -->
|
||||
<g id="edge12" class="edge">
|
||||
<title>qa_query->search</title>
|
||||
<path fill="none" stroke="black" d="M581.62,-274.28C625.53,-265.18 689.76,-251.87 738.34,-241.8"/>
|
||||
<polygon fill="black" stroke="black" points="739.06,-245.22 748.14,-239.77 737.64,-238.37 739.06,-245.22"/>
|
||||
</g>
|
||||
<!-- qa_query->concepts -->
|
||||
<g id="edge13" class="edge">
|
||||
<title>qa_query->concepts</title>
|
||||
<path fill="none" stroke="black" d="M562.57,-302.69C595.39,-321.16 649.07,-349.78 698,-368.67 704.31,-371.11 710.89,-373.43 717.57,-375.63"/>
|
||||
<polygon fill="black" stroke="black" points="716.79,-379.06 727.38,-378.77 718.92,-372.39 716.79,-379.06"/>
|
||||
<text text-anchor="middle" x="645" y="-376.47" font-family="Helvetica,sans-Serif" font-size="9.00">synonym_expand /</text>
|
||||
<text text-anchor="middle" x="645" y="-366.47" font-family="Helvetica,sans-Serif" font-size="9.00">rivalry_excluded</text>
|
||||
</g>
|
||||
<!-- qa_verify -->
|
||||
<g id="node16" class="node">
|
||||
<title>qa_verify</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M880,-191.17C880,-191.17 735,-191.17 735,-191.17 729,-191.17 723,-185.17 723,-179.17 723,-179.17 723,-162.17 723,-162.17 723,-156.17 729,-150.17 735,-150.17 735,-150.17 880,-150.17 880,-150.17 886,-150.17 892,-156.17 892,-162.17 892,-162.17 892,-179.17 892,-179.17 892,-185.17 886,-191.17 880,-191.17"/>
|
||||
<text text-anchor="middle" x="807.5" y="-179.17" font-family="Helvetica,sans-Serif" font-size="10.00">qa/verify.py</text>
|
||||
<text text-anchor="middle" x="807.5" y="-168.17" font-family="Helvetica,sans-Serif" font-size="10.00">quote/span/entity/paraphrase</text>
|
||||
<text text-anchor="middle" x="807.5" y="-157.17" font-family="Helvetica,sans-Serif" font-size="10.00">+ claim_lattice (7 hard checks)</text>
|
||||
</g>
|
||||
<!-- qa_query->qa_verify -->
|
||||
<g id="edge15" class="edge">
|
||||
<title>qa_query->qa_verify</title>
|
||||
<path fill="none" stroke="black" d="M562.57,-266.65C595.39,-248.18 649.07,-219.57 698,-200.67 703.74,-198.45 709.72,-196.33 715.77,-194.3"/>
|
||||
<polygon fill="black" stroke="black" points="716.91,-197.62 725.34,-191.21 714.75,-190.96 716.91,-197.62"/>
|
||||
</g>
|
||||
<!-- qa_evidence -->
|
||||
<g id="node18" class="node">
|
||||
<title>qa_evidence</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M1192,-165.67C1192,-165.67 1067,-165.67 1067,-165.67 1061,-165.67 1055,-159.67 1055,-153.67 1055,-153.67 1055,-141.67 1055,-141.67 1055,-135.67 1061,-129.67 1067,-129.67 1067,-129.67 1192,-129.67 1192,-129.67 1198,-129.67 1204,-135.67 1204,-141.67 1204,-141.67 1204,-153.67 1204,-153.67 1204,-159.67 1198,-165.67 1192,-165.67"/>
|
||||
<text text-anchor="middle" x="1129.5" y="-150.67" font-family="Helvetica,sans-Serif" font-size="10.00">qa/evidence.py</text>
|
||||
<text text-anchor="middle" x="1129.5" y="-139.67" font-family="Helvetica,sans-Serif" font-size="10.00">EvidenceObject + spotlight</text>
|
||||
</g>
|
||||
<!-- qa_query->qa_evidence -->
|
||||
<g id="edge16" class="edge">
|
||||
<title>qa_query->qa_evidence</title>
|
||||
<path fill="none" stroke="black" d="M543.93,-266.59C568.54,-232.79 627.25,-161.53 698,-140.67 814.5,-106.32 956.55,-118.4 1044.77,-131.8"/>
|
||||
<polygon fill="black" stroke="black" points="1044.31,-135.27 1054.73,-133.36 1045.39,-128.36 1044.31,-135.27"/>
|
||||
</g>
|
||||
<!-- qa_dag -->
|
||||
<g id="node20" class="node">
|
||||
<title>qa_dag</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M852,-305.17C852,-305.17 763,-305.17 763,-305.17 757,-305.17 751,-299.17 751,-293.17 751,-293.17 751,-276.17 751,-276.17 751,-270.17 757,-264.17 763,-264.17 763,-264.17 852,-264.17 852,-264.17 858,-264.17 864,-270.17 864,-276.17 864,-276.17 864,-293.17 864,-293.17 864,-299.17 858,-305.17 852,-305.17"/>
|
||||
<text text-anchor="middle" x="807.5" y="-293.17" font-family="Helvetica,sans-Serif" font-size="10.00">qa/dag.py</text>
|
||||
<text text-anchor="middle" x="807.5" y="-282.17" font-family="Helvetica,sans-Serif" font-size="10.00">per-run Merkle DAG</text>
|
||||
<text text-anchor="middle" x="807.5" y="-271.17" font-family="Helvetica,sans-Serif" font-size="10.00">(7 / 9 stages)</text>
|
||||
</g>
|
||||
<!-- qa_query->qa_dag -->
|
||||
<g id="edge17" class="edge">
|
||||
<title>qa_query->qa_dag</title>
|
||||
<path fill="none" stroke="black" d="M581.62,-284.67C626.17,-284.67 691.61,-284.67 740.42,-284.67"/>
|
||||
<polygon fill="black" stroke="black" points="740.57,-288.17 750.57,-284.67 740.57,-281.17 740.57,-288.17"/>
|
||||
</g>
|
||||
<!-- qa_keys -->
|
||||
<g id="node21" class="node">
|
||||
<title>qa_keys</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M887.5,-359.67C887.5,-359.67 727.5,-359.67 727.5,-359.67 721.5,-359.67 715.5,-353.67 715.5,-347.67 715.5,-347.67 715.5,-335.67 715.5,-335.67 715.5,-329.67 721.5,-323.67 727.5,-323.67 727.5,-323.67 887.5,-323.67 887.5,-323.67 893.5,-323.67 899.5,-329.67 899.5,-335.67 899.5,-335.67 899.5,-347.67 899.5,-347.67 899.5,-353.67 893.5,-359.67 887.5,-359.67"/>
|
||||
<text text-anchor="middle" x="807.5" y="-344.67" font-family="Helvetica,sans-Serif" font-size="10.00">qa/keys.py</text>
|
||||
<text text-anchor="middle" x="807.5" y="-333.67" font-family="Helvetica,sans-Serif" font-size="10.00">8-dim cache_key + question_hash</text>
|
||||
</g>
|
||||
<!-- qa_query->qa_keys -->
|
||||
<g id="edge14" class="edge">
|
||||
<title>qa_query->qa_keys</title>
|
||||
<path fill="none" stroke="black" d="M581.62,-295.06C617.26,-302.45 666.28,-312.61 709.46,-321.56"/>
|
||||
<polygon fill="black" stroke="black" points="708.93,-325.02 719.44,-323.63 710.35,-318.17 708.93,-325.02"/>
|
||||
</g>
|
||||
<!-- qa_warrant -->
|
||||
<g id="node17" class="node">
|
||||
<title>qa_warrant</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M1190.5,-235.67C1190.5,-235.67 1068.5,-235.67 1068.5,-235.67 1062.5,-235.67 1056.5,-229.67 1056.5,-223.67 1056.5,-223.67 1056.5,-195.67 1056.5,-195.67 1056.5,-189.67 1062.5,-183.67 1068.5,-183.67 1068.5,-183.67 1190.5,-183.67 1190.5,-183.67 1196.5,-183.67 1202.5,-189.67 1202.5,-195.67 1202.5,-195.67 1202.5,-223.67 1202.5,-223.67 1202.5,-229.67 1196.5,-235.67 1190.5,-235.67"/>
|
||||
<text text-anchor="middle" x="1129.5" y="-223.67" font-family="Helvetica,sans-Serif" font-size="10.00">qa/warrant.py</text>
|
||||
<text text-anchor="middle" x="1129.5" y="-212.67" font-family="Helvetica,sans-Serif" font-size="10.00">5 anchor classes</text>
|
||||
<text text-anchor="middle" x="1129.5" y="-201.67" font-family="Helvetica,sans-Serif" font-size="10.00">(proper-noun · date · count</text>
|
||||
<text text-anchor="middle" x="1129.5" y="-190.67" font-family="Helvetica,sans-Serif" font-size="10.00"> · entity-list · cause)</text>
|
||||
</g>
|
||||
<!-- qa_verify->qa_warrant -->
|
||||
<g id="edge19" class="edge">
|
||||
<title>qa_verify->qa_warrant</title>
|
||||
<path fill="none" stroke="black" d="M892.16,-180.87C939.69,-186.66 998.96,-193.88 1046.25,-199.65"/>
|
||||
<polygon fill="black" stroke="black" points="1046.11,-203.16 1056.46,-200.89 1046.96,-196.21 1046.11,-203.16"/>
|
||||
</g>
|
||||
<!-- qa_verify->qa_evidence -->
|
||||
<g id="edge20" class="edge">
|
||||
<title>qa_verify->qa_evidence</title>
|
||||
<path fill="none" stroke="black" d="M892.16,-164.66C939.17,-161.28 997.67,-157.08 1044.7,-153.7"/>
|
||||
<polygon fill="black" stroke="black" points="1045.14,-157.17 1054.86,-152.96 1044.64,-150.19 1045.14,-157.17"/>
|
||||
</g>
|
||||
<!-- qa_runner -->
|
||||
<g id="node19" class="node">
|
||||
<title>qa_runner</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M326,-361.67C326,-361.67 228,-361.67 228,-361.67 222,-361.67 216,-355.67 216,-349.67 216,-349.67 216,-337.67 216,-337.67 216,-331.67 222,-325.67 228,-325.67 228,-325.67 326,-325.67 326,-325.67 332,-325.67 338,-331.67 338,-337.67 338,-337.67 338,-349.67 338,-349.67 338,-355.67 332,-361.67 326,-361.67"/>
|
||||
<text text-anchor="middle" x="277" y="-346.67" font-family="Helvetica,sans-Serif" font-size="10.00">qa/runner.py</text>
|
||||
<text text-anchor="middle" x="277" y="-335.67" font-family="Helvetica,sans-Serif" font-size="10.00">ask(): single-doc Q&A</text>
|
||||
</g>
|
||||
<!-- qa_runner->qa_query -->
|
||||
<g id="edge21" class="edge">
|
||||
<title>qa_runner->qa_query</title>
|
||||
<path fill="none" stroke="black" d="M338.24,-329.54C378.03,-320.21 429.96,-308.02 469.66,-298.71"/>
|
||||
<polygon fill="black" stroke="black" points="470.54,-302.1 479.48,-296.41 468.94,-295.28 470.54,-302.1"/>
|
||||
</g>
|
||||
<!-- qa_inspect -->
|
||||
<g id="node22" class="node">
|
||||
<title>qa_inspect</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M317.5,-305.17C317.5,-305.17 236.5,-305.17 236.5,-305.17 230.5,-305.17 224.5,-299.17 224.5,-293.17 224.5,-293.17 224.5,-276.17 224.5,-276.17 224.5,-270.17 230.5,-264.17 236.5,-264.17 236.5,-264.17 317.5,-264.17 317.5,-264.17 323.5,-264.17 329.5,-270.17 329.5,-276.17 329.5,-276.17 329.5,-293.17 329.5,-293.17 329.5,-299.17 323.5,-305.17 317.5,-305.17"/>
|
||||
<text text-anchor="middle" x="277" y="-293.17" font-family="Helvetica,sans-Serif" font-size="10.00">qa/inspect.py</text>
|
||||
<text text-anchor="middle" x="277" y="-282.17" font-family="Helvetica,sans-Serif" font-size="10.00">sidecar diagnostic</text>
|
||||
<text text-anchor="middle" x="277" y="-271.17" font-family="Helvetica,sans-Serif" font-size="10.00">(read-only)</text>
|
||||
</g>
|
||||
<!-- qa_inspect->qa_query -->
|
||||
<g id="edge22" class="edge">
|
||||
<title>qa_inspect->qa_query</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M329.5,-284.67C369.99,-284.67 426.49,-284.67 469.11,-284.67"/>
|
||||
<polygon fill="black" stroke="black" points="469.35,-288.17 479.35,-284.67 469.35,-281.17 469.35,-288.17"/>
|
||||
<text text-anchor="middle" x="419.5" y="-287.47" font-family="Helvetica,sans-Serif" font-size="9.00">read-only sidecar</text>
|
||||
</g>
|
||||
<!-- distill -->
|
||||
<g id="node23" class="node">
|
||||
<title>distill</title>
|
||||
<path fill="#fff0e0" stroke="black" d="M892.5,-679.67C892.5,-679.67 722.5,-679.67 722.5,-679.67 716.5,-679.67 710.5,-673.67 710.5,-667.67 710.5,-667.67 710.5,-655.67 710.5,-655.67 710.5,-649.67 716.5,-643.67 722.5,-643.67 722.5,-643.67 892.5,-643.67 892.5,-643.67 898.5,-643.67 904.5,-649.67 904.5,-655.67 904.5,-655.67 904.5,-667.67 904.5,-667.67 904.5,-673.67 898.5,-679.67 892.5,-679.67"/>
|
||||
<text text-anchor="middle" x="807.5" y="-664.67" font-family="Helvetica,sans-Serif" font-size="10.00">distill/</text>
|
||||
<text text-anchor="middle" x="807.5" y="-653.67" font-family="Helvetica,sans-Serif" font-size="10.00">Distiller ABC + tfidf + first_sentence</text>
|
||||
</g>
|
||||
<!-- distill->document -->
|
||||
<g id="edge26" class="edge">
|
||||
<title>distill->document</title>
|
||||
<path fill="none" stroke="black" d="M904.87,-667.65C987.77,-670.16 1109.79,-667.5 1212,-639.67 1235.36,-633.31 1259.51,-621.14 1278.5,-610.07"/>
|
||||
<polygon fill="black" stroke="black" points="1280.32,-613.06 1287.1,-604.92 1276.73,-607.05 1280.32,-613.06"/>
|
||||
</g>
|
||||
<!-- distill->store -->
|
||||
<g id="edge25" class="edge">
|
||||
<title>distill->store</title>
|
||||
<path fill="none" stroke="black" d="M896.52,-643.63C900.76,-642.38 904.94,-641.06 909,-639.67 971.41,-618.37 1039.47,-584.28 1083,-560.98"/>
|
||||
<polygon fill="black" stroke="black" points="1084.76,-564.01 1091.91,-556.18 1081.44,-557.85 1084.76,-564.01"/>
|
||||
</g>
|
||||
<!-- mesh -->
|
||||
<g id="node24" class="node">
|
||||
<title>mesh</title>
|
||||
<path fill="#f0e6ff" stroke="black" d="M350,-500.17C350,-500.17 204,-500.17 204,-500.17 198,-500.17 192,-494.17 192,-488.17 192,-488.17 192,-471.17 192,-471.17 192,-465.17 198,-459.17 204,-459.17 204,-459.17 350,-459.17 350,-459.17 356,-459.17 362,-465.17 362,-471.17 362,-471.17 362,-488.17 362,-488.17 362,-494.17 356,-500.17 350,-500.17"/>
|
||||
<text text-anchor="middle" x="277" y="-488.17" font-family="Helvetica,sans-Serif" font-size="10.00">mesh/</text>
|
||||
<text text-anchor="middle" x="277" y="-477.17" font-family="Helvetica,sans-Serif" font-size="10.00">federation: identity, roster,</text>
|
||||
<text text-anchor="middle" x="277" y="-466.17" font-family="Helvetica,sans-Serif" font-size="10.00">AEAD epoch secret, gossip wire</text>
|
||||
</g>
|
||||
<!-- mesh->store -->
|
||||
<g id="edge28" class="edge">
|
||||
<title>mesh->store</title>
|
||||
<path fill="none" stroke="black" d="M362.11,-480.61C483.22,-482.53 713.65,-488.34 909,-505.67 958.48,-510.06 1013.9,-517.66 1056.67,-524.11"/>
|
||||
<polygon fill="black" stroke="black" points="1056.29,-527.59 1066.71,-525.63 1057.35,-520.67 1056.29,-527.59"/>
|
||||
</g>
|
||||
<!-- cli -->
|
||||
<g id="node25" class="node">
|
||||
<title>cli</title>
|
||||
<polygon fill="#dddddd" stroke="black" points="149,-475.17 0,-475.17 0,-434.17 155,-434.17 155,-469.17 149,-475.17"/>
|
||||
<polyline fill="none" stroke="black" points="149,-475.17 149,-469.17 "/>
|
||||
<polyline fill="none" stroke="black" points="155,-469.17 149,-469.17 "/>
|
||||
<text text-anchor="middle" x="77.5" y="-463.17" font-family="Helvetica,sans-Serif" font-size="10.00">cli.py</text>
|
||||
<text text-anchor="middle" x="77.5" y="-452.17" font-family="Helvetica,sans-Serif" font-size="10.00">argparse entrypoint</text>
|
||||
<text text-anchor="middle" x="77.5" y="-441.17" font-family="Helvetica,sans-Serif" font-size="10.00">(make targets call into here)</text>
|
||||
</g>
|
||||
<!-- cli->ingest -->
|
||||
<g id="edge29" class="edge">
|
||||
<title>cli->ingest</title>
|
||||
<path fill="none" stroke="black" d="M99.43,-475.28C120.75,-494.85 155.76,-523.23 192,-536.67 281.45,-569.86 544.33,-581.07 695.71,-584.82"/>
|
||||
<polygon fill="black" stroke="black" points="695.88,-588.33 705.96,-585.07 696.05,-581.33 695.88,-588.33"/>
|
||||
</g>
|
||||
<!-- cli->evict -->
|
||||
<g id="edge34" class="edge">
|
||||
<title>cli->evict</title>
|
||||
<path fill="none" stroke="black" d="M111.87,-475.4C133.77,-487.84 163.53,-502.57 192,-509.67 394.76,-560.22 646.69,-546.92 754.08,-537.85"/>
|
||||
<polygon fill="black" stroke="black" points="754.65,-541.32 764.31,-536.96 754.05,-534.34 754.65,-541.32"/>
|
||||
</g>
|
||||
<!-- cli->concepts -->
|
||||
<g id="edge36" class="edge">
|
||||
<title>cli->concepts</title>
|
||||
<path fill="none" stroke="black" d="M155.03,-448.79C290.77,-438.34 572.94,-416.64 717.27,-405.54"/>
|
||||
<polygon fill="black" stroke="black" points="717.72,-409.01 727.42,-404.76 717.18,-402.03 717.72,-409.01"/>
|
||||
</g>
|
||||
<!-- cli->qa_query -->
|
||||
<g id="edge31" class="edge">
|
||||
<title>cli->qa_query</title>
|
||||
<path fill="none" stroke="black" d="M124.1,-434.15C139.44,-427.71 156.79,-420.91 173,-415.67 255.16,-389.12 281.01,-400.61 362,-370.67 407.96,-353.68 457.53,-327.1 490.72,-307.99"/>
|
||||
<polygon fill="black" stroke="black" points="492.75,-310.86 499.63,-302.81 489.23,-304.81 492.75,-310.86"/>
|
||||
</g>
|
||||
<!-- cli->qa_runner -->
|
||||
<g id="edge30" class="edge">
|
||||
<title>cli->qa_runner</title>
|
||||
<path fill="none" stroke="black" d="M98.93,-433.84C117,-416.44 144.81,-392.07 173,-376.67 183.36,-371.01 194.85,-366.14 206.29,-362.01"/>
|
||||
<polygon fill="black" stroke="black" points="207.54,-365.28 215.85,-358.72 205.26,-358.66 207.54,-365.28"/>
|
||||
</g>
|
||||
<!-- cli->qa_inspect -->
|
||||
<g id="edge32" class="edge">
|
||||
<title>cli->qa_inspect</title>
|
||||
<path fill="none" stroke="black" d="M90.58,-434.02C108.97,-404.25 146.66,-349.26 192,-316.67 198.92,-311.7 206.71,-307.39 214.69,-303.69"/>
|
||||
<polygon fill="black" stroke="black" points="216.31,-306.8 224.1,-299.62 213.53,-300.38 216.31,-306.8"/>
|
||||
</g>
|
||||
<!-- cli->distill -->
|
||||
<g id="edge33" class="edge">
|
||||
<title>cli->distill</title>
|
||||
<path fill="none" stroke="black" d="M80.72,-475.38C89.61,-553.75 125.34,-830.42 192,-881.67 332.93,-990.04 422.6,-935.59 592,-881.67 690.58,-850.3 762.9,-740.21 792.17,-688.67"/>
|
||||
<polygon fill="black" stroke="black" points="795.32,-690.2 797.13,-679.76 789.21,-686.79 795.32,-690.2"/>
|
||||
</g>
|
||||
<!-- cli->mesh -->
|
||||
<g id="edge35" class="edge">
|
||||
<title>cli->mesh</title>
|
||||
<path fill="none" stroke="black" d="M155.14,-464.37C163.89,-465.48 172.87,-466.62 181.81,-467.75"/>
|
||||
<polygon fill="black" stroke="black" points="181.51,-471.24 191.87,-469.02 182.39,-464.29 181.51,-471.24"/>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 34 KiB |
55
docs/diagrams/ingest-pipeline.dot
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
// Ingest pipeline: how a source document becomes Merkle-committed
|
||||
// content-addressed storage in a v9.8 shard.
|
||||
//
|
||||
// Render: dot -Tsvg ingest-pipeline.dot -o ingest-pipeline.svg
|
||||
|
||||
digraph ingest_pipeline {
|
||||
rankdir=TB
|
||||
node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10]
|
||||
edge [fontname="Helvetica", fontsize=9]
|
||||
bgcolor="white"
|
||||
|
||||
source [label="SOURCE\nWikipedia SQL dump · XML page · HTML URL ·\nGrok export · git/hg repo · crawled site", fillcolor="#fff7e6", shape=note]
|
||||
|
||||
iter_documents [label="Source.iter_documents()\nABC method per source kind\nyields Document(uri, raw_content, kind, ...)", fillcolor="#e7ffe7"]
|
||||
|
||||
parse [label="source-specific parse\n• wikipedia.py: SQL row → Document\n• html_page.py: selectolax → text + edges\n• vcs.py: HEAD walk + supersedes chain\n• grok.py: conversation → Document(s)", fillcolor="#e7ffe7"]
|
||||
|
||||
canonicalize [label="canonicalize(text)\nNFC + ws-collapse + strip ends\n(pinned by canonicalization_version)", fillcolor="#fff0e0"]
|
||||
|
||||
chunker [label="Chunker.chunk(text)\ntok-512-v1 default\n(pinned by chunking_version)", fillcolor="#fff0e0"]
|
||||
|
||||
leaf_hash [label="per-chunk leaf hash\nsha256(0x00 || canonical_chunk_bytes)\nleaves are content-addressed", fillcolor="#fff0e0"]
|
||||
|
||||
merkle_tree [label="MerkleTree.build(leaves)\nnon-commutative HashCombine 0x03\nodd-element rule = self-duplicate\n→ document_root (32 bytes)", fillcolor="#fff0e0"]
|
||||
|
||||
upsert [label="store.upsert_document(doc)\n• documents row keyed on document_root\n• chunks rows with leaf_hash + tier\n• merkle_nodes for proof reconstruction\n• edges per outbound link\n• audit_event chained", fillcolor="#e6f0ff"]
|
||||
|
||||
chunks_fts [label="chunks_fts INSERT\nFTS5 contentless index\nrowid = chunks.chunk_id", fillcolor="#e6f0ff"]
|
||||
|
||||
supersedes [label="supersedes edge\nif same uri but different content,\nlink new → old (lossless history)", fillcolor="#e6f0ff"]
|
||||
|
||||
audit [label="audit_events INSERT\nevent_hash = sha256(prev_hash || canonical(body))\nlinear chain — 0 breaks invariant", fillcolor="#e6f0ff"]
|
||||
|
||||
// Optional: distill cores AFTER ingest
|
||||
distill [label="distill/runner.py\n(optional, separate phase)\n• first_sentence\n• tfidf keyword core\n• per-contrib-chunk Merkle proof", fillcolor="#fff0e0", style="rounded,filled,dashed"]
|
||||
|
||||
derivations [label="derivations row\ncore_root ← src_root\nproof_blob = inclusion proofs\nfor every contributing chunk", fillcolor="#fff0e0", style="rounded,filled,dashed"]
|
||||
|
||||
end [label="SHARD READY\ndocuments + chunks + edges + merkle_nodes\n+ audit_events + (concept_relations\nback-derived later)", fillcolor="#d4edda", shape=note]
|
||||
|
||||
source -> iter_documents
|
||||
iter_documents -> parse
|
||||
parse -> canonicalize
|
||||
canonicalize -> chunker
|
||||
chunker -> leaf_hash
|
||||
leaf_hash -> merkle_tree
|
||||
merkle_tree -> upsert
|
||||
upsert -> chunks_fts
|
||||
upsert -> supersedes
|
||||
upsert -> audit
|
||||
upsert -> end
|
||||
upsert -> distill [style=dashed]
|
||||
distill -> derivations [style=dashed]
|
||||
derivations -> end [style=dashed]
|
||||
}
|
||||
BIN
docs/diagrams/ingest-pipeline.png
Normal file
|
After Width: | Height: | Size: 153 KiB |
223
docs/diagrams/ingest-pipeline.svg
Normal file
|
|
@ -0,0 +1,223 @@
|
|||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<!-- Generated by graphviz version 2.43.0 (0)
|
||||
-->
|
||||
<!-- Title: ingest_pipeline Pages: 1 -->
|
||||
<svg width="917pt" height="929pt"
|
||||
viewBox="0.00 0.00 916.50 929.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 925)">
|
||||
<title>ingest_pipeline</title>
|
||||
<polygon fill="white" stroke="transparent" points="-4,4 -4,-925 912.5,-925 912.5,4 -4,4"/>
|
||||
<!-- source -->
|
||||
<g id="node1" class="node">
|
||||
<title>source</title>
|
||||
<polygon fill="#fff7e6" stroke="black" points="599.5,-921 365.5,-921 365.5,-880 605.5,-880 605.5,-915 599.5,-921"/>
|
||||
<polyline fill="none" stroke="black" points="599.5,-921 599.5,-915 "/>
|
||||
<polyline fill="none" stroke="black" points="605.5,-915 599.5,-915 "/>
|
||||
<text text-anchor="middle" x="485.5" y="-909" font-family="Helvetica,sans-Serif" font-size="10.00">SOURCE</text>
|
||||
<text text-anchor="middle" x="485.5" y="-898" font-family="Helvetica,sans-Serif" font-size="10.00">Wikipedia SQL dump · XML page · HTML URL ·</text>
|
||||
<text text-anchor="middle" x="485.5" y="-887" font-family="Helvetica,sans-Serif" font-size="10.00">Grok export · git/hg repo · crawled site</text>
|
||||
</g>
|
||||
<!-- iter_documents -->
|
||||
<g id="node2" class="node">
|
||||
<title>iter_documents</title>
|
||||
<path fill="#e7ffe7" stroke="black" d="M587,-844C587,-844 384,-844 384,-844 378,-844 372,-838 372,-832 372,-832 372,-815 372,-815 372,-809 378,-803 384,-803 384,-803 587,-803 587,-803 593,-803 599,-809 599,-815 599,-815 599,-832 599,-832 599,-838 593,-844 587,-844"/>
|
||||
<text text-anchor="middle" x="485.5" y="-832" font-family="Helvetica,sans-Serif" font-size="10.00">Source.iter_documents()</text>
|
||||
<text text-anchor="middle" x="485.5" y="-821" font-family="Helvetica,sans-Serif" font-size="10.00">ABC method per source kind</text>
|
||||
<text text-anchor="middle" x="485.5" y="-810" font-family="Helvetica,sans-Serif" font-size="10.00">yields Document(uri, raw_content, kind, ...)</text>
|
||||
</g>
|
||||
<!-- source->iter_documents -->
|
||||
<g id="edge1" class="edge">
|
||||
<title>source->iter_documents</title>
|
||||
<path fill="none" stroke="black" d="M485.5,-879.79C485.5,-871.96 485.5,-862.77 485.5,-854.16"/>
|
||||
<polygon fill="black" stroke="black" points="489,-854.07 485.5,-844.07 482,-854.07 489,-854.07"/>
|
||||
</g>
|
||||
<!-- parse -->
|
||||
<g id="node3" class="node">
|
||||
<title>parse</title>
|
||||
<path fill="#e7ffe7" stroke="black" d="M586,-767C586,-767 385,-767 385,-767 379,-767 373,-761 373,-755 373,-755 373,-716 373,-716 373,-710 379,-704 385,-704 385,-704 586,-704 586,-704 592,-704 598,-710 598,-716 598,-716 598,-755 598,-755 598,-761 592,-767 586,-767"/>
|
||||
<text text-anchor="middle" x="485.5" y="-755" font-family="Helvetica,sans-Serif" font-size="10.00">source-specific parse</text>
|
||||
<text text-anchor="middle" x="485.5" y="-744" font-family="Helvetica,sans-Serif" font-size="10.00">• wikipedia.py: SQL row → Document</text>
|
||||
<text text-anchor="middle" x="485.5" y="-733" font-family="Helvetica,sans-Serif" font-size="10.00">• html_page.py: selectolax → text + edges</text>
|
||||
<text text-anchor="middle" x="485.5" y="-722" font-family="Helvetica,sans-Serif" font-size="10.00">• vcs.py: HEAD walk + supersedes chain</text>
|
||||
<text text-anchor="middle" x="485.5" y="-711" font-family="Helvetica,sans-Serif" font-size="10.00">• grok.py: conversation → Document(s)</text>
|
||||
</g>
|
||||
<!-- iter_documents->parse -->
|
||||
<g id="edge2" class="edge">
|
||||
<title>iter_documents->parse</title>
|
||||
<path fill="none" stroke="black" d="M485.5,-802.99C485.5,-795.28 485.5,-786.12 485.5,-777.11"/>
|
||||
<polygon fill="black" stroke="black" points="489,-777.07 485.5,-767.07 482,-777.07 489,-777.07"/>
|
||||
</g>
|
||||
<!-- canonicalize -->
|
||||
<g id="node4" class="node">
|
||||
<title>canonicalize</title>
|
||||
<path fill="#fff0e0" stroke="black" d="M571.5,-668C571.5,-668 399.5,-668 399.5,-668 393.5,-668 387.5,-662 387.5,-656 387.5,-656 387.5,-639 387.5,-639 387.5,-633 393.5,-627 399.5,-627 399.5,-627 571.5,-627 571.5,-627 577.5,-627 583.5,-633 583.5,-639 583.5,-639 583.5,-656 583.5,-656 583.5,-662 577.5,-668 571.5,-668"/>
|
||||
<text text-anchor="middle" x="485.5" y="-656" font-family="Helvetica,sans-Serif" font-size="10.00">canonicalize(text)</text>
|
||||
<text text-anchor="middle" x="485.5" y="-645" font-family="Helvetica,sans-Serif" font-size="10.00">NFC + ws-collapse + strip ends</text>
|
||||
<text text-anchor="middle" x="485.5" y="-634" font-family="Helvetica,sans-Serif" font-size="10.00">(pinned by canonicalization_version)</text>
|
||||
</g>
|
||||
<!-- parse->canonicalize -->
|
||||
<g id="edge3" class="edge">
|
||||
<title>parse->canonicalize</title>
|
||||
<path fill="none" stroke="black" d="M485.5,-703.94C485.5,-695.59 485.5,-686.55 485.5,-678.25"/>
|
||||
<polygon fill="black" stroke="black" points="489,-678.23 485.5,-668.23 482,-678.23 489,-678.23"/>
|
||||
</g>
|
||||
<!-- chunker -->
|
||||
<g id="node5" class="node">
|
||||
<title>chunker</title>
|
||||
<path fill="#fff0e0" stroke="black" d="M554,-591C554,-591 417,-591 417,-591 411,-591 405,-585 405,-579 405,-579 405,-562 405,-562 405,-556 411,-550 417,-550 417,-550 554,-550 554,-550 560,-550 566,-556 566,-562 566,-562 566,-579 566,-579 566,-585 560,-591 554,-591"/>
|
||||
<text text-anchor="middle" x="485.5" y="-579" font-family="Helvetica,sans-Serif" font-size="10.00">Chunker.chunk(text)</text>
|
||||
<text text-anchor="middle" x="485.5" y="-568" font-family="Helvetica,sans-Serif" font-size="10.00">tok-512-v1 default</text>
|
||||
<text text-anchor="middle" x="485.5" y="-557" font-family="Helvetica,sans-Serif" font-size="10.00">(pinned by chunking_version)</text>
|
||||
</g>
|
||||
<!-- canonicalize->chunker -->
|
||||
<g id="edge4" class="edge">
|
||||
<title>canonicalize->chunker</title>
|
||||
<path fill="none" stroke="black" d="M485.5,-626.79C485.5,-618.96 485.5,-609.77 485.5,-601.16"/>
|
||||
<polygon fill="black" stroke="black" points="489,-601.07 485.5,-591.07 482,-601.07 489,-601.07"/>
|
||||
</g>
|
||||
<!-- leaf_hash -->
|
||||
<g id="node6" class="node">
|
||||
<title>leaf_hash</title>
|
||||
<path fill="#fff0e0" stroke="black" d="M578,-514C578,-514 393,-514 393,-514 387,-514 381,-508 381,-502 381,-502 381,-485 381,-485 381,-479 387,-473 393,-473 393,-473 578,-473 578,-473 584,-473 590,-479 590,-485 590,-485 590,-502 590,-502 590,-508 584,-514 578,-514"/>
|
||||
<text text-anchor="middle" x="485.5" y="-502" font-family="Helvetica,sans-Serif" font-size="10.00">per-chunk leaf hash</text>
|
||||
<text text-anchor="middle" x="485.5" y="-491" font-family="Helvetica,sans-Serif" font-size="10.00">sha256(0x00 || canonical_chunk_bytes)</text>
|
||||
<text text-anchor="middle" x="485.5" y="-480" font-family="Helvetica,sans-Serif" font-size="10.00">leaves are content-addressed</text>
|
||||
</g>
|
||||
<!-- chunker->leaf_hash -->
|
||||
<g id="edge5" class="edge">
|
||||
<title>chunker->leaf_hash</title>
|
||||
<path fill="none" stroke="black" d="M485.5,-549.79C485.5,-541.96 485.5,-532.77 485.5,-524.16"/>
|
||||
<polygon fill="black" stroke="black" points="489,-524.07 485.5,-514.07 482,-524.07 489,-524.07"/>
|
||||
</g>
|
||||
<!-- merkle_tree -->
|
||||
<g id="node7" class="node">
|
||||
<title>merkle_tree</title>
|
||||
<path fill="#fff0e0" stroke="black" d="M574.5,-437C574.5,-437 396.5,-437 396.5,-437 390.5,-437 384.5,-431 384.5,-425 384.5,-425 384.5,-397 384.5,-397 384.5,-391 390.5,-385 396.5,-385 396.5,-385 574.5,-385 574.5,-385 580.5,-385 586.5,-391 586.5,-397 586.5,-397 586.5,-425 586.5,-425 586.5,-431 580.5,-437 574.5,-437"/>
|
||||
<text text-anchor="middle" x="485.5" y="-425" font-family="Helvetica,sans-Serif" font-size="10.00">MerkleTree.build(leaves)</text>
|
||||
<text text-anchor="middle" x="485.5" y="-414" font-family="Helvetica,sans-Serif" font-size="10.00">non-commutative HashCombine 0x03</text>
|
||||
<text text-anchor="middle" x="485.5" y="-403" font-family="Helvetica,sans-Serif" font-size="10.00">odd-element rule = self-duplicate</text>
|
||||
<text text-anchor="middle" x="485.5" y="-392" font-family="Helvetica,sans-Serif" font-size="10.00">→ document_root (32 bytes)</text>
|
||||
</g>
|
||||
<!-- leaf_hash->merkle_tree -->
|
||||
<g id="edge6" class="edge">
|
||||
<title>leaf_hash->merkle_tree</title>
|
||||
<path fill="none" stroke="black" d="M485.5,-472.6C485.5,-464.99 485.5,-456.06 485.5,-447.43"/>
|
||||
<polygon fill="black" stroke="black" points="489,-447.16 485.5,-437.16 482,-447.16 489,-447.16"/>
|
||||
</g>
|
||||
<!-- upsert -->
|
||||
<g id="node8" class="node">
|
||||
<title>upsert</title>
|
||||
<path fill="#e6f0ff" stroke="black" d="M585.5,-349C585.5,-349 385.5,-349 385.5,-349 379.5,-349 373.5,-343 373.5,-337 373.5,-337 373.5,-287 373.5,-287 373.5,-281 379.5,-275 385.5,-275 385.5,-275 585.5,-275 585.5,-275 591.5,-275 597.5,-281 597.5,-287 597.5,-287 597.5,-337 597.5,-337 597.5,-343 591.5,-349 585.5,-349"/>
|
||||
<text text-anchor="middle" x="485.5" y="-337" font-family="Helvetica,sans-Serif" font-size="10.00">store.upsert_document(doc)</text>
|
||||
<text text-anchor="middle" x="485.5" y="-326" font-family="Helvetica,sans-Serif" font-size="10.00">• documents row keyed on document_root</text>
|
||||
<text text-anchor="middle" x="485.5" y="-315" font-family="Helvetica,sans-Serif" font-size="10.00">• chunks rows with leaf_hash + tier</text>
|
||||
<text text-anchor="middle" x="485.5" y="-304" font-family="Helvetica,sans-Serif" font-size="10.00">• merkle_nodes for proof reconstruction</text>
|
||||
<text text-anchor="middle" x="485.5" y="-293" font-family="Helvetica,sans-Serif" font-size="10.00">• edges per outbound link</text>
|
||||
<text text-anchor="middle" x="485.5" y="-282" font-family="Helvetica,sans-Serif" font-size="10.00">• audit_event chained</text>
|
||||
</g>
|
||||
<!-- merkle_tree->upsert -->
|
||||
<g id="edge7" class="edge">
|
||||
<title>merkle_tree->upsert</title>
|
||||
<path fill="none" stroke="black" d="M485.5,-384.77C485.5,-376.94 485.5,-368.05 485.5,-359.26"/>
|
||||
<polygon fill="black" stroke="black" points="489,-359.06 485.5,-349.06 482,-359.06 489,-359.06"/>
|
||||
</g>
|
||||
<!-- chunks_fts -->
|
||||
<g id="node9" class="node">
|
||||
<title>chunks_fts</title>
|
||||
<path fill="#e6f0ff" stroke="black" d="M127,-228C127,-228 12,-228 12,-228 6,-228 0,-222 0,-216 0,-216 0,-199 0,-199 0,-193 6,-187 12,-187 12,-187 127,-187 127,-187 133,-187 139,-193 139,-199 139,-199 139,-216 139,-216 139,-222 133,-228 127,-228"/>
|
||||
<text text-anchor="middle" x="69.5" y="-216" font-family="Helvetica,sans-Serif" font-size="10.00">chunks_fts INSERT</text>
|
||||
<text text-anchor="middle" x="69.5" y="-205" font-family="Helvetica,sans-Serif" font-size="10.00">FTS5 contentless index</text>
|
||||
<text text-anchor="middle" x="69.5" y="-194" font-family="Helvetica,sans-Serif" font-size="10.00">rowid = chunks.chunk_id</text>
|
||||
</g>
|
||||
<!-- upsert->chunks_fts -->
|
||||
<g id="edge8" class="edge">
|
||||
<title>upsert->chunks_fts</title>
|
||||
<path fill="none" stroke="black" d="M373.22,-291.83C306.88,-279.34 221.64,-261.23 147.5,-239 140.41,-236.88 133.06,-234.39 125.84,-231.77"/>
|
||||
<polygon fill="black" stroke="black" points="126.68,-228.35 116.09,-228.14 124.24,-234.91 126.68,-228.35"/>
|
||||
</g>
|
||||
<!-- supersedes -->
|
||||
<g id="node10" class="node">
|
||||
<title>supersedes</title>
|
||||
<path fill="#e6f0ff" stroke="black" d="M320,-228C320,-228 169,-228 169,-228 163,-228 157,-222 157,-216 157,-216 157,-199 157,-199 157,-193 163,-187 169,-187 169,-187 320,-187 320,-187 326,-187 332,-193 332,-199 332,-199 332,-216 332,-216 332,-222 326,-228 320,-228"/>
|
||||
<text text-anchor="middle" x="244.5" y="-216" font-family="Helvetica,sans-Serif" font-size="10.00">supersedes edge</text>
|
||||
<text text-anchor="middle" x="244.5" y="-205" font-family="Helvetica,sans-Serif" font-size="10.00">if same uri but different content,</text>
|
||||
<text text-anchor="middle" x="244.5" y="-194" font-family="Helvetica,sans-Serif" font-size="10.00">link new → old (lossless history)</text>
|
||||
</g>
|
||||
<!-- upsert->supersedes -->
|
||||
<g id="edge9" class="edge">
|
||||
<title>upsert->supersedes</title>
|
||||
<path fill="none" stroke="black" d="M400.53,-274.86C367.15,-260.66 329.77,-244.77 299.97,-232.09"/>
|
||||
<polygon fill="black" stroke="black" points="301.14,-228.79 290.57,-228.09 298.4,-235.23 301.14,-228.79"/>
|
||||
</g>
|
||||
<!-- audit -->
|
||||
<g id="node11" class="node">
|
||||
<title>audit</title>
|
||||
<path fill="#e6f0ff" stroke="black" d="M608.5,-228C608.5,-228 362.5,-228 362.5,-228 356.5,-228 350.5,-222 350.5,-216 350.5,-216 350.5,-199 350.5,-199 350.5,-193 356.5,-187 362.5,-187 362.5,-187 608.5,-187 608.5,-187 614.5,-187 620.5,-193 620.5,-199 620.5,-199 620.5,-216 620.5,-216 620.5,-222 614.5,-228 608.5,-228"/>
|
||||
<text text-anchor="middle" x="485.5" y="-216" font-family="Helvetica,sans-Serif" font-size="10.00">audit_events INSERT</text>
|
||||
<text text-anchor="middle" x="485.5" y="-205" font-family="Helvetica,sans-Serif" font-size="10.00">event_hash = sha256(prev_hash || canonical(body))</text>
|
||||
<text text-anchor="middle" x="485.5" y="-194" font-family="Helvetica,sans-Serif" font-size="10.00">linear chain — 0 breaks invariant</text>
|
||||
</g>
|
||||
<!-- upsert->audit -->
|
||||
<g id="edge10" class="edge">
|
||||
<title>upsert->audit</title>
|
||||
<path fill="none" stroke="black" d="M485.5,-274.86C485.5,-262.98 485.5,-249.91 485.5,-238.53"/>
|
||||
<polygon fill="black" stroke="black" points="489,-238.24 485.5,-228.24 482,-238.24 489,-238.24"/>
|
||||
</g>
|
||||
<!-- distill -->
|
||||
<g id="node12" class="node">
|
||||
<title>distill</title>
|
||||
<path fill="#fff0e0" stroke="black" stroke-dasharray="5,2" d="M804,-239C804,-239 651,-239 651,-239 645,-239 639,-233 639,-227 639,-227 639,-188 639,-188 639,-182 645,-176 651,-176 651,-176 804,-176 804,-176 810,-176 816,-182 816,-188 816,-188 816,-227 816,-227 816,-233 810,-239 804,-239"/>
|
||||
<text text-anchor="middle" x="727.5" y="-227" font-family="Helvetica,sans-Serif" font-size="10.00">distill/runner.py</text>
|
||||
<text text-anchor="middle" x="727.5" y="-216" font-family="Helvetica,sans-Serif" font-size="10.00">(optional, separate phase)</text>
|
||||
<text text-anchor="middle" x="727.5" y="-205" font-family="Helvetica,sans-Serif" font-size="10.00">• first_sentence</text>
|
||||
<text text-anchor="middle" x="727.5" y="-194" font-family="Helvetica,sans-Serif" font-size="10.00">• tfidf keyword core</text>
|
||||
<text text-anchor="middle" x="727.5" y="-183" font-family="Helvetica,sans-Serif" font-size="10.00">• per-contrib-chunk Merkle proof</text>
|
||||
</g>
|
||||
<!-- upsert->distill -->
|
||||
<g id="edge12" class="edge">
|
||||
<title>upsert->distill</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M570.82,-274.86C595.21,-264.53 621.72,-253.3 645.88,-243.07"/>
|
||||
<polygon fill="black" stroke="black" points="647.52,-246.18 655.36,-239.05 644.79,-239.73 647.52,-246.18"/>
|
||||
</g>
|
||||
<!-- end -->
|
||||
<g id="node14" class="node">
|
||||
<title>end</title>
|
||||
<polygon fill="#d4edda" stroke="black" points="902.5,-52 662.5,-52 662.5,0 908.5,0 908.5,-46 902.5,-52"/>
|
||||
<polyline fill="none" stroke="black" points="902.5,-52 902.5,-46 "/>
|
||||
<polyline fill="none" stroke="black" points="908.5,-46 902.5,-46 "/>
|
||||
<text text-anchor="middle" x="785.5" y="-40" font-family="Helvetica,sans-Serif" font-size="10.00">SHARD READY</text>
|
||||
<text text-anchor="middle" x="785.5" y="-29" font-family="Helvetica,sans-Serif" font-size="10.00">documents + chunks + edges + merkle_nodes</text>
|
||||
<text text-anchor="middle" x="785.5" y="-18" font-family="Helvetica,sans-Serif" font-size="10.00">+ audit_events + (concept_relations</text>
|
||||
<text text-anchor="middle" x="785.5" y="-7" font-family="Helvetica,sans-Serif" font-size="10.00">back-derived later)</text>
|
||||
</g>
|
||||
<!-- upsert->end -->
|
||||
<g id="edge11" class="edge">
|
||||
<title>upsert->end</title>
|
||||
<path fill="none" stroke="black" d="M597.56,-302.18C684.97,-292.7 796.15,-274.01 824.5,-239 832.09,-229.63 824.51,-101.79 820.5,-88 817.82,-78.79 813.49,-69.48 808.77,-61.01"/>
|
||||
<polygon fill="black" stroke="black" points="811.66,-59.02 803.56,-52.19 805.63,-62.58 811.66,-59.02"/>
|
||||
</g>
|
||||
<!-- derivations -->
|
||||
<g id="node13" class="node">
|
||||
<title>derivations</title>
|
||||
<path fill="#fff0e0" stroke="black" stroke-dasharray="5,2" d="M799.5,-140C799.5,-140 663.5,-140 663.5,-140 657.5,-140 651.5,-134 651.5,-128 651.5,-128 651.5,-100 651.5,-100 651.5,-94 657.5,-88 663.5,-88 663.5,-88 799.5,-88 799.5,-88 805.5,-88 811.5,-94 811.5,-100 811.5,-100 811.5,-128 811.5,-128 811.5,-134 805.5,-140 799.5,-140"/>
|
||||
<text text-anchor="middle" x="731.5" y="-128" font-family="Helvetica,sans-Serif" font-size="10.00">derivations row</text>
|
||||
<text text-anchor="middle" x="731.5" y="-117" font-family="Helvetica,sans-Serif" font-size="10.00">core_root ← src_root</text>
|
||||
<text text-anchor="middle" x="731.5" y="-106" font-family="Helvetica,sans-Serif" font-size="10.00">proof_blob = inclusion proofs</text>
|
||||
<text text-anchor="middle" x="731.5" y="-95" font-family="Helvetica,sans-Serif" font-size="10.00">for every contributing chunk</text>
|
||||
</g>
|
||||
<!-- distill->derivations -->
|
||||
<g id="edge13" class="edge">
|
||||
<title>distill->derivations</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M728.84,-175.78C729.2,-167.51 729.6,-158.51 729.97,-150.02"/>
|
||||
<polygon fill="black" stroke="black" points="733.47,-150.16 730.41,-140.02 726.47,-149.86 733.47,-150.16"/>
|
||||
</g>
|
||||
<!-- derivations->end -->
|
||||
<g id="edge14" class="edge">
|
||||
<title>derivations->end</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M747.41,-87.66C752.68,-79.26 758.64,-69.78 764.25,-60.84"/>
|
||||
<polygon fill="black" stroke="black" points="767.35,-62.49 769.7,-52.16 761.42,-58.76 767.35,-62.49"/>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 17 KiB |
224
docs/diagrams/mesh-data-flow.svg
Normal file
|
|
@ -0,0 +1,224 @@
|
|||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<!-- Generated by graphviz version 2.43.0 (0)
|
||||
-->
|
||||
<!-- Title: mesh_data_flow Pages: 1 -->
|
||||
<svg width="912pt" height="509pt"
|
||||
viewBox="0.00 0.00 912.00 509.38" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 505.38)">
|
||||
<title>mesh_data_flow</title>
|
||||
<polygon fill="white" stroke="transparent" points="-4,4 -4,-505.38 908,-505.38 908,4 -4,4"/>
|
||||
<g id="clust1" class="cluster">
|
||||
<title>cluster_alice</title>
|
||||
<path fill="#fff7d6" stroke="black" d="M20,-92.38C20,-92.38 245,-92.38 245,-92.38 251,-92.38 257,-98.38 257,-104.38 257,-104.38 257,-381.38 257,-381.38 257,-387.38 251,-393.38 245,-393.38 245,-393.38 20,-393.38 20,-393.38 14,-393.38 8,-387.38 8,-381.38 8,-381.38 8,-104.38 8,-104.38 8,-98.38 14,-92.38 20,-92.38"/>
|
||||
<text text-anchor="middle" x="132.5" y="-378.18" font-family="Times,serif" font-size="14.00">peer: alice (admin)</text>
|
||||
</g>
|
||||
<g id="clust2" class="cluster">
|
||||
<title>cluster_bob</title>
|
||||
<path fill="#d6ecff" stroke="black" d="M751,-100.38C751,-100.38 885,-100.38 885,-100.38 891,-100.38 897,-106.38 897,-112.38 897,-112.38 897,-381.38 897,-381.38 897,-387.38 891,-393.38 885,-393.38 885,-393.38 751,-393.38 751,-393.38 745,-393.38 739,-387.38 739,-381.38 739,-381.38 739,-112.38 739,-112.38 739,-106.38 745,-100.38 751,-100.38"/>
|
||||
<text text-anchor="middle" x="818" y="-378.18" font-family="Times,serif" font-size="14.00">peer: bob (member)</text>
|
||||
</g>
|
||||
<g id="clust3" class="cluster">
|
||||
<title>cluster_wire</title>
|
||||
<path fill="white" stroke="#666666" stroke-dasharray="5,2" d="M290,-59.38C290,-59.38 646,-59.38 646,-59.38 652,-59.38 658,-65.38 658,-71.38 658,-71.38 658,-481.38 658,-481.38 658,-487.38 652,-493.38 646,-493.38 646,-493.38 290,-493.38 290,-493.38 284,-493.38 278,-487.38 278,-481.38 278,-481.38 278,-71.38 278,-71.38 278,-65.38 284,-59.38 290,-59.38"/>
|
||||
<text text-anchor="middle" x="468" y="-478.18" font-family="Times,serif" font-size="14.00">gossip messages (Ed25519-signed, optional AEAD)</text>
|
||||
</g>
|
||||
<!-- a_docs -->
|
||||
<g id="node1" class="node">
|
||||
<title>a_docs</title>
|
||||
<path fill="#ffffff" stroke="black" d="M185,-362.38C185,-362.38 80,-362.38 80,-362.38 74,-362.38 68,-356.38 68,-350.38 68,-350.38 68,-336.38 68,-336.38 68,-330.38 74,-324.38 80,-324.38 80,-324.38 185,-324.38 185,-324.38 191,-324.38 197,-330.38 197,-336.38 197,-336.38 197,-350.38 197,-350.38 197,-356.38 191,-362.38 185,-362.38"/>
|
||||
<text text-anchor="middle" x="132.5" y="-347.18" font-family="Helvetica,sans-Serif" font-size="14.00">documents</text>
|
||||
<text text-anchor="middle" x="132.5" y="-332.18" font-family="Helvetica,sans-Serif" font-size="14.00">(surface + core)</text>
|
||||
</g>
|
||||
<!-- m_root -->
|
||||
<g id="node11" class="node">
|
||||
<title>m_root</title>
|
||||
<path fill="#e8ffe8" stroke="black" d="M586,-461.88C586,-461.88 350,-461.88 350,-461.88 344,-461.88 338,-455.88 338,-449.88 338,-449.88 338,-420.88 338,-420.88 338,-414.88 344,-408.88 350,-408.88 350,-408.88 586,-408.88 586,-408.88 592,-408.88 598,-414.88 598,-420.88 598,-420.88 598,-449.88 598,-449.88 598,-455.88 592,-461.88 586,-461.88"/>
|
||||
<text text-anchor="middle" x="468" y="-446.68" font-family="Helvetica,sans-Serif" font-size="14.00">ANNOUNCE_ROOT</text>
|
||||
<text text-anchor="middle" x="468" y="-431.68" font-family="Helvetica,sans-Serif" font-size="14.00">document_root, source_uri,</text>
|
||||
<text text-anchor="middle" x="468" y="-416.68" font-family="Helvetica,sans-Serif" font-size="14.00">chunking_version, schema_version</text>
|
||||
</g>
|
||||
<!-- a_docs->m_root -->
|
||||
<g id="edge1" class="edge">
|
||||
<title>a_docs->m_root</title>
|
||||
<path fill="none" stroke="black" d="M175.43,-362.41C201.33,-373.59 235.74,-387.46 268.39,-397.52"/>
|
||||
<polygon fill="black" stroke="black" points="267.42,-400.88 278,-400.38 269.41,-394.17 267.42,-400.88"/>
|
||||
</g>
|
||||
<!-- m_resp -->
|
||||
<g id="node16" class="node">
|
||||
<title>m_resp</title>
|
||||
<path fill="#fff7d6" stroke="black" d="M579.5,-390.88C579.5,-390.88 356.5,-390.88 356.5,-390.88 350.5,-390.88 344.5,-384.88 344.5,-378.88 344.5,-378.88 344.5,-349.88 344.5,-349.88 344.5,-343.88 350.5,-337.88 356.5,-337.88 356.5,-337.88 579.5,-337.88 579.5,-337.88 585.5,-337.88 591.5,-343.88 591.5,-349.88 591.5,-349.88 591.5,-378.88 591.5,-378.88 591.5,-384.88 585.5,-390.88 579.5,-390.88"/>
|
||||
<text text-anchor="middle" x="468" y="-375.68" font-family="Helvetica,sans-Serif" font-size="14.00">DELIVER_BODY</text>
|
||||
<text text-anchor="middle" x="468" y="-360.68" font-family="Helvetica,sans-Serif" font-size="14.00">bytes + Merkle proof of inclusion</text>
|
||||
<text text-anchor="middle" x="468" y="-345.68" font-family="Helvetica,sans-Serif" font-size="14.00">recipient verifies before insert</text>
|
||||
</g>
|
||||
<!-- a_docs->m_resp -->
|
||||
<g id="edge11" class="edge">
|
||||
<title>a_docs->m_resp</title>
|
||||
<path fill="none" stroke="#cc6600" d="M197.01,-347.38C235.83,-349.82 286.95,-353.04 334.26,-356.02"/>
|
||||
<polygon fill="#cc6600" stroke="#cc6600" points="334.09,-359.52 344.29,-356.65 334.53,-352.53 334.09,-359.52"/>
|
||||
</g>
|
||||
<!-- a_deriv -->
|
||||
<g id="node2" class="node">
|
||||
<title>a_deriv</title>
|
||||
<path fill="#ffffff" stroke="black" d="M226.5,-306.38C226.5,-306.38 38.5,-306.38 38.5,-306.38 32.5,-306.38 26.5,-300.38 26.5,-294.38 26.5,-294.38 26.5,-280.38 26.5,-280.38 26.5,-274.38 32.5,-268.38 38.5,-268.38 38.5,-268.38 226.5,-268.38 226.5,-268.38 232.5,-268.38 238.5,-274.38 238.5,-280.38 238.5,-280.38 238.5,-294.38 238.5,-294.38 238.5,-300.38 232.5,-306.38 226.5,-306.38"/>
|
||||
<text text-anchor="middle" x="132.5" y="-291.18" font-family="Helvetica,sans-Serif" font-size="14.00">derivations</text>
|
||||
<text text-anchor="middle" x="132.5" y="-276.18" font-family="Helvetica,sans-Serif" font-size="14.00">(surface -> core proof_blob)</text>
|
||||
</g>
|
||||
<!-- m_der -->
|
||||
<g id="node12" class="node">
|
||||
<title>m_der</title>
|
||||
<path fill="#e8ffe8" stroke="black" d="M563,-319.88C563,-319.88 373,-319.88 373,-319.88 367,-319.88 361,-313.88 361,-307.88 361,-307.88 361,-278.88 361,-278.88 361,-272.88 367,-266.88 373,-266.88 373,-266.88 563,-266.88 563,-266.88 569,-266.88 575,-272.88 575,-278.88 575,-278.88 575,-307.88 575,-307.88 575,-313.88 569,-319.88 563,-319.88"/>
|
||||
<text text-anchor="middle" x="468" y="-304.68" font-family="Helvetica,sans-Serif" font-size="14.00">ANNOUNCE_DERIVATION</text>
|
||||
<text text-anchor="middle" x="468" y="-289.68" font-family="Helvetica,sans-Serif" font-size="14.00">core_root <- surface_root(s),</text>
|
||||
<text text-anchor="middle" x="468" y="-274.68" font-family="Helvetica,sans-Serif" font-size="14.00">distiller_id, proof_blob hash</text>
|
||||
</g>
|
||||
<!-- a_deriv->m_der -->
|
||||
<g id="edge2" class="edge">
|
||||
<title>a_deriv->m_der</title>
|
||||
<path fill="none" stroke="black" d="M238.56,-289.27C248.1,-289.44 257.94,-289.62 267.95,-289.8"/>
|
||||
<polygon fill="black" stroke="black" points="267.94,-293.3 278,-289.98 268.07,-286.3 267.94,-293.3"/>
|
||||
</g>
|
||||
<!-- a_prov -->
|
||||
<g id="node3" class="node">
|
||||
<title>a_prov</title>
|
||||
<path fill="#ffffff" stroke="black" d="M237,-250.38C237,-250.38 28,-250.38 28,-250.38 22,-250.38 16,-244.38 16,-238.38 16,-238.38 16,-224.38 16,-224.38 16,-218.38 22,-212.38 28,-212.38 28,-212.38 237,-212.38 237,-212.38 243,-212.38 249,-218.38 249,-224.38 249,-224.38 249,-238.38 249,-238.38 249,-244.38 243,-250.38 237,-250.38"/>
|
||||
<text text-anchor="middle" x="132.5" y="-235.18" font-family="Helvetica,sans-Serif" font-size="14.00">providence_cache</text>
|
||||
<text text-anchor="middle" x="132.5" y="-220.18" font-family="Helvetica,sans-Serif" font-size="14.00">(8-dim cache_key, audit_mode)</text>
|
||||
</g>
|
||||
<!-- m_prov -->
|
||||
<g id="node13" class="node">
|
||||
<title>m_prov</title>
|
||||
<path fill="#e8ffe8" stroke="black" d="M554.5,-248.88C554.5,-248.88 381.5,-248.88 381.5,-248.88 375.5,-248.88 369.5,-242.88 369.5,-236.88 369.5,-236.88 369.5,-207.88 369.5,-207.88 369.5,-201.88 375.5,-195.88 381.5,-195.88 381.5,-195.88 554.5,-195.88 554.5,-195.88 560.5,-195.88 566.5,-201.88 566.5,-207.88 566.5,-207.88 566.5,-236.88 566.5,-236.88 566.5,-242.88 560.5,-248.88 554.5,-248.88"/>
|
||||
<text text-anchor="middle" x="468" y="-233.68" font-family="Helvetica,sans-Serif" font-size="14.00">ANNOUNCE_PROVIDENCE</text>
|
||||
<text text-anchor="middle" x="468" y="-218.68" font-family="Helvetica,sans-Serif" font-size="14.00">cache_key, audit_mode,</text>
|
||||
<text text-anchor="middle" x="468" y="-203.68" font-family="Helvetica,sans-Serif" font-size="14.00">answer_hash</text>
|
||||
</g>
|
||||
<!-- a_prov->m_prov -->
|
||||
<g id="edge3" class="edge">
|
||||
<title>a_prov->m_prov</title>
|
||||
<path fill="none" stroke="black" d="M249.21,-228.26C255.38,-228.09 261.65,-227.92 267.99,-227.75"/>
|
||||
<polygon fill="black" stroke="black" points="268.1,-231.25 278,-227.48 267.91,-224.25 268.1,-231.25"/>
|
||||
</g>
|
||||
<!-- a_falsif -->
|
||||
<g id="node4" class="node">
|
||||
<title>a_falsif</title>
|
||||
<path fill="#ffffff" stroke="black" d="M209.5,-194.38C209.5,-194.38 55.5,-194.38 55.5,-194.38 49.5,-194.38 43.5,-188.38 43.5,-182.38 43.5,-182.38 43.5,-168.38 43.5,-168.38 43.5,-162.38 49.5,-156.38 55.5,-156.38 55.5,-156.38 209.5,-156.38 209.5,-156.38 215.5,-156.38 221.5,-162.38 221.5,-168.38 221.5,-168.38 221.5,-182.38 221.5,-182.38 221.5,-188.38 215.5,-194.38 209.5,-194.38"/>
|
||||
<text text-anchor="middle" x="132.5" y="-179.18" font-family="Helvetica,sans-Serif" font-size="14.00">falsifications</text>
|
||||
<text text-anchor="middle" x="132.5" y="-164.18" font-family="Helvetica,sans-Serif" font-size="14.00">('this answer is wrong')</text>
|
||||
</g>
|
||||
<!-- m_fals -->
|
||||
<g id="node14" class="node">
|
||||
<title>m_fals</title>
|
||||
<path fill="#ffe0e0" stroke="black" d="M595,-177.38C595,-177.38 341,-177.38 341,-177.38 335,-177.38 329,-171.38 329,-165.38 329,-165.38 329,-151.38 329,-151.38 329,-145.38 335,-139.38 341,-139.38 341,-139.38 595,-139.38 595,-139.38 601,-139.38 607,-145.38 607,-151.38 607,-151.38 607,-165.38 607,-165.38 607,-171.38 601,-177.38 595,-177.38"/>
|
||||
<text text-anchor="middle" x="468" y="-162.18" font-family="Helvetica,sans-Serif" font-size="14.00">ANNOUNCE_FALSIFICATION</text>
|
||||
<text text-anchor="middle" x="468" y="-147.18" font-family="Helvetica,sans-Serif" font-size="14.00">cache_key, reason, signed by witness</text>
|
||||
</g>
|
||||
<!-- a_falsif->m_fals -->
|
||||
<g id="edge4" class="edge">
|
||||
<title>a_falsif->m_fals</title>
|
||||
<path fill="none" stroke="black" d="M221.56,-170.89C236.18,-170.15 251.85,-169.35 267.94,-168.53"/>
|
||||
<polygon fill="black" stroke="black" points="268.2,-172.02 278,-168.01 267.84,-165.03 268.2,-172.02"/>
|
||||
</g>
|
||||
<!-- a_audit -->
|
||||
<g id="node5" class="node">
|
||||
<title>a_audit</title>
|
||||
<path fill="#ffffff" stroke="black" d="M198.5,-138.38C198.5,-138.38 66.5,-138.38 66.5,-138.38 60.5,-138.38 54.5,-132.38 54.5,-126.38 54.5,-126.38 54.5,-112.38 54.5,-112.38 54.5,-106.38 60.5,-100.38 66.5,-100.38 66.5,-100.38 198.5,-100.38 198.5,-100.38 204.5,-100.38 210.5,-106.38 210.5,-112.38 210.5,-112.38 210.5,-126.38 210.5,-126.38 210.5,-132.38 204.5,-138.38 198.5,-138.38"/>
|
||||
<text text-anchor="middle" x="132.5" y="-123.18" font-family="Helvetica,sans-Serif" font-size="14.00">audit_events</text>
|
||||
<text text-anchor="middle" x="132.5" y="-108.18" font-family="Helvetica,sans-Serif" font-size="14.00">(append-only chain)</text>
|
||||
</g>
|
||||
<!-- b_audit -->
|
||||
<g id="node10" class="node">
|
||||
<title>b_audit</title>
|
||||
<path fill="#ffffff" stroke="black" d="M858.5,-144.38C858.5,-144.38 777.5,-144.38 777.5,-144.38 771.5,-144.38 765.5,-138.38 765.5,-132.38 765.5,-132.38 765.5,-120.38 765.5,-120.38 765.5,-114.38 771.5,-108.38 777.5,-108.38 777.5,-108.38 858.5,-108.38 858.5,-108.38 864.5,-108.38 870.5,-114.38 870.5,-120.38 870.5,-120.38 870.5,-132.38 870.5,-132.38 870.5,-138.38 864.5,-144.38 858.5,-144.38"/>
|
||||
<text text-anchor="middle" x="818" y="-122.68" font-family="Helvetica,sans-Serif" font-size="14.00">audit_events</text>
|
||||
</g>
|
||||
<!-- a_audit->b_audit -->
|
||||
<g id="edge13" class="edge">
|
||||
<title>a_audit->b_audit</title>
|
||||
<path fill="none" stroke="#0066cc" stroke-dasharray="5,2" d="M156.71,-100.11C183.82,-78.97 230.99,-46.31 278,-33.38 440.84,11.41 494.99,10.79 658,-33.38 707.09,-46.68 756.56,-79.35 786.98,-102.19"/>
|
||||
<polygon fill="#0066cc" stroke="#0066cc" points="784.97,-105.06 795.04,-108.35 789.22,-99.5 784.97,-105.06"/>
|
||||
<text text-anchor="middle" x="468" y="-47.38" font-family="Helvetica,sans-Serif" font-size="10.00">audit chain merge</text>
|
||||
<text text-anchor="middle" x="468" y="-36.38" font-family="Helvetica,sans-Serif" font-size="10.00">(prev_event_hash gates inserts)</text>
|
||||
</g>
|
||||
<!-- b_docs -->
|
||||
<g id="node6" class="node">
|
||||
<title>b_docs</title>
|
||||
<path fill="#ffffff" stroke="black" d="M870.5,-362.38C870.5,-362.38 765.5,-362.38 765.5,-362.38 759.5,-362.38 753.5,-356.38 753.5,-350.38 753.5,-350.38 753.5,-336.38 753.5,-336.38 753.5,-330.38 759.5,-324.38 765.5,-324.38 765.5,-324.38 870.5,-324.38 870.5,-324.38 876.5,-324.38 882.5,-330.38 882.5,-336.38 882.5,-336.38 882.5,-350.38 882.5,-350.38 882.5,-356.38 876.5,-362.38 870.5,-362.38"/>
|
||||
<text text-anchor="middle" x="818" y="-347.18" font-family="Helvetica,sans-Serif" font-size="14.00">documents</text>
|
||||
<text text-anchor="middle" x="818" y="-332.18" font-family="Helvetica,sans-Serif" font-size="14.00">(surface + core)</text>
|
||||
</g>
|
||||
<!-- m_pull -->
|
||||
<g id="node15" class="node">
|
||||
<title>m_pull</title>
|
||||
<path fill="#fff7d6" stroke="black" d="M566,-120.88C566,-120.88 370,-120.88 370,-120.88 364,-120.88 358,-114.88 358,-108.88 358,-108.88 358,-79.88 358,-79.88 358,-73.88 364,-67.88 370,-67.88 370,-67.88 566,-67.88 566,-67.88 572,-67.88 578,-73.88 578,-79.88 578,-79.88 578,-108.88 578,-108.88 578,-114.88 572,-120.88 566,-120.88"/>
|
||||
<text text-anchor="middle" x="468" y="-105.68" font-family="Helvetica,sans-Serif" font-size="14.00">REQUEST_BODY</text>
|
||||
<text text-anchor="middle" x="468" y="-90.68" font-family="Helvetica,sans-Serif" font-size="14.00">document_root or chunk_root</text>
|
||||
<text text-anchor="middle" x="468" y="-75.68" font-family="Helvetica,sans-Serif" font-size="14.00">(only on local miss)</text>
|
||||
</g>
|
||||
<!-- b_docs->m_pull -->
|
||||
<g id="edge9" class="edge">
|
||||
<title>b_docs->m_pull</title>
|
||||
<path fill="none" stroke="#cc6600" d="M753.48,-325.83C748.24,-322.86 743.31,-319.41 739,-315.38 673.41,-254.11 728.18,-186.34 658,-130.38 638.06,-114.48 613.19,-104.86 588.12,-99.19"/>
|
||||
<polygon fill="#cc6600" stroke="#cc6600" points="588.64,-95.72 578.14,-97.14 587.23,-102.58 588.64,-95.72"/>
|
||||
<text text-anchor="middle" x="698.5" y="-306.38" font-family="Helvetica,sans-Serif" font-size="10.00">miss on root</text>
|
||||
</g>
|
||||
<!-- b_deriv -->
|
||||
<g id="node7" class="node">
|
||||
<title>b_deriv</title>
|
||||
<path fill="#ffffff" stroke="black" d="M853,-306.38C853,-306.38 783,-306.38 783,-306.38 777,-306.38 771,-300.38 771,-294.38 771,-294.38 771,-282.38 771,-282.38 771,-276.38 777,-270.38 783,-270.38 783,-270.38 853,-270.38 853,-270.38 859,-270.38 865,-276.38 865,-282.38 865,-282.38 865,-294.38 865,-294.38 865,-300.38 859,-306.38 853,-306.38"/>
|
||||
<text text-anchor="middle" x="818" y="-284.68" font-family="Helvetica,sans-Serif" font-size="14.00">derivations</text>
|
||||
</g>
|
||||
<!-- b_prov -->
|
||||
<g id="node8" class="node">
|
||||
<title>b_prov</title>
|
||||
<path fill="#ffffff" stroke="black" d="M876,-252.38C876,-252.38 760,-252.38 760,-252.38 754,-252.38 748,-246.38 748,-240.38 748,-240.38 748,-228.38 748,-228.38 748,-222.38 754,-216.38 760,-216.38 760,-216.38 876,-216.38 876,-216.38 882,-216.38 888,-222.38 888,-228.38 888,-228.38 888,-240.38 888,-240.38 888,-246.38 882,-252.38 876,-252.38"/>
|
||||
<text text-anchor="middle" x="818" y="-230.68" font-family="Helvetica,sans-Serif" font-size="14.00">providence_cache</text>
|
||||
</g>
|
||||
<!-- b_falsif -->
|
||||
<g id="node9" class="node">
|
||||
<title>b_falsif</title>
|
||||
<path fill="#ffffff" stroke="black" d="M858,-198.38C858,-198.38 778,-198.38 778,-198.38 772,-198.38 766,-192.38 766,-186.38 766,-186.38 766,-174.38 766,-174.38 766,-168.38 772,-162.38 778,-162.38 778,-162.38 858,-162.38 858,-162.38 864,-162.38 870,-168.38 870,-174.38 870,-174.38 870,-186.38 870,-186.38 870,-192.38 864,-198.38 858,-198.38"/>
|
||||
<text text-anchor="middle" x="818" y="-176.68" font-family="Helvetica,sans-Serif" font-size="14.00">falsifications</text>
|
||||
</g>
|
||||
<!-- m_root->b_docs -->
|
||||
<g id="edge5" class="edge">
|
||||
<title>m_root->b_docs</title>
|
||||
<path fill="none" stroke="black" d="M658,-400.38C690.15,-392.23 697.74,-388.47 729,-377.38 738.94,-373.85 749.45,-369.95 759.62,-366.08"/>
|
||||
<polygon fill="black" stroke="black" points="761.14,-369.24 769.22,-362.39 758.63,-362.71 761.14,-369.24"/>
|
||||
</g>
|
||||
<!-- m_der->b_deriv -->
|
||||
<g id="edge6" class="edge">
|
||||
<title>m_der->b_deriv</title>
|
||||
<path fill="none" stroke="black" d="M658,-290.67C695.41,-290.13 731.9,-289.6 760.46,-289.19"/>
|
||||
<polygon fill="black" stroke="black" points="760.79,-292.69 770.74,-289.05 760.69,-285.69 760.79,-292.69"/>
|
||||
</g>
|
||||
<!-- m_prov->b_prov -->
|
||||
<g id="edge7" class="edge">
|
||||
<title>m_prov->b_prov</title>
|
||||
<path fill="none" stroke="black" d="M658,-228.9C686.08,-229.87 713.64,-230.82 737.71,-231.65"/>
|
||||
<polygon fill="black" stroke="black" points="737.85,-235.15 747.97,-232 738.09,-228.16 737.85,-235.15"/>
|
||||
</g>
|
||||
<!-- m_fals->b_falsif -->
|
||||
<g id="edge8" class="edge">
|
||||
<title>m_fals->b_falsif</title>
|
||||
<path fill="none" stroke="black" d="M658,-170.33C693.43,-172.57 728.03,-174.76 755.85,-176.52"/>
|
||||
<polygon fill="black" stroke="black" points="755.79,-180.02 765.99,-177.16 756.23,-173.03 755.79,-180.02"/>
|
||||
</g>
|
||||
<!-- m_pull->a_docs -->
|
||||
<g id="edge10" class="edge">
|
||||
<title>m_pull->a_docs</title>
|
||||
<path fill="none" stroke="#cc6600" d="M357.93,-92.28C328.2,-97.22 298.56,-108.22 278,-130.38 221.73,-191.05 312.21,-253.74 257,-315.38 244.26,-329.61 225.94,-337.4 207.29,-341.5"/>
|
||||
<polygon fill="#cc6600" stroke="#cc6600" points="206.44,-338.09 197.24,-343.36 207.71,-344.98 206.44,-338.09"/>
|
||||
</g>
|
||||
<!-- m_resp->b_docs -->
|
||||
<g id="edge12" class="edge">
|
||||
<title>m_resp->b_docs</title>
|
||||
<path fill="none" stroke="#cc6600" d="M591.69,-356.98C642.34,-353.92 699.36,-350.48 743.37,-347.82"/>
|
||||
<polygon fill="#cc6600" stroke="#cc6600" points="743.64,-351.31 753.41,-347.22 743.21,-344.33 743.64,-351.31"/>
|
||||
<text text-anchor="middle" x="698.5" y="-365.38" font-family="Helvetica,sans-Serif" font-size="10.00">verify proof,</text>
|
||||
<text text-anchor="middle" x="698.5" y="-354.38" font-family="Helvetica,sans-Serif" font-size="10.00">then ingest</text>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 18 KiB |
126
docs/diagrams/mesh-epoch-lifecycle.svg
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<!-- Generated by graphviz version 2.43.0 (0)
|
||||
-->
|
||||
<!-- Title: mesh_epoch_lifecycle Pages: 1 -->
|
||||
<svg width="1347pt" height="296pt"
|
||||
viewBox="0.00 0.00 1347.00 295.50" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 291.5)">
|
||||
<title>mesh_epoch_lifecycle</title>
|
||||
<polygon fill="white" stroke="transparent" points="-4,4 -4,-291.5 1343,-291.5 1343,4 -4,4"/>
|
||||
<g id="clust1" class="cluster">
|
||||
<title>cluster_ops</title>
|
||||
<path fill="white" stroke="#666666" stroke-dasharray="5,2" d="M22,-59.5C22,-59.5 173,-59.5 173,-59.5 179,-59.5 185,-65.5 185,-71.5 185,-71.5 185,-267.5 185,-267.5 185,-273.5 179,-279.5 173,-279.5 173,-279.5 22,-279.5 22,-279.5 16,-279.5 10,-273.5 10,-267.5 10,-267.5 10,-71.5 10,-71.5 10,-65.5 16,-59.5 22,-59.5"/>
|
||||
<text text-anchor="middle" x="97.5" y="-264.3" font-family="Times,serif" font-size="14.00">Operator commands</text>
|
||||
</g>
|
||||
<!-- none -->
|
||||
<g id="node1" class="node">
|
||||
<title>none</title>
|
||||
<path fill="#eeeeee" stroke="black" d="M183,-49.5C183,-49.5 12,-49.5 12,-49.5 6,-49.5 0,-43.5 0,-37.5 0,-37.5 0,-23.5 0,-23.5 0,-17.5 6,-11.5 12,-11.5 12,-11.5 183,-11.5 183,-11.5 189,-11.5 195,-17.5 195,-23.5 195,-23.5 195,-37.5 195,-37.5 195,-43.5 189,-49.5 183,-49.5"/>
|
||||
<text text-anchor="middle" x="97.5" y="-34.3" font-family="Helvetica,sans-Serif" font-size="14.00">no mesh state</text>
|
||||
<text text-anchor="middle" x="97.5" y="-19.3" font-family="Helvetica,sans-Serif" font-size="14.00">(meta.mesh.enabled = 0)</text>
|
||||
</g>
|
||||
<!-- e0 -->
|
||||
<g id="node2" class="node">
|
||||
<title>e0</title>
|
||||
<path fill="#fff7d6" stroke="black" d="M556,-57C556,-57 412,-57 412,-57 406,-57 400,-51 400,-45 400,-45 400,-16 400,-16 400,-10 406,-4 412,-4 412,-4 556,-4 556,-4 562,-4 568,-10 568,-16 568,-16 568,-45 568,-45 568,-51 562,-57 556,-57"/>
|
||||
<text text-anchor="middle" x="484" y="-41.8" font-family="Helvetica,sans-Serif" font-size="14.00">epoch 0</text>
|
||||
<text text-anchor="middle" x="484" y="-26.8" font-family="Helvetica,sans-Serif" font-size="14.00">founder = sole admin</text>
|
||||
<text text-anchor="middle" x="484" y="-11.8" font-family="Helvetica,sans-Serif" font-size="14.00">reason: 'genesis'</text>
|
||||
</g>
|
||||
<!-- none->e0 -->
|
||||
<g id="edge1" class="edge">
|
||||
<title>none->e0</title>
|
||||
<path fill="none" stroke="black" d="M195.03,-30.5C254.65,-30.5 330.75,-30.5 389.75,-30.5"/>
|
||||
<polygon fill="black" stroke="black" points="389.81,-34 399.81,-30.5 389.81,-27 389.81,-34"/>
|
||||
<text text-anchor="middle" x="297.5" y="-33.5" font-family="Helvetica,sans-Serif" font-size="10.00">aborist mesh init --group <name></text>
|
||||
</g>
|
||||
<!-- e1 -->
|
||||
<g id="node3" class="node">
|
||||
<title>e1</title>
|
||||
<path fill="#d6ffd6" stroke="black" d="M819,-57C819,-57 702,-57 702,-57 696,-57 690,-51 690,-45 690,-45 690,-16 690,-16 690,-10 696,-4 702,-4 702,-4 819,-4 819,-4 825,-4 831,-10 831,-16 831,-16 831,-45 831,-45 831,-51 825,-57 819,-57"/>
|
||||
<text text-anchor="middle" x="760.5" y="-41.8" font-family="Helvetica,sans-Serif" font-size="14.00">epoch N</text>
|
||||
<text text-anchor="middle" x="760.5" y="-26.8" font-family="Helvetica,sans-Serif" font-size="14.00">roster + envelope</text>
|
||||
<text text-anchor="middle" x="760.5" y="-11.8" font-family="Helvetica,sans-Serif" font-size="14.00">signed by actor</text>
|
||||
</g>
|
||||
<!-- e0->e1 -->
|
||||
<g id="edge2" class="edge">
|
||||
<title>e0->e1</title>
|
||||
<path fill="none" stroke="black" d="M568.1,-30.5C603.43,-30.5 644.47,-30.5 679.46,-30.5"/>
|
||||
<polygon fill="black" stroke="black" points="679.63,-34 689.63,-30.5 679.63,-27 679.63,-34"/>
|
||||
<text text-anchor="middle" x="629" y="-33.5" font-family="Helvetica,sans-Serif" font-size="10.00">add | kick | rotate</text>
|
||||
</g>
|
||||
<!-- eN -->
|
||||
<g id="node4" class="node">
|
||||
<title>eN</title>
|
||||
<path fill="#d6ffd6" stroke="black" d="M1060,-96C1060,-96 965,-96 965,-96 959,-96 953,-90 953,-84 953,-84 953,-55 953,-55 953,-49 959,-43 965,-43 965,-43 1060,-43 1060,-43 1066,-43 1072,-49 1072,-55 1072,-55 1072,-84 1072,-84 1072,-90 1066,-96 1060,-96"/>
|
||||
<text text-anchor="middle" x="1012.5" y="-80.8" font-family="Helvetica,sans-Serif" font-size="14.00">epoch N+1</text>
|
||||
<text text-anchor="middle" x="1012.5" y="-65.8" font-family="Helvetica,sans-Serif" font-size="14.00">fresh secret</text>
|
||||
<text text-anchor="middle" x="1012.5" y="-50.8" font-family="Helvetica,sans-Serif" font-size="14.00">fresh envelope</text>
|
||||
</g>
|
||||
<!-- e1->eN -->
|
||||
<g id="edge3" class="edge">
|
||||
<title>e1->eN</title>
|
||||
<path fill="none" stroke="black" d="M831.14,-41.36C865.99,-46.8 908.03,-53.36 942.75,-58.77"/>
|
||||
<polygon fill="black" stroke="black" points="942.38,-62.26 952.8,-60.34 943.46,-55.34 942.38,-62.26"/>
|
||||
<text text-anchor="middle" x="892" y="-59.5" font-family="Helvetica,sans-Serif" font-size="10.00">add | kick | rotate</text>
|
||||
</g>
|
||||
<!-- historic -->
|
||||
<g id="node5" class="node">
|
||||
<title>historic</title>
|
||||
<polygon fill="#ffffff" stroke="black" points="1333,-53 1109,-53 1109,0 1339,0 1339,-47 1333,-53"/>
|
||||
<polyline fill="none" stroke="black" points="1333,-53 1333,-47 "/>
|
||||
<polyline fill="none" stroke="black" points="1339,-47 1333,-47 "/>
|
||||
<text text-anchor="middle" x="1224" y="-37.8" font-family="Helvetica,sans-Serif" font-size="14.00">every prior epoch</text>
|
||||
<text text-anchor="middle" x="1224" y="-22.8" font-family="Helvetica,sans-Serif" font-size="14.00">stays on disk</text>
|
||||
<text text-anchor="middle" x="1224" y="-7.8" font-family="Helvetica,sans-Serif" font-size="14.00">(roster + signatures verifiable)</text>
|
||||
</g>
|
||||
<!-- e1->historic -->
|
||||
<g id="edge5" class="edge">
|
||||
<title>e1->historic</title>
|
||||
<path fill="none" stroke="#888888" stroke-dasharray="5,2" d="M831.09,-29.9C904.62,-29.26 1021.75,-28.24 1108.83,-27.49"/>
|
||||
</g>
|
||||
<!-- eN->eN -->
|
||||
<g id="edge4" class="edge">
|
||||
<title>eN->eN</title>
|
||||
<path fill="none" stroke="black" d="M997.89,-96.44C997.58,-106.11 1002.45,-114 1012.5,-114 1018.63,-114 1022.83,-111.07 1025.11,-106.59"/>
|
||||
<polygon fill="black" stroke="black" points="1028.61,-106.93 1027.11,-96.44 1021.74,-105.57 1028.61,-106.93"/>
|
||||
<text text-anchor="middle" x="1012.5" y="-117" font-family="Helvetica,sans-Serif" font-size="10.00">rotate (no roster change)</text>
|
||||
</g>
|
||||
<!-- eN->historic -->
|
||||
<g id="edge6" class="edge">
|
||||
<title>eN->historic</title>
|
||||
<path fill="none" stroke="#888888" stroke-dasharray="5,2" d="M1072.14,-57.46C1083.64,-55.1 1096.06,-52.55 1108.67,-49.97"/>
|
||||
</g>
|
||||
<!-- op_add -->
|
||||
<g id="node6" class="node">
|
||||
<title>op_add</title>
|
||||
<polygon fill="#e8f0ff" stroke="black" points="167,-248 22,-248 22,-195 173,-195 173,-242 167,-248"/>
|
||||
<polyline fill="none" stroke="black" points="167,-248 167,-242 "/>
|
||||
<polyline fill="none" stroke="black" points="173,-242 167,-242 "/>
|
||||
<text text-anchor="middle" x="97.5" y="-232.8" font-family="Helvetica,sans-Serif" font-size="14.00">aborist mesh add</text>
|
||||
<text text-anchor="middle" x="97.5" y="-217.8" font-family="Helvetica,sans-Serif" font-size="14.00">--member-id bob ...</text>
|
||||
<text text-anchor="middle" x="97.5" y="-202.8" font-family="Helvetica,sans-Serif" font-size="14.00">admin only</text>
|
||||
</g>
|
||||
<!-- op_kick -->
|
||||
<g id="node7" class="node">
|
||||
<title>op_kick</title>
|
||||
<polygon fill="#e8f0ff" stroke="black" points="167,-177 22,-177 22,-124 173,-124 173,-171 167,-177"/>
|
||||
<polyline fill="none" stroke="black" points="167,-177 167,-171 "/>
|
||||
<polyline fill="none" stroke="black" points="173,-171 167,-171 "/>
|
||||
<text text-anchor="middle" x="97.5" y="-161.8" font-family="Helvetica,sans-Serif" font-size="14.00">aborist mesh kick</text>
|
||||
<text text-anchor="middle" x="97.5" y="-146.8" font-family="Helvetica,sans-Serif" font-size="14.00">--member-id bob ...</text>
|
||||
<text text-anchor="middle" x="97.5" y="-131.8" font-family="Helvetica,sans-Serif" font-size="14.00">admin only</text>
|
||||
</g>
|
||||
<!-- op_rotate -->
|
||||
<g id="node8" class="node">
|
||||
<title>op_rotate</title>
|
||||
<polygon fill="#e8f0ff" stroke="black" points="171,-105.5 18,-105.5 18,-67.5 177,-67.5 177,-99.5 171,-105.5"/>
|
||||
<polyline fill="none" stroke="black" points="171,-105.5 171,-99.5 "/>
|
||||
<polyline fill="none" stroke="black" points="177,-99.5 171,-99.5 "/>
|
||||
<text text-anchor="middle" x="97.5" y="-90.3" font-family="Helvetica,sans-Serif" font-size="14.00">aborist mesh rotate</text>
|
||||
<text text-anchor="middle" x="97.5" y="-75.3" font-family="Helvetica,sans-Serif" font-size="14.00">any current member</text>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 8.3 KiB |
206
docs/diagrams/mesh-group-decisions.svg
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<!-- Generated by graphviz version 2.43.0 (0)
|
||||
-->
|
||||
<!-- Title: mesh_group_decisions Pages: 1 -->
|
||||
<svg width="1124pt" height="836pt"
|
||||
viewBox="0.00 0.00 1124.00 836.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 832)">
|
||||
<title>mesh_group_decisions</title>
|
||||
<polygon fill="white" stroke="transparent" points="-4,4 -4,-832 1120,-832 1120,4 -4,4"/>
|
||||
<!-- start -->
|
||||
<g id="node1" class="node">
|
||||
<title>start</title>
|
||||
<ellipse fill="#eeeeee" stroke="black" cx="824" cy="-810" rx="144.87" ry="18"/>
|
||||
<text text-anchor="middle" x="824" y="-806.3" font-family="Helvetica,sans-Serif" font-size="14.00">group steward has a question</text>
|
||||
</g>
|
||||
<!-- q_first -->
|
||||
<g id="node2" class="node">
|
||||
<title>q_first</title>
|
||||
<polygon fill="#fff7d6" stroke="black" points="824,-755 741,-702 824,-649 907,-702 824,-755"/>
|
||||
<text text-anchor="middle" x="824" y="-713.3" font-family="Helvetica,sans-Serif" font-size="14.00">is mesh</text>
|
||||
<text text-anchor="middle" x="824" y="-698.3" font-family="Helvetica,sans-Serif" font-size="14.00">initialized</text>
|
||||
<text text-anchor="middle" x="824" y="-683.3" font-family="Helvetica,sans-Serif" font-size="14.00">yet?</text>
|
||||
</g>
|
||||
<!-- start->q_first -->
|
||||
<g id="edge1" class="edge">
|
||||
<title>start->q_first</title>
|
||||
<path fill="none" stroke="black" d="M824,-791.97C824,-784.5 824,-775.22 824,-765.51"/>
|
||||
<polygon fill="black" stroke="black" points="827.5,-765.28 824,-755.28 820.5,-765.28 827.5,-765.28"/>
|
||||
</g>
|
||||
<!-- q_role -->
|
||||
<g id="node3" class="node">
|
||||
<title>q_role</title>
|
||||
<polygon fill="#fff7d6" stroke="black" points="681,-602 562,-549 681,-496 800,-549 681,-602"/>
|
||||
<text text-anchor="middle" x="681" y="-560.3" font-family="Helvetica,sans-Serif" font-size="14.00">is the question</text>
|
||||
<text text-anchor="middle" x="681" y="-545.3" font-family="Helvetica,sans-Serif" font-size="14.00">about people</text>
|
||||
<text text-anchor="middle" x="681" y="-530.3" font-family="Helvetica,sans-Serif" font-size="14.00">or secrets?</text>
|
||||
</g>
|
||||
<!-- q_first->q_role -->
|
||||
<g id="edge3" class="edge">
|
||||
<title>q_first->q_role</title>
|
||||
<path fill="none" stroke="black" d="M792.97,-668.24C772.44,-646.55 745.27,-617.86 722.83,-594.18"/>
|
||||
<polygon fill="black" stroke="black" points="725.15,-591.53 715.73,-586.67 720.07,-596.34 725.15,-591.53"/>
|
||||
<text text-anchor="middle" x="766" y="-623" font-family="Helvetica,sans-Serif" font-size="10.00">yes</text>
|
||||
</g>
|
||||
<!-- a_init -->
|
||||
<g id="node6" class="node">
|
||||
<title>a_init</title>
|
||||
<path fill="#d6ffd6" stroke="black" d="M1104,-575.5C1104,-575.5 830,-575.5 830,-575.5 824,-575.5 818,-569.5 818,-563.5 818,-563.5 818,-534.5 818,-534.5 818,-528.5 824,-522.5 830,-522.5 830,-522.5 1104,-522.5 1104,-522.5 1110,-522.5 1116,-528.5 1116,-534.5 1116,-534.5 1116,-563.5 1116,-563.5 1116,-569.5 1110,-575.5 1104,-575.5"/>
|
||||
<text text-anchor="middle" x="967" y="-560.3" font-family="Helvetica,sans-Serif" font-size="14.00">aborist mesh init --group <name></text>
|
||||
<text text-anchor="middle" x="967" y="-545.3" font-family="Helvetica,sans-Serif" font-size="14.00">then mesh enable</text>
|
||||
<text text-anchor="middle" x="967" y="-530.3" font-family="Helvetica,sans-Serif" font-size="14.00">(creates epoch 0; founder = sole admin)</text>
|
||||
</g>
|
||||
<!-- q_first->a_init -->
|
||||
<g id="edge2" class="edge">
|
||||
<title>q_first->a_init</title>
|
||||
<path fill="none" stroke="black" d="M855.03,-668.24C878.85,-643.08 911.61,-608.49 935.54,-583.22"/>
|
||||
<polygon fill="black" stroke="black" points="938.33,-585.37 942.66,-575.7 933.24,-580.55 938.33,-585.37"/>
|
||||
<text text-anchor="middle" x="906.5" y="-623" font-family="Helvetica,sans-Serif" font-size="10.00">no</text>
|
||||
</g>
|
||||
<!-- q_who -->
|
||||
<g id="node4" class="node">
|
||||
<title>q_who</title>
|
||||
<polygon fill="#fff7d6" stroke="black" points="609,-449 501,-411 609,-373 717,-411 609,-449"/>
|
||||
<text text-anchor="middle" x="609" y="-414.8" font-family="Helvetica,sans-Serif" font-size="14.00">add, remove,</text>
|
||||
<text text-anchor="middle" x="609" y="-399.8" font-family="Helvetica,sans-Serif" font-size="14.00">or refresh?</text>
|
||||
</g>
|
||||
<!-- q_role->q_who -->
|
||||
<g id="edge4" class="edge">
|
||||
<title>q_role->q_who</title>
|
||||
<path fill="none" stroke="black" d="M658.61,-505.71C649.63,-488.74 639.31,-469.26 630.43,-452.47"/>
|
||||
<polygon fill="black" stroke="black" points="633.42,-450.65 625.65,-443.45 627.24,-453.93 633.42,-450.65"/>
|
||||
<text text-anchor="middle" x="660" y="-470" font-family="Helvetica,sans-Serif" font-size="10.00">people</text>
|
||||
</g>
|
||||
<!-- a_rotate -->
|
||||
<g id="node9" class="node">
|
||||
<title>a_rotate</title>
|
||||
<path fill="#d6ffd6" stroke="black" d="M826,-299.5C826,-299.5 680,-299.5 680,-299.5 674,-299.5 668,-293.5 668,-287.5 668,-287.5 668,-258.5 668,-258.5 668,-252.5 674,-246.5 680,-246.5 680,-246.5 826,-246.5 826,-246.5 832,-246.5 838,-252.5 838,-258.5 838,-258.5 838,-287.5 838,-287.5 838,-293.5 832,-299.5 826,-299.5"/>
|
||||
<text text-anchor="middle" x="753" y="-284.3" font-family="Helvetica,sans-Serif" font-size="14.00">aborist mesh rotate</text>
|
||||
<text text-anchor="middle" x="753" y="-269.3" font-family="Helvetica,sans-Serif" font-size="14.00">--reason '...'</text>
|
||||
<text text-anchor="middle" x="753" y="-254.3" font-family="Helvetica,sans-Serif" font-size="14.00">(any current member)</text>
|
||||
</g>
|
||||
<!-- q_role->a_rotate -->
|
||||
<g id="edge5" class="edge">
|
||||
<title>q_role->a_rotate</title>
|
||||
<path fill="none" stroke="black" d="M703.35,-505.62C711.58,-488.45 720.28,-468.18 726,-449 740.03,-401.94 747.07,-345.48 750.37,-309.67"/>
|
||||
<polygon fill="black" stroke="black" points="753.86,-309.96 751.24,-299.69 746.89,-309.35 753.86,-309.96"/>
|
||||
<text text-anchor="middle" x="794.5" y="-408.5" font-family="Helvetica,sans-Serif" font-size="10.00">just rotate the secret</text>
|
||||
</g>
|
||||
<!-- q_admin -->
|
||||
<g id="node5" class="node">
|
||||
<title>q_admin</title>
|
||||
<polygon fill="#fff7d6" stroke="black" points="462,-326 342,-273 462,-220 582,-273 462,-326"/>
|
||||
<text text-anchor="middle" x="462" y="-284.3" font-family="Helvetica,sans-Serif" font-size="14.00">are you</text>
|
||||
<text text-anchor="middle" x="462" y="-269.3" font-family="Helvetica,sans-Serif" font-size="14.00">admin in</text>
|
||||
<text text-anchor="middle" x="462" y="-254.3" font-family="Helvetica,sans-Serif" font-size="14.00">current epoch?</text>
|
||||
</g>
|
||||
<!-- q_who->q_admin -->
|
||||
<g id="edge6" class="edge">
|
||||
<title>q_who->q_admin</title>
|
||||
<path fill="none" stroke="black" d="M579.96,-383.13C559.35,-364.06 531.26,-338.08 507.67,-316.25"/>
|
||||
<polygon fill="black" stroke="black" points="509.9,-313.55 500.18,-309.32 505.14,-318.68 509.9,-313.55"/>
|
||||
<text text-anchor="middle" x="576.5" y="-347" font-family="Helvetica,sans-Serif" font-size="10.00">add or kick</text>
|
||||
</g>
|
||||
<!-- q_who->a_rotate -->
|
||||
<g id="edge7" class="edge">
|
||||
<title>q_who->a_rotate</title>
|
||||
<path fill="none" stroke="black" d="M623.5,-378.09C629.7,-366.54 637.67,-353.92 647,-344 660.96,-329.16 678.65,-315.91 695.56,-305.04"/>
|
||||
<polygon fill="black" stroke="black" points="697.68,-307.84 704.31,-299.58 693.97,-301.91 697.68,-307.84"/>
|
||||
<text text-anchor="middle" x="693" y="-347" font-family="Helvetica,sans-Serif" font-size="10.00">refresh secret only</text>
|
||||
</g>
|
||||
<!-- a_add -->
|
||||
<g id="node7" class="node">
|
||||
<title>a_add</title>
|
||||
<path fill="#d6ffd6" stroke="black" d="M180.5,-173C180.5,-173 61.5,-173 61.5,-173 55.5,-173 49.5,-167 49.5,-161 49.5,-161 49.5,-102 49.5,-102 49.5,-96 55.5,-90 61.5,-90 61.5,-90 180.5,-90 180.5,-90 186.5,-90 192.5,-96 192.5,-102 192.5,-102 192.5,-161 192.5,-161 192.5,-167 186.5,-173 180.5,-173"/>
|
||||
<text text-anchor="middle" x="121" y="-157.8" font-family="Helvetica,sans-Serif" font-size="14.00">aborist mesh add</text>
|
||||
<text text-anchor="middle" x="121" y="-142.8" font-family="Helvetica,sans-Serif" font-size="14.00">--member-id <id></text>
|
||||
<text text-anchor="middle" x="121" y="-127.8" font-family="Helvetica,sans-Serif" font-size="14.00">--sign-pub <hex></text>
|
||||
<text text-anchor="middle" x="121" y="-112.8" font-family="Helvetica,sans-Serif" font-size="14.00">--dh-pub <hex></text>
|
||||
<text text-anchor="middle" x="121" y="-97.8" font-family="Helvetica,sans-Serif" font-size="14.00">[--role admin]</text>
|
||||
</g>
|
||||
<!-- q_admin->a_add -->
|
||||
<g id="edge8" class="edge">
|
||||
<title>q_admin->a_add</title>
|
||||
<path fill="none" stroke="black" d="M399.54,-247.3C366.1,-234 324.25,-217.25 287,-202 259.32,-190.67 229.14,-178.1 202.19,-166.8"/>
|
||||
<polygon fill="black" stroke="black" points="203.38,-163.5 192.8,-162.86 200.67,-169.95 203.38,-163.5"/>
|
||||
<text text-anchor="middle" x="334.5" y="-194" font-family="Helvetica,sans-Serif" font-size="10.00">yes (admin) -> add</text>
|
||||
</g>
|
||||
<!-- a_kick -->
|
||||
<g id="node8" class="node">
|
||||
<title>a_kick</title>
|
||||
<path fill="#d6ffd6" stroke="black" d="M384.5,-158C384.5,-158 265.5,-158 265.5,-158 259.5,-158 253.5,-152 253.5,-146 253.5,-146 253.5,-117 253.5,-117 253.5,-111 259.5,-105 265.5,-105 265.5,-105 384.5,-105 384.5,-105 390.5,-105 396.5,-111 396.5,-117 396.5,-117 396.5,-146 396.5,-146 396.5,-152 390.5,-158 384.5,-158"/>
|
||||
<text text-anchor="middle" x="325" y="-142.8" font-family="Helvetica,sans-Serif" font-size="14.00">aborist mesh kick</text>
|
||||
<text text-anchor="middle" x="325" y="-127.8" font-family="Helvetica,sans-Serif" font-size="14.00">--member-id <id></text>
|
||||
<text text-anchor="middle" x="325" y="-112.8" font-family="Helvetica,sans-Serif" font-size="14.00">--reason '...'</text>
|
||||
</g>
|
||||
<!-- q_admin->a_kick -->
|
||||
<g id="edge9" class="edge">
|
||||
<title>q_admin->a_kick</title>
|
||||
<path fill="none" stroke="black" d="M426.36,-235.71C404.94,-213.9 377.91,-186.37 357.15,-165.23"/>
|
||||
<polygon fill="black" stroke="black" points="359.59,-162.72 350.08,-158.04 354.59,-167.63 359.59,-162.72"/>
|
||||
<text text-anchor="middle" x="438.5" y="-194" font-family="Helvetica,sans-Serif" font-size="10.00">yes (admin) -> kick</text>
|
||||
</g>
|
||||
<!-- g_admin -->
|
||||
<g id="node10" class="node">
|
||||
<title>g_admin</title>
|
||||
<path fill="#ffe0e0" stroke="black" d="M637,-158C637,-158 427,-158 427,-158 421,-158 415,-152 415,-146 415,-146 415,-117 415,-117 415,-111 421,-105 427,-105 427,-105 637,-105 637,-105 643,-105 649,-111 649,-117 649,-117 649,-146 649,-146 649,-152 643,-158 637,-158"/>
|
||||
<text text-anchor="middle" x="532" y="-142.8" font-family="Helvetica,sans-Serif" font-size="14.00">permission denied</text>
|
||||
<text text-anchor="middle" x="532" y="-127.8" font-family="Helvetica,sans-Serif" font-size="14.00">(only admins of the</text>
|
||||
<text text-anchor="middle" x="532" y="-112.8" font-family="Helvetica,sans-Serif" font-size="14.00">current epoch may add or kick)</text>
|
||||
</g>
|
||||
<!-- q_admin->g_admin -->
|
||||
<g id="edge10" class="edge">
|
||||
<title>q_admin->g_admin</title>
|
||||
<path fill="none" stroke="black" d="M483.38,-229.39C493.36,-209.51 505.08,-186.15 514.52,-167.34"/>
|
||||
<polygon fill="black" stroke="black" points="517.73,-168.75 519.08,-158.24 511.47,-165.61 517.73,-168.75"/>
|
||||
<text text-anchor="middle" x="508.5" y="-194" font-family="Helvetica,sans-Serif" font-size="10.00">no</text>
|
||||
</g>
|
||||
<!-- g_add -->
|
||||
<g id="node11" class="node">
|
||||
<title>g_add</title>
|
||||
<polygon fill="#ffffff" stroke="black" points="192,-53 0,-53 0,0 198,0 198,-47 192,-53"/>
|
||||
<polyline fill="none" stroke="black" points="192,-53 192,-47 "/>
|
||||
<polyline fill="none" stroke="black" points="198,-47 192,-47 "/>
|
||||
<text text-anchor="middle" x="99" y="-37.8" font-family="Helvetica,sans-Serif" font-size="14.00">new member can decrypt</text>
|
||||
<text text-anchor="middle" x="99" y="-22.8" font-family="Helvetica,sans-Serif" font-size="14.00">epoch+1 forward;</text>
|
||||
<text text-anchor="middle" x="99" y="-7.8" font-family="Helvetica,sans-Serif" font-size="14.00">cannot reach prior epochs</text>
|
||||
</g>
|
||||
<!-- a_add->g_add -->
|
||||
<g id="edge11" class="edge">
|
||||
<title>a_add->g_add</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M112.31,-89.79C109.66,-77.42 106.83,-64.18 104.47,-53.1"/>
|
||||
</g>
|
||||
<!-- g_kick -->
|
||||
<g id="node12" class="node">
|
||||
<title>g_kick</title>
|
||||
<polygon fill="#ffffff" stroke="black" points="472,-53 216,-53 216,0 478,0 478,-47 472,-53"/>
|
||||
<polyline fill="none" stroke="black" points="472,-53 472,-47 "/>
|
||||
<polyline fill="none" stroke="black" points="478,-47 472,-47 "/>
|
||||
<text text-anchor="middle" x="347" y="-37.8" font-family="Helvetica,sans-Serif" font-size="14.00">kicked member's prior</text>
|
||||
<text text-anchor="middle" x="347" y="-22.8" font-family="Helvetica,sans-Serif" font-size="14.00">signatures stay valid forever;</text>
|
||||
<text text-anchor="middle" x="347" y="-7.8" font-family="Helvetica,sans-Serif" font-size="14.00">opaque to gossip from epoch+1 on</text>
|
||||
</g>
|
||||
<!-- a_kick->g_kick -->
|
||||
<g id="edge12" class="edge">
|
||||
<title>a_kick->g_kick</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M330.49,-104.77C333.85,-89.04 338.14,-68.99 341.5,-53.25"/>
|
||||
</g>
|
||||
<!-- g_rot -->
|
||||
<g id="node13" class="node">
|
||||
<title>g_rot</title>
|
||||
<polygon fill="#ffffff" stroke="black" points="900.5,-158 667.5,-158 667.5,-105 906.5,-105 906.5,-152 900.5,-158"/>
|
||||
<polyline fill="none" stroke="black" points="900.5,-158 900.5,-152 "/>
|
||||
<polyline fill="none" stroke="black" points="906.5,-152 900.5,-152 "/>
|
||||
<text text-anchor="middle" x="787" y="-142.8" font-family="Helvetica,sans-Serif" font-size="14.00">fresh secret on the same roster;</text>
|
||||
<text text-anchor="middle" x="787" y="-127.8" font-family="Helvetica,sans-Serif" font-size="14.00">use on suspected secret leak</text>
|
||||
<text text-anchor="middle" x="787" y="-112.8" font-family="Helvetica,sans-Serif" font-size="14.00">or scheduled hygiene</text>
|
||||
</g>
|
||||
<!-- a_rotate->g_rot -->
|
||||
<g id="edge13" class="edge">
|
||||
<title>a_rotate->g_rot</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M759.24,-246.41C765.33,-221.4 774.57,-183.52 780.69,-158.4"/>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 14 KiB |
150
docs/diagrams/mesh-identity-stack.svg
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<!-- Generated by graphviz version 2.43.0 (0)
|
||||
-->
|
||||
<!-- Title: mesh_identity_stack Pages: 1 -->
|
||||
<svg width="886pt" height="404pt"
|
||||
viewBox="0.00 0.00 886.00 403.50" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 399.5)">
|
||||
<title>mesh_identity_stack</title>
|
||||
<polygon fill="white" stroke="transparent" points="-4,4 -4,-399.5 882,-399.5 882,4 -4,4"/>
|
||||
<g id="clust1" class="cluster">
|
||||
<title>cluster_peer</title>
|
||||
<path fill="white" stroke="#666666" stroke-dasharray="5,2" d="M607,-232.5C607,-232.5 744,-232.5 744,-232.5 750,-232.5 756,-238.5 756,-244.5 756,-244.5 756,-297.5 756,-297.5 756,-303.5 750,-309.5 744,-309.5 744,-309.5 607,-309.5 607,-309.5 601,-309.5 595,-303.5 595,-297.5 595,-297.5 595,-244.5 595,-244.5 595,-238.5 601,-232.5 607,-232.5"/>
|
||||
<text text-anchor="middle" x="675.5" y="-294.3" font-family="Times,serif" font-size="14.00">One peer (e.g. alice)</text>
|
||||
</g>
|
||||
<g id="clust2" class="cluster">
|
||||
<title>cluster_group</title>
|
||||
<path fill="white" stroke="#666666" stroke-dasharray="5,2" d="M20,-8C20,-8 708,-8 708,-8 714,-8 720,-14 720,-20 720,-20 720,-173 720,-173 720,-179 714,-185 708,-185 708,-185 20,-185 20,-185 14,-185 8,-179 8,-173 8,-173 8,-20 8,-20 8,-14 14,-8 20,-8"/>
|
||||
<text text-anchor="middle" x="364" y="-169.8" font-family="Times,serif" font-size="14.00">Per-epoch group state</text>
|
||||
</g>
|
||||
<!-- sign_priv -->
|
||||
<g id="node1" class="node">
|
||||
<title>sign_priv</title>
|
||||
<path fill="#ffe8a3" stroke="black" d="M302.5,-395.5C302.5,-395.5 199.5,-395.5 199.5,-395.5 193.5,-395.5 187.5,-389.5 187.5,-383.5 187.5,-383.5 187.5,-369.5 187.5,-369.5 187.5,-363.5 193.5,-357.5 199.5,-357.5 199.5,-357.5 302.5,-357.5 302.5,-357.5 308.5,-357.5 314.5,-363.5 314.5,-369.5 314.5,-369.5 314.5,-383.5 314.5,-383.5 314.5,-389.5 308.5,-395.5 302.5,-395.5"/>
|
||||
<text text-anchor="middle" x="251" y="-380.3" font-family="Helvetica,sans-Serif" font-size="14.00">Ed25519 priv</text>
|
||||
<text text-anchor="middle" x="251" y="-365.3" font-family="Helvetica,sans-Serif" font-size="14.00">(sign_priv, 32B)</text>
|
||||
</g>
|
||||
<!-- sign_pub -->
|
||||
<g id="node2" class="node">
|
||||
<title>sign_pub</title>
|
||||
<path fill="#fff7d6" stroke="black" d="M413.5,-286C413.5,-286 294.5,-286 294.5,-286 288.5,-286 282.5,-280 282.5,-274 282.5,-274 282.5,-245 282.5,-245 282.5,-239 288.5,-233 294.5,-233 294.5,-233 413.5,-233 413.5,-233 419.5,-233 425.5,-239 425.5,-245 425.5,-245 425.5,-274 425.5,-274 425.5,-280 419.5,-286 413.5,-286"/>
|
||||
<text text-anchor="middle" x="354" y="-270.8" font-family="Helvetica,sans-Serif" font-size="14.00">Ed25519 pub</text>
|
||||
<text text-anchor="middle" x="354" y="-255.8" font-family="Helvetica,sans-Serif" font-size="14.00">(sign_pub, 32B)</text>
|
||||
<text text-anchor="middle" x="354" y="-240.8" font-family="Helvetica,sans-Serif" font-size="14.00">shared with group</text>
|
||||
</g>
|
||||
<!-- sign_priv->sign_pub -->
|
||||
<g id="edge1" class="edge">
|
||||
<title>sign_priv->sign_pub</title>
|
||||
<path fill="none" stroke="black" d="M267.32,-357.28C282.51,-340.32 305.52,-314.62 324.02,-293.97"/>
|
||||
<polygon fill="black" stroke="black" points="326.8,-296.12 330.86,-286.33 321.58,-291.45 326.8,-296.12"/>
|
||||
<text text-anchor="middle" x="332" y="-326" font-family="Helvetica,sans-Serif" font-size="10.00">public_key()</text>
|
||||
</g>
|
||||
<!-- audit -->
|
||||
<g id="node9" class="node">
|
||||
<title>audit</title>
|
||||
<polygon fill="#ffffff" stroke="black" points="258.5,-286 31.5,-286 31.5,-233 264.5,-233 264.5,-280 258.5,-286"/>
|
||||
<polyline fill="none" stroke="black" points="258.5,-286 258.5,-280 "/>
|
||||
<polyline fill="none" stroke="black" points="264.5,-280 258.5,-280 "/>
|
||||
<text text-anchor="middle" x="148" y="-270.8" font-family="Helvetica,sans-Serif" font-size="14.00">audit_events</text>
|
||||
<text text-anchor="middle" x="148" y="-255.8" font-family="Helvetica,sans-Serif" font-size="14.00">(prev_event_hash, event_hash)</text>
|
||||
<text text-anchor="middle" x="148" y="-240.8" font-family="Helvetica,sans-Serif" font-size="14.00">append-only Merkle chain</text>
|
||||
</g>
|
||||
<!-- sign_priv->audit -->
|
||||
<g id="edge4" class="edge">
|
||||
<title>sign_priv->audit</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="5,2" d="M223.84,-357.49C216.68,-352.12 209.22,-345.95 203,-339.5 190.12,-326.16 178.07,-309.54 168.55,-294.99"/>
|
||||
<polygon fill="black" stroke="black" points="171.3,-292.79 162.97,-286.25 165.4,-296.56 171.3,-292.79"/>
|
||||
<text text-anchor="middle" x="241.5" y="-331.5" font-family="Helvetica,sans-Serif" font-size="10.00">signs each</text>
|
||||
<text text-anchor="middle" x="241.5" y="-320.5" font-family="Helvetica,sans-Serif" font-size="10.00">membership op</text>
|
||||
</g>
|
||||
<!-- roster -->
|
||||
<g id="node6" class="node">
|
||||
<title>roster</title>
|
||||
<path fill="#e8ffe8" stroke="black" d="M699.5,-154C699.5,-154 400.5,-154 400.5,-154 394.5,-154 388.5,-148 388.5,-142 388.5,-142 388.5,-128 388.5,-128 388.5,-122 394.5,-116 400.5,-116 400.5,-116 699.5,-116 699.5,-116 705.5,-116 711.5,-122 711.5,-128 711.5,-128 711.5,-142 711.5,-142 711.5,-148 705.5,-154 699.5,-154"/>
|
||||
<text text-anchor="middle" x="550" y="-138.8" font-family="Helvetica,sans-Serif" font-size="14.00">mesh_roster</text>
|
||||
<text text-anchor="middle" x="550" y="-123.8" font-family="Helvetica,sans-Serif" font-size="14.00">(epoch, member_id, sign_pub, dh_pub, role)</text>
|
||||
</g>
|
||||
<!-- sign_pub->roster -->
|
||||
<g id="edge5" class="edge">
|
||||
<title>sign_pub->roster</title>
|
||||
<path fill="none" stroke="black" d="M395.09,-232.82C429.71,-211.18 478.84,-180.48 512.43,-159.48"/>
|
||||
<polygon fill="black" stroke="black" points="514.42,-162.37 521.04,-154.1 510.7,-156.43 514.42,-162.37"/>
|
||||
<text text-anchor="middle" x="483.5" y="-201.5" font-family="Helvetica,sans-Serif" font-size="10.00">enrolled at</text>
|
||||
</g>
|
||||
<!-- dh_priv -->
|
||||
<g id="node3" class="node">
|
||||
<title>dh_priv</title>
|
||||
<path fill="#a3d8ff" stroke="black" d="M755,-395.5C755,-395.5 663,-395.5 663,-395.5 657,-395.5 651,-389.5 651,-383.5 651,-383.5 651,-369.5 651,-369.5 651,-363.5 657,-357.5 663,-357.5 663,-357.5 755,-357.5 755,-357.5 761,-357.5 767,-363.5 767,-369.5 767,-369.5 767,-383.5 767,-383.5 767,-389.5 761,-395.5 755,-395.5"/>
|
||||
<text text-anchor="middle" x="709" y="-380.3" font-family="Helvetica,sans-Serif" font-size="14.00">X25519 priv</text>
|
||||
<text text-anchor="middle" x="709" y="-365.3" font-family="Helvetica,sans-Serif" font-size="14.00">(dh_priv, 32B)</text>
|
||||
</g>
|
||||
<!-- dh_pub -->
|
||||
<g id="node4" class="node">
|
||||
<title>dh_pub</title>
|
||||
<path fill="#d6ecff" stroke="black" d="M574.5,-286C574.5,-286 455.5,-286 455.5,-286 449.5,-286 443.5,-280 443.5,-274 443.5,-274 443.5,-245 443.5,-245 443.5,-239 449.5,-233 455.5,-233 455.5,-233 574.5,-233 574.5,-233 580.5,-233 586.5,-239 586.5,-245 586.5,-245 586.5,-274 586.5,-274 586.5,-280 580.5,-286 574.5,-286"/>
|
||||
<text text-anchor="middle" x="515" y="-270.8" font-family="Helvetica,sans-Serif" font-size="14.00">X25519 pub</text>
|
||||
<text text-anchor="middle" x="515" y="-255.8" font-family="Helvetica,sans-Serif" font-size="14.00">(dh_pub, 32B)</text>
|
||||
<text text-anchor="middle" x="515" y="-240.8" font-family="Helvetica,sans-Serif" font-size="14.00">shared with group</text>
|
||||
</g>
|
||||
<!-- dh_priv->dh_pub -->
|
||||
<g id="edge2" class="edge">
|
||||
<title>dh_priv->dh_pub</title>
|
||||
<path fill="none" stroke="black" d="M675.37,-357.42C651.58,-344.45 619.06,-326.37 591,-309.5 581.63,-303.86 571.73,-297.67 562.28,-291.63"/>
|
||||
<polygon fill="black" stroke="black" points="564.06,-288.62 553.76,-286.15 560.28,-294.51 564.06,-288.62"/>
|
||||
<text text-anchor="middle" x="672" y="-326" font-family="Helvetica,sans-Serif" font-size="10.00">public_key()</text>
|
||||
</g>
|
||||
<!-- secret -->
|
||||
<g id="node8" class="node">
|
||||
<title>secret</title>
|
||||
<path fill="#ffd6d6" stroke="black" d="M592.5,-69C592.5,-69 433.5,-69 433.5,-69 427.5,-69 421.5,-63 421.5,-57 421.5,-57 421.5,-28 421.5,-28 421.5,-22 427.5,-16 433.5,-16 433.5,-16 592.5,-16 592.5,-16 598.5,-16 604.5,-22 604.5,-28 604.5,-28 604.5,-57 604.5,-57 604.5,-63 598.5,-69 592.5,-69"/>
|
||||
<text text-anchor="middle" x="513" y="-53.8" font-family="Helvetica,sans-Serif" font-size="14.00">symmetric epoch secret</text>
|
||||
<text text-anchor="middle" x="513" y="-38.8" font-family="Helvetica,sans-Serif" font-size="14.00">32 random bytes</text>
|
||||
<text text-anchor="middle" x="513" y="-23.8" font-family="Helvetica,sans-Serif" font-size="14.00">never stored in clear</text>
|
||||
</g>
|
||||
<!-- dh_priv->secret -->
|
||||
<g id="edge7" class="edge">
|
||||
<title>dh_priv->secret</title>
|
||||
<path fill="none" stroke="#0066cc" stroke-dasharray="5,2" d="M728.44,-357.33C740.05,-345.09 753.73,-327.86 760,-309.5 788.34,-226.47 782.65,-178.42 721,-116 692.63,-87.28 652.29,-69.92 614.67,-59.43"/>
|
||||
<polygon fill="#0066cc" stroke="#0066cc" points="615.23,-55.95 604.66,-56.78 613.43,-62.72 615.23,-55.95"/>
|
||||
<text text-anchor="middle" x="826.5" y="-207" font-family="Helvetica,sans-Serif" font-size="10.00">ECDH-unwraps</text>
|
||||
<text text-anchor="middle" x="826.5" y="-196" font-family="Helvetica,sans-Serif" font-size="10.00">self's envelope entry</text>
|
||||
</g>
|
||||
<!-- dh_pub->roster -->
|
||||
<g id="edge6" class="edge">
|
||||
<title>dh_pub->roster</title>
|
||||
<path fill="none" stroke="black" d="M522.34,-232.82C528.1,-212.65 536.11,-184.62 542.03,-163.9"/>
|
||||
<polygon fill="black" stroke="black" points="545.45,-164.68 544.83,-154.1 538.72,-162.75 545.45,-164.68"/>
|
||||
<text text-anchor="middle" x="559.5" y="-201.5" font-family="Helvetica,sans-Serif" font-size="10.00">enrolled at</text>
|
||||
</g>
|
||||
<!-- member_id -->
|
||||
<g id="node5" class="node">
|
||||
<title>member_id</title>
|
||||
<path fill="#eeeeee" stroke="black" d="M691,-278.5C691,-278.5 617,-278.5 617,-278.5 611,-278.5 605,-272.5 605,-266.5 605,-266.5 605,-252.5 605,-252.5 605,-246.5 611,-240.5 617,-240.5 617,-240.5 691,-240.5 691,-240.5 697,-240.5 703,-246.5 703,-252.5 703,-252.5 703,-266.5 703,-266.5 703,-272.5 697,-278.5 691,-278.5"/>
|
||||
<text text-anchor="middle" x="654" y="-263.3" font-family="Helvetica,sans-Serif" font-size="14.00">member_id</text>
|
||||
<text text-anchor="middle" x="654" y="-248.3" font-family="Helvetica,sans-Serif" font-size="14.00">8 hex chars</text>
|
||||
</g>
|
||||
<!-- member_id->roster -->
|
||||
<g id="edge8" class="edge">
|
||||
<title>member_id->roster</title>
|
||||
<path fill="none" stroke="black" d="M638.65,-240.42C621.07,-219.71 592.09,-185.58 572.08,-162.01"/>
|
||||
<polygon fill="black" stroke="black" points="574.55,-159.51 565.41,-154.15 569.21,-164.04 574.55,-159.51"/>
|
||||
<text text-anchor="middle" x="633" y="-201.5" font-family="Helvetica,sans-Serif" font-size="10.00">keys by</text>
|
||||
</g>
|
||||
<!-- epoch -->
|
||||
<g id="node7" class="node">
|
||||
<title>epoch</title>
|
||||
<path fill="#e8ffe8" stroke="black" d="M358,-154C358,-154 28,-154 28,-154 22,-154 16,-148 16,-142 16,-142 16,-128 16,-128 16,-122 22,-116 28,-116 28,-116 358,-116 358,-116 364,-116 370,-122 370,-128 370,-128 370,-142 370,-142 370,-148 364,-154 358,-154"/>
|
||||
<text text-anchor="middle" x="193" y="-138.8" font-family="Helvetica,sans-Serif" font-size="14.00">mesh_epochs</text>
|
||||
<text text-anchor="middle" x="193" y="-123.8" font-family="Helvetica,sans-Serif" font-size="14.00">(epoch_id, secret_envelope, started_event_hash)</text>
|
||||
</g>
|
||||
<!-- epoch->secret -->
|
||||
<g id="edge3" class="edge">
|
||||
<title>epoch->secret</title>
|
||||
<path fill="none" stroke="black" d="M256.99,-115.9C301.62,-103.28 362.08,-86.18 413.03,-71.77"/>
|
||||
<polygon fill="black" stroke="black" points="414.09,-75.11 422.76,-69.02 412.19,-68.37 414.09,-75.11"/>
|
||||
<text text-anchor="middle" x="430" y="-90" font-family="Helvetica,sans-Serif" font-size="10.00">wrapped per member via ECDH</text>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 11 KiB |
212
docs/diagrams/mesh-secret-envelope.svg
Normal file
|
|
@ -0,0 +1,212 @@
|
|||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<!-- Generated by graphviz version 2.43.0 (0)
|
||||
-->
|
||||
<!-- Title: mesh_secret_envelope Pages: 1 -->
|
||||
<svg width="1237pt" height="387pt"
|
||||
viewBox="0.00 0.00 1237.32 386.56" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 382.56)">
|
||||
<title>mesh_secret_envelope</title>
|
||||
<polygon fill="white" stroke="transparent" points="-4,4 -4,-382.56 1233.32,-382.56 1233.32,4 -4,4"/>
|
||||
<g id="clust1" class="cluster">
|
||||
<title>cluster_wrap</title>
|
||||
<path fill="white" stroke="#666666" stroke-dasharray="5,2" d="M20,-64.56C20,-64.56 444,-64.56 444,-64.56 450,-64.56 456,-70.56 456,-76.56 456,-76.56 456,-358.56 456,-358.56 456,-364.56 450,-370.56 444,-370.56 444,-370.56 20,-370.56 20,-370.56 14,-370.56 8,-364.56 8,-358.56 8,-358.56 8,-76.56 8,-76.56 8,-70.56 14,-64.56 20,-64.56"/>
|
||||
<text text-anchor="middle" x="232" y="-355.36" font-family="Times,serif" font-size="14.00">rotate_epoch() — done by actor (admin)</text>
|
||||
</g>
|
||||
<g id="clust2" class="cluster">
|
||||
<title>cluster_unwrap</title>
|
||||
<path fill="white" stroke="#666666" stroke-dasharray="5,2" d="M948.32,-91.56C948.32,-91.56 1217.32,-91.56 1217.32,-91.56 1223.32,-91.56 1229.32,-97.56 1229.32,-103.56 1229.32,-103.56 1229.32,-358.56 1229.32,-358.56 1229.32,-364.56 1223.32,-370.56 1217.32,-370.56 1217.32,-370.56 948.32,-370.56 948.32,-370.56 942.32,-370.56 936.32,-364.56 936.32,-358.56 936.32,-358.56 936.32,-103.56 936.32,-103.56 936.32,-97.56 942.32,-91.56 948.32,-91.56"/>
|
||||
<text text-anchor="middle" x="1082.82" y="-355.36" font-family="Times,serif" font-size="14.00">Each peer unwraps only their own slot</text>
|
||||
</g>
|
||||
<!-- secret -->
|
||||
<g id="node1" class="node">
|
||||
<title>secret</title>
|
||||
<path fill="#ffd6d6" stroke="black" d="M169.5,-263.04C169.5,-265.68 139.02,-267.82 101.5,-267.82 63.98,-267.82 33.5,-265.68 33.5,-263.04 33.5,-263.04 33.5,-220.09 33.5,-220.09 33.5,-217.45 63.98,-215.31 101.5,-215.31 139.02,-215.31 169.5,-217.45 169.5,-220.09 169.5,-220.09 169.5,-263.04 169.5,-263.04"/>
|
||||
<path fill="none" stroke="black" d="M169.5,-263.04C169.5,-260.41 139.02,-258.27 101.5,-258.27 63.98,-258.27 33.5,-260.41 33.5,-263.04"/>
|
||||
<text text-anchor="middle" x="101.5" y="-245.36" font-family="Helvetica,sans-Serif" font-size="14.00">epoch_N_secret</text>
|
||||
<text text-anchor="middle" x="101.5" y="-230.36" font-family="Helvetica,sans-Serif" font-size="14.00">32 random bytes</text>
|
||||
</g>
|
||||
<!-- wrap_a -->
|
||||
<g id="node5" class="node">
|
||||
<title>wrap_a</title>
|
||||
<path fill="lightgrey" stroke="black" d="M435.5,-290.06C435.5,-290.06 253.5,-290.06 253.5,-290.06 247.5,-290.06 241.5,-284.06 241.5,-278.06 241.5,-278.06 241.5,-249.06 241.5,-249.06 241.5,-243.06 247.5,-237.06 253.5,-237.06 253.5,-237.06 435.5,-237.06 435.5,-237.06 441.5,-237.06 447.5,-243.06 447.5,-249.06 447.5,-249.06 447.5,-278.06 447.5,-278.06 447.5,-284.06 441.5,-290.06 435.5,-290.06"/>
|
||||
<text text-anchor="middle" x="344.5" y="-274.86" font-family="Helvetica,sans-Serif" font-size="14.00">ChaCha20-Poly1305</text>
|
||||
<text text-anchor="middle" x="344.5" y="-259.86" font-family="Helvetica,sans-Serif" font-size="14.00">(shared_a, nonce_a, secret,</text>
|
||||
<text text-anchor="middle" x="344.5" y="-244.86" font-family="Helvetica,sans-Serif" font-size="14.00"> aad=alice.member_id)</text>
|
||||
</g>
|
||||
<!-- secret->wrap_a -->
|
||||
<g id="edge1" class="edge">
|
||||
<title>secret->wrap_a</title>
|
||||
<path fill="none" stroke="black" d="M169.63,-247.69C188.66,-249.43 209.92,-251.37 230.86,-253.28"/>
|
||||
<polygon fill="black" stroke="black" points="230.82,-256.79 241.1,-254.22 231.46,-249.82 230.82,-256.79"/>
|
||||
</g>
|
||||
<!-- wrap_b -->
|
||||
<g id="node6" class="node">
|
||||
<title>wrap_b</title>
|
||||
<path fill="lightgrey" stroke="black" d="M436,-219.06C436,-219.06 253,-219.06 253,-219.06 247,-219.06 241,-213.06 241,-207.06 241,-207.06 241,-178.06 241,-178.06 241,-172.06 247,-166.06 253,-166.06 253,-166.06 436,-166.06 436,-166.06 442,-166.06 448,-172.06 448,-178.06 448,-178.06 448,-207.06 448,-207.06 448,-213.06 442,-219.06 436,-219.06"/>
|
||||
<text text-anchor="middle" x="344.5" y="-203.86" font-family="Helvetica,sans-Serif" font-size="14.00">ChaCha20-Poly1305</text>
|
||||
<text text-anchor="middle" x="344.5" y="-188.86" font-family="Helvetica,sans-Serif" font-size="14.00">(shared_b, nonce_b, secret,</text>
|
||||
<text text-anchor="middle" x="344.5" y="-173.86" font-family="Helvetica,sans-Serif" font-size="14.00"> aad=bob.member_id)</text>
|
||||
</g>
|
||||
<!-- secret->wrap_b -->
|
||||
<g id="edge2" class="edge">
|
||||
<title>secret->wrap_b</title>
|
||||
<path fill="none" stroke="black" d="M169.63,-227.92C188.63,-224.05 209.84,-219.74 230.75,-215.49"/>
|
||||
<polygon fill="black" stroke="black" points="231.54,-218.9 240.64,-213.48 230.15,-212.04 231.54,-218.9"/>
|
||||
</g>
|
||||
<!-- wrap_c -->
|
||||
<g id="node7" class="node">
|
||||
<title>wrap_c</title>
|
||||
<path fill="lightgrey" stroke="black" d="M434.5,-148.06C434.5,-148.06 254.5,-148.06 254.5,-148.06 248.5,-148.06 242.5,-142.06 242.5,-136.06 242.5,-136.06 242.5,-107.06 242.5,-107.06 242.5,-101.06 248.5,-95.06 254.5,-95.06 254.5,-95.06 434.5,-95.06 434.5,-95.06 440.5,-95.06 446.5,-101.06 446.5,-107.06 446.5,-107.06 446.5,-136.06 446.5,-136.06 446.5,-142.06 440.5,-148.06 434.5,-148.06"/>
|
||||
<text text-anchor="middle" x="344.5" y="-132.86" font-family="Helvetica,sans-Serif" font-size="14.00">ChaCha20-Poly1305</text>
|
||||
<text text-anchor="middle" x="344.5" y="-117.86" font-family="Helvetica,sans-Serif" font-size="14.00">(shared_c, nonce_c, secret,</text>
|
||||
<text text-anchor="middle" x="344.5" y="-102.86" font-family="Helvetica,sans-Serif" font-size="14.00"> aad=carol.member_id)</text>
|
||||
</g>
|
||||
<!-- secret->wrap_c -->
|
||||
<g id="edge3" class="edge">
|
||||
<title>secret->wrap_c</title>
|
||||
<path fill="none" stroke="black" d="M169.58,-226.8C188.57,-220.07 208.1,-210.39 223,-196.56 237,-183.58 226.88,-170.41 241,-157.56 242.29,-156.39 243.63,-155.25 245,-154.14"/>
|
||||
<polygon fill="black" stroke="black" points="247.16,-156.91 253.22,-148.22 243.06,-151.23 247.16,-156.91"/>
|
||||
</g>
|
||||
<!-- ecdh_a -->
|
||||
<g id="node2" class="node">
|
||||
<title>ecdh_a</title>
|
||||
<path fill="#e8f0ff" stroke="black" d="M175,-339.06C175,-339.06 28,-339.06 28,-339.06 22,-339.06 16,-333.06 16,-327.06 16,-327.06 16,-298.06 16,-298.06 16,-292.06 22,-286.06 28,-286.06 28,-286.06 175,-286.06 175,-286.06 181,-286.06 187,-292.06 187,-298.06 187,-298.06 187,-327.06 187,-327.06 187,-333.06 181,-339.06 175,-339.06"/>
|
||||
<text text-anchor="middle" x="101.5" y="-323.86" font-family="Helvetica,sans-Serif" font-size="14.00">ECDH(rotator_dh_priv,</text>
|
||||
<text text-anchor="middle" x="101.5" y="-308.86" font-family="Helvetica,sans-Serif" font-size="14.00">      alice_dh_pub)</text>
|
||||
<text text-anchor="middle" x="101.5" y="-293.86" font-family="Helvetica,sans-Serif" font-size="14.00">  -> shared_a (HKDF)</text>
|
||||
</g>
|
||||
<!-- ecdh_a->wrap_a -->
|
||||
<g id="edge4" class="edge">
|
||||
<title>ecdh_a->wrap_a</title>
|
||||
<path fill="none" stroke="black" d="M187.13,-295.36C201.34,-292.47 216.29,-289.43 231.08,-286.42"/>
|
||||
<polygon fill="black" stroke="black" points="232.18,-289.77 241.28,-284.35 230.78,-282.91 232.18,-289.77"/>
|
||||
<text text-anchor="middle" x="214" y="-293.56" font-family="Helvetica,sans-Serif" font-size="10.00">key</text>
|
||||
</g>
|
||||
<!-- ecdh_b -->
|
||||
<g id="node3" class="node">
|
||||
<title>ecdh_b</title>
|
||||
<path fill="#e8f0ff" stroke="black" d="M175,-197.06C175,-197.06 28,-197.06 28,-197.06 22,-197.06 16,-191.06 16,-185.06 16,-185.06 16,-156.06 16,-156.06 16,-150.06 22,-144.06 28,-144.06 28,-144.06 175,-144.06 175,-144.06 181,-144.06 187,-150.06 187,-156.06 187,-156.06 187,-185.06 187,-185.06 187,-191.06 181,-197.06 175,-197.06"/>
|
||||
<text text-anchor="middle" x="101.5" y="-181.86" font-family="Helvetica,sans-Serif" font-size="14.00">ECDH(rotator_dh_priv,</text>
|
||||
<text text-anchor="middle" x="101.5" y="-166.86" font-family="Helvetica,sans-Serif" font-size="14.00">      bob_dh_pub)</text>
|
||||
<text text-anchor="middle" x="101.5" y="-151.86" font-family="Helvetica,sans-Serif" font-size="14.00">  -> shared_b (HKDF)</text>
|
||||
</g>
|
||||
<!-- ecdh_b->wrap_b -->
|
||||
<g id="edge5" class="edge">
|
||||
<title>ecdh_b->wrap_b</title>
|
||||
<path fill="none" stroke="black" d="M187.13,-178.29C201.23,-179.58 216.06,-180.93 230.73,-182.27"/>
|
||||
<polygon fill="black" stroke="black" points="230.58,-185.77 240.86,-183.19 231.22,-178.8 230.58,-185.77"/>
|
||||
<text text-anchor="middle" x="214" y="-184.56" font-family="Helvetica,sans-Serif" font-size="10.00">key</text>
|
||||
</g>
|
||||
<!-- ecdh_c -->
|
||||
<g id="node4" class="node">
|
||||
<title>ecdh_c</title>
|
||||
<path fill="#e8f0ff" stroke="black" d="M175,-126.06C175,-126.06 28,-126.06 28,-126.06 22,-126.06 16,-120.06 16,-114.06 16,-114.06 16,-85.06 16,-85.06 16,-79.06 22,-73.06 28,-73.06 28,-73.06 175,-73.06 175,-73.06 181,-73.06 187,-79.06 187,-85.06 187,-85.06 187,-114.06 187,-114.06 187,-120.06 181,-126.06 175,-126.06"/>
|
||||
<text text-anchor="middle" x="101.5" y="-110.86" font-family="Helvetica,sans-Serif" font-size="14.00">ECDH(rotator_dh_priv,</text>
|
||||
<text text-anchor="middle" x="101.5" y="-95.86" font-family="Helvetica,sans-Serif" font-size="14.00">      carol_dh_pub)</text>
|
||||
<text text-anchor="middle" x="101.5" y="-80.86" font-family="Helvetica,sans-Serif" font-size="14.00">  -> shared_c (HKDF)</text>
|
||||
</g>
|
||||
<!-- ecdh_c->wrap_c -->
|
||||
<g id="edge6" class="edge">
|
||||
<title>ecdh_c->wrap_c</title>
|
||||
<path fill="none" stroke="black" d="M187.13,-107.29C201.56,-108.61 216.76,-109.99 231.77,-111.36"/>
|
||||
<polygon fill="black" stroke="black" points="231.84,-114.89 242.12,-112.31 232.48,-107.91 231.84,-114.89"/>
|
||||
<text text-anchor="middle" x="214" y="-113.56" font-family="Helvetica,sans-Serif" font-size="10.00">key</text>
|
||||
</g>
|
||||
<!-- envelope -->
|
||||
<g id="node8" class="node">
|
||||
<title>envelope</title>
|
||||
<polygon fill="#fff7d6" stroke="black" points="802,-241.56 530,-241.56 530,-143.56 808,-143.56 808,-235.56 802,-241.56"/>
|
||||
<polyline fill="none" stroke="black" points="802,-241.56 802,-235.56 "/>
|
||||
<polyline fill="none" stroke="black" points="808,-235.56 802,-235.56 "/>
|
||||
<text text-anchor="middle" x="669" y="-226.36" font-family="Helvetica,sans-Serif" font-size="14.00">mesh_epochs.secret_envelope (JSON)</text>
|
||||
<text text-anchor="middle" x="669" y="-211.36" font-family="Helvetica,sans-Serif" font-size="14.00">{</text>
|
||||
<text text-anchor="middle" x="669" y="-196.36" font-family="Helvetica,sans-Serif" font-size="14.00">  alice: {nonce_b64, ct_b64},</text>
|
||||
<text text-anchor="middle" x="669" y="-181.36" font-family="Helvetica,sans-Serif" font-size="14.00">  bob:   {nonce_b64, ct_b64},</text>
|
||||
<text text-anchor="middle" x="669" y="-166.36" font-family="Helvetica,sans-Serif" font-size="14.00">  carol: {nonce_b64, ct_b64}</text>
|
||||
<text text-anchor="middle" x="669" y="-151.36" font-family="Helvetica,sans-Serif" font-size="14.00">}</text>
|
||||
</g>
|
||||
<!-- wrap_a->envelope -->
|
||||
<g id="edge7" class="edge">
|
||||
<title>wrap_a->envelope</title>
|
||||
<path fill="none" stroke="black" d="M447.54,-241.1C470.49,-236.05 495.3,-230.59 519.71,-225.21"/>
|
||||
<polygon fill="black" stroke="black" points="520.76,-228.56 529.77,-223 519.25,-221.73 520.76,-228.56"/>
|
||||
<text text-anchor="middle" x="489" y="-238.56" font-family="Helvetica,sans-Serif" font-size="10.00">alice slot</text>
|
||||
</g>
|
||||
<!-- wrap_b->envelope -->
|
||||
<g id="edge8" class="edge">
|
||||
<title>wrap_b->envelope</title>
|
||||
<path fill="none" stroke="black" d="M448.43,-192.56C471.09,-192.56 495.53,-192.56 519.58,-192.56"/>
|
||||
<polygon fill="black" stroke="black" points="519.86,-196.06 529.86,-192.56 519.86,-189.06 519.86,-196.06"/>
|
||||
<text text-anchor="middle" x="489" y="-195.56" font-family="Helvetica,sans-Serif" font-size="10.00">bob slot</text>
|
||||
</g>
|
||||
<!-- wrap_c->envelope -->
|
||||
<g id="edge9" class="edge">
|
||||
<title>wrap_c->envelope</title>
|
||||
<path fill="none" stroke="black" d="M446.66,-143.83C469.98,-148.97 495.25,-154.53 520.1,-160"/>
|
||||
<polygon fill="black" stroke="black" points="519.45,-163.44 529.97,-162.18 520.96,-156.61 519.45,-163.44"/>
|
||||
<text text-anchor="middle" x="489" y="-159.56" font-family="Helvetica,sans-Serif" font-size="10.00">carol slot</text>
|
||||
</g>
|
||||
<!-- u_alice -->
|
||||
<g id="node9" class="node">
|
||||
<title>u_alice</title>
|
||||
<path fill="#d6ffd6" stroke="black" d="M1178.32,-339.56C1178.32,-339.56 987.32,-339.56 987.32,-339.56 981.32,-339.56 975.32,-333.56 975.32,-327.56 975.32,-327.56 975.32,-283.56 975.32,-283.56 975.32,-277.56 981.32,-271.56 987.32,-271.56 987.32,-271.56 1178.32,-271.56 1178.32,-271.56 1184.32,-271.56 1190.32,-277.56 1190.32,-283.56 1190.32,-283.56 1190.32,-327.56 1190.32,-327.56 1190.32,-333.56 1184.32,-339.56 1178.32,-339.56"/>
|
||||
<text text-anchor="middle" x="1082.82" y="-324.36" font-family="Helvetica,sans-Serif" font-size="14.00">alice:</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-309.36" font-family="Helvetica,sans-Serif" font-size="14.00">ECDH(alice_dh_priv,</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-294.36" font-family="Helvetica,sans-Serif" font-size="14.00">     rotator_dh_pub)</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-279.36" font-family="Helvetica,sans-Serif" font-size="14.00">then AEAD-decrypt slot.alice</text>
|
||||
</g>
|
||||
<!-- envelope->u_alice -->
|
||||
<g id="edge10" class="edge">
|
||||
<title>envelope->u_alice</title>
|
||||
<path fill="none" stroke="black" d="M808.27,-230.51C859.32,-244.51 916.59,-260.23 965.5,-273.65"/>
|
||||
<polygon fill="black" stroke="black" points="964.59,-277.03 975.16,-276.3 966.45,-270.28 964.59,-277.03"/>
|
||||
</g>
|
||||
<!-- u_bob -->
|
||||
<g id="node10" class="node">
|
||||
<title>u_bob</title>
|
||||
<path fill="#d6ffd6" stroke="black" d="M1175.82,-253.56C1175.82,-253.56 989.82,-253.56 989.82,-253.56 983.82,-253.56 977.82,-247.56 977.82,-241.56 977.82,-241.56 977.82,-197.56 977.82,-197.56 977.82,-191.56 983.82,-185.56 989.82,-185.56 989.82,-185.56 1175.82,-185.56 1175.82,-185.56 1181.82,-185.56 1187.82,-191.56 1187.82,-197.56 1187.82,-197.56 1187.82,-241.56 1187.82,-241.56 1187.82,-247.56 1181.82,-253.56 1175.82,-253.56"/>
|
||||
<text text-anchor="middle" x="1082.82" y="-238.36" font-family="Helvetica,sans-Serif" font-size="14.00">bob:</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-223.36" font-family="Helvetica,sans-Serif" font-size="14.00">ECDH(bob_dh_priv,</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-208.36" font-family="Helvetica,sans-Serif" font-size="14.00">     rotator_dh_pub)</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-193.36" font-family="Helvetica,sans-Serif" font-size="14.00">then AEAD-decrypt slot.bob</text>
|
||||
</g>
|
||||
<!-- envelope->u_bob -->
|
||||
<g id="edge11" class="edge">
|
||||
<title>envelope->u_bob</title>
|
||||
<path fill="none" stroke="black" d="M808.27,-201.63C860.08,-205.03 918.32,-208.85 967.7,-212.08"/>
|
||||
<polygon fill="black" stroke="black" points="967.57,-215.58 977.78,-212.74 968.03,-208.6 967.57,-215.58"/>
|
||||
</g>
|
||||
<!-- u_carol -->
|
||||
<g id="node11" class="node">
|
||||
<title>u_carol</title>
|
||||
<path fill="#d6ffd6" stroke="black" d="M1179.32,-167.56C1179.32,-167.56 986.32,-167.56 986.32,-167.56 980.32,-167.56 974.32,-161.56 974.32,-155.56 974.32,-155.56 974.32,-111.56 974.32,-111.56 974.32,-105.56 980.32,-99.56 986.32,-99.56 986.32,-99.56 1179.32,-99.56 1179.32,-99.56 1185.32,-99.56 1191.32,-105.56 1191.32,-111.56 1191.32,-111.56 1191.32,-155.56 1191.32,-155.56 1191.32,-161.56 1185.32,-167.56 1179.32,-167.56"/>
|
||||
<text text-anchor="middle" x="1082.82" y="-152.36" font-family="Helvetica,sans-Serif" font-size="14.00">carol:</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-137.36" font-family="Helvetica,sans-Serif" font-size="14.00">ECDH(carol_dh_priv,</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-122.36" font-family="Helvetica,sans-Serif" font-size="14.00">     rotator_dh_pub)</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-107.36" font-family="Helvetica,sans-Serif" font-size="14.00">then AEAD-decrypt slot.carol</text>
|
||||
</g>
|
||||
<!-- envelope->u_carol -->
|
||||
<g id="edge12" class="edge">
|
||||
<title>envelope->u_carol</title>
|
||||
<path fill="none" stroke="black" d="M808.27,-172.75C858.88,-165.5 915.62,-157.37 964.25,-150.41"/>
|
||||
<polygon fill="black" stroke="black" points="964.78,-153.87 974.18,-148.99 963.78,-146.94 964.78,-153.87"/>
|
||||
</g>
|
||||
<!-- evicted -->
|
||||
<g id="node12" class="node">
|
||||
<title>evicted</title>
|
||||
<path fill="#ffe0e0" stroke="black" d="M1210.45,-34.95C1210.45,-34.95 1210.45,-46.17 1210.45,-46.17 1210.45,-51.78 1205.1,-59.1 1199.76,-60.8 1199.76,-60.8 1146.37,-77.79 1146.37,-77.79 1141.03,-79.49 1130.07,-81.19 1124.46,-81.19 1124.46,-81.19 1041.17,-81.19 1041.17,-81.19 1035.56,-81.19 1024.6,-79.49 1019.26,-77.79 1019.26,-77.79 965.87,-60.8 965.87,-60.8 960.53,-59.1 955.18,-51.78 955.18,-46.17 955.18,-46.17 955.18,-34.95 955.18,-34.95 955.18,-29.35 960.53,-22.03 965.87,-20.33 965.87,-20.33 1019.26,-3.34 1019.26,-3.34 1024.6,-1.64 1035.56,0.06 1041.17,0.06 1041.17,0.06 1124.46,0.06 1124.46,0.06 1130.07,0.06 1141.03,-1.64 1146.37,-3.34 1146.37,-3.34 1199.76,-20.33 1199.76,-20.33 1205.1,-22.03 1210.45,-29.35 1210.45,-34.95"/>
|
||||
<text text-anchor="middle" x="1082.82" y="-51.86" font-family="Helvetica,sans-Serif" font-size="14.00">evicted dave</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-36.86" font-family="Helvetica,sans-Serif" font-size="14.00">no slot in envelope</text>
|
||||
<text text-anchor="middle" x="1082.82" y="-21.86" font-family="Helvetica,sans-Serif" font-size="14.00">ValueError on unwrap</text>
|
||||
</g>
|
||||
<!-- envelope->evicted -->
|
||||
<g id="edge13" class="edge">
|
||||
<title>envelope->evicted</title>
|
||||
<path fill="none" stroke="#cc0000" stroke-dasharray="5,2" d="M789.7,-143.51C835.35,-125.3 888.01,-104.87 936.32,-87.56 950.77,-82.39 966.12,-77.15 981.22,-72.15"/>
|
||||
<polygon fill="#cc0000" stroke="#cc0000" points="982.69,-75.34 991.09,-68.89 980.5,-68.7 982.69,-75.34"/>
|
||||
<text text-anchor="middle" x="872.5" y="-128.56" font-family="Helvetica,sans-Serif" font-size="10.00">dave NOT included</text>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 18 KiB |
70
docs/diagrams/query-pipeline.dot
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
// Query pipeline: how a question becomes a verified answer.
|
||||
//
|
||||
// Per-stage timing budget at the top of each box (Hermes ~6-8s
|
||||
// dominates; everything else target sub-second per phase).
|
||||
//
|
||||
// Render: dot -Tsvg query-pipeline.dot -o query-pipeline.svg
|
||||
|
||||
digraph query_pipeline {
|
||||
rankdir=TB
|
||||
node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10]
|
||||
edge [fontname="Helvetica", fontsize=9]
|
||||
bgcolor="white"
|
||||
|
||||
question [label="QUESTION\n\"what tech can reconstruct\nanother person's thoughts?\"", fillcolor="#fff7e6", shape=note]
|
||||
|
||||
// Cache lookup first (free hit if previously asked)
|
||||
cache_check [label="cache lookup\nproperence_cache by 8-dim cache_key\n• equivalence_class fallback\n• ~0.3ms warm", fillcolor="#e6f0ff"]
|
||||
cache_hit [label="CACHE HIT\nreplay answer + Merkle proof\n~100ms total", fillcolor="#d4edda", shape=note]
|
||||
|
||||
// Retrieval
|
||||
fts5 [label="FTS5 BM25 search\n• AND-mode first (strict)\n• OR-mode fallback (top-5 longest +\n synonym pool)\n• ~0.25s/shard", fillcolor="#ffe6f0"]
|
||||
title_like [label="title-LIKE backup\n(skipped if >5 tokens)\n• ~0.5s/shard for short queries", fillcolor="#ffe6f0"]
|
||||
phrase_route [label="phrase-pattern route\n4-token sliding-window match\n(closes allusion gap)", fillcolor="#ffe6f0"]
|
||||
concepts_lookup [label="concepts/ overlay\n• synonym_expand(qtokens)\n• rivalry_excluded()\n• per-process cache, ~1ms warm", fillcolor="#ffe6f0"]
|
||||
|
||||
// Filter
|
||||
filter [label="title-relevance filter\n4 accept paths:\n1. title-token overlap\n2. TF-IDF core match\n3. body density\n4. phrase match", fillcolor="#ffe6f0"]
|
||||
rerank [label="rerank stack\n• body coverage (sqrt)\n• title token boost\n• source role boost\n• title purity", fillcolor="#ffe6f0"]
|
||||
|
||||
// Evidence map (claim_lattice modes)
|
||||
evidence_map [label="evidence map\n(claim_lattice modes)\nEvidenceObject per chunk\nrun-stable evidence_id (sha256)\nprompt-facing pointer_id (E1, E2…)", fillcolor="#fff0e0"]
|
||||
|
||||
// Context assembly
|
||||
context [label="context assembly\n• per-source cap\n• 60KB total budget\n• wikitext.to_base() prose-strip", fillcolor="#fff0e0"]
|
||||
|
||||
// LLM
|
||||
llm [label="LLM call (Hermes-3 / vLLM)\n6–8s — DOMINANT cost\nclaim_lattice_pointer or\nclaim_lattice (JSON)", fillcolor="#ffd4d4"]
|
||||
|
||||
// Verification
|
||||
verify [label="verify_claim_lattice\n7 hard checks:\n1. parser succeeded\n2. evidence_id resolves\n3. source_role allowed\n4. claim text non-empty\n5. citation coverage\n6. pointer count cap\n7. anchor-class warrant", fillcolor="#e7ffe7"]
|
||||
warrant [label="warrant_check (5 anchor classes)\n• proper-noun (relation-gated)\n• date (year + month)\n• count (digit↔word)\n• entity-list\n• cause (why-shape)", fillcolor="#e7ffe7"]
|
||||
|
||||
// Render
|
||||
render [label="render\n• spotlight excerpt (density rank)\n• [E5 | Title | chunk_prefix: \"...\"]\n• four-rung label ladder", fillcolor="#fff7e6"]
|
||||
|
||||
// Persistence
|
||||
cache_write [label="providence_cache write\n• 8-dim cache_key\n• run_dag_root + run_dag_blob\n• audit_event chained\n• context_root committed", fillcolor="#e6f0ff"]
|
||||
|
||||
answer [label="ANSWER\nlabel: POINTER-LINKED-PARTIAL ·\nvia claim_lattice · 2/2 · 11.6s", fillcolor="#d4edda", shape=note]
|
||||
|
||||
// Edges
|
||||
question -> cache_check
|
||||
cache_check -> cache_hit [label="match"]
|
||||
cache_check -> concepts_lookup [label="miss"]
|
||||
concepts_lookup -> fts5
|
||||
concepts_lookup -> title_like
|
||||
concepts_lookup -> phrase_route
|
||||
fts5 -> filter
|
||||
title_like -> filter
|
||||
phrase_route -> filter
|
||||
filter -> rerank
|
||||
rerank -> evidence_map
|
||||
evidence_map -> context
|
||||
context -> llm
|
||||
llm -> verify
|
||||
verify -> warrant
|
||||
warrant -> render
|
||||
render -> cache_write
|
||||
cache_write -> answer
|
||||
}
|
||||
BIN
docs/diagrams/query-pipeline.png
Normal file
|
After Width: | Height: | Size: 164 KiB |
290
docs/diagrams/query-pipeline.svg
Normal file
|
|
@ -0,0 +1,290 @@
|
|||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<!-- Generated by graphviz version 2.43.0 (0)
|
||||
-->
|
||||
<!-- Title: query_pipeline Pages: 1 -->
|
||||
<svg width="571pt" height="1347pt"
|
||||
viewBox="0.00 0.00 571.00 1347.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 1343)">
|
||||
<title>query_pipeline</title>
|
||||
<polygon fill="white" stroke="transparent" points="-4,4 -4,-1343 567,-1343 567,4 -4,4"/>
|
||||
<!-- question -->
|
||||
<g id="node1" class="node">
|
||||
<title>question</title>
|
||||
<polygon fill="#fff7e6" stroke="black" points="205,-1339 57,-1339 57,-1298 211,-1298 211,-1333 205,-1339"/>
|
||||
<polyline fill="none" stroke="black" points="205,-1339 205,-1333 "/>
|
||||
<polyline fill="none" stroke="black" points="211,-1333 205,-1333 "/>
|
||||
<text text-anchor="middle" x="134" y="-1327" font-family="Helvetica,sans-Serif" font-size="10.00">QUESTION</text>
|
||||
<text text-anchor="middle" x="134" y="-1316" font-family="Helvetica,sans-Serif" font-size="10.00">"what tech can reconstruct</text>
|
||||
<text text-anchor="middle" x="134" y="-1305" font-family="Helvetica,sans-Serif" font-size="10.00">another person's thoughts?"</text>
|
||||
</g>
|
||||
<!-- cache_check -->
|
||||
<g id="node2" class="node">
|
||||
<title>cache_check</title>
|
||||
<path fill="#e6f0ff" stroke="black" d="M225,-1261C225,-1261 43,-1261 43,-1261 37,-1261 31,-1255 31,-1249 31,-1249 31,-1221 31,-1221 31,-1215 37,-1209 43,-1209 43,-1209 225,-1209 225,-1209 231,-1209 237,-1215 237,-1221 237,-1221 237,-1249 237,-1249 237,-1255 231,-1261 225,-1261"/>
|
||||
<text text-anchor="middle" x="134" y="-1249" font-family="Helvetica,sans-Serif" font-size="10.00">cache lookup</text>
|
||||
<text text-anchor="middle" x="134" y="-1238" font-family="Helvetica,sans-Serif" font-size="10.00">properence_cache by 8-dim cache_key</text>
|
||||
<text text-anchor="middle" x="134" y="-1227" font-family="Helvetica,sans-Serif" font-size="10.00">• equivalence_class fallback</text>
|
||||
<text text-anchor="middle" x="134" y="-1216" font-family="Helvetica,sans-Serif" font-size="10.00">• ~0.3ms warm</text>
|
||||
</g>
|
||||
<!-- question->cache_check -->
|
||||
<g id="edge1" class="edge">
|
||||
<title>question->cache_check</title>
|
||||
<path fill="none" stroke="black" d="M134,-1297.77C134,-1289.76 134,-1280.26 134,-1271.14"/>
|
||||
<polygon fill="black" stroke="black" points="137.5,-1271.11 134,-1261.11 130.5,-1271.11 137.5,-1271.11"/>
|
||||
</g>
|
||||
<!-- cache_hit -->
|
||||
<g id="node3" class="node">
|
||||
<title>cache_hit</title>
|
||||
<polygon fill="#d4edda" stroke="black" points="183,-1157.5 27,-1157.5 27,-1116.5 189,-1116.5 189,-1151.5 183,-1157.5"/>
|
||||
<polyline fill="none" stroke="black" points="183,-1157.5 183,-1151.5 "/>
|
||||
<polyline fill="none" stroke="black" points="189,-1151.5 183,-1151.5 "/>
|
||||
<text text-anchor="middle" x="108" y="-1145.5" font-family="Helvetica,sans-Serif" font-size="10.00">CACHE HIT</text>
|
||||
<text text-anchor="middle" x="108" y="-1134.5" font-family="Helvetica,sans-Serif" font-size="10.00">replay answer + Merkle proof</text>
|
||||
<text text-anchor="middle" x="108" y="-1123.5" font-family="Helvetica,sans-Serif" font-size="10.00">~100ms total</text>
|
||||
</g>
|
||||
<!-- cache_check->cache_hit -->
|
||||
<g id="edge2" class="edge">
|
||||
<title>cache_check->cache_hit</title>
|
||||
<path fill="none" stroke="black" d="M127.17,-1208.78C123.75,-1196.15 119.59,-1180.79 116.02,-1167.61"/>
|
||||
<polygon fill="black" stroke="black" points="119.3,-1166.34 113.31,-1157.6 112.54,-1168.17 119.3,-1166.34"/>
|
||||
<text text-anchor="middle" x="137" y="-1183.8" font-family="Helvetica,sans-Serif" font-size="9.00">match</text>
|
||||
</g>
|
||||
<!-- concepts_lookup -->
|
||||
<g id="node7" class="node">
|
||||
<title>concepts_lookup</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M375,-1163C375,-1163 219,-1163 219,-1163 213,-1163 207,-1157 207,-1151 207,-1151 207,-1123 207,-1123 207,-1117 213,-1111 219,-1111 219,-1111 375,-1111 375,-1111 381,-1111 387,-1117 387,-1123 387,-1123 387,-1151 387,-1151 387,-1157 381,-1163 375,-1163"/>
|
||||
<text text-anchor="middle" x="297" y="-1151" font-family="Helvetica,sans-Serif" font-size="10.00">concepts/ overlay</text>
|
||||
<text text-anchor="middle" x="297" y="-1140" font-family="Helvetica,sans-Serif" font-size="10.00">• synonym_expand(qtokens)</text>
|
||||
<text text-anchor="middle" x="297" y="-1129" font-family="Helvetica,sans-Serif" font-size="10.00">• rivalry_excluded()</text>
|
||||
<text text-anchor="middle" x="297" y="-1118" font-family="Helvetica,sans-Serif" font-size="10.00">• per-process cache, ~1ms warm</text>
|
||||
</g>
|
||||
<!-- cache_check->concepts_lookup -->
|
||||
<g id="edge3" class="edge">
|
||||
<title>cache_check->concepts_lookup</title>
|
||||
<path fill="none" stroke="black" d="M176.83,-1208.78C197.98,-1196.32 223.65,-1181.2 245.83,-1168.14"/>
|
||||
<polygon fill="black" stroke="black" points="247.62,-1171.15 254.46,-1163.05 244.07,-1165.11 247.62,-1171.15"/>
|
||||
<text text-anchor="middle" x="231.5" y="-1183.8" font-family="Helvetica,sans-Serif" font-size="9.00">miss</text>
|
||||
</g>
|
||||
<!-- fts5 -->
|
||||
<g id="node4" class="node">
|
||||
<title>fts5</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M184,-1074C184,-1074 12,-1074 12,-1074 6,-1074 0,-1068 0,-1062 0,-1062 0,-1023 0,-1023 0,-1017 6,-1011 12,-1011 12,-1011 184,-1011 184,-1011 190,-1011 196,-1017 196,-1023 196,-1023 196,-1062 196,-1062 196,-1068 190,-1074 184,-1074"/>
|
||||
<text text-anchor="middle" x="98" y="-1062" font-family="Helvetica,sans-Serif" font-size="10.00">FTS5 BM25 search</text>
|
||||
<text text-anchor="middle" x="98" y="-1051" font-family="Helvetica,sans-Serif" font-size="10.00">• AND-mode first (strict)</text>
|
||||
<text text-anchor="middle" x="98" y="-1040" font-family="Helvetica,sans-Serif" font-size="10.00">• OR-mode fallback (top-5 longest +</text>
|
||||
<text text-anchor="middle" x="98" y="-1029" font-family="Helvetica,sans-Serif" font-size="10.00">  synonym pool)</text>
|
||||
<text text-anchor="middle" x="98" y="-1018" font-family="Helvetica,sans-Serif" font-size="10.00">• ~0.25s/shard</text>
|
||||
</g>
|
||||
<!-- filter -->
|
||||
<g id="node8" class="node">
|
||||
<title>filter</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M344.5,-974C344.5,-974 249.5,-974 249.5,-974 243.5,-974 237.5,-968 237.5,-962 237.5,-962 237.5,-912 237.5,-912 237.5,-906 243.5,-900 249.5,-900 249.5,-900 344.5,-900 344.5,-900 350.5,-900 356.5,-906 356.5,-912 356.5,-912 356.5,-962 356.5,-962 356.5,-968 350.5,-974 344.5,-974"/>
|
||||
<text text-anchor="middle" x="297" y="-962" font-family="Helvetica,sans-Serif" font-size="10.00">title-relevance filter</text>
|
||||
<text text-anchor="middle" x="297" y="-951" font-family="Helvetica,sans-Serif" font-size="10.00">4 accept paths:</text>
|
||||
<text text-anchor="middle" x="297" y="-940" font-family="Helvetica,sans-Serif" font-size="10.00">1. title-token overlap</text>
|
||||
<text text-anchor="middle" x="297" y="-929" font-family="Helvetica,sans-Serif" font-size="10.00">2. TF-IDF core match</text>
|
||||
<text text-anchor="middle" x="297" y="-918" font-family="Helvetica,sans-Serif" font-size="10.00">3. body density</text>
|
||||
<text text-anchor="middle" x="297" y="-907" font-family="Helvetica,sans-Serif" font-size="10.00">4. phrase match</text>
|
||||
</g>
|
||||
<!-- fts5->filter -->
|
||||
<g id="edge7" class="edge">
|
||||
<title>fts5->filter</title>
|
||||
<path fill="none" stroke="black" d="M156.9,-1010.87C179.26,-999.24 204.92,-985.89 228.12,-973.82"/>
|
||||
<polygon fill="black" stroke="black" points="229.96,-976.81 237.22,-969.09 226.73,-970.6 229.96,-976.81"/>
|
||||
</g>
|
||||
<!-- title_like -->
|
||||
<g id="node5" class="node">
|
||||
<title>title_like</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M368.5,-1063C368.5,-1063 225.5,-1063 225.5,-1063 219.5,-1063 213.5,-1057 213.5,-1051 213.5,-1051 213.5,-1034 213.5,-1034 213.5,-1028 219.5,-1022 225.5,-1022 225.5,-1022 368.5,-1022 368.5,-1022 374.5,-1022 380.5,-1028 380.5,-1034 380.5,-1034 380.5,-1051 380.5,-1051 380.5,-1057 374.5,-1063 368.5,-1063"/>
|
||||
<text text-anchor="middle" x="297" y="-1051" font-family="Helvetica,sans-Serif" font-size="10.00">title-LIKE backup</text>
|
||||
<text text-anchor="middle" x="297" y="-1040" font-family="Helvetica,sans-Serif" font-size="10.00">(skipped if >5 tokens)</text>
|
||||
<text text-anchor="middle" x="297" y="-1029" font-family="Helvetica,sans-Serif" font-size="10.00">• ~0.5s/shard for short queries</text>
|
||||
</g>
|
||||
<!-- title_like->filter -->
|
||||
<g id="edge8" class="edge">
|
||||
<title>title_like->filter</title>
|
||||
<path fill="none" stroke="black" d="M297,-1021.54C297,-1010.98 297,-997.52 297,-984.51"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-984.24 297,-974.24 293.5,-984.24 300.5,-984.24"/>
|
||||
</g>
|
||||
<!-- phrase_route -->
|
||||
<g id="node6" class="node">
|
||||
<title>phrase_route</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M551,-1063C551,-1063 411,-1063 411,-1063 405,-1063 399,-1057 399,-1051 399,-1051 399,-1034 399,-1034 399,-1028 405,-1022 411,-1022 411,-1022 551,-1022 551,-1022 557,-1022 563,-1028 563,-1034 563,-1034 563,-1051 563,-1051 563,-1057 557,-1063 551,-1063"/>
|
||||
<text text-anchor="middle" x="481" y="-1051" font-family="Helvetica,sans-Serif" font-size="10.00">phrase-pattern route</text>
|
||||
<text text-anchor="middle" x="481" y="-1040" font-family="Helvetica,sans-Serif" font-size="10.00">4-token sliding-window match</text>
|
||||
<text text-anchor="middle" x="481" y="-1029" font-family="Helvetica,sans-Serif" font-size="10.00">(closes allusion gap)</text>
|
||||
</g>
|
||||
<!-- phrase_route->filter -->
|
||||
<g id="edge9" class="edge">
|
||||
<title>phrase_route->filter</title>
|
||||
<path fill="none" stroke="black" d="M445.96,-1021.79C423.19,-1008.98 392.75,-991.86 365.32,-976.43"/>
|
||||
<polygon fill="black" stroke="black" points="366.96,-973.34 356.53,-971.49 363.53,-979.44 366.96,-973.34"/>
|
||||
</g>
|
||||
<!-- concepts_lookup->fts5 -->
|
||||
<g id="edge4" class="edge">
|
||||
<title>concepts_lookup->fts5</title>
|
||||
<path fill="none" stroke="black" d="M243.15,-1110.97C221.6,-1100.95 196.42,-1089.25 173.13,-1078.42"/>
|
||||
<polygon fill="black" stroke="black" points="174.51,-1075.2 163.97,-1074.16 171.56,-1081.55 174.51,-1075.2"/>
|
||||
</g>
|
||||
<!-- concepts_lookup->title_like -->
|
||||
<g id="edge5" class="edge">
|
||||
<title>concepts_lookup->title_like</title>
|
||||
<path fill="none" stroke="black" d="M297,-1110.72C297,-1099.14 297,-1085.33 297,-1073.25"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-1073.17 297,-1063.17 293.5,-1073.17 300.5,-1073.17"/>
|
||||
</g>
|
||||
<!-- concepts_lookup->phrase_route -->
|
||||
<g id="edge6" class="edge">
|
||||
<title>concepts_lookup->phrase_route</title>
|
||||
<path fill="none" stroke="black" d="M346.79,-1110.97C373.55,-1097.52 406.38,-1081.01 432.87,-1067.69"/>
|
||||
<polygon fill="black" stroke="black" points="434.6,-1070.75 441.96,-1063.13 431.45,-1064.49 434.6,-1070.75"/>
|
||||
</g>
|
||||
<!-- rerank -->
|
||||
<g id="node9" class="node">
|
||||
<title>rerank</title>
|
||||
<path fill="#ffe6f0" stroke="black" d="M348.5,-863C348.5,-863 245.5,-863 245.5,-863 239.5,-863 233.5,-857 233.5,-851 233.5,-851 233.5,-812 233.5,-812 233.5,-806 239.5,-800 245.5,-800 245.5,-800 348.5,-800 348.5,-800 354.5,-800 360.5,-806 360.5,-812 360.5,-812 360.5,-851 360.5,-851 360.5,-857 354.5,-863 348.5,-863"/>
|
||||
<text text-anchor="middle" x="297" y="-851" font-family="Helvetica,sans-Serif" font-size="10.00">rerank stack</text>
|
||||
<text text-anchor="middle" x="297" y="-840" font-family="Helvetica,sans-Serif" font-size="10.00">• body coverage (sqrt)</text>
|
||||
<text text-anchor="middle" x="297" y="-829" font-family="Helvetica,sans-Serif" font-size="10.00">• title token boost</text>
|
||||
<text text-anchor="middle" x="297" y="-818" font-family="Helvetica,sans-Serif" font-size="10.00">• source role boost</text>
|
||||
<text text-anchor="middle" x="297" y="-807" font-family="Helvetica,sans-Serif" font-size="10.00">• title purity</text>
|
||||
</g>
|
||||
<!-- filter->rerank -->
|
||||
<g id="edge10" class="edge">
|
||||
<title>filter->rerank</title>
|
||||
<path fill="none" stroke="black" d="M297,-899.8C297,-891.17 297,-881.9 297,-873.09"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-873.02 297,-863.02 293.5,-873.02 300.5,-873.02"/>
|
||||
</g>
|
||||
<!-- evidence_map -->
|
||||
<g id="node10" class="node">
|
||||
<title>evidence_map</title>
|
||||
<path fill="#fff0e0" stroke="black" d="M379,-763C379,-763 215,-763 215,-763 209,-763 203,-757 203,-751 203,-751 203,-712 203,-712 203,-706 209,-700 215,-700 215,-700 379,-700 379,-700 385,-700 391,-706 391,-712 391,-712 391,-751 391,-751 391,-757 385,-763 379,-763"/>
|
||||
<text text-anchor="middle" x="297" y="-751" font-family="Helvetica,sans-Serif" font-size="10.00">evidence map</text>
|
||||
<text text-anchor="middle" x="297" y="-740" font-family="Helvetica,sans-Serif" font-size="10.00">(claim_lattice modes)</text>
|
||||
<text text-anchor="middle" x="297" y="-729" font-family="Helvetica,sans-Serif" font-size="10.00">EvidenceObject per chunk</text>
|
||||
<text text-anchor="middle" x="297" y="-718" font-family="Helvetica,sans-Serif" font-size="10.00">run-stable evidence_id (sha256)</text>
|
||||
<text text-anchor="middle" x="297" y="-707" font-family="Helvetica,sans-Serif" font-size="10.00">prompt-facing pointer_id (E1, E2…)</text>
|
||||
</g>
|
||||
<!-- rerank->evidence_map -->
|
||||
<g id="edge11" class="edge">
|
||||
<title>rerank->evidence_map</title>
|
||||
<path fill="none" stroke="black" d="M297,-799.76C297,-791.37 297,-782.14 297,-773.29"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-773.13 297,-763.13 293.5,-773.13 300.5,-773.13"/>
|
||||
</g>
|
||||
<!-- context -->
|
||||
<g id="node11" class="node">
|
||||
<title>context</title>
|
||||
<path fill="#fff0e0" stroke="black" d="M370,-663C370,-663 224,-663 224,-663 218,-663 212,-657 212,-651 212,-651 212,-623 212,-623 212,-617 218,-611 224,-611 224,-611 370,-611 370,-611 376,-611 382,-617 382,-623 382,-623 382,-651 382,-651 382,-657 376,-663 370,-663"/>
|
||||
<text text-anchor="middle" x="297" y="-651" font-family="Helvetica,sans-Serif" font-size="10.00">context assembly</text>
|
||||
<text text-anchor="middle" x="297" y="-640" font-family="Helvetica,sans-Serif" font-size="10.00">• per-source cap</text>
|
||||
<text text-anchor="middle" x="297" y="-629" font-family="Helvetica,sans-Serif" font-size="10.00">• 60KB total budget</text>
|
||||
<text text-anchor="middle" x="297" y="-618" font-family="Helvetica,sans-Serif" font-size="10.00">• wikitext.to_base() prose-strip</text>
|
||||
</g>
|
||||
<!-- evidence_map->context -->
|
||||
<g id="edge12" class="edge">
|
||||
<title>evidence_map->context</title>
|
||||
<path fill="none" stroke="black" d="M297,-699.96C297,-691.5 297,-682.25 297,-673.54"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-673.27 297,-663.27 293.5,-673.27 300.5,-673.27"/>
|
||||
</g>
|
||||
<!-- llm -->
|
||||
<g id="node12" class="node">
|
||||
<title>llm</title>
|
||||
<path fill="#ffd4d4" stroke="black" d="M359.5,-574C359.5,-574 234.5,-574 234.5,-574 228.5,-574 222.5,-568 222.5,-562 222.5,-562 222.5,-534 222.5,-534 222.5,-528 228.5,-522 234.5,-522 234.5,-522 359.5,-522 359.5,-522 365.5,-522 371.5,-528 371.5,-534 371.5,-534 371.5,-562 371.5,-562 371.5,-568 365.5,-574 359.5,-574"/>
|
||||
<text text-anchor="middle" x="297" y="-562" font-family="Helvetica,sans-Serif" font-size="10.00">LLM call (Hermes-3 / vLLM)</text>
|
||||
<text text-anchor="middle" x="297" y="-551" font-family="Helvetica,sans-Serif" font-size="10.00">6–8s — DOMINANT cost</text>
|
||||
<text text-anchor="middle" x="297" y="-540" font-family="Helvetica,sans-Serif" font-size="10.00">claim_lattice_pointer or</text>
|
||||
<text text-anchor="middle" x="297" y="-529" font-family="Helvetica,sans-Serif" font-size="10.00">claim_lattice (JSON)</text>
|
||||
</g>
|
||||
<!-- context->llm -->
|
||||
<g id="edge13" class="edge">
|
||||
<title>context->llm</title>
|
||||
<path fill="none" stroke="black" d="M297,-610.84C297,-602.54 297,-593.15 297,-584.25"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-584.11 297,-574.11 293.5,-584.11 300.5,-584.11"/>
|
||||
</g>
|
||||
<!-- verify -->
|
||||
<g id="node13" class="node">
|
||||
<title>verify</title>
|
||||
<path fill="#e7ffe7" stroke="black" d="M352,-485C352,-485 242,-485 242,-485 236,-485 230,-479 230,-473 230,-473 230,-390 230,-390 230,-384 236,-378 242,-378 242,-378 352,-378 352,-378 358,-378 364,-384 364,-390 364,-390 364,-473 364,-473 364,-479 358,-485 352,-485"/>
|
||||
<text text-anchor="middle" x="297" y="-473" font-family="Helvetica,sans-Serif" font-size="10.00">verify_claim_lattice</text>
|
||||
<text text-anchor="middle" x="297" y="-462" font-family="Helvetica,sans-Serif" font-size="10.00">7 hard checks:</text>
|
||||
<text text-anchor="middle" x="297" y="-451" font-family="Helvetica,sans-Serif" font-size="10.00">1. parser succeeded</text>
|
||||
<text text-anchor="middle" x="297" y="-440" font-family="Helvetica,sans-Serif" font-size="10.00">2. evidence_id resolves</text>
|
||||
<text text-anchor="middle" x="297" y="-429" font-family="Helvetica,sans-Serif" font-size="10.00">3. source_role allowed</text>
|
||||
<text text-anchor="middle" x="297" y="-418" font-family="Helvetica,sans-Serif" font-size="10.00">4. claim text non-empty</text>
|
||||
<text text-anchor="middle" x="297" y="-407" font-family="Helvetica,sans-Serif" font-size="10.00">5. citation coverage</text>
|
||||
<text text-anchor="middle" x="297" y="-396" font-family="Helvetica,sans-Serif" font-size="10.00">6. pointer count cap</text>
|
||||
<text text-anchor="middle" x="297" y="-385" font-family="Helvetica,sans-Serif" font-size="10.00">7. anchor-class warrant</text>
|
||||
</g>
|
||||
<!-- llm->verify -->
|
||||
<g id="edge14" class="edge">
|
||||
<title>llm->verify</title>
|
||||
<path fill="none" stroke="black" d="M297,-521.87C297,-513.94 297,-504.8 297,-495.45"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-495.28 297,-485.28 293.5,-495.28 300.5,-495.28"/>
|
||||
</g>
|
||||
<!-- warrant -->
|
||||
<g id="node14" class="node">
|
||||
<title>warrant</title>
|
||||
<path fill="#e7ffe7" stroke="black" d="M374.5,-341C374.5,-341 219.5,-341 219.5,-341 213.5,-341 207.5,-335 207.5,-329 207.5,-329 207.5,-279 207.5,-279 207.5,-273 213.5,-267 219.5,-267 219.5,-267 374.5,-267 374.5,-267 380.5,-267 386.5,-273 386.5,-279 386.5,-279 386.5,-329 386.5,-329 386.5,-335 380.5,-341 374.5,-341"/>
|
||||
<text text-anchor="middle" x="297" y="-329" font-family="Helvetica,sans-Serif" font-size="10.00">warrant_check (5 anchor classes)</text>
|
||||
<text text-anchor="middle" x="297" y="-318" font-family="Helvetica,sans-Serif" font-size="10.00">• proper-noun (relation-gated)</text>
|
||||
<text text-anchor="middle" x="297" y="-307" font-family="Helvetica,sans-Serif" font-size="10.00">• date (year + month)</text>
|
||||
<text text-anchor="middle" x="297" y="-296" font-family="Helvetica,sans-Serif" font-size="10.00">• count (digit↔word)</text>
|
||||
<text text-anchor="middle" x="297" y="-285" font-family="Helvetica,sans-Serif" font-size="10.00">• entity-list</text>
|
||||
<text text-anchor="middle" x="297" y="-274" font-family="Helvetica,sans-Serif" font-size="10.00">• cause (why-shape)</text>
|
||||
</g>
|
||||
<!-- verify->warrant -->
|
||||
<g id="edge15" class="edge">
|
||||
<title>verify->warrant</title>
|
||||
<path fill="none" stroke="black" d="M297,-377.64C297,-368.88 297,-359.85 297,-351.27"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-351.06 297,-341.06 293.5,-351.06 300.5,-351.06"/>
|
||||
</g>
|
||||
<!-- render -->
|
||||
<g id="node15" class="node">
|
||||
<title>render</title>
|
||||
<path fill="#fff7e6" stroke="black" d="M374.5,-230C374.5,-230 219.5,-230 219.5,-230 213.5,-230 207.5,-224 207.5,-218 207.5,-218 207.5,-190 207.5,-190 207.5,-184 213.5,-178 219.5,-178 219.5,-178 374.5,-178 374.5,-178 380.5,-178 386.5,-184 386.5,-190 386.5,-190 386.5,-218 386.5,-218 386.5,-224 380.5,-230 374.5,-230"/>
|
||||
<text text-anchor="middle" x="297" y="-218" font-family="Helvetica,sans-Serif" font-size="10.00">render</text>
|
||||
<text text-anchor="middle" x="297" y="-207" font-family="Helvetica,sans-Serif" font-size="10.00">• spotlight excerpt (density rank)</text>
|
||||
<text text-anchor="middle" x="297" y="-196" font-family="Helvetica,sans-Serif" font-size="10.00">• [E5 | Title | chunk_prefix: "..."]</text>
|
||||
<text text-anchor="middle" x="297" y="-185" font-family="Helvetica,sans-Serif" font-size="10.00">• four-rung label ladder</text>
|
||||
</g>
|
||||
<!-- warrant->render -->
|
||||
<g id="edge16" class="edge">
|
||||
<title>warrant->render</title>
|
||||
<path fill="none" stroke="black" d="M297,-266.79C297,-258.1 297,-248.84 297,-240.21"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-240.06 297,-230.06 293.5,-240.06 300.5,-240.06"/>
|
||||
</g>
|
||||
<!-- cache_write -->
|
||||
<g id="node16" class="node">
|
||||
<title>cache_write</title>
|
||||
<path fill="#e6f0ff" stroke="black" d="M369.5,-141C369.5,-141 224.5,-141 224.5,-141 218.5,-141 212.5,-135 212.5,-129 212.5,-129 212.5,-90 212.5,-90 212.5,-84 218.5,-78 224.5,-78 224.5,-78 369.5,-78 369.5,-78 375.5,-78 381.5,-84 381.5,-90 381.5,-90 381.5,-129 381.5,-129 381.5,-135 375.5,-141 369.5,-141"/>
|
||||
<text text-anchor="middle" x="297" y="-129" font-family="Helvetica,sans-Serif" font-size="10.00">providence_cache write</text>
|
||||
<text text-anchor="middle" x="297" y="-118" font-family="Helvetica,sans-Serif" font-size="10.00">• 8-dim cache_key</text>
|
||||
<text text-anchor="middle" x="297" y="-107" font-family="Helvetica,sans-Serif" font-size="10.00">• run_dag_root + run_dag_blob</text>
|
||||
<text text-anchor="middle" x="297" y="-96" font-family="Helvetica,sans-Serif" font-size="10.00">• audit_event chained</text>
|
||||
<text text-anchor="middle" x="297" y="-85" font-family="Helvetica,sans-Serif" font-size="10.00">• context_root committed</text>
|
||||
</g>
|
||||
<!-- render->cache_write -->
|
||||
<g id="edge17" class="edge">
|
||||
<title>render->cache_write</title>
|
||||
<path fill="none" stroke="black" d="M297,-177.72C297,-169.48 297,-160.12 297,-151.05"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-151.01 297,-141.01 293.5,-151.01 300.5,-151.01"/>
|
||||
</g>
|
||||
<!-- answer -->
|
||||
<g id="node17" class="node">
|
||||
<title>answer</title>
|
||||
<polygon fill="#d4edda" stroke="black" points="380,-41 208,-41 208,0 386,0 386,-35 380,-41"/>
|
||||
<polyline fill="none" stroke="black" points="380,-41 380,-35 "/>
|
||||
<polyline fill="none" stroke="black" points="386,-35 380,-35 "/>
|
||||
<text text-anchor="middle" x="297" y="-29" font-family="Helvetica,sans-Serif" font-size="10.00">ANSWER</text>
|
||||
<text text-anchor="middle" x="297" y="-18" font-family="Helvetica,sans-Serif" font-size="10.00">label: POINTER-LINKED-PARTIAL ·</text>
|
||||
<text text-anchor="middle" x="297" y="-7" font-family="Helvetica,sans-Serif" font-size="10.00">via claim_lattice · 2/2 · 11.6s</text>
|
||||
</g>
|
||||
<!-- cache_write->answer -->
|
||||
<g id="edge18" class="edge">
|
||||
<title>cache_write->answer</title>
|
||||
<path fill="none" stroke="black" d="M297,-77.58C297,-69.07 297,-59.84 297,-51.39"/>
|
||||
<polygon fill="black" stroke="black" points="300.5,-51.19 297,-41.19 293.5,-51.19 300.5,-51.19"/>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 22 KiB |
49
docs/diagrams/verifier-ladder.dot
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
// Verifier label ladder: how (audit_mode, violations) → display rung.
|
||||
//
|
||||
// Schema column `audit_mode` keeps the v9.8 trichotomy
|
||||
// (STRICT / HYBRID / UNGROUNDED). The display ladder is a renderer-
|
||||
// only transformation that names what's actually proved at each rung.
|
||||
//
|
||||
// Render: dot -Tsvg verifier-ladder.dot -o verifier-ladder.svg
|
||||
|
||||
digraph verifier_ladder {
|
||||
rankdir=TB
|
||||
node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=10]
|
||||
edge [fontname="Helvetica", fontsize=9]
|
||||
bgcolor="white"
|
||||
|
||||
// Inputs
|
||||
inputs [label="VERIFIER OUTPUT\naudit_mode ∈ {STRICT, HYBRID, UNGROUNDED}\nviolations: list[dict]", fillcolor="#fff7e6", shape=note]
|
||||
|
||||
// Decision tree
|
||||
is_ungrounded [label="audit_mode == UNGROUNDED?", fillcolor="#dddddd", shape=diamond]
|
||||
has_warrant_miss [label="WARRANT_MISSING or\nTITLE_MISMATCH in violations?", fillcolor="#dddddd", shape=diamond]
|
||||
has_soft_demote [label="any soft-demote violation?\n(LAZY_ANCHOR_DEMOTED,\nPOINTER_OVERFLOW_TRIMMED,\nTOO_MANY_CLAIMS, …)", fillcolor="#dddddd", shape=diamond]
|
||||
is_hybrid [label="audit_mode == HYBRID?", fillcolor="#dddddd", shape=diamond]
|
||||
|
||||
// Rungs (top to bottom = strongest to weakest)
|
||||
evidence_warranted [label="EVIDENCE-WARRANTED\npointer verified +\nwarrant ran & passed +\nNO soft demotes", fillcolor="#28a745", fontcolor="white"]
|
||||
anchor_warranted [label="ANCHOR-WARRANTED\npointer-linked +\nwarrant passed where it ran;\nsoft demotes present", fillcolor="#7cc66e"]
|
||||
pointer_linked [label="POINTER-LINKED\npointer/source/chunk verified;\nwarrant didn't apply OR failed\nfor at least one claim", fillcolor="#ffc107"]
|
||||
ungrounded [label="UNGROUNDED\nno verified pairs\nat all", fillcolor="#dc3545", fontcolor="white"]
|
||||
|
||||
// Suffix annotation
|
||||
partial [label="HYBRID → add `-PARTIAL` suffix\non whichever rung applies", fillcolor="#fff0e0", shape=note]
|
||||
|
||||
// Decision flow
|
||||
inputs -> is_ungrounded
|
||||
is_ungrounded -> ungrounded [label="yes"]
|
||||
is_ungrounded -> has_warrant_miss [label="no"]
|
||||
has_warrant_miss -> pointer_linked [label="yes"]
|
||||
has_warrant_miss -> has_soft_demote [label="no"]
|
||||
has_soft_demote -> anchor_warranted [label="yes"]
|
||||
has_soft_demote -> evidence_warranted [label="no"]
|
||||
|
||||
// HYBRID adds suffix to all 3 non-UNGROUNDED rungs
|
||||
evidence_warranted -> is_hybrid [style=dotted]
|
||||
anchor_warranted -> is_hybrid [style=dotted]
|
||||
pointer_linked -> is_hybrid [style=dotted]
|
||||
is_hybrid -> partial [label="yes" style=dotted]
|
||||
|
||||
{rank=same; evidence_warranted; anchor_warranted; pointer_linked; ungrounded}
|
||||
}
|
||||
BIN
docs/diagrams/verifier-ladder.png
Normal file
|
After Width: | Height: | Size: 104 KiB |
168
docs/diagrams/verifier-ladder.svg
Normal file
|
|
@ -0,0 +1,168 @@
|
|||
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
|
||||
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<!-- Generated by graphviz version 2.43.0 (0)
|
||||
-->
|
||||
<!-- Title: verifier_ladder Pages: 1 -->
|
||||
<svg width="634pt" height="631pt"
|
||||
viewBox="0.00 0.00 633.50 631.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 627)">
|
||||
<title>verifier_ladder</title>
|
||||
<polygon fill="white" stroke="transparent" points="-4,4 -4,-627 629.5,-627 629.5,4 -4,4"/>
|
||||
<!-- inputs -->
|
||||
<g id="node1" class="node">
|
||||
<title>inputs</title>
|
||||
<polygon fill="#fff7e6" stroke="black" points="602,-623 350,-623 350,-582 608,-582 608,-617 602,-623"/>
|
||||
<polyline fill="none" stroke="black" points="602,-623 602,-617 "/>
|
||||
<polyline fill="none" stroke="black" points="608,-617 602,-617 "/>
|
||||
<text text-anchor="middle" x="479" y="-611" font-family="Helvetica,sans-Serif" font-size="10.00">VERIFIER OUTPUT</text>
|
||||
<text text-anchor="middle" x="479" y="-600" font-family="Helvetica,sans-Serif" font-size="10.00">audit_mode ∈ {STRICT, HYBRID, UNGROUNDED}</text>
|
||||
<text text-anchor="middle" x="479" y="-589" font-family="Helvetica,sans-Serif" font-size="10.00">violations: list[dict]</text>
|
||||
</g>
|
||||
<!-- is_ungrounded -->
|
||||
<g id="node2" class="node">
|
||||
<title>is_ungrounded</title>
|
||||
<path fill="#dddddd" stroke="black" d="M467.09,-543.53C467.09,-543.53 345.04,-528.47 345.04,-528.47 339.08,-527.73 339.08,-526.27 345.04,-525.53 345.04,-525.53 467.09,-510.47 467.09,-510.47 473.05,-509.73 484.95,-509.73 490.91,-510.47 490.91,-510.47 612.96,-525.53 612.96,-525.53 618.92,-526.27 618.92,-527.73 612.96,-528.47 612.96,-528.47 490.91,-543.53 490.91,-543.53 484.95,-544.27 473.05,-544.27 467.09,-543.53"/>
|
||||
<text text-anchor="middle" x="479" y="-524.5" font-family="Helvetica,sans-Serif" font-size="10.00">audit_mode == UNGROUNDED?</text>
|
||||
</g>
|
||||
<!-- inputs->is_ungrounded -->
|
||||
<g id="edge1" class="edge">
|
||||
<title>inputs->is_ungrounded</title>
|
||||
<path fill="none" stroke="black" d="M479,-581.8C479,-573.7 479,-564.18 479,-555.41"/>
|
||||
<polygon fill="black" stroke="black" points="482.5,-555.25 479,-545.25 475.5,-555.25 482.5,-555.25"/>
|
||||
</g>
|
||||
<!-- has_warrant_miss -->
|
||||
<g id="node3" class="node">
|
||||
<title>has_warrant_miss</title>
|
||||
<path fill="#dddddd" stroke="black" d="M369.18,-460.9C369.18,-460.9 223.82,-435.1 223.82,-435.1 217.91,-434.05 217.91,-431.95 223.82,-430.9 223.82,-430.9 369.18,-405.1 369.18,-405.1 375.09,-404.05 386.91,-404.05 392.82,-405.1 392.82,-405.1 538.18,-430.9 538.18,-430.9 544.09,-431.95 544.09,-434.05 538.18,-435.1 538.18,-435.1 392.82,-460.9 392.82,-460.9 386.91,-461.95 375.09,-461.95 369.18,-460.9"/>
|
||||
<text text-anchor="middle" x="381" y="-436" font-family="Helvetica,sans-Serif" font-size="10.00">WARRANT_MISSING or</text>
|
||||
<text text-anchor="middle" x="381" y="-425" font-family="Helvetica,sans-Serif" font-size="10.00">TITLE_MISMATCH in violations?</text>
|
||||
</g>
|
||||
<!-- is_ungrounded->has_warrant_miss -->
|
||||
<g id="edge3" class="edge">
|
||||
<title>is_ungrounded->has_warrant_miss</title>
|
||||
<path fill="none" stroke="black" d="M462.82,-510.81C449.71,-498.5 430.81,-480.76 414.47,-465.42"/>
|
||||
<polygon fill="black" stroke="black" points="416.86,-462.87 407.18,-458.57 412.07,-467.97 416.86,-462.87"/>
|
||||
<text text-anchor="middle" x="448" y="-483.8" font-family="Helvetica,sans-Serif" font-size="9.00">no</text>
|
||||
</g>
|
||||
<!-- ungrounded -->
|
||||
<g id="node9" class="node">
|
||||
<title>ungrounded</title>
|
||||
<path fill="#dc3545" stroke="black" d="M613.5,-201.5C613.5,-201.5 542.5,-201.5 542.5,-201.5 536.5,-201.5 530.5,-195.5 530.5,-189.5 530.5,-189.5 530.5,-172.5 530.5,-172.5 530.5,-166.5 536.5,-160.5 542.5,-160.5 542.5,-160.5 613.5,-160.5 613.5,-160.5 619.5,-160.5 625.5,-166.5 625.5,-172.5 625.5,-172.5 625.5,-189.5 625.5,-189.5 625.5,-195.5 619.5,-201.5 613.5,-201.5"/>
|
||||
<text text-anchor="middle" x="578" y="-189.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="white">UNGROUNDED</text>
|
||||
<text text-anchor="middle" x="578" y="-178.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="white">no verified pairs</text>
|
||||
<text text-anchor="middle" x="578" y="-167.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="white">at all</text>
|
||||
</g>
|
||||
<!-- is_ungrounded->ungrounded -->
|
||||
<g id="edge2" class="edge">
|
||||
<title>is_ungrounded->ungrounded</title>
|
||||
<path fill="none" stroke="black" d="M506.67,-512.17C524.93,-501.44 547.53,-484.73 559,-463 602.45,-380.67 591.78,-265.38 583.46,-211.69"/>
|
||||
<polygon fill="black" stroke="black" points="586.86,-210.81 581.79,-201.5 579.95,-211.94 586.86,-210.81"/>
|
||||
<text text-anchor="middle" x="593" y="-377.8" font-family="Helvetica,sans-Serif" font-size="9.00">yes</text>
|
||||
</g>
|
||||
<!-- has_soft_demote -->
|
||||
<g id="node4" class="node">
|
||||
<title>has_soft_demote</title>
|
||||
<path fill="#dddddd" stroke="black" d="M182.49,-353.6C182.49,-353.6 29.51,-308.4 29.51,-308.4 23.75,-306.7 23.75,-303.3 29.51,-301.6 29.51,-301.6 182.49,-256.4 182.49,-256.4 188.25,-254.7 199.75,-254.7 205.51,-256.4 205.51,-256.4 358.49,-301.6 358.49,-301.6 364.25,-303.3 364.25,-306.7 358.49,-308.4 358.49,-308.4 205.51,-353.6 205.51,-353.6 199.75,-355.3 188.25,-355.3 182.49,-353.6"/>
|
||||
<text text-anchor="middle" x="194" y="-319" font-family="Helvetica,sans-Serif" font-size="10.00">any soft-demote violation?</text>
|
||||
<text text-anchor="middle" x="194" y="-308" font-family="Helvetica,sans-Serif" font-size="10.00">(LAZY_ANCHOR_DEMOTED,</text>
|
||||
<text text-anchor="middle" x="194" y="-297" font-family="Helvetica,sans-Serif" font-size="10.00">POINTER_OVERFLOW_TRIMMED,</text>
|
||||
<text text-anchor="middle" x="194" y="-286" font-family="Helvetica,sans-Serif" font-size="10.00">TOO_MANY_CLAIMS, …)</text>
|
||||
</g>
|
||||
<!-- has_warrant_miss->has_soft_demote -->
|
||||
<g id="edge5" class="edge">
|
||||
<title>has_warrant_miss->has_soft_demote</title>
|
||||
<path fill="none" stroke="black" d="M346.7,-408.89C321.16,-391.68 285.61,-367.73 255.23,-347.26"/>
|
||||
<polygon fill="black" stroke="black" points="257.07,-344.27 246.82,-341.59 253.15,-350.08 257.07,-344.27"/>
|
||||
<text text-anchor="middle" x="316" y="-377.8" font-family="Helvetica,sans-Serif" font-size="9.00">no</text>
|
||||
</g>
|
||||
<!-- pointer_linked -->
|
||||
<g id="node8" class="node">
|
||||
<title>pointer_linked</title>
|
||||
<path fill="#ffc107" stroke="black" d="M475.5,-207C475.5,-207 336.5,-207 336.5,-207 330.5,-207 324.5,-201 324.5,-195 324.5,-195 324.5,-167 324.5,-167 324.5,-161 330.5,-155 336.5,-155 336.5,-155 475.5,-155 475.5,-155 481.5,-155 487.5,-161 487.5,-167 487.5,-167 487.5,-195 487.5,-195 487.5,-201 481.5,-207 475.5,-207"/>
|
||||
<text text-anchor="middle" x="406" y="-195" font-family="Helvetica,sans-Serif" font-size="10.00">POINTER-LINKED</text>
|
||||
<text text-anchor="middle" x="406" y="-184" font-family="Helvetica,sans-Serif" font-size="10.00">pointer/source/chunk verified;</text>
|
||||
<text text-anchor="middle" x="406" y="-173" font-family="Helvetica,sans-Serif" font-size="10.00">warrant didn't apply OR failed</text>
|
||||
<text text-anchor="middle" x="406" y="-162" font-family="Helvetica,sans-Serif" font-size="10.00">for at least one claim</text>
|
||||
</g>
|
||||
<!-- has_warrant_miss->pointer_linked -->
|
||||
<g id="edge4" class="edge">
|
||||
<title>has_warrant_miss->pointer_linked</title>
|
||||
<path fill="none" stroke="black" d="M383.87,-403.32C388.47,-357.34 397.48,-267.24 402.46,-217.35"/>
|
||||
<polygon fill="black" stroke="black" points="405.95,-217.65 403.47,-207.35 398.99,-216.95 405.95,-217.65"/>
|
||||
<text text-anchor="middle" x="407" y="-302.8" font-family="Helvetica,sans-Serif" font-size="9.00">yes</text>
|
||||
</g>
|
||||
<!-- evidence_warranted -->
|
||||
<g id="node6" class="node">
|
||||
<title>evidence_warranted</title>
|
||||
<path fill="#28a745" stroke="black" d="M120,-207C120,-207 12,-207 12,-207 6,-207 0,-201 0,-195 0,-195 0,-167 0,-167 0,-161 6,-155 12,-155 12,-155 120,-155 120,-155 126,-155 132,-161 132,-167 132,-167 132,-195 132,-195 132,-201 126,-207 120,-207"/>
|
||||
<text text-anchor="middle" x="66" y="-195" font-family="Helvetica,sans-Serif" font-size="10.00" fill="white">EVIDENCE-WARRANTED</text>
|
||||
<text text-anchor="middle" x="66" y="-184" font-family="Helvetica,sans-Serif" font-size="10.00" fill="white">pointer verified +</text>
|
||||
<text text-anchor="middle" x="66" y="-173" font-family="Helvetica,sans-Serif" font-size="10.00" fill="white">warrant ran & passed +</text>
|
||||
<text text-anchor="middle" x="66" y="-162" font-family="Helvetica,sans-Serif" font-size="10.00" fill="white">NO soft demotes</text>
|
||||
</g>
|
||||
<!-- has_soft_demote->evidence_warranted -->
|
||||
<g id="edge7" class="edge">
|
||||
<title>has_soft_demote->evidence_warranted</title>
|
||||
<path fill="none" stroke="black" d="M153.15,-265.06C135.89,-248.61 116.04,-229.69 99.64,-214.07"/>
|
||||
<polygon fill="black" stroke="black" points="102.03,-211.51 92.38,-207.14 97.2,-216.57 102.03,-211.51"/>
|
||||
<text text-anchor="middle" x="125" y="-227.8" font-family="Helvetica,sans-Serif" font-size="9.00">no</text>
|
||||
</g>
|
||||
<!-- anchor_warranted -->
|
||||
<g id="node7" class="node">
|
||||
<title>anchor_warranted</title>
|
||||
<path fill="#7cc66e" stroke="black" d="M294,-207C294,-207 162,-207 162,-207 156,-207 150,-201 150,-195 150,-195 150,-167 150,-167 150,-161 156,-155 162,-155 162,-155 294,-155 294,-155 300,-155 306,-161 306,-167 306,-167 306,-195 306,-195 306,-201 300,-207 294,-207"/>
|
||||
<text text-anchor="middle" x="228" y="-195" font-family="Helvetica,sans-Serif" font-size="10.00">ANCHOR-WARRANTED</text>
|
||||
<text text-anchor="middle" x="228" y="-184" font-family="Helvetica,sans-Serif" font-size="10.00">pointer-linked +</text>
|
||||
<text text-anchor="middle" x="228" y="-173" font-family="Helvetica,sans-Serif" font-size="10.00">warrant passed where it ran;</text>
|
||||
<text text-anchor="middle" x="228" y="-162" font-family="Helvetica,sans-Serif" font-size="10.00">soft demotes present</text>
|
||||
</g>
|
||||
<!-- has_soft_demote->anchor_warranted -->
|
||||
<g id="edge6" class="edge">
|
||||
<title>has_soft_demote->anchor_warranted</title>
|
||||
<path fill="none" stroke="black" d="M207.15,-256.83C210.82,-243.63 214.76,-229.52 218.21,-217.15"/>
|
||||
<polygon fill="black" stroke="black" points="221.66,-217.79 220.97,-207.22 214.92,-215.91 221.66,-217.79"/>
|
||||
<text text-anchor="middle" x="224" y="-227.8" font-family="Helvetica,sans-Serif" font-size="9.00">yes</text>
|
||||
</g>
|
||||
<!-- is_hybrid -->
|
||||
<g id="node5" class="node">
|
||||
<title>is_hybrid</title>
|
||||
<path fill="#dddddd" stroke="black" d="M216.14,-116.17C216.14,-116.17 123.26,-101.83 123.26,-101.83 117.33,-100.92 117.33,-99.08 123.26,-98.17 123.26,-98.17 216.14,-83.83 216.14,-83.83 222.07,-82.92 233.93,-82.92 239.86,-83.83 239.86,-83.83 332.74,-98.17 332.74,-98.17 338.67,-99.08 338.67,-100.92 332.74,-101.83 332.74,-101.83 239.86,-116.17 239.86,-116.17 233.93,-117.08 222.07,-117.08 216.14,-116.17"/>
|
||||
<text text-anchor="middle" x="228" y="-97.5" font-family="Helvetica,sans-Serif" font-size="10.00">audit_mode == HYBRID?</text>
|
||||
</g>
|
||||
<!-- partial -->
|
||||
<g id="node10" class="node">
|
||||
<title>partial</title>
|
||||
<polygon fill="#fff0e0" stroke="black" points="308.5,-36 141.5,-36 141.5,0 314.5,0 314.5,-30 308.5,-36"/>
|
||||
<polyline fill="none" stroke="black" points="308.5,-36 308.5,-30 "/>
|
||||
<polyline fill="none" stroke="black" points="314.5,-30 308.5,-30 "/>
|
||||
<text text-anchor="middle" x="228" y="-21" font-family="Helvetica,sans-Serif" font-size="10.00">HYBRID → add `-PARTIAL` suffix</text>
|
||||
<text text-anchor="middle" x="228" y="-10" font-family="Helvetica,sans-Serif" font-size="10.00">on whichever rung applies</text>
|
||||
</g>
|
||||
<!-- is_hybrid->partial -->
|
||||
<g id="edge11" class="edge">
|
||||
<title>is_hybrid->partial</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M228,-81.64C228,-71.3 228,-57.94 228,-46.22"/>
|
||||
<polygon fill="black" stroke="black" points="231.5,-46.05 228,-36.05 224.5,-46.05 231.5,-46.05"/>
|
||||
<text text-anchor="middle" x="236" y="-56.8" font-family="Helvetica,sans-Serif" font-size="9.00">yes</text>
|
||||
</g>
|
||||
<!-- evidence_warranted->is_hybrid -->
|
||||
<g id="edge8" class="edge">
|
||||
<title>evidence_warranted->is_hybrid</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M117.7,-154.79C141.9,-142.99 170.13,-129.22 191.97,-118.57"/>
|
||||
<polygon fill="black" stroke="black" points="193.76,-121.59 201.22,-114.06 190.7,-115.3 193.76,-121.59"/>
|
||||
</g>
|
||||
<!-- anchor_warranted->is_hybrid -->
|
||||
<g id="edge9" class="edge">
|
||||
<title>anchor_warranted->is_hybrid</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M228,-154.57C228,-146.2 228,-136.85 228,-128.34"/>
|
||||
<polygon fill="black" stroke="black" points="231.5,-128.15 228,-118.15 224.5,-128.15 231.5,-128.15"/>
|
||||
</g>
|
||||
<!-- pointer_linked->is_hybrid -->
|
||||
<g id="edge10" class="edge">
|
||||
<title>pointer_linked->is_hybrid</title>
|
||||
<path fill="none" stroke="black" stroke-dasharray="1,5" d="M349.19,-154.79C321.88,-142.67 289.91,-128.48 265.66,-117.71"/>
|
||||
<polygon fill="black" stroke="black" points="267,-114.48 256.44,-113.62 264.16,-120.88 267,-114.48"/>
|
||||
</g>
|
||||
</g>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 13 KiB |
179
docs/modules/concepts.md
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
# `aborist.concepts`
|
||||
|
||||
Per-shard `concept_relations` SQLite table — the corpus-derived
|
||||
synonym, rivalry, antonym & category overlay that replaces the
|
||||
hand-curated frozensets that lived in `aborist/qa/concepts.py`
|
||||
through April 2026.
|
||||
|
||||
**Full architecture rationale lives in
|
||||
[`../concept-relations-design.md`](../concept-relations-design.md)**
|
||||
including the 1.6% storage-tax measurement and the three-compactions-
|
||||
considered-and-rejected analysis. This page is the API reference.
|
||||
|
||||
## Sub-modules
|
||||
|
||||
### `concepts.store` — append-only CRUD
|
||||
|
||||
```python
|
||||
from aborist.concepts.store import (
|
||||
add_concept_relation, # idempotent INSERT OR IGNORE
|
||||
concept_relations_for_token, # read all relations for a token
|
||||
purge_by_evidence_kind, # the only DELETE path
|
||||
list_evidence_kinds, # diagnostic
|
||||
RELATION_KINDS, # ('synonym','antonym','rivalry','category')
|
||||
)
|
||||
```
|
||||
|
||||
UNIQUE on `(source_root, relation_kind, token, target, evidence_kind)`
|
||||
makes re-derivation idempotent. `purge_by_evidence_kind` lets an
|
||||
operator revoke a single extractor's output without touching manual
|
||||
or other-extractor rows.
|
||||
|
||||
[`aborist/concepts/store.py`](../../aborist/concepts/store.py)
|
||||
|
||||
### `concepts.query` — cross-shard lookup
|
||||
|
||||
```python
|
||||
from aborist.concepts import (
|
||||
synonym_expand, # query tokens → expanded set
|
||||
rivalry_excluded, # query tokens → tokens to drop from results
|
||||
has_compare_phrasing, # bool — does the query say "vs", "compare", etc.
|
||||
invalidate_cache, # drop the per-process LRU
|
||||
)
|
||||
|
||||
expanded = synonym_expand({"thoughts"}, shards_dir=shards_dir)
|
||||
# → {"thoughts", "telepathy", "neurotechnology", "mind", "cognition", ...}
|
||||
```
|
||||
|
||||
Two synonym indices are loaded:
|
||||
|
||||
- **`manual_index`** — `manual_legacy` + `manual` rows. Curated;
|
||||
always expanded regardless of per-token degree. Captures the
|
||||
brain-tech / AMD-family / Mac / Linux / etc. seed groups.
|
||||
- **`derived_index`** — `link_reciprocity` & other corpus-derived
|
||||
edges. Subject to **`MAX_NEIGHBORS_PER_TOKEN=8`** cap because the
|
||||
Wikipedia link graph carries topic-adjacency noise on generic
|
||||
tokens (person, thoughts, language).
|
||||
|
||||
Overall **`MAX_TOTAL_TOKENS=50`** cap on expanded set bounds the
|
||||
SQL clause count downstream so retrieval stays sub-second.
|
||||
|
||||
A per-process LRU keyed on shard mtime avoids re-loading the index
|
||||
on every query (290k rows across 4 shards loads in ~1.8s cold).
|
||||
|
||||
[`aborist/concepts/query.py`](../../aborist/concepts/query.py)
|
||||
|
||||
### `concepts.extract` — pluggable extractor framework
|
||||
|
||||
```python
|
||||
from aborist.concepts.extract import (
|
||||
EXTRACTORS, # registry: evidence_kind → callable
|
||||
link_reciprocity_synonym, # built-in extractor
|
||||
)
|
||||
|
||||
# Run an extractor against a shard:
|
||||
result = link_reciprocity_synonym(conn, derived_from="backfill@2026-05-01")
|
||||
# → {"reciprocal_pairs": N, "synonyms_inserted": M, "synonyms_skipped": K}
|
||||
```
|
||||
|
||||
Each extractor walks the shard's existing rows (`documents`,
|
||||
`chunks`, `edges`, `derivations`) — **no new crawler needed** — &
|
||||
emits concept relations under a stable `evidence_kind` string that
|
||||
supports targeted purge.
|
||||
|
||||
The built-in `link_reciprocity_synonym` reads the existing `edges`
|
||||
table for reciprocal A↔B link pairs and emits a synonym edge between
|
||||
every (title-token-of-A, title-token-of-B) pair. Works for
|
||||
Wikipedia (See-also bidirectional), HTML site internal links
|
||||
(russell.ballestrini.net pattern), or any document graph with
|
||||
bidirectional links. Title-tokens are filtered to ≥4 chars +
|
||||
stopword-stripped.
|
||||
|
||||
To add a new extractor:
|
||||
|
||||
1. Implement `(conn, *, derived_from) -> dict[str, int]` that calls
|
||||
`add_concept_relation` for each finding.
|
||||
2. Pick a stable `evidence_kind` string.
|
||||
3. Register in `EXTRACTORS`.
|
||||
|
||||
[`aborist/concepts/extract.py`](../../aborist/concepts/extract.py)
|
||||
|
||||
### `concepts.seed` — legacy frozenset migration
|
||||
|
||||
One-shot migration of the 8 hand-curated frozenset groups
|
||||
(AMD-family, Intel-family, HTTP, FTP, Mac, Windows, Linux,
|
||||
brain-tech) to `evidence_kind='manual_legacy'` rows.
|
||||
|
||||
Writes **clique edges** within each group — every `(a, b)` pair —
|
||||
so any member retrieves every other member (preserves the legacy
|
||||
frozenset semantic where lookups didn't depend on which token in
|
||||
the group was the anchor).
|
||||
|
||||
```python
|
||||
from aborist.concepts.seed import seed_legacy_concepts
|
||||
result = seed_legacy_concepts(conn)
|
||||
# → {"synonyms_inserted": N, "rivalries_inserted": M, "skipped": K}
|
||||
```
|
||||
|
||||
Idempotent — re-running adds nothing if all the rows already exist.
|
||||
|
||||
[`aborist/concepts/seed.py`](../../aborist/concepts/seed.py)
|
||||
|
||||
## Data model
|
||||
|
||||
```sql
|
||||
CREATE TABLE concept_relations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_root TEXT NOT NULL,
|
||||
relation_kind TEXT NOT NULL CHECK (relation_kind IN
|
||||
('synonym','antonym','rivalry','category')),
|
||||
token TEXT NOT NULL,
|
||||
target TEXT NOT NULL,
|
||||
evidence_kind TEXT NOT NULL,
|
||||
confidence REAL NOT NULL DEFAULT 1.0,
|
||||
derived_at INTEGER NOT NULL,
|
||||
derived_from TEXT,
|
||||
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
|
||||
);
|
||||
```
|
||||
|
||||
**Append-only by construction.** Re-running an extractor adds
|
||||
nothing if every relation already exists. No `UPDATE` path; only
|
||||
`add_concept_relation` (insert) and `purge_by_evidence_kind`
|
||||
(targeted delete).
|
||||
|
||||
**Per-shard storage.** Concept relations live in the shard whose
|
||||
document derived them. Mesh sync moves shards between peers;
|
||||
relations come along.
|
||||
|
||||
**Orthogonal to Merkle.** Writes to `concept_relations` NEVER
|
||||
affect `document_root`, `chunk_root`, or `cache_key`. Backfilling
|
||||
relations is safe across the entire corpus without invalidating
|
||||
any cached answer or breaking any audit chain.
|
||||
|
||||
## Storage cost — measured
|
||||
|
||||
Backfill on 4 wiki shards (3.47M docs, 10.75M resolved edges):
|
||||
|
||||
| Shard | Reciprocal pairs | Synonyms | Storage |
|
||||
|---|---:|---:|---:|
|
||||
| 000.db | 13,562 | 71,288 | 23.52 MB |
|
||||
| 001.db | 14,078 | 73,351 | 24.20 MB |
|
||||
| 002.db | 13,708 | 72,576 | 23.92 MB |
|
||||
| 003.db | 13,800 | 72,633 | 23.94 MB |
|
||||
| **total** | **55,148** | **289,848** | **95.58 MB** |
|
||||
|
||||
**1.6% storage tax** on the 6 GB corpus. Backfill takes ~4 min
|
||||
wall-clock total. Cost is paid once at backfill; every retrieval-
|
||||
time lookup benefits.
|
||||
|
||||
## Diagrams
|
||||
|
||||

|
||||

|
||||
|
||||
## Source
|
||||
|
||||
- [`aborist/concepts/`](../../aborist/concepts/) — package
|
||||
- [`docs/concept-relations-design.md`](../concept-relations-design.md) — full design doc
|
||||
- Whitepaper §13.4.11 — public-facing summary
|
||||
76
docs/modules/distill.md
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
# `aborist.distill`
|
||||
|
||||
Surface → core distillation. Takes a set of surface documents
|
||||
(the original ingest layer) and produces "core" documents — shorter,
|
||||
more focused, Merkle-bound back to their contributing surface
|
||||
chunks via per-chunk inclusion proofs.
|
||||
|
||||
The "trees and forests of cross-linked information" tagline aborist
|
||||
takes its name from comes from this layer: planet-toward-center
|
||||
compression where each layer of cores derives from the previous,
|
||||
recursively.
|
||||
|
||||
## Layered design
|
||||
|
||||
```
|
||||
distill/
|
||||
├── base.py Distiller ABC + DistillationResult dataclass
|
||||
├── first_sentence.py no-ML stub: take the first sentence of each doc
|
||||
├── tfidf.py pure-Python TF-IDF top-keyword extraction
|
||||
└── runner.py batched distillation + per-contrib-chunk proofs
|
||||
```
|
||||
|
||||
## `Distiller` ABC
|
||||
|
||||
```python
|
||||
from aborist.distill import Distiller, DistillationResult
|
||||
|
||||
class Distiller(ABC):
|
||||
@abstractmethod
|
||||
def distill(self, docs: list[Document]) -> DistillationResult: ...
|
||||
```
|
||||
|
||||
Each `DistillationResult` carries the new core's content +
|
||||
references to every contributing surface chunk by `(document_root,
|
||||
chunk_root)`. The runner writes one `derivations` row per core,
|
||||
with `proof_blob = json.dumps(per_chunk_inclusion_proofs)`.
|
||||
|
||||
## Built-in distillers
|
||||
|
||||
### `FirstSentenceDistiller` (no-ML stub)
|
||||
|
||||
Take the first sentence of each input doc, concatenate. Used as a
|
||||
sanity-check for the pipeline + a baseline for measuring the
|
||||
benefit of richer distillers.
|
||||
|
||||
### `TfidfKeywordDistiller`
|
||||
|
||||
Pure-Python TF-IDF. Computes term frequencies across the input
|
||||
doc set + inverse document frequencies; emits the top-K terms per
|
||||
doc as the core's content. The "permacomputer" neologism case
|
||||
(every Grok conversation has the word, no Wikipedia article does)
|
||||
is the canonical TF-IDF win — surfaces the topic that title-search
|
||||
can't catch.
|
||||
|
||||
## Why distill
|
||||
|
||||
Three use cases:
|
||||
|
||||
1. **Retrieval signal.** Cores feed the third accept path in
|
||||
`_filter_by_title_relevance` — `core_match_roots` (TF-IDF top-
|
||||
keywords contain a query token). Closes the gap for neologisms
|
||||
that never make Wikipedia titles but ARE distinctive.
|
||||
|
||||
2. **Hot/cold tier discipline.** `evict_to_cold` only touches
|
||||
`kind='surface'` — cores never evict. Distilling surface to
|
||||
cores then evicting surfaces gives a "long tail keeps small
|
||||
cache" pattern with full provenance preserved.
|
||||
|
||||
3. **Recursive abstraction.** Cores can themselves be distilled
|
||||
into shorter cores. Each generation Merkle-binds back to the
|
||||
previous via `derivations.proof_blob` — the audit chain stays
|
||||
intact across an arbitrary distillation depth.
|
||||
|
||||
## Source
|
||||
|
||||
[`aborist/distill/`](../../aborist/distill/)
|
||||
68
docs/modules/document.md
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
# `aborist.document`
|
||||
|
||||
The data structures every source produces and every storage layer
|
||||
consumes. Three core types: `Document`, `Edge`, `Chunker`.
|
||||
|
||||
## `Document`
|
||||
|
||||
A single ingest unit: a Wikipedia article, an HTML page, a Grok
|
||||
conversation, a git commit message, etc. Carries both the raw
|
||||
content AND the version tags that determine its identity:
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Document:
|
||||
document_uri: str # canonical URI (or stable surrogate for non-URI sources)
|
||||
raw_content: str # source-of-truth bytes pre-canonicalization
|
||||
kind: str # 'surface' / 'core' / 'visual' / etc.
|
||||
chunking_version: str # e.g. 'tok-512-v1' — pinned by the Chunker
|
||||
canonicalization_version: str # e.g. 'norm-v1' — pinned by canonicalize()
|
||||
schema_version: str # e.g. 'v9.8.0' — store schema generation
|
||||
title: str | None = None
|
||||
edges: list[Edge] = () # outbound link graph
|
||||
metadata: dict = ... # source-specific opaque payload
|
||||
```
|
||||
|
||||
`document_root` is computed at ingest time as the Merkle root over
|
||||
the canonicalized chunks. Two peers ingesting the same source +
|
||||
running the same `chunking_version` + `canonicalization_version`
|
||||
get bit-identical `document_root`s — the v9.8 admissibility property.
|
||||
|
||||
## `Edge`
|
||||
|
||||
One outbound link. `aborist/sources/wikipedia.py` emits one Edge
|
||||
per `[[wikilink]]`; `aborist/sources/html_page.py` emits one per
|
||||
`<a href>`. The link graph IS the corpus topology — `concepts/extract.py`
|
||||
later reads `edges` rows to derive synonym relations from
|
||||
reciprocal links (no separate crawler needed).
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class Edge:
|
||||
src_root: str # source document_root
|
||||
dst_uri: str # always present
|
||||
dst_root: str # '' (unresolved) until the dst doc is also ingested
|
||||
edge_type: str # 'wikilink' / 'href' / 'citation' / 'derived_from' / ...
|
||||
anchor: str # chunk index or fragment, '' if N/A
|
||||
```
|
||||
|
||||
## `Chunker`
|
||||
|
||||
ABC with one method `chunk(text: str) -> list[str]`. Default impl
|
||||
is `TokenChunker` (`name='tok-512-v1'`) — splits on token-rough
|
||||
windows so the resulting chunks are predictable for downstream FTS5
|
||||
indexing & for the LLM context budget.
|
||||
|
||||
**Changing the chunker bumps `chunking_version` AND stales every
|
||||
prior cache record** (chunking is one of the 8 cache_key dimensions).
|
||||
Don't redefine `tok-512-v1`; add a new chunker as a new `name`
|
||||
instead.
|
||||
|
||||
## Diagrams
|
||||
|
||||

|
||||

|
||||
|
||||
## Source
|
||||
|
||||
[`aborist/document.py`](../../aborist/document.py)
|
||||
50
docs/modules/evict.md
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
# `aborist.evict`
|
||||
|
||||
Hot ↔ cold tier transitions. The corpus is large (3.47M Wikipedia
|
||||
docs); not every chunk fits in working memory. `evict.py` is the
|
||||
mechanism that moves rarely-touched chunks to a cold tier (still
|
||||
indexed, just stored separately) and rehydrates them on demand
|
||||
from the original source.
|
||||
|
||||
## API surface
|
||||
|
||||
```python
|
||||
from aborist.evict import evict_to_cold, rehydrate
|
||||
|
||||
# Move chunks unused for >threshold days to cold tier
|
||||
evict_to_cold(conn, max_age_days=90, max_evictions=10000)
|
||||
|
||||
# Pull a cold chunk back to hot from its original source
|
||||
rehydrate(conn, document_root="abc123...")
|
||||
```
|
||||
|
||||
## Invariant: cores never evict
|
||||
|
||||
`evict_to_cold` filters `WHERE kind='surface'`. Cores are always
|
||||
hot — they're the long-tail-friendly compression layer that justifies
|
||||
evicting their underlying surfaces. Evicting cores would defeat the
|
||||
purpose.
|
||||
|
||||
## v9.8 falsification on drift
|
||||
|
||||
When `rehydrate()` re-fetches a document and the recomputed
|
||||
`document_root` differs from the stored one, the source has
|
||||
changed since ingest (Wikipedia article was edited, HTML page was
|
||||
republished, etc.). The cache record's `falsification_state` flips
|
||||
from `live` to `stale` — every providence record keyed on that
|
||||
`source_root` is no longer admissible to lookups.
|
||||
|
||||
This is the **drift-detection-as-falsification** discipline: cache
|
||||
hits don't blindly trust historical answers; they trust answers
|
||||
that the SAME source still grounds.
|
||||
|
||||
## Tier values
|
||||
|
||||
`chunks.tier ∈ {'hot', 'cold'}`. Hot chunks live in `chunks.content`;
|
||||
cold chunks live with NULL `content` and a `cold_uri` pointing at
|
||||
the source. The QA pipeline's chunk-fetch path checks `tier`; on
|
||||
'cold', it triggers `rehydrate` before continuing.
|
||||
|
||||
## Source
|
||||
|
||||
[`aborist/evict.py`](../../aborist/evict.py)
|
||||
97
docs/modules/index.md
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
# Aborist module reference
|
||||
|
||||
One page per top-level package + one diagram per major data flow.
|
||||
This index is the entry point — every module page links back here
|
||||
and to the relevant diagrams.
|
||||
|
||||
## Diagrams
|
||||
|
||||
| Diagram | What it shows | File |
|
||||
|---|---|---|
|
||||
| **Module graph** | Top-level packages & how they import each other | [`aborist-modules.svg`](../diagrams/aborist-modules.svg) ([dot](../diagrams/aborist-modules.dot)) |
|
||||
| **Query pipeline** | Question → cache → retrieval → LLM → verify → render | [`query-pipeline.svg`](../diagrams/query-pipeline.svg) ([dot](../diagrams/query-pipeline.dot)) |
|
||||
| **Ingest pipeline** | Source document → Merkle-committed shard | [`ingest-pipeline.svg`](../diagrams/ingest-pipeline.svg) ([dot](../diagrams/ingest-pipeline.dot)) |
|
||||
| **Verifier ladder** | (audit_mode, violations) → display rung | [`verifier-ladder.svg`](../diagrams/verifier-ladder.svg) ([dot](../diagrams/verifier-ladder.dot)) |
|
||||
| **Mesh data flow** | Federation: roster, gossip, AEAD envelope | [`mesh-data-flow.svg`](../diagrams/mesh-data-flow.svg) |
|
||||
| **Mesh epoch lifecycle** | Epoch advance via add/kick/rotate | [`mesh-epoch-lifecycle.svg`](../diagrams/mesh-epoch-lifecycle.svg) |
|
||||
| **Mesh identity stack** | Ed25519 sign + X25519 DH key derivation | [`mesh-identity-stack.svg`](../diagrams/mesh-identity-stack.svg) |
|
||||
| **Mesh secret envelope** | AEAD-wrapped epoch secret per peer | [`mesh-secret-envelope.svg`](../diagrams/mesh-secret-envelope.svg) |
|
||||
| **Mesh group decisions** | Membership change voting & quorum | [`mesh-group-decisions.svg`](../diagrams/mesh-group-decisions.svg) |
|
||||
|
||||
Render diagrams locally:
|
||||
|
||||
```
|
||||
make docs # runs `dot -Tsvg` and `-Tpng` on every docs/diagrams/*.dot
|
||||
```
|
||||
|
||||
## Substrate (no SQL, pure data structures)
|
||||
|
||||
| Module | One-line role | Doc |
|
||||
|---|---|---|
|
||||
| [`merkle.py`](../../aborist/merkle.py) | Merkle tree + proof — Python port of `proxy.unturf.com/pkg/verified/merkle.go` | [merkle.md](./merkle.md) |
|
||||
| [`document.py`](../../aborist/document.py) | `Document`, `Edge`, `Chunker` (default `tok-512-v1`) | [document.md](./document.md) |
|
||||
| [`wikitext.py`](../../aborist/wikitext.py) | `to_base()` — wikitext → plain prose, BASE_VERSION-pinned | [wikitext.md](./wikitext.md) |
|
||||
|
||||
## Storage
|
||||
|
||||
| Module | One-line role | Doc |
|
||||
|---|---|---|
|
||||
| [`store.py`](../../aborist/store.py) | v9.8 SQLite schema + audit chain helpers | [store.md](./store.md) |
|
||||
| [`ingest.py`](../../aborist/ingest.py) | normalize → chunk → merkle → upsert (bulk-batched) | [ingest.md](./ingest.md) |
|
||||
| [`evict.py`](../../aborist/evict.py) | hot ↔ cold tier transitions; rehydrate via source | [evict.md](./evict.md) |
|
||||
|
||||
## Sources (corpus producers)
|
||||
|
||||
| Module | One-line role | Doc |
|
||||
|---|---|---|
|
||||
| [`sources/wikipedia.py`](../../aborist/sources/wikipedia.py) | Wikipedia 2003 cur + old SQL dumps (bz2-streamed) | [sources.md](./sources.md) |
|
||||
| [`sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py) | Phase IV XML dumps (iterparse, page + history) | [sources.md](./sources.md) |
|
||||
| [`sources/html_page.py`](../../aborist/sources/html_page.py) | URL list + selectolax + httpx (robots-aware) | [sources.md](./sources.md) |
|
||||
| [`sources/crawler/`](../../aborist/sources/crawler/) | verbatim AsyncWebFetcher lift + ingest bridge | [sources.md](./sources.md) |
|
||||
| [`sources/grok.py`](../../aborist/sources/grok.py) | xAI data export (conversations + media prompts) | [sources.md](./sources.md) |
|
||||
| [`sources/vcs.py`](../../aborist/sources/vcs.py) | git + Mercurial repos (HEAD walk, supersedes chain) | [sources.md](./sources.md) |
|
||||
|
||||
## Search & retrieval
|
||||
|
||||
| Module | One-line role | Doc |
|
||||
|---|---|---|
|
||||
| [`search/`](../../aborist/search/) | FTS5 backend + `SearchBackend` ABC + `AuditMode` enum | [search.md](./search.md) |
|
||||
| [`concepts/`](../../aborist/concepts/) | Per-shard `concept_relations` synonym/rivalry overlay | [concepts.md](./concepts.md) |
|
||||
|
||||
## Q&A pipeline
|
||||
|
||||
| Module | One-line role | Doc |
|
||||
|---|---|---|
|
||||
| [`qa/keys.py`](../../aborist/qa/keys.py) | 8-dim cache_key + `question_hash` | [qa.md](./qa.md) |
|
||||
| [`qa/client.py`](../../aborist/qa/client.py) | `ChatClient` + `StubClient` + `OpenAICompatibleClient` | [qa.md](./qa.md) |
|
||||
| [`qa/runner.py`](../../aborist/qa/runner.py) | `ask()`: single-doc Q&A + cache + verify | [qa.md](./qa.md) |
|
||||
| [`qa/query.py`](../../aborist/qa/query.py) | `query()`: multi-source RAG + concept overlay | [qa.md](./qa.md) |
|
||||
| [`qa/verify.py`](../../aborist/qa/verify.py) | quote/span/entity/paraphrase + claim_lattice (7 hard checks) | [qa.md](./qa.md) |
|
||||
| [`qa/warrant.py`](../../aborist/qa/warrant.py) | 5 anchor classes (proper-noun · date · count · entity-list · cause) | [qa.md](./qa.md) |
|
||||
| [`qa/evidence.py`](../../aborist/qa/evidence.py) | `EvidenceObject` + spotlight excerpt (density rank) | [qa.md](./qa.md) |
|
||||
| [`qa/parse_claims.py`](../../aborist/qa/parse_claims.py) | pointer-line parser (`claim. [E1,E2]`) | [qa.md](./qa.md) |
|
||||
| [`qa/dag.py`](../../aborist/qa/dag.py) | per-run Merkle-DAG (7-stage quote / 9-stage CTI) | [qa.md](./qa.md) |
|
||||
| [`qa/inspect.py`](../../aborist/qa/inspect.py) | sidecar diagnostic (read-only span classifier) | [qa.md](./qa.md) |
|
||||
|
||||
## Distillation
|
||||
|
||||
| Module | One-line role | Doc |
|
||||
|---|---|---|
|
||||
| [`distill/`](../../aborist/distill/) | `Distiller` ABC + `first_sentence` + `tfidf` + runner | [distill.md](./distill.md) |
|
||||
|
||||
## Federation (off by default)
|
||||
|
||||
| Module | One-line role | Doc |
|
||||
|---|---|---|
|
||||
| [`mesh/`](../../aborist/mesh/) | identity (Ed25519/X25519), per-epoch roster, AEAD envelope, gossip wire | [../mesh.md](../mesh.md), [../mesh-deploy.md](../mesh-deploy.md) |
|
||||
|
||||
## Entry point
|
||||
|
||||
| Module | One-line role | Doc |
|
||||
|---|---|---|
|
||||
| [`cli.py`](../../aborist/cli.py) | `argparse` entrypoint — every `make` target dispatches here | run `aborist --help` or any `make help` target |
|
||||
|
||||
## Tickets, design docs, journals
|
||||
|
||||
See [`../TICKETS.md`](../TICKETS.md) for the ticket index and the
|
||||
list of design-reference docs that aren't tickets.
|
||||
64
docs/modules/ingest.md
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
# `aborist.ingest`
|
||||
|
||||
The bulk-batched pipeline that turns documents from a source into
|
||||
Merkle-committed shard storage. Source-agnostic: anything that
|
||||
implements `Source.iter_documents()` flows through here.
|
||||
|
||||

|
||||
|
||||
## Public API
|
||||
|
||||
```python
|
||||
from aborist.ingest import ingest_source
|
||||
from aborist.sources.wikipedia import WikipediaSqlDump
|
||||
|
||||
source = WikipediaSqlDump("/path/to/cur.sql.bz2")
|
||||
ingest_source(
|
||||
source,
|
||||
db_path=Path("~/.aborist/shards/000.db"),
|
||||
batch_size=200, # docs per transaction
|
||||
progress_every=1000,
|
||||
)
|
||||
```
|
||||
|
||||
## What happens per document
|
||||
|
||||
1. **Canonicalize** — `canonicalize(text)`: NFC + ws-collapse +
|
||||
strip ends. Pinned by `canonicalization_version='norm-v1'`.
|
||||
2. **Chunk** — `Chunker.chunk(canonical_text)` → list of token-
|
||||
bounded substrings. Default `tok-512-v1` chunker.
|
||||
3. **Hash leaves** — `sha256(0x00 || canonical_chunk_bytes)` per
|
||||
chunk.
|
||||
4. **Merkle tree** — `MerkleTree.build(leaves).root` →
|
||||
`document_root`. Two peers running the same chunker on the same
|
||||
canonicalized content compute bit-identical roots.
|
||||
5. **Upsert** — `documents` row keyed on `document_root` (idempotent
|
||||
re-ingest), `chunks` rows with leaf hashes, `merkle_nodes` for
|
||||
proof reconstruction, `edges` per outbound link.
|
||||
6. **FTS5** — `chunks_fts` insert with rowid = `chunks.chunk_id`
|
||||
so the search-time JOIN lines up.
|
||||
7. **Audit event** — one row per ingest batch in `audit_events`,
|
||||
chained on `prev_event_hash`.
|
||||
|
||||
## Batching discipline
|
||||
|
||||
Default `batch_size=200`: balances Python GIL overhead vs SQLite
|
||||
transaction commit cost. Lower it (e.g. 50) only to bound peak
|
||||
memory on a low-RAM host. Higher (e.g. 1000) for ETL throughput on
|
||||
SSD storage when memory isn't tight.
|
||||
|
||||
`progress_every` prints a stderr line every N docs so long ingests
|
||||
are observable. Use `PYTHONUNBUFFERED=1` for tail-able output.
|
||||
|
||||
## Resumability
|
||||
|
||||
Idempotent re-ingest: same content + same chunker + same canonicalize
|
||||
= same `document_root` = no-op insert. So a crashed ingest can be
|
||||
restarted from the source's beginning without duplicating rows.
|
||||
|
||||
Different content at the same URI gets a new `document_root` AND a
|
||||
`supersedes` edge linking new → old (lossless history).
|
||||
|
||||
## Source
|
||||
|
||||
[`aborist/ingest.py`](../../aborist/ingest.py)
|
||||
63
docs/modules/merkle.md
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
# `aborist.merkle`
|
||||
|
||||
Pure Merkle tree + proof primitives. Python port of
|
||||
`proxy.unturf.com/pkg/verified/merkle.go` — convention-identical.
|
||||
Used everywhere a content-addressable handle is needed: per-chunk
|
||||
leaves, document_root, evidence_map_root, run_dag_root, snapshots.
|
||||
|
||||
## Conventions (do not silently change)
|
||||
|
||||
These match the Go reference & are load-bearing for cross-language
|
||||
verification (Go peer ↔ Python peer compute bit-identical roots):
|
||||
|
||||
- **Leaf hash:** `sha256(0x00 || canonical_chunk_bytes)`. The `0x00`
|
||||
prefix domain-separates leaves from internal nodes.
|
||||
- **Internal hash:** `sha256(0x03 || left || right)`. The `0x03`
|
||||
prefix is the **non-commutative** combine — `H(L,R) ≠ H(R,L)`.
|
||||
Order matters.
|
||||
- **Odd-element rule:** when a level has an odd count, the last
|
||||
leaf is **self-duplicated** before pairing. NOT zero-padded.
|
||||
- **Proof path:** each step carries an explicit `is_left: bool`
|
||||
alongside the sibling hash so a verifier knows which side to put
|
||||
the sibling on. Never sort siblings lexically — the order tells
|
||||
the verifier the tree topology.
|
||||
|
||||
## API surface
|
||||
|
||||
```python
|
||||
from aborist.merkle import MerkleTree, MerkleProof
|
||||
|
||||
tree = MerkleTree.build([b"chunk_0_bytes", b"chunk_1_bytes", ...])
|
||||
tree.root # bytes(32) — sha256 of the whole tree
|
||||
tree.leaves # list[bytes(32)] — leaf hashes in input order
|
||||
|
||||
proof = tree.proof_for(leaf_index=2)
|
||||
proof.siblings # list[(sibling_hash, is_left)]
|
||||
proof.verify(leaf_hash=tree.leaves[2], root=tree.root) # bool
|
||||
```
|
||||
|
||||
## When to read the source
|
||||
|
||||
- Adding a new content-addressable artifact (cores, evidence maps,
|
||||
snapshots, run-DAGs all touch this).
|
||||
- Cross-language verification debugging (Go peer says one root,
|
||||
Python peer says another — the difference is always in canonical
|
||||
encoding, ordering, or one of the three prefix bytes above).
|
||||
- Performance work — the Python build is ~3× slower than the Go
|
||||
reference; if it ever shows up in profiling, that's the file.
|
||||
|
||||
## Diagrams
|
||||
|
||||
The module graph shows what depends on `merkle.py` (a lot — it's
|
||||
substrate):
|
||||
|
||||

|
||||
|
||||
The ingest pipeline shows where leaf & root hashes get computed:
|
||||
|
||||

|
||||
|
||||
## Source
|
||||
|
||||
[`aborist/merkle.py`](../../aborist/merkle.py) ·
|
||||
Reference: [`proxy.unturf.com/pkg/verified/merkle.go`](https://git.unturf.com/engineering/unturf/proxy.unturf.com/-/blob/main/pkg/verified/merkle.go)
|
||||
185
docs/modules/qa.md
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
# `aborist.qa`
|
||||
|
||||
The Q&A pipeline. Question → cache → retrieval → LLM → verify →
|
||||
render → cache write. Lives in 9 sub-modules; this page is the
|
||||
map.
|
||||
|
||||

|
||||
|
||||
## Sub-modules
|
||||
|
||||
### `qa.client` — LLM transport
|
||||
|
||||
`ChatClient` ABC with three concrete implementations:
|
||||
|
||||
- `StubClient` — deterministic test fixture; returns canned
|
||||
responses keyed on the input. Used in unit tests to avoid network.
|
||||
- `OpenAICompatibleClient` — talks to any OpenAI-shape `/v1/chat/completions`
|
||||
endpoint (Hermes-3 on vLLM by default). Includes HTTP retry layer
|
||||
(3× exponential backoff on 5xx).
|
||||
- (Future) `AnthropicClient` — Claude API direct.
|
||||
|
||||
[`aborist/qa/client.py`](../../aborist/qa/client.py)
|
||||
|
||||
### `qa.keys` — the 8-dim cache_key
|
||||
|
||||
```
|
||||
cache_key = sha256(
|
||||
source_root | question_hash | model_profile_hash |
|
||||
conversation_hash | governance_policy_hash |
|
||||
schema_version | canonicalization_version | chunking_version
|
||||
)
|
||||
```
|
||||
|
||||
Two question-hash modes (`strict` vs `equivalence_class`) live here.
|
||||
Bumping any of these eight dimensions invalidates prior records on
|
||||
lookup. The `verifier_policy_hash` (v9.9 9th dim) is also
|
||||
implemented here.
|
||||
|
||||
[`aborist/qa/keys.py`](../../aborist/qa/keys.py)
|
||||
|
||||
### `qa.runner` — `ask()` for single-doc Q&A
|
||||
|
||||
The simplest entry point. Take one document, ask one question, get
|
||||
back an answer + audit_mode + cache record. Used by the CLI for
|
||||
focused queries against one URI.
|
||||
|
||||
[`aborist/qa/runner.py`](../../aborist/qa/runner.py)
|
||||
|
||||
### `qa.query` — `query()` for multi-source RAG
|
||||
|
||||
The main retrieval entry point. Walks shards, runs FTS5 BM25 with
|
||||
AND→OR fallback (with synonym-pool injection in OR mode), filters
|
||||
by title relevance with 4 accept paths, reranks by body coverage +
|
||||
title boost + source role + title purity, assembles a 60 KB context
|
||||
budget, calls the LLM, runs the verifier, persists to
|
||||
`providence_cache`.
|
||||
|
||||
[`aborist/qa/query.py`](../../aborist/qa/query.py)
|
||||
|
||||
### `qa.verify` — the layered verifier
|
||||
|
||||
Five strategies run in sequence; first to find evidence classifies:
|
||||
1. `quote` — `"..."`-wrapped claims tested verbatim
|
||||
2. `span` — bullet/sentence units substring-tested
|
||||
3. `entity` — multi-word proper nouns with proximity gating
|
||||
4. `paraphrase` — token coverage on prose-shaped spans (≥85%)
|
||||
5. `claim_lattice` — pointer-line `[E1,E2]` or JSON; runs **seven
|
||||
deterministic hard checks**:
|
||||
1. parser succeeded
|
||||
2. evidence_id resolves
|
||||
3. source_role allowed
|
||||
4. claim text non-empty
|
||||
5. citation coverage threshold
|
||||
6. pointer count cap (trim-and-verify)
|
||||
7. anchor-class warrant (see `qa.warrant`)
|
||||
|
||||
The classifier output rolls up into the v9.8 trichotomy
|
||||
`audit_mode ∈ {STRICT, HYBRID, UNGROUNDED}`. Display layer (in
|
||||
`cli.py`) maps `(audit_mode, violations) → four-rung ladder`.
|
||||
|
||||

|
||||
|
||||
[`aborist/qa/verify.py`](../../aborist/qa/verify.py)
|
||||
|
||||
### `qa.warrant` — anchor-class warrant
|
||||
|
||||
Five lexical anchor classes the verifier composes:
|
||||
|
||||
- **Proper-noun** — relation-question shape; at least one
|
||||
Title-Case anchor must appear in some cited span
|
||||
- **Date** — claim has a 4-digit year + month name; ALL components
|
||||
required in some cited span
|
||||
- **Entity-list** — entity-list-shape question; ≥1 named entity
|
||||
must anchor (demote-don't-reject)
|
||||
- **Count** — count-shape question; count token must appear in
|
||||
word OR digit form (digit↔word equivalence)
|
||||
- **Cause** — why-shape question; ≥1 cause anchor (proper noun OR
|
||||
≥5-char common noun outside stopword pool)
|
||||
|
||||
The warrant layer earns proof-path entry by staying **lexical** —
|
||||
no NLI, no embeddings. Substring tests over already-canonicalized
|
||||
spans. See `docs/concept-relations-design.md` (sibling section)
|
||||
for the relationship to retrieval-time synonym expansion.
|
||||
|
||||
[`aborist/qa/warrant.py`](../../aborist/qa/warrant.py)
|
||||
|
||||
### `qa.evidence` — EvidenceObject + spotlight
|
||||
|
||||
Builds the runtime evidence map for claim-lattice modes. Each
|
||||
chunk becomes one `EvidenceObject` carrying TWO ids:
|
||||
|
||||
- `pointer_id` — short prompt-facing tag (`E1`, `E2`, …)
|
||||
- `evidence_id` — content-addressed `E########` (sha256-derived)
|
||||
|
||||
The model sees only `pointer_id`s in the prompt; the runtime maps
|
||||
to `evidence_id` for the cache & run-DAG (run-stable identity).
|
||||
|
||||
The spotlight excerpt picks the load-bearing slice via **density
|
||||
rank** — find ALL match positions for ALL claim content tokens,
|
||||
pick the position with maximum distinct-token cluster within
|
||||
±half-window. Replaces the older first-match-of-longest-token
|
||||
approach which lost the load-bearing slice on noisy chunks.
|
||||
|
||||
[`aborist/qa/evidence.py`](../../aborist/qa/evidence.py)
|
||||
|
||||
### `qa.parse_claims` — pointer-line parser
|
||||
|
||||
Walks lines of the model output, pulls every `[E\d+]` and
|
||||
`[E\d+,E\d+,…]` bracket payload, returns
|
||||
`(claim_text, pointer_ids[])` per line. Lines without a tag get
|
||||
`parse_status='NO_EVIDENCE_POINTER'` & count toward the denominator
|
||||
so unsourced prose can't smuggle past the verifier.
|
||||
|
||||
[`aborist/qa/parse_claims.py`](../../aborist/qa/parse_claims.py)
|
||||
|
||||
### `qa.dag` — per-run Merkle DAG
|
||||
|
||||
Commits each provenance step independently as a stage hash. Two
|
||||
shapes:
|
||||
|
||||
- **7-stage (quote mode):** question / retrieval / context /
|
||||
prompt / answer / verify / final_label
|
||||
- **9-stage (claim-lattice / CTI):** question / retrieval /
|
||||
evidence_map / prompt / raw_answer / parsed_claim_lattice /
|
||||
verify / render / final_label
|
||||
|
||||
The `run_dag_root` is persisted alongside every providence record;
|
||||
`run_dag_blob` carries the full `{root, nodes}` JSON so an auditor
|
||||
can recompute & verify any step.
|
||||
|
||||
[`aborist/qa/dag.py`](../../aborist/qa/dag.py)
|
||||
|
||||
### `qa.inspect` — read-only sidecar
|
||||
|
||||
Pulls source chunks for a given cache_key & classifies each
|
||||
unverified span: `verbatim_in_base` / `verbatim_in_raw_only` /
|
||||
`trailing_artifact` / `paraphrase` / `partial_paraphrase` /
|
||||
`no_overlap`. Also includes the deflection-detection sidecar
|
||||
(subject-anchor heuristic for adversarial-premise topic shift).
|
||||
|
||||
**Sidecars never write to `providence_cache` or `audit_events`** —
|
||||
they're diagnostic only. That invariant is what keeps `audit_mode`
|
||||
a binary classification rather than a soft score.
|
||||
|
||||
[`aborist/qa/inspect.py`](../../aborist/qa/inspect.py)
|
||||
|
||||
### `qa.concepts` — backwards-compat shim
|
||||
|
||||
Delegates to `aborist.concepts` (the corpus-derived synonym/rivalry
|
||||
layer). Pre-2026-05-01 the data lived as hand-curated frozensets in
|
||||
this file; now it's a per-shard SQLite table. The shim preserves
|
||||
the legacy public API (`synonym_expand`, `rivalry_excluded`,
|
||||
`has_compare_phrasing`) so call sites in `qa/query.py` didn't have
|
||||
to change.
|
||||
|
||||
[`aborist/qa/concepts.py`](../../aborist/qa/concepts.py) →
|
||||
[`aborist/concepts/`](../../aborist/concepts/)
|
||||
|
||||
## Source papers
|
||||
|
||||
- Whitepaper §13.8 covers the layered verifier in depth
|
||||
- Whitepaper §13.9 covers claim-lattice / CTI mode
|
||||
- `docs/cti-architecture.md` is the architecture reference
|
||||
- `docs/seven-point-program.md` enumerates the seven hard checks
|
||||
- `docs/concept-relations-design.md` covers the synonym layer
|
||||
75
docs/modules/search.md
Normal file
|
|
@ -0,0 +1,75 @@
|
|||
# `aborist.search`
|
||||
|
||||
The retrieval primitive. Today's only backend is FTS5 over the
|
||||
chunks table; the `SearchBackend` ABC is in place so additional
|
||||
backends (BM25 over titles, embedding-based vector search) can be
|
||||
added without touching the rest of the QA pipeline.
|
||||
|
||||
## `SearchBackend` ABC
|
||||
|
||||
```python
|
||||
from aborist.search import SearchBackend, AuditMode, Hit
|
||||
|
||||
class SearchBackend(ABC):
|
||||
@abstractmethod
|
||||
def search(self, query: str, limit: int = 20) -> list[Hit]: ...
|
||||
```
|
||||
|
||||
Each `Hit` carries `(document_root, document_uri, chunk_idx,
|
||||
snippet, score, audit_mode, title)`. `audit_mode` is the sticky
|
||||
provenance label that tracks how the chunk made it into the index;
|
||||
FTS5 backend always sets `UNGROUNDED` (search itself doesn't verify
|
||||
anything — that's the QA pipeline's job).
|
||||
|
||||
## `FTS5Backend`
|
||||
|
||||
Wraps the contentless `chunks_fts` virtual table. Two-mode query:
|
||||
|
||||
- **AND-mode (strict, primary):** every content token must appear
|
||||
in the doc. Keeps unrelated docs out of the context window.
|
||||
- **OR-mode (fallback):** when AND returns 0 hits, fall back to OR
|
||||
but **capped to top-5 longest tokens** (proxy for rarity). Long
|
||||
topical synonyms fed via `extra_or_tokens` join the pool —
|
||||
`neurotechnology` (15 chars) outranks `thoughts` (8) by length
|
||||
and surfaces brain-tech titles for vocabulary-mismatch queries.
|
||||
|
||||
```python
|
||||
from aborist.search import FTS5Backend
|
||||
backend = FTS5Backend(conn)
|
||||
|
||||
# Plain search
|
||||
hits = backend.search("permacomputer", limit=32)
|
||||
|
||||
# Search with synonym pool injection (used by qa.query._search_corpus)
|
||||
hits = backend.search(
|
||||
long_query,
|
||||
limit=32,
|
||||
extra_or_tokens=synonym_expand(qtokens, shards_dir=shards_dir),
|
||||
)
|
||||
```
|
||||
|
||||
## Stopword & stopword-cap discipline
|
||||
|
||||
`_FTS5_STOPWORDS` filters question words (`what`, `tell`, `please`)
|
||||
+ generic connectors (`one`, `some`, `another`, `without`, `soon`,
|
||||
`currently`) before AND/OR construction. Two principles:
|
||||
|
||||
- **Stay in sync with `_TITLE_STOPWORDS`** in `qa/query.py`. A token
|
||||
filtered at retrieval time but kept at title-relevance check (or
|
||||
vice versa) creates ranking incoherence.
|
||||
- **`_OR_FALLBACK_MAX_TOKENS=5`** caps the OR-mode pool. Without
|
||||
this, a 19-token OR clause matches millions of docs and forces
|
||||
BM25 to rank them all — 13s/shard observed pre-cap. Now 0.25s/shard.
|
||||
|
||||
## Snippet building
|
||||
|
||||
FTS5 contentless mode means SQLite's built-in `snippet()` and
|
||||
`highlight()` return empty. Aborist builds snippets in Python by
|
||||
joining `chunks_fts.rowid = chunks.chunk_id`, decompressing the
|
||||
chunk content, and locating query tokens locally
|
||||
(`_build_snippet`).
|
||||
|
||||
## Source
|
||||
|
||||
[`aborist/search/fts5.py`](../../aborist/search/fts5.py) ·
|
||||
[`aborist/search/__init__.py`](../../aborist/search/__init__.py)
|
||||
80
docs/modules/sources.md
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
# `aborist.sources`
|
||||
|
||||
Corpus producers. Each is a `Source` ABC implementation that yields
|
||||
`Document` instances; the standard `ingest.ingest_source(source, db)`
|
||||
pipeline takes them from there.
|
||||
|
||||
The `Source` ABC lives in [`aborist/source.py`](../../aborist/source.py):
|
||||
|
||||
```python
|
||||
class Source(ABC):
|
||||
@abstractmethod
|
||||
def iter_documents(self) -> Iterator[Document]: ...
|
||||
```
|
||||
|
||||
## Built-in sources
|
||||
|
||||
### `wikipedia.py` — Phase III SQL dumps (the canonical bootstrap)
|
||||
|
||||
Streams the Wikipedia 2003 `cur` (current revisions) and `old`
|
||||
(revision history) SQL dumps. Hand-rolled escape-aware parser
|
||||
(no `sqlite3` import — the dump is MySQL syntax). 4× speedup vs
|
||||
char-by-char loops via `str.find` + slicing. cProfile any change.
|
||||
|
||||
Default Wikipedia 2003-05-16 dump source:
|
||||
`https://dumps.wikimedia.org/archive/2003/2003-05-16/en/`.
|
||||
robots.txt returned 404 → no rules.
|
||||
|
||||
[`aborist/sources/wikipedia.py`](../../aborist/sources/wikipedia.py)
|
||||
|
||||
### `wikipedia_xml.py` — Phase IV XML dumps
|
||||
|
||||
Modern Wikipedia dump format (`enwiki-YYYYMMDD-pages-articles.xml.bz2`,
|
||||
`enwiki-YYYYMMDD-pages-meta-history*.xml.bz2`). Uses `xml.etree.ElementTree.iterparse`
|
||||
to stream-parse without loading the whole tree.
|
||||
|
||||
[`aborist/sources/wikipedia_xml.py`](../../aborist/sources/wikipedia_xml.py)
|
||||
|
||||
### `html_page.py` — single-URL or URL-list HTML ingest
|
||||
|
||||
Robots-aware (`urllib.robotparser`). Uses `selectolax` for fast
|
||||
HTML parsing (CSS-selector based; ~10× faster than `lxml`). Pulls
|
||||
the main body text + every `<a href>` as an `Edge` row.
|
||||
|
||||
The `edges` rows are what the corpus-derived synonym extractor
|
||||
later reads — no separate crawler needed for site-internal link
|
||||
graphs.
|
||||
|
||||
Optional dep: `pip install '.[html]'` for `selectolax` + `httpx`.
|
||||
|
||||
[`aborist/sources/html_page.py`](../../aborist/sources/html_page.py)
|
||||
|
||||
### `crawler/` — async BFS web crawl
|
||||
|
||||
Verbatim lift of an `AsyncWebFetcher` implementation + an `ingest`
|
||||
bridge. BFS-discovers same-domain URLs from a seed, respecting
|
||||
`robots.txt` + crawl delays. Captures ETag + Last-Modified per URL
|
||||
into `document_http_meta` so a future recrawl can send conditional
|
||||
HEAD requests.
|
||||
|
||||
[`aborist/sources/crawler/bridge.py`](../../aborist/sources/crawler/bridge.py)
|
||||
|
||||
### `grok.py` — xAI Grok export
|
||||
|
||||
Reads the `xAI-conversations.json` data export shape. Each
|
||||
conversation becomes one `Document`; media prompts are kept
|
||||
inline.
|
||||
|
||||
[`aborist/sources/grok.py`](../../aborist/sources/grok.py)
|
||||
|
||||
### `vcs.py` — git + Mercurial repositories
|
||||
|
||||
HEAD walk. Each commit becomes a `Document` (commit message + diff
|
||||
stat). The supersedes chain captures commit ancestry as edges.
|
||||
|
||||
[`aborist/sources/vcs.py`](../../aborist/sources/vcs.py)
|
||||
|
||||
## Source
|
||||
|
||||
[`aborist/sources/`](../../aborist/sources/) ·
|
||||
[`aborist/source.py`](../../aborist/source.py) (ABC)
|
||||
81
docs/modules/store.md
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
# `aborist.store`
|
||||
|
||||
The v9.8 SQLite schema, the audit chain, and the cross-shard
|
||||
read-only view. Every table that holds runtime state lives here.
|
||||
|
||||
## Schema overview (per shard)
|
||||
|
||||
```
|
||||
documents – one row per source document, keyed on document_root
|
||||
chunks – per-document chunk content + tier (hot/cold)
|
||||
chunks_fts – FTS5 contentless index, rowid = chunks.chunk_id
|
||||
merkle_nodes – internal-node hashes for proof reconstruction
|
||||
edges – src_root → dst_root link graph (wikilink, href, …)
|
||||
derivations – core_root ← src_root with proof_blob (Merkle)
|
||||
providence_cache – Q&A records keyed on the v9.8 8-dim cache_key
|
||||
audit_events – linear chain; event_hash = sha256(prev || canonical(body))
|
||||
falsifications – record_id → state transition + reason + actor
|
||||
snapshots – named corpus roots (one hash names a forest)
|
||||
document_http_meta – ETag + Last-Modified for crawler conditional fetches
|
||||
concept_relations – per-shard synonym/rivalry/category/antonym overlay
|
||||
mesh_* – federation tables (off by default)
|
||||
```
|
||||
|
||||
## v9.8 invariants (do not break)
|
||||
|
||||
- **Aborist is a v9.8 store.** Every providence record carries the
|
||||
full **8-dim cache_key**: `source_root | question_hash |
|
||||
model_profile_hash | conversation_hash | governance_policy_hash |
|
||||
schema_version | canonicalization_version | chunking_version`.
|
||||
Bumping any one invalidates prior records on lookup.
|
||||
- **`falsification_state ∈ {live, failed, stale, quarantined}`.**
|
||||
Cache lookups must filter on `state='live'`. Drift detection
|
||||
flips to `stale`.
|
||||
- **Audit chain.** Every state-changing op writes one row in
|
||||
`audit_events` with `event_hash = sha256(prev_event_hash ||
|
||||
canonical(body))`. Chain integrity is verified in
|
||||
`make analyze-shards`. **Never insert into `audit_events`
|
||||
directly — use `aborist.store.append_audit`.**
|
||||
- **Cores never evict.** `evict_to_cold` only touches `kind='surface'`.
|
||||
- **Idempotent re-ingest.** Same content → same `document_root` →
|
||||
no-op insert. Same URI + different content → new doc + `supersedes`
|
||||
edge linking new → old (lossless history).
|
||||
|
||||
## API surface
|
||||
|
||||
```python
|
||||
from aborist.store import (
|
||||
connect, # writable connection to a single shard
|
||||
connect_query, # read-only UNION view across all shards
|
||||
discover_shards, # list *.db files in a shards_dir
|
||||
transaction, # BEGIN IMMEDIATE / COMMIT / ROLLBACK context manager
|
||||
get_meta, set_meta,
|
||||
append_audit, # the ONLY way to write audit_events
|
||||
)
|
||||
```
|
||||
|
||||
Cross-shard reads use `connect_query(shards_dir=...)` which ATTACHes
|
||||
every `*.db` and creates UNION views over the shardable tables
|
||||
(`documents`, `chunks`, `merkle_nodes`, `edges`, `derivations`,
|
||||
`providence_cache`, `audit_events`, `falsifications`,
|
||||
`concept_relations`).
|
||||
|
||||
## Performance pragmas
|
||||
|
||||
`connect()` applies these per-connection:
|
||||
- `journal_mode=WAL` (set in SCHEMA_SQL once at first ingest)
|
||||
- `synchronous=NORMAL` (skip per-commit fsync; safe under WAL)
|
||||
- `cache_size=-65536` (64 MB page cache)
|
||||
- `temp_store=MEMORY` (no /tmp churn for temp tables)
|
||||
|
||||
Don't downgrade to `synchronous=FULL` without a measured reason —
|
||||
costs ~5× throughput.
|
||||
|
||||
## Diagrams
|
||||
|
||||

|
||||

|
||||
|
||||
## Source
|
||||
|
||||
[`aborist/store.py`](../../aborist/store.py)
|
||||
46
docs/modules/wikitext.md
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
# `aborist.wikitext`
|
||||
|
||||
A single function: `to_base(raw)`. Converts MediaWiki wikitext to
|
||||
plain prose deterministically.
|
||||
|
||||
```python
|
||||
from aborist.wikitext import to_base, BASE_VERSION
|
||||
|
||||
prose = to_base("[[The Beatles]] are an [[English rock band]] from [[Liverpool]].")
|
||||
# → "The Beatles are an English rock band from Liverpool."
|
||||
```
|
||||
|
||||
## Why it exists
|
||||
|
||||
The corpus stores raw wikitext (so the link graph is recoverable
|
||||
on demand) but the LLM and verifier both want plain prose. Reasons:
|
||||
|
||||
1. **Token efficiency.** Wikipedia chunks ship to Hermes with ~43%
|
||||
fewer tokens after wikitext-strip — bigger context window for
|
||||
the same chars budget.
|
||||
2. **Verbatim citation.** The model can quote source paragraphs
|
||||
verbatim instead of escaping `[[wikilinks]]`. The verifier's
|
||||
substring test then matches cleanly.
|
||||
3. **Pinned identity.** `BASE_VERSION='wikitext-base-v1'` lives in
|
||||
`policy["base_version"]`, which folds into
|
||||
`governance_policy_hash`. Bumping `BASE_VERSION` invalidates
|
||||
every prior cache record on next lookup — same discipline as
|
||||
`chunking_version` and `canonicalization_version`.
|
||||
|
||||
## Hot-path discipline
|
||||
|
||||
`to_base()` runs on the assembled context **before** the LLM call
|
||||
in `aborist/qa/runner.py` and `aborist/qa/query.py`, **and again
|
||||
inside `verify_quotes`** so the verifier compares like-against-like.
|
||||
Both sides see prose.
|
||||
|
||||
## Optional dependency
|
||||
|
||||
Backed by `mwparserfromhell`. Install via `pip install '.[wikitext]'`
|
||||
to enable. Without the dep, `_wikitext_to_base = None` and
|
||||
`policy["base_version"] = None` — graceful fallback leaves raw
|
||||
wikitext in both context and verifier (works, just less efficient).
|
||||
|
||||
## Source
|
||||
|
||||
[`aborist/wikitext.py`](../../aborist/wikitext.py)
|
||||