russell@unturf.com
326badf6d8
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).
(1) README — label refresh:
- Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
- Verifier section spells out both layers (schema trichotomy +
display ladder), the seven hard checks of claim_lattice, and
the five anchor classes of warrant.
- Architecture tree updated: concepts/ package added, qa/
sub-modules expanded (warrant, evidence, parse_claims, dag),
verify.py described as quote/span/entity/paraphrase + claim_lattice.
- Concept overlay description updated for corpus-derived layer
(concept_relations table, link_reciprocity extractor, 1.6%
tax cite).
- Test count: 326+ → 641+.
(2) docs/diagrams/ — Graphviz dot sources:
- aborist-modules.dot — top-level package graph (substrate /
storage / sources / retrieval / qa / mesh / cli)
- query-pipeline.dot — question → cache → retrieval → LLM →
verify → render → cache write, with phase budgets
- ingest-pipeline.dot — source doc → canonicalize → chunk →
Merkle → upsert (+ optional distill)
- verifier-ladder.dot — (audit_mode, violations) → display rung
decision tree
Existing mesh-*.dot kept as-is. Makefile `make docs` target
extended to also emit .svg alongside the existing .png so the
diagrams render in markdown viewers.
(3) docs/modules/ — per-module reference pages:
- index.md (links to every diagram + every module page)
- merkle.md, document.md, store.md, ingest.md, evict.md,
sources.md, search.md, concepts.md, qa.md, distill.md,
wikitext.md
Each page is a one-screenful concise reference: what the
module is for, public API, key invariants, embedded diagrams
where useful, link to source. Mesh stays at the existing
docs/mesh.md + docs/mesh-deploy.md (already comprehensive).
Tests: 641 passed (no code change).
68 lines
2.5 KiB
Markdown
68 lines
2.5 KiB
Markdown
# `aborist.document`
|
|
|
|
The data structures every source produces and every storage layer
|
|
consumes. Three core types: `Document`, `Edge`, `Chunker`.
|
|
|
|
## `Document`
|
|
|
|
A single ingest unit: a Wikipedia article, an HTML page, a Grok
|
|
conversation, a git commit message, etc. Carries both the raw
|
|
content AND the version tags that determine its identity:
|
|
|
|
```python
|
|
@dataclass(frozen=True)
|
|
class Document:
|
|
document_uri: str # canonical URI (or stable surrogate for non-URI sources)
|
|
raw_content: str # source-of-truth bytes pre-canonicalization
|
|
kind: str # 'surface' / 'core' / 'visual' / etc.
|
|
chunking_version: str # e.g. 'tok-512-v1' — pinned by the Chunker
|
|
canonicalization_version: str # e.g. 'norm-v1' — pinned by canonicalize()
|
|
schema_version: str # e.g. 'v9.8.0' — store schema generation
|
|
title: str | None = None
|
|
edges: list[Edge] = () # outbound link graph
|
|
metadata: dict = ... # source-specific opaque payload
|
|
```
|
|
|
|
`document_root` is computed at ingest time as the Merkle root over
|
|
the canonicalized chunks. Two peers ingesting the same source +
|
|
running the same `chunking_version` + `canonicalization_version`
|
|
get bit-identical `document_root`s — the v9.8 admissibility property.
|
|
|
|
## `Edge`
|
|
|
|
One outbound link. `aborist/sources/wikipedia.py` emits one Edge
|
|
per `[[wikilink]]`; `aborist/sources/html_page.py` emits one per
|
|
`<a href>`. The link graph IS the corpus topology — `concepts/extract.py`
|
|
later reads `edges` rows to derive synonym relations from
|
|
reciprocal links (no separate crawler needed).
|
|
|
|
```python
|
|
@dataclass(frozen=True)
|
|
class Edge:
|
|
src_root: str # source document_root
|
|
dst_uri: str # always present
|
|
dst_root: str # '' (unresolved) until the dst doc is also ingested
|
|
edge_type: str # 'wikilink' / 'href' / 'citation' / 'derived_from' / ...
|
|
anchor: str # chunk index or fragment, '' if N/A
|
|
```
|
|
|
|
## `Chunker`
|
|
|
|
ABC with one method `chunk(text: str) -> list[str]`. Default impl
|
|
is `TokenChunker` (`name='tok-512-v1'`) — splits on token-rough
|
|
windows so the resulting chunks are predictable for downstream FTS5
|
|
indexing & for the LLM context budget.
|
|
|
|
**Changing the chunker bumps `chunking_version` AND stales every
|
|
prior cache record** (chunking is one of the 8 cache_key dimensions).
|
|
Don't redefine `tok-512-v1`; add a new chunker as a new `name`
|
|
instead.
|
|
|
|
## Diagrams
|
|
|
|
|
|
|
|
|
|
## Source
|
|
|
|
[`aborist/document.py`](../../aborist/document.py)
|