russell@unturf.com
326badf6d8
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).
(1) README — label refresh:
- Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
- Verifier section spells out both layers (schema trichotomy +
display ladder), the seven hard checks of claim_lattice, and
the five anchor classes of warrant.
- Architecture tree updated: concepts/ package added, qa/
sub-modules expanded (warrant, evidence, parse_claims, dag),
verify.py described as quote/span/entity/paraphrase + claim_lattice.
- Concept overlay description updated for corpus-derived layer
(concept_relations table, link_reciprocity extractor, 1.6%
tax cite).
- Test count: 326+ → 641+.
(2) docs/diagrams/ — Graphviz dot sources:
- aborist-modules.dot — top-level package graph (substrate /
storage / sources / retrieval / qa / mesh / cli)
- query-pipeline.dot — question → cache → retrieval → LLM →
verify → render → cache write, with phase budgets
- ingest-pipeline.dot — source doc → canonicalize → chunk →
Merkle → upsert (+ optional distill)
- verifier-ladder.dot — (audit_mode, violations) → display rung
decision tree
Existing mesh-*.dot kept as-is. Makefile `make docs` target
extended to also emit .svg alongside the existing .png so the
diagrams render in markdown viewers.
(3) docs/modules/ — per-module reference pages:
- index.md (links to every diagram + every module page)
- merkle.md, document.md, store.md, ingest.md, evict.md,
sources.md, search.md, concepts.md, qa.md, distill.md,
wikitext.md
Each page is a one-screenful concise reference: what the
module is for, public API, key invariants, embedded diagrams
where useful, link to source. Mesh stays at the existing
docs/mesh.md + docs/mesh-deploy.md (already comprehensive).
Tests: 641 passed (no code change).
2.6 KiB
aborist.distill
Surface → core distillation. Takes a set of surface documents (the original ingest layer) and produces "core" documents — shorter, more focused, Merkle-bound back to their contributing surface chunks via per-chunk inclusion proofs.
The "trees and forests of cross-linked information" tagline aborist takes its name from comes from this layer: planet-toward-center compression where each layer of cores derives from the previous, recursively.
Layered design
distill/
├── base.py Distiller ABC + DistillationResult dataclass
├── first_sentence.py no-ML stub: take the first sentence of each doc
├── tfidf.py pure-Python TF-IDF top-keyword extraction
└── runner.py batched distillation + per-contrib-chunk proofs
Distiller ABC
from aborist.distill import Distiller, DistillationResult
class Distiller(ABC):
@abstractmethod
def distill(self, docs: list[Document]) -> DistillationResult: ...
Each DistillationResult carries the new core's content +
references to every contributing surface chunk by (document_root, chunk_root). The runner writes one derivations row per core,
with proof_blob = json.dumps(per_chunk_inclusion_proofs).
Built-in distillers
FirstSentenceDistiller (no-ML stub)
Take the first sentence of each input doc, concatenate. Used as a sanity-check for the pipeline + a baseline for measuring the benefit of richer distillers.
TfidfKeywordDistiller
Pure-Python TF-IDF. Computes term frequencies across the input doc set + inverse document frequencies; emits the top-K terms per doc as the core's content. The "permacomputer" neologism case (every Grok conversation has the word, no Wikipedia article does) is the canonical TF-IDF win — surfaces the topic that title-search can't catch.
Why distill
Three use cases:
-
Retrieval signal. Cores feed the third accept path in
_filter_by_title_relevance—core_match_roots(TF-IDF top- keywords contain a query token). Closes the gap for neologisms that never make Wikipedia titles but ARE distinctive. -
Hot/cold tier discipline.
evict_to_coldonly toucheskind='surface'— cores never evict. Distilling surface to cores then evicting surfaces gives a "long tail keeps small cache" pattern with full provenance preserved. -
Recursive abstraction. Cores can themselves be distilled into shorter cores. Each generation Merkle-binds back to the previous via
derivations.proof_blob— the audit chain stays intact across an arbitrary distillation depth.