# `aborist.distill`

Surface → core distillation. Takes a set of surface documents
(the original ingest layer) and produces "core" documents — shorter,
more focused, Merkle-bound back to their contributing surface
chunks via per-chunk inclusion proofs.

The "trees and forests of cross-linked information" tagline aborist
takes its name from comes from this layer: planet-toward-center
compression where each layer of cores derives from the previous,
recursively.

## Layered design

```
distill/
├── base.py            Distiller ABC + DistillationResult dataclass
├── first_sentence.py  no-ML stub: take the first sentence of each doc
├── tfidf.py           pure-Python TF-IDF top-keyword extraction
└── runner.py          batched distillation + per-contrib-chunk proofs
```

## `Distiller` ABC

```python
from aborist.distill import Distiller, DistillationResult

class Distiller(ABC):
    @abstractmethod
    def distill(self, docs: list[Document]) -> DistillationResult: ...
```

Each `DistillationResult` carries the new core's content +
references to every contributing surface chunk by `(document_root,
chunk_root)`. The runner writes one `derivations` row per core,
with `proof_blob = json.dumps(per_chunk_inclusion_proofs)`.

## Built-in distillers

### `FirstSentenceDistiller` (no-ML stub)

Take the first sentence of each input doc, concatenate. Used as a
sanity-check for the pipeline + a baseline for measuring the
benefit of richer distillers.

### `TfidfKeywordDistiller`

Pure-Python TF-IDF. Computes term frequencies across the input
doc set + inverse document frequencies; emits the top-K terms per
doc as the core's content. The "permacomputer" neologism case
(every Grok conversation has the word, no Wikipedia article does)
is the canonical TF-IDF win — surfaces the topic that title-search
can't catch.

## Why distill

Three use cases:

1. **Retrieval signal.** Cores feed the third accept path in
   `_filter_by_title_relevance` — `core_match_roots` (TF-IDF top-
   keywords contain a query token). Closes the gap for neologisms
   that never make Wikipedia titles but ARE distinctive.

2. **Hot/cold tier discipline.** `evict_to_cold` only touches
   `kind='surface'` — cores never evict. Distilling surface to
   cores then evicting surfaces gives a "long tail keeps small
   cache" pattern with full provenance preserved.

3. **Recursive abstraction.** Cores can themselves be distilled
   into shorter cores. Each generation Merkle-binds back to the
   previous via `derivations.proof_blob` — the audit chain stays
   intact across an arbitrary distillation depth.

## Source

[`aborist/distill/`](../../aborist/distill/)
