# Self-reference distillation — STRICT facts as Merkle-bound substrate

**Date:** 2026-05-01
**Status:** design proposal. Successor to `docs/self-reference-thought-chains-design.md` (the flat-source MVP). Implementation scoped to a follow-on commit pass.
**Audience:** fox + future blackops shifts.
**Hard constraint:** STRICT records become Cores via the existing Distiller / Core / derivations infrastructure. Soft signals never enter the proof path. The audit chain extends recursively without schema change.

---

## 1. Where the flat MVP falls short

`docs/self-reference-thought-chains-design.md` (commit `8de0044`, 2026-05-01) lands a `ProvidenceSource(Source)` that promotes STRICT live providence_cache records into the document corpus. New retrieval surfaces them as `self_reference_source` citations. That's the surface step — necessary scaffolding so questions about aborist itself can ground at all.

But fox's deeper framing is that the MVP makes records "another flat document source," when the architecture wants them to be **Merkle-bound facts that compose into new claims**. The flat MVP retrieves a record's text; it doesn't let the record's *evidence chain* attach as substrate for new reasoning. New claims about the same topic don't compose with old claims; they just see them as more context.

The deeper play is: STRICT claims become **Cores** via the existing distillation pipeline. New claims derive from cores via the recursive-cores layer ("planet toward center compression" in CLAUDE.md). The fact-graph grows.

## 2. The existing infrastructure already does most of this

The Surface → Core → Recursive-Core layering is in `aborist/distill/`:

```
aborist/distill/
├── base.py              # Distiller ABC + DistillationResult
├── first_sentence.py    # FirstSentenceDistiller (no-ML stub)
├── tfidf.py             # TfidfKeywordDistiller (pure-Python TF-IDF)
└── runner.py            # batched: derive + per-contrib-chunk proofs
```

The contract:

```python
class Distiller(ABC):
    name: str
    def distill(self, source: Document, source_chunks: list[str]) -> DistillationResult:
        ...
```

The runner takes a surface Document, runs the Distiller, gets back a Core Document plus `contributing_chunk_indices`. The runner generates Merkle inclusion proofs for each contributing chunk against `document_root` and stores them in `derivations.proof_blob` — so the Core is **cryptographically bound** to its source surface.

What this gives us for free:
- The Core is itself a Document with its own `document_root`, chunkable + retrievable like Wikipedia content
- The recursive-core mode (cores derive from cores) already exists in the runner
- Every Core carries explicit lineage back to its source via per-chunk proofs

The missing piece: a `ProvidenceDistiller` that takes a STRICT providence record and produces a Core, where the "source chunks" are the cited evidence spans the record verified against.

## 3. Architecture

### 3.1 ProvidenceDistiller

A new `Distiller` subclass:

```python
class ProvidenceDistiller(Distiller):
    name = "providence-claim-v1"
    def distill(self, source, source_chunks):
        # source.uri = "aborist://providence/<cache_key>"
        # source.content = "Q: ...\n\nA: ..." (from ProvidenceSource)
        # source_chunks = the chunks of THIS providence record's content
        ...
```

But the right shape is subtler. The Distiller contract today maps `(surface Document, surface chunks) → core Document`. For self-reference we want `(STRICT record, cited evidence spans from OTHER documents) → fact-Core`. The "source chunks" the Core derives from aren't the providence record's own chunks — they're the EVIDENCE SPANS the record cited in its claim_statuses.

Two options:

**Option A — distill from the providence record's own content.**
ProvidenceDistiller treats the record's `Q: ... A: ...` text as the surface, distills it to a Core. The contributing-chunk-indices point into the providence record's own chunking. Simple, fits the Distiller contract directly.

**Option B — distill from the cited evidence spans, with the record as an indirection.**
ProvidenceDistiller looks up the record's `claim_statuses[].evidence_ids`, fetches the cited evidence chunks from THEIR source documents, treats those as the "source chunks," and emits a Core that's bound by inclusion proof to the cited chunks of the cited Wikipedia documents. The fact-Core carries direct provenance to the Wikipedia spans, not just to the providence record.

**Option B is the right deep version.** A fact-Core derived from a STRICT claim is a Merkle-bound assertion that "claim text C is supported by chunk_root Cr1 in document_root Dr1." Future claims attaching to this Core inherit that evidence chain transparently. Option A would make Cores derive from the record's text (which already says what the answer is), losing the direct connection to the underlying Wikipedia facts.

Implementation note: Option B requires the Distiller (or its runner) to fetch chunks from documents the providence record cites. The cross-shard attach machinery (`--shards-dir`) already lets a single `connect()` see all shards as UNION views, so the fetch is just a SELECT.

### 3.2 Core content shape

A fact-Core's content is the structured assertion. Three candidate shapes:

```text
SHAPE A — claim text only (haiku-like)
    "Joey Potter is the girl across the creek in Dawson's Creek."

SHAPE B — claim + per-source pointer
    "Joey Potter is the girl across the creek in Dawson's Creek."
    [Dawson Leery (Wikipedia): "...the central fictional character..."]

SHAPE C — structured triple form
    SUBJECT: Joey Potter
    PREDICATE: is the girl across the creek
    OBJECT: in Dawson's Creek
    EVIDENCE: chunk_root=ab12... offset_start=4032 offset_end=4189
```

Shape A is the simplest — pure text, chunkable, retrievable as prose. Shape B carries the cited span inline so retrieval surfaces it without extra DB lookups. Shape C is most useful for fact-graph composition (other claims can find the SUBJECT in their queries) but requires NER + relation extraction we don't have.

Recommendation: **Shape B for MVP**. Pure prose with a tagged citation. Retrieval finds the prose; the cited span is right there. Shape C lands later if a fact-graph traversal becomes a real need.

### 3.3 Recursive cores — facts grow new ideas

The runner's recursive-core mode lets a Core be the input to another Distiller pass. So:

```text
Surface (Wikipedia chunk)
    ↓ TfidfKeywordDistiller
Core-tfidf (keywords from Wikipedia chunk)

STRICT providence record
    ↓ ProvidenceDistiller (with Option B — bound to cited Wikipedia chunks)
Fact-Core

[Later] N related Fact-Cores
    ↓ ?CompositionDistiller (future, out of MVP scope)
Composite-Fact-Core (claims that combine multiple facts)
```

The MVP does only the first ProvidenceDistiller pass. CompositionDistiller is the future shape that makes facts compose into new ideas — that's where "the substrate forms new claims from its own facts" lives. CompositionDistiller is hard because deciding which facts to compose, and how, is the actual reasoning step. Today's Hermes doesn't do that reliably. That's why we're not building CompositionDistiller in this MVP.

What we ARE building: each STRICT claim becomes a Merkle-bound Fact-Core whose proof chain reaches all the way back to a Wikipedia chunk_root. Retrieval over Cores returns Fact-Cores alongside Wikipedia surface — the lattice grows. Composition is deferred but the substrate is in shape for it when we land it.

### 3.4 Recursive Merkle proof (the part that's already free)

When a future Q3 cites a Fact-Core that itself derived from a STRICT Q1 record that cited Wikipedia chunk Cr1:

```
Q3 claim → cites evidence_id E_x in Q3's run-DAG
E_x → Fact-Core's evidence_id (content-addressed)
Fact-Core's derivation row → has proof_blob containing inclusion proof
                              of chunk Cr1 against document_root Dr1
Wikipedia document Dr1 → has source_root Sr1 in source corpus
```

The chain is:

```
Q3 claim → Fact-Core → Cr1 → Dr1 → Sr1
```

Each link is a Merkle inclusion proof or a content-addressed lookup. No new schema is needed — `derivations.proof_blob` already holds the per-chunk inclusion proofs; the per-claim → Fact-Core → derivation walk just composes existing primitives.

Per CLAUDE.md "Merkle: non-commutative HashCombine with prefix `0x03`" — the same hash discipline applies all the way down. v9.8's audit chain extends naturally; we don't need v9.9.

### 3.5 Trust + falsification

Same as the flat MVP, sharper:

- A Fact-Core is created only from a STRICT-live providence record past the kindergarten window
- If the record is later falsified (`falsification_state != live`), the Fact-Core's `derivations` row is marked stale on next promotion run. Idempotent: same record → same Core hash. Falsified records don't promote.
- A future Q3 citing a stale Fact-Core fails verification at the source-state check (verifier checks `falsification_state` of the cited source's underlying records, not just the surface document)
- **Fail-closed: a falsified Fact-Core CANNOT serve as substrate even if it's still in the documents table.**

This is the key trust-model add: STRICT-as-fact unless falsified, AND the falsification cascades — falsifying Q1 stales Q1's Fact-Core, which stales Q2 if Q2 had cited the Fact-Core. The Merkle chain makes the cascade traceable.

### 3.6 Anti-recursion (kept from MVP)

A providence record whose own answer text already cites a Fact-Core — i.e. a record that was answered by composing existing facts — gets ONE level of self-reference but cannot itself be promoted to a NEW Fact-Core. First-generation only. This kills echo-chamber chains where a wrong-but-STRICT record keeps recompiling itself into deeper claims.

This is conservative; the right relaxation is "promote when the lazy-anchor sidecar AND the NLI sidecar both pass" — but that's after both signals are in place.

## 4. Implementation plan (high-level, 8 steps)

1. **`aborist/distill/providence.py`** — new module, `ProvidenceDistiller(Distiller)`. Reads STRICT live providence records past kindergarten, fetches the cited evidence chunks (Option B), builds Shape-B Core content (claim text + tagged citation span), returns `DistillationResult` with contributing_chunk_indices pointing to the cited Wikipedia chunks.
2. **Wire into `aborist/distill/runner.py`** — registers ProvidenceDistiller as a known kind. The existing batched-distill flow handles the per-chunk-proof generation transparently.
3. **CLI: `aborist distill --kind providence`** — adds the new kind to the `distill` subcommand's choices. Plumbs through the `--kindergarten-seconds` knob from the flat-MVP CLI work.
4. **Makefile: `distill-self-providence`** — runs `aborist distill --kind providence` against each shard. Hourly cron candidate.
5. **Source-role classifier** — Fact-Cores are tagged `self_reference_source` via the existing URI-prefix path (`aborist://providence/...` from the flat MVP carries through). Cores derived from those records inherit the role. No classifier change needed.
6. **Falsification cascade** — when `aborist providence --falsify` flips a record's state, also mark the corresponding Fact-Core's derivation row as stale. New CLI flag or implicit on next ingest pass; tradeoff: explicit is debuggable, implicit is less coordinated.
7. **Tests** — unit tests for ProvidenceDistiller (correctly fetches cited chunks, builds Shape-B content, generates valid inclusion proofs), falsification cascade (falsified record → stale Core → rejected citation in new run).
8. **Bench validation** — re-run `make bench-qa` after `make ingest-self-providence` AND `make distill-self-providence` have populated some Fact-Cores. Compare to baseline. Questions about aborist itself (currently UNGROUNDED) should ground; questions tangential to past STRICT answers should gain new anchors.

## 5. What's deliberately NOT in this design

- **CompositionDistiller** — combining multiple Fact-Cores into a new claim. Reasoning machinery, not infrastructure. Defer until the soft-signal taxonomy (NLI, predicate compatibility) is mature enough that compositions can be sanity-checked.
- **Shape-C structured triples** — needs NER + relation extraction. Land Shape B first; promote to Shape C if a fact-graph use case actually needs subject-predicate-object retrieval.
- **HYBRID record promotion** — flat MVP and this design both gate on STRICT only. HYBRID could become a `self_reference_hybrid_source` role with lower trust, separately gated.
- **Cross-shard cascading falsification** — falsifying a record on one shard doesn't auto-falsify a Fact-Core derived from it on another shard. Mesh-sync handles cross-shard coherence eventually, but the immediate cascade is per-shard. Acceptable.

## 6. Risks + mitigations

| risk | mitigation |
|------|-----------|
| Lazy-anchor compounding: STRICT-but-bogus record gets promoted to Fact-Core; new claims cite it; the lattice grows around a wrong fact. | Anti-recursion (first-generation only). Once `lazy_anchor_demoted` and the NLI sidecar (per `docs/verifier-semantic-gap-design.md`) ship, gate promotion on those passing too. |
| Storage bloat: every STRICT record produces a Core with its own document_root, chunks, derivation row. | Same chunker + Merkle as Wikipedia ingestion; per-record cost is small. The kindergarten window + STRICT-only filter keep volume low. Periodic `burn-kindergarten` trims. |
| Schema/policy drift: a Core promoted under v9.8.0 stops being valid when chunking_version bumps. | Same schema-invariant rule as Wikipedia; `chunking_version` change stales every Core on lookup. Re-promote with the new version on next pass. |
| Fact-Core text leaks information from the cited evidence span — the renderer's spotlight excerpt becomes load-bearing. | Shape B inlines the spotlight span, which IS the source content. Acceptable; the substrate's whole purpose is making cited content reachable. |
| Cross-shard fetching: ProvidenceDistiller's Option-B fetch needs chunks from documents that may live on a different shard than the providence record. | The existing `--shards-dir` UNION views handle this; distill runs against the unioned read connection. |

## 7. Bench impact (speculative, disciplined)

After a few hundred STRICT records have promoted to Fact-Cores:

- Questions about aborist itself (today's ~UNGROUNDED) start grounding against Fact-Cores derived from past Q&A about aborist
- Questions tangentially related to past STRICT answers gain anchors that reach back to Wikipedia transparently
- Strict-rate creeps up as the fact-substrate matures; honest-grounded count rises faster
- New failure modes: bad anchors landing inside Fact-Cores. The lazy-anchor sidecar already covers this layer-recursively because Fact-Cores look just like other documents to the verifier.
- Latency: same as Wikipedia retrieval. No new path; just more documents indexed.

The forest grows. The trees stay individually verifiable.

## 8. The architectural payoff

Today the substrate is a one-shot answerer: every query starts cold, retrieves Wikipedia, prompts Hermes, verifies, caches. The cache is a key-value lookup, not a substrate for reasoning.

After this design lands, the substrate becomes recursively-deepening: every STRICT answer becomes a Merkle-bound fact in the tree. New questions retrieve old facts as substrate. The fact-graph compounds. Wrong facts get falsified and the cascade reaches the dependent records. Right facts stay grounded and become the foundation for deeper claims.

That's "tends trees and forests of cross-linked information" — literally. The naming wasn't aspirational; it was load-bearing for the architecture.
