arborist/docs/self-reference-distillation-design.md

Self-reference distillation — STRICT facts as Merkle-bound substrate

Date: 2026-05-01 Status: design proposal. Successor to docs/self-reference-thought-chains-design.md (the flat-source MVP). Implementation scoped to a follow-on commit pass. Audience: fox + future blackops shifts. Hard constraint: STRICT records become Cores via the existing Distiller / Core / derivations infrastructure. Soft signals never enter the proof path. The audit chain extends recursively without schema change.

---

1. Where the flat MVP falls short

docs/self-reference-thought-chains-design.md (commit 8de0044, 2026-05-01) lands a ProvidenceSource(Source) that promotes STRICT live providence_cache records into the document corpus. New retrieval surfaces them as self_reference_source citations. That's the surface step — necessary scaffolding so questions about aborist itself can ground at all.

But fox's deeper framing is that the MVP makes records "another flat document source," when the architecture wants them to be Merkle-bound facts that compose into new claims. The flat MVP retrieves a record's text; it doesn't let the record's evidence chain attach as substrate for new reasoning. New claims about the same topic don't compose with old claims; they just see them as more context.

The deeper play is: STRICT claims become Cores via the existing distillation pipeline. New claims derive from cores via the recursive-cores layer ("planet toward center compression" in CLAUDE.md). The fact-graph grows.

2. The existing infrastructure already does most of this

The Surface → Core → Recursive-Core layering is in aborist/distill/:

aborist/distill/
├── base.py              # Distiller ABC + DistillationResult
├── first_sentence.py    # FirstSentenceDistiller (no-ML stub)
├── tfidf.py             # TfidfKeywordDistiller (pure-Python TF-IDF)
└── runner.py            # batched: derive + per-contrib-chunk proofs

The contract:

class Distiller(ABC):
    name: str
    def distill(self, source: Document, source_chunks: list[str]) -> DistillationResult:
        ...

The runner takes a surface Document, runs the Distiller, gets back a Core Document plus contributing_chunk_indices. The runner generates Merkle inclusion proofs for each contributing chunk against document_root and stores them in derivations.proof_blob — so the Core is cryptographically bound to its source surface.

What this gives us for free:

• The Core is itself a Document with its own document_root, chunkable + retrievable like Wikipedia content
• The recursive-core mode (cores derive from cores) already exists in the runner
• Every Core carries explicit lineage back to its source via per-chunk proofs

The missing piece: a ProvidenceDistiller that takes a STRICT providence record and produces a Core, where the "source chunks" are the cited evidence spans the record verified against.

3. Architecture

3.1 ProvidenceDistiller

A new Distiller subclass:

class ProvidenceDistiller(Distiller):
    name = "providence-claim-v1"
    def distill(self, source, source_chunks):
        # source.uri = "aborist://providence/<cache_key>"
        # source.content = "Q: ...\n\nA: ..." (from ProvidenceSource)
        # source_chunks = the chunks of THIS providence record's content
        ...

But the right shape is subtler. The Distiller contract today maps (surface Document, surface chunks) → core Document. For self-reference we want (STRICT record, cited evidence spans from OTHER documents) → fact-Core. The "source chunks" the Core derives from aren't the providence record's own chunks — they're the EVIDENCE SPANS the record cited in its claim_statuses.

Two options:

Option A — distill from the providence record's own content. ProvidenceDistiller treats the record's Q: ... A: ... text as the surface, distills it to a Core. The contributing-chunk-indices point into the providence record's own chunking. Simple, fits the Distiller contract directly.

Option B — distill from the cited evidence spans, with the record as an indirection. ProvidenceDistiller looks up the record's claim_statuses[].evidence_ids, fetches the cited evidence chunks from THEIR source documents, treats those as the "source chunks," and emits a Core that's bound by inclusion proof to the cited chunks of the cited Wikipedia documents. The fact-Core carries direct provenance to the Wikipedia spans, not just to the providence record.

Option B is the right deep version. A fact-Core derived from a STRICT claim is a Merkle-bound assertion that "claim text C is supported by chunk_root Cr1 in document_root Dr1." Future claims attaching to this Core inherit that evidence chain transparently. Option A would make Cores derive from the record's text (which already says what the answer is), losing the direct connection to the underlying Wikipedia facts.

Implementation note: Option B requires the Distiller (or its runner) to fetch chunks from documents the providence record cites. The cross-shard attach machinery (--shards-dir) already lets a single connect() see all shards as UNION views, so the fetch is just a SELECT.

3.2 Core content shape

A fact-Core's content is the structured assertion. Three candidate shapes:

SHAPE A — claim text only (haiku-like)
    "Joey Potter is the girl across the creek in Dawson's Creek."

SHAPE B — claim + per-source pointer
    "Joey Potter is the girl across the creek in Dawson's Creek."
    [Dawson Leery (Wikipedia): "...the central fictional character..."]

SHAPE C — structured triple form
    SUBJECT: Joey Potter
    PREDICATE: is the girl across the creek
    OBJECT: in Dawson's Creek
    EVIDENCE: chunk_root=ab12... offset_start=4032 offset_end=4189

Shape A is the simplest — pure text, chunkable, retrievable as prose. Shape B carries the cited span inline so retrieval surfaces it without extra DB lookups. Shape C is most useful for fact-graph composition (other claims can find the SUBJECT in their queries) but requires NER + relation extraction we don't have.

Recommendation: Shape B for MVP. Pure prose with a tagged citation. Retrieval finds the prose; the cited span is right there. Shape C lands later if a fact-graph traversal becomes a real need.

3.3 Recursive cores — facts grow new ideas

The runner's recursive-core mode lets a Core be the input to another Distiller pass. So:

Surface (Wikipedia chunk)
    ↓ TfidfKeywordDistiller
Core-tfidf (keywords from Wikipedia chunk)

STRICT providence record
    ↓ ProvidenceDistiller (with Option B — bound to cited Wikipedia chunks)
Fact-Core

[Later] N related Fact-Cores
    ↓ ?CompositionDistiller (future, out of MVP scope)
Composite-Fact-Core (claims that combine multiple facts)

The MVP does only the first ProvidenceDistiller pass. CompositionDistiller is the future shape that makes facts compose into new ideas — that's where "the substrate forms new claims from its own facts" lives. CompositionDistiller is hard because deciding which facts to compose, and how, is the actual reasoning step. Today's Hermes doesn't do that reliably. That's why we're not building CompositionDistiller in this MVP.

What we ARE building: each STRICT claim becomes a Merkle-bound Fact-Core whose proof chain reaches all the way back to a Wikipedia chunk_root. Retrieval over Cores returns Fact-Cores alongside Wikipedia surface — the lattice grows. Composition is deferred but the substrate is in shape for it when we land it.

3.4 Recursive Merkle proof (the part that's already free)

When a future Q3 cites a Fact-Core that itself derived from a STRICT Q1 record that cited Wikipedia chunk Cr1:

Q3 claim → cites evidence_id E_x in Q3's run-DAG
E_x → Fact-Core's evidence_id (content-addressed)
Fact-Core's derivation row → has proof_blob containing inclusion proof
                              of chunk Cr1 against document_root Dr1
Wikipedia document Dr1 → has source_root Sr1 in source corpus

The chain is:

Q3 claim → Fact-Core → Cr1 → Dr1 → Sr1

Each link is a Merkle inclusion proof or a content-addressed lookup. No new schema is needed — derivations.proof_blob already holds the per-chunk inclusion proofs; the per-claim → Fact-Core → derivation walk just composes existing primitives.

Per CLAUDE.md "Merkle: non-commutative HashCombine with prefix 0x03" — the same hash discipline applies all the way down. v9.8's audit chain extends naturally; we don't need v9.9.

3.5 Trust + falsification

Same as the flat MVP, sharper:

• A Fact-Core is created only from a STRICT-live providence record past the kindergarten window
• If the record is later falsified (falsification_state != live), the Fact-Core's derivations row is marked stale on next promotion run. Idempotent: same record → same Core hash. Falsified records don't promote.
• A future Q3 citing a stale Fact-Core fails verification at the source-state check (verifier checks falsification_state of the cited source's underlying records, not just the surface document)
• Fail-closed: a falsified Fact-Core CANNOT serve as substrate even if it's still in the documents table.

This is the key trust-model add: STRICT-as-fact unless falsified, AND the falsification cascades — falsifying Q1 stales Q1's Fact-Core, which stales Q2 if Q2 had cited the Fact-Core. The Merkle chain makes the cascade traceable.

3.6 Anti-recursion (kept from MVP)

A providence record whose own answer text already cites a Fact-Core — i.e. a record that was answered by composing existing facts — gets ONE level of self-reference but cannot itself be promoted to a NEW Fact-Core. First-generation only. This kills echo-chamber chains where a wrong-but-STRICT record keeps recompiling itself into deeper claims.

This is conservative; the right relaxation is "promote when the lazy-anchor sidecar AND the NLI sidecar both pass" — but that's after both signals are in place.

4. Implementation plan (high-level, 8 steps)

1. aborist/distill/providence.py — new module, ProvidenceDistiller(Distiller). Reads STRICT live providence records past kindergarten, fetches the cited evidence chunks (Option B), builds Shape-B Core content (claim text + tagged citation span), returns DistillationResult with contributing_chunk_indices pointing to the cited Wikipedia chunks.
2. Wire into aborist/distill/runner.py — registers ProvidenceDistiller as a known kind. The existing batched-distill flow handles the per-chunk-proof generation transparently.
3. CLI: aborist distill --kind providence — adds the new kind to the distill subcommand's choices. Plumbs through the --kindergarten-seconds knob from the flat-MVP CLI work.
4. Makefile: distill-self-providence — runs aborist distill --kind providence against each shard. Hourly cron candidate.
5. Source-role classifier — Fact-Cores are tagged self_reference_source via the existing URI-prefix path (aborist://providence/... from the flat MVP carries through). Cores derived from those records inherit the role. No classifier change needed.
6. Falsification cascade — when aborist providence --falsify flips a record's state, also mark the corresponding Fact-Core's derivation row as stale. New CLI flag or implicit on next ingest pass; tradeoff: explicit is debuggable, implicit is less coordinated.
7. Tests — unit tests for ProvidenceDistiller (correctly fetches cited chunks, builds Shape-B content, generates valid inclusion proofs), falsification cascade (falsified record → stale Core → rejected citation in new run).
8. Bench validation — re-run make bench-qa after make ingest-self-providence AND make distill-self-providence have populated some Fact-Cores. Compare to baseline. Questions about aborist itself (currently UNGROUNDED) should ground; questions tangential to past STRICT answers should gain new anchors.

5. What's deliberately NOT in this design

• CompositionDistiller — combining multiple Fact-Cores into a new claim. Reasoning machinery, not infrastructure. Defer until the soft-signal taxonomy (NLI, predicate compatibility) is mature enough that compositions can be sanity-checked.
• Shape-C structured triples — needs NER + relation extraction. Land Shape B first; promote to Shape C if a fact-graph use case actually needs subject-predicate-object retrieval.
• HYBRID record promotion — flat MVP and this design both gate on STRICT only. HYBRID could become a self_reference_hybrid_source role with lower trust, separately gated.
• Cross-shard cascading falsification — falsifying a record on one shard doesn't auto-falsify a Fact-Core derived from it on another shard. Mesh-sync handles cross-shard coherence eventually, but the immediate cascade is per-shard. Acceptable.

6. Risks + mitigations

risk	mitigation
Lazy-anchor compounding: STRICT-but-bogus record gets promoted to Fact-Core; new claims cite it; the lattice grows around a wrong fact.	Anti-recursion (first-generation only). Once lazy_anchor_demoted and the NLI sidecar (per docs/verifier-semantic-gap-design.md) ship, gate promotion on those passing too.
Storage bloat: every STRICT record produces a Core with its own document_root, chunks, derivation row.	Same chunker + Merkle as Wikipedia ingestion; per-record cost is small. The kindergarten window + STRICT-only filter keep volume low. Periodic burn-kindergarten trims.
Schema/policy drift: a Core promoted under v9.8.0 stops being valid when chunking_version bumps.	Same schema-invariant rule as Wikipedia; chunking_version change stales every Core on lookup. Re-promote with the new version on next pass.
Fact-Core text leaks information from the cited evidence span — the renderer's spotlight excerpt becomes load-bearing.	Shape B inlines the spotlight span, which IS the source content. Acceptable; the substrate's whole purpose is making cited content reachable.
Cross-shard fetching: ProvidenceDistiller's Option-B fetch needs chunks from documents that may live on a different shard than the providence record.	The existing --shards-dir UNION views handle this; distill runs against the unioned read connection.

7. Bench impact (speculative, disciplined)

After a few hundred STRICT records have promoted to Fact-Cores:

• Questions about aborist itself (today's ~UNGROUNDED) start grounding against Fact-Cores derived from past Q&A about aborist
• Questions tangentially related to past STRICT answers gain anchors that reach back to Wikipedia transparently
• Strict-rate creeps up as the fact-substrate matures; honest-grounded count rises faster
• New failure modes: bad anchors landing inside Fact-Cores. The lazy-anchor sidecar already covers this layer-recursively because Fact-Cores look just like other documents to the verifier.
• Latency: same as Wikipedia retrieval. No new path; just more documents indexed.

The forest grows. The trees stay individually verifiable.

8. The architectural payoff

Today the substrate is a one-shot answerer: every query starts cold, retrieves Wikipedia, prompts Hermes, verifies, caches. The cache is a key-value lookup, not a substrate for reasoning.

After this design lands, the substrate becomes recursively-deepening: every STRICT answer becomes a Merkle-bound fact in the tree. New questions retrieve old facts as substrate. The fact-graph compounds. Wrong facts get falsified and the cascade reaches the dependent records. Right facts stay grounded and become the foundation for deeper claims.

That's "tends trees and forests of cross-linked information" — literally. The naming wasn't aspirational; it was load-bearing for the architecture.