arborist/docs/self-reference-design.md

Self-reference design — recursive grounding on the providence ledger

Date opened: 2026-05-01 Status: v1 (flat MVP) shipped — aborist/sources/providence.py, make ingest-self-providence, allowlist update for claim_lattice_allowed_source_roles. v2 (fact-Core distillation) — design proposal, implementation scoped to a follow-on commit pass. Audience: fox + future blackops shifts. Hard constraint: STRICT records are trusted as fact unless a verifier falsifies them. Other audit_modes stay opaque to retrieval until promoted. v2 extends the audit chain recursively without schema change.

---

1. Problem statement

Aborist's namesake is "tends trees and forests of cross-linked information." Today the system tends Wikipedia trees but never grafts its own past Q&A records into the forest. Every query starts from cold corpus retrieval; prior providence records sit in providence_cache unread. The system answers a question, stores the answer, and never looks at that answer again unless someone re-asks the exact same question (cache_key match).

The "kindergarten thought chains" framing names the gap: the system has a kindergarten of thoughts (early STRICT records) that should mature into citable substrate as they cool, then serve as anchors for new thoughts. Without that loop the substrate is a one-shot answerer, not a recursively-deepening reasoner.

Concretely: ask "what is verify_claim_lattice?" today and Hermes guesses from training. The right primary source — the verify.py source code or the design docs in docs/ — isn't in the corpus. Even after make ingest-self, only the code gets ingested; the system's answers about its own code stay invisible to retrieval.

Two design layers solve this. v1 flat MVP lands the surface step (records become flat documents). v2 fact-Core distillation lands the deep step (records become Merkle-bound facts that compose).

---

2. v1 — flat MVP (shipped)

2.1 Architecture

A new Source subclass — ProvidenceSource — iterates providence_cache records and yields each as a Document:

• URI: aborist://providence/<cache_key> — content-addressed, stable, distinguishable from Wikipedia / external URIs at retrieval time.
• Title: the question text (truncated to ~120 chars).
• Content: a canonical layout of Q: <question> then A: <answer_text> then per-claim line [E#: cite] if available. The content gets chunked + Merkle-rooted via the standard ingest pipeline.
• source_type: "providence".

Records are filtered at iteration time by:

1. audit_mode == "STRICT" — only fully-grounded records become substrate. HYBRID and UNGROUNDED stay opaque to retrieval (noisy or speculative).
2. falsification_state == "live" — falsified records (state ∈ {failed, stale, quarantined}) are excluded. The existing falsification machinery is the verifier-falsification mechanism: when aborist providence --falsify flips a record's state, it stops being substrate on next ingest.
3. now - created_at >= kindergarten_seconds (default 3600s = 1h). Fresh thoughts cool before they're recyclable. Mirrors the mesh-sync kindergarten window. Without this, the system would self-cite a record from 30 seconds ago — echo-chamber loops.
4. Anti-recursion: records whose own answer cited a self_reference_source are excluded — first-generation only. A wrong-but-STRICT record otherwise repeatedly recompiles into deeper claims and the chain rots silently.

2.2 Retrieval integration

The source-role classifier in aborist/qa/query.py:_classify_source_role recognizes the aborist://providence/ URI scheme and tags those documents self_reference_source. This role gets added to the default claim_lattice_allowed_source_roles allowlist so claims can verify against self-reference spans. SOURCE_ROLE_BUDGET_WEIGHTS for self_reference_source = 1.0 (same as background; deliberately not boosted — Wikipedia is still the canonical primary).

2.3 Recursive Merkle proof

When Q2 cites Q1's answer span, the audit chain becomes "Q2 → Q1 → Wikipedia chunk." The original chunk_root remains the leaf; Q1's run_dag_root becomes an intermediate node. v9.8's admissibility ledger already supports this layering — the recursive-cores insight ("planet toward center compression" in CLAUDE.md). No schema change needed; the providence record's merkle_proof blob carries the parent chain.

2.4 Operational model

• make ingest-self-providence — promote STRICT-live providence records older than the kindergarten window into the document corpus.
• Run on a cron (every hour, mirroring the mesh kindergarten window).
• Idempotent: same record → same document_root → no-op insert. Replaced records get a supersedes edge linking new → old.

2.5 What v1 does NOT do

• Aggregation of multiple Q&A records into a synthesized "summary" record.
• Cross-shard self-reference: each shard self-promotes within itself; cross-shard cites work via the existing --shards-dir UNION. No new code.
• Live retrieval from providence_cache (Option B from the design discussion). The MVP uses snapshot ingestion (Option A) for simplicity.
• HYBRID-record self-promotion. STRICT only.
• Compositional reasoning — records become reachable, but they don't yet COMPOSE into new claims. v2 addresses the composition gap.

2.6 v1 risks (still open)

risk	mitigation
Lazy-anchor false-STRICT compounds — Q2 inherits Q1's bogus cite.	lazy_anchor_demoted records skipped on promotion. NLI sidecar (docs/verifier-semantic-gap-design.md) adds another gate when it lands.
Echo-chamber: same fact recycled across many records.	Kindergarten window (1h) + first-gen-only anti-recursion check.
Falsified record stays in retrieval until re-ingest.	Verifier checks falsification_state of the cited source_root at verify time; non-live cites get rejected. Fail-closed.
Storage bloat: every Q&A becomes a document.	Same chunker + Merkle as everything else; per-record cost is small. Periodic burn-kindergarten trims.

---

3. v2 — fact-Core distillation (proposal)

3.1 Where v1 falls short

v1 makes records "another flat document source." The architecture wants them to be Merkle-bound facts that compose into new claims. v1 retrieves a record's text; it doesn't let the record's evidence chain attach as substrate for new reasoning. New claims about the same topic don't compose with old claims; they just see them as more context.

The deeper play: STRICT claims become Cores via the existing distillation pipeline. New claims derive from cores via the recursive-cores layer. The fact-graph grows.

3.2 The existing infrastructure already does most of this

The Surface → Core → Recursive-Core layering is in aborist/distill/:

aborist/distill/
├── base.py              # Distiller ABC + DistillationResult
├── first_sentence.py    # FirstSentenceDistiller (no-ML stub)
├── tfidf.py             # TfidfKeywordDistiller (pure-Python TF-IDF)
└── runner.py            # batched: derive + per-contrib-chunk proofs

Contract:

class Distiller(ABC):
    name: str
    def distill(self, source: Document, source_chunks: list[str]) -> DistillationResult:
        ...

The runner takes a surface Document, runs the Distiller, returns a Core Document plus contributing_chunk_indices. Per-chunk Merkle inclusion proofs against document_root get stored in derivations.proof_blob — Cores are cryptographically bound to their source surfaces.

What this gives us for free:

• A Core is itself a Document with its own document_root, chunkable + retrievable like Wikipedia content.
• The recursive-core mode (cores derive from cores) already exists in the runner.
• Every Core carries explicit lineage back to its source via per-chunk proofs.

The missing piece: a ProvidenceDistiller that takes a STRICT providence record and produces a Core, where the "source chunks" are the cited evidence spans the record verified against.

3.3 ProvidenceDistiller — cited-evidence-bound (Option B)

The Distiller contract today maps (surface Document, surface chunks) → core Document. For self-reference we want (STRICT record, cited evidence spans from OTHER documents) → fact-Core. The "source chunks" the Core derives from aren't the providence record's own chunks — they're the EVIDENCE SPANS the record cited in its claim_statuses.

Two options were considered:

• Option A — distill from the providence record's own content. ProvidenceDistiller treats the record's Q: ... A: ... text as the surface, distills it to a Core. Simple, fits the Distiller contract directly.
• Option B — distill from the cited evidence spans, with the record as an indirection. ProvidenceDistiller looks up the record's claim_statuses[].evidence_ids, fetches the cited evidence chunks from THEIR source documents, treats those as the "source chunks," and emits a Core that's bound by inclusion proof to the cited chunks of the cited Wikipedia documents.

Option B is the right deep version. A fact-Core derived from a STRICT claim is a Merkle-bound assertion that "claim text C is supported by chunk_root Cr1 in document_root Dr1." Future claims attaching to this Core inherit that evidence chain transparently. Option A would make Cores derive from the record's text (which already says what the answer is), losing the direct connection to the underlying Wikipedia facts.

Option B's cross-document fetch is supported today: --shards-dir UNION views let a single connect() see all shards as one read connection.

3.4 Core content shape

Three candidate shapes for the fact-Core's content:

SHAPE A — claim text only
    "Joey Potter is the girl across the creek in Dawson's Creek."

SHAPE B — claim + per-source pointer
    "Joey Potter is the girl across the creek in Dawson's Creek."
    [Dawson Leery (Wikipedia): "...the central fictional character..."]

SHAPE C — structured triple
    SUBJECT: Joey Potter
    PREDICATE: is the girl across the creek
    OBJECT: in Dawson's Creek
    EVIDENCE: chunk_root=ab12... offset_start=4032 offset_end=4189

Recommendation: Shape B for MVP. Pure prose with a tagged citation. Retrieval finds the prose; the cited span is right there. Shape C lands later if a fact-graph traversal becomes a real need (NER + relation extraction — out of scope today).

3.5 Recursive cores — facts grow new ideas

Surface (Wikipedia chunk)
    ↓ TfidfKeywordDistiller
Core-tfidf (keywords from Wikipedia chunk)

STRICT providence record
    ↓ ProvidenceDistiller (Option B — bound to cited Wikipedia chunks)
Fact-Core

[Future] N related Fact-Cores
    ↓ ?CompositionDistiller (deferred)
Composite-Fact-Core (claims combining multiple facts)

The v2 MVP does only the first ProvidenceDistiller pass. CompositionDistiller is the future shape that makes facts compose into new ideas — that's where "the substrate forms new claims from its own facts" lives. CompositionDistiller is hard because deciding which facts to compose, and how, is the actual reasoning step. Today's Hermes doesn't do that reliably. Defer.

What v2 ships: each STRICT claim becomes a Merkle-bound Fact-Core whose proof chain reaches all the way back to a Wikipedia chunk_root. Retrieval over Cores returns Fact-Cores alongside Wikipedia surfaces — the lattice grows. Composition is deferred but the substrate is in shape for it when we land it.

3.6 Recursive Merkle proof (the part already free)

When a future Q3 cites a Fact-Core that derived from a STRICT Q1 record citing Wikipedia chunk Cr1:

Q3 claim → Fact-Core → Cr1 → Dr1 → Sr1

Each link is a Merkle inclusion proof or a content-addressed lookup. No new schema is needed — derivations.proof_blob already holds the per-chunk inclusion proofs; the per-claim → Fact-Core → derivation walk just composes existing primitives. v9.8's audit chain extends naturally; we don't need v9.9.

3.7 Trust + falsification (sharper than v1)

• A Fact-Core is created only from a STRICT-live providence record past the kindergarten window.
• If the record is later falsified (falsification_state != live), the Fact-Core's derivations row is marked stale on next promotion run. Idempotent: same record → same Core hash. Falsified records don't promote.
• A future Q3 citing a stale Fact-Core fails verification at the source-state check (verifier checks falsification_state of the cited source's underlying records, not just the surface document).
• Fail-closed: a falsified Fact-Core CANNOT serve as substrate even if it's still in the documents table.

The key trust-model add over v1: falsification cascades. Falsifying Q1 stales Q1's Fact-Core, which stales Q2 if Q2 had cited the Fact-Core. The Merkle chain makes the cascade traceable.

3.8 Anti-recursion (kept from v1)

A providence record whose own answer text already cites a Fact-Core — i.e. a record answered by composing existing facts — gets ONE level of self-reference but cannot itself be promoted to a NEW Fact-Core. First-generation only. This kills echo-chamber chains where a wrong-but-STRICT record keeps recompiling itself into deeper claims. Conservative; the right relaxation is "promote when the lazy-anchor sidecar AND the NLI sidecar both pass" — but that's after both signals are in place.

---

4. v2 implementation plan (8 steps)

1. aborist/distill/providence.py — new module, ProvidenceDistiller(Distiller). Reads STRICT live providence records past kindergarten, fetches the cited evidence chunks (Option B), builds Shape-B Core content (claim text + tagged citation span), returns DistillationResult with contributing_chunk_indices pointing to the cited Wikipedia chunks.
2. Wire into aborist/distill/runner.py — register ProvidenceDistiller as a known kind. The existing batched-distill flow handles the per-chunk-proof generation transparently.
3. CLI: aborist distill --kind providence — adds the new kind to the distill subcommand's choices. Plumbs the --kindergarten-seconds knob from the v1 CLI work.
4. Makefile: distill-self-providence — runs aborist distill --kind providence against each shard. Hourly cron candidate.
5. Source-role classifier — Fact-Cores are tagged self_reference_source via the existing URI-prefix path (aborist://providence/... from v1 carries through). Cores derived from those records inherit the role. No classifier change needed.
6. Falsification cascade — when aborist providence --falsify flips a record's state, also mark the corresponding Fact-Core's derivation row as stale. New CLI flag or implicit on next ingest pass; tradeoff: explicit is debuggable, implicit is less coordinated.
7. Tests — unit tests for ProvidenceDistiller (correctly fetches cited chunks, builds Shape-B content, generates valid inclusion proofs), falsification cascade (falsified record → stale Core → rejected citation in new run).
8. Bench validation — re-run make bench-qa after make ingest-self-providence AND make distill-self-providence have populated some Fact-Cores. Compare to baseline. Questions about aborist itself (currently UNGROUNDED) should ground; questions tangential to past STRICT answers should gain new anchors.

---

5. What's deliberately NOT in this design

• CompositionDistiller — combining multiple Fact-Cores into a new claim. Reasoning machinery, not infrastructure. Defer until the soft-signal taxonomy (NLI, predicate compatibility) is mature enough that compositions can be sanity-checked.
• Shape-C structured triples — needs NER + relation extraction. Land Shape B first; promote to Shape C if a fact-graph use case actually needs subject-predicate-object retrieval.
• HYBRID record promotion — both v1 and v2 gate on STRICT only. HYBRID could become a self_reference_hybrid_source role with lower trust, separately gated.
• Cross-shard cascading falsification — falsifying a record on one shard doesn't auto-falsify a Fact-Core derived from it on another shard. Mesh-sync handles cross-shard coherence eventually; the immediate cascade is per-shard. Acceptable.

---

6. Bench impact (speculative, disciplined)

After a few hundred STRICT records have promoted to Fact-Cores:

• Questions about aborist itself (today's mostly UNGROUNDED) start grounding against Fact-Cores derived from past Q&A about aborist.
• Questions tangentially related to past STRICT answers gain anchors that reach back to Wikipedia transparently.
• Strict-rate creeps up as the fact-substrate matures; honest-grounded count rises faster.
• New failure modes: bad anchors landing inside Fact-Cores. The lazy-anchor sidecar already covers this layer-recursively because Fact-Cores look just like other documents to the verifier.
• Latency: same as Wikipedia retrieval. No new path; just more documents indexed.

---

7. The architectural payoff

Today the substrate is a one-shot answerer: every query starts cold, retrieves Wikipedia, prompts Hermes, verifies, caches. The cache is a key-value lookup, not a substrate for reasoning.

After v2 lands, the substrate becomes recursively-deepening: every STRICT answer becomes a Merkle-bound fact in the tree. New questions retrieve old facts as substrate. The fact-graph compounds. Wrong facts get falsified and the cascade reaches the dependent records. Right facts stay grounded and become the foundation for deeper claims.

That's "tends trees and forests of cross-linked information" — literally. The naming wasn't aspirational; it was load-bearing for the architecture.