# Self-reference thought chains — recursive grounding on the providence ledger

**Date:** 2026-05-01
**Status:** design + MVP implementation. New module `aborist/sources/providence.py`, new Makefile target `ingest-self-providence`, allowlist update for `claim_lattice_allowed_source_roles`. Unit-tested. Live integration deferred (requires running ingest then bench).
**Audience:** fox + future blackops shifts.
**Hard constraint:** STRICT records are trusted as fact unless a verifier falsifies them. Other audit_modes stay opaque to retrieval until promoted.

---

## 1. Problem statement

Aborist's namesake is "tends trees and forests of cross-linked information." Today the system tends Wikipedia trees but never grafts its own past Q&A records into the forest. Every query starts from cold corpus retrieval; prior providence records sit in `providence_cache` unread. The system answers a question, stores the answer in the cache, and never looks at that answer again unless someone re-asks the exact same question (cache_key match).

The "kindergarten thought chains" framing names the gap: the system has a kindergarten of thoughts (early STRICT records) that should mature into citable substrate as they cool, then serve as anchors for new thoughts. Without that loop the substrate is a one-shot answerer, not a recursively-deepening reasoner.

Concretely: ask "what is verify_claim_lattice?" today and Hermes guesses from training. The right primary source — the verify.py source code or the design docs in `docs/` — isn't in the corpus. Even if you `make ingest-self`, only the *code* gets ingested; the system's *answers about its own code* stay invisible to retrieval.

## 2. Design

### 2.1 Architecture

A new `Source` subclass — `ProvidenceSource` — iterates `providence_cache` records and yields each as a `Document`:

- **URI**: `aborist://providence/<cache_key>` — content-addressed, stable, distinguishable from Wikipedia/external URIs at retrieval time.
- **Title**: the question text (truncated to ~120 chars).
- **Content**: a canonical layout of `Q: <question>` then `A: <answer_text>` then per-claim line `[E#: cite]` if available. The content gets chunked + Merkle-rooted via the standard ingest pipeline.
- **source_type**: `"providence"`.

Records are filtered at iteration time by:

1. **`audit_mode == "STRICT"`** — only fully-grounded records become substrate. HYBRID and UNGROUNDED stay opaque to retrieval (they're noisy or speculative).
2. **`falsification_state == "live"`** — falsified records (state ∈ {failed, stale, quarantined}) are excluded. The existing falsification machinery is the verifier-falsification mechanism fox asked for: when `aborist providence --falsify` flips a record's state, it stops being substrate on next ingest.
3. **`now - created_at >= kindergarten_seconds`** (default 3600s = 1h). Fresh thoughts cool before they're recyclable. Mirrors the mesh-sync kindergarten window. Without this, the system would self-cite a record from 30 seconds ago and you get echo-chamber loops.
4. **Anti-recursion**: records whose own answer cited a `self_reference_source` are excluded — first-generation only. Otherwise a wrong-but-STRICT record gets repeatedly recompiled into deeper claims and the chain rots silently. (This is conservative; second-gen self-citation may land later if the lazy-anchor / NLI-sidecar work proves the error rate is low enough.)

### 2.2 Retrieval integration

The source-role classifier in `aborist/qa/query.py:_classify_source_role` recognizes the `aborist://providence/` URI scheme and tags those documents `self_reference_source`. This role gets added to the default `claim_lattice_allowed_source_roles` allowlist so claims can verify against self-reference spans.

`SOURCE_ROLE_BUDGET_WEIGHTS` for `self_reference_source` = 1.0 (same as background; deliberately not boosted — Wikipedia is still the canonical primary).

### 2.3 Recursive Merkle proof

When Q2 cites Q1's answer span, the audit chain becomes "Q2 → Q1 → Wikipedia chunk." The original `chunk_root` remains the leaf; Q1's `run_dag_root` becomes an intermediate node. v9.8's admissibility ledger already supports this layering — that's the recursive-cores insight ("planet toward center compression" in CLAUDE.md). No schema change needed; the providence record's `merkle_proof` blob carries the parent chain.

### 2.4 Falsification trust model

Per fox: "we trust strict statements as fact unless a verifier falsifies it."

- STRICT live records → trusted, ingested as substrate
- A falsifier (manual via `aborist providence --falsify`, drift-detection in `evict_to_cold` rehydrate, mesh-sync mismatch) flips state to `failed` / `stale` / `quarantined`
- On next `ingest-self-providence` run, the falsified record's document gets removed from FTS (idempotent re-ingest with `supersedes` edge — the documents table already supports this)
- Until that re-ingest, the stale record stays in retrieval but the verifier on a Q2 citing it sees `falsification_state != live` for the underlying `source_root` and rejects the citation. **Fail-closed: a falsified record CANNOT serve as STRICT substrate even if it's still in the documents table.**

### 2.5 Operational model

- `make ingest-self-providence` — promote STRICT-live providence records older than the kindergarten window into the document corpus
- Run on a cron (every hour, mirroring the mesh kindergarten window)
- Idempotent: same record → same document_root → no-op insert. Replaced records get a `supersedes` edge linking new → old.

### 2.6 What's NOT in this design

- **Aggregation** of multiple Q&A records into a synthesized "summary" record. Possible follow-on but not MVP.
- **Cross-shard self-reference**: each shard can self-promote within itself; cross-shard cites work via the existing shards-dir UNION. No new code.
- **Live retrieval from providence_cache** (Option B from the design discussion). The MVP uses snapshot ingestion (Option A) for simplicity. Live virtual sourcing would be an optimization if duplication becomes painful.
- **Self-reference for HYBRID records**. STRICT only. HYBRID could be added as a soft-anchor source role with lower trust (own role: `self_reference_hybrid_source`, role weight 0.5). Holding for fox's call.

## 3. Risks & mitigations

| risk | mitigation |
|------|-----------|
| Lazy-anchor false-STRICT compounds — Q2 inherits Q1's bogus cite. | `lazy_anchor_demoted` records skipped on promotion. Once the NLI sidecar lands, `semantic_demoted` adds another gate. |
| Echo-chamber: same fact recycled across many records. | Kindergarten window (1h) + first-gen-only anti-recursion check. |
| Falsified record stays in retrieval until re-ingest. | Verifier checks `falsification_state` of the cited source_root at verify time; non-live cites get rejected. Fail-closed. |
| Storage bloat: every Q&A becomes a document. | Same chunker + Merkle as everything else; per-record cost is small. Periodic burn (`burn-kindergarten`) trims old or low-quality rows. |
| Schema/policy drift: a record promoted under v9.8.0 becomes invalid when chunking_version bumps. | Same schema-invariant rule as Wikipedia ingestion; `governance_policy_hash` migration applies. |

## 4. Bench impact estimate

Before any self-reference is in the corpus, no impact. Once `make ingest-self-providence` has been run for a while and a few hundred STRICT records are substrate:

- Questions about aborist itself (mostly UNGROUNDED today) start grounding
- Questions tangentially related to past STRICT answers gain new anchors
- Strict-rate slowly climbs as the substrate accumulates trusted self-knowledge
- New failure modes: badly-anchored citations to providence chunks (the Wall + JP-dinos pattern repeats inside the self-reference layer). The lazy-anchor smell sidecar already covers this.

## 5. Implementation summary

Files added / modified:
- `aborist/sources/providence.py` — new module, `ProvidenceSource(Source)` with kindergarten / state filters
- `aborist/qa/query.py` — `_classify_source_role` recognizes `aborist://providence/` URIs; `claim_lattice_allowed_source_roles` adds `self_reference_source`
- `aborist/qa/runner.py` — same allowlist update for the per-document `ask` path
- `Makefile` — new `ingest-self-providence` target
- `tests/test_providence_source.py` — unit tests for the source filter + classifier

CLI wire-up + bench validation deferred to follow-up commits once we've ingested-and-bench-validated on real data.
