qa: self-reference thought chains — STRICT-as-fact substrate

Closes the recursive-grounding gap fox surfaced today: aborist
tends Wikipedia trees but never grafts its own past Q&A records
into the forest. Each query starts from cold corpus retrieval;
prior providence_cache records sit unread until the same question
is re-asked (cache_key match). The "kindergarten thought chains"
framing names the gap — the system has a kindergarten of thoughts
(early STRICT records) that should mature into citable substrate
as they cool.

This commit lands the MVP — STRICT live providence records past
the kindergarten window become substrate via a new Source
subclass. Trust model per fox: "we trust strict statements as
fact unless a verifier falsifies it."

NEW
---
- docs/self-reference-thought-chains-design.md — full architecture
  doc covering the four iteration-time gates, the recursive Merkle
  proof story (Q2 → Q1 → Wikipedia chunk per v9.8's recursive-
  cores insight), the falsification trust model (state=live as the
  fail-closed gate), risks (lazy-anchor compounding, echo chambers,
  storage bloat), and bench-impact estimate.
- aborist/sources/providence.py — ProvidenceSource(Source) with
  four hard gates:
    1. audit_mode == 'STRICT' (HYBRID/UNGROUNDED stay opaque)
    2. falsification_state == 'live' (failed/stale/quarantined
       excluded — verifier-falsification mechanism per fox)
    3. now - created_at >= kindergarten_seconds (default 1h —
       fresh thoughts cool first; kills tight echo loops)
    4. anti-recursion: records whose answer text contains a
       self-reference URI are skipped — first-generation only
- tests/test_providence_source.py — 10 unit tests covering each
  gate plus the URI-scheme source-role classifier
- Makefile target `ingest-self-providence` (KG_SECONDS=3600
  default; iterates each shard and self-promotes its STRICT live
  records — cross-shard sharing happens via the existing
  shards-dir UNION at retrieval time)

WIRE-UP
-------
- aborist/qa/query.py
  - SOURCE_ROLE_BUDGET_WEIGHTS: self_reference_source = 1.0
    (same as background — Wikipedia stays canonical primary;
    self-reference is supplementary anchoring)
  - SOURCE_ROLE_RANK_WEIGHTS: self_reference_source = 0.9
  - _classify_source_role: short-circuits on aborist://providence/
    URI prefix → self_reference_source regardless of title shape
  - DEFAULT_QUERY_POLICY['claim_lattice_allowed_source_roles']
    += 'self_reference_source'
- aborist/qa/runner.py — same allowlist update for the
  per-document `ask` path
- aborist/cli.py — `aborist ingest --source providence` reads the
  providence_cache from the same shard it writes into;
  --kindergarten-seconds flag plumbed through

NOT IN THIS COMMIT
------------------
- Aggregation of multiple Q&A records into synthesized summary
  records (follow-on)
- Self-reference for HYBRID records (only STRICT is substrate
  today; HYBRID could land later as a soft-anchor role with
  lower trust)
- Live virtual sourcing (the design discusses it; MVP uses
  snapshot ingestion so existing FTS / chunker / Merkle apply
  with zero schema change)
- A live bench validating actual lift on self-reference questions
  (requires running ingest-self-providence then bench; deferred
  to follow-on commit on real data)

10 new unit tests pass; full suite at 482 passed / 21 skipped
(live fixtures gated).

2026-05-01 10:16:47 -04:00

8.3 KiB

Raw Blame History

Self-reference thought chains — recursive grounding on the providence ledger

Date: 2026-05-01 Status: design + MVP implementation. New module aborist/sources/providence.py, new Makefile target ingest-self-providence, allowlist update for claim_lattice_allowed_source_roles. Unit-tested. Live integration deferred (requires running ingest then bench). Audience: fox + future blackops shifts. Hard constraint: STRICT records are trusted as fact unless a verifier falsifies them. Other audit_modes stay opaque to retrieval until promoted.

1. Problem statement

Aborist's namesake is "tends trees and forests of cross-linked information." Today the system tends Wikipedia trees but never grafts its own past Q&A records into the forest. Every query starts from cold corpus retrieval; prior providence records sit in providence_cache unread. The system answers a question, stores the answer in the cache, and never looks at that answer again unless someone re-asks the exact same question (cache_key match).

The "kindergarten thought chains" framing names the gap: the system has a kindergarten of thoughts (early STRICT records) that should mature into citable substrate as they cool, then serve as anchors for new thoughts. Without that loop the substrate is a one-shot answerer, not a recursively-deepening reasoner.

Concretely: ask "what is verify_claim_lattice?" today and Hermes guesses from training. The right primary source — the verify.py source code or the design docs in docs/ — isn't in the corpus. Even if you make ingest-self, only the code gets ingested; the system's answers about its own code stay invisible to retrieval.

2. Design

2.1 Architecture

A new Source subclass — ProvidenceSource — iterates providence_cache records and yields each as a Document:

URI: aborist://providence/<cache_key> — content-addressed, stable, distinguishable from Wikipedia/external URIs at retrieval time.
Title: the question text (truncated to ~120 chars).
Content: a canonical layout of Q: <question> then A: <answer_text> then per-claim line [E#: cite] if available. The content gets chunked + Merkle-rooted via the standard ingest pipeline.
source_type: "providence".

Records are filtered at iteration time by:

audit_mode == "STRICT" — only fully-grounded records become substrate. HYBRID and UNGROUNDED stay opaque to retrieval (they're noisy or speculative).
falsification_state == "live" — falsified records (state ∈ {failed, stale, quarantined}) are excluded. The existing falsification machinery is the verifier-falsification mechanism fox asked for: when aborist providence --falsify flips a record's state, it stops being substrate on next ingest.
now - created_at >= kindergarten_seconds (default 3600s = 1h). Fresh thoughts cool before they're recyclable. Mirrors the mesh-sync kindergarten window. Without this, the system would self-cite a record from 30 seconds ago and you get echo-chamber loops.
Anti-recursion: records whose own answer cited a self_reference_source are excluded — first-generation only. Otherwise a wrong-but-STRICT record gets repeatedly recompiled into deeper claims and the chain rots silently. (This is conservative; second-gen self-citation may land later if the lazy-anchor / NLI-sidecar work proves the error rate is low enough.)

2.2 Retrieval integration

The source-role classifier in aborist/qa/query.py:_classify_source_role recognizes the aborist://providence/ URI scheme and tags those documents self_reference_source. This role gets added to the default claim_lattice_allowed_source_roles allowlist so claims can verify against self-reference spans.

SOURCE_ROLE_BUDGET_WEIGHTS for self_reference_source = 1.0 (same as background; deliberately not boosted — Wikipedia is still the canonical primary).

2.3 Recursive Merkle proof

When Q2 cites Q1's answer span, the audit chain becomes "Q2 → Q1 → Wikipedia chunk." The original chunk_root remains the leaf; Q1's run_dag_root becomes an intermediate node. v9.8's admissibility ledger already supports this layering — that's the recursive-cores insight ("planet toward center compression" in CLAUDE.md). No schema change needed; the providence record's merkle_proof blob carries the parent chain.

2.4 Falsification trust model

Per fox: "we trust strict statements as fact unless a verifier falsifies it."

STRICT live records → trusted, ingested as substrate
A falsifier (manual via aborist providence --falsify, drift-detection in evict_to_cold rehydrate, mesh-sync mismatch) flips state to failed / stale / quarantined
On next ingest-self-providence run, the falsified record's document gets removed from FTS (idempotent re-ingest with supersedes edge — the documents table already supports this)
Until that re-ingest, the stale record stays in retrieval but the verifier on a Q2 citing it sees falsification_state != live for the underlying source_root and rejects the citation. Fail-closed: a falsified record CANNOT serve as STRICT substrate even if it's still in the documents table.

2.5 Operational model

make ingest-self-providence — promote STRICT-live providence records older than the kindergarten window into the document corpus
Run on a cron (every hour, mirroring the mesh kindergarten window)
Idempotent: same record → same document_root → no-op insert. Replaced records get a supersedes edge linking new → old.

2.6 What's NOT in this design

Aggregation of multiple Q&A records into a synthesized "summary" record. Possible follow-on but not MVP.
Cross-shard self-reference: each shard can self-promote within itself; cross-shard cites work via the existing shards-dir UNION. No new code.
Live retrieval from providence_cache (Option B from the design discussion). The MVP uses snapshot ingestion (Option A) for simplicity. Live virtual sourcing would be an optimization if duplication becomes painful.
Self-reference for HYBRID records. STRICT only. HYBRID could be added as a soft-anchor source role with lower trust (own role: self_reference_hybrid_source, role weight 0.5). Holding for fox's call.

3. Risks & mitigations

risk	mitigation
Lazy-anchor false-STRICT compounds — Q2 inherits Q1's bogus cite.	`lazy_anchor_demoted` records skipped on promotion. Once the NLI sidecar lands, `semantic_demoted` adds another gate.
Echo-chamber: same fact recycled across many records.	Kindergarten window (1h) + first-gen-only anti-recursion check.
Falsified record stays in retrieval until re-ingest.	Verifier checks `falsification_state` of the cited source_root at verify time; non-live cites get rejected. Fail-closed.
Storage bloat: every Q&A becomes a document.	Same chunker + Merkle as everything else; per-record cost is small. Periodic burn (`burn-kindergarten`) trims old or low-quality rows.
Schema/policy drift: a record promoted under v9.8.0 becomes invalid when chunking_version bumps.	Same schema-invariant rule as Wikipedia ingestion; `governance_policy_hash` migration applies.

4. Bench impact estimate

Before any self-reference is in the corpus, no impact. Once make ingest-self-providence has been run for a while and a few hundred STRICT records are substrate:

Questions about aborist itself (mostly UNGROUNDED today) start grounding
Questions tangentially related to past STRICT answers gain new anchors
Strict-rate slowly climbs as the substrate accumulates trusted self-knowledge
New failure modes: badly-anchored citations to providence chunks (the Wall + JP-dinos pattern repeats inside the self-reference layer). The lazy-anchor smell sidecar already covers this.

5. Implementation summary

Files added / modified:

aborist/sources/providence.py — new module, ProvidenceSource(Source) with kindergarten / state filters
aborist/qa/query.py — _classify_source_role recognizes aborist://providence/ URIs; claim_lattice_allowed_source_roles adds self_reference_source
aborist/qa/runner.py — same allowlist update for the per-document ask path
Makefile — new ingest-self-providence target
tests/test_providence_source.py — unit tests for the source filter + classifier

CLI wire-up + bench validation deferred to follow-up commits once we've ingested-and-bench-validated on real data.

8.3 KiB Raw Blame History