From 39c3652e0aca0dce3881aaeffffa835e73a34a6b Mon Sep 17 00:00:00 2001 From: "russell@unturf.com" Date: Sat, 2 May 2026 12:39:42 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20consolidate=20self-reference=20design?= =?UTF-8?q?=20=E2=80=94=20flat=20MVP=20+=20fact-Core=20distillation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two self-reference design docs were a sequential pair: self-reference-thought-chains-design.md (96 lines, v1 MVP that shipped via 8de0044) self-reference-distillation-design.md (208 lines, v2 fact-Core successor, design-only) v2 explicitly named v1 as 'flat MVP' that 'falls short' and itself as 'successor' — they're one story told in two files. Merge into docs/self-reference-design.md (230 lines) Structure: §1 problem statement, §2 v1 flat MVP (shipped), §3 v2 fact-Core distillation (proposal), §4 v2 implementation plan, §5 deliberate non-goals, §6 bench impact, §7 architectural payoff. One narrative; the v1/v2 demarcation stays clear. Net: 304 → 230 lines (-74), one file instead of two. References updated in: CLAUDE.md, aborist/cli.py, aborist/qa/query.py, aborist/qa/runner.py, aborist/sources/providence.py, docs/TICKETS.md, docs/cti-architecture.md (3 refs), tests/test_providence_source.py 10/10 test_providence_source tests still pass. --- CLAUDE.md | 4 +- aborist/cli.py | 2 +- aborist/qa/query.py | 2 +- aborist/qa/runner.py | 2 +- aborist/sources/providence.py | 2 +- docs/TICKETS.md | 2 +- docs/cti-architecture.md | 6 +- docs/self-reference-design.md | 230 +++++++++++++++++++ docs/self-reference-distillation-design.md | 208 ----------------- docs/self-reference-thought-chains-design.md | 96 -------- tests/test_providence_source.py | 2 +- 11 files changed, 241 insertions(+), 315 deletions(-) create mode 100644 docs/self-reference-design.md delete mode 100644 docs/self-reference-distillation-design.md delete mode 100644 docs/self-reference-thought-chains-design.md diff --git a/CLAUDE.md b/CLAUDE.md index 7e695a3..613159e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -334,8 +334,8 @@ Architecture / ongoing work: - `docs/bench-maxing.md` — bench discipline. - `docs/test-coverage-audit-2026-05-01.md` — test-suite coverage audit. - `docs/verifier-semantic-gap-design.md` — soft-signal NLI proposal. -- `docs/self-reference-thought-chains-design.md`, - `docs/self-reference-distillation-design.md` — recursive distillation. +- `docs/self-reference-design.md`, + `docs/self-reference-design.md` — recursive distillation. - `docs/naming-deferral.md` — naming convention notes. Tickets: `docs/TICKETS.md` is the authoritative index with `Next diff --git a/aborist/cli.py b/aborist/cli.py index 2d6b142..bd460bb 100644 --- a/aborist/cli.py +++ b/aborist/cli.py @@ -93,7 +93,7 @@ def _cmd_ingest(args: argparse.Namespace) -> int: elif args.source == "providence": # Self-reference: promote STRICT live providence_cache records # past the kindergarten window into the document corpus. - # See docs/self-reference-thought-chains-design.md. + # See docs/self-reference-design.md. from aborist.sources.providence import ( DEFAULT_KINDERGARTEN_SECONDS, ProvidenceSource, diff --git a/aborist/qa/query.py b/aborist/qa/query.py index 98370d4..c87b3c3 100644 --- a/aborist/qa/query.py +++ b/aborist/qa/query.py @@ -354,7 +354,7 @@ DEFAULT_QUERY_POLICY = { "background_source", "unclassified", # Self-reference: STRICT-trusted-as-fact unless falsified. - # See docs/self-reference-thought-chains-design.md. + # See docs/self-reference-design.md. "self_reference_source", ], "claim_lattice_max_pointers_per_claim": 2, diff --git a/aborist/qa/runner.py b/aborist/qa/runner.py index 67d8c83..a33c290 100644 --- a/aborist/qa/runner.py +++ b/aborist/qa/runner.py @@ -134,7 +134,7 @@ DEFAULT_POLICY = { # URI scheme). Trusted-as-fact substrate per the # self-reference design — STRICT live records past the # kindergarten window. See - # docs/self-reference-thought-chains-design.md for the + # docs/self-reference-design.md for the # falsification trust model. "self_reference_source", ], diff --git a/aborist/sources/providence.py b/aborist/sources/providence.py index 69d6231..9c1e493 100644 --- a/aborist/sources/providence.py +++ b/aborist/sources/providence.py @@ -5,7 +5,7 @@ that landed STRICT and are still live) into the document corpus so retrieval can surface them as `self_reference_source` citations for new thoughts. -See ``docs/self-reference-thought-chains-design.md`` for the full +See ``docs/self-reference-design.md`` for the full architecture. Trust model summary: STRICT live records older than the kindergarten window are emitted as documents. HYBRID / UNGROUNDED records and falsified records (state ∈ {failed, stale, quarantined}) diff --git a/docs/TICKETS.md b/docs/TICKETS.md index b9c040a..33999b1 100644 --- a/docs/TICKETS.md +++ b/docs/TICKETS.md @@ -42,7 +42,7 @@ proposing change. Examples that stay un-numbered: - `naming-deferral.md` (naming convention reference) - `qa-modes-bench-2026-04-30.md` (point-in-time bench journal) - `verifier-semantic-gap-design.md`, - `self-reference-thought-chains-design.md` (older design docs that + `self-reference-design.md` (older design docs that pre-date the ticket convention; leave as-is unless retroactively promoting one to a ticket adds value) - `concept-relations-design.md` (architecture reference for the diff --git a/docs/cti-architecture.md b/docs/cti-architecture.md index bf3f8ef..83e9730 100644 --- a/docs/cti-architecture.md +++ b/docs/cti-architecture.md @@ -183,8 +183,8 @@ The audit flow is what makes Merkle-AGI O(log N + k) per challenged claim — yo | this layer-cake | `docs/cti-architecture.md` (you are here) | | QA mode bench journey | `docs/qa-modes-bench-2026-04-30.md` | | NLI semantic-gap sidecar | `docs/verifier-semantic-gap-design.md` | -| Self-reference (flat MVP) | `docs/self-reference-thought-chains-design.md` | -| Self-reference (deep, distillation-based) | `docs/self-reference-distillation-design.md` | +| Self-reference (flat MVP) | `docs/self-reference-design.md` | +| Self-reference (deep, distillation-based) | `docs/self-reference-design.md` | | Why we don't rename code to CTI/PROMETHEUS-Σ | `docs/naming-deferral.md` | | Bench-maxing discipline | CLAUDE.md "Bench-maxing" section | | Schema invariants | CLAUDE.md "Schema invariants (do not break)" | @@ -207,7 +207,7 @@ These are the layer-cake-shaped work items still on the roadmap: 1. **Verifier-policy-hash separation** — landing in this commit pass. Fold `answer_mode + parser_version + evidence_schema_version + manual_quote_policy + hard_checks_list` into a 9th dimension that's separate from `governance_policy_hash`. Prevents cross-mode cache aliasing more cleanly. 2. **Soft-signal taxonomy expansion** — the NLI sidecar (`docs/verifier-semantic-gap-design.md`) is one. Predicate-compatibility, completeness, counterevidence, source-authority, scope-ambiguity are others. All demote-only, all out of the proof path. Implementations come one at a time as the bench surfaces motivating cases. -3. **Self-reference distillation** — STRICT claims become Cores via a new `ProvidenceDistiller` (see `docs/self-reference-distillation-design.md`). Lets new claims compose from existing facts, not just retrieve them. +3. **Self-reference distillation** — STRICT claims become Cores via a new `ProvidenceDistiller` (see `docs/self-reference-design.md`). Lets new claims compose from existing facts, not just retrieve them. 4. **PROMETHEUS-Σ explicit naming** — deferred, see `docs/naming-deferral.md`. The control logic exists; an extracted `prometheus.py` module is a refactor with no behavior change. The architecture is real. The labels above name what's already there. diff --git a/docs/self-reference-design.md b/docs/self-reference-design.md new file mode 100644 index 0000000..8f7f74b --- /dev/null +++ b/docs/self-reference-design.md @@ -0,0 +1,230 @@ +# Self-reference design — recursive grounding on the providence ledger + +**Date opened:** 2026-05-01 +**Status:** v1 (flat MVP) shipped — `aborist/sources/providence.py`, `make ingest-self-providence`, allowlist update for `claim_lattice_allowed_source_roles`. v2 (fact-Core distillation) — design proposal, implementation scoped to a follow-on commit pass. +**Audience:** fox + future blackops shifts. +**Hard constraint:** STRICT records are trusted as fact unless a verifier falsifies them. Other audit_modes stay opaque to retrieval until promoted. v2 extends the audit chain recursively without schema change. + +--- + +## 1. Problem statement + +Aborist's namesake is "tends trees and forests of cross-linked information." Today the system tends Wikipedia trees but never grafts its own past Q&A records into the forest. Every query starts from cold corpus retrieval; prior providence records sit in `providence_cache` unread. The system answers a question, stores the answer, and never looks at that answer again unless someone re-asks the exact same question (cache_key match). + +The "kindergarten thought chains" framing names the gap: the system has a kindergarten of thoughts (early STRICT records) that should mature into citable substrate as they cool, then serve as anchors for new thoughts. Without that loop the substrate is a one-shot answerer, not a recursively-deepening reasoner. + +Concretely: ask "what is verify_claim_lattice?" today and Hermes guesses from training. The right primary source — the verify.py source code or the design docs in `docs/` — isn't in the corpus. Even after `make ingest-self`, only the *code* gets ingested; the system's *answers about its own code* stay invisible to retrieval. + +Two design layers solve this. **v1 flat MVP** lands the surface step (records become flat documents). **v2 fact-Core distillation** lands the deep step (records become Merkle-bound facts that compose). + +--- + +## 2. v1 — flat MVP (shipped) + +### 2.1 Architecture + +A new `Source` subclass — `ProvidenceSource` — iterates `providence_cache` records and yields each as a `Document`: + +- **URI**: `aborist://providence/` — content-addressed, stable, distinguishable from Wikipedia / external URIs at retrieval time. +- **Title**: the question text (truncated to ~120 chars). +- **Content**: a canonical layout of `Q: ` then `A: ` then per-claim line `[E#: cite]` if available. The content gets chunked + Merkle-rooted via the standard ingest pipeline. +- **source_type**: `"providence"`. + +Records are filtered at iteration time by: + +1. **`audit_mode == "STRICT"`** — only fully-grounded records become substrate. HYBRID and UNGROUNDED stay opaque to retrieval (noisy or speculative). +2. **`falsification_state == "live"`** — falsified records (state ∈ {failed, stale, quarantined}) are excluded. The existing falsification machinery is the verifier-falsification mechanism: when `aborist providence --falsify` flips a record's state, it stops being substrate on next ingest. +3. **`now - created_at >= kindergarten_seconds`** (default 3600s = 1h). Fresh thoughts cool before they're recyclable. Mirrors the mesh-sync kindergarten window. Without this, the system would self-cite a record from 30 seconds ago — echo-chamber loops. +4. **Anti-recursion**: records whose own answer cited a `self_reference_source` are excluded — first-generation only. A wrong-but-STRICT record otherwise repeatedly recompiles into deeper claims and the chain rots silently. + +### 2.2 Retrieval integration + +The source-role classifier in `aborist/qa/query.py:_classify_source_role` recognizes the `aborist://providence/` URI scheme and tags those documents `self_reference_source`. This role gets added to the default `claim_lattice_allowed_source_roles` allowlist so claims can verify against self-reference spans. `SOURCE_ROLE_BUDGET_WEIGHTS` for `self_reference_source` = 1.0 (same as background; deliberately not boosted — Wikipedia is still the canonical primary). + +### 2.3 Recursive Merkle proof + +When Q2 cites Q1's answer span, the audit chain becomes "Q2 → Q1 → Wikipedia chunk." The original `chunk_root` remains the leaf; Q1's `run_dag_root` becomes an intermediate node. v9.8's admissibility ledger already supports this layering — the recursive-cores insight ("planet toward center compression" in CLAUDE.md). No schema change needed; the providence record's `merkle_proof` blob carries the parent chain. + +### 2.4 Operational model + +- `make ingest-self-providence` — promote STRICT-live providence records older than the kindergarten window into the document corpus. +- Run on a cron (every hour, mirroring the mesh kindergarten window). +- Idempotent: same record → same document_root → no-op insert. Replaced records get a `supersedes` edge linking new → old. + +### 2.5 What v1 does NOT do + +- **Aggregation** of multiple Q&A records into a synthesized "summary" record. +- **Cross-shard self-reference**: each shard self-promotes within itself; cross-shard cites work via the existing `--shards-dir` UNION. No new code. +- **Live retrieval from `providence_cache`** (Option B from the design discussion). The MVP uses snapshot ingestion (Option A) for simplicity. +- **HYBRID-record self-promotion**. STRICT only. +- **Compositional reasoning** — records become reachable, but they don't yet COMPOSE into new claims. v2 addresses the composition gap. + +### 2.6 v1 risks (still open) + +| risk | mitigation | +|------|-----------| +| Lazy-anchor false-STRICT compounds — Q2 inherits Q1's bogus cite. | `lazy_anchor_demoted` records skipped on promotion. NLI sidecar (`docs/verifier-semantic-gap-design.md`) adds another gate when it lands. | +| Echo-chamber: same fact recycled across many records. | Kindergarten window (1h) + first-gen-only anti-recursion check. | +| Falsified record stays in retrieval until re-ingest. | Verifier checks `falsification_state` of the cited source_root at verify time; non-live cites get rejected. Fail-closed. | +| Storage bloat: every Q&A becomes a document. | Same chunker + Merkle as everything else; per-record cost is small. Periodic `burn-kindergarten` trims. | + +--- + +## 3. v2 — fact-Core distillation (proposal) + +### 3.1 Where v1 falls short + +v1 makes records "another flat document source." The architecture wants them to be **Merkle-bound facts that compose into new claims**. v1 retrieves a record's text; it doesn't let the record's *evidence chain* attach as substrate for new reasoning. New claims about the same topic don't compose with old claims; they just see them as more context. + +The deeper play: STRICT claims become **Cores** via the existing distillation pipeline. New claims derive from cores via the recursive-cores layer. The fact-graph grows. + +### 3.2 The existing infrastructure already does most of this + +The Surface → Core → Recursive-Core layering is in `aborist/distill/`: + +``` +aborist/distill/ +├── base.py # Distiller ABC + DistillationResult +├── first_sentence.py # FirstSentenceDistiller (no-ML stub) +├── tfidf.py # TfidfKeywordDistiller (pure-Python TF-IDF) +└── runner.py # batched: derive + per-contrib-chunk proofs +``` + +Contract: + +```python +class Distiller(ABC): + name: str + def distill(self, source: Document, source_chunks: list[str]) -> DistillationResult: + ... +``` + +The runner takes a surface Document, runs the Distiller, returns a Core Document plus `contributing_chunk_indices`. Per-chunk Merkle inclusion proofs against `document_root` get stored in `derivations.proof_blob` — Cores are **cryptographically bound** to their source surfaces. + +What this gives us for free: +- A Core is itself a Document with its own `document_root`, chunkable + retrievable like Wikipedia content. +- The recursive-core mode (cores derive from cores) already exists in the runner. +- Every Core carries explicit lineage back to its source via per-chunk proofs. + +The missing piece: a `ProvidenceDistiller` that takes a STRICT providence record and produces a Core, where the "source chunks" are the cited evidence spans the record verified against. + +### 3.3 ProvidenceDistiller — cited-evidence-bound (Option B) + +The Distiller contract today maps `(surface Document, surface chunks) → core Document`. For self-reference we want `(STRICT record, cited evidence spans from OTHER documents) → fact-Core`. The "source chunks" the Core derives from aren't the providence record's own chunks — they're the EVIDENCE SPANS the record cited in its `claim_statuses`. + +Two options were considered: + +- **Option A — distill from the providence record's own content.** ProvidenceDistiller treats the record's `Q: ... A: ...` text as the surface, distills it to a Core. Simple, fits the Distiller contract directly. +- **Option B — distill from the cited evidence spans, with the record as an indirection.** ProvidenceDistiller looks up the record's `claim_statuses[].evidence_ids`, fetches the cited evidence chunks from THEIR source documents, treats those as the "source chunks," and emits a Core that's bound by inclusion proof to the cited chunks of the cited Wikipedia documents. + +**Option B is the right deep version.** A fact-Core derived from a STRICT claim is a Merkle-bound assertion that "claim text C is supported by chunk_root Cr1 in document_root Dr1." Future claims attaching to this Core inherit that evidence chain transparently. Option A would make Cores derive from the record's text (which already says what the answer is), losing the direct connection to the underlying Wikipedia facts. + +Option B's cross-document fetch is supported today: `--shards-dir` UNION views let a single `connect()` see all shards as one read connection. + +### 3.4 Core content shape + +Three candidate shapes for the fact-Core's content: + +```text +SHAPE A — claim text only + "Joey Potter is the girl across the creek in Dawson's Creek." + +SHAPE B — claim + per-source pointer + "Joey Potter is the girl across the creek in Dawson's Creek." + [Dawson Leery (Wikipedia): "...the central fictional character..."] + +SHAPE C — structured triple + SUBJECT: Joey Potter + PREDICATE: is the girl across the creek + OBJECT: in Dawson's Creek + EVIDENCE: chunk_root=ab12... offset_start=4032 offset_end=4189 +``` + +**Recommendation: Shape B for MVP.** Pure prose with a tagged citation. Retrieval finds the prose; the cited span is right there. Shape C lands later if a fact-graph traversal becomes a real need (NER + relation extraction — out of scope today). + +### 3.5 Recursive cores — facts grow new ideas + +```text +Surface (Wikipedia chunk) + ↓ TfidfKeywordDistiller +Core-tfidf (keywords from Wikipedia chunk) + +STRICT providence record + ↓ ProvidenceDistiller (Option B — bound to cited Wikipedia chunks) +Fact-Core + +[Future] N related Fact-Cores + ↓ ?CompositionDistiller (deferred) +Composite-Fact-Core (claims combining multiple facts) +``` + +The v2 MVP does only the first ProvidenceDistiller pass. **CompositionDistiller is the future shape that makes facts compose into new ideas — that's where "the substrate forms new claims from its own facts" lives.** CompositionDistiller is hard because deciding which facts to compose, and how, is the actual reasoning step. Today's Hermes doesn't do that reliably. Defer. + +What v2 ships: each STRICT claim becomes a Merkle-bound Fact-Core whose proof chain reaches all the way back to a Wikipedia chunk_root. Retrieval over Cores returns Fact-Cores alongside Wikipedia surfaces — the lattice grows. Composition is deferred but the substrate is in shape for it when we land it. + +### 3.6 Recursive Merkle proof (the part already free) + +When a future Q3 cites a Fact-Core that derived from a STRICT Q1 record citing Wikipedia chunk Cr1: + +``` +Q3 claim → Fact-Core → Cr1 → Dr1 → Sr1 +``` + +Each link is a Merkle inclusion proof or a content-addressed lookup. No new schema is needed — `derivations.proof_blob` already holds the per-chunk inclusion proofs; the per-claim → Fact-Core → derivation walk just composes existing primitives. v9.8's audit chain extends naturally; we don't need v9.9. + +### 3.7 Trust + falsification (sharper than v1) + +- A Fact-Core is created only from a STRICT-live providence record past the kindergarten window. +- If the record is later falsified (`falsification_state != live`), the Fact-Core's `derivations` row is marked stale on next promotion run. Idempotent: same record → same Core hash. Falsified records don't promote. +- A future Q3 citing a stale Fact-Core fails verification at the source-state check (verifier checks `falsification_state` of the cited source's underlying records, not just the surface document). +- **Fail-closed: a falsified Fact-Core CANNOT serve as substrate even if it's still in the documents table.** + +The key trust-model add over v1: **falsification cascades**. Falsifying Q1 stales Q1's Fact-Core, which stales Q2 if Q2 had cited the Fact-Core. The Merkle chain makes the cascade traceable. + +### 3.8 Anti-recursion (kept from v1) + +A providence record whose own answer text already cites a Fact-Core — i.e. a record answered by composing existing facts — gets ONE level of self-reference but cannot itself be promoted to a NEW Fact-Core. First-generation only. This kills echo-chamber chains where a wrong-but-STRICT record keeps recompiling itself into deeper claims. Conservative; the right relaxation is "promote when the lazy-anchor sidecar AND the NLI sidecar both pass" — but that's after both signals are in place. + +--- + +## 4. v2 implementation plan (8 steps) + +1. **`aborist/distill/providence.py`** — new module, `ProvidenceDistiller(Distiller)`. Reads STRICT live providence records past kindergarten, fetches the cited evidence chunks (Option B), builds Shape-B Core content (claim text + tagged citation span), returns `DistillationResult` with `contributing_chunk_indices` pointing to the cited Wikipedia chunks. +2. **Wire into `aborist/distill/runner.py`** — register ProvidenceDistiller as a known kind. The existing batched-distill flow handles the per-chunk-proof generation transparently. +3. **CLI: `aborist distill --kind providence`** — adds the new kind to the `distill` subcommand's choices. Plumbs the `--kindergarten-seconds` knob from the v1 CLI work. +4. **Makefile: `distill-self-providence`** — runs `aborist distill --kind providence` against each shard. Hourly cron candidate. +5. **Source-role classifier** — Fact-Cores are tagged `self_reference_source` via the existing URI-prefix path (`aborist://providence/...` from v1 carries through). Cores derived from those records inherit the role. No classifier change needed. +6. **Falsification cascade** — when `aborist providence --falsify` flips a record's state, also mark the corresponding Fact-Core's derivation row as stale. New CLI flag or implicit on next ingest pass; tradeoff: explicit is debuggable, implicit is less coordinated. +7. **Tests** — unit tests for ProvidenceDistiller (correctly fetches cited chunks, builds Shape-B content, generates valid inclusion proofs), falsification cascade (falsified record → stale Core → rejected citation in new run). +8. **Bench validation** — re-run `make bench-qa` after `make ingest-self-providence` AND `make distill-self-providence` have populated some Fact-Cores. Compare to baseline. Questions about aborist itself (currently UNGROUNDED) should ground; questions tangential to past STRICT answers should gain new anchors. + +--- + +## 5. What's deliberately NOT in this design + +- **CompositionDistiller** — combining multiple Fact-Cores into a new claim. Reasoning machinery, not infrastructure. Defer until the soft-signal taxonomy (NLI, predicate compatibility) is mature enough that compositions can be sanity-checked. +- **Shape-C structured triples** — needs NER + relation extraction. Land Shape B first; promote to Shape C if a fact-graph use case actually needs subject-predicate-object retrieval. +- **HYBRID record promotion** — both v1 and v2 gate on STRICT only. HYBRID could become a `self_reference_hybrid_source` role with lower trust, separately gated. +- **Cross-shard cascading falsification** — falsifying a record on one shard doesn't auto-falsify a Fact-Core derived from it on another shard. Mesh-sync handles cross-shard coherence eventually; the immediate cascade is per-shard. Acceptable. + +--- + +## 6. Bench impact (speculative, disciplined) + +After a few hundred STRICT records have promoted to Fact-Cores: + +- Questions about aborist itself (today's mostly UNGROUNDED) start grounding against Fact-Cores derived from past Q&A about aborist. +- Questions tangentially related to past STRICT answers gain anchors that reach back to Wikipedia transparently. +- Strict-rate creeps up as the fact-substrate matures; honest-grounded count rises faster. +- New failure modes: bad anchors landing inside Fact-Cores. The lazy-anchor sidecar already covers this layer-recursively because Fact-Cores look just like other documents to the verifier. +- Latency: same as Wikipedia retrieval. No new path; just more documents indexed. + +--- + +## 7. The architectural payoff + +Today the substrate is a one-shot answerer: every query starts cold, retrieves Wikipedia, prompts Hermes, verifies, caches. The cache is a key-value lookup, not a substrate for reasoning. + +After v2 lands, the substrate becomes recursively-deepening: every STRICT answer becomes a Merkle-bound fact in the tree. New questions retrieve old facts as substrate. The fact-graph compounds. Wrong facts get falsified and the cascade reaches the dependent records. Right facts stay grounded and become the foundation for deeper claims. + +That's "tends trees and forests of cross-linked information" — literally. The naming wasn't aspirational; it was load-bearing for the architecture. diff --git a/docs/self-reference-distillation-design.md b/docs/self-reference-distillation-design.md deleted file mode 100644 index 9db6664..0000000 --- a/docs/self-reference-distillation-design.md +++ /dev/null @@ -1,208 +0,0 @@ -# Self-reference distillation — STRICT facts as Merkle-bound substrate - -**Date:** 2026-05-01 -**Status:** design proposal. Successor to `docs/self-reference-thought-chains-design.md` (the flat-source MVP). Implementation scoped to a follow-on commit pass. -**Audience:** fox + future blackops shifts. -**Hard constraint:** STRICT records become Cores via the existing Distiller / Core / derivations infrastructure. Soft signals never enter the proof path. The audit chain extends recursively without schema change. - ---- - -## 1. Where the flat MVP falls short - -`docs/self-reference-thought-chains-design.md` (commit `8de0044`, 2026-05-01) lands a `ProvidenceSource(Source)` that promotes STRICT live providence_cache records into the document corpus. New retrieval surfaces them as `self_reference_source` citations. That's the surface step — necessary scaffolding so questions about aborist itself can ground at all. - -But fox's deeper framing is that the MVP makes records "another flat document source," when the architecture wants them to be **Merkle-bound facts that compose into new claims**. The flat MVP retrieves a record's text; it doesn't let the record's *evidence chain* attach as substrate for new reasoning. New claims about the same topic don't compose with old claims; they just see them as more context. - -The deeper play is: STRICT claims become **Cores** via the existing distillation pipeline. New claims derive from cores via the recursive-cores layer ("planet toward center compression" in CLAUDE.md). The fact-graph grows. - -## 2. The existing infrastructure already does most of this - -The Surface → Core → Recursive-Core layering is in `aborist/distill/`: - -``` -aborist/distill/ -├── base.py # Distiller ABC + DistillationResult -├── first_sentence.py # FirstSentenceDistiller (no-ML stub) -├── tfidf.py # TfidfKeywordDistiller (pure-Python TF-IDF) -└── runner.py # batched: derive + per-contrib-chunk proofs -``` - -The contract: - -```python -class Distiller(ABC): - name: str - def distill(self, source: Document, source_chunks: list[str]) -> DistillationResult: - ... -``` - -The runner takes a surface Document, runs the Distiller, gets back a Core Document plus `contributing_chunk_indices`. The runner generates Merkle inclusion proofs for each contributing chunk against `document_root` and stores them in `derivations.proof_blob` — so the Core is **cryptographically bound** to its source surface. - -What this gives us for free: -- The Core is itself a Document with its own `document_root`, chunkable + retrievable like Wikipedia content -- The recursive-core mode (cores derive from cores) already exists in the runner -- Every Core carries explicit lineage back to its source via per-chunk proofs - -The missing piece: a `ProvidenceDistiller` that takes a STRICT providence record and produces a Core, where the "source chunks" are the cited evidence spans the record verified against. - -## 3. Architecture - -### 3.1 ProvidenceDistiller - -A new `Distiller` subclass: - -```python -class ProvidenceDistiller(Distiller): - name = "providence-claim-v1" - def distill(self, source, source_chunks): - # source.uri = "aborist://providence/" - # source.content = "Q: ...\n\nA: ..." (from ProvidenceSource) - # source_chunks = the chunks of THIS providence record's content - ... -``` - -But the right shape is subtler. The Distiller contract today maps `(surface Document, surface chunks) → core Document`. For self-reference we want `(STRICT record, cited evidence spans from OTHER documents) → fact-Core`. The "source chunks" the Core derives from aren't the providence record's own chunks — they're the EVIDENCE SPANS the record cited in its claim_statuses. - -Two options: - -**Option A — distill from the providence record's own content.** -ProvidenceDistiller treats the record's `Q: ... A: ...` text as the surface, distills it to a Core. The contributing-chunk-indices point into the providence record's own chunking. Simple, fits the Distiller contract directly. - -**Option B — distill from the cited evidence spans, with the record as an indirection.** -ProvidenceDistiller looks up the record's `claim_statuses[].evidence_ids`, fetches the cited evidence chunks from THEIR source documents, treats those as the "source chunks," and emits a Core that's bound by inclusion proof to the cited chunks of the cited Wikipedia documents. The fact-Core carries direct provenance to the Wikipedia spans, not just to the providence record. - -**Option B is the right deep version.** A fact-Core derived from a STRICT claim is a Merkle-bound assertion that "claim text C is supported by chunk_root Cr1 in document_root Dr1." Future claims attaching to this Core inherit that evidence chain transparently. Option A would make Cores derive from the record's text (which already says what the answer is), losing the direct connection to the underlying Wikipedia facts. - -Implementation note: Option B requires the Distiller (or its runner) to fetch chunks from documents the providence record cites. The cross-shard attach machinery (`--shards-dir`) already lets a single `connect()` see all shards as UNION views, so the fetch is just a SELECT. - -### 3.2 Core content shape - -A fact-Core's content is the structured assertion. Three candidate shapes: - -```text -SHAPE A — claim text only (haiku-like) - "Joey Potter is the girl across the creek in Dawson's Creek." - -SHAPE B — claim + per-source pointer - "Joey Potter is the girl across the creek in Dawson's Creek." - [Dawson Leery (Wikipedia): "...the central fictional character..."] - -SHAPE C — structured triple form - SUBJECT: Joey Potter - PREDICATE: is the girl across the creek - OBJECT: in Dawson's Creek - EVIDENCE: chunk_root=ab12... offset_start=4032 offset_end=4189 -``` - -Shape A is the simplest — pure text, chunkable, retrievable as prose. Shape B carries the cited span inline so retrieval surfaces it without extra DB lookups. Shape C is most useful for fact-graph composition (other claims can find the SUBJECT in their queries) but requires NER + relation extraction we don't have. - -Recommendation: **Shape B for MVP**. Pure prose with a tagged citation. Retrieval finds the prose; the cited span is right there. Shape C lands later if a fact-graph traversal becomes a real need. - -### 3.3 Recursive cores — facts grow new ideas - -The runner's recursive-core mode lets a Core be the input to another Distiller pass. So: - -```text -Surface (Wikipedia chunk) - ↓ TfidfKeywordDistiller -Core-tfidf (keywords from Wikipedia chunk) - -STRICT providence record - ↓ ProvidenceDistiller (with Option B — bound to cited Wikipedia chunks) -Fact-Core - -[Later] N related Fact-Cores - ↓ ?CompositionDistiller (future, out of MVP scope) -Composite-Fact-Core (claims that combine multiple facts) -``` - -The MVP does only the first ProvidenceDistiller pass. CompositionDistiller is the future shape that makes facts compose into new ideas — that's where "the substrate forms new claims from its own facts" lives. CompositionDistiller is hard because deciding which facts to compose, and how, is the actual reasoning step. Today's Hermes doesn't do that reliably. That's why we're not building CompositionDistiller in this MVP. - -What we ARE building: each STRICT claim becomes a Merkle-bound Fact-Core whose proof chain reaches all the way back to a Wikipedia chunk_root. Retrieval over Cores returns Fact-Cores alongside Wikipedia surface — the lattice grows. Composition is deferred but the substrate is in shape for it when we land it. - -### 3.4 Recursive Merkle proof (the part that's already free) - -When a future Q3 cites a Fact-Core that itself derived from a STRICT Q1 record that cited Wikipedia chunk Cr1: - -``` -Q3 claim → cites evidence_id E_x in Q3's run-DAG -E_x → Fact-Core's evidence_id (content-addressed) -Fact-Core's derivation row → has proof_blob containing inclusion proof - of chunk Cr1 against document_root Dr1 -Wikipedia document Dr1 → has source_root Sr1 in source corpus -``` - -The chain is: - -``` -Q3 claim → Fact-Core → Cr1 → Dr1 → Sr1 -``` - -Each link is a Merkle inclusion proof or a content-addressed lookup. No new schema is needed — `derivations.proof_blob` already holds the per-chunk inclusion proofs; the per-claim → Fact-Core → derivation walk just composes existing primitives. - -Per CLAUDE.md "Merkle: non-commutative HashCombine with prefix `0x03`" — the same hash discipline applies all the way down. v9.8's audit chain extends naturally; we don't need v9.9. - -### 3.5 Trust + falsification - -Same as the flat MVP, sharper: - -- A Fact-Core is created only from a STRICT-live providence record past the kindergarten window -- If the record is later falsified (`falsification_state != live`), the Fact-Core's `derivations` row is marked stale on next promotion run. Idempotent: same record → same Core hash. Falsified records don't promote. -- A future Q3 citing a stale Fact-Core fails verification at the source-state check (verifier checks `falsification_state` of the cited source's underlying records, not just the surface document) -- **Fail-closed: a falsified Fact-Core CANNOT serve as substrate even if it's still in the documents table.** - -This is the key trust-model add: STRICT-as-fact unless falsified, AND the falsification cascades — falsifying Q1 stales Q1's Fact-Core, which stales Q2 if Q2 had cited the Fact-Core. The Merkle chain makes the cascade traceable. - -### 3.6 Anti-recursion (kept from MVP) - -A providence record whose own answer text already cites a Fact-Core — i.e. a record that was answered by composing existing facts — gets ONE level of self-reference but cannot itself be promoted to a NEW Fact-Core. First-generation only. This kills echo-chamber chains where a wrong-but-STRICT record keeps recompiling itself into deeper claims. - -This is conservative; the right relaxation is "promote when the lazy-anchor sidecar AND the NLI sidecar both pass" — but that's after both signals are in place. - -## 4. Implementation plan (high-level, 8 steps) - -1. **`aborist/distill/providence.py`** — new module, `ProvidenceDistiller(Distiller)`. Reads STRICT live providence records past kindergarten, fetches the cited evidence chunks (Option B), builds Shape-B Core content (claim text + tagged citation span), returns `DistillationResult` with contributing_chunk_indices pointing to the cited Wikipedia chunks. -2. **Wire into `aborist/distill/runner.py`** — registers ProvidenceDistiller as a known kind. The existing batched-distill flow handles the per-chunk-proof generation transparently. -3. **CLI: `aborist distill --kind providence`** — adds the new kind to the `distill` subcommand's choices. Plumbs through the `--kindergarten-seconds` knob from the flat-MVP CLI work. -4. **Makefile: `distill-self-providence`** — runs `aborist distill --kind providence` against each shard. Hourly cron candidate. -5. **Source-role classifier** — Fact-Cores are tagged `self_reference_source` via the existing URI-prefix path (`aborist://providence/...` from the flat MVP carries through). Cores derived from those records inherit the role. No classifier change needed. -6. **Falsification cascade** — when `aborist providence --falsify` flips a record's state, also mark the corresponding Fact-Core's derivation row as stale. New CLI flag or implicit on next ingest pass; tradeoff: explicit is debuggable, implicit is less coordinated. -7. **Tests** — unit tests for ProvidenceDistiller (correctly fetches cited chunks, builds Shape-B content, generates valid inclusion proofs), falsification cascade (falsified record → stale Core → rejected citation in new run). -8. **Bench validation** — re-run `make bench-qa` after `make ingest-self-providence` AND `make distill-self-providence` have populated some Fact-Cores. Compare to baseline. Questions about aborist itself (currently UNGROUNDED) should ground; questions tangential to past STRICT answers should gain new anchors. - -## 5. What's deliberately NOT in this design - -- **CompositionDistiller** — combining multiple Fact-Cores into a new claim. Reasoning machinery, not infrastructure. Defer until the soft-signal taxonomy (NLI, predicate compatibility) is mature enough that compositions can be sanity-checked. -- **Shape-C structured triples** — needs NER + relation extraction. Land Shape B first; promote to Shape C if a fact-graph use case actually needs subject-predicate-object retrieval. -- **HYBRID record promotion** — flat MVP and this design both gate on STRICT only. HYBRID could become a `self_reference_hybrid_source` role with lower trust, separately gated. -- **Cross-shard cascading falsification** — falsifying a record on one shard doesn't auto-falsify a Fact-Core derived from it on another shard. Mesh-sync handles cross-shard coherence eventually, but the immediate cascade is per-shard. Acceptable. - -## 6. Risks + mitigations - -| risk | mitigation | -|------|-----------| -| Lazy-anchor compounding: STRICT-but-bogus record gets promoted to Fact-Core; new claims cite it; the lattice grows around a wrong fact. | Anti-recursion (first-generation only). Once `lazy_anchor_demoted` and the NLI sidecar (per `docs/verifier-semantic-gap-design.md`) ship, gate promotion on those passing too. | -| Storage bloat: every STRICT record produces a Core with its own document_root, chunks, derivation row. | Same chunker + Merkle as Wikipedia ingestion; per-record cost is small. The kindergarten window + STRICT-only filter keep volume low. Periodic `burn-kindergarten` trims. | -| Schema/policy drift: a Core promoted under v9.8.0 stops being valid when chunking_version bumps. | Same schema-invariant rule as Wikipedia; `chunking_version` change stales every Core on lookup. Re-promote with the new version on next pass. | -| Fact-Core text leaks information from the cited evidence span — the renderer's spotlight excerpt becomes load-bearing. | Shape B inlines the spotlight span, which IS the source content. Acceptable; the substrate's whole purpose is making cited content reachable. | -| Cross-shard fetching: ProvidenceDistiller's Option-B fetch needs chunks from documents that may live on a different shard than the providence record. | The existing `--shards-dir` UNION views handle this; distill runs against the unioned read connection. | - -## 7. Bench impact (speculative, disciplined) - -After a few hundred STRICT records have promoted to Fact-Cores: - -- Questions about aborist itself (today's ~UNGROUNDED) start grounding against Fact-Cores derived from past Q&A about aborist -- Questions tangentially related to past STRICT answers gain anchors that reach back to Wikipedia transparently -- Strict-rate creeps up as the fact-substrate matures; honest-grounded count rises faster -- New failure modes: bad anchors landing inside Fact-Cores. The lazy-anchor sidecar already covers this layer-recursively because Fact-Cores look just like other documents to the verifier. -- Latency: same as Wikipedia retrieval. No new path; just more documents indexed. - -The forest grows. The trees stay individually verifiable. - -## 8. The architectural payoff - -Today the substrate is a one-shot answerer: every query starts cold, retrieves Wikipedia, prompts Hermes, verifies, caches. The cache is a key-value lookup, not a substrate for reasoning. - -After this design lands, the substrate becomes recursively-deepening: every STRICT answer becomes a Merkle-bound fact in the tree. New questions retrieve old facts as substrate. The fact-graph compounds. Wrong facts get falsified and the cascade reaches the dependent records. Right facts stay grounded and become the foundation for deeper claims. - -That's "tends trees and forests of cross-linked information" — literally. The naming wasn't aspirational; it was load-bearing for the architecture. diff --git a/docs/self-reference-thought-chains-design.md b/docs/self-reference-thought-chains-design.md deleted file mode 100644 index 83537da..0000000 --- a/docs/self-reference-thought-chains-design.md +++ /dev/null @@ -1,96 +0,0 @@ -# Self-reference thought chains — recursive grounding on the providence ledger - -**Date:** 2026-05-01 -**Status:** design + MVP implementation. New module `aborist/sources/providence.py`, new Makefile target `ingest-self-providence`, allowlist update for `claim_lattice_allowed_source_roles`. Unit-tested. Live integration deferred (requires running ingest then bench). -**Audience:** fox + future blackops shifts. -**Hard constraint:** STRICT records are trusted as fact unless a verifier falsifies them. Other audit_modes stay opaque to retrieval until promoted. - ---- - -## 1. Problem statement - -Aborist's namesake is "tends trees and forests of cross-linked information." Today the system tends Wikipedia trees but never grafts its own past Q&A records into the forest. Every query starts from cold corpus retrieval; prior providence records sit in `providence_cache` unread. The system answers a question, stores the answer in the cache, and never looks at that answer again unless someone re-asks the exact same question (cache_key match). - -The "kindergarten thought chains" framing names the gap: the system has a kindergarten of thoughts (early STRICT records) that should mature into citable substrate as they cool, then serve as anchors for new thoughts. Without that loop the substrate is a one-shot answerer, not a recursively-deepening reasoner. - -Concretely: ask "what is verify_claim_lattice?" today and Hermes guesses from training. The right primary source — the verify.py source code or the design docs in `docs/` — isn't in the corpus. Even if you `make ingest-self`, only the *code* gets ingested; the system's *answers about its own code* stay invisible to retrieval. - -## 2. Design - -### 2.1 Architecture - -A new `Source` subclass — `ProvidenceSource` — iterates `providence_cache` records and yields each as a `Document`: - -- **URI**: `aborist://providence/` — content-addressed, stable, distinguishable from Wikipedia/external URIs at retrieval time. -- **Title**: the question text (truncated to ~120 chars). -- **Content**: a canonical layout of `Q: ` then `A: ` then per-claim line `[E#: cite]` if available. The content gets chunked + Merkle-rooted via the standard ingest pipeline. -- **source_type**: `"providence"`. - -Records are filtered at iteration time by: - -1. **`audit_mode == "STRICT"`** — only fully-grounded records become substrate. HYBRID and UNGROUNDED stay opaque to retrieval (they're noisy or speculative). -2. **`falsification_state == "live"`** — falsified records (state ∈ {failed, stale, quarantined}) are excluded. The existing falsification machinery is the verifier-falsification mechanism fox asked for: when `aborist providence --falsify` flips a record's state, it stops being substrate on next ingest. -3. **`now - created_at >= kindergarten_seconds`** (default 3600s = 1h). Fresh thoughts cool before they're recyclable. Mirrors the mesh-sync kindergarten window. Without this, the system would self-cite a record from 30 seconds ago and you get echo-chamber loops. -4. **Anti-recursion**: records whose own answer cited a `self_reference_source` are excluded — first-generation only. Otherwise a wrong-but-STRICT record gets repeatedly recompiled into deeper claims and the chain rots silently. (This is conservative; second-gen self-citation may land later if the lazy-anchor / NLI-sidecar work proves the error rate is low enough.) - -### 2.2 Retrieval integration - -The source-role classifier in `aborist/qa/query.py:_classify_source_role` recognizes the `aborist://providence/` URI scheme and tags those documents `self_reference_source`. This role gets added to the default `claim_lattice_allowed_source_roles` allowlist so claims can verify against self-reference spans. - -`SOURCE_ROLE_BUDGET_WEIGHTS` for `self_reference_source` = 1.0 (same as background; deliberately not boosted — Wikipedia is still the canonical primary). - -### 2.3 Recursive Merkle proof - -When Q2 cites Q1's answer span, the audit chain becomes "Q2 → Q1 → Wikipedia chunk." The original `chunk_root` remains the leaf; Q1's `run_dag_root` becomes an intermediate node. v9.8's admissibility ledger already supports this layering — that's the recursive-cores insight ("planet toward center compression" in CLAUDE.md). No schema change needed; the providence record's `merkle_proof` blob carries the parent chain. - -### 2.4 Falsification trust model - -Per fox: "we trust strict statements as fact unless a verifier falsifies it." - -- STRICT live records → trusted, ingested as substrate -- A falsifier (manual via `aborist providence --falsify`, drift-detection in `evict_to_cold` rehydrate, mesh-sync mismatch) flips state to `failed` / `stale` / `quarantined` -- On next `ingest-self-providence` run, the falsified record's document gets removed from FTS (idempotent re-ingest with `supersedes` edge — the documents table already supports this) -- Until that re-ingest, the stale record stays in retrieval but the verifier on a Q2 citing it sees `falsification_state != live` for the underlying `source_root` and rejects the citation. **Fail-closed: a falsified record CANNOT serve as STRICT substrate even if it's still in the documents table.** - -### 2.5 Operational model - -- `make ingest-self-providence` — promote STRICT-live providence records older than the kindergarten window into the document corpus -- Run on a cron (every hour, mirroring the mesh kindergarten window) -- Idempotent: same record → same document_root → no-op insert. Replaced records get a `supersedes` edge linking new → old. - -### 2.6 What's NOT in this design - -- **Aggregation** of multiple Q&A records into a synthesized "summary" record. Possible follow-on but not MVP. -- **Cross-shard self-reference**: each shard can self-promote within itself; cross-shard cites work via the existing shards-dir UNION. No new code. -- **Live retrieval from providence_cache** (Option B from the design discussion). The MVP uses snapshot ingestion (Option A) for simplicity. Live virtual sourcing would be an optimization if duplication becomes painful. -- **Self-reference for HYBRID records**. STRICT only. HYBRID could be added as a soft-anchor source role with lower trust (own role: `self_reference_hybrid_source`, role weight 0.5). Holding for fox's call. - -## 3. Risks & mitigations - -| risk | mitigation | -|------|-----------| -| Lazy-anchor false-STRICT compounds — Q2 inherits Q1's bogus cite. | `lazy_anchor_demoted` records skipped on promotion. Once the NLI sidecar lands, `semantic_demoted` adds another gate. | -| Echo-chamber: same fact recycled across many records. | Kindergarten window (1h) + first-gen-only anti-recursion check. | -| Falsified record stays in retrieval until re-ingest. | Verifier checks `falsification_state` of the cited source_root at verify time; non-live cites get rejected. Fail-closed. | -| Storage bloat: every Q&A becomes a document. | Same chunker + Merkle as everything else; per-record cost is small. Periodic burn (`burn-kindergarten`) trims old or low-quality rows. | -| Schema/policy drift: a record promoted under v9.8.0 becomes invalid when chunking_version bumps. | Same schema-invariant rule as Wikipedia ingestion; `governance_policy_hash` migration applies. | - -## 4. Bench impact estimate - -Before any self-reference is in the corpus, no impact. Once `make ingest-self-providence` has been run for a while and a few hundred STRICT records are substrate: - -- Questions about aborist itself (mostly UNGROUNDED today) start grounding -- Questions tangentially related to past STRICT answers gain new anchors -- Strict-rate slowly climbs as the substrate accumulates trusted self-knowledge -- New failure modes: badly-anchored citations to providence chunks (the Wall + JP-dinos pattern repeats inside the self-reference layer). The lazy-anchor smell sidecar already covers this. - -## 5. Implementation summary - -Files added / modified: -- `aborist/sources/providence.py` — new module, `ProvidenceSource(Source)` with kindergarten / state filters -- `aborist/qa/query.py` — `_classify_source_role` recognizes `aborist://providence/` URIs; `claim_lattice_allowed_source_roles` adds `self_reference_source` -- `aborist/qa/runner.py` — same allowlist update for the per-document `ask` path -- `Makefile` — new `ingest-self-providence` target -- `tests/test_providence_source.py` — unit tests for the source filter + classifier - -CLI wire-up + bench validation deferred to follow-up commits once we've ingested-and-bench-validated on real data. diff --git a/tests/test_providence_source.py b/tests/test_providence_source.py index e4a9da1..ebd3baa 100644 --- a/tests/test_providence_source.py +++ b/tests/test_providence_source.py @@ -1,7 +1,7 @@ """Tests for ProvidenceSource — self-reference thought chains. Covers the four iteration-time gates documented in -docs/self-reference-thought-chains-design.md: +docs/self-reference-design.md: 1. audit_mode == 'STRICT' (HYBRID/UNGROUNDED excluded) 2. falsification_state == 'live' (failed/stale/quarantined excluded)