qa: self-reference thought chains — STRICT-as-fact substrate

Closes the recursive-grounding gap fox surfaced today: aborist
tends Wikipedia trees but never grafts its own past Q&A records
into the forest. Each query starts from cold corpus retrieval;
prior providence_cache records sit unread until the same question
is re-asked (cache_key match). The "kindergarten thought chains"
framing names the gap — the system has a kindergarten of thoughts
(early STRICT records) that should mature into citable substrate
as they cool.

This commit lands the MVP — STRICT live providence records past
the kindergarten window become substrate via a new Source
subclass. Trust model per fox: "we trust strict statements as
fact unless a verifier falsifies it."

NEW
---
- docs/self-reference-thought-chains-design.md — full architecture
  doc covering the four iteration-time gates, the recursive Merkle
  proof story (Q2 → Q1 → Wikipedia chunk per v9.8's recursive-
  cores insight), the falsification trust model (state=live as the
  fail-closed gate), risks (lazy-anchor compounding, echo chambers,
  storage bloat), and bench-impact estimate.
- aborist/sources/providence.py — ProvidenceSource(Source) with
  four hard gates:
    1. audit_mode == 'STRICT' (HYBRID/UNGROUNDED stay opaque)
    2. falsification_state == 'live' (failed/stale/quarantined
       excluded — verifier-falsification mechanism per fox)
    3. now - created_at >= kindergarten_seconds (default 1h —
       fresh thoughts cool first; kills tight echo loops)
    4. anti-recursion: records whose answer text contains a
       self-reference URI are skipped — first-generation only
- tests/test_providence_source.py — 10 unit tests covering each
  gate plus the URI-scheme source-role classifier
- Makefile target `ingest-self-providence` (KG_SECONDS=3600
  default; iterates each shard and self-promotes its STRICT live
  records — cross-shard sharing happens via the existing
  shards-dir UNION at retrieval time)

WIRE-UP
-------
- aborist/qa/query.py
  - SOURCE_ROLE_BUDGET_WEIGHTS: self_reference_source = 1.0
    (same as background — Wikipedia stays canonical primary;
    self-reference is supplementary anchoring)
  - SOURCE_ROLE_RANK_WEIGHTS: self_reference_source = 0.9
  - _classify_source_role: short-circuits on aborist://providence/
    URI prefix → self_reference_source regardless of title shape
  - DEFAULT_QUERY_POLICY['claim_lattice_allowed_source_roles']
    += 'self_reference_source'
- aborist/qa/runner.py — same allowlist update for the
  per-document `ask` path
- aborist/cli.py — `aborist ingest --source providence` reads the
  providence_cache from the same shard it writes into;
  --kindergarten-seconds flag plumbed through

NOT IN THIS COMMIT
------------------
- Aggregation of multiple Q&A records into synthesized summary
  records (follow-on)
- Self-reference for HYBRID records (only STRICT is substrate
  today; HYBRID could land later as a soft-anchor role with
  lower trust)
- Live virtual sourcing (the design discusses it; MVP uses
  snapshot ingestion so existing FTS / chunker / Merkle apply
  with zero schema change)
- A live bench validating actual lift on self-reference questions
  (requires running ingest-self-providence then bench; deferred
  to follow-on commit on real data)

10 new unit tests pass; full suite at 482 passed / 21 skipped
(live fixtures gated).
This commit is contained in:
russell@unturf.com 2026-05-01 10:16:47 -04:00
parent 8506a7068f
commit 8de00442f1
No known key found for this signature in database
7 changed files with 643 additions and 30 deletions

View file

@ -29,7 +29,7 @@ SEARCH_Q ?= computer
ingest ingest-cur ingest-old ingest-xml ingest-xml-history \
ingest-xml-attached ingest-abstract \
ingest-grok ingest-grok-media \
ingest-self ingest-git ingest-hg \
ingest-self ingest-self-providence ingest-git ingest-hg \
verify search stats test test-live docs chain-check chain-check-shards \
falsify burn burn-kindergarten inspect bootstrap-crawler test-crawler crawl-ingest \
recrawl-check bench-qa clean clean-db clean-data help
@ -332,6 +332,20 @@ ingest-self: bootstrap ## ingest this repo's HEAD into a dedicated shard
@mkdir -p $(SHARDS_DIR)
$(ABORIST) --db $(SELF_SHARD) ingest --source git_repo --path $(SELF_REPO)
# Self-reference: promote STRICT live providence records past the
# kindergarten window into each shard's documents table. Each shard
# self-promotes only its own records; cross-shard sharing happens
# via the existing shards-dir UNION at retrieval time. Run on a cron
# (e.g. hourly) to keep the substrate fresh.
# See docs/self-reference-thought-chains-design.md.
KG_SECONDS ?= 3600
ingest-self-providence: bootstrap ## promote STRICT live providence records into the document corpus [KG_SECONDS=3600]
@mkdir -p $(SHARDS_DIR)
@for db in $(SHARDS_DIR)/*.db; do \
echo ">> promoting providence records: $$db"; \
$(ABORIST) --db $$db ingest --source providence --kindergarten-seconds $(KG_SECONDS); \
done
# Generic git-repo ingest: aim it at any local clone via GIT_REPO=...
GIT_REPO ?= $(CURDIR)
GIT_SHARD := $(SHARDS_DIR)/$(notdir $(GIT_REPO))-git.db