arborist/docs/cti-architecture.md

CTI architecture — layer cake mapped onto today's codebase

Date: 2026-05-01 Audience: fox + future blackops shifts. Purpose: name the architectural layers fox's de novo synthesis identifies (PROMETHEUS-Σ / CTI / Merkle-AGI-DAG / Reverse-RAG / v9.8 / Hermes) against the modules that already exist, so future work has a stable vocabulary for which layer it touches.

This is a mapping doc, not a rename. The codebase keeps its internal terms (claim_lattice, claim_lattice_pointer, verify_claim_lattice); the architectural labels (CTI, PROMETHEUS-Σ, Merkle-AGI) live in docs and commit messages where the cross-cutting story matters.

---

1. The layer cake

┌────────────────────────────────────────────────────────────┐
│   PROMETHEUS-Σ                                             │
│   policy / admissibility / one-shot mode / cache reuse     │
└────────────────────────────────────────────────────────────┘
            │
            ▼
┌────────────────────────────────────────────────────────────┐
│   CTI — Clause Tree Intelligence                           │
│   claim nodes · evidence edges · statuses · falsifiers     │
│   (the runtime IR for reasoning, NOT a model output format)│
└────────────────────────────────────────────────────────────┘
            │
            ▼
┌────────────────────────────────────────────────────────────┐
│   Merkle-AGI-DAG                                           │
│   roots · run-DAG · path proofs · audit lineage            │
└────────────────────────────────────────────────────────────┘
            │
            ▼
┌────────────────────────────────────────────────────────────┐
│   Reverse-RAG (Merkle Providence)                          │
│   answer claim → evidence pointer → source span → root     │
└────────────────────────────────────────────────────────────┘
            │
            ▼
┌────────────────────────────────────────────────────────────┐
│   v9.8 Merkle Providence Runtime                           │
│   8-dim cache_key · falsification states · audit chain     │
└────────────────────────────────────────────────────────────┘
            │
            ▼
┌────────────────────────────────────────────────────────────┐
│   Hermes (and any other base model)                        │
│   weak proposer of natural-language pointer-line clauses   │
└────────────────────────────────────────────────────────────┘

The animating principle: the model proposes, the runtime structures, the verifier falsifies, the renderer quotes, the Merkle-DAG commits, PROMETHEUS-Σ admits. Authority moves DOWN the stack — from training-time priors into runtime artifacts.

2. Per-layer responsibility + module map

2.1 PROMETHEUS-Σ (controller)

Owns:

• which answer_mode is active
• whether cache read / write is allowed
• whether a record is admissible for reuse (the 8-dim match + state filter)
• whether a verifier policy changed (governance hash bump)
• whether a run is scoreable under one-shot rules
• which falsification triggers stale a record

Today's codebase:

• aborist/qa/runner.py:ask — orchestrates the full lookup → infer → verify → admissibility flow for the per-document path
• aborist/qa/query.py:query — same orchestration for the multi-source RAG path
• DEFAULT_POLICY (runner) and DEFAULT_QUERY_POLICY (query) — the policy dict
• aborist.qa.keys.governance_policy_hash — folds policy into the cache_key

Status: Implicit. The layer exists as code but is not named. The de-novo doc proposes naming it; the codebase doesn't need a prometheus.py module today — the dispatch logic in runner.py and query.py IS PROMETHEUS-Σ.

2.2 CTI — Clause Tree Intelligence (reasoning IR)

Owns:

• claim nodes with text + evidence edges
• per-claim statuses (PARSED, NO_EVIDENCE_POINTER, EVIDENCE_LINKED, EVIDENCE_LINKED_PARTIAL, UNKNOWN_EVIDENCE_ID, SOURCE_ROLE_BLOCKED, CITATION_MISMATCH, SCHEMA_INVALID, etc.)
• soft-signal sidecars: lazy-anchor smell, deflection, partial-grounding split, semantic-entailment (designed in docs/verifier-semantic-gap-design.md)
• the boundary between hard checks (lexical, deterministic) and soft checks (heuristic, demote-only)

Today's codebase:

• aborist/qa/parse_claims.py:parse_pointer_claims — the model's pointer-line output → ParsedClaim records (the CTI compile step)
• aborist/qa/evidence.py:EvidenceObject — typed evidence nodes with evidence_id (content-addressed) + pointer_id (prompt-facing)
• aborist/qa/evidence.py:render_claim_lattice — the renderer that interpolates source spans by offset
• aborist/qa/verify.py:verify_claim_lattice — the deterministic verifier (six hard checks + sidecar signals)
• aborist/qa/verify.py:verify_claim_lattice_json — same for the JSON variant

Status: Real. CTI is what claim_lattice already is. The architectural insight from fox's synthesis — that this is an INTERNAL bytecode for reasoning, not a model output format — is the right reading: the model emits weak pointer lines; the runtime parses them into the CTI lattice; the lattice is what the verifier and renderer operate on.

2.3 Merkle-AGI-DAG (commitment)

Owns:

• per-run Merkle DAG over the 9 stages (question / retrieval / evidence_map / prompt / raw_answer / parsed_claim_lattice / verify / render / final_label)
• run_dag_root (sha256 of the canonical DAG)
• run_dag_blob (the structured data for inspection)
• the audit-chain integrity (each audit_event_hash chains to the previous)
• inclusion proofs from chunk → document_root → source corpus

Today's codebase:

• aborist/qa/dag.py:build_run_dag — emits the 9-stage DAG; returns {root, nodes, blob}
• aborist/merkle.py — the non-commutative HashCombine + odd-self-duplicate Merkle conventions (Python port of proxy.unturf.com/pkg/verified/merkle.go)
• aborist.store.append_audit — the only legal entry point for audit_events; computes event_hash = sha256(prev_event_hash || canonical(body))
• aborist/qa/keys.py:cache_key — the 8-dim composite hash that gates record reuse

Status: Real. The 9-stage DAG was the F-track work earlier in the session.

2.4 Reverse-RAG / Merkle Providence (evidence direction)

Owns:

• the claim → evidence_id → source_span → source_root → corpus reverse path
• per-claim provenance: every claim links forward to an evidence edge that maps to a chunk_root that proves into a document_root that proves into a source_root
• the renderer pulls source text by (chunk_root, offset_start, offset_end) — the model NEVER owns the quote text; the runtime interpolates it

Today's codebase:

• aborist/qa/evidence.py — EvidenceObject(source_root, document_uri, chunk_root, offset_start, offset_end, source_role, text_hash, span, evidence_id, pointer_id). The pointer_id (E1, E2, …) is what the model sees in the prompt; the evidence_id (content-addressed) is the cache/run-DAG handle. The runtime maps pointer_id → object → content-addressed evidence_id internally.
• aborist/qa/verify.py:verify_claim_lattice step 6 (citation-overlap check) — the claim → cited_span lexical-coverage gate
• aborist/qa/evidence.py:render_claim_lattice — interpolates source spans by offset, not Hermes text. Synthetic-elision is impossible by construction in pointer mode because the model never produces the quote string.

Status: Real. Pointer mode makes the reverse-RAG direction explicit; the model produces the LEFT side of the chain (claim) and points to the RIGHT side (evidence_id) — the runtime walks the rest.

2.5 v9.8 Merkle Providence Runtime (admissibility ledger)

Owns:

• the 8-dim cache_key: source_root | question_hash | model_profile_hash | conversation_hash | governance_policy_hash | schema_version | canonicalization_version | chunking_version
• falsification_state ∈ {live, failed, stale, quarantined} and the rules for transitions
• the providence_cache table + the audit_events chain
• the deliberate honesty boundary: NOT semantic truth, NOT hallucination elimination, NOT formal proof — just admissible provenance

Today's codebase:

• aborist/qa/keys.py — all eight dimensions hashed into cache_key
• aborist/store.py — schema, audit chain, transaction helpers
• aborist/qa/runner.py + aborist/qa/query.py — the cache-lookup-then-infer-then-write flow

Status: Real and honest. CLAUDE.md "Schema invariants (do not break)" enforces this boundary.

2.6 Hermes (proposer)

Owns:

• emitting weak pointer-line claims like Steve Jobs co-founded Apple. [E1]
• emitting JSON with evidence_ids referencing the same pointer ids
• nothing else — every other artifact (the lattice, the verdict, the rendered prose, the run DAG) comes from the runtime

Today's codebase:

• aborist/qa/client.py:OpenAICompatibleClient — HTTP client with retry on 502/503/504
• aborist/qa/client.py:StubClient — offline test stub
• claim_lattice_system_prompt + claim_lattice_grounding_reminder — the prompt fields that frame Hermes' role
• claim_lattice_json_system_prompt + claim_lattice_json_grounding_reminder — JSON variant

Status: Real. The negation-removal + atomic-claim work earlier in the session was about tightening the proposer's output shape so the runtime has cleaner input to compile.

3. The two information-flow directions

Two flows traverse the layers in opposite directions:

Inference flow (top-to-bottom-then-up):

  question → PROMETHEUS-Σ (admissibility check) → cache miss → 
  retrieval (Reverse-RAG path: titles → chunks → evidence map) →
  Hermes prompt (with pointer ids) → 
  Hermes response (pointer-line clauses) →
  CTI compile (parse_pointer_claims) →
  CTI verify (verify_claim_lattice) →
  CTI render (render_claim_lattice + spotlight excerpts) →
  Merkle-AGI commit (build_run_dag + audit_event) →
  v9.8 cache write (providence_cache row) →
  PROMETHEUS-Σ admit (audit_mode + falsification_state)

Audit flow (bottom-to-top, on demand):

  cache_key lookup → record + run_dag_blob →
  rebuild CTI lattice from run_dag stages →
  re-verify hard checks against current evidence map →
  if mismatch: PROMETHEUS-Σ falsifies → state flips

The audit flow is what makes Merkle-AGI O(log N + k) per challenged claim — you don't re-run Hermes; you replay the deterministic CTI verify path against the committed run-DAG.

4. Where each design doc lives

concern	doc
this layer-cake	docs/cti-architecture.md (you are here)
QA mode bench journey	docs/qa-modes-bench-2026-04-30.md
NLI semantic-gap sidecar	docs/verifier-semantic-gap-design.md
Self-reference (flat MVP)	docs/self-reference-design.md
Self-reference (deep, distillation-based)	docs/self-reference-design.md
Bench-maxing discipline	CLAUDE.md "Bench-maxing" section
Schema invariants	CLAUDE.md "Schema invariants (do not break)"
Convention list	CLAUDE.md "Conventions (do not silently change)"

5. What this layering DOESN'T claim

Per fox's de-novo synthesis (and CLAUDE.md's existing honesty boundary):

• Merkle-AGI does NOT make full-model verification free. It moves construction to O(N) once; targeted audit becomes O(log N + k). Full-model proof remains expensive.
• CTI is NOT semantic truth. The hard checks are lexical (substring, source-role, evidence-id resolution). Semantic checks (NLI, entailment, predicate compatibility) live in the soft-signal sidecar layer and never enter the proof path.
• v9.8 admissibility is NOT correctness. A STRICT record means "every claim grounded under the current verifier policy" — the policy is fallible (lazy-anchor false-positives, the Great Wall case in the bench journey doc). PROMETHEUS-Σ falsification is the corrective mechanism.
• Reverse-RAG does NOT prove sources are authoritative. It proves the answer pointed at the source the runtime committed to. Source quality is a separate problem (handled at retrieval time via _classify_source_role, noisy markers, title-purity rerank).

The substrate's honesty boundary stays where v9.8 placed it. The layer cake names the machinery; it does not extend the claims.

6. What's still open

These are the layer-cake-shaped work items still on the roadmap:

1. Verifier-policy-hash separation — landing in this commit pass. Fold answer_mode + parser_version + evidence_schema_version + manual_quote_policy + hard_checks_list into a 9th dimension that's separate from governance_policy_hash. Prevents cross-mode cache aliasing more cleanly.
2. Soft-signal taxonomy expansion — the NLI sidecar (docs/verifier-semantic-gap-design.md) is one. Predicate-compatibility, completeness, counterevidence, source-authority, scope-ambiguity are others. All demote-only, all out of the proof path. Implementations come one at a time as the bench surfaces motivating cases.
3. Self-reference distillation — STRICT claims become Cores via a new ProvidenceDistiller (see docs/self-reference-design.md). Lets new claims compose from existing facts, not just retrieve them.
4. PROMETHEUS-Σ explicit naming — deferred. The control logic exists; an extracted prometheus.py module is a refactor with no behavior change.

The architecture is real. The labels above name what's already there.