arborist/docs/tickets/ticket-000017-memory-root-lifelong-learning.md

Ticket #000017 — Memory-root: lifelong learning audit chain

Status: closed · landed 2026-05-07 Opened: 2026-05-07 Closed: 2026-05-07 Scope: Spec + initial wire-up of a memory_root commitment that binds an arborist-hosted agent's evolving cross-query memory into the audit chain. Distinct from per-query providence cache (which is keyed on cache_key dimensions); memory_root captures lifelong, cross-query state — accepted/rejected belief updates, recurring failure motifs, domain-specific corrections. Audience: fox + future blackops shifts. Hard constraint: memory_root is hard-hash committed; no soft state in its preimage. Memory falsification cascades MUST NOT invalidate the entire prior providence cache; cascade bounded by explicit policy.

---

1. Problem statement

DNA analogy thread (Asia/Kuala_Lumpur, 2026-05-04) lists the heritable units a Merkle-AGI organism passes to its child:

what worked, what failed, why it failed, which benchmark proved the fix, which policy allowed the fix, which verifier accepted the fix, which SelfModel update resulted.

The genome schema includes memory_root: .... arborist has no such object. Today's memory model:

Memory layer	Today
Providence cache	Per-cache_key answers; no cross-query continuity.
Audit events	Per-shard chain; not summarized.
Concepts table	Per-shard rivalry/synonym graph; static once built.
SelfModel (ticket #000014)	Identity, not behavior history.
Long-term failure recall	None. Same failure shape repeats across queries with no surfacing.

Without memory_root, an agent that hallucinates about spin glass on Monday has no audit-bound memory of that on Tuesday. The falsification record exists in audit_events but is not summarized, not surfaced to the verifier on the next related query, and not inheritable when a child shard forks.

1.1 Concrete failure scenarios

Scenario A — Repeating a known motif. Query 1 hallucinates "spin glass cited from Quantum chromodynamics article" → audited as TITLE_MISMATCH → falsification recorded. Query 2 (related topic) runs through the same retrieval path, hits the same chunk, makes a similar mistake. Without memory_root, the verifier has no way to say "this retrieval path produced a falsified claim before; up- weight skepticism."

Scenario B — Forked agent loses context. A new shard splits off with parent_shard_root pointing to the existing chain. The new shard has no record of recurring failure motifs unless the audit chain is replayed. memory_root would let the new shard inherit a compact summary without replaying every event.

Scenario C — Operator query "what does this agent struggle with?" Today: no answer. With memory_root + a small CLI surface: a queryable summary backed by audit history.

---

2. Design choices

2.1 What goes in memory_root

A. Just a digest of the audit-event chain. Trivial; redundant with audit chain.

B. Curated summaries: failure-motif counts, recurring-error topic clusters, accepted-correction list. Useful, but how is the curation deterministic?

C. Hierarchical: chain-digest at root, multiple branch-digests for domain-specific summaries (e.g., "physics-domain-failures", "language-domain-corrections"). Compositional. Selectively inheritable.

Recommendation: C. Each summary is itself a deterministic projection over a slice of audit_events; the slice rule is committed in the memory-root manifest so verifiers can recompute.

2.2 Update cadence

A. Every audit_event triggers re-summary. High churn; expensive.

B. Periodic snapshot (every N events, or every N seconds). Bounded cost; staleness window.

C. On-demand. Operator triggers re-snapshot. Lowest churn, risk of stale state at critical moments.

Recommendation: B at default cadence (every 1000 audit_events, configurable). Operators can also force-trigger via CLI.

2.3 Falsification cascade scope

If a memory-summary entry is invalidated (e.g., a previously- accepted correction turns out to be wrong), what gets re-evaluated?

A. Nothing. Memory is purely advisory; falsification recorded but no cache impact. Safest. Loses the "memory shapes future decisions" benefit.

B. Only future queries. Existing providence cache stays valid. Practical. Cache continues to reflect what was known at the time.

C. Cascade re-evaluation. Falsification triggers re-evaluation of every providence record that cites the falsified memory entry. Strongest semantic guarantee. Expensive at scale.

Recommendation: B. Memory falsification flips state on upstream-citing records but doesn't auto-rerun them. Operator can trigger cascade re-evaluation explicitly when they want correctness

efficiency.

2.4 Cache_key folding

Same options as ticket #000014:

A. Fold into governance_policy_hash. Memory changes invalidate prior cache.

B. Sibling field on providence record, not in cache_key. Cache hits across memory shifts; memory changes are advisory.

Recommendation: B. Memory is advisory by default. Operator can opt into A via policy["memory_binding"]=True in deployments where memory changes should invalidate prior answers.

---

3. Recommendation

Hierarchical memory-root with periodic snapshot at every-1000-events cadence, advisory-by-default cache impact (sibling field, not in cache_key by default), explicit cascade-re-evaluation only on operator request.

---

4. Implementation sketch

4.1 Schema

CREATE TABLE memory_records (
  memory_root TEXT PRIMARY KEY,
  schema_version TEXT NOT NULL,
  parent_memory_root TEXT,
  audit_events_high_water TEXT NOT NULL,    -- last event_hash included
  branch_summaries_blob BLOB NOT NULL,      -- canonical-JSON of branch digests
  state TEXT NOT NULL DEFAULT 'live'
        CHECK(state IN ('live','stale','falsified')),
  created_at TEXT NOT NULL,
  falsified_at TEXT,
  falsified_reason TEXT
);

CREATE TABLE memory_branch_summaries (
  branch_id TEXT NOT NULL,        -- e.g., "failure-motif:title-mismatch"
  memory_root TEXT NOT NULL,
  summary_digest TEXT NOT NULL,
  summary_blob BLOB NOT NULL,
  count INTEGER NOT NULL,
  PRIMARY KEY(branch_id, memory_root),
  FOREIGN KEY(memory_root) REFERENCES memory_records(memory_root)
);

4.2 Module layout

arborist/memory/
├── __init__.py
├── snapshot.py         # build memory_root from audit_events slice
├── projections.py      # branch projection rules (deterministic)
├── falsify.py          # mark memory entries falsified
└── store.py            # CRUD

4.3 Initial branch projections (curated set, expandable)

• failure-motif:title-mismatch — count of TITLE_MISMATCH violations grouped by source title.
• failure-motif:warrant-missing — count grouped by claim shape.
• failure-motif:deflection — deflection signals grouped by topic.
• correction:operator-applied — corrections explicitly logged via CLI.

Each projection rule lives in projections.py and is canonicalized

• hashed; the hash is part of the branch summary's preimage so projection-rule changes produce a different memory_root.

4.4 CLI surface

arborist memory snapshot --shards-dir DIR
arborist memory show --root ROOT
arborist memory branches --root ROOT
arborist memory falsify --branch BR --reason TXT
arborist memory cascade --memory-root ROOT  # re-evaluate dependent records

4.5 SelfModel relationship

memory_root and selfmodel_root are sibling commitments. SelfModel is identity (capability claims, profile fingerprint). memory_root is behavior history (recurring motifs, accepted corrections). A SelfModel snapshot may cite a memory_root in its body (via memory_root field) so the audit reader can trace from "agent claims X" to "agent's behavior history when claim was made."

4.6 Tests

• Unit: snapshot deterministic given fixed audit_events high-water.
• Unit: branch projection rule changes produce different memory_root.
• Unit: parent_memory_root chain traverses cleanly.
• Integration: memory falsification flips state; cascade is no-op unless explicitly invoked.
• Audit chain: chain-check stays clean after memory_root landing.

4.7 Bumps

• New schema (memory_records, memory_branch_summaries). Additive.
• Optional governance_policy.memory_binding flag. Default off.
• New CLI subcommand arborist memory.

---

5. Out of scope

• Cross-shard memory reconciliation. Each shard maintains its own memory_records.
• Soft memory (e.g., recent-failure cache, sliding-window popularity counts). Sidecar territory; not in this ticket.
• Memory-driven retrieval ("avoid this source — it produced a falsified claim recently"). Future capability layer.
• Fork inheritance (child shard inherits memory_root from parent). Mesh-level concern; depends on v8 consensus (ticket #000012).

---

6. Status

Closed 2026-05-07. Scope delivered:

• Schema migration _migrate_memory_root adds memory_records + memory_branch_summaries tables. Sibling state — does NOT enter cache_key by default (per §2.4 advisory mode).
• Module arborist.memory:
  • canonical.py — MemorySnapshot + BranchSummary dataclasses, branch_digest, memory_root SHA-256 derivation. Branch order invariant (sorted by branch_id before hashing).
  • projections.py — three deterministic projections at v1: failure-motif:violations, audit-mode-distribution, falsification-state. Projection rule version pinned via PROJECTION_VERSION = "projections-v1".
  • snapshot.py — snapshot(conn) runs default branch set against current state, returns a MemorySnapshot. Side-effect-free.
  • store.py — store_snapshot persists + emits memory_snapshot_landed audit event. Idempotent on same root.
  • falsify.py — falsify and mark_stale flip state and emit memory_falsified / memory_marked_stale audit events.
• CLI: arborist memory snapshot|show|branches|falsify.
• SelfModel integration (ticket #000014 already shipped): SelfModel snapshot reads the latest live memory_root and folds it into the SelfModel canonical body. SelfModel root therefore changes when memory_root changes.
• Tests: tests/test_memory_root.py — 15 cases covering canonical body stability, root invariance under branch order, snapshot determinism + high-water progression, store idempotency, falsify/mark_stale, projection determinism, audit-chain integrity. Full suite: 1040 passed, 36 skipped.

Out-of-scope items (deferred to follow-ups):

• Cascade re-evaluation (arborist memory cascade). Memory falsification stays advisory (option B in §2.3); explicit cascade is a follow-up.
• Cross-shard memory reconciliation. Per-shard for now.
• Memory-driven retrieval routing.
• Fork inheritance (mesh / v8 concern).