# Ticket #000028 — Multi-modality witness for canonical shapes

**Status:** closed · landed 2026-05-09 (cache-leg wired post-#000027; STRICT-WITNESSED reachable)
**Opened:** 2026-05-08
**Scope:** When a question matches a canonical shape (arithmetic@v1,
logic-kernel@v1, future kernels), fan out three independent answer
paths in parallel — **python kernel**, **cache lookup**, **alien
LLM** — and verify cross-modality agreement via kernel-canonicalization
of every answer. Disagreement is recorded as falsification signal,
not an error. Agreement is the strongest grounding signal arborist
can offer.
**Audience:** fox + future blackops shifts.
**Hard constraint:** opt-in (default off). Tripling latency on every
arithmetic question would tax operators who only want the kernel
answer. The witness path runs only when explicitly enabled per call
or per-policy.

---

## 1. Problem statement

> **Terminology note** (per §8.1): in this ticket *modality* refers
> to a **witness channel** — an independent epistemic source
> (deterministic kernel, persisted cache, LLM output) — not to a
> *carrier* (text / image / audio / world state). Cross-carrier
> witness extensions are a later ticket once π* libraries support
> non-text carriers.

Canonical projections (post-#000027) provide deterministic ground
truth for arithmetic and propositional logic. The LLM, in contrast,
is trained to produce plausible text and can hallucinate even on
trivial math (`0.1 + 0.2 = 0.30000000000000004` is famous LLM
training data; the kernel returns `3/10` exactly). Today we have no
systematic way to:

- **Detect when the LLM disagrees with ground truth** for canonical
  questions. A bench could measure this offline, but every live
  call that asks a canonical-shape question is also a calibration
  data point we throw away.
- **Catch cache drift.** If the kernel was bumped (`@v1` → `@v2`)
  without invalidating prior rows, the cache silently serves stale
  answers. A live cross-check at lookup time catches this.
- **Strengthen the warrant for canonical answers.** Today the
  warrant is "kernel-deterministic" — strong, but mono-modal.
  Three-path agreement is mono-modal × independent-training × prior-
  commitment — strictly stronger.

### 1.1 Architecture sketch

```
question
  ├─ python kernel  (arithmetic@v1)        → "3/10"   evidence: kernel-deterministic
  ├─ cache lookup   (providence_cache)     → "3/10"   evidence: prior commitment
  └─ alien (LLM)    (Hermes/Claude/etc)    → "0.3"    evidence: training-derived
                    │
                    └─ claim-lattice cross-modality verifier:
                       canonicalize("0.3") via arithmetic@v1 → "3/10"
                       all three byte-equal → STRICT-CANONICAL-WITNESSED
```

### 1.2 Agreement matrix

For three modalities {kernel (K), cache (C), llm (L)} the agreement
matrix has 7 distinct outcomes (kernel always present when this path
fires; cache may be absent pre-#000027; LLM may be absent if
canonicalization fails):

| K | C | L | Outcome label | Meaning |
|---|---|---|---|---|
| ✓ | ✓ | ✓ | `STRICT-WITNESSED` | All three byte-equal — strongest possible warrant |
| ✓ | ✓ | ✗ | `LLM-DIVERGED` | Cache & kernel agree, LLM hallucinated — bench data |
| ✓ | ✗ | ✓ | `CACHE-DRIFT` | Kernel & LLM agree, cache row stale — invalidate row |
| ✓ | ✗ | ✗ | `LLM-AND-CACHE-DIVERGED` | Only kernel matches itself (degenerate) |
| ✓ | absent | ✓ | `KERNEL-LLM-AGREE` | No prior cache row; kernel ≡ LLM |
| ✓ | absent | ✗ | `KERNEL-LLM-DIVERGED` | No prior cache row; LLM disagrees with kernel |
| ✓ | ✓ | absent | `KERNEL-CACHE-AGREE` | LLM produced unparseable text; kernel ≡ cache |

Kernel is always the ground truth. Cache and LLM are checked
*against* the kernel. The kernel never participates in byte-equal
comparison with itself — it's the reference.

---

## 2. Design choices

### 2.1 Parallelism

LLM call is ~2-5s I/O-bound. Kernel + cache are ~10ms CPU/SQL.
Sequential = sum (slow); parallel = max (LLM only).

**A. ThreadPoolExecutor (RECOMMENDED).** Three threads, one per
modality. The LLM call doesn't block kernel/cache. Total wall time
≈ LLM time + small overhead. Plays well with the existing sync
codebase (no asyncio refactor).

**B. asyncio.** Cleaner conceptually but the existing codebase is
sync top-to-bottom. Pulling asyncio in for one call creates an
event-loop boundary the rest of the code doesn't know about.

**C. Sequential.** Adds latency = kernel + cache + LLM ≈ LLM × 1.01.
Simpler. Loses the "free LLM hit" framing (you wait for it anyway).

→ **A.** ~30 LOC delta. Standard library. Threads are I/O-bound so
the GIL doesn't block.

### 2.2 LLM prompt design

The LLM must emit a canonicalize-able answer for byte-comparison
to work. Two prompt strategies:

**A. Strict-format (RECOMMENDED).** System prompt explicitly
constrains the output:

```
You are a deterministic math/logic computer. Reply with ONLY the
canonical answer — no prose, no explanation, no LaTeX, no leading
"=", no trailing punctuation. For arithmetic emit a fraction (e.g.
3/10) or integer. For propositional logic emit TRUE or FALSE.
```

If the LLM disobeys, the canonicalizer rejects the answer →
classified as `L=✗`. This itself is a valuable signal: the LLM
ignored instructions on a question with a known-correct format.

**B. Extraction.** Accept arbitrary LLM output, run a regex/heuristic
extractor to pull a numeric portion, then canonicalize. More
permissive but introduces extraction errors as a confounder.

→ **A.** Cleaner separation. If we want extraction later, layer it
on as a fallback when strict mode rejects.

### 2.3 Canonicalization for cross-modality comparison

The kernel canonicalizes each modality's answer to bytes:

```python
kernel_bytes  = arithmetic_v1.canonicalize(b"0.1 + 0.2")   # "3/10"
llm_bytes     = arithmetic_v1.canonicalize(b"0.3")         # "3/10"
cache_bytes   = arithmetic_v1.canonicalize(b"3/10")        # "3/10"  (idempotent)
agreement = (kernel_bytes == llm_bytes == cache_bytes)
```

**Idempotence is the key property.** The kernel is a deterministic
projection — running it on its own output yields the same bytes.
This is why we can compare across modalities by canonicalizing each
to the same canonical form.

Failure modes:
- LLM produces `"the answer is 0.3"` → `_CANONICAL_ARITHMETIC_RE`
  rejects (letters present) → `L=absent`.
- LLM produces `"0.3"` → canonicalize succeeds → `L=✓`.
- LLM produces `"0.4"` (wrong) → canonicalize succeeds → `L=✓` but
  `llm_bytes != kernel_bytes` → `LLM-DIVERGED`.

### 2.4 Default behavior

**Default OFF.** Tripling LLM cost on every arithmetic question is a
tax operators don't want by default. Witness path enabled per-call
via `--witness` CLI flag or per-policy via
`policy["canonical_witness_enabled"] = True`.

Sampling (e.g., 5% of canonical questions get the witness path for
opportunistic calibration) is **out of scope for this ticket**.
Easy to add later as `policy["canonical_witness_sample_rate"] =
0.05`.

### 2.5 What gets persisted

This ticket does **NOT** persist witness rows. Reasons:

1. #000027 (canonical persistence) is not yet implemented. Until
   then there's no canonical row to attach a witness to.
2. The witness data model needs to settle — the agreement matrix
   may grow new outcomes as more kernels land.

**For MVP**: witness result surfaces on the return dict + audit-line
render. When #000027 lands, witness data folds into the
`providence_canonical` audit event body as a `witness` clause.

### 2.6 What `audit_mode` does the witnessed answer carry?

The runtime `audit_mode` stays `"CANONICAL_PROJECTION"`. The witness
outcome is a render-layer label tail:

```
0.1 + 0.2
  CANONICAL · via arithmetic@v1  [STRICT-WITNESSED · 3/3 modalities agree]  0.0s

3/10
```

vs. divergence:

```
2 + 2
  CANONICAL · via arithmetic@v1  [LLM-DIVERGED · llm said "5"]  3.2s

4
```

The CLI render adds a witness tail; cache_key + governance_policy_hash
+ programmatic callers see `audit_mode = "CANONICAL_PROJECTION"`
unchanged. **Same pattern as the four-rung ladder render layer.**

---

## 3. Implementation sketch

### 3.1 New module `arborist/qa/witness.py` (~200 LOC)

```python
@dataclass(frozen=True)
class ModalityResult:
    modality: str                      # "kernel" | "cache" | "llm"
    raw_answer: str | None             # what the modality said (display)
    canonical_bytes: bytes | None      # post-canonicalization bytes
    error: str | None                  # "PIS_REJECT" / "TIMEOUT" / etc.
    elapsed_ms: float

@dataclass(frozen=True)
class Witness:
    pi_star_ref: str
    modalities: dict[str, ModalityResult]
    agreement_label: str               # "STRICT-WITNESSED" / "LLM-DIVERGED" / etc.
    canonical_answer_bytes: bytes      # the kernel's output (ground truth)


def run_witness(
    *,
    question: str,
    pi_star_ref: str,
    canonical_answer_bytes: bytes,     # already-computed kernel output
    cache_lookup: Callable[[], bytes | None],  # closure that returns cached bytes or None
    chat_client: ChatClient | None,
    model_id: str,
    timeout_s: float = 10.0,
    progress: Progress | None = None,
) -> Witness:
    """Fan out kernel + cache + LLM in parallel; return Witness."""
    ...


def _canonicalize_via_kernel(text: str, pi_star_ref: str) -> bytes | None:
    """Run the named kernel on `text`. Returns canonical bytes, or None
    if the kernel rejects (PiStarError / non-canonical shape)."""
    ...


def _classify_agreement(modalities: dict[str, ModalityResult]) -> str:
    """Map modality outcomes → STRICT-WITNESSED / LLM-DIVERGED / etc."""
    ...
```

### 3.2 Edit `arborist/qa/query.py` canonical branch (~40 LOC delta)

```python
if canonical_match is not None:
    pi_star_ref, canonical_bytes = canonical_match
    answer_text = canonical_bytes.decode("utf-8", errors="replace")

    witness = None
    if policy.get("canonical_witness_enabled", False):
        from arborist.qa.witness import run_witness
        # Cache lookup closes over the prior persisted row's answer
        # bytes (post-#000027 / e19aed8). The real implementation in
        # arborist/qa/query.py:2158-2172 reads the cached_row picked
        # up at the top of the canonical branch — comparing against
        # that prior row, not anything we'd write in the same call,
        # so the leg is non-tautological.
        _cached_bytes = (
            cached_row["answer_text"].encode("utf-8", errors="surrogatepass")
            if cached_row is not None
            else None
        )
        _cache_lookup = lambda: _cached_bytes  # noqa: E731
        witness = run_witness(
            question=question,
            pi_star_ref=pi_star_ref,
            canonical_answer_bytes=canonical_bytes,
            cache_lookup=_cache_lookup,
            chat_client=chat_client,
            model_id=model_id,
            timeout_s=float(policy.get("canonical_witness_timeout_s", 10.0)),
            progress=progress,
        )

    return {
        "status": "canonical_projection",
        "audit_mode": "CANONICAL_PROJECTION",
        "verifier_method": "canonical_projection",
        "pi_star_ref": pi_star_ref,
        "answer_text": answer_text,
        "witness": witness.to_dict() if witness else None,
        ...
    }
```

### 3.3 CLI surface

```bash
make query Q="0.1 + 0.2" WITNESS=1
.venv/bin/arborist query --witness "0.1 + 0.2"
```

Mutex group `--witness` / `--no-witness` on `arborist query`.
Make passthrough: `WITNESS=1` → `--witness`.

### 3.4 Render layer

`arborist/cli.py` audit-line render gains a witness tail when
`result.get("witness")` is set:

```
CANONICAL · via arithmetic@v1  [STRICT-WITNESSED · 3/3]  0.0s
```

Detail block under the answer (when `--json` not used) prints each
modality:

```
witness:
  kernel: "3/10"  (10ms)
  cache:  absent  (no prior commitment)
  llm:    "3/10"  (2841ms · agrees)
```

### 3.5 Progress emitter

```
[arborist 0.02s] canonical_projection.match pi_star_ref=arithmetic@v1
[arborist 0.02s] witness.start parallel=3 timeout_s=10
[arborist 0.03s] witness.modality.kernel ms=8 ok=true
[arborist 0.03s] witness.modality.cache ms=1 ok=false reason=absent
[arborist 2.84s] witness.modality.llm ms=2812 ok=true canonicalized=true
[arborist 2.84s] witness.done label=STRICT-WITNESSED
```

### 3.6 Tests `tests/qa/test_witness.py` (~120 LOC)

- `test_three_way_agreement` — kernel + cache + LLM all return same canonical bytes → `STRICT-WITNESSED`.
- `test_llm_diverges` — kernel + cache agree, LLM returns different value → `LLM-DIVERGED`.
- `test_llm_not_canonical` — LLM emits prose; canonicalizer rejects → `KERNEL-CACHE-AGREE` (LLM absent).
- `test_cache_absent` — cache lookup returns None → `KERNEL-LLM-AGREE`.
- `test_cache_drift` — kernel + LLM agree, cache returns stale value → `CACHE-DRIFT`.
- `test_llm_timeout` — chat_client raises TimeoutError → LLM modality reports error, agreement label reflects absence.
- `test_default_off` — witness only runs when policy flag is True.
- `test_parallel_timing` — total wall time ≤ max(kernel, llm) + overhead, not sum.

### 3.7 Stub LLM client for tests

Existing `arborist.qa.client.StubClient(answer="...")` works.
`StubClient(answer="3/10")` for agreement; `StubClient(answer="0.4")`
for divergence; `StubClient(answer="the answer is 0.3")` for
not-canonical; `StubClient(raises=TimeoutError)` for timeout.

---

## 4. Out of scope

- **Persistence of witness rows.** Wires in via #000027 once that ticket
  lands. For MVP, witness is runtime-only.
- **Sampling rate** for opportunistic calibration. Add as `policy
  ["canonical_witness_sample_rate"] = 0.05` later if useful.
- **Falsification of LLM modality.** When `LLM-DIVERGED`, we record
  the divergence on the result dict but don't mutate any LLM
  reliability ledger. Future ticket: `LLMReliability` with
  per-shape accuracy tracking.
- **Re-asking the LLM with feedback.** No retry / repair loop for
  canonical shapes — the kernel is ground truth, the LLM gets one
  shot.
- **Multi-LLM witness** (Hermes + Claude + GPT4 vote). Single LLM
  for now. The "alien" leg is one independently-trained model.
  Ensemble is a later ticket.
- **Cache-leg implementation.** Stub closure that returns None.
  Real implementation lands with #000027.

---

## 5. Risks and rollback

**Risk 1 — LLM cost.** Witness adds one full LLM call per canonical
question. Default-off mitigates. Operators who flip `--witness` on
accept the cost knowingly. Sampling rate (out of scope) addresses
opportunistic-calibration use.

**Risk 2 — LLM canonicalization rate.** If the LLM ignores the
strict-format prompt > 30% of the time, the witness is mostly
noise. Bench fixture in §3.6 includes adherence measurement;
fox decides the floor.

**Risk 3 — Threading correctness.** ThreadPoolExecutor on three
short-lived threads is well-trodden territory; primary concern is
clean shutdown on timeout. `concurrent.futures.wait(timeout=...)`
+ `future.cancel()` handles it.

**Risk 4 — Render-layer divergence.** The audit-line tail is render-
only. Cache_key, governance_policy_hash, and run_dag stay
byte-identical to non-witness canonical answers. Same render-layer
discipline as the four-rung ladder.

**Rollback**: set `policy["canonical_witness_enabled"] = False`
(default). Witness module stays installed but never fires.
Pre-existing canonical answers unaffected.

---

## 6. Acceptance criteria

1. `make query Q="0.1 + 0.2" WITNESS=1` returns the same `3/10`
   answer with a `witness` field on the result dict + a witness
   tail on the audit-line render.
2. When the LLM emits a canonicalize-able answer that matches the
   kernel: `agreement_label = "KERNEL-LLM-AGREE"` (cache absent
   pre-#000027) or `"STRICT-WITNESSED"` (cache present, post-
   #000027).
3. When the LLM emits a wrong value: `agreement_label =
   "KERNEL-LLM-DIVERGED"` and the LLM's raw answer is preserved
   in the witness dict for bench analysis.
4. Default `make query Q="0.1 + 0.2"` (no `WITNESS=1`) does NOT
   call the LLM — preserves the ~10ms canonical fast-path
   latency.
5. `tests/qa/test_witness.py` passes (8 cases listed in §3.6).
6. Existing tests unchanged.
7. Total wall time when witness=on ≈ LLM call + 50ms overhead
   (parallel, not sequential).
8. Progress emitter emits witness.start / witness.modality.* /
   witness.done lines.

---

## 7. Dependencies

- Builds on `arborist/pi_star/arithmetic.py` and `arborist/pi_star/
  logic.py`.
- Reuses `arborist/qa/client.py` ChatClient + StubClient.
- Reuses progress emitter from 2026-05-08 state-machine wiring.
- Independent of #000027 (witness can ship before persistence;
  cache-leg is a no-op closure until #000027 lands).

---

## 8. Post-MVP follow-ups + design notes

Added 2026-05-09. MVP shipped in commit `656b573`
(`arborist/qa/witness.py` 394 LOC + `tests/test_witness.py` 354
LOC + query.py / cli.py / Makefile integration; 28/28 tests pass).
Captures items from the 2026-05-09 review
(`response_ticket-000027-canonical-projections-in-providence-cache.txt`)
that are deferred or worth flagging.

### 8.1 Witness vs modality terminology (review §2)

The review argued that `kernel / cache / LLM` are not "modalities"
in the carrier-modality sense (text / image / audio / world-state).
They are **epistemic witness channels**:

```
carrier modality:   text, image, code, table, audio, world state
witness channel:    deterministic kernel, persisted cache,
                    LLM output, external tool, human validator
```

The ticket title and body use "modality" throughout, which is
fine as terminology fox chose — but a future reader looking for
cross-carrier (text↔image) work might land here by mistake.
Suggest a single clarifying line near §1: *"In this ticket
'modality' refers to a witness channel — independent epistemic
source — not to a carrier (text/image/audio). Cross-carrier
witness extensions are a later ticket once π* libraries support
non-text carriers."*

Pure docs change; no code impact.

### 8.2 Capital-ledger integration (#000020) — landed in 708aa45

Witness mode adds one full LLM call per canonical question
(§5 Risk 1). The MVP recorded LLM latency on the witness result
dict but did NOT thread the cost into the capital ledger.

**Landed**: commit `708aa45` ("fan-out: warrant ladder wiring ·
witness follow-ups · 5F Phase 1d", 2026-05-09 12:42 EDT). For
each FIRED witness (sample-rate gating respected), one
``capital_ledger`` row is written under
``op_type='canonical_witness'`` with the cost shape inherited
from the ``qa`` estimator (LLM is LLM). Estimator inputs include
``prompt_chars``, ``answer_chars``, ``llm_seconds``,
``agreement_label``, and ``pi_star_ref`` so retroactive cost
re-estimation can rebuild cleanly. Best-effort: ledger-write
failure never fails the query (sidecar discipline).

Implementation in ``arborist/qa/query.py`` lines 2237-2285;
test coverage in ``tests/test_witness.py`` includes
``test_query_canonical_witness_records_capital_ledger`` which
verifies a row lands under the right op_type with the full
input blob. ForkScore (#000012) can now compare witness-on vs
witness-off forks honestly via ``capital_ledger.summary()``.

### 8.3 Sample-rate policy field — landed in 708aa45

§2.4 explicitly said sampling was out of scope for the MVP. MVP
fired witness on every canonical question when the master flag
was on, which made calibration-data collection an all-or-nothing
LLM-cost decision.

**Landed**: commit `708aa45` ("fan-out: warrant ladder wiring ·
witness follow-ups · 5F Phase 1d", 2026-05-09 12:42 EDT) added
the policy field:

```
policy["canonical_witness_sample_rate"] = 0.05  # 0.0 = effectively off, 1.0 = always on
```

Defaults to ``1.0`` (current always-on behavior preserved).
Operators wanting passive calibration set ``0.05`` to fire
witness on 5% of canonical questions while paying 5% of LLM
cost; ``0.0`` is effectively off. Gating uses
``random.random()`` for uniform distribution; clamped to
``[0, 1]``. The field folds into ``governance_policy_hash``
naturally via the existing policy-hash machinery — flipping it
invalidates prior records as expected.

Implementation in ``arborist/qa/query.py`` lines 2140-2155.
Test coverage in ``tests/test_witness.py`` includes
``test_query_canonical_witness_sample_rate_zero_skips`` and
``test_query_canonical_witness_sample_rate_one_always_fires``.

### 8.4 Dependency relationship vs #000027

§7 says #000028 is "independent of #000027" — true for shipping
order, but worth disambiguating design coupling:

- **Shipping**: MVP shipped before #000027 with the cache-leg
  stubbed to return None. `STRICT-WITNESSED` is structurally
  unreachable today — only `KERNEL-LLM-AGREE` /
  `KERNEL-LLM-DIVERGED` / etc. fire.
- **Full witness value**: requires #000027 to land. The strongest
  warrant in the matrix (3-of-3 byte-equal) is dormant until
  the cache leg has real rows to look up.

When #000027 lands, swap the no-op cache-lookup closure for the
real `lookup_canonical(qa_conn, ckey)` call — likely a one-line
change in the canonical branch of `arborist/qa/query.py`.

### 8.5 5F / 5T bench-data integration

Review §13 frames this as ongoing downstream value: every
witness-on canonical question is supervised calibration data.
Divergence events become natural fixtures for:

- **5F Falsification**: `LLM-DIVERGED` rows are falsification
  samples; metric `llm_canonical_error_rate`.
- **5F Feedback Loop**: divergence → improve prompt /
  canonicalizer / extractor; metric
  `feedback_efficiency = downstream_effect_count / Δcapital_cost`.
- **5T Truthtables**: LLM truth-table-row agreement with
  `logic-kernel@v1`.
- **5T Transfer Learning**: does the LLM improve across canonical
  shapes after calibration?
- **5S Semantics**: does free-form answer preserve invariant
  meaning after canonicalization?

Suggest a follow-up `make bench-witness-divergence` Makefile
target that dumps divergence audit events as bench fixtures.
Low-cost; defers naturally when fox decides to invest in
calibration-driven self-improvement.

### 8.6 Threat model — caveats from review §7

The review flags two qualifying claims worth pinning here:

- The strongest correct claim is **"no clean single-channel
  adversarial path"** — NOT "no adversarial path." Coordinated
  multi-channel compromise (e.g. shared canonicalizer bug
  affecting kernel and LLM canonicalization simultaneously) is
  still possible but requires strictly stronger adversary
  assumptions.
- Future hardening: **dual-kernel witness** —
  `arithmetic@v1_python` (Fraction + Decimal — current) plus
  `arithmetic@v1_sympy` (independent implementation). If two
  independent deterministic kernels agree alongside cache + LLM,
  warrant is even stronger. Out of scope; captured for future
  reference.

These aren't action items; they're calibrations of the warrant
strength claims so future readers don't overstate them.
