modified: .gitignore
modified: Makefile modified: arborist/cli.py new file: arborist/qa/progress.py modified: arborist/qa/query.py new file: arborist/qa/witness.py modified: bench/results/real-shard-baseline.json modified: bench/results/real-shard-baseline.md modified: docs/TICKETS.md new file: docs/tickets/ticket-000028-multi-modality-witness.md new file: greatest-live-rock-and-roll-song-ever-played.md new file: tests/test_witness.py
This commit is contained in:
parent
7e86774e44
commit
656b573198
12 changed files with 1420 additions and 63 deletions
|
|
@ -61,6 +61,8 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000028 | Multi-modality witness for canonical shapes | open · implementation in progress | 2026-05-08 | — |
|
||||
| #000027 | Canonical projections persist to providence_cache | open · design phase | 2026-05-08 | — |
|
||||
| #000026 | Real-shard workload baseline + search latency | in progress · Phase 1 + 2 landed 2026-05-08 | 2026-05-08 | — |
|
||||
| #000025 | 5F battery (Function · Finetuning · Falsification · Formulate · Feedback Loop) | in progress · Phase 1a landed 2026-05-08 | 2026-05-07 | — |
|
||||
| #000024 | 5T Phase 1b + Dav1DPrometheus vocabulary alignment | closed · landed 2026-05-08 | 2026-05-07 | — |
|
||||
|
|
@ -90,4 +92,4 @@ Newest first. Update on every open/close.
|
|||
|
||||
## Next ID
|
||||
|
||||
`000027`
|
||||
`000029`
|
||||
|
|
|
|||
423
docs/tickets/ticket-000028-multi-modality-witness.md
Normal file
423
docs/tickets/ticket-000028-multi-modality-witness.md
Normal file
|
|
@ -0,0 +1,423 @@
|
|||
# Ticket #000028 — Multi-modality witness for canonical shapes
|
||||
|
||||
**Status:** open · implementation in progress
|
||||
**Opened:** 2026-05-08
|
||||
**Scope:** When a question matches a canonical shape (arithmetic@v1,
|
||||
logic-kernel@v1, future kernels), fan out three independent answer
|
||||
paths in parallel — **python kernel**, **cache lookup**, **alien
|
||||
LLM** — and verify cross-modality agreement via kernel-canonicalization
|
||||
of every answer. Disagreement is recorded as falsification signal,
|
||||
not an error. Agreement is the strongest grounding signal arborist
|
||||
can offer.
|
||||
**Audience:** fox + future blackops shifts.
|
||||
**Hard constraint:** opt-in (default off). Tripling latency on every
|
||||
arithmetic question would tax operators who only want the kernel
|
||||
answer. The witness path runs only when explicitly enabled per call
|
||||
or per-policy.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
Canonical projections (post-#000027) provide deterministic ground
|
||||
truth for arithmetic and propositional logic. The LLM, in contrast,
|
||||
is trained to produce plausible text and can hallucinate even on
|
||||
trivial math (`0.1 + 0.2 = 0.30000000000000004` is famous LLM
|
||||
training data; the kernel returns `3/10` exactly). Today we have no
|
||||
systematic way to:
|
||||
|
||||
- **Detect when the LLM disagrees with ground truth** for canonical
|
||||
questions. A bench could measure this offline, but every live
|
||||
call that asks a canonical-shape question is also a calibration
|
||||
data point we throw away.
|
||||
- **Catch cache drift.** If the kernel was bumped (`@v1` → `@v2`)
|
||||
without invalidating prior rows, the cache silently serves stale
|
||||
answers. A live cross-check at lookup time catches this.
|
||||
- **Strengthen the warrant for canonical answers.** Today the
|
||||
warrant is "kernel-deterministic" — strong, but mono-modal.
|
||||
Three-path agreement is mono-modal × independent-training × prior-
|
||||
commitment — strictly stronger.
|
||||
|
||||
### 1.1 Architecture sketch
|
||||
|
||||
```
|
||||
question
|
||||
├─ python kernel (arithmetic@v1) → "3/10" evidence: kernel-deterministic
|
||||
├─ cache lookup (providence_cache) → "3/10" evidence: prior commitment
|
||||
└─ alien (LLM) (Hermes/Claude/etc) → "0.3" evidence: training-derived
|
||||
│
|
||||
└─ claim-lattice cross-modality verifier:
|
||||
canonicalize("0.3") via arithmetic@v1 → "3/10"
|
||||
all three byte-equal → STRICT-CANONICAL-WITNESSED
|
||||
```
|
||||
|
||||
### 1.2 Agreement matrix
|
||||
|
||||
For three modalities {kernel (K), cache (C), llm (L)} the agreement
|
||||
matrix has 7 distinct outcomes (kernel always present when this path
|
||||
fires; cache may be absent pre-#000027; LLM may be absent if
|
||||
canonicalization fails):
|
||||
|
||||
| K | C | L | Outcome label | Meaning |
|
||||
|---|---|---|---|---|
|
||||
| ✓ | ✓ | ✓ | `STRICT-WITNESSED` | All three byte-equal — strongest possible warrant |
|
||||
| ✓ | ✓ | ✗ | `LLM-DIVERGED` | Cache & kernel agree, LLM hallucinated — bench data |
|
||||
| ✓ | ✗ | ✓ | `CACHE-DRIFT` | Kernel & LLM agree, cache row stale — invalidate row |
|
||||
| ✓ | ✗ | ✗ | `LLM-AND-CACHE-DIVERGED` | Only kernel matches itself (degenerate) |
|
||||
| ✓ | absent | ✓ | `KERNEL-LLM-AGREE` | No prior cache row; kernel ≡ LLM |
|
||||
| ✓ | absent | ✗ | `KERNEL-LLM-DIVERGED` | No prior cache row; LLM disagrees with kernel |
|
||||
| ✓ | ✓ | absent | `KERNEL-CACHE-AGREE` | LLM produced unparseable text; kernel ≡ cache |
|
||||
|
||||
Kernel is always the ground truth. Cache and LLM are checked
|
||||
*against* the kernel. The kernel never participates in byte-equal
|
||||
comparison with itself — it's the reference.
|
||||
|
||||
---
|
||||
|
||||
## 2. Design choices
|
||||
|
||||
### 2.1 Parallelism
|
||||
|
||||
LLM call is ~2-5s I/O-bound. Kernel + cache are ~10ms CPU/SQL.
|
||||
Sequential = sum (slow); parallel = max (LLM only).
|
||||
|
||||
**A. ThreadPoolExecutor (RECOMMENDED).** Three threads, one per
|
||||
modality. The LLM call doesn't block kernel/cache. Total wall time
|
||||
≈ LLM time + small overhead. Plays well with the existing sync
|
||||
codebase (no asyncio refactor).
|
||||
|
||||
**B. asyncio.** Cleaner conceptually but the existing codebase is
|
||||
sync top-to-bottom. Pulling asyncio in for one call creates an
|
||||
event-loop boundary the rest of the code doesn't know about.
|
||||
|
||||
**C. Sequential.** Adds latency = kernel + cache + LLM ≈ LLM × 1.01.
|
||||
Simpler. Loses the "free LLM hit" framing (you wait for it anyway).
|
||||
|
||||
→ **A.** ~30 LOC delta. Standard library. Threads are I/O-bound so
|
||||
the GIL doesn't block.
|
||||
|
||||
### 2.2 LLM prompt design
|
||||
|
||||
The LLM must emit a canonicalize-able answer for byte-comparison
|
||||
to work. Two prompt strategies:
|
||||
|
||||
**A. Strict-format (RECOMMENDED).** System prompt explicitly
|
||||
constrains the output:
|
||||
|
||||
```
|
||||
You are a deterministic math/logic computer. Reply with ONLY the
|
||||
canonical answer — no prose, no explanation, no LaTeX, no leading
|
||||
"=", no trailing punctuation. For arithmetic emit a fraction (e.g.
|
||||
3/10) or integer. For propositional logic emit TRUE or FALSE.
|
||||
```
|
||||
|
||||
If the LLM disobeys, the canonicalizer rejects the answer →
|
||||
classified as `L=✗`. This itself is a valuable signal: the LLM
|
||||
ignored instructions on a question with a known-correct format.
|
||||
|
||||
**B. Extraction.** Accept arbitrary LLM output, run a regex/heuristic
|
||||
extractor to pull a numeric portion, then canonicalize. More
|
||||
permissive but introduces extraction errors as a confounder.
|
||||
|
||||
→ **A.** Cleaner separation. If we want extraction later, layer it
|
||||
on as a fallback when strict mode rejects.
|
||||
|
||||
### 2.3 Canonicalization for cross-modality comparison
|
||||
|
||||
The kernel canonicalizes each modality's answer to bytes:
|
||||
|
||||
```python
|
||||
kernel_bytes = arithmetic_v1.canonicalize(b"0.1 + 0.2") # "3/10"
|
||||
llm_bytes = arithmetic_v1.canonicalize(b"0.3") # "3/10"
|
||||
cache_bytes = arithmetic_v1.canonicalize(b"3/10") # "3/10" (idempotent)
|
||||
agreement = (kernel_bytes == llm_bytes == cache_bytes)
|
||||
```
|
||||
|
||||
**Idempotence is the key property.** The kernel is a deterministic
|
||||
projection — running it on its own output yields the same bytes.
|
||||
This is why we can compare across modalities by canonicalizing each
|
||||
to the same canonical form.
|
||||
|
||||
Failure modes:
|
||||
- LLM produces `"the answer is 0.3"` → `_CANONICAL_ARITHMETIC_RE`
|
||||
rejects (letters present) → `L=absent`.
|
||||
- LLM produces `"0.3"` → canonicalize succeeds → `L=✓`.
|
||||
- LLM produces `"0.4"` (wrong) → canonicalize succeeds → `L=✓` but
|
||||
`llm_bytes != kernel_bytes` → `LLM-DIVERGED`.
|
||||
|
||||
### 2.4 Default behavior
|
||||
|
||||
**Default OFF.** Tripling LLM cost on every arithmetic question is a
|
||||
tax operators don't want by default. Witness path enabled per-call
|
||||
via `--witness` CLI flag or per-policy via
|
||||
`policy["canonical_witness_enabled"] = True`.
|
||||
|
||||
Sampling (e.g., 5% of canonical questions get the witness path for
|
||||
opportunistic calibration) is **out of scope for this ticket**.
|
||||
Easy to add later as `policy["canonical_witness_sample_rate"] =
|
||||
0.05`.
|
||||
|
||||
### 2.5 What gets persisted
|
||||
|
||||
This ticket does **NOT** persist witness rows. Reasons:
|
||||
|
||||
1. #000027 (canonical persistence) is not yet implemented. Until
|
||||
then there's no canonical row to attach a witness to.
|
||||
2. The witness data model needs to settle — the agreement matrix
|
||||
may grow new outcomes as more kernels land.
|
||||
|
||||
**For MVP**: witness result surfaces on the return dict + audit-line
|
||||
render. When #000027 lands, witness data folds into the
|
||||
`providence_canonical` audit event body as a `witness` clause.
|
||||
|
||||
### 2.6 What `audit_mode` does the witnessed answer carry?
|
||||
|
||||
The runtime `audit_mode` stays `"CANONICAL_PROJECTION"`. The witness
|
||||
outcome is a render-layer label tail:
|
||||
|
||||
```
|
||||
0.1 + 0.2
|
||||
CANONICAL · via arithmetic@v1 [STRICT-WITNESSED · 3/3 modalities agree] 0.0s
|
||||
|
||||
3/10
|
||||
```
|
||||
|
||||
vs. divergence:
|
||||
|
||||
```
|
||||
2 + 2
|
||||
CANONICAL · via arithmetic@v1 [LLM-DIVERGED · llm said "5"] 3.2s
|
||||
|
||||
4
|
||||
```
|
||||
|
||||
The CLI render adds a witness tail; cache_key + governance_policy_hash
|
||||
+ programmatic callers see `audit_mode = "CANONICAL_PROJECTION"`
|
||||
unchanged. **Same pattern as the four-rung ladder render layer.**
|
||||
|
||||
---
|
||||
|
||||
## 3. Implementation sketch
|
||||
|
||||
### 3.1 New module `arborist/qa/witness.py` (~200 LOC)
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class ModalityResult:
|
||||
modality: str # "kernel" | "cache" | "llm"
|
||||
raw_answer: str | None # what the modality said (display)
|
||||
canonical_bytes: bytes | None # post-canonicalization bytes
|
||||
error: str | None # "PIS_REJECT" / "TIMEOUT" / etc.
|
||||
elapsed_ms: float
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Witness:
|
||||
pi_star_ref: str
|
||||
modalities: dict[str, ModalityResult]
|
||||
agreement_label: str # "STRICT-WITNESSED" / "LLM-DIVERGED" / etc.
|
||||
canonical_answer_bytes: bytes # the kernel's output (ground truth)
|
||||
|
||||
|
||||
def run_witness(
|
||||
*,
|
||||
question: str,
|
||||
pi_star_ref: str,
|
||||
canonical_answer_bytes: bytes, # already-computed kernel output
|
||||
cache_lookup: Callable[[], bytes | None], # closure that returns cached bytes or None
|
||||
chat_client: ChatClient | None,
|
||||
model_id: str,
|
||||
timeout_s: float = 10.0,
|
||||
progress: Progress | None = None,
|
||||
) -> Witness:
|
||||
"""Fan out kernel + cache + LLM in parallel; return Witness."""
|
||||
...
|
||||
|
||||
|
||||
def _canonicalize_via_kernel(text: str, pi_star_ref: str) -> bytes | None:
|
||||
"""Run the named kernel on `text`. Returns canonical bytes, or None
|
||||
if the kernel rejects (PiStarError / non-canonical shape)."""
|
||||
...
|
||||
|
||||
|
||||
def _classify_agreement(modalities: dict[str, ModalityResult]) -> str:
|
||||
"""Map modality outcomes → STRICT-WITNESSED / LLM-DIVERGED / etc."""
|
||||
...
|
||||
```
|
||||
|
||||
### 3.2 Edit `arborist/qa/query.py` canonical branch (~40 LOC delta)
|
||||
|
||||
```python
|
||||
if canonical_match is not None:
|
||||
pi_star_ref, canonical_bytes = canonical_match
|
||||
answer_text = canonical_bytes.decode("utf-8", errors="replace")
|
||||
|
||||
witness = None
|
||||
if policy.get("canonical_witness_enabled", False):
|
||||
from arborist.qa.witness import run_witness
|
||||
# Cache lookup is a no-op closure pre-#000027 (always returns
|
||||
# None). Post-#000027 it's the actual canonical_cache lookup.
|
||||
def _cache_lookup() -> bytes | None:
|
||||
return None # TODO: wire post-#000027
|
||||
witness = run_witness(
|
||||
question=question,
|
||||
pi_star_ref=pi_star_ref,
|
||||
canonical_answer_bytes=canonical_bytes,
|
||||
cache_lookup=_cache_lookup,
|
||||
chat_client=chat_client,
|
||||
model_id=model_id,
|
||||
timeout_s=float(policy.get("canonical_witness_timeout_s", 10.0)),
|
||||
progress=progress,
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "canonical_projection",
|
||||
"audit_mode": "CANONICAL_PROJECTION",
|
||||
"verifier_method": "canonical_projection",
|
||||
"pi_star_ref": pi_star_ref,
|
||||
"answer_text": answer_text,
|
||||
"witness": witness.to_dict() if witness else None,
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
### 3.3 CLI surface
|
||||
|
||||
```bash
|
||||
make query Q="0.1 + 0.2" WITNESS=1
|
||||
.venv/bin/arborist query --witness "0.1 + 0.2"
|
||||
```
|
||||
|
||||
Mutex group `--witness` / `--no-witness` on `arborist query`.
|
||||
Make passthrough: `WITNESS=1` → `--witness`.
|
||||
|
||||
### 3.4 Render layer
|
||||
|
||||
`arborist/cli.py` audit-line render gains a witness tail when
|
||||
`result.get("witness")` is set:
|
||||
|
||||
```
|
||||
CANONICAL · via arithmetic@v1 [STRICT-WITNESSED · 3/3] 0.0s
|
||||
```
|
||||
|
||||
Detail block under the answer (when `--json` not used) prints each
|
||||
modality:
|
||||
|
||||
```
|
||||
witness:
|
||||
kernel: "3/10" (10ms)
|
||||
cache: absent (no prior commitment)
|
||||
llm: "3/10" (2841ms · agrees)
|
||||
```
|
||||
|
||||
### 3.5 Progress emitter
|
||||
|
||||
```
|
||||
[arborist 0.02s] canonical_projection.match pi_star_ref=arithmetic@v1
|
||||
[arborist 0.02s] witness.start parallel=3 timeout_s=10
|
||||
[arborist 0.03s] witness.modality.kernel ms=8 ok=true
|
||||
[arborist 0.03s] witness.modality.cache ms=1 ok=false reason=absent
|
||||
[arborist 2.84s] witness.modality.llm ms=2812 ok=true canonicalized=true
|
||||
[arborist 2.84s] witness.done label=STRICT-WITNESSED
|
||||
```
|
||||
|
||||
### 3.6 Tests `tests/qa/test_witness.py` (~120 LOC)
|
||||
|
||||
- `test_three_way_agreement` — kernel + cache + LLM all return same canonical bytes → `STRICT-WITNESSED`.
|
||||
- `test_llm_diverges` — kernel + cache agree, LLM returns different value → `LLM-DIVERGED`.
|
||||
- `test_llm_not_canonical` — LLM emits prose; canonicalizer rejects → `KERNEL-CACHE-AGREE` (LLM absent).
|
||||
- `test_cache_absent` — cache lookup returns None → `KERNEL-LLM-AGREE`.
|
||||
- `test_cache_drift` — kernel + LLM agree, cache returns stale value → `CACHE-DRIFT`.
|
||||
- `test_llm_timeout` — chat_client raises TimeoutError → LLM modality reports error, agreement label reflects absence.
|
||||
- `test_default_off` — witness only runs when policy flag is True.
|
||||
- `test_parallel_timing` — total wall time ≤ max(kernel, llm) + overhead, not sum.
|
||||
|
||||
### 3.7 Stub LLM client for tests
|
||||
|
||||
Existing `arborist.qa.client.StubClient(answer="...")` works.
|
||||
`StubClient(answer="3/10")` for agreement; `StubClient(answer="0.4")`
|
||||
for divergence; `StubClient(answer="the answer is 0.3")` for
|
||||
not-canonical; `StubClient(raises=TimeoutError)` for timeout.
|
||||
|
||||
---
|
||||
|
||||
## 4. Out of scope
|
||||
|
||||
- **Persistence of witness rows.** Wires in via #000027 once that ticket
|
||||
lands. For MVP, witness is runtime-only.
|
||||
- **Sampling rate** for opportunistic calibration. Add as `policy
|
||||
["canonical_witness_sample_rate"] = 0.05` later if useful.
|
||||
- **Falsification of LLM modality.** When `LLM-DIVERGED`, we record
|
||||
the divergence on the result dict but don't mutate any LLM
|
||||
reliability ledger. Future ticket: `LLMReliability` with
|
||||
per-shape accuracy tracking.
|
||||
- **Re-asking the LLM with feedback.** No retry / repair loop for
|
||||
canonical shapes — the kernel is ground truth, the LLM gets one
|
||||
shot.
|
||||
- **Multi-LLM witness** (Hermes + Claude + GPT4 vote). Single LLM
|
||||
for now. The "alien" leg is one independently-trained model.
|
||||
Ensemble is a later ticket.
|
||||
- **Cache-leg implementation.** Stub closure that returns None.
|
||||
Real implementation lands with #000027.
|
||||
|
||||
---
|
||||
|
||||
## 5. Risks and rollback
|
||||
|
||||
**Risk 1 — LLM cost.** Witness adds one full LLM call per canonical
|
||||
question. Default-off mitigates. Operators who flip `--witness` on
|
||||
accept the cost knowingly. Sampling rate (out of scope) addresses
|
||||
opportunistic-calibration use.
|
||||
|
||||
**Risk 2 — LLM canonicalization rate.** If the LLM ignores the
|
||||
strict-format prompt > 30% of the time, the witness is mostly
|
||||
noise. Bench fixture in §3.6 includes adherence measurement;
|
||||
fox decides the floor.
|
||||
|
||||
**Risk 3 — Threading correctness.** ThreadPoolExecutor on three
|
||||
short-lived threads is well-trodden territory; primary concern is
|
||||
clean shutdown on timeout. `concurrent.futures.wait(timeout=...)`
|
||||
+ `future.cancel()` handles it.
|
||||
|
||||
**Risk 4 — Render-layer divergence.** The audit-line tail is render-
|
||||
only. Cache_key, governance_policy_hash, and run_dag stay
|
||||
byte-identical to non-witness canonical answers. Same render-layer
|
||||
discipline as the four-rung ladder.
|
||||
|
||||
**Rollback**: set `policy["canonical_witness_enabled"] = False`
|
||||
(default). Witness module stays installed but never fires.
|
||||
Pre-existing canonical answers unaffected.
|
||||
|
||||
---
|
||||
|
||||
## 6. Acceptance criteria
|
||||
|
||||
1. `make query Q="0.1 + 0.2" WITNESS=1` returns the same `3/10`
|
||||
answer with a `witness` field on the result dict + a witness
|
||||
tail on the audit-line render.
|
||||
2. When the LLM emits a canonicalize-able answer that matches the
|
||||
kernel: `agreement_label = "KERNEL-LLM-AGREE"` (cache absent
|
||||
pre-#000027) or `"STRICT-WITNESSED"` (cache present, post-
|
||||
#000027).
|
||||
3. When the LLM emits a wrong value: `agreement_label =
|
||||
"KERNEL-LLM-DIVERGED"` and the LLM's raw answer is preserved
|
||||
in the witness dict for bench analysis.
|
||||
4. Default `make query Q="0.1 + 0.2"` (no `WITNESS=1`) does NOT
|
||||
call the LLM — preserves the ~10ms canonical fast-path
|
||||
latency.
|
||||
5. `tests/qa/test_witness.py` passes (8 cases listed in §3.6).
|
||||
6. Existing tests unchanged.
|
||||
7. Total wall time when witness=on ≈ LLM call + 50ms overhead
|
||||
(parallel, not sequential).
|
||||
8. Progress emitter emits witness.start / witness.modality.* /
|
||||
witness.done lines.
|
||||
|
||||
---
|
||||
|
||||
## 7. Dependencies
|
||||
|
||||
- Builds on `arborist/pi_star/arithmetic.py` and `arborist/pi_star/
|
||||
logic.py`.
|
||||
- Reuses `arborist/qa/client.py` ChatClient + StubClient.
|
||||
- Reuses progress emitter from 2026-05-08 state-machine wiring.
|
||||
- Independent of #000027 (witness can ship before persistence;
|
||||
cache-leg is a no-op closure until #000027 lands).
|
||||
Loading…
Add table
Add a link
Reference in a new issue