diff --git a/docs/tickets/ticket-000028-multi-modality-witness.md b/docs/tickets/ticket-000028-multi-modality-witness.md index e90cd72..944767d 100644 --- a/docs/tickets/ticket-000028-multi-modality-witness.md +++ b/docs/tickets/ticket-000028-multi-modality-witness.md @@ -19,6 +19,13 @@ or per-policy. ## 1. Problem statement +> **Terminology note** (per §8.1): in this ticket *modality* refers +> to a **witness channel** — an independent epistemic source +> (deterministic kernel, persisted cache, LLM output) — not to a +> *carrier* (text / image / audio / world state). Cross-carrier +> witness extensions are a later ticket once π* libraries support +> non-text carriers. + Canonical projections (post-#000027) provide deterministic ground truth for arithmetic and propositional logic. The LLM, in contrast, is trained to produce plausible text and can hallucinate even on @@ -456,43 +463,58 @@ non-text carriers."* Pure docs change; no code impact. -### 8.2 Capital-ledger integration (#000020) — deferred +### 8.2 Capital-ledger integration (#000020) — landed in 708aa45 Witness mode adds one full LLM call per canonical question -(§5 Risk 1). The MVP records the LLM latency on the witness -result dict but does NOT thread the cost into the capital -ledger. +(§5 Risk 1). The MVP recorded LLM latency on the witness result +dict but did NOT thread the cost into the capital ledger. -Recommend a follow-up commit to wire it: +**Landed**: commit `708aa45` ("fan-out: warrant ladder wiring · +witness follow-ups · 5F Phase 1d", 2026-05-09 12:42 EDT). For +each FIRED witness (sample-rate gating respected), one +``capital_ledger`` row is written under +``op_type='canonical_witness'`` with the cost shape inherited +from the ``qa`` estimator (LLM is LLM). Estimator inputs include +``prompt_chars``, ``answer_chars``, ``llm_seconds``, +``agreement_label``, and ``pi_star_ref`` so retroactive cost +re-estimation can rebuild cleanly. Best-effort: ledger-write +failure never fails the query (sidecar discipline). -```python -if witness is not None: - capital_ledger.record( - kind="canonical_witness", - llm_tokens=witness.modalities["llm"].tokens or 0, - llm_latency_ms=witness.modalities["llm"].elapsed_ms, - agreement_label=witness.agreement_label, - ) -``` +Implementation in ``arborist/qa/query.py`` lines 2237-2285; +test coverage in ``tests/test_witness.py`` includes +``test_query_canonical_witness_records_capital_ledger`` which +verifies a row lands under the right op_type with the full +input blob. ForkScore (#000012) can now compare witness-on vs +witness-off forks honestly via ``capital_ledger.summary()``. -Without this, ForkScore (#000012) cannot compare witness-on vs -witness-off forks honestly, and operators have no mechanism to -cap witness-mode spend per shard / per session. ~15 LOC; queued -as Phase 2 of this ticket or a new sub-ticket. +### 8.3 Sample-rate policy field — landed in 708aa45 -### 8.3 Sample-rate policy field — deferred +§2.4 explicitly said sampling was out of scope for the MVP. MVP +fired witness on every canonical question when the master flag +was on, which made calibration-data collection an all-or-nothing +LLM-cost decision. -§2.4 explicitly says sampling is out of scope. MVP fires witness -on every canonical question when the master flag is on. For -passive calibration data without full witness cost on every -question, a future bolt-on: +**Landed**: commit `708aa45` ("fan-out: warrant ladder wiring · +witness follow-ups · 5F Phase 1d", 2026-05-09 12:42 EDT) added +the policy field: ``` -policy["canonical_witness_sample_rate"] = 0.05 # 0.0 = off +policy["canonical_witness_sample_rate"] = 0.05 # 0.0 = effectively off, 1.0 = always on ``` -Captured here so it doesn't get re-discovered as a new -requirement when calibration-data hunger surfaces. +Defaults to ``1.0`` (current always-on behavior preserved). +Operators wanting passive calibration set ``0.05`` to fire +witness on 5% of canonical questions while paying 5% of LLM +cost; ``0.0`` is effectively off. Gating uses +``random.random()`` for uniform distribution; clamped to +``[0, 1]``. The field folds into ``governance_policy_hash`` +naturally via the existing policy-hash machinery — flipping it +invalidates prior records as expected. + +Implementation in ``arborist/qa/query.py`` lines 2140-2155. +Test coverage in ``tests/test_witness.py`` includes +``test_query_canonical_witness_sample_rate_zero_skips`` and +``test_query_canonical_witness_sample_rate_one_always_fires``. ### 8.4 Dependency relationship vs #000027