docs: fold Dav1d review (2026-05-26) into ticket stack

Three doc-only housekeeping items from Dav1d's de-novo reconciliation
of the architecture stack (no code changes; the active build is
#000061 cold-pack work, which is unaffected by this review):

1. Accepted-error formula. Dav1d corrected p_raw × (1-d)(1-r) to
   p_raw × (1 - dr) where r is conditional on detection. Searched
   docs/ and arborist/ — the wrong form does NOT appear in this
   tree (it lives in the external recursive-truth-maintenance / RCO
   manuscripts Dav1d cited). Nothing to fix here; recorded for the
   manuscript authors.

2. #000060 H-ABCDEFG split. Folded the M/C/X axis split into the
   harness scope: M (mechanism — does the substrate's plumbing work),
   C (capability — does it improve task performance), X (external
   adversarial — does it generalize outside author-designed fixtures).
   ACCEPT requires clearing all three. Without X, the harness risks
   self-validating benchmark theology — passing tests its own designers
   picked. Tagged at fixture-definition time; aggregator emits per-axis
   pass rates + combined ACCEPT verdict. Doesn't change the existing
   BatteryResult row schema.

3. #000062 Mechanistic Witness — new scaffold-only ticket. Specifies
   a content-addressed MechanisticWitnessRoot over (model_config,
   capture_policy, contrastive_prompts, features/neurons, intervention
   result, behavioral delta, safety policy) as a DIAGNOSTIC sidecar
   feeding SelfModel + benchmark fixtures. Four hard guardrails:
   diagnostic-only by default; sandbox intervention only; no production
   steering without governance ACCEPT via #000060 M+C+X; feature labels
   never become semantic proof. No code until a real falsifier use case
   exists + guardrails are CLAUDE.md rules + #000060 harness gates
   promotion. The dual-use risk (Pan et al. 2025: 0.1% MLP ablation
   breaks refusal in 72B models) makes governance-first framing
   load-bearing.

Next ID bumped to 000063.

No code change to arborist/. The in-flight v3 SPV corpus pack
(bmq47x6t3) continues unaffected.
This commit is contained in:
russell@unturf.com 2026-05-26 10:22:17 -04:00
parent d57ab41989
commit eba08beb61
No known key found for this signature in database
3 changed files with 194 additions and 2 deletions

View file

@ -111,8 +111,9 @@ Newest first. Update on every open/close.
| ID | Title | Status | Opened | Directive |
|----------|------------------------------------------------|-----------------------|------------|-----------|
| #000062 | Mechanistic Witness: governed diagnostic sidecar (CNA/SAE/Neuronpedia) | **scaffold-only · awaiting go/no-go** (2026-05-26; Dav1d de-novo review §4.7 / §9.1.F). Specification of a mechanistic-interpretability sidecar that produces a content-addressed `MechanisticWitnessRoot` over (model, prompts, capture policy, neurons/features, intervention deltas), used as a **diagnostic input** to SelfModel (#000014/#000017) + benchmark-fixture generation. **Hard constraint:** soft signals never enter the hard proof path — `audit_mode` does NOT move based on witness output, `providence_cache` is untouched, `governance_policy_hash` only moves via explicit ForkScore ACCEPT with M+C+X axes passing (#000060 §7). Four guardrails (diagnostic-only by default · sandbox intervention only · no production steering without governance · feature labels never semantic proof). Witness root TLV-encodes `model_config_root | activation_capture_policy_root | contrastive_prompt_set_root | feature_or_neuron_set_root | intervention_result_root | behavioral_delta_root | safety_policy_root`. Scaffold only — no code until a real falsifier-in-hand use case exists + the four guardrails are restated in CLAUDE.md as rules + #000060 H-ABCDEFG-M+C+X harness exists to gate promotion. Captured to keep mechanistic-interp tooling out of the substrate unless and until it earns its place; the dual-use risk (Pan et al. 2025 CNA: 0.1% MLP ablation breaks refusal in 72B models) makes the governance-first framing load-bearing. | 2026-05-26 | — |
| #000061 | Cold-pack distribution tier (boto3 S3-compat, DO Spaces + DVD-R targets) | **in progress** — opened 2026-05-25 (fox: "implement it now … target digital ocean first as a test"; later "I wanted a way to hydrate using tarballs (the core and important data) for bringing new machines up"). Tarball-only distribution mechanism — bucket holds `tar.zst` packs keyed by `hash_leaf(manifest)`, no individual-chunk blobs. New peers hydrate by downloading packs from the bucket's CDN edge (~4 HTTPS GETs for the current ~14.1M-chunk corpus, packs filled to 4.4 GB compressed each via streaming zstd, vs ~14M for individual blobs). Same artifact ≤4.4 GB safe-fit (~6.5 % buffer below DVD-R's 4.7 GB marketing capacity, accommodating ISO9660 overhead + media variance + drive-edge refusal) burns directly to physical media via `--local-dir` + `growisofs`. Packs are *delayed* snapshots: each pack pins the corpus `snapshot_root` it covers in audit + body, so falsifications between repacks produce new pack_hashes and stale packs stay in the bucket until explicit GC (future ticket). Packs include cores AND surfaces (full-corpus hydration). Default selection covers every hot chunk with local content in the shard. Multi-pack splitting via `stream_packs` (streaming zstd, FLUSH_BLOCK peek of compressed buffer after each chunk, cut at cap) fills each disc to ~4.4 GB compressed instead of leaving ~50% empty. One backend class (`S3CompatibleBackend` via boto3 + `endpoint_url`) covers AWS S3, DO Spaces, GCS S3-interop, R2, B2, MinIO. CDN public-read makes packs accessible to anyone; hash binding via in-tar `leaf_hash` member names makes hostile-bucket scenarios safe. Optional dep `[object-store]` = boto3>=1.34. Voyeur: credentials via standard `AWS_ACCESS_KEY_ID`/`_SECRET_ACCESS_KEY` (env or `~/.aws/credentials`), never printed; only endpoint URL + bucket name surface in logs. Initial individual-blob path (per-chunk S3 objects) was scoped+landed then **deleted same day** (fox: "what ever was blobs? I wanted a way to hydrate using tarballs"); the five-step deletion record lives inline in the doc — we'd added 14M-object storage and ~$70/hydrate request cost for a workflow that needed neither. Sizing math for current shards: ~4 packs total (17.2 GB compressed ÷ 4.4 GB compressed per pack), ~17 GB bucket storage, ~$0.34/mo DO Spaces. | 2026-05-25 | — |
| #000060 | H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + curvature) | open · awaiting go/no-go (2026-05-20; from Dav1dPrometheus *Protocol-Layer AGI* working report §26/§50/§82-84). The report's "decisive proof": run the SAME base model substrate-OFF vs substrate-ON over long-horizon/adversarial/non-jagged batteries, report the delta. Two new metrics: jaggedness tensor `J_norm` (§73 — variance across nearby variants, normalized by difficulty) + discrete performance curvature `κ_t` (§5.2, with the honest no-global-convexity bound, Erratum 5). A-vs-C spine (B optional, D=mesh OUT → #000012/#000016). Curvature-aware ForkScore extension folds into **#000012** (NOT a new ticket — reserved `iota`/`kappa` weight slots already exist). Budget: control arms = Hermes/Qwen, never Opus without go; heavy passes on GPU box. Held-out/mechanism-agnostic variants required so ABCDEFG doesn't self-validate. | 2026-05-20 | — |
| #000060 | H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + curvature) | open · awaiting go/no-go (2026-05-20; from Dav1dPrometheus *Protocol-Layer AGI* working report §26/§50/§82-84). The report's "decisive proof": run the SAME base model substrate-OFF vs substrate-ON over long-horizon/adversarial/non-jagged batteries, report the delta. Two new metrics: jaggedness tensor `J_norm` (§73 — variance across nearby variants, normalized by difficulty) + discrete performance curvature `κ_t` (§5.2, with the honest no-global-convexity bound, Erratum 5). A-vs-C spine (B optional, D=mesh OUT → #000012/#000016). Curvature-aware ForkScore extension folds into **#000012** (NOT a new ticket — reserved `iota`/`kappa` weight slots already exist). Budget: control arms = Hermes/Qwen, never Opus without go; heavy passes on GPU box. Held-out/mechanism-agnostic variants required so ABCDEFG doesn't self-validate. **2026-05-26 scope refinement (Dav1d review §4 / §7):** split the harness output into three axes — **H-ABCDEFG-M** (mechanism tests: does the substrate work?), **H-ABCDEFG-C** (capability tests: does the substrate improve task performance?), **H-ABCDEFG-X** (external adversarial: does it generalize outside author-designed fixtures?). The split prevents "self-validating benchmark theology" — a harness that reports only M+C with no X can pass while still failing on held-out adversarial generalization. Fold into the harness design before any code lands; doesn't change the bench-row schema (`carrier`/`domain`/`pi_star_ref`) but does change what "ACCEPT" requires (must clear all three axes). | 2026-05-20 | — |
| #000059 | Admission discipline: claim-graveyard query + self-providence quarantine | open · awaiting go/no-go (2026-05-20; Dav1dPrometheus report §11 Priority 2 + §57 "admissible state transition" thesis). Two coupled write-path mechanisms. **(A) GraveyardCheck:** storage already exists (`falsification_state` failed/stale/quarantined records ARE the graveyard); the gap is burden-shifting — a re-asked claim family with a known `failed` history should require stronger evidence (§42.9/§62-63). **(B) Self-providence quarantine:** `make ingest-self-providence` (Makefile:769) deliberately promotes STRICT records into the corpus — the exact self-confirmation loop §70 warns of — and ships with NO guard; detect/lineage-tag self-providence-descended evidence + quarantine for high-impact claims. Both advisory-sidecar-first (run-DAG only, never `providence_cache`/`audit_events`); demote-hooks bench-gated + unwired pending net win (would fold `governance_policy_hash`). Soft signals never enter the hard proof path. **Bounded-ingestion hard constraint (fox 2026-05-20, §7):** the graveyard MUST reach a steady-state size ∝ the *recurring*-error surface, never queries-ever — earn-to-enter (recurrence-gated), fingerprints not transcripts (UTXO-set analogy), decay/compact (evicts like a surface), off the hot path. BTC's lesson is bounded self-regulating ingestion, not "store everything." Gossip-group falsifier admission inherits difficulty-adjusted stable-rate + per-window budget (#000036) → enforced in #000012/`mesh/`. If it can't be bounded, it isn't built. | 2026-05-20 | — |
| #000058 | `cache_key_9` verifier-policy: mandatory-vs-legible decision + doc reconciliation | open · awaiting go/no-go (2026-05-20; Dav1dPrometheus report §2 Erratum 1 / §11 Priority 1 "mandatory cache_key_9"). **Five-step #1 correction:** the report's *correctness* premise is already false in arborist — verifier fields are a subset of the policy dict and so already fold into `governance_policy_hash` (`keys.py:269-275`); a verifier-rule change ALREADY changes the cache_key today. The explicit 9th `verifier_policy_hash` buys **audit legibility**, not correctness — so "mandatory" would stale every prior record for zero correctness gain. Decision: A leave-as-is (8-dim default, 9th optional) + doc reconcile [recommended] · B default-write 9-dim · C flag-staged bench-gated default-write — never a hard mandatory flip. Doc reconcile (CLAUDE.md/concepts "8-dim" → "8 + optional legible 9th") is the do-regardless. | 2026-05-20 | — |
| #000057 | Witness-preserving vs ordinary recursive loop: minimal deterministic drift A/B | **open · awaiting go/no-go · doc-only scaffold** (2026-05-19; fox relaying a Hamming-framed GPT-5.5 de-novo review). The review's one ticket-worthy nugget: prove the whitepaper's headline claim — *untracked evidence loss → unbounded recursive drift; witness-preserving state bounds it (detectable+reversible)*. Everything else in the review (rename, corpus hierarchy, IQ, ToE/Riemann/identity) **deliberately NOT ticketed** — narrative, not arborist engineering; don't-proliferate. Hard discipline encoded from the 2026-05-18 precedent: **instrument before experiment** (deterministic, no-LLM-judge, ground-truth-carrying, noise-resolvable — the `recall_at_k` discipline), **one task not eight** (recursive stale-source-invalidation *or* contradiction-repair — the falsification-state-exercising ones), minimal A/B (witness-binding ON vs OFF, N iterations, deterministic surviving-unsupported-claim count), non-claims pinned (necessary substrate, NOT AGI). **Sharpened by fox 2026-05-19 (§4b/§5):** the review's strongest point — every measurement so far is *within* Arborist, no control arm. v1 (smallest proof) = single-shot **Hermes-solo vs Arborist** on a fabrication-bait fixture; headline = fabrication-vs-honest-abstention delta (NOT raw accuracy — common-knowledge would falsely show null via parametric memory). **Judge correction (fox):** "no LLM-as-judge" is a *proof-path* rule (LLM judge inside Arborist's verifier = the circularity it kills); the experiment's judge is **external science** outside *both* arms → use a **strong SOTA model**, blinded + reference-grounded + transcript-logged. That lifts the atomic-only limit (judge handles essay grounding too — #000049's exclusion was right for the proof path, wrong for external science); deterministic gold-key demoted to a cheap atomic cross-check. Recursive-drift loop = v2, gated on v1. Still open (fox calls): fixture population, **which SOTA judge** (API cost + Voyeur egress note), spend. Scaffold; no build past gate-1. One ticket, not ten. | 2026-05-19 | — |
@ -175,4 +176,4 @@ Newest first. Update on every open/close.
## Next ID
`000062`
`000063`

View file

@ -114,3 +114,52 @@ over the batteries and reports per-battery delta + `J_norm` + curvature,
with gate numbers from bench-qa pipeline output (n≥3); the jaggedness
and curvature outputs are consumable by the #000012 ForkScore
extension. `docs/benchmarks.md` gains the new harness + metric schema.
## 7. Scope refinement — H-ABCDEFG-M / -C / -X split (2026-05-26)
Folded in from Dav1d's de-novo review of the architecture stack
(§1.2, §4.7, §9.1). The harness output is split along three axes so
"the harness ACCEPTed the upgrade" can't mean "we passed the tests we
designed for ourselves":
- **H-ABCDEFG-M (mechanism)** — does the substrate work?
Cache-key invalidation correctness, audit-chain integrity, π*
canonicalization round-trip, falsification-state transitions, ForkScore
scoring on synthetic per-axis deltas. These are tests of the
*plumbing* — pass/fail tells you whether the substrate's primitives
function as specified, not whether they help cognition.
- **H-ABCDEFG-C (capability)** — does the substrate improve task
performance?
The current 5S / 5T / 5F / 5R bench surface over real fixtures.
This is the "does it actually help?" axis. ACCEPT here on its own
is the load-bearing improvement claim — but only one of three.
- **H-ABCDEFG-X (external adversarial)** — does it generalize outside
author-designed fixtures?
Held-out / mechanism-agnostic / adversarial variants drawn from
outside the substrate's own design parameters. Fixtures that the
substrate's designers did NOT pick. Without this axis, the harness
is at risk of self-validating benchmark theology (§27.2 in the
ABCDEFG report; Erratum 4 / Anomaly 4 already noted in §5 of this
ticket but stated weaker than Dav1d's framing).
ACCEPT requires clearing all three axes. M-only or M+C with X failing
is a *partial* acceptance that should still surface in the harness
output, but cannot trigger ForkScore ACCEPT for substrate upgrades.
The split doesn't change `BatteryResult` row schema (already carries
`carrier` / `domain` / `pi_star_ref`); it changes what the harness
aggregator reports + what ForkScore consumes. Each fixture is tagged
with its axis at fixture-definition time; the harness reports per-axis
roll-ups + a combined verdict.
Implementation order when this ticket gates open:
1. Tag every existing 5S/5T/5F/5R fixture with `{M, C}` (current
fixtures are mechanism + capability by construction; X is the new
bucket).
2. Add X axis fixtures: held-out variants, prompt rewording, adversarial
carrier swaps, π*-collision attempts.
3. Aggregator emits `M_pass_rate`, `C_pass_rate`, `X_pass_rate` +
combined `ACCEPT_M_C_X` boolean. ForkScore consumes the per-axis
numbers (already lined up with the reserved weight slots in #000012).

View file

@ -0,0 +1,142 @@
# Ticket #000062 — Mechanistic Witness: governed diagnostic sidecar
**Status:** open · awaiting go/no-go · scaffold-only
**Opened:** 2026-05-26
**Scope:** Specification of a mechanistic-interpretability sidecar
(CNA / SAE / Neuronpedia-compatible) that produces a content-addressed
`MechanisticWitnessRoot` over (model, prompts, capture policy, neurons/
features, intervention deltas), used as a **diagnostic input** to
SelfModel + benchmark-fixture generation. **Never** a proof of semantic
truth; **never** a production-steering signal without explicit
governance promotion through ForkScore.
**Audience:** dav1d (architectural inflection — first time arborist has
framed interpretability tooling as part of the substrate at all; needs
the four guardrails below to be load-bearing before any code lands).
**Hard constraint:** soft signals never enter the hard proof path.
`audit_mode` does not move based on mechanistic-witness output;
`governance_policy_hash` only moves if/when a promotion is explicitly
ACCEPT'd via ForkScore with M+C+X axes passing (cf. #000060 §7).
## Problem
The recursive truth-maintenance framing (Dav1d review, 2026-05-26) puts
arborist's verifier + falsifier layers at the "what" of truth: was a
claim grounded, did its evidence support it, did a falsifier defeat it.
None of that surfaces the *internal* "why" of a model's behavior — the
neurons / SAE features / activation patterns that fire when the model
produces a refusal, a hallucination, or a low-confidence answer.
Two reasons this matters for arborist:
1. **SelfModel calibration improves** when the system has internal
evidence of its own failure modes (e.g. "this question class
activates the refusal cluster — expect higher refusal rate"). The
5R workspace (#000017 SelfModel) currently relies on behavioral
evidence only; a mechanistic channel adds an orthogonal signal.
2. **Benchmark fixtures become falsifiers.** Contrastive-neuron-
attribution work (Pan et al., 2025; the CNA paper Dav1d cited) shows
that ablating 0.1% of MLP neurons reduces refusal rate by >50%
across Llama / Qwen up to 72B. That kind of finding is itself a
*falsifier* for safety claims — "this model is safe because it
refuses X" is undermined by "ablating these N neurons breaks the
refusal." Treating that finding as a benchmark fixture lets the
substrate use it.
But — and this is the load-bearing caveat — mechanistic interpretability
is **dual-use** by construction. The same technique that identifies a
refusal circuit can be used to break it. arborist must absorb the
diagnostic value without giving up the governance discipline that keeps
soft signals out of the hard proof path.
## Design — four guardrails
Dav1d's review (§4.7) frames the integration rule. Restated for this
ticket:
1. **Diagnostic only by default.** Mechanistic-witness output feeds
SelfModel + benchmark fixtures + safety audits. It does NOT promote
audit_mode, write to providence_cache, or move governance_policy_hash.
2. **Sandbox intervention only.** Activation patching / neuron clamping
runs in a sandbox process, against a sandbox model load, against a
sandbox eval set. Production inference paths see no intervention.
3. **No production steering without governance.** If a mechanistic
finding suggests "amplify these neurons to reduce hallucination,"
that's a *proposal*, not an action. The proposal goes through
ForkScore acceptance (M+C+X axes per #000060 §7) with the result
logged as a separate `MechanisticWitnessRoot` and explicit policy
change.
4. **Feature labels never become semantic proof.** A neuron labeled
"refusal_cluster" in Neuronpedia is a hypothesis about its function,
not a guarantee. `MechanisticWitnessRoot` records the LABEL provider
+ version + evidence; the substrate treats labels as fallible
metadata, not as ground truth.
## Artifact — `MechanisticWitnessRoot`
Following the v9.8 content-addressing discipline, a witness is a Merkle
root over the following preimages (TLV-encoded, hash-chained):
```
MechanisticWitnessRoot = hash_leaf(
model_config_root # model name + version + weights hash
| activation_capture_policy_root # which layers, which sites, what dtype
| contrastive_prompt_set_root # the prompt pair the witness used
| feature_or_neuron_set_root # the discovered features/neurons
| intervention_result_root # ablation/amplification + delta
| behavioral_delta_root # measured output change
| safety_policy_root # what governance bucket this lives under
)
```
Two writers running the same prompt set against the same model produce
identical `MechanisticWitnessRoot`. Witnesses can be replayed
(reproducibility), audited (governance), and gossiped (mesh) without
trusting the producer — the same content-addressing discipline that
already covers documents, chunks, and providence cache.
## Restrictions in scope of this ticket
This ticket is a **scaffold only**, like #000057. No code lands until:
- An actual mechanistic finding is sitting in a fixture that arborist
could use as a SelfModel input or benchmark fixture. (Without a real
use case, this is speculative tooling.)
- The four guardrails above are formally restated in CLAUDE.md (as
rules, not just doc).
- An #000060 H-ABCDEFG-M+C+X harness exists to gate any promotion.
Open before any of those land: this is a *placeholder* in the design
log so the architectural framing is captured. Don't proliferate
sub-tickets, don't write a `arborist/witness/` directory.
## Explicitly NOT in scope
- Replacing the verifier or falsifier layers. Mechanistic Witness is
**additive diagnostic**, never substitutive.
- Direct integration with `providence_cache` or `audit_events` for
output rows. Witness data lives in its own table (when implemented)
with explicit hard-vs-soft separation per CLAUDE.md.
- Adversarial robustness claims. The substrate does NOT make claims
like "this model is safe because the refusal cluster fires" — that
would itself be a mechanistic-as-proof regression.
- LLM-as-judge over witness output. Witness analysis is deterministic
+ transparent — labels can come from external tools (Neuronpedia
cross-references, SAE labels with provenance), but interpretation
stays in code-level rules, not in a model loop.
## Status
Scaffold. Not started. No code expected until the prerequisites land.
Captured 2026-05-26 from Dav1d's de-novo review (§4.7, §9.1.F) so the
framing is in the design log before the first real use case forces an
implementation decision.
If the use case never materializes, this ticket stays scaffold
indefinitely — and that's the correct state. Mechanistic interpretability
is one of the most-promised + least-delivered areas in ML; arborist's
substrate is at risk of absorbing speculative tooling unless we keep
the bar at "real falsifier in hand + four guardrails locked in CLAUDE.md."