#000061: fold 3 gaps from Dav1d review (manifest/latest, license_class, cold_pending)

Dav1d's reviews of #000061 (Response A + Response B/FINAL in
~/Downloads, 2026-05-26) flagged a long list of items — most already
shipped in the SPV-split work. Three were genuine gaps worth folding
into #000061 before close:

Gap 1: manifest/latest pointer for new-peer discovery.

  A fresh peer doing `cold list` got a list of metadata-pack hashes
  but no obvious "which one is current for shard X." Added
  get_latest_pointer + update_latest_pointer to the backend ABC.
  push_pack writes manifest/latest.json on every successful metadata
  pack push (read-modify-write keyed by snapshot_root). Mutable
  pointer; content addressing of the packs themselves preserves the
  trust root. Last-writer-wins on contention.

Gap 2: license_class field + producer-side refuse for public buckets.

  Maps documents.source_type to a license bucket (wikipedia_cur /
  textbook_tex → public_redistributable; html / grok / vcs → unknown;
  anything else → unknown). Strictness order: public < unknown <
  private. compute_shard_license_class() walks DISTINCT source_type
  in documents. push_pack now refuses to upload if the shard's
  strictest license is more restrictive than the operator's
  allow_license_class (default: public_redistributable). The
  metadata pack's manifest carries _license_class so consumers /
  auditors can see the producer's classification without inspecting
  source documents. ValueError on refusal — the bucket ACL is the
  operator's call, but arborist refuses to participate in a
  licensing/membership leak unless explicitly opted in.

Gap 3: cold_pending table for resumable uploads.

  Killed mid-upload, push_pack left orphan multi-GB tempfiles in
  /tmp with no DB trace. Added schema:

    CREATE TABLE cold_pending (
        tempfile_path TEXT PRIMARY KEY,
        pack_hash TEXT NOT NULL,
        kind TEXT NOT NULL,
        backend_endpoint TEXT NOT NULL,
        backend_bucket TEXT NOT NULL,
        object_key TEXT NOT NULL,
        started_at INTEGER NOT NULL,
        state TEXT NOT NULL DEFAULT 'pending'
    );

  push_pack INSERTs a row before each upload + DELETEs on success.
  A killed process leaves the row pointing at the orphan tempfile;
  a recovery script (future) reads cold_pending, checks bucket for
  the object, either deletes the row + tempfile (success was just
  unreported) or re-uploads from the tempfile if it still exists.
  Matches the same pattern as the audit chain — explicit state
  rows beat inferring from chunks.content IS NULL.

Sibling tickets opened for the larger items the reviews flagged
(scaffold-only, no code; opening them captures the design in the
log without proliferating, per CLAUDE.md):

- #000063 Cold-object private-ciphertext mode (mesh-keyed object
  keys for non-public corpora on public-read buckets). Needs mesh
  group-key ABI + real non-public corpus before code.

- #000064 Cold-object operations toolkit (verify / diff / doctor /
  repair-fts / gc-plan CLI + expanded audit-event taxonomy).
  Bundled so the audit-event vocabulary gets one design pass.

5 new tests:
  test_gap2_license_gate_refuses_unknown_class_to_public_bucket
  test_gap2_license_class_in_metadata_manifest
  test_gap1_latest_pointer_resolves_metadata_pack_per_snapshot
  test_gap3_cold_pending_clears_on_successful_upload
  test_gap3_cold_pending_records_inflight_upload

26 cold-object + 7 evict tests pass (33/33 green incl. boto3 wire).

Next ID bumped to 000065.

Live v3 SPV corpus run (bmq47x6t3) completed cleanly during this work.
Will report sizing + memory profile in the next message.
This commit is contained in:
russell@unturf.com 2026-05-26 10:50:57 -04:00
parent eba08beb61
commit 576cb0eeaf
No known key found for this signature in database
8 changed files with 667 additions and 4 deletions

View file

@ -111,6 +111,8 @@ Newest first. Update on every open/close.
| ID | Title | Status | Opened | Directive |
|----------|------------------------------------------------|-----------------------|------------|-----------|
| #000064 | Cold-object operations toolkit (verify/diff/doctor/repair-fts/gc-plan + audit taxonomy) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §11/§12/§14). Operator-facing observability + repair tools on top of #000061: `cold verify` (sample/full integrity check), `cold diff` (local vs remote manifest), `cold doctor` (one-shot health: connectivity / credentials / manifest age / missing-object count / tamper sample / audit-chain integrity), `cold repair-fts` (rebuild FTS5 from chunks.content), `cold gc-plan` (orphan bucket objects, read-only by default — destructive only with `--apply` + confirm). Plus expanded audit-event taxonomy: per-PUT/HEAD/GET success/failure events, manifest-pointer events, verify/doctor/gc events. All read-mostly; destructive ops require `--apply`. Bundled so the audit-taxonomy gets one design pass instead of five-way drift. Sequence: doctor → verify → diff → repair-fts → gc-plan. No code until #000061 closes. | 2026-05-26 | — |
| #000063 | Cold-object private-ciphertext mode (mesh-keyed object keys) | **scaffold-only · awaiting go/no-go** (2026-05-26; from Dav1d #000061 review §9 / response A §13.3). Adds private mode to #000061 cold-object format so chunk bodies + manifest can be uploaded to public-read bucket without leaking corpus membership. Two strategies: (A) deterministic `object_key = HMAC(group_key, leaf_hash)` + AEAD-encrypted body — supports lookup-by-leaf-hash given the key; (B) random-key ciphertext + encrypted private manifest — stronger membership hiding, needs manifest fetch first. Strategy A default; B opt-in. Group key from existing `arborist/mesh/crypto.py`; pack manifest carries `epoch_id` for rotation. Verifier path unchanged: consumer decrypts, then `hash_leaf(plaintext) == leaf_hash` as in public mode. No code until (1) a real non-public corpus needs cold-object shipping, (2) mesh group-key ABI is stable enough to reference, (3) threat-model split between A vs B is settled by real adversary. | 2026-05-26 | — |
| #000062 | Mechanistic Witness: governed diagnostic sidecar (CNA/SAE/Neuronpedia) | **scaffold-only · awaiting go/no-go** (2026-05-26; Dav1d de-novo review §4.7 / §9.1.F). Specification of a mechanistic-interpretability sidecar that produces a content-addressed `MechanisticWitnessRoot` over (model, prompts, capture policy, neurons/features, intervention deltas), used as a **diagnostic input** to SelfModel (#000014/#000017) + benchmark-fixture generation. **Hard constraint:** soft signals never enter the hard proof path — `audit_mode` does NOT move based on witness output, `providence_cache` is untouched, `governance_policy_hash` only moves via explicit ForkScore ACCEPT with M+C+X axes passing (#000060 §7). Four guardrails (diagnostic-only by default · sandbox intervention only · no production steering without governance · feature labels never semantic proof). Witness root TLV-encodes `model_config_root | activation_capture_policy_root | contrastive_prompt_set_root | feature_or_neuron_set_root | intervention_result_root | behavioral_delta_root | safety_policy_root`. Scaffold only — no code until a real falsifier-in-hand use case exists + the four guardrails are restated in CLAUDE.md as rules + #000060 H-ABCDEFG-M+C+X harness exists to gate promotion. Captured to keep mechanistic-interp tooling out of the substrate unless and until it earns its place; the dual-use risk (Pan et al. 2025 CNA: 0.1% MLP ablation breaks refusal in 72B models) makes the governance-first framing load-bearing. | 2026-05-26 | — |
| #000061 | Cold-pack distribution tier (boto3 S3-compat, DO Spaces + DVD-R targets) | **in progress** — opened 2026-05-25 (fox: "implement it now … target digital ocean first as a test"; later "I wanted a way to hydrate using tarballs (the core and important data) for bringing new machines up"). Tarball-only distribution mechanism — bucket holds `tar.zst` packs keyed by `hash_leaf(manifest)`, no individual-chunk blobs. New peers hydrate by downloading packs from the bucket's CDN edge (~4 HTTPS GETs for the current ~14.1M-chunk corpus, packs filled to 4.4 GB compressed each via streaming zstd, vs ~14M for individual blobs). Same artifact ≤4.4 GB safe-fit (~6.5 % buffer below DVD-R's 4.7 GB marketing capacity, accommodating ISO9660 overhead + media variance + drive-edge refusal) burns directly to physical media via `--local-dir` + `growisofs`. Packs are *delayed* snapshots: each pack pins the corpus `snapshot_root` it covers in audit + body, so falsifications between repacks produce new pack_hashes and stale packs stay in the bucket until explicit GC (future ticket). Packs include cores AND surfaces (full-corpus hydration). Default selection covers every hot chunk with local content in the shard. Multi-pack splitting via `stream_packs` (streaming zstd, FLUSH_BLOCK peek of compressed buffer after each chunk, cut at cap) fills each disc to ~4.4 GB compressed instead of leaving ~50% empty. One backend class (`S3CompatibleBackend` via boto3 + `endpoint_url`) covers AWS S3, DO Spaces, GCS S3-interop, R2, B2, MinIO. CDN public-read makes packs accessible to anyone; hash binding via in-tar `leaf_hash` member names makes hostile-bucket scenarios safe. Optional dep `[object-store]` = boto3>=1.34. Voyeur: credentials via standard `AWS_ACCESS_KEY_ID`/`_SECRET_ACCESS_KEY` (env or `~/.aws/credentials`), never printed; only endpoint URL + bucket name surface in logs. Initial individual-blob path (per-chunk S3 objects) was scoped+landed then **deleted same day** (fox: "what ever was blobs? I wanted a way to hydrate using tarballs"); the five-step deletion record lives inline in the doc — we'd added 14M-object storage and ~$70/hydrate request cost for a workflow that needed neither. Sizing math for current shards: ~4 packs total (17.2 GB compressed ÷ 4.4 GB compressed per pack), ~17 GB bucket storage, ~$0.34/mo DO Spaces. | 2026-05-25 | — |
| #000060 | H-ABCDEFG same-model substrate-delta harness (+ jaggedness tensor + curvature) | open · awaiting go/no-go (2026-05-20; from Dav1dPrometheus *Protocol-Layer AGI* working report §26/§50/§82-84). The report's "decisive proof": run the SAME base model substrate-OFF vs substrate-ON over long-horizon/adversarial/non-jagged batteries, report the delta. Two new metrics: jaggedness tensor `J_norm` (§73 — variance across nearby variants, normalized by difficulty) + discrete performance curvature `κ_t` (§5.2, with the honest no-global-convexity bound, Erratum 5). A-vs-C spine (B optional, D=mesh OUT → #000012/#000016). Curvature-aware ForkScore extension folds into **#000012** (NOT a new ticket — reserved `iota`/`kappa` weight slots already exist). Budget: control arms = Hermes/Qwen, never Opus without go; heavy passes on GPU box. Held-out/mechanism-agnostic variants required so ABCDEFG doesn't self-validate. **2026-05-26 scope refinement (Dav1d review §4 / §7):** split the harness output into three axes — **H-ABCDEFG-M** (mechanism tests: does the substrate work?), **H-ABCDEFG-C** (capability tests: does the substrate improve task performance?), **H-ABCDEFG-X** (external adversarial: does it generalize outside author-designed fixtures?). The split prevents "self-validating benchmark theology" — a harness that reports only M+C with no X can pass while still failing on held-out adversarial generalization. Fold into the harness design before any code lands; doesn't change the bench-row schema (`carrier`/`domain`/`pi_star_ref`) but does change what "ACCEPT" requires (must clear all three axes). | 2026-05-20 | — |
@ -176,4 +178,4 @@ Newest first. Update on every open/close.
## Next ID
`000063`
`000065`

View file

@ -0,0 +1,120 @@
# Ticket #000063 — Cold-object private-ciphertext mode (mesh-keyed)
**Status:** open · awaiting go/no-go · scaffold-only
**Opened:** 2026-05-26
**Scope:** add a private mode to the #000061 cold-object pack format so
chunk bodies + manifest can be uploaded to a public-read bucket
without leaking corpus membership or document content. Object keys
derive from a mesh group key via HMAC, not from plaintext leaf_hash;
bodies are ciphertext, manifest is encrypted or access-controlled.
**Audience:** dav1d (cold-object security model extension; spec is the
load-bearing artifact, not code).
**Hard constraint:** the integrity invariant from #000061 stays —
verifier recovers plaintext bytes, hashes via `hash_leaf`, compares
to local `chunks.leaf_hash`. Encryption sits between disk and bucket;
inside the verifier path, behavior is identical to public mode.
## Problem
#000061 ships `public_plaintext` mode: object key = `leaf_hash`, body =
raw UTF-8 plaintext. Safe for public-redistributable corpora (Wikipedia,
PG textbooks). **Not safe** when the bucket has any chunk whose source
is private, licensed, or membership-sensitive:
- `object_key = sha256(0x00 ‖ plaintext)` is a membership oracle. An
attacker who guesses a chunk's plaintext can probe the bucket and
learn whether that chunk is in the corpus.
- Plaintext bodies on a public-read bucket are a redistribution of
every chunk to the world.
#000061 v3 sidesteps this by refusing to push private content to a
public bucket (#000061 Gap 2 — `license_class` policy gate). This
ticket adds the *other* path: ship private content under proper
ciphertext, so the bucket can stay public-read without leaking.
## Design (spec only — no code yet)
Two keying strategies:
**Strategy A — deterministic-key ciphertext** (preferred for indexing):
```
plaintext_leaf_hash = chunks.leaf_hash # local DB value
object_key = "blobs/" + HMAC(group_key, plaintext_leaf_hash)
ciphertext_body = AEAD-encrypt(group_key, plaintext_bytes,
aad = plaintext_leaf_hash)
```
- Deterministic key allows lookup-by-leaf_hash without a separate
manifest fetch (given the group key).
- AAD binds the ciphertext to the plaintext identity.
- Bucket can be public-read; without `group_key`, neither key nor body
reveals membership.
**Strategy B — random-key ciphertext + manifest map** (stronger
membership hiding):
```
ciphertext_body = AEAD-encrypt(group_key, plaintext_bytes)
ciphertext_hash = sha256(ciphertext_bytes)
object_key = "blobs/" + ciphertext_hash
private_manifest binds plaintext_leaf_hash -> object_key
private_manifest itself encrypted with group_key
```
- Membership requires both the encrypted manifest AND the group key.
- Lookup requires manifest fetch; one extra round-trip.
- Defeats prefix-enumeration of `blobs/`.
Strategy A is the default; Strategy B is opt-in for the most sensitive
corpora.
### Mesh integration
The group key lives where it already lives for arborist:
`arborist/mesh/crypto.py` (group-key state machine, epoch rotation,
member wrap/unwrap). This ticket plumbs that key into the pack flow:
- Producer: `push_pack(..., mode='private', mesh_conn=...)` reads the
current epoch's group key, uses it for HMAC + AEAD.
- Consumer: `hydrate_from_metadata_pack(..., mesh_conn=...)` reads the
same key, decrypts/verifies.
- Key rotation: epoch advance means old packs use old key. Pack
manifest carries `epoch_id` so consumers know which key to use.
### Verifier path unchanged
After decryption, the consumer holds plaintext bytes. From that point
on, the cold-object pull path is **byte-identical** to public mode:
`hash_leaf(plaintext) == leaf_hash`, restore into `chunks.content`,
populate FTS5, etc. Ciphertext does not enter `chunks.content`.
## Why scaffold-only
This ticket lands when:
1. There's a real corpus with non-public content that arborist needs
to ship via cold-object. (No use case today — fox's corpora are
Wikipedia + open textbooks.)
2. The mesh group-key ABI is stable enough to be referenced by pack
code. (Today the mesh state lives behind `arborist/mesh/` with its
own state machine.)
3. The threat-model split between Strategy A and Strategy B is settled
by a real adversary scenario, not a paper one.
Open as scaffold so the design is in the log; defer code until at
least #1 lands.
## Out of scope
- Server-side encryption (KMS / SSE-S3). That's bucket-level
encryption-at-rest, orthogonal to membership hiding. Mesh ciphertext
is the client-side path that survives a hostile bucket.
- Key rotation logistics beyond reading the current epoch_id from
`mesh_epochs`. Old packs stay decryptable with old keys (already a
mesh primitive).
- Public/private mixed-mode shards. A shard is one mode or the other.
## Status
Scaffold. No code expected until the prerequisites above land.

View file

@ -0,0 +1,160 @@
# Ticket #000064 — Cold-object operations toolkit (verify / diff / doctor / repair-fts / gc-plan + expanded audit taxonomy)
**Status:** open · awaiting go/no-go · scaffold-only
**Opened:** 2026-05-26
**Scope:** operator-facing observability + repair tools on top of the
#000061 cold-pack tier. Bundles five CLI subcommands and a fine-
grained audit event vocabulary into one ticket so each tool can be
designed against a shared event model rather than five-way drift.
**Audience:** operator (fox, or anyone running an arborist node with
cold-object storage). dav1d secondary — the audit taxonomy is the
thing that goes through the audit chain.
**Hard constraint:** every tool here is *read-mostly* by default.
Destructive operations (gc, repair) print proposed actions and exit
unless `--apply` is given. Cold-object packs are durable artifacts;
ops tools must never enter a state where they'd accidentally orphan
bytes or break replay.
## Problem
#000061 ships a producer (`cold pack`) and a consumer (`cold unpack`
via `hydrate_from_metadata_pack`). It doesn't ship operator tools:
- **No way to spot-check a remote pack** without fully unpacking it.
- **No way to diff local DB chunks against a remote manifest** to see
what would be lost if local disk failed.
- **No health check** that exercises the auth + connectivity +
manifest-age + missing-object signals in one command.
- **No way to rebuild FTS5** from local content if the index gets
corrupted (independent of cold-object pulls).
- **No GC plan** showing which bucket objects are no longer referenced
by any local manifest.
And the audit chain currently has one event per pull (`cold_pack_pulled`)
with everything stuffed into the body. Fine for the proof property
(audit chain stays intact); too coarse for operators trying to diagnose
"why did 3 of 4 shards complete but the 4th failed?"
## The five tools (spec only)
### `arborist cold verify [--sample N | --full]`
Spot-check object integrity without restoring. Default samples N
random objects from the bucket; `--full` checks every object. For
each: HEAD for size + content-length, GET body (or HEAD-only with
`--head-only`), recompute `hash_leaf(body).hex()`, compare to expected
`leaf_hash`. Audit `cold_verify_sample` with pass/fail counts.
### `arborist cold diff`
Compare local DB chunks against a remote `manifest/latest.json` (or
explicit `--metadata-pack <hash>`). Output four buckets:
- Local has, remote has (intersection)
- Local has, remote missing (would-be-lost-if-local-gone)
- Remote has, local missing (could-be-pulled)
- Hash disagreement between local and remote
JSON output for scripting; `--summary` for human.
### `arborist cold doctor`
One-shot health check. Runs in order:
- Backend connectivity (HEAD a known key)
- Credential validity (signed request returns 200)
- Manifest age (`manifest/latest.json` last-modified vs now)
- Object existence (sample N HEADs, count 404s)
- Tamper sample (N body GETs + hash-check, count mismatches)
- Audit chain integrity (`make chain-check-shards` equivalent)
- Disk space at temp dir + shard dir
- Memory available
Exit code reflects overall health. JSON output for monitoring scripts.
### `arborist cold repair-fts [--shard X]`
Rebuild FTS5 rows from `chunks.content` for every hot chunk. Useful
after an interrupted pull or a corrupted index. Idempotent —
`DELETE FROM chunks_fts; INSERT INTO chunks_fts (rowid, content) SELECT
chunk_id, content FROM chunks WHERE content IS NOT NULL`. Audits one
`cold_repair_fts` event.
### `arborist cold gc-plan`
List bucket objects that no manifest references. Read every manifest
in the bucket (latest pointer + every historical metadata pack
manifest), union their referenced `chunk_pack_hashes`. Diff against
the bucket's actual pack listing. Output candidates with sizes.
**Read-only by default** — actual deletion requires `--apply` + an
audit row + the operator typing the orphan count back to confirm.
## Expanded audit event taxonomy
Current: `cold_pack_pushed` + `cold_pack_pulled` (with `pack_kind`
distinguishing metadata vs chunks). Load-bearing for proof properties.
Add (operability, lower information density per event):
```
cold_object_put_started per upload, before bytes leave
cold_object_put_verified after HEAD confirms object exists
cold_object_evict_committed after local DB commit drops content
cold_object_put_failed backend rejected the PUT
cold_object_head_failed HEAD after PUT returned an error
cold_object_get_started per fetch
cold_object_missing bucket returned 404
cold_object_hash_mismatch hash check failed; never restored
cold_object_decode_error UTF-8 decode failed; never restored
cold_pack_uploaded pack body confirmed in bucket
cold_pack_unpacked pack body extracted to local
cold_pack_blob_hash_mismatch pack tar bytes don't match expected
cold_manifest_pointer_updated latest.json was rewritten
cold_manifest_conflict conditional write lost a race
cold_verify_sample N objects probed, K mismatches
cold_doctor_run health-check verdict
cold_repair_fts_complete FTS rebuild done, N rows
cold_gc_plan_computed N orphans identified
cold_gc_executed orphans deleted (only via --apply)
```
Audit body fields stay minimal — endpoint URL + bucket + object key +
result + error_class. Never credentials. Goes through the same
`append_audit` path as everything else.
## Why scaffold-only
Each of these tools is its own design problem:
- `verify` sample size needs a statistical justification.
- `diff` needs a clear answer to "what does 'missing' mean across
shards / manifests / packs?"
- `doctor` needs a defined exit-code vocabulary for monitoring
integration.
- `repair-fts` needs to handle running concurrently with `cold pack`
(it touches the same FTS5 index).
- `gc-plan` needs the manifest-history walk to be efficient (could be
many manifests over time).
Bundling so the audit-event taxonomy gets designed once instead of
five times. Sequence the implementations in this order: `doctor` first
(fastest, no destructive), `verify` second (sampling, read-only),
`diff` third (operator clarity), `repair-fts` fourth (touches FTS),
`gc-plan` last (destructive even in plan mode if `--apply` ever fires).
## Out of scope
- A web UI / dashboard. Output is JSON; integrate elsewhere.
- Cross-arborist-fleet aggregation (e.g., "doctor on all peers"). Each
peer runs its own toolkit.
- Continuous monitoring daemon. `make` target + cron is the operator
recipe.
## Status
Scaffold. No code expected until #000061 is fully closed and there's
a clear operator pain that justifies a specific tool first.