# Ticket #000064 — Cold-object operations toolkit (verify / diff / doctor / repair-fts / gc-plan + expanded audit taxonomy)

**Status:** open · awaiting go/no-go · scaffold-only
**Opened:** 2026-05-26
**Scope:** operator-facing observability + repair tools on top of the
  #000061 cold-pack tier. Bundles five CLI subcommands and a fine-
  grained audit event vocabulary into one ticket so each tool can be
  designed against a shared event model rather than five-way drift.
**Audience:** operator (fox, or anyone running an arborist node with
  cold-object storage). dav1d secondary — the audit taxonomy is the
  thing that goes through the audit chain.
**Hard constraint:** every tool here is *read-mostly* by default.
  Destructive operations (gc, repair) print proposed actions and exit
  unless `--apply` is given. Cold-object packs are durable artifacts;
  ops tools must never enter a state where they'd accidentally orphan
  bytes or break replay.

## Problem

#000061 ships a producer (`cold pack`) and a consumer (`cold unpack`
via `hydrate_from_metadata_pack`). It doesn't ship operator tools:

- **No way to spot-check a remote pack** without fully unpacking it.
- **No way to diff local DB chunks against a remote manifest** to see
  what would be lost if local disk failed.
- **No health check** that exercises the auth + connectivity +
  manifest-age + missing-object signals in one command.
- **No way to rebuild FTS5** from local content if the index gets
  corrupted (independent of cold-object pulls).
- **No GC plan** showing which bucket objects are no longer referenced
  by any local manifest.

And the audit chain currently has one event per pull (`cold_pack_pulled`)
with everything stuffed into the body. Fine for the proof property
(audit chain stays intact); too coarse for operators trying to diagnose
"why did 3 of 4 shards complete but the 4th failed?"

## The five tools (spec only)

### `arborist cold verify [--sample N | --full]`

Spot-check object integrity without restoring. Default samples N
random objects from the bucket; `--full` checks every object. For
each: HEAD for size + content-length, GET body (or HEAD-only with
`--head-only`), recompute `hash_leaf(body).hex()`, compare to expected
`leaf_hash`. Audit `cold_verify_sample` with pass/fail counts.

### `arborist cold diff`

Compare local DB chunks against a remote `manifest/latest.json` (or
explicit `--metadata-pack <hash>`). Output four buckets:

- Local has, remote has (intersection)
- Local has, remote missing (would-be-lost-if-local-gone)
- Remote has, local missing (could-be-pulled)
- Hash disagreement between local and remote

JSON output for scripting; `--summary` for human.

### `arborist cold doctor`

One-shot health check. Runs in order:

- Backend connectivity (HEAD a known key)
- Credential validity (signed request returns 200)
- Manifest age (`manifest/latest.json` last-modified vs now)
- Object existence (sample N HEADs, count 404s)
- Tamper sample (N body GETs + hash-check, count mismatches)
- Audit chain integrity (`make chain-check-shards` equivalent)
- Disk space at temp dir + shard dir
- Memory available

Exit code reflects overall health. JSON output for monitoring scripts.

### `arborist cold repair-fts [--shard X]`

Rebuild FTS5 rows from `chunks.content` for every hot chunk. Useful
after an interrupted pull or a corrupted index. Idempotent —
`DELETE FROM chunks_fts; INSERT INTO chunks_fts (rowid, content) SELECT
chunk_id, content FROM chunks WHERE content IS NOT NULL`. Audits one
`cold_repair_fts` event.

### `arborist cold gc-plan`

List bucket objects that no manifest references. Read every manifest
in the bucket (latest pointer + every historical metadata pack
manifest), union their referenced `chunk_pack_hashes`. Diff against
the bucket's actual pack listing. Output candidates with sizes.
**Read-only by default** — actual deletion requires `--apply` + an
audit row + the operator typing the orphan count back to confirm.

## Expanded audit event taxonomy

Current: `cold_pack_pushed` + `cold_pack_pulled` (with `pack_kind`
distinguishing metadata vs chunks). Load-bearing for proof properties.

Add (operability, lower information density per event):

```
cold_object_put_started         per upload, before bytes leave
cold_object_put_verified        after HEAD confirms object exists
cold_object_evict_committed     after local DB commit drops content
cold_object_put_failed          backend rejected the PUT
cold_object_head_failed         HEAD after PUT returned an error

cold_object_get_started         per fetch
cold_object_missing             bucket returned 404
cold_object_hash_mismatch       hash check failed; never restored
cold_object_decode_error        UTF-8 decode failed; never restored

cold_pack_uploaded              pack body confirmed in bucket
cold_pack_unpacked              pack body extracted to local
cold_pack_blob_hash_mismatch    pack tar bytes don't match expected

cold_manifest_pointer_updated   latest.json was rewritten
cold_manifest_conflict          conditional write lost a race

cold_verify_sample              N objects probed, K mismatches
cold_doctor_run                 health-check verdict
cold_repair_fts_complete        FTS rebuild done, N rows
cold_gc_plan_computed           N orphans identified
cold_gc_executed                orphans deleted (only via --apply)
```

Audit body fields stay minimal — endpoint URL + bucket + object key +
result + error_class. Never credentials. Goes through the same
`append_audit` path as everything else.

## Why scaffold-only

Each of these tools is its own design problem:

- `verify` sample size needs a statistical justification.
- `diff` needs a clear answer to "what does 'missing' mean across
  shards / manifests / packs?"
- `doctor` needs a defined exit-code vocabulary for monitoring
  integration.
- `repair-fts` needs to handle running concurrently with `cold pack`
  (it touches the same FTS5 index).
- `gc-plan` needs the manifest-history walk to be efficient (could be
  many manifests over time).

Bundling so the audit-event taxonomy gets designed once instead of
five times. Sequence the implementations in this order: `doctor` first
(fastest, no destructive), `verify` second (sampling, read-only),
`diff` third (operator clarity), `repair-fts` fourth (touches FTS),
`gc-plan` last (destructive even in plan mode if `--apply` ever fires).

## Out of scope

- A web UI / dashboard. Output is JSON; integrate elsewhere.
- Cross-arborist-fleet aggregation (e.g., "doctor on all peers"). Each
  peer runs its own toolkit.
- Continuous monitoring daemon. `make` target + cron is the operator
  recipe.

## Status

Scaffold. No code expected until #000061 is fully closed and there's
a clear operator pain that justifies a specific tool first.
