# Ticket #000022 — Adapter LossReport (PRD I9 analogue)

**Status:** closed · landed 2026-05-07
**Opened:** 2026-05-07
**Closed:** 2026-05-07 (amended pre-land per GPT-5.5 review:
schema columns added, policy split out of `governance_policy_hash`,
property test rewritten to be loss-mode-aware)
**Scope:** Design proposal for a typed loss ledger that every source
adapter emits during normalization, so an auditor can answer "what
content did we drop on the way from raw bytes to the prose we
verified?" today's adapters silently strip `<ref>` tags, image
wikilinks, HTML chrome, and whitespace runs without recording any
of it. Doc-only — no code in this ticket. Provenance gap analogous
to Donto's PRD I9 LossReport contract surfaced by the
2026-05-07 arborist-vs-donto comparison report.
**Audience:** fox + future blackops shifts + future adapter authors.
**Hard constraint:** adapters never block on loss reporting; LossReport
is additive metadata, not a gate. The Merkle commitment of `chunks.content`
(raw wikitext, raw HTML body) stays bit-identical to today — LossReport
captures *post-chunk* normalization losses, not anything that would
shift `document_root`. Cache-key dimensionality stays at 8.

---

## 1. Problem statement

Arborist's adapters perform two layers of lossy transformation between
raw bytes and prose the verifier sees:

| Layer | Adapter | What gets dropped | Where |
|-------|---------|--------------------|-------|
| Ingest-time | `WikipediaSqlDump` family | None at chunk level — raw wikitext is preserved verbatim in `chunks.content`. | `arborist/sources/wikipedia.py` |
| Ingest-time | `HtmlPageSource` | `<script>`, `<style>`, navigation chrome, `&nbsp;` runs, leading/trailing whitespace per line. Body text is normalized into `chunks.content`. | `arborist/sources/html_page.py:_normalize_text` |
| Pre-LLM | `arborist.wikitext.to_base()` | `<ref>...</ref>` tags, self-closing `<ref/>`, `[[File:...]]` / `[[Image:...]]` / `[[Category:...]]` wikilinks, template residues, header markers, whitespace runs, 3+ newlines collapsed to 2. | `arborist/wikitext.py` |

Today none of those drop events surface anywhere. The only audit
trail is a version pin (`canonicalization_version`,
`wikitext-base-v1`) hashed into `governance_policy_hash` — which
proves "the algorithm version was X" but not "X dropped these N
bytes from this particular chunk."

### 1.1 Concrete failure cases

**Case A — phantom omission.** A Wikipedia article cites a key claim
in a `<ref>` tag. `to_base()` drops the entire `<ref>` block. The
verifier never sees the citation. An auditor asking "did the LLM
have access to this citation?" has no record either way.

**Case B — drift after upstream change.** A new wikitext-base version
bumps the algorithm. The bump invalidates all prior cache_keys (good)
but doesn't tell an auditor what *changed* about the prose — only
that the version is different. To diff, you have to re-run both
versions on the same chunk and compare outputs.

**Case C — HTML chrome poisoning.** A scraped HTML page has a
navigation banner ("Most read articles: ...") that survives
`_normalize_text` and ends up as a leading prose chunk. The
verifier matches a "fact" against banner text. No audit record
distinguishes prose-from-article from prose-from-chrome.

### 1.2 Donto comparison

Donto's PRD invariant I9 mandates that every adapter produce a
typed LossReport documenting what was dropped, transformed, or
quarantined. Arborist's posture is currently "adapter authors
self-discipline, version pin in hash" — strong on identity, blank
on inspectability.

This is the same gap shape as #000001 (retrieval-keywords audit
gap): the *map* is committed (canonicalization_version) but the
*territory dropped* (the bytes that didn't survive normalization)
is not.

---

## 2. Design choices

### 2.1 Option A — sidecar table, no proof-path entry (RECOMMENDED)

New table `adapter_loss_reports`, one row per (chunk, normalization-stage,
loss-kind) tuple emitted at ingest and again at pre-LLM normalization.
Sidecar — never enters `cache_key`, never extends `audit_events`.
Operator queries it, verifier doesn't.

```sql
CREATE TABLE adapter_loss_reports (
    chunk_id                  INTEGER NOT NULL,
    document_root             BLOB    NOT NULL,
    stage                     TEXT    NOT NULL,  -- 'ingest' | 'wikitext_base' | 'html_normalize'
    canonicalization_version  TEXT    NOT NULL,  -- pins the algorithm (e.g. 'wikitext-base-v1')
    loss_kind                 TEXT    NOT NULL,  -- 'ref_tag' | 'file_link' | 'category_link' |
                                                 -- 'template_residue' | 'whitespace_run' |
                                                 -- 'html_chrome' | 'script_block' | ...
    loss_mode                 TEXT    NOT NULL,  -- 'pure_drop' | 'transform' | 'quarantine' |
                                                 -- 'normalize' (see §2.1.1 for semantics)
    bytes_dropped             INTEGER NOT NULL,  -- post-canonicalization byte count removed
                                                 -- (only meaningful for pure_drop / quarantine;
                                                 -- transform/normalize record byte delta but
                                                 -- the property test excludes them)
    occurrence_count          INTEGER NOT NULL,  -- how many events of this kind in this chunk
    input_length_bytes        INTEGER,           -- chunk input byte count at this stage
    output_length_bytes       INTEGER,           -- chunk output byte count at this stage
    sample_excerpt            TEXT,              -- truncated representative excerpt (PII risk;
                                                 -- truncate at 200 bytes max, NULL when
                                                 -- loss_report_excerpts=False)
    sample_hash               BLOB,              -- sha256(canonical(sample_bytes)) — populated
                                                 -- even in PII-safe mode so loss-signature
                                                 -- recurrence stays detectable without storing
                                                 -- raw bytes
    adapter_name              TEXT,              -- e.g. 'WikipediaSqlDump', 'HtmlPageSource'
    adapter_version           TEXT,              -- adapter package version (independent of
                                                 -- canonicalization_version — adapter taxonomy
                                                 -- expands faster than canonicalizer versions)
    loss_report_policy_hash   BLOB    NOT NULL,  -- pins the loss-reporting policy (see §3.7);
                                                 -- separate from governance_policy_hash so
                                                 -- toggling reporting does NOT invalidate QA
                                                 -- cache_keys
    created_at                TEXT    NOT NULL,
    PRIMARY KEY (chunk_id, stage, canonicalization_version, loss_kind)
);
CREATE INDEX idx_adapter_loss_doc   ON adapter_loss_reports(document_root, stage);
CREATE INDEX idx_adapter_loss_kind  ON adapter_loss_reports(loss_kind, stage);
CREATE INDEX idx_adapter_loss_chunk ON adapter_loss_reports(chunk_id, stage);
```

#### 2.1.1 `loss_mode` semantics

A loss event is one of four shapes. The shape determines whether
`bytes_dropped` participates in the property test (§3.8):

| `loss_mode`  | Meaning                                                        | Counts toward bytes-dropped sanity check? |
|--------------|----------------------------------------------------------------|-------------------------------------------|
| `pure_drop`  | Content removed entirely. `<ref>...</ref>` blocks, script/style blocks, `[[Category:...]]` links. | Yes — `sum(bytes_dropped) ≤ input_length_bytes`. |
| `transform`  | Content rewritten to different bytes. `[[Velociraptor]]` → `Velociraptor` drops markup but keeps the link text. | No — bytes-dropped here is a delta, not a deletion. |
| `quarantine` | Content preserved elsewhere (e.g., a sidecar) but withheld from prose. Reserved for future adapters; no v0 emitter. | Yes — same shape as `pure_drop` for accounting. |
| `normalize`  | Semantically low-content formatting normalization. Whitespace runs, newline collapse, `&nbsp;` runs. | No — emitted for visibility, not for byte-conservation accounting. |

Free-string `loss_kind` keeps the per-adapter naming flexibility from
the original §4 out-of-scope list; `loss_mode` is the closed enum that
keeps the property test sound across adapters.

**Pros:**
- Direct SQL queryability — "show me every chunk where wikitext-base
  dropped a `<ref>` tag" is one statement.
- Additive migration; no existing data touched.
- Fully optional — adapters that don't emit a LossReport degrade
  gracefully (same as Donto's "sidecar absence degrades shape/rule/
  cert calls only" stance).
- Honors arborist's "soft signals are sidecars" invariant — loss
  data is descriptive, never prescriptive.

**Cons:**
- Storage cost. Wikipedia ingest at 6 GB shard scale: rough
  back-of-envelope at 5 loss events/chunk * ~120 bytes/row (post
  schema additions in §2.1.1) gives ~2-3% storage tax. Within
  the same order as `concept_relations`'s 1.6%; not a blocker.
- Free-string `loss_kind` plus heuristic kinds (`html_chrome`)
  mean two adapters could emit the same dropped bytes under
  different names. Mitigated by §4 promotion-to-enum after three
  adapters; loss data is descriptive, not prescriptive, so
  inconsistent naming is a triage cost not a correctness cost.

### 2.2 Option B — Merkle-bound `loss_root` per chunk

Hash each chunk's loss-report list into a `loss_root`, fold into
`document_root` derivation alongside chunk content.

**Pros:**
- Cryptographically committed; two peers that ingest the same
  source compute the same `loss_root`.
- Loss becomes proof-path data, not sidecar.

**Cons:**
- Forces every adapter to emit a LossReport — no graceful degradation.
- Bumps `document_root` for every existing chunk on next ingest →
  full re-merkle of every shard. 5-8 GB old + 2 GB cur = days of
  rebuild for a feature that doesn't move the verifier needle.
- Couples loss reporting to identity. A bug in a single adapter's
  loss accounting now invalidates document identity, not just an
  auxiliary report.
- Violates "soft signals stay out of the proof path" — loss-counting
  heuristics (what counts as a "template residue"? what's a
  "whitespace run"?) are exactly the kind of soft signal that
  shouldn't decide cryptographic identity.

### 2.3 Option C — append to `audit_events`

Emit a `loss_report_emitted` audit event per adapter run.

**Pros:**
- Loss data inherits the audit chain's tamper-evidence.
- Existing audit machinery; no new table.

**Cons:**
- Audit chain is currently ~100 events per shard's lifecycle. A
  Wikipedia ingest with N chunks * M loss kinds = N*M new events.
  Shard-scale chain bloat — `chain-check-shards` slows linearly.
- Conflates state-changing operations (ingest, falsify, burn) with
  descriptive metadata (loss reports). The audit chain semantics
  blur.
- `governance_policy_hash` already pins the canonicalization
  version. Re-emitting per-chunk loss reports as state-change
  events overstates what loss reporting *is*.

### 2.4 Recommendation

**Option A.** A sidecar table mirrors the verifier's binary-vs-soft-signals
discipline: hard hashes commit identity (chunk content, document_root),
soft sidecars (loss reports, deflection diagnostics, title-relevance
diagnostics) describe properties without entering the proof path.
Same pattern as `arborist.qa.inspect`. Storage tax matches
`concept_relations` (~1.6%). Cache-key dimensionality stays at 8.

---

## 3. Implementation sketch (when scheduled)

1. **`arborist/store.py`** — schema migration adds `adapter_loss_reports`
   table + indexes. Bump schema-version-suffix only (e.g., `v9.8.1`)
   if the migration touches existing tables; pure add = no version
   bump per the existing additive-column policy.

2. **`arborist/sources/loss_report.py`** (new) — pure data layer:

   ```python
   @dataclass(frozen=True)
   class LossEvent:
       stage: str
       canonicalization_version: str
       loss_kind: str
       loss_mode: str  # 'pure_drop' | 'transform' | 'quarantine' | 'normalize'
       bytes_dropped: int
       occurrence_count: int
       input_length_bytes: int | None
       output_length_bytes: int | None
       sample_excerpt: str | None  # truncated to 200 bytes; None when
                                   # policy["loss_report_excerpts"] is False
       sample_hash: bytes | None   # sha256 of canonical sample bytes;
                                   # populated even in PII-safe mode
       adapter_name: str | None
       adapter_version: str | None
   ```

   Plus `record_losses(conn, chunk_id, document_root, events,
   loss_report_policy_hash)` — batched INSERT, idempotent on the
   `(chunk_id, stage, canonicalization_version, loss_kind)` primary
   key.

3. **`arborist/wikitext.py`** — `to_base()` gains an optional
   `loss_collector` parameter. When provided, the existing drop
   passes (`_drop_ref_tags`, `_drop_namespace_wikilinks`, the
   `strip_code` step, the whitespace pass) accumulate `LossEvent`s
   into the collector. When `None`, behavior is identical to today
   (the LLM-call path uses `None`; the ingest path passes a
   collector). Default `None` keeps every existing test stable.

4. **`arborist/sources/html_page.py`** — `_normalize_text` and
   `parse_html` likewise accept an optional collector; emit
   `loss_kind='html_chrome'`, `'script_block'`, `'whitespace_run'`,
   etc.

5. **`arborist/ingest.py`** — at the chunk-emission site, if
   `policy.get("loss_report_enabled", True)`, instantiate a
   collector per chunk and persist via `record_losses`.

6. **`arborist/cli.py`** — new `arborist losses --document-root X`
   sub-command (read-only, mirrors the existing
   `arborist inspect` shape) renders the loss table for a given
   document or chunk.

7. **Policy fields — separate `loss_report_policy_hash`, NOT folded
   into `governance_policy_hash`:**
   - `loss_report_enabled` (bool, default `True`) — master switch.
   - `loss_report_excerpts` (bool, default `True`) — when False,
     `sample_excerpt` is always NULL (PII-paranoid mode);
     `sample_hash` stays populated.
   - `loss_report_max_excerpt_bytes` (int, default `200`) — hard cap.

   These three fields canonicalize-then-hash into
   `loss_report_policy_hash`, stored as a column on
   `adapter_loss_reports` (per §2.1). They do **not** participate
   in the QA-side `cache_key` because LossReport is sidecar
   metadata that does not change normalized prose, retrieval,
   prompt, verifier behavior, or answer text. Toggling
   `loss_report_enabled` from True→False (or back) MUST NOT
   invalidate prior QA cache entries — that would convert a
   diagnostic toggle into answer-path churn, exactly the failure
   mode §2.2 rejects Option B for.

   Promotion path: if a future loss-reporting policy DOES change
   normalized prose (e.g., a `loss_quarantine_strict` mode that
   alters which bytes survive into the prose stream), it stops
   being a LossReport policy and becomes a canonicalization policy.
   At that point — and only at that point — it folds into
   `governance_policy_hash` and gets its own
   `canonicalization_version` bump. v0 reporting keeps prose
   bit-identical to today, so no fold is needed.

8. **Tests:**
   - Unit: `to_base()` with collector populates the expected
     `LossEvent` set on a fixture wikitext blob with `<ref>`,
     `[[File:]]`, `[[Category:]]`, and template residues. Each
     event carries the right `loss_mode`.
   - Unit: `to_base()` without collector is bit-identical to the
     current implementation (regression guard).
   - Unit: `record_losses` is idempotent on duplicate emit.
   - Unit: `sample_excerpt` truncates at
     `loss_report_max_excerpt_bytes`; `sample_hash` stays
     populated when `loss_report_excerpts=False`.
   - Unit: toggling `loss_report_enabled` from True→False on the
     same shard does NOT change any `cache_key` — confirms the
     hash split (§3.7).
   - Integration: ingest a small Wikipedia fixture and verify
     `adapter_loss_reports` rows match expected loss kinds and
     modes per chunk.
   - Property (loss-mode-aware byte conservation):
     - For each `(chunk_id, stage)`:
       `sum(bytes_dropped) WHERE loss_mode IN ('pure_drop','quarantine')
        <= input_length_bytes`.
     - For each `(chunk_id, stage)`:
       `output_length_bytes <= input_length_bytes` for every adapter
       whose `canonicalization_version` advertises pure-strip semantics
       (wikitext-base-v1 and html-normalize-v1 both qualify).
     - `loss_mode='transform'` and `loss_mode='normalize'` rows are
       excluded from the sum because their `bytes_dropped` records a
       byte delta, not a deletion. The original ticket's
       `sum(bytes_dropped) <= len(raw) - len(prose)` test fails on
       transforms like `[[Velociraptor]] → Velociraptor` (markup
       drops, link text survives) and is replaced by the above.

9. **No bumps to:**
   - `cache_key` (8 dimensions unchanged; `governance_policy_hash`
     unchanged because LossReport policy lives in its own
     `loss_report_policy_hash` per §3.7).
   - `document_root` (chunks.content stays raw).
   - `audit_events` chain (sidecar, not state-changing).
   - `run_dag_root` (verifier never sees loss reports).

---

## 4. Out of scope

- **Generalizing `loss_kind` to a typed enum across adapters.**
  Adapter authors should be free to mint new loss kinds (ad-hoc
  string identity) until we see naming patterns settle. Promote
  to enum after at least three adapters are in tree.
- **Cross-shard loss aggregation.** Per-shard tables only; no
  cross-shard analytics in this ticket.
- **PII-aware sample-excerpt redaction beyond the
  `loss_report_excerpts=False` master switch.** Per-loss-kind
  redaction policies (e.g., always-redact for `html_chrome`,
  truncate-only for `template_residue`) earn their own ticket
  when an operator surfaces a real PII case.
- **Replaying loss reports during cache rehydrate.** The audit
  property "was this chunk's prose derived under canonicalization
  version V?" is already answered by `governance_policy_hash`;
  loss reports add granularity, not new identity.
- **Donto's full PRD I9 surface** — `LossReport` as a typed
  Rust object with serde-bound JSON schema, pgrx integration,
  Lean 4 kernel-checked emptiness theorem. We're shipping the
  audit primitive (a queryable sidecar table); shape it like
  arborist's other sidecars, not like Donto's.

---

## 5. Status

Open · awaiting go/no-go. Surfaced by the 2026-05-07
arborist-vs-donto comparison report (`/home/fox/Downloads/arborist_vs_donto.pdf`).
Pass 5 of that report names "surface area asymmetry" as the
defining cost difference — Donto adapters carry a `LossReport`
contract; arborist adapters carry "self-discipline plus version
pin." Closing that gap with a sidecar table preserves arborist's
"one engineer can read it end-to-end" property while giving an
auditor a real answer to "what did the adapter drop?"

### 5.1 Amendment trail

**2026-05-07 (GPT-5.5 review pass).** Original draft contradicted
its own §2.2 "soft signals stay out of the proof path" reasoning by
folding `loss_report_enabled` and `loss_report_excerpts` into
`governance_policy_hash` in §3.7 — a diagnostic toggle that would
have invalidated every prior QA cache entry on landing. Amendments:

1. **§3.7** — split policy into a separate `loss_report_policy_hash`
   stored on `adapter_loss_reports`. QA `cache_key` stays at 8
   dimensions; `governance_policy_hash` unchanged. Promotion path
   spelled out: if a future loss policy ever changes normalized
   prose, it stops being a LossReport policy and gets a
   `canonicalization_version` bump.
2. **§2.1** — schema gains `loss_mode`, `input_length_bytes`,
   `output_length_bytes`, `sample_hash`, `adapter_name`,
   `adapter_version`, `loss_report_policy_hash`, `created_at`.
   PK extends to `(chunk_id, stage, canonicalization_version,
   loss_kind)` so multiple normalizer versions can coexist for
   the same chunk during canonicalization migrations.
3. **§2.1.1 (new)** — `loss_mode ∈ {pure_drop, transform,
   quarantine, normalize}` semantics defined with a participation
   table for the byte-conservation property test.
4. **§3.8** — replaced the naive
   `sum(bytes_dropped) <= len(raw) - len(prose)` property test
   with a loss-mode-aware version that excludes `transform` and
   `normalize` rows (their `bytes_dropped` records a byte delta,
   not a deletion — `[[Velociraptor]] → Velociraptor` is the
   canonical breakage).
5. **§2.1 cons** — struck the "anyone with shard write access could
   update the table without breaking any merkle invariant" line.
   That's the design intent of a sidecar, not a downside. Replaced
   with the real con: free-string `loss_kind` plus heuristic kinds
   (`html_chrome`) admit naming drift across adapters.

Out-of-scope tightening (rejected for v0):

- LossReport as a generalized π* loss ledger — speculative bridge
  to #000015. Defer until #000022 has shipped and at least three
  adapters use it.
- Loss-aware UI warnings on `arborist query` output — blurs the
  soft/hard line we just defended. CLI `arborist losses` covers
  the auditor case.
- Per-occurrence `adapter_loss_occurrences` table — defer per
  the existing aggregation-first discipline.

---

## 6. Source

- `/home/fox/Downloads/arborist_vs_donto.pdf`,
  §6.5 "Pass 5 — Stack and surface area," "Surface area asymmetry"
  bullet:

  > Anyone could write Arborist's adapter for a new corpus in a
  > weekend. Donto's adapter contract requires a LossReport type
  > (PRD I9), per-anchor-kind validators, idempotency guarantees,
  > and quarantine-instead-of-fail semantics — this is a different
  > scale of investment.

- `/home/fox/Downloads/arborist_vs_donto.pdf`,
  §7.1 "Where they converge," "Audit-preserving removal" bullet
  — establishes that arborist's `falsify` and Donto's `donto_retract`
  share the same shape, which makes the LossReport gap genuinely
  one-sided rather than a deliberate identity choice.

- Donto PRD invariant I9 (cited in PDF §4.2) — "adapters must
  report information loss." The shape this ticket adopts is
  arborist's read of that invariant under the
  "soft-signals-as-sidecars" discipline.
