arborist/docs/tickets/ticket-000022-adapter-loss-report.md

Ticket #000022 — Adapter LossReport (PRD I9 analogue)

Status: open · awaiting go/no-go Opened: 2026-05-07 Scope: Design proposal for a typed loss ledger that every source adapter emits during normalization, so an auditor can answer "what content did we drop on the way from raw bytes to the prose we verified?" today's adapters silently strip <ref> tags, image wikilinks, HTML chrome, and whitespace runs without recording any of it. Doc-only — no code in this ticket. Provenance gap analogous to Donto's PRD I9 LossReport contract surfaced by the 2026-05-07 arborist-vs-donto comparison report. Audience: fox + future blackops shifts + future adapter authors. Hard constraint: adapters never block on loss reporting; LossReport is additive metadata, not a gate. The Merkle commitment of chunks.content (raw wikitext, raw HTML body) stays bit-identical to today — LossReport captures post-chunk normalization losses, not anything that would shift document_root. Cache-key dimensionality stays at 8.

---

1. Problem statement

Arborist's adapters perform two layers of lossy transformation between raw bytes and prose the verifier sees:

Layer	Adapter	What gets dropped	Where
Ingest-time	WikipediaSqlDump family	None at chunk level — raw wikitext is preserved verbatim in chunks.content.	arborist/sources/wikipedia.py
Ingest-time	HtmlPageSource	<script>, <style>, navigation chrome, &nbsp; runs, leading/trailing whitespace per line. Body text is normalized into chunks.content.	arborist/sources/html_page.py:_normalize_text
Pre-LLM	arborist.wikitext.to_base()	<ref>...</ref> tags, self-closing <ref/>, [[File:...]] / [[Image:...]] / [[Category:...]] wikilinks, template residues, header markers, whitespace runs, 3+ newlines collapsed to 2.	arborist/wikitext.py

Today none of those drop events surface anywhere. The only audit trail is a version pin (canonicalization_version, wikitext-base-v1) hashed into governance_policy_hash — which proves "the algorithm version was X" but not "X dropped these N bytes from this particular chunk."

1.1 Concrete failure cases

Case A — phantom omission. A Wikipedia article cites a key claim in a <ref> tag. to_base() drops the entire <ref> block. The verifier never sees the citation. An auditor asking "did the LLM have access to this citation?" has no record either way.

Case B — drift after upstream change. A new wikitext-base version bumps the algorithm. The bump invalidates all prior cache_keys (good) but doesn't tell an auditor what changed about the prose — only that the version is different. To diff, you have to re-run both versions on the same chunk and compare outputs.

Case C — HTML chrome poisoning. A scraped HTML page has a navigation banner ("Most read articles: ...") that survives _normalize_text and ends up as a leading prose chunk. The verifier matches a "fact" against banner text. No audit record distinguishes prose-from-article from prose-from-chrome.

1.2 Donto comparison

Donto's PRD invariant I9 mandates that every adapter produce a typed LossReport documenting what was dropped, transformed, or quarantined. Arborist's posture is currently "adapter authors self-discipline, version pin in hash" — strong on identity, blank on inspectability.

This is the same gap shape as #000001 (retrieval-keywords audit gap): the map is committed (canonicalization_version) but the territory dropped (the bytes that didn't survive normalization) is not.

---

2. Design choices

2.1 Option A — sidecar table, no proof-path entry (RECOMMENDED)

New table adapter_loss_reports, one row per (chunk, normalization-stage, loss-kind) tuple emitted at ingest and again at pre-LLM normalization. Sidecar — never enters cache_key, never extends audit_events. Operator queries it, verifier doesn't.

CREATE TABLE adapter_loss_reports (
    chunk_id            INTEGER NOT NULL,
    document_root       BLOB    NOT NULL,
    stage               TEXT    NOT NULL,  -- 'ingest' | 'wikitext_base' | 'html_normalize'
    canonicalization_version TEXT NOT NULL, -- pins the algorithm
    loss_kind           TEXT    NOT NULL,  -- 'ref_tag' | 'file_link' | 'category_link' |
                                           -- 'template_residue' | 'whitespace_run' |
                                           -- 'html_chrome' | 'script_block' | ...
    bytes_dropped       INTEGER NOT NULL,  -- post-canonicalization byte count removed
    occurrence_count    INTEGER NOT NULL,  -- how many drops of this kind in this chunk
    sample_excerpt      TEXT,              -- truncated representative excerpt (PII risk;
                                           -- truncate at 200 bytes max, opt-out via policy)
    PRIMARY KEY (chunk_id, stage, loss_kind)
);
CREATE INDEX idx_adapter_loss_doc ON adapter_loss_reports(document_root, stage);
CREATE INDEX idx_adapter_loss_kind ON adapter_loss_reports(loss_kind, stage);

Pros:

• Direct SQL queryability — "show me every chunk where wikitext-base dropped a <ref> tag" is one statement.
• Additive migration; no existing data touched.
• Fully optional — adapters that don't emit a LossReport degrade gracefully (same as Donto's "sidecar absence degrades shape/rule/ cert calls only" stance).
• Honors arborist's "soft signals are sidecars" invariant — loss data is descriptive, never prescriptive.

Cons:

• Storage cost. Wikipedia ingest at 6 GB shard scale: rough back-of-envelope at 5 loss events/chunk * 64 bytes/row gives ~1-2% storage tax (consistent with concept_relations's 1.6%).
• Anyone with shard write access could update the table without breaking any merkle invariant (acceptable — it's a sidecar by design).

2.2 Option B — Merkle-bound loss_root per chunk

Hash each chunk's loss-report list into a loss_root, fold into document_root derivation alongside chunk content.

Pros:

• Cryptographically committed; two peers that ingest the same source compute the same loss_root.
• Loss becomes proof-path data, not sidecar.

Cons:

• Forces every adapter to emit a LossReport — no graceful degradation.
• Bumps document_root for every existing chunk on next ingest → full re-merkle of every shard. 5-8 GB old + 2 GB cur = days of rebuild for a feature that doesn't move the verifier needle.
• Couples loss reporting to identity. A bug in a single adapter's loss accounting now invalidates document identity, not just an auxiliary report.
• Violates "soft signals stay out of the proof path" — loss-counting heuristics (what counts as a "template residue"? what's a "whitespace run"?) are exactly the kind of soft signal that shouldn't decide cryptographic identity.

2.3 Option C — append to audit_events

Emit a loss_report_emitted audit event per adapter run.

Pros:

• Loss data inherits the audit chain's tamper-evidence.
• Existing audit machinery; no new table.

Cons:

• Audit chain is currently ~100 events per shard's lifecycle. A Wikipedia ingest with N chunks * M loss kinds = N*M new events. Shard-scale chain bloat — chain-check-shards slows linearly.
• Conflates state-changing operations (ingest, falsify, burn) with descriptive metadata (loss reports). The audit chain semantics blur.
• governance_policy_hash already pins the canonicalization version. Re-emitting per-chunk loss reports as state-change events overstates what loss reporting is.

2.4 Recommendation

Option A. A sidecar table mirrors the verifier's binary-vs-soft-signals discipline: hard hashes commit identity (chunk content, document_root), soft sidecars (loss reports, deflection diagnostics, title-relevance diagnostics) describe properties without entering the proof path. Same pattern as arborist.qa.inspect. Storage tax matches concept_relations (~1.6%). Cache-key dimensionality stays at 8.

---

3. Implementation sketch (when scheduled)

1. arborist/store.py — schema migration adds adapter_loss_reports table + indexes. Bump schema-version-suffix only (e.g., v9.8.1) if the migration touches existing tables; pure add = no version bump per the existing additive-column policy.

2. arborist/sources/loss_report.py (new) — pure data layer:

   @dataclass(frozen=True)
   class LossEvent:
       stage: str
       canonicalization_version: str
       loss_kind: str
       bytes_dropped: int
       occurrence_count: int
       sample_excerpt: str | None  # truncated to 200 bytes; None when policy["loss_report_excerpts"] is False
   

   Plus record_losses(conn, chunk_id, document_root, events) — batched INSERT, idempotent on the (chunk_id, stage, loss_kind) primary key.

3. arborist/wikitext.py — to_base() gains an optional loss_collector parameter. When provided, the existing drop passes (_drop_ref_tags, _drop_namespace_wikilinks, the strip_code step, the whitespace pass) accumulate LossEvents into the collector. When None, behavior is identical to today (the LLM-call path uses None; the ingest path passes a collector). Default None keeps every existing test stable.

4. arborist/sources/html_page.py — _normalize_text and parse_html likewise accept an optional collector; emit loss_kind='html_chrome', 'script_block', 'whitespace_run', etc.

5. arborist/ingest.py — at the chunk-emission site, if policy.get("loss_report_enabled", True), instantiate a collector per chunk and persist via record_losses.

6. arborist/cli.py — new arborist losses --document-root X sub-command (read-only, mirrors the existing arborist inspect shape) renders the loss table for a given document or chunk.

7. Policy fields (fold into governance_policy_hash):

   • loss_report_enabled (bool, default True) — master switch.
   • loss_report_excerpts (bool, default True) — when False, sample_excerpt is always NULL (PII-paranoid mode).

   Adding two policy fields invalidates every prior cache_key on next lookup. Acceptable; loss reporting is a one-time bump.

8. Tests:

   • Unit: to_base() with collector populates the expected LossEvent set on a fixture wikitext blob with <ref>, [[File:]], [[Category:]], and template residues.
   • Unit: to_base() without collector is bit-identical to the current implementation (regression guard).
   • Unit: record_losses is idempotent on duplicate emit.
   • Integration: ingest a small Wikipedia fixture and verify adapter_loss_reports rows match expected loss kinds per chunk.
   • Property: sum(bytes_dropped) <= len(raw) - len(prose) on every chunk (loss accounting can't exceed the actual length delta).

9. No bumps to:

   • cache_key (8 dimensions unchanged).
   • document_root (chunks.content stays raw).
   • audit_events chain (sidecar, not state-changing).
   • run_dag_root (verifier never sees loss reports).

---

4. Out of scope

• Generalizing loss_kind to a typed enum across adapters. Adapter authors should be free to mint new loss kinds (ad-hoc string identity) until we see naming patterns settle. Promote to enum after at least three adapters are in tree.
• Cross-shard loss aggregation. Per-shard tables only; no cross-shard analytics in this ticket.
• PII-aware sample-excerpt redaction beyond the loss_report_excerpts=False master switch. Per-loss-kind redaction policies (e.g., always-redact for html_chrome, truncate-only for template_residue) earn their own ticket when an operator surfaces a real PII case.
• Replaying loss reports during cache rehydrate. The audit property "was this chunk's prose derived under canonicalization version V?" is already answered by governance_policy_hash; loss reports add granularity, not new identity.
• Donto's full PRD I9 surface — LossReport as a typed Rust object with serde-bound JSON schema, pgrx integration, Lean 4 kernel-checked emptiness theorem. We're shipping the audit primitive (a queryable sidecar table); shape it like arborist's other sidecars, not like Donto's.

---

5. Status

Open · awaiting go/no-go. Surfaced by the 2026-05-07 arborist-vs-donto comparison report (/home/fox/Downloads/arborist_vs_donto.pdf). Pass 5 of that report names "surface area asymmetry" as the defining cost difference — Donto adapters carry a LossReport contract; arborist adapters carry "self-discipline plus version pin." Closing that gap with a sidecar table preserves arborist's "one engineer can read it end-to-end" property while giving an auditor a real answer to "what did the adapter drop?"

---

6. Source

• /home/fox/Downloads/arborist_vs_donto.pdf, §6.5 "Pass 5 — Stack and surface area," "Surface area asymmetry" bullet:

  Anyone could write Arborist's adapter for a new corpus in a weekend. Donto's adapter contract requires a LossReport type (PRD I9), per-anchor-kind validators, idempotency guarantees, and quarantine-instead-of-fail semantics — this is a different scale of investment.

• /home/fox/Downloads/arborist_vs_donto.pdf, §7.1 "Where they converge," "Audit-preserving removal" bullet — establishes that arborist's falsify and Donto's donto_retract share the same shape, which makes the LossReport gap genuinely one-sided rather than a deliberate identity choice.

• Donto PRD invariant I9 (cited in PDF §4.2) — "adapters must report information loss." The shape this ticket adopts is arborist's read of that invariant under the "soft-signals-as-sidecars" discipline.