From a9ee8596575e99b4292d3135ec1d4ffc55f255ca Mon Sep 17 00:00:00 2001
From: "russell@unturf.com" <russell@unturf.com>
Date: Thu, 7 May 2026 16:36:13 -0400
Subject: [PATCH] docs: open ticket #000022 (adapter LossReport) + federation
 doc-discoverability fix
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Both items surfaced by the 2026-05-07 arborist-vs-donto comparison
report (/home/fox/Downloads/arborist_vs_donto.pdf).

Ticket #000022 — adapter LossReport (PRD I9 analogue). Today wikitext
to_base() and html_page _normalize_text drop <ref> tags, image/category
wikilinks, HTML chrome, whitespace runs without recording any of it;
only the canonicalization-version pin survives. Ticket proposes a
sidecar adapter_loss_reports table (Option A) over Merkle-bound
loss_root (B) or audit-chain entries (C), preserving arborist's
"soft signals are sidecars" discipline. ~1.6% storage tax expected,
matching concept_relations. Doc-only; no code in this commit.

Federation doc-discoverability: PDF author concluded "federation
exists in code but the public docs page returned 404" because the
mesh page lives at /api/mesh.html and the concepts orientation
never surfaces the topic. Adds a "Federation across peers" section
to concepts.rst pointing at api/mesh, a :ref:\`federation\` anchor
+ context lead on api/mesh.rst, and a footer link in concepts.rst's
"Where to go next." Sphinx build passes; api/mesh.html#federation
deep link resolves.
---
 docs/TICKETS.md                               |   3 +-
 docs/_source/api/mesh.rst                     |   8 +-
 docs/_source/concepts.rst                     |  15 +
 .../ticket-000022-adapter-loss-report.md      | 307 ++++++++++++++++++
 4 files changed, 331 insertions(+), 2 deletions(-)
 create mode 100644 docs/tickets/ticket-000022-adapter-loss-report.md

diff --git a/docs/TICKETS.md b/docs/TICKETS.md
index 69d9bfe..639ce31 100644
--- a/docs/TICKETS.md
+++ b/docs/TICKETS.md
@@ -57,6 +57,7 @@ Newest first. Update on every open/close.
 
 | ID       | Title                                          | Status                | Opened     | Directive |
 |----------|------------------------------------------------|-----------------------|------------|-----------|
+| #000022  | Adapter LossReport (PRD I9 analogue)           | open · awaiting go/no-go | 2026-05-07 | —         |
 | #000021  | 5S/5T/5R benchmark fixtures + harness          | open · awaiting go/no-go | 2026-05-07 | —         |
 | #000020  | Capital-cost ledger (8-capital queues)         | open · awaiting go/no-go | 2026-05-07 | —         |
 | #000019  | Specification methodology for π* and V         | open · awaiting go/no-go | 2026-05-07 | —         |
@@ -81,4 +82,4 @@ Newest first. Update on every open/close.
 
 ## Next ID
 
-`000022`
+`000023`
diff --git a/docs/_source/api/mesh.rst b/docs/_source/api/mesh.rst
index 934da7c..8d5a255 100644
--- a/docs/_source/api/mesh.rst
+++ b/docs/_source/api/mesh.rst
@@ -1,7 +1,13 @@
+.. _federation:
+
 Federation: multiplayer arborist
 ================================
 
-Gossip-based mesh for cross-peer data sharing.
+Gossip-based mesh for cross-peer data sharing. Arborist peers
+that ingest the same source under the same chunker and
+canonicalization compute bit-identical ``document_root`` hashes;
+the mesh layer is what lets peers exchange and verify each other's
+records without a central authority.
 
 .. automodule:: arborist.mesh
    :members:
diff --git a/docs/_source/concepts.rst b/docs/_source/concepts.rst
index aac6597..ce8c39a 100644
--- a/docs/_source/concepts.rst
+++ b/docs/_source/concepts.rst
@@ -52,6 +52,20 @@ Arborist uses fox's existing Go Merkle conventions verbatim
 
 See :doc:`api/substrate` for the Python port.
 
+Federation across peers
+-----------------------
+
+Identity-by-content is the federation primitive. Two peers that
+ingest the same source with the same chunker and canonicalization
+compute the same ``document_root`` and the same ``cache_key``;
+that property is what lets peers exchange and verify each other's
+providence records without a central authority. The mesh layer
+implements gossip-based sync, group key management, and per-peer
+audit-chain reconciliation on top of that.
+
+See :doc:`api/mesh` (also reachable as :ref:`federation`) for the
+mesh wire format, member-set semantics, and deploy notes.
+
 The 8-dim cache key
 -------------------
 
@@ -177,4 +191,5 @@ Where to go next
 * :doc:`api/makefile` — every workflow as a ``make`` target
 * :doc:`api/substrate` — Merkle tree + document primitives
 * :doc:`api/qa` — Q&A pipeline, verifier, evidence map, run-DAG
+* :doc:`api/mesh` — federation primitives across peers
 * :doc:`license` — full AGPL + Permacomputer Preamble
diff --git a/docs/tickets/ticket-000022-adapter-loss-report.md b/docs/tickets/ticket-000022-adapter-loss-report.md
new file mode 100644
index 0000000..6030350
--- /dev/null
+++ b/docs/tickets/ticket-000022-adapter-loss-report.md
@@ -0,0 +1,307 @@
+# Ticket #000022 — Adapter LossReport (PRD I9 analogue)
+
+**Status:** open · awaiting go/no-go
+**Opened:** 2026-05-07
+**Scope:** Design proposal for a typed loss ledger that every source
+adapter emits during normalization, so an auditor can answer "what
+content did we drop on the way from raw bytes to the prose we
+verified?" today's adapters silently strip `<ref>` tags, image
+wikilinks, HTML chrome, and whitespace runs without recording any
+of it. Doc-only — no code in this ticket. Provenance gap analogous
+to Donto's PRD I9 LossReport contract surfaced by the
+2026-05-07 arborist-vs-donto comparison report.
+**Audience:** fox + future blackops shifts + future adapter authors.
+**Hard constraint:** adapters never block on loss reporting; LossReport
+is additive metadata, not a gate. The Merkle commitment of `chunks.content`
+(raw wikitext, raw HTML body) stays bit-identical to today — LossReport
+captures *post-chunk* normalization losses, not anything that would
+shift `document_root`. Cache-key dimensionality stays at 8.
+
+---
+
+## 1. Problem statement
+
+Arborist's adapters perform two layers of lossy transformation between
+raw bytes and prose the verifier sees:
+
+| Layer | Adapter | What gets dropped | Where |
+|-------|---------|--------------------|-------|
+| Ingest-time | `WikipediaSqlDump` family | None at chunk level — raw wikitext is preserved verbatim in `chunks.content`. | `arborist/sources/wikipedia.py` |
+| Ingest-time | `HtmlPageSource` | `<script>`, `<style>`, navigation chrome, `&nbsp;` runs, leading/trailing whitespace per line. Body text is normalized into `chunks.content`. | `arborist/sources/html_page.py:_normalize_text` |
+| Pre-LLM | `arborist.wikitext.to_base()` | `<ref>...</ref>` tags, self-closing `<ref/>`, `[[File:...]]` / `[[Image:...]]` / `[[Category:...]]` wikilinks, template residues, header markers, whitespace runs, 3+ newlines collapsed to 2. | `arborist/wikitext.py` |
+
+Today none of those drop events surface anywhere. The only audit
+trail is a version pin (`canonicalization_version`,
+`wikitext-base-v1`) hashed into `governance_policy_hash` — which
+proves "the algorithm version was X" but not "X dropped these N
+bytes from this particular chunk."
+
+### 1.1 Concrete failure cases
+
+**Case A — phantom omission.** A Wikipedia article cites a key claim
+in a `<ref>` tag. `to_base()` drops the entire `<ref>` block. The
+verifier never sees the citation. An auditor asking "did the LLM
+have access to this citation?" has no record either way.
+
+**Case B — drift after upstream change.** A new wikitext-base version
+bumps the algorithm. The bump invalidates all prior cache_keys (good)
+but doesn't tell an auditor what *changed* about the prose — only
+that the version is different. To diff, you have to re-run both
+versions on the same chunk and compare outputs.
+
+**Case C — HTML chrome poisoning.** A scraped HTML page has a
+navigation banner ("Most read articles: ...") that survives
+`_normalize_text` and ends up as a leading prose chunk. The
+verifier matches a "fact" against banner text. No audit record
+distinguishes prose-from-article from prose-from-chrome.
+
+### 1.2 Donto comparison
+
+Donto's PRD invariant I9 mandates that every adapter produce a
+typed LossReport documenting what was dropped, transformed, or
+quarantined. Arborist's posture is currently "adapter authors
+self-discipline, version pin in hash" — strong on identity, blank
+on inspectability.
+
+This is the same gap shape as #000001 (retrieval-keywords audit
+gap): the *map* is committed (canonicalization_version) but the
+*territory dropped* (the bytes that didn't survive normalization)
+is not.
+
+---
+
+## 2. Design choices
+
+### 2.1 Option A — sidecar table, no proof-path entry (RECOMMENDED)
+
+New table `adapter_loss_reports`, one row per (chunk, normalization-stage,
+loss-kind) tuple emitted at ingest and again at pre-LLM normalization.
+Sidecar — never enters `cache_key`, never extends `audit_events`.
+Operator queries it, verifier doesn't.
+
+```sql
+CREATE TABLE adapter_loss_reports (
+    chunk_id            INTEGER NOT NULL,
+    document_root       BLOB    NOT NULL,
+    stage               TEXT    NOT NULL,  -- 'ingest' | 'wikitext_base' | 'html_normalize'
+    canonicalization_version TEXT NOT NULL, -- pins the algorithm
+    loss_kind           TEXT    NOT NULL,  -- 'ref_tag' | 'file_link' | 'category_link' |
+                                           -- 'template_residue' | 'whitespace_run' |
+                                           -- 'html_chrome' | 'script_block' | ...
+    bytes_dropped       INTEGER NOT NULL,  -- post-canonicalization byte count removed
+    occurrence_count    INTEGER NOT NULL,  -- how many drops of this kind in this chunk
+    sample_excerpt      TEXT,              -- truncated representative excerpt (PII risk;
+                                           -- truncate at 200 bytes max, opt-out via policy)
+    PRIMARY KEY (chunk_id, stage, loss_kind)
+);
+CREATE INDEX idx_adapter_loss_doc ON adapter_loss_reports(document_root, stage);
+CREATE INDEX idx_adapter_loss_kind ON adapter_loss_reports(loss_kind, stage);
+```
+
+**Pros:**
+- Direct SQL queryability — "show me every chunk where wikitext-base
+  dropped a `<ref>` tag" is one statement.
+- Additive migration; no existing data touched.
+- Fully optional — adapters that don't emit a LossReport degrade
+  gracefully (same as Donto's "sidecar absence degrades shape/rule/
+  cert calls only" stance).
+- Honors arborist's "soft signals are sidecars" invariant — loss
+  data is descriptive, never prescriptive.
+
+**Cons:**
+- Storage cost. Wikipedia ingest at 6 GB shard scale: rough
+  back-of-envelope at 5 loss events/chunk * 64 bytes/row gives
+  ~1-2% storage tax (consistent with `concept_relations`'s 1.6%).
+- Anyone with shard write access could update the table without
+  breaking any merkle invariant (acceptable — it's a sidecar by
+  design).
+
+### 2.2 Option B — Merkle-bound `loss_root` per chunk
+
+Hash each chunk's loss-report list into a `loss_root`, fold into
+`document_root` derivation alongside chunk content.
+
+**Pros:**
+- Cryptographically committed; two peers that ingest the same
+  source compute the same `loss_root`.
+- Loss becomes proof-path data, not sidecar.
+
+**Cons:**
+- Forces every adapter to emit a LossReport — no graceful degradation.
+- Bumps `document_root` for every existing chunk on next ingest →
+  full re-merkle of every shard. 5-8 GB old + 2 GB cur = days of
+  rebuild for a feature that doesn't move the verifier needle.
+- Couples loss reporting to identity. A bug in a single adapter's
+  loss accounting now invalidates document identity, not just an
+  auxiliary report.
+- Violates "soft signals stay out of the proof path" — loss-counting
+  heuristics (what counts as a "template residue"? what's a
+  "whitespace run"?) are exactly the kind of soft signal that
+  shouldn't decide cryptographic identity.
+
+### 2.3 Option C — append to `audit_events`
+
+Emit a `loss_report_emitted` audit event per adapter run.
+
+**Pros:**
+- Loss data inherits the audit chain's tamper-evidence.
+- Existing audit machinery; no new table.
+
+**Cons:**
+- Audit chain is currently ~100 events per shard's lifecycle. A
+  Wikipedia ingest with N chunks * M loss kinds = N*M new events.
+  Shard-scale chain bloat — `chain-check-shards` slows linearly.
+- Conflates state-changing operations (ingest, falsify, burn) with
+  descriptive metadata (loss reports). The audit chain semantics
+  blur.
+- `governance_policy_hash` already pins the canonicalization
+  version. Re-emitting per-chunk loss reports as state-change
+  events overstates what loss reporting *is*.
+
+### 2.4 Recommendation
+
+**Option A.** A sidecar table mirrors the verifier's binary-vs-soft-signals
+discipline: hard hashes commit identity (chunk content, document_root),
+soft sidecars (loss reports, deflection diagnostics, title-relevance
+diagnostics) describe properties without entering the proof path.
+Same pattern as `arborist.qa.inspect`. Storage tax matches
+`concept_relations` (~1.6%). Cache-key dimensionality stays at 8.
+
+---
+
+## 3. Implementation sketch (when scheduled)
+
+1. **`arborist/store.py`** — schema migration adds `adapter_loss_reports`
+   table + indexes. Bump schema-version-suffix only (e.g., `v9.8.1`)
+   if the migration touches existing tables; pure add = no version
+   bump per the existing additive-column policy.
+
+2. **`arborist/sources/loss_report.py`** (new) — pure data layer:
+
+   ```python
+   @dataclass(frozen=True)
+   class LossEvent:
+       stage: str
+       canonicalization_version: str
+       loss_kind: str
+       bytes_dropped: int
+       occurrence_count: int
+       sample_excerpt: str | None  # truncated to 200 bytes; None when policy["loss_report_excerpts"] is False
+   ```
+
+   Plus `record_losses(conn, chunk_id, document_root, events)` —
+   batched INSERT, idempotent on the (chunk_id, stage, loss_kind)
+   primary key.
+
+3. **`arborist/wikitext.py`** — `to_base()` gains an optional
+   `loss_collector` parameter. When provided, the existing drop
+   passes (`_drop_ref_tags`, `_drop_namespace_wikilinks`, the
+   `strip_code` step, the whitespace pass) accumulate `LossEvent`s
+   into the collector. When `None`, behavior is identical to today
+   (the LLM-call path uses `None`; the ingest path passes a
+   collector). Default `None` keeps every existing test stable.
+
+4. **`arborist/sources/html_page.py`** — `_normalize_text` and
+   `parse_html` likewise accept an optional collector; emit
+   `loss_kind='html_chrome'`, `'script_block'`, `'whitespace_run'`,
+   etc.
+
+5. **`arborist/ingest.py`** — at the chunk-emission site, if
+   `policy.get("loss_report_enabled", True)`, instantiate a
+   collector per chunk and persist via `record_losses`.
+
+6. **`arborist/cli.py`** — new `arborist losses --document-root X`
+   sub-command (read-only, mirrors the existing
+   `arborist inspect` shape) renders the loss table for a given
+   document or chunk.
+
+7. **Policy fields** (fold into `governance_policy_hash`):
+   - `loss_report_enabled` (bool, default `True`) — master switch.
+   - `loss_report_excerpts` (bool, default `True`) — when False,
+     `sample_excerpt` is always NULL (PII-paranoid mode).
+
+   Adding two policy fields invalidates every prior cache_key on
+   next lookup. Acceptable; loss reporting is a one-time bump.
+
+8. **Tests:**
+   - Unit: `to_base()` with collector populates the expected
+     `LossEvent` set on a fixture wikitext blob with
+     `<ref>`, `[[File:]]`, `[[Category:]]`, and template residues.
+   - Unit: `to_base()` without collector is bit-identical to the
+     current implementation (regression guard).
+   - Unit: `record_losses` is idempotent on duplicate emit.
+   - Integration: ingest a small Wikipedia fixture and verify
+     `adapter_loss_reports` rows match expected loss kinds
+     per chunk.
+   - Property: `sum(bytes_dropped) <= len(raw) - len(prose)` on
+     every chunk (loss accounting can't exceed the actual length
+     delta).
+
+9. **No bumps to:**
+   - `cache_key` (8 dimensions unchanged).
+   - `document_root` (chunks.content stays raw).
+   - `audit_events` chain (sidecar, not state-changing).
+   - `run_dag_root` (verifier never sees loss reports).
+
+---
+
+## 4. Out of scope
+
+- **Generalizing `loss_kind` to a typed enum across adapters.**
+  Adapter authors should be free to mint new loss kinds (ad-hoc
+  string identity) until we see naming patterns settle. Promote
+  to enum after at least three adapters are in tree.
+- **Cross-shard loss aggregation.** Per-shard tables only; no
+  cross-shard analytics in this ticket.
+- **PII-aware sample-excerpt redaction beyond the
+  `loss_report_excerpts=False` master switch.** Per-loss-kind
+  redaction policies (e.g., always-redact for `html_chrome`,
+  truncate-only for `template_residue`) earn their own ticket
+  when an operator surfaces a real PII case.
+- **Replaying loss reports during cache rehydrate.** The audit
+  property "was this chunk's prose derived under canonicalization
+  version V?" is already answered by `governance_policy_hash`;
+  loss reports add granularity, not new identity.
+- **Donto's full PRD I9 surface** — `LossReport` as a typed
+  Rust object with serde-bound JSON schema, pgrx integration,
+  Lean 4 kernel-checked emptiness theorem. We're shipping the
+  audit primitive (a queryable sidecar table); shape it like
+  arborist's other sidecars, not like Donto's.
+
+---
+
+## 5. Status
+
+Open · awaiting go/no-go. Surfaced by the 2026-05-07
+arborist-vs-donto comparison report (`/home/fox/Downloads/arborist_vs_donto.pdf`).
+Pass 5 of that report names "surface area asymmetry" as the
+defining cost difference — Donto adapters carry a `LossReport`
+contract; arborist adapters carry "self-discipline plus version
+pin." Closing that gap with a sidecar table preserves arborist's
+"one engineer can read it end-to-end" property while giving an
+auditor a real answer to "what did the adapter drop?"
+
+---
+
+## 6. Source
+
+- `/home/fox/Downloads/arborist_vs_donto.pdf`,
+  §6.5 "Pass 5 — Stack and surface area," "Surface area asymmetry"
+  bullet:
+
+  > Anyone could write Arborist's adapter for a new corpus in a
+  > weekend. Donto's adapter contract requires a LossReport type
+  > (PRD I9), per-anchor-kind validators, idempotency guarantees,
+  > and quarantine-instead-of-fail semantics — this is a different
+  > scale of investment.
+
+- `/home/fox/Downloads/arborist_vs_donto.pdf`,
+  §7.1 "Where they converge," "Audit-preserving removal" bullet
+  — establishes that arborist's `falsify` and Donto's `donto_retract`
+  share the same shape, which makes the LossReport gap genuinely
+  one-sided rather than a deliberate identity choice.
+
+- Donto PRD invariant I9 (cited in PDF §4.2) — "adapters must
+  report information loss." The shape this ticket adopts is
+  arborist's read of that invariant under the
+  "soft-signals-as-sidecars" discipline.