docs: open ticket #000022 (adapter LossReport) + federation doc-discoverability fix
Both items surfaced by the 2026-05-07 arborist-vs-donto comparison report (/home/fox/Downloads/arborist_vs_donto.pdf). Ticket #000022 — adapter LossReport (PRD I9 analogue). Today wikitext to_base() and html_page _normalize_text drop <ref> tags, image/category wikilinks, HTML chrome, whitespace runs without recording any of it; only the canonicalization-version pin survives. Ticket proposes a sidecar adapter_loss_reports table (Option A) over Merkle-bound loss_root (B) or audit-chain entries (C), preserving arborist's "soft signals are sidecars" discipline. ~1.6% storage tax expected, matching concept_relations. Doc-only; no code in this commit. Federation doc-discoverability: PDF author concluded "federation exists in code but the public docs page returned 404" because the mesh page lives at /api/mesh.html and the concepts orientation never surfaces the topic. Adds a "Federation across peers" section to concepts.rst pointing at api/mesh, a :ref:\`federation\` anchor + context lead on api/mesh.rst, and a footer link in concepts.rst's "Where to go next." Sphinx build passes; api/mesh.html#federation deep link resolves.
This commit is contained in:
parent
8fe0144d81
commit
a9ee859657
4 changed files with 331 additions and 2 deletions
|
|
@ -57,6 +57,7 @@ Newest first. Update on every open/close.
|
|||
|
||||
| ID | Title | Status | Opened | Directive |
|
||||
|----------|------------------------------------------------|-----------------------|------------|-----------|
|
||||
| #000022 | Adapter LossReport (PRD I9 analogue) | open · awaiting go/no-go | 2026-05-07 | — |
|
||||
| #000021 | 5S/5T/5R benchmark fixtures + harness | open · awaiting go/no-go | 2026-05-07 | — |
|
||||
| #000020 | Capital-cost ledger (8-capital queues) | open · awaiting go/no-go | 2026-05-07 | — |
|
||||
| #000019 | Specification methodology for π* and V | open · awaiting go/no-go | 2026-05-07 | — |
|
||||
|
|
@ -81,4 +82,4 @@ Newest first. Update on every open/close.
|
|||
|
||||
## Next ID
|
||||
|
||||
`000022`
|
||||
`000023`
|
||||
|
|
|
|||
|
|
@ -1,7 +1,13 @@
|
|||
.. _federation:
|
||||
|
||||
Federation: multiplayer arborist
|
||||
================================
|
||||
|
||||
Gossip-based mesh for cross-peer data sharing.
|
||||
Gossip-based mesh for cross-peer data sharing. Arborist peers
|
||||
that ingest the same source under the same chunker and
|
||||
canonicalization compute bit-identical ``document_root`` hashes;
|
||||
the mesh layer is what lets peers exchange and verify each other's
|
||||
records without a central authority.
|
||||
|
||||
.. automodule:: arborist.mesh
|
||||
:members:
|
||||
|
|
|
|||
|
|
@ -52,6 +52,20 @@ Arborist uses fox's existing Go Merkle conventions verbatim
|
|||
|
||||
See :doc:`api/substrate` for the Python port.
|
||||
|
||||
Federation across peers
|
||||
-----------------------
|
||||
|
||||
Identity-by-content is the federation primitive. Two peers that
|
||||
ingest the same source with the same chunker and canonicalization
|
||||
compute the same ``document_root`` and the same ``cache_key``;
|
||||
that property is what lets peers exchange and verify each other's
|
||||
providence records without a central authority. The mesh layer
|
||||
implements gossip-based sync, group key management, and per-peer
|
||||
audit-chain reconciliation on top of that.
|
||||
|
||||
See :doc:`api/mesh` (also reachable as :ref:`federation`) for the
|
||||
mesh wire format, member-set semantics, and deploy notes.
|
||||
|
||||
The 8-dim cache key
|
||||
-------------------
|
||||
|
||||
|
|
@ -177,4 +191,5 @@ Where to go next
|
|||
* :doc:`api/makefile` — every workflow as a ``make`` target
|
||||
* :doc:`api/substrate` — Merkle tree + document primitives
|
||||
* :doc:`api/qa` — Q&A pipeline, verifier, evidence map, run-DAG
|
||||
* :doc:`api/mesh` — federation primitives across peers
|
||||
* :doc:`license` — full AGPL + Permacomputer Preamble
|
||||
|
|
|
|||
307
docs/tickets/ticket-000022-adapter-loss-report.md
Normal file
307
docs/tickets/ticket-000022-adapter-loss-report.md
Normal file
|
|
@ -0,0 +1,307 @@
|
|||
# Ticket #000022 — Adapter LossReport (PRD I9 analogue)
|
||||
|
||||
**Status:** open · awaiting go/no-go
|
||||
**Opened:** 2026-05-07
|
||||
**Scope:** Design proposal for a typed loss ledger that every source
|
||||
adapter emits during normalization, so an auditor can answer "what
|
||||
content did we drop on the way from raw bytes to the prose we
|
||||
verified?" today's adapters silently strip `<ref>` tags, image
|
||||
wikilinks, HTML chrome, and whitespace runs without recording any
|
||||
of it. Doc-only — no code in this ticket. Provenance gap analogous
|
||||
to Donto's PRD I9 LossReport contract surfaced by the
|
||||
2026-05-07 arborist-vs-donto comparison report.
|
||||
**Audience:** fox + future blackops shifts + future adapter authors.
|
||||
**Hard constraint:** adapters never block on loss reporting; LossReport
|
||||
is additive metadata, not a gate. The Merkle commitment of `chunks.content`
|
||||
(raw wikitext, raw HTML body) stays bit-identical to today — LossReport
|
||||
captures *post-chunk* normalization losses, not anything that would
|
||||
shift `document_root`. Cache-key dimensionality stays at 8.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
Arborist's adapters perform two layers of lossy transformation between
|
||||
raw bytes and prose the verifier sees:
|
||||
|
||||
| Layer | Adapter | What gets dropped | Where |
|
||||
|-------|---------|--------------------|-------|
|
||||
| Ingest-time | `WikipediaSqlDump` family | None at chunk level — raw wikitext is preserved verbatim in `chunks.content`. | `arborist/sources/wikipedia.py` |
|
||||
| Ingest-time | `HtmlPageSource` | `<script>`, `<style>`, navigation chrome, ` ` runs, leading/trailing whitespace per line. Body text is normalized into `chunks.content`. | `arborist/sources/html_page.py:_normalize_text` |
|
||||
| Pre-LLM | `arborist.wikitext.to_base()` | `<ref>...</ref>` tags, self-closing `<ref/>`, `[[File:...]]` / `[[Image:...]]` / `[[Category:...]]` wikilinks, template residues, header markers, whitespace runs, 3+ newlines collapsed to 2. | `arborist/wikitext.py` |
|
||||
|
||||
Today none of those drop events surface anywhere. The only audit
|
||||
trail is a version pin (`canonicalization_version`,
|
||||
`wikitext-base-v1`) hashed into `governance_policy_hash` — which
|
||||
proves "the algorithm version was X" but not "X dropped these N
|
||||
bytes from this particular chunk."
|
||||
|
||||
### 1.1 Concrete failure cases
|
||||
|
||||
**Case A — phantom omission.** A Wikipedia article cites a key claim
|
||||
in a `<ref>` tag. `to_base()` drops the entire `<ref>` block. The
|
||||
verifier never sees the citation. An auditor asking "did the LLM
|
||||
have access to this citation?" has no record either way.
|
||||
|
||||
**Case B — drift after upstream change.** A new wikitext-base version
|
||||
bumps the algorithm. The bump invalidates all prior cache_keys (good)
|
||||
but doesn't tell an auditor what *changed* about the prose — only
|
||||
that the version is different. To diff, you have to re-run both
|
||||
versions on the same chunk and compare outputs.
|
||||
|
||||
**Case C — HTML chrome poisoning.** A scraped HTML page has a
|
||||
navigation banner ("Most read articles: ...") that survives
|
||||
`_normalize_text` and ends up as a leading prose chunk. The
|
||||
verifier matches a "fact" against banner text. No audit record
|
||||
distinguishes prose-from-article from prose-from-chrome.
|
||||
|
||||
### 1.2 Donto comparison
|
||||
|
||||
Donto's PRD invariant I9 mandates that every adapter produce a
|
||||
typed LossReport documenting what was dropped, transformed, or
|
||||
quarantined. Arborist's posture is currently "adapter authors
|
||||
self-discipline, version pin in hash" — strong on identity, blank
|
||||
on inspectability.
|
||||
|
||||
This is the same gap shape as #000001 (retrieval-keywords audit
|
||||
gap): the *map* is committed (canonicalization_version) but the
|
||||
*territory dropped* (the bytes that didn't survive normalization)
|
||||
is not.
|
||||
|
||||
---
|
||||
|
||||
## 2. Design choices
|
||||
|
||||
### 2.1 Option A — sidecar table, no proof-path entry (RECOMMENDED)
|
||||
|
||||
New table `adapter_loss_reports`, one row per (chunk, normalization-stage,
|
||||
loss-kind) tuple emitted at ingest and again at pre-LLM normalization.
|
||||
Sidecar — never enters `cache_key`, never extends `audit_events`.
|
||||
Operator queries it, verifier doesn't.
|
||||
|
||||
```sql
|
||||
CREATE TABLE adapter_loss_reports (
|
||||
chunk_id INTEGER NOT NULL,
|
||||
document_root BLOB NOT NULL,
|
||||
stage TEXT NOT NULL, -- 'ingest' | 'wikitext_base' | 'html_normalize'
|
||||
canonicalization_version TEXT NOT NULL, -- pins the algorithm
|
||||
loss_kind TEXT NOT NULL, -- 'ref_tag' | 'file_link' | 'category_link' |
|
||||
-- 'template_residue' | 'whitespace_run' |
|
||||
-- 'html_chrome' | 'script_block' | ...
|
||||
bytes_dropped INTEGER NOT NULL, -- post-canonicalization byte count removed
|
||||
occurrence_count INTEGER NOT NULL, -- how many drops of this kind in this chunk
|
||||
sample_excerpt TEXT, -- truncated representative excerpt (PII risk;
|
||||
-- truncate at 200 bytes max, opt-out via policy)
|
||||
PRIMARY KEY (chunk_id, stage, loss_kind)
|
||||
);
|
||||
CREATE INDEX idx_adapter_loss_doc ON adapter_loss_reports(document_root, stage);
|
||||
CREATE INDEX idx_adapter_loss_kind ON adapter_loss_reports(loss_kind, stage);
|
||||
```
|
||||
|
||||
**Pros:**
|
||||
- Direct SQL queryability — "show me every chunk where wikitext-base
|
||||
dropped a `<ref>` tag" is one statement.
|
||||
- Additive migration; no existing data touched.
|
||||
- Fully optional — adapters that don't emit a LossReport degrade
|
||||
gracefully (same as Donto's "sidecar absence degrades shape/rule/
|
||||
cert calls only" stance).
|
||||
- Honors arborist's "soft signals are sidecars" invariant — loss
|
||||
data is descriptive, never prescriptive.
|
||||
|
||||
**Cons:**
|
||||
- Storage cost. Wikipedia ingest at 6 GB shard scale: rough
|
||||
back-of-envelope at 5 loss events/chunk * 64 bytes/row gives
|
||||
~1-2% storage tax (consistent with `concept_relations`'s 1.6%).
|
||||
- Anyone with shard write access could update the table without
|
||||
breaking any merkle invariant (acceptable — it's a sidecar by
|
||||
design).
|
||||
|
||||
### 2.2 Option B — Merkle-bound `loss_root` per chunk
|
||||
|
||||
Hash each chunk's loss-report list into a `loss_root`, fold into
|
||||
`document_root` derivation alongside chunk content.
|
||||
|
||||
**Pros:**
|
||||
- Cryptographically committed; two peers that ingest the same
|
||||
source compute the same `loss_root`.
|
||||
- Loss becomes proof-path data, not sidecar.
|
||||
|
||||
**Cons:**
|
||||
- Forces every adapter to emit a LossReport — no graceful degradation.
|
||||
- Bumps `document_root` for every existing chunk on next ingest →
|
||||
full re-merkle of every shard. 5-8 GB old + 2 GB cur = days of
|
||||
rebuild for a feature that doesn't move the verifier needle.
|
||||
- Couples loss reporting to identity. A bug in a single adapter's
|
||||
loss accounting now invalidates document identity, not just an
|
||||
auxiliary report.
|
||||
- Violates "soft signals stay out of the proof path" — loss-counting
|
||||
heuristics (what counts as a "template residue"? what's a
|
||||
"whitespace run"?) are exactly the kind of soft signal that
|
||||
shouldn't decide cryptographic identity.
|
||||
|
||||
### 2.3 Option C — append to `audit_events`
|
||||
|
||||
Emit a `loss_report_emitted` audit event per adapter run.
|
||||
|
||||
**Pros:**
|
||||
- Loss data inherits the audit chain's tamper-evidence.
|
||||
- Existing audit machinery; no new table.
|
||||
|
||||
**Cons:**
|
||||
- Audit chain is currently ~100 events per shard's lifecycle. A
|
||||
Wikipedia ingest with N chunks * M loss kinds = N*M new events.
|
||||
Shard-scale chain bloat — `chain-check-shards` slows linearly.
|
||||
- Conflates state-changing operations (ingest, falsify, burn) with
|
||||
descriptive metadata (loss reports). The audit chain semantics
|
||||
blur.
|
||||
- `governance_policy_hash` already pins the canonicalization
|
||||
version. Re-emitting per-chunk loss reports as state-change
|
||||
events overstates what loss reporting *is*.
|
||||
|
||||
### 2.4 Recommendation
|
||||
|
||||
**Option A.** A sidecar table mirrors the verifier's binary-vs-soft-signals
|
||||
discipline: hard hashes commit identity (chunk content, document_root),
|
||||
soft sidecars (loss reports, deflection diagnostics, title-relevance
|
||||
diagnostics) describe properties without entering the proof path.
|
||||
Same pattern as `arborist.qa.inspect`. Storage tax matches
|
||||
`concept_relations` (~1.6%). Cache-key dimensionality stays at 8.
|
||||
|
||||
---
|
||||
|
||||
## 3. Implementation sketch (when scheduled)
|
||||
|
||||
1. **`arborist/store.py`** — schema migration adds `adapter_loss_reports`
|
||||
table + indexes. Bump schema-version-suffix only (e.g., `v9.8.1`)
|
||||
if the migration touches existing tables; pure add = no version
|
||||
bump per the existing additive-column policy.
|
||||
|
||||
2. **`arborist/sources/loss_report.py`** (new) — pure data layer:
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class LossEvent:
|
||||
stage: str
|
||||
canonicalization_version: str
|
||||
loss_kind: str
|
||||
bytes_dropped: int
|
||||
occurrence_count: int
|
||||
sample_excerpt: str | None # truncated to 200 bytes; None when policy["loss_report_excerpts"] is False
|
||||
```
|
||||
|
||||
Plus `record_losses(conn, chunk_id, document_root, events)` —
|
||||
batched INSERT, idempotent on the (chunk_id, stage, loss_kind)
|
||||
primary key.
|
||||
|
||||
3. **`arborist/wikitext.py`** — `to_base()` gains an optional
|
||||
`loss_collector` parameter. When provided, the existing drop
|
||||
passes (`_drop_ref_tags`, `_drop_namespace_wikilinks`, the
|
||||
`strip_code` step, the whitespace pass) accumulate `LossEvent`s
|
||||
into the collector. When `None`, behavior is identical to today
|
||||
(the LLM-call path uses `None`; the ingest path passes a
|
||||
collector). Default `None` keeps every existing test stable.
|
||||
|
||||
4. **`arborist/sources/html_page.py`** — `_normalize_text` and
|
||||
`parse_html` likewise accept an optional collector; emit
|
||||
`loss_kind='html_chrome'`, `'script_block'`, `'whitespace_run'`,
|
||||
etc.
|
||||
|
||||
5. **`arborist/ingest.py`** — at the chunk-emission site, if
|
||||
`policy.get("loss_report_enabled", True)`, instantiate a
|
||||
collector per chunk and persist via `record_losses`.
|
||||
|
||||
6. **`arborist/cli.py`** — new `arborist losses --document-root X`
|
||||
sub-command (read-only, mirrors the existing
|
||||
`arborist inspect` shape) renders the loss table for a given
|
||||
document or chunk.
|
||||
|
||||
7. **Policy fields** (fold into `governance_policy_hash`):
|
||||
- `loss_report_enabled` (bool, default `True`) — master switch.
|
||||
- `loss_report_excerpts` (bool, default `True`) — when False,
|
||||
`sample_excerpt` is always NULL (PII-paranoid mode).
|
||||
|
||||
Adding two policy fields invalidates every prior cache_key on
|
||||
next lookup. Acceptable; loss reporting is a one-time bump.
|
||||
|
||||
8. **Tests:**
|
||||
- Unit: `to_base()` with collector populates the expected
|
||||
`LossEvent` set on a fixture wikitext blob with
|
||||
`<ref>`, `[[File:]]`, `[[Category:]]`, and template residues.
|
||||
- Unit: `to_base()` without collector is bit-identical to the
|
||||
current implementation (regression guard).
|
||||
- Unit: `record_losses` is idempotent on duplicate emit.
|
||||
- Integration: ingest a small Wikipedia fixture and verify
|
||||
`adapter_loss_reports` rows match expected loss kinds
|
||||
per chunk.
|
||||
- Property: `sum(bytes_dropped) <= len(raw) - len(prose)` on
|
||||
every chunk (loss accounting can't exceed the actual length
|
||||
delta).
|
||||
|
||||
9. **No bumps to:**
|
||||
- `cache_key` (8 dimensions unchanged).
|
||||
- `document_root` (chunks.content stays raw).
|
||||
- `audit_events` chain (sidecar, not state-changing).
|
||||
- `run_dag_root` (verifier never sees loss reports).
|
||||
|
||||
---
|
||||
|
||||
## 4. Out of scope
|
||||
|
||||
- **Generalizing `loss_kind` to a typed enum across adapters.**
|
||||
Adapter authors should be free to mint new loss kinds (ad-hoc
|
||||
string identity) until we see naming patterns settle. Promote
|
||||
to enum after at least three adapters are in tree.
|
||||
- **Cross-shard loss aggregation.** Per-shard tables only; no
|
||||
cross-shard analytics in this ticket.
|
||||
- **PII-aware sample-excerpt redaction beyond the
|
||||
`loss_report_excerpts=False` master switch.** Per-loss-kind
|
||||
redaction policies (e.g., always-redact for `html_chrome`,
|
||||
truncate-only for `template_residue`) earn their own ticket
|
||||
when an operator surfaces a real PII case.
|
||||
- **Replaying loss reports during cache rehydrate.** The audit
|
||||
property "was this chunk's prose derived under canonicalization
|
||||
version V?" is already answered by `governance_policy_hash`;
|
||||
loss reports add granularity, not new identity.
|
||||
- **Donto's full PRD I9 surface** — `LossReport` as a typed
|
||||
Rust object with serde-bound JSON schema, pgrx integration,
|
||||
Lean 4 kernel-checked emptiness theorem. We're shipping the
|
||||
audit primitive (a queryable sidecar table); shape it like
|
||||
arborist's other sidecars, not like Donto's.
|
||||
|
||||
---
|
||||
|
||||
## 5. Status
|
||||
|
||||
Open · awaiting go/no-go. Surfaced by the 2026-05-07
|
||||
arborist-vs-donto comparison report (`/home/fox/Downloads/arborist_vs_donto.pdf`).
|
||||
Pass 5 of that report names "surface area asymmetry" as the
|
||||
defining cost difference — Donto adapters carry a `LossReport`
|
||||
contract; arborist adapters carry "self-discipline plus version
|
||||
pin." Closing that gap with a sidecar table preserves arborist's
|
||||
"one engineer can read it end-to-end" property while giving an
|
||||
auditor a real answer to "what did the adapter drop?"
|
||||
|
||||
---
|
||||
|
||||
## 6. Source
|
||||
|
||||
- `/home/fox/Downloads/arborist_vs_donto.pdf`,
|
||||
§6.5 "Pass 5 — Stack and surface area," "Surface area asymmetry"
|
||||
bullet:
|
||||
|
||||
> Anyone could write Arborist's adapter for a new corpus in a
|
||||
> weekend. Donto's adapter contract requires a LossReport type
|
||||
> (PRD I9), per-anchor-kind validators, idempotency guarantees,
|
||||
> and quarantine-instead-of-fail semantics — this is a different
|
||||
> scale of investment.
|
||||
|
||||
- `/home/fox/Downloads/arborist_vs_donto.pdf`,
|
||||
§7.1 "Where they converge," "Audit-preserving removal" bullet
|
||||
— establishes that arborist's `falsify` and Donto's `donto_retract`
|
||||
share the same shape, which makes the LossReport gap genuinely
|
||||
one-sided rather than a deliberate identity choice.
|
||||
|
||||
- Donto PRD invariant I9 (cited in PDF §4.2) — "adapters must
|
||||
report information loss." The shape this ticket adopts is
|
||||
arborist's read of that invariant under the
|
||||
"soft-signals-as-sidecars" discipline.
|
||||
Loading…
Add table
Add a link
Reference in a new issue