engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

loss_report: land ticket #000022 (adapter LossReport sidecar)

Browse source 
Typed loss ledger for adapter / canonicalizer drops, transforms, and
normalizations. Sidecar — never enters cache_key, document_root,
run_dag_root, or audit_events. Loss policy lives in its own
loss_report_policy_hash so toggling reporting does NOT invalidate
prior QA cache entries (corrected pre-land per GPT-5.5 review).

- adapter_loss_reports table: PK (chunk_id, stage, canonicalization_version,
  loss_kind); columns include loss_mode {pure_drop|transform|quarantine|
  normalize}, bytes_dropped, occurrence_count, input/output_length_bytes,
  sample_excerpt, sample_hash, adapter_name/version, loss_report_policy_hash
- arborist/sources/loss_report.py: LossEvent, LossCollector with
  add()/record_delta()/set_lengths()/events(), record_losses() batched
  idempotent insert, compute_loss_report_policy_hash() pure function
- wikitext.to_base() emits ref_tag, self_closing_ref_tag, file_link,
  image_link, category_link, strip_code_transform, whitespace_run.
  loss_collector=None default keeps verifier/runner/query path unchanged
- html_page parse_html / _normalize_text emit script_block, style_block,
  html_chrome, whitespace_run; HtmlPageSource gains loss_report_*
  __init__ flags. Document-scope events anchor to first chunk_id at
  ingest via Document.extra['loss_events']
- ingest.ingest_source: per-chunk to_base() with collector for
  wikipedia_* sources; persisted via record_losses inside the same
  transaction as chunk inserts. Default loss_report_enabled=True
- arborist losses CLI subcommand: --document-root / --chunk-id /
  --kind / --stage / --summary / --json. arborist ingest gains
  --no-loss-report / --no-loss-excerpts / --loss-excerpt-bytes
- tests/test_loss_report.py: 15 tests covering bit-identical
  regression, loss-kind taxonomy, byte-conservation property test
  (loss-mode-aware), idempotent persistence, document_root invariant
  under toggle, policy hash purity

1091 tests pass, 0 audit-chain breaks across all 7 shards.
This commit is contained in:
russell@unturf.com 2026-05-07 17:58:50 -04:00
parent 10d2db621c
commit 02c7e41ef8
No known key found for this signature in database
9 changed files with 1460 additions and 49 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/TICKETS.md
View file

@ -60,7 +60,7 @@ Newest first. Update on every open/close.
| #000025 | 5F battery (Function · Finetuning · Falsification · Formulate · Feedback Loop) | open · awaiting go/no-go | 2026-05-07 | — |
| #000024 | 5T Phase 1b + Dav1DPrometheus vocabulary alignment | open · awaiting go/no-go | 2026-05-07 | — |
| #000023 | 5S Phase 1b: Syllogism · Synthesis · Semiotics | open · awaiting go/no-go | 2026-05-07 | — |
| #000022 | Adapter LossReport (PRD I9 analogue) | open · awaiting go/no-go | 2026-05-07 | — |
| #000022 | Adapter LossReport (PRD I9 analogue) | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000021 | 5S/5T/5R benchmark fixtures + harness | in progress · Phase 1a landed 2026-05-07 | 2026-05-07 | — |
| #000020 | Capital-cost ledger (8-capital queues) | closed · landed 2026-05-07 | 2026-05-07 | — |
| #000019 | Specification methodology for π* and V | closed · landed 2026-05-07 | 2026-05-07 | — |

211
docs/tickets/ticket-000022-adapter-loss-report.md
View file

@ -1,7 +1,10 @@
# Ticket #000022 — Adapter LossReport (PRD I9 analogue)
**Status:** open · awaiting go/no-go
**Status:** closed · landed 2026-05-07
**Opened:** 2026-05-07
**Closed:** 2026-05-07 (amended pre-land per GPT-5.5 review:
schema columns added, policy split out of `governance_policy_hash`,
property test rewritten to be loss-mode-aware)
**Scope:** Design proposal for a typed loss ledger that every source
adapter emits during normalization, so an auditor can answer "what
content did we drop on the way from raw bytes to the prose we
@ -81,23 +84,61 @@ Operator queries it, verifier doesn't.
```sql
CREATE TABLE adapter_loss_reports (
chunk_id INTEGER NOT NULL,
document_root BLOB NOT NULL,
stage TEXT NOT NULL, -- 'ingest' | 'wikitext_base' | 'html_normalize'
canonicalization_version TEXT NOT NULL, -- pins the algorithm
loss_kind TEXT NOT NULL, -- 'ref_tag' | 'file_link' | 'category_link' |
-- 'template_residue' | 'whitespace_run' |
-- 'html_chrome' | 'script_block' | ...
bytes_dropped INTEGER NOT NULL, -- post-canonicalization byte count removed
occurrence_count INTEGER NOT NULL, -- how many drops of this kind in this chunk
sample_excerpt TEXT, -- truncated representative excerpt (PII risk;
-- truncate at 200 bytes max, opt-out via policy)
PRIMARY KEY (chunk_id, stage, loss_kind)
chunk_id INTEGER NOT NULL,
document_root BLOB NOT NULL,
stage TEXT NOT NULL, -- 'ingest' | 'wikitext_base' | 'html_normalize'
canonicalization_version TEXT NOT NULL, -- pins the algorithm (e.g. 'wikitext-base-v1')
loss_kind TEXT NOT NULL, -- 'ref_tag' | 'file_link' | 'category_link' |
-- 'template_residue' | 'whitespace_run' |
-- 'html_chrome' | 'script_block' | ...
loss_mode TEXT NOT NULL, -- 'pure_drop' | 'transform' | 'quarantine' |
-- 'normalize' (see §2.1.1 for semantics)
bytes_dropped INTEGER NOT NULL, -- post-canonicalization byte count removed
-- (only meaningful for pure_drop / quarantine;
-- transform/normalize record byte delta but
-- the property test excludes them)
occurrence_count INTEGER NOT NULL, -- how many events of this kind in this chunk
input_length_bytes INTEGER, -- chunk input byte count at this stage
output_length_bytes INTEGER, -- chunk output byte count at this stage
sample_excerpt TEXT, -- truncated representative excerpt (PII risk;
-- truncate at 200 bytes max, NULL when
-- loss_report_excerpts=False)
sample_hash BLOB, -- sha256(canonical(sample_bytes)) — populated
-- even in PII-safe mode so loss-signature
-- recurrence stays detectable without storing
-- raw bytes
adapter_name TEXT, -- e.g. 'WikipediaSqlDump', 'HtmlPageSource'
adapter_version TEXT, -- adapter package version (independent of
-- canonicalization_version — adapter taxonomy
-- expands faster than canonicalizer versions)
loss_report_policy_hash BLOB NOT NULL, -- pins the loss-reporting policy (see §3.7);
-- separate from governance_policy_hash so
-- toggling reporting does NOT invalidate QA
-- cache_keys
created_at TEXT NOT NULL,
PRIMARY KEY (chunk_id, stage, canonicalization_version, loss_kind)
);
CREATE INDEX idx_adapter_loss_doc ON adapter_loss_reports(document_root, stage);
CREATE INDEX idx_adapter_loss_kind ON adapter_loss_reports(loss_kind, stage);
CREATE INDEX idx_adapter_loss_doc ON adapter_loss_reports(document_root, stage);
CREATE INDEX idx_adapter_loss_kind ON adapter_loss_reports(loss_kind, stage);
CREATE INDEX idx_adapter_loss_chunk ON adapter_loss_reports(chunk_id, stage);
```
#### 2.1.1 `loss_mode` semantics
A loss event is one of four shapes. The shape determines whether
`bytes_dropped` participates in the property test (§3.8):
| `loss_mode` | Meaning | Counts toward bytes-dropped sanity check? |
|--------------|----------------------------------------------------------------|-------------------------------------------|
| `pure_drop` | Content removed entirely. `<ref>...</ref>` blocks, script/style blocks, `[[Category:...]]` links. | Yes — `sum(bytes_dropped) ≤ input_length_bytes`. |
| `transform` | Content rewritten to different bytes. `[[Velociraptor]]` → `Velociraptor` drops markup but keeps the link text. | No — bytes-dropped here is a delta, not a deletion. |
| `quarantine` | Content preserved elsewhere (e.g., a sidecar) but withheld from prose. Reserved for future adapters; no v0 emitter. | Yes — same shape as `pure_drop` for accounting. |
| `normalize` | Semantically low-content formatting normalization. Whitespace runs, newline collapse, `&nbsp;` runs. | No — emitted for visibility, not for byte-conservation accounting. |
Free-string `loss_kind` keeps the per-adapter naming flexibility from
the original §4 out-of-scope list; `loss_mode` is the closed enum that
keeps the property test sound across adapters.
**Pros:**
- Direct SQL queryability — "show me every chunk where wikitext-base
dropped a `<ref>` tag" is one statement.
@ -110,11 +151,14 @@ CREATE INDEX idx_adapter_loss_kind ON adapter_loss_reports(loss_kind, stage);
**Cons:**
- Storage cost. Wikipedia ingest at 6 GB shard scale: rough
back-of-envelope at 5 loss events/chunk * 64 bytes/row gives
~1-2% storage tax (consistent with `concept_relations`'s 1.6%).
- Anyone with shard write access could update the table without
breaking any merkle invariant (acceptable — it's a sidecar by
design).
back-of-envelope at 5 loss events/chunk * ~120 bytes/row (post
schema additions in §2.1.1) gives ~2-3% storage tax. Within
the same order as `concept_relations`'s 1.6%; not a blocker.
- Free-string `loss_kind` plus heuristic kinds (`html_chrome`)
mean two adapters could emit the same dropped bytes under
different names. Mitigated by §4 promotion-to-enum after three
adapters; loss data is descriptive, not prescriptive, so
inconsistent naming is a triage cost not a correctness cost.
### 2.2 Option B — Merkle-bound `loss_root` per chunk
@ -184,14 +228,23 @@ Same pattern as `arborist.qa.inspect`. Storage tax matches
stage: str
canonicalization_version: str
loss_kind: str
loss_mode: str # 'pure_drop' | 'transform' | 'quarantine' | 'normalize'
bytes_dropped: int
occurrence_count: int
sample_excerpt: str | None # truncated to 200 bytes; None when policy["loss_report_excerpts"] is False
input_length_bytes: int | None
output_length_bytes: int | None
sample_excerpt: str | None # truncated to 200 bytes; None when
# policy["loss_report_excerpts"] is False
sample_hash: bytes | None # sha256 of canonical sample bytes;
# populated even in PII-safe mode
adapter_name: str | None
adapter_version: str | None
```
Plus `record_losses(conn, chunk_id, document_root, events)` —
batched INSERT, idempotent on the (chunk_id, stage, loss_kind)
primary key.
Plus `record_losses(conn, chunk_id, document_root, events,
loss_report_policy_hash)` — batched INSERT, idempotent on the
`(chunk_id, stage, canonicalization_version, loss_kind)` primary
key.
3. **`arborist/wikitext.py`** — `to_base()` gains an optional
`loss_collector` parameter. When provided, the existing drop
@ -215,30 +268,70 @@ Same pattern as `arborist.qa.inspect`. Storage tax matches
`arborist inspect` shape) renders the loss table for a given
document or chunk.
7. **Policy fields** (fold into `governance_policy_hash`):
7. **Policy fields — separate `loss_report_policy_hash`, NOT folded
into `governance_policy_hash`:**
- `loss_report_enabled` (bool, default `True`) — master switch.
- `loss_report_excerpts` (bool, default `True`) — when False,
`sample_excerpt` is always NULL (PII-paranoid mode).
`sample_excerpt` is always NULL (PII-paranoid mode);
`sample_hash` stays populated.
- `loss_report_max_excerpt_bytes` (int, default `200`) — hard cap.
Adding two policy fields invalidates every prior cache_key on
next lookup. Acceptable; loss reporting is a one-time bump.
These three fields canonicalize-then-hash into
`loss_report_policy_hash`, stored as a column on
`adapter_loss_reports` (per §2.1). They do **not** participate
in the QA-side `cache_key` because LossReport is sidecar
metadata that does not change normalized prose, retrieval,
prompt, verifier behavior, or answer text. Toggling
`loss_report_enabled` from True→False (or back) MUST NOT
invalidate prior QA cache entries — that would convert a
diagnostic toggle into answer-path churn, exactly the failure
mode §2.2 rejects Option B for.
Promotion path: if a future loss-reporting policy DOES change
normalized prose (e.g., a `loss_quarantine_strict` mode that
alters which bytes survive into the prose stream), it stops
being a LossReport policy and becomes a canonicalization policy.
At that point — and only at that point — it folds into
`governance_policy_hash` and gets its own
`canonicalization_version` bump. v0 reporting keeps prose
bit-identical to today, so no fold is needed.
8. **Tests:**
- Unit: `to_base()` with collector populates the expected
`LossEvent` set on a fixture wikitext blob with
`<ref>`, `[[File:]]`, `[[Category:]]`, and template residues.
`LossEvent` set on a fixture wikitext blob with `<ref>`,
`[[File:]]`, `[[Category:]]`, and template residues. Each
event carries the right `loss_mode`.
- Unit: `to_base()` without collector is bit-identical to the
current implementation (regression guard).
- Unit: `record_losses` is idempotent on duplicate emit.
- Unit: `sample_excerpt` truncates at
`loss_report_max_excerpt_bytes`; `sample_hash` stays
populated when `loss_report_excerpts=False`.
- Unit: toggling `loss_report_enabled` from True→False on the
same shard does NOT change any `cache_key` — confirms the
hash split (§3.7).
- Integration: ingest a small Wikipedia fixture and verify
`adapter_loss_reports` rows match expected loss kinds
per chunk.
- Property: `sum(bytes_dropped) <= len(raw) - len(prose)` on
every chunk (loss accounting can't exceed the actual length
delta).
`adapter_loss_reports` rows match expected loss kinds and
modes per chunk.
- Property (loss-mode-aware byte conservation):
- For each `(chunk_id, stage)`:
`sum(bytes_dropped) WHERE loss_mode IN ('pure_drop','quarantine')
<= input_length_bytes`.
- For each `(chunk_id, stage)`:
`output_length_bytes <= input_length_bytes` for every adapter
whose `canonicalization_version` advertises pure-strip semantics
(wikitext-base-v1 and html-normalize-v1 both qualify).
- `loss_mode='transform'` and `loss_mode='normalize'` rows are
excluded from the sum because their `bytes_dropped` records a
byte delta, not a deletion. The original ticket's
`sum(bytes_dropped) <= len(raw) - len(prose)` test fails on
transforms like `[[Velociraptor]] → Velociraptor` (markup
drops, link text survives) and is replaced by the above.
9. **No bumps to:**
- `cache_key` (8 dimensions unchanged).
- `cache_key` (8 dimensions unchanged; `governance_policy_hash`
unchanged because LossReport policy lives in its own
`loss_report_policy_hash` per §3.7).
- `document_root` (chunks.content stays raw).
- `audit_events` chain (sidecar, not state-changing).
- `run_dag_root` (verifier never sees loss reports).
@ -281,6 +374,52 @@ pin." Closing that gap with a sidecar table preserves arborist's
"one engineer can read it end-to-end" property while giving an
auditor a real answer to "what did the adapter drop?"
### 5.1 Amendment trail
**2026-05-07 (GPT-5.5 review pass).** Original draft contradicted
its own §2.2 "soft signals stay out of the proof path" reasoning by
folding `loss_report_enabled` and `loss_report_excerpts` into
`governance_policy_hash` in §3.7 — a diagnostic toggle that would
have invalidated every prior QA cache entry on landing. Amendments:
1. **§3.7** — split policy into a separate `loss_report_policy_hash`
stored on `adapter_loss_reports`. QA `cache_key` stays at 8
dimensions; `governance_policy_hash` unchanged. Promotion path
spelled out: if a future loss policy ever changes normalized
prose, it stops being a LossReport policy and gets a
`canonicalization_version` bump.
2. **§2.1** — schema gains `loss_mode`, `input_length_bytes`,
`output_length_bytes`, `sample_hash`, `adapter_name`,
`adapter_version`, `loss_report_policy_hash`, `created_at`.
PK extends to `(chunk_id, stage, canonicalization_version,
loss_kind)` so multiple normalizer versions can coexist for
the same chunk during canonicalization migrations.
3. **§2.1.1 (new)** — `loss_mode ∈ {pure_drop, transform,
quarantine, normalize}` semantics defined with a participation
table for the byte-conservation property test.
4. **§3.8** — replaced the naive
`sum(bytes_dropped) <= len(raw) - len(prose)` property test
with a loss-mode-aware version that excludes `transform` and
`normalize` rows (their `bytes_dropped` records a byte delta,
not a deletion — `[[Velociraptor]] → Velociraptor` is the
canonical breakage).
5. **§2.1 cons** — struck the "anyone with shard write access could
update the table without breaking any merkle invariant" line.
That's the design intent of a sidecar, not a downside. Replaced
with the real con: free-string `loss_kind` plus heuristic kinds
(`html_chrome`) admit naming drift across adapters.
Out-of-scope tightening (rejected for v0):
- LossReport as a generalized π* loss ledger — speculative bridge
to #000015. Defer until #000022 has shipped and at least three
adapters use it.
- Loss-aware UI warnings on `arborist query` output — blurs the
soft/hard line we just defended. CLI `arborist losses` covers
the auditor case.
- Per-occurrence `adapter_loss_occurrences` table — defer per
the existing aggregation-first discipline.
---
## 6. Source