|
|
02c7e41ef8
|
loss_report: land ticket #000022 (adapter LossReport sidecar)
Typed loss ledger for adapter / canonicalizer drops, transforms, and
normalizations. Sidecar — never enters cache_key, document_root,
run_dag_root, or audit_events. Loss policy lives in its own
loss_report_policy_hash so toggling reporting does NOT invalidate
prior QA cache entries (corrected pre-land per GPT-5.5 review).
- adapter_loss_reports table: PK (chunk_id, stage, canonicalization_version,
loss_kind); columns include loss_mode {pure_drop|transform|quarantine|
normalize}, bytes_dropped, occurrence_count, input/output_length_bytes,
sample_excerpt, sample_hash, adapter_name/version, loss_report_policy_hash
- arborist/sources/loss_report.py: LossEvent, LossCollector with
add()/record_delta()/set_lengths()/events(), record_losses() batched
idempotent insert, compute_loss_report_policy_hash() pure function
- wikitext.to_base() emits ref_tag, self_closing_ref_tag, file_link,
image_link, category_link, strip_code_transform, whitespace_run.
loss_collector=None default keeps verifier/runner/query path unchanged
- html_page parse_html / _normalize_text emit script_block, style_block,
html_chrome, whitespace_run; HtmlPageSource gains loss_report_*
__init__ flags. Document-scope events anchor to first chunk_id at
ingest via Document.extra['loss_events']
- ingest.ingest_source: per-chunk to_base() with collector for
wikipedia_* sources; persisted via record_losses inside the same
transaction as chunk inserts. Default loss_report_enabled=True
- arborist losses CLI subcommand: --document-root / --chunk-id /
--kind / --stage / --summary / --json. arborist ingest gains
--no-loss-report / --no-loss-excerpts / --loss-excerpt-bytes
- tests/test_loss_report.py: 15 tests covering bit-identical
regression, loss-kind taxonomy, byte-conservation property test
(loss-mode-aware), idempotent persistence, document_root invariant
under toggle, policy hash purity
1091 tests pass, 0 audit-chain breaks across all 7 shards.
|
2026-05-07 17:58:50 -04:00 |
|
