# `aborist.ingest`

The bulk-batched pipeline that turns documents from a source into
Merkle-committed shard storage. Source-agnostic: anything that
implements `Source.iter_documents()` flows through here.

![ingest pipeline](../diagrams/ingest-pipeline.svg)

## Public API

```python
from aborist.ingest import ingest_source
from aborist.sources.wikipedia import WikipediaSqlDump

source = WikipediaSqlDump("/path/to/cur.sql.bz2")
ingest_source(
    source,
    db_path=Path("~/.aborist/shards/000.db"),
    batch_size=200,        # docs per transaction
    progress_every=1000,
)
```

## What happens per document

1. **Canonicalize** — `canonicalize(text)`: NFC + ws-collapse +
   strip ends. Pinned by `canonicalization_version='norm-v1'`.
2. **Chunk** — `Chunker.chunk(canonical_text)` → list of token-
   bounded substrings. Default `tok-512-v1` chunker.
3. **Hash leaves** — `sha256(0x00 || canonical_chunk_bytes)` per
   chunk.
4. **Merkle tree** — `MerkleTree.build(leaves).root` →
   `document_root`. Two peers running the same chunker on the same
   canonicalized content compute bit-identical roots.
5. **Upsert** — `documents` row keyed on `document_root` (idempotent
   re-ingest), `chunks` rows with leaf hashes, `merkle_nodes` for
   proof reconstruction, `edges` per outbound link.
6. **FTS5** — `chunks_fts` insert with rowid = `chunks.chunk_id`
   so the search-time JOIN lines up.
7. **Audit event** — one row per ingest batch in `audit_events`,
   chained on `prev_event_hash`.

## Batching discipline

Default `batch_size=200`: balances Python GIL overhead vs SQLite
transaction commit cost. Lower it (e.g. 50) only to bound peak
memory on a low-RAM host. Higher (e.g. 1000) for ETL throughput on
SSD storage when memory isn't tight.

`progress_every` prints a stderr line every N docs so long ingests
are observable. Use `PYTHONUNBUFFERED=1` for tail-able output.

## Resumability

Idempotent re-ingest: same content + same chunker + same canonicalize
= same `document_root` = no-op insert. So a crashed ingest can be
restarted from the source's beginning without duplicating rows.

Different content at the same URI gets a new `document_root` AND a
`supersedes` edge linking new → old (lossless history).

## Source

[`aborist/ingest.py`](../../aborist/ingest.py)