russell@unturf.com
326badf6d8
Three things in one commit because they're tightly coupled (README
points at the diagrams; diagrams index in modules/index.md points
back at README; module pages embed the diagrams).
(1) README — label refresh:
- Quickstart label changed from STRICT/HYBRID/UNGROUNDED to the
four-rung ladder POINTER-LINKED → ANCHOR-WARRANTED →
EVIDENCE-WARRANTED → UNGROUNDED with -PARTIAL suffix on HYBRID.
- Verifier section spells out both layers (schema trichotomy +
display ladder), the seven hard checks of claim_lattice, and
the five anchor classes of warrant.
- Architecture tree updated: concepts/ package added, qa/
sub-modules expanded (warrant, evidence, parse_claims, dag),
verify.py described as quote/span/entity/paraphrase + claim_lattice.
- Concept overlay description updated for corpus-derived layer
(concept_relations table, link_reciprocity extractor, 1.6%
tax cite).
- Test count: 326+ → 641+.
(2) docs/diagrams/ — Graphviz dot sources:
- aborist-modules.dot — top-level package graph (substrate /
storage / sources / retrieval / qa / mesh / cli)
- query-pipeline.dot — question → cache → retrieval → LLM →
verify → render → cache write, with phase budgets
- ingest-pipeline.dot — source doc → canonicalize → chunk →
Merkle → upsert (+ optional distill)
- verifier-ladder.dot — (audit_mode, violations) → display rung
decision tree
Existing mesh-*.dot kept as-is. Makefile `make docs` target
extended to also emit .svg alongside the existing .png so the
diagrams render in markdown viewers.
(3) docs/modules/ — per-module reference pages:
- index.md (links to every diagram + every module page)
- merkle.md, document.md, store.md, ingest.md, evict.md,
sources.md, search.md, concepts.md, qa.md, distill.md,
wikitext.md
Each page is a one-screenful concise reference: what the
module is for, public API, key invariants, embedded diagrams
where useful, link to source. Mesh stays at the existing
docs/mesh.md + docs/mesh-deploy.md (already comprehensive).
Tests: 641 passed (no code change).
64 lines
2.3 KiB
Markdown
64 lines
2.3 KiB
Markdown
# `aborist.ingest`
|
|
|
|
The bulk-batched pipeline that turns documents from a source into
|
|
Merkle-committed shard storage. Source-agnostic: anything that
|
|
implements `Source.iter_documents()` flows through here.
|
|
|
|
|
|
|
|
## Public API
|
|
|
|
```python
|
|
from aborist.ingest import ingest_source
|
|
from aborist.sources.wikipedia import WikipediaSqlDump
|
|
|
|
source = WikipediaSqlDump("/path/to/cur.sql.bz2")
|
|
ingest_source(
|
|
source,
|
|
db_path=Path("~/.aborist/shards/000.db"),
|
|
batch_size=200, # docs per transaction
|
|
progress_every=1000,
|
|
)
|
|
```
|
|
|
|
## What happens per document
|
|
|
|
1. **Canonicalize** — `canonicalize(text)`: NFC + ws-collapse +
|
|
strip ends. Pinned by `canonicalization_version='norm-v1'`.
|
|
2. **Chunk** — `Chunker.chunk(canonical_text)` → list of token-
|
|
bounded substrings. Default `tok-512-v1` chunker.
|
|
3. **Hash leaves** — `sha256(0x00 || canonical_chunk_bytes)` per
|
|
chunk.
|
|
4. **Merkle tree** — `MerkleTree.build(leaves).root` →
|
|
`document_root`. Two peers running the same chunker on the same
|
|
canonicalized content compute bit-identical roots.
|
|
5. **Upsert** — `documents` row keyed on `document_root` (idempotent
|
|
re-ingest), `chunks` rows with leaf hashes, `merkle_nodes` for
|
|
proof reconstruction, `edges` per outbound link.
|
|
6. **FTS5** — `chunks_fts` insert with rowid = `chunks.chunk_id`
|
|
so the search-time JOIN lines up.
|
|
7. **Audit event** — one row per ingest batch in `audit_events`,
|
|
chained on `prev_event_hash`.
|
|
|
|
## Batching discipline
|
|
|
|
Default `batch_size=200`: balances Python GIL overhead vs SQLite
|
|
transaction commit cost. Lower it (e.g. 50) only to bound peak
|
|
memory on a low-RAM host. Higher (e.g. 1000) for ETL throughput on
|
|
SSD storage when memory isn't tight.
|
|
|
|
`progress_every` prints a stderr line every N docs so long ingests
|
|
are observable. Use `PYTHONUNBUFFERED=1` for tail-able output.
|
|
|
|
## Resumability
|
|
|
|
Idempotent re-ingest: same content + same chunker + same canonicalize
|
|
= same `document_root` = no-op insert. So a crashed ingest can be
|
|
restarted from the source's beginning without duplicating rows.
|
|
|
|
Different content at the same URI gets a new `document_root` AND a
|
|
`supersedes` edge linking new → old (lossless history).
|
|
|
|
## Source
|
|
|
|
[`aborist/ingest.py`](../../aborist/ingest.py)
|