russell@unturf.com
d0857fb43d
Workers each parse the full SQL stream but only emit rows whose index satisfies (idx % total == rank). N processes cooperate against a single shared SQLite file; WAL+synchronous=NORMAL handles writer contention via BEGIN IMMEDIATE serialization. Implementation chose OS-level processes over in-process mp.Queue: the gain ceiling is identical (writer is the bottleneck either way) and the surface area is one new flag plus stride-filter logic instead of a queue-based pipeline with picklable workers. Makefile gains: ingest-cur-parallel SHARDS=4 ingest-old-parallel SHARDS=4 Benchmark on 20k cur articles: serial 55.3 s 361 docs/s 1.0x CPU usage 4-way parallel 43.1 s 464 docs/s ~3.0x CPU usage 1.28x wall Both runs landed 19,999 unique documents — correctness verified. The write-lock is now the bottleneck; further gains require Phase 2 attach-forever sharding (each shard owns its own SQLite file, queries fan out via ATTACH DATABASE, zero merge cost). |
||
|---|---|---|
| aborist | ||
| tests | ||
| .gitignore | ||
| CLAUDE.md | ||
| LICENSE | ||
| Makefile | ||
| pyproject.toml | ||
| README.md | ||
aborist
An arborist for trees and forests of cross-linked information.
Aborist ingests documents into a content-addressed, Merkle-committed store. Every
document carries a URI for backtrack and cross-link. Search hooks return hits with
declared audit modes (STRICT / HYBRID / VISUAL) so callers never overclaim
what a result actually proves.
Design
Aborist implements the runtime spec from Merkle-AGI v9.8 / Merkle Providence
Reverse RAG. The 8-dim admissibility key (source_root, question_hash,
model_profile_hash, conversation_hash, governance_policy_hash,
schema_version, canonicalization_version, chunking_version) plus
falsification state ensures cached records are never reused under drift.
Two layered document kinds:
- surface — diverse ingested content (full chunks, FTS-indexed)
- core — distilled records (haiku/equation/snippet) Merkle-signed back to
source surface roots via
derivations
Reversible eviction via chunk tier ∈ {hot, warm, cold}. Cold = leaf hash
- URI only; rehydratable from URI, identity verifiable via leaf hash.
Quick start
make all # bootstrap + fetch + ingest + verify + stats
make search Q='…'
License
AGPL-3.0-only.