russell@unturf.com
856b3116d7
A content-addressed, Merkle-committed document store implementing the runtime spec from Merkle Providence Reverse RAG (April 2026 whitepaper) and Merkle-AGI v9.8 admissibility ledger. Ports proxy.unturf.com Go merkle conventions to Python: non-commutative HashCombine with 0x03 prefix, explicit IsLeft per sibling, self-duplicate odd elements. What's in: - merkle.py — proof generation/verification, JSON serialization - store.py — v9.8 SQLite schema: 8-dim providence_cache key, falsification_state, append-only audit chain, surface/core kind, hot/warm/cold tier, derivations, edges - ingest.py — Source -> normalize -> chunk -> merkle -> upsert, idempotent on document_root collision - search/ — SearchBackend ABC with explicit AuditMode (STRICT/HYBRID/ VISUAL), FTS5 backend returning VISUAL hits - sources/ — wikipedia.py (streaming bz2/MySQL extended-INSERT parser for 2003-era cur dumps); html_page.py (selectolax + httpx, robots.txt honored automatically) - distill/ — Distiller ABC + first-sentence-v1 stub. Runner generates per-contributing-chunk Merkle proofs binding cores back to source document_root. - evict.py — hot->cold demote (NULLs content, drops FTS row, retains leaf_hash). rehydrate() refetches via source pipeline; matching root restores content, mismatching root marks providence stale and writes rehydrate_drift event. Cores never evict. - cli.py — ingest / search / verify / stats / distill / evict / rehydrate - 31 tests covering merkle round-trip, ingest+audit, chunker version binding, html parse, distillation proof verification, evict+ rehydrate including drift detection. Smoke: 503 Wikipedia 2003-05-16 + 3 fox-owned HTML pages ingested, 478 cores produced (24 surface->core merkle dedups), 7 chunks evicted to cold and round-tripped via rehydrate, 987 audit events chained 0 breaks.
70 lines
2.3 KiB
Makefile
70 lines
2.3 KiB
Makefile
# aborist — Makefile entry points
|
|
# Every workflow lives behind a `make` target. Bare python commands are not
|
|
# the user interface.
|
|
|
|
# Tools and config
|
|
PYTHON ?= python3
|
|
VENV ?= .venv
|
|
PIP := $(VENV)/bin/pip
|
|
PY := $(VENV)/bin/python
|
|
ABORIST := $(VENV)/bin/aborist
|
|
|
|
# Data + DB
|
|
DATA_DIR ?= data
|
|
WP_DUMP_URL ?= https://dumps.wikimedia.org/archive/2003/2003-05-16/en/20030516_cur_tablesql.bz2
|
|
WP_DUMP := $(DATA_DIR)/20030516_cur_tablesql.bz2
|
|
DB ?= $(HOME)/.aborist/aborist.db
|
|
|
|
# Smoke-test caps so make all stays fast
|
|
INGEST_LIMIT ?= 500
|
|
VERIFY_N ?= 10
|
|
SEARCH_Q ?= computer
|
|
|
|
.PHONY: all bootstrap fetch ingest verify search stats test clean clean-db clean-data help
|
|
|
|
all: bootstrap fetch ingest verify stats ## bootstrap → fetch → ingest → verify → stats
|
|
|
|
help: ## show this help
|
|
@awk 'BEGIN{FS=":.*##"} /^[a-zA-Z0-9_-]+:.*##/{printf " %-14s %s\n", $$1, $$2}' $(MAKEFILE_LIST)
|
|
|
|
$(VENV)/bin/activate: pyproject.toml
|
|
$(PYTHON) -m venv $(VENV)
|
|
$(PIP) install --upgrade pip wheel
|
|
$(PIP) install -e '.[dev]'
|
|
@touch $(VENV)/bin/activate
|
|
|
|
bootstrap: $(VENV)/bin/activate ## create venv and install editable package
|
|
|
|
$(WP_DUMP): | $(DATA_DIR)
|
|
@echo ">> fetching $(WP_DUMP_URL)"
|
|
curl -fL --retry 3 -o $@ "$(WP_DUMP_URL)"
|
|
|
|
$(DATA_DIR):
|
|
mkdir -p $(DATA_DIR)
|
|
|
|
fetch: $(WP_DUMP) ## download Wikipedia 2003-05-16 cur dump (cached)
|
|
|
|
ingest: bootstrap fetch ## ingest INGEST_LIMIT articles into $(DB)
|
|
$(ABORIST) --db $(DB) ingest --source wikipedia_cur --path $(WP_DUMP) --limit $(INGEST_LIMIT)
|
|
|
|
verify: bootstrap ## round-trip Merkle proofs for VERIFY_N random documents
|
|
$(ABORIST) --db $(DB) verify -n $(VERIFY_N)
|
|
|
|
search: bootstrap ## keyword search; override SEARCH_Q (or pass Q=...)
|
|
$(ABORIST) --db $(DB) search '$(if $(Q),$(Q),$(SEARCH_Q))'
|
|
|
|
stats: bootstrap ## counts: documents, chunks, edges, audit chain
|
|
$(ABORIST) --db $(DB) stats
|
|
|
|
test: bootstrap ## run pytest suite
|
|
$(VENV)/bin/pytest -q
|
|
|
|
clean: ## remove venv + caches (keeps fetched data and db)
|
|
rm -rf $(VENV) .pytest_cache **/__pycache__ aborist.egg-info
|
|
find . -type d -name __pycache__ -prune -exec rm -rf {} +
|
|
|
|
clean-db: ## drop the aborist db (keeps fetched data and venv)
|
|
rm -f $(DB) $(DB)-journal $(DB)-wal $(DB)-shm
|
|
|
|
clean-data: ## remove fetched dumps
|
|
rm -rf $(DATA_DIR)
|