russell@unturf.com
856b3116d7
A content-addressed, Merkle-committed document store implementing the runtime spec from Merkle Providence Reverse RAG (April 2026 whitepaper) and Merkle-AGI v9.8 admissibility ledger. Ports proxy.unturf.com Go merkle conventions to Python: non-commutative HashCombine with 0x03 prefix, explicit IsLeft per sibling, self-duplicate odd elements. What's in: - merkle.py — proof generation/verification, JSON serialization - store.py — v9.8 SQLite schema: 8-dim providence_cache key, falsification_state, append-only audit chain, surface/core kind, hot/warm/cold tier, derivations, edges - ingest.py — Source -> normalize -> chunk -> merkle -> upsert, idempotent on document_root collision - search/ — SearchBackend ABC with explicit AuditMode (STRICT/HYBRID/ VISUAL), FTS5 backend returning VISUAL hits - sources/ — wikipedia.py (streaming bz2/MySQL extended-INSERT parser for 2003-era cur dumps); html_page.py (selectolax + httpx, robots.txt honored automatically) - distill/ — Distiller ABC + first-sentence-v1 stub. Runner generates per-contributing-chunk Merkle proofs binding cores back to source document_root. - evict.py — hot->cold demote (NULLs content, drops FTS row, retains leaf_hash). rehydrate() refetches via source pipeline; matching root restores content, mismatching root marks providence stale and writes rehydrate_drift event. Cores never evict. - cli.py — ingest / search / verify / stats / distill / evict / rehydrate - 31 tests covering merkle round-trip, ingest+audit, chunker version binding, html parse, distillation proof verification, evict+ rehydrate including drift detection. Smoke: 503 Wikipedia 2003-05-16 + 3 fox-owned HTML pages ingested, 478 cores produced (24 surface->core merkle dedups), 7 chunks evicted to cold and round-tripped via rehydrate, 987 audit events chained 0 breaks.
47 lines
1.2 KiB
Python
47 lines
1.2 KiB
Python
"""Search backend ABC + Hit dataclass with explicit audit mode.
|
|
|
|
Every search hit carries an `audit_mode` so callers never overclaim. Per
|
|
Merkle-AGI v7 §:
|
|
- STRICT — Merkle-verified, deterministic, full local reveal. Supports formal
|
|
claims (e.g., providence_cache hit with verified proof).
|
|
- HYBRID — random-challenge spot check (RCA(K)). Non-normative.
|
|
- VISUAL — exploration / debug only. No proof claim. Keyword search lives here.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import enum
|
|
import sqlite3
|
|
from abc import ABC, abstractmethod
|
|
from dataclasses import dataclass
|
|
|
|
|
|
class AuditMode(str, enum.Enum):
|
|
STRICT = "STRICT"
|
|
HYBRID = "HYBRID"
|
|
VISUAL = "VISUAL"
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Hit:
|
|
document_root: str
|
|
document_uri: str
|
|
chunk_idx: int
|
|
snippet: str
|
|
score: float
|
|
audit_mode: AuditMode
|
|
title: str | None = None
|
|
|
|
|
|
class SearchBackend(ABC):
|
|
"""A search hook over the chunk store."""
|
|
|
|
name: str
|
|
audit_mode: AuditMode # default mode this backend reports
|
|
|
|
def __init__(self, conn: sqlite3.Connection):
|
|
self.conn = conn
|
|
|
|
@abstractmethod
|
|
def search(self, query: str, limit: int = 20) -> list[Hit]:
|
|
...
|