trim Makefile (-6 redundant targets) + better RTD docs
Makefile cuts (64 → 58 documented targets): - ingest-cur-parallel, ingest-old-parallel: parallel-shared mode superseded by attached (no WAL contention) - distill-shards: sequential never preferred over parallel variant - bench-qa-quick: bench-qa-smoke covers same use case (~30s vs ~10s) - ingest-grok, ingest-grok-media: single-DB grok rare; -attached is canonical path All cuts land in code that the underlying CLI still exposes — operators who need the dropped variant call '.venv/bin/aborist ingest --shard ...' directly. No behavior loss, just shortcut removal. Docs improvements: - New Concepts page (docs/_source/concepts.rst): orientation on what aborist is, three layers (surface/core/providence), Merkle commitment, 8-dim cache key, audit chain, trichotomy + four-rung ladder, layered verifier, falsification state, sidecars. Embeds module-graph and verifier-ladder SVG diagrams. - New Cookbook page (docs/_source/cookbook.rst): 8 recipes — recrawl, falsify, ingest-self-providence, mixed-corpus query, LLM endpoint override, integrity after bulk ops, bench, retrieval tuning. - Quickstart embeds query-pipeline SVG diagram. - docs/_source/diagrams symlinks to docs/diagrams so Sphinx can include the SVGs (was orphaned, only README referenced them). Better Makefile RTD page (docs/_source/_ext/makefile_targets.py): - Group by workflow phase (Setup → Fetch → Ingest → Distill → Query → Verify → Operations → Tests → Docs → Clean) instead of alphabetical prefix. Tells a new operator the order they'd actually run things. - Phase descriptions added; targets prefixed with 'make ' for copy-paste. - Uncategorized leftover surfaces missing entries in PHASES list.
This commit is contained in:
parent
09fb5e4f2d
commit
850e59d16e
7 changed files with 496 additions and 61 deletions
36
Makefile
36
Makefile
|
|
@ -28,7 +28,6 @@ SEARCH_Q ?= computer
|
|||
.PHONY: all bootstrap fetch fetch-cur fetch-old fetch-xml fetch-abstract \
|
||||
ingest ingest-cur ingest-old ingest-xml ingest-xml-history \
|
||||
ingest-xml-attached ingest-abstract \
|
||||
ingest-grok ingest-grok-media \
|
||||
ingest-self ingest-self-providence ingest-git ingest-hg \
|
||||
verify search stats test test-live docs docs-api docs-api-clean \
|
||||
chain-check chain-check-shards \
|
||||
|
|
@ -81,24 +80,10 @@ ingest-cur: bootstrap fetch-cur ## ingest INGEST_LIMIT cur articles
|
|||
ingest-old: bootstrap fetch-old ## ingest INGEST_LIMIT old (history) revisions
|
||||
$(ABORIST) --db $(DB) ingest --source wikipedia_old --path $(WP_OLD) --limit $(INGEST_LIMIT)
|
||||
|
||||
# Phase 1b: parallel shards into ONE shared SQLite. Workers parallelize
|
||||
# parser CPU; writes serialize at the WAL writer-lock. ~1.3x wall on 4 cores.
|
||||
SHARDS ?= 4
|
||||
ingest-cur-parallel: bootstrap fetch-cur ## ingest cur with SHARDS=N processes -> one shared DB
|
||||
@for i in $$(seq 0 $$(($(SHARDS) - 1))); do \
|
||||
$(ABORIST) --db $(DB) ingest --source wikipedia_cur --path $(WP_CUR) \
|
||||
--shard $$i/$(SHARDS) & \
|
||||
done; wait
|
||||
|
||||
ingest-old-parallel: bootstrap fetch-old ## ingest old history with SHARDS=N processes -> one shared DB
|
||||
@for i in $$(seq 0 $$(($(SHARDS) - 1))); do \
|
||||
$(ABORIST) --db $(DB) ingest --source wikipedia_old --path $(WP_OLD) \
|
||||
--shard $$i/$(SHARDS) & \
|
||||
done; wait
|
||||
|
||||
# Phase 2: attach-forever sharding. Each shard owns its own SQLite file —
|
||||
# no WAL contention. Reads via `aborist --shards-dir <dir> <cmd>` attach
|
||||
# all shards as UNION views. "Merge cost" = 0.
|
||||
SHARDS ?= 4
|
||||
SHARDS_DIR ?= $(HOME)/.aborist/shards
|
||||
ingest-cur-attached: bootstrap fetch-cur ## sharded ingest, no WAL contention (Phase 2)
|
||||
@mkdir -p $(SHARDS_DIR)
|
||||
|
|
@ -256,15 +241,6 @@ bench-emergent: bootstrap ## blue-moon emergent stress test (3-word triangulatio
|
|||
bench-emergent-pending: bootstrap ## print log entries awaiting teacher review
|
||||
$(PY) scripts/bench_emergent.py --print-pending
|
||||
|
||||
bench-qa-quick: bootstrap ## fastest bench (1 sample × 5 questions × 3 modes; ~10s)
|
||||
PYTHONUNBUFFERED=1 $(PY) bench/qa_sweep.py \
|
||||
--questions bench/qa_questions_smoke.txt \
|
||||
--shards-dir $(SHARDS_DIR) \
|
||||
--out-dir $(BENCH_QA_OUT) \
|
||||
--top-k $(QUERY_TOP_K) \
|
||||
--modes $(BENCH_QA_MODES) \
|
||||
--n 1 \
|
||||
--concurrency $(BENCH_QA_CONCURRENCY)
|
||||
|
||||
verify-shards: bootstrap ## cross-shard Merkle round-trip on a random sample
|
||||
$(ABORIST) --shards-dir $(SHARDS_DIR) verify -n $(VERIFY_N)
|
||||
|
|
@ -289,10 +265,6 @@ chain-check-shards: ## audit-chain break count for every *.db in $(SHARDS_DIR)
|
|||
printf '%s ' "$$db"; sqlite3 "$$db" "$$CHAIN_CHECK_SQL"; \
|
||||
done
|
||||
|
||||
# Sequential per-shard distill (one process iterates all shards).
|
||||
distill-shards: bootstrap ## distill every shard in $(SHARDS_DIR), surface -> depth=1 cores
|
||||
$(ABORIST) --shards-dir $(SHARDS_DIR) distill --process first-sentence-v1 --kind surface
|
||||
|
||||
# Parallel per-shard distill: one process per shard. No DB contention
|
||||
# because each shard is its own file.
|
||||
distill-shards-parallel: bootstrap ## one distill process per shard (parallel)
|
||||
|
|
@ -315,12 +287,6 @@ ingest: ingest-cur ## default ingest = cur (use ingest-old or *-parallel for ful
|
|||
# `ttl/30d/export_data/<user-id>/prod-grok-backend.json`). The source class
|
||||
# auto-walks down to find the JSON.
|
||||
GROK_EXPORT ?= $(HOME)/Downloads/ab8ef1f0-0d08-4f87-89c2-d4509e18115b
|
||||
ingest-grok: bootstrap ## ingest Grok conversations into $(DB) (single-DB mode)
|
||||
$(ABORIST) --db $(DB) ingest --source grok_export --path $(GROK_EXPORT)
|
||||
|
||||
ingest-grok-media: bootstrap ## ingest Grok media-generation prompts into $(DB)
|
||||
$(ABORIST) --db $(DB) ingest --source grok_media --path $(GROK_EXPORT)
|
||||
|
||||
# Grok lives in its own shard inside the attach-forever cluster so cross-
|
||||
# shard queries (`make query`) see it alongside the Wikipedia 2003 corpus.
|
||||
# Single shard (rank 0/1) — Grok conversations are private and small.
|
||||
|
|
|
|||
|
|
@ -3,8 +3,9 @@
|
|||
Convention: every documented target has the form
|
||||
target-name: deps ## one-line description
|
||||
|
||||
This extension parses the project's Makefile and writes an RST page
|
||||
listing every documented target, grouped by category prefix.
|
||||
Targets are grouped into workflow phases (manual mapping below) rather
|
||||
than alphabetical prefix, because phase tells a new reader the order
|
||||
they'd actually run things.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
|
@ -12,29 +13,128 @@ import re
|
|||
|
||||
TARGET_RE = re.compile(r"^([a-zA-Z0-9_-]+):.*?##\s*(.*)$")
|
||||
|
||||
# Workflow phases. Order = the order a new operator usually runs them.
|
||||
# Targets not listed here go into UNCATEGORIZED so we notice on review.
|
||||
PHASES: list[tuple[str, str, list[str]]] = [
|
||||
(
|
||||
"Setup",
|
||||
"One-time installation. Creates the venv and (optionally) the "
|
||||
"crawler's heavy extras. Re-running is a no-op when up to date.",
|
||||
["bootstrap", "bootstrap-crawler", "all"],
|
||||
),
|
||||
(
|
||||
"Fetch",
|
||||
"Download corpus snapshots into ``data/``. Idempotent — ``curl`` "
|
||||
"skips files already present.",
|
||||
["fetch", "fetch-cur", "fetch-old", "fetch-xml", "fetch-abstract"],
|
||||
),
|
||||
(
|
||||
"Ingest",
|
||||
"Parse a source into Merkle-committed shards. ``-attached`` "
|
||||
"variants are the canonical sharded path (one SQLite per shard, "
|
||||
"no WAL contention). Single-DB variants are for experiments.",
|
||||
[
|
||||
"ingest",
|
||||
"ingest-cur",
|
||||
"ingest-cur-attached",
|
||||
"ingest-old",
|
||||
"ingest-old-attached",
|
||||
"ingest-xml",
|
||||
"ingest-xml-history",
|
||||
"ingest-xml-attached",
|
||||
"ingest-abstract",
|
||||
"ingest-grok-attached",
|
||||
"ingest-grok-media-attached",
|
||||
"ingest-self",
|
||||
"ingest-self-providence",
|
||||
"ingest-git",
|
||||
"ingest-hg",
|
||||
"crawl-ingest",
|
||||
"recrawl-check",
|
||||
],
|
||||
),
|
||||
(
|
||||
"Distill",
|
||||
"Surface → core distillation. Cores are Merkle-bound back to "
|
||||
"their source chunks via inclusion proofs.",
|
||||
[
|
||||
"distill-shards-parallel",
|
||||
"distill-shards-tfidf-parallel",
|
||||
"backfill-concepts",
|
||||
],
|
||||
),
|
||||
(
|
||||
"Query",
|
||||
"Ask the corpus a question. ``query-dry`` skips the LLM call "
|
||||
"and returns the assembled context — useful for prompt iteration.",
|
||||
["query", "query-dry", "search"],
|
||||
),
|
||||
(
|
||||
"Verify and inspect",
|
||||
"Round-trip Merkle proofs, audit chain integrity, sidecar "
|
||||
"diagnostics on cached answers.",
|
||||
[
|
||||
"verify",
|
||||
"verify-shards",
|
||||
"chain-check",
|
||||
"chain-check-shards",
|
||||
"analyze-shards",
|
||||
"stats",
|
||||
"stats-shards",
|
||||
"activity",
|
||||
"inspect",
|
||||
],
|
||||
),
|
||||
(
|
||||
"Operations on cached records",
|
||||
"Mark a record falsified (audit-preserving) or burn it from "
|
||||
"the database (refuses if it has children unless ``FORCE=1``).",
|
||||
["falsify", "burn", "burn-kindergarten"],
|
||||
),
|
||||
(
|
||||
"Tests and benches",
|
||||
"Default test suite excludes opt-in crawler tests. ``bench-qa`` "
|
||||
"runs the full QA-quality sweep; ``bench-qa-smoke`` is the 5-question "
|
||||
"fast loop.",
|
||||
[
|
||||
"test",
|
||||
"test-crawler",
|
||||
"test-live",
|
||||
"bench",
|
||||
"bench-qa",
|
||||
"bench-qa-smoke",
|
||||
"bench-emergent",
|
||||
"bench-emergent-pending",
|
||||
],
|
||||
),
|
||||
(
|
||||
"Docs",
|
||||
"Render diagrams (graphviz) and build the Sphinx API reference. "
|
||||
"RTD rebuilds on push; these targets are for local previews.",
|
||||
["docs", "docs-api", "docs-api-clean"],
|
||||
),
|
||||
(
|
||||
"Clean",
|
||||
"Reversible by re-running the matching ``bootstrap`` / ``fetch`` "
|
||||
"/ ``ingest`` target. ``clean-data`` deletes the largest payload "
|
||||
"(downloaded dumps).",
|
||||
["clean", "clean-db", "clean-data", "help"],
|
||||
),
|
||||
]
|
||||
|
||||
def parse_makefile(makefile_path: Path) -> list[tuple[str, str]]:
|
||||
"""Return list of (target, description) tuples from a Makefile."""
|
||||
targets = []
|
||||
|
||||
def parse_makefile(makefile_path: Path) -> dict[str, str]:
|
||||
"""Return {target: description} from a Makefile."""
|
||||
targets = {}
|
||||
for line in makefile_path.read_text(encoding="utf-8").splitlines():
|
||||
m = TARGET_RE.match(line)
|
||||
if m:
|
||||
targets.append((m.group(1), m.group(2).strip()))
|
||||
targets[m.group(1)] = m.group(2).strip()
|
||||
return targets
|
||||
|
||||
|
||||
def categorize(targets: list[tuple[str, str]]) -> dict[str, list[tuple[str, str]]]:
|
||||
"""Group targets by leading prefix (e.g. 'fetch-', 'ingest-', 'docs-')."""
|
||||
groups: dict[str, list[tuple[str, str]]] = {}
|
||||
for name, desc in targets:
|
||||
prefix = name.split("-", 1)[0] if "-" in name else name
|
||||
groups.setdefault(prefix, []).append((name, desc))
|
||||
return groups
|
||||
|
||||
|
||||
def generate_rst(targets: list[tuple[str, str]], output_path: Path) -> None:
|
||||
"""Write a single RST page listing every documented target."""
|
||||
groups = categorize(targets)
|
||||
def generate_rst(all_targets: dict[str, str], output_path: Path) -> None:
|
||||
"""Write a single RST page grouping every target by workflow phase."""
|
||||
lines = [
|
||||
"Makefile reference",
|
||||
"==================",
|
||||
|
|
@ -43,23 +143,62 @@ def generate_rst(targets: list[tuple[str, str]], output_path: Path) -> None:
|
|||
"auto-generated from the project ``Makefile``'s ``## description``",
|
||||
"annotations at Sphinx build time, so it stays in sync with the source.",
|
||||
"",
|
||||
"Run ``make help`` locally for the same listing.",
|
||||
"Run ``make help`` locally for a flat alphabetized listing.",
|
||||
"",
|
||||
".. note::",
|
||||
"",
|
||||
" Targets are grouped below by **workflow phase**, in the order a",
|
||||
" new operator typically runs them. The first row of each table is",
|
||||
" the most common entry point for that phase.",
|
||||
"",
|
||||
]
|
||||
for prefix in sorted(groups):
|
||||
lines.append(prefix)
|
||||
lines.append("-" * len(prefix))
|
||||
|
||||
used: set[str] = set()
|
||||
for phase_title, blurb, target_names in PHASES:
|
||||
# Filter to targets that actually exist in the parsed Makefile.
|
||||
present = [(t, all_targets[t]) for t in target_names if t in all_targets]
|
||||
if not present:
|
||||
continue
|
||||
used.update(t for t, _ in present)
|
||||
|
||||
lines.append(phase_title)
|
||||
lines.append("-" * len(phase_title))
|
||||
lines.append("")
|
||||
lines.append(blurb)
|
||||
lines.append("")
|
||||
lines.append(".. list-table::")
|
||||
lines.append(" :widths: 25 75")
|
||||
lines.append(" :widths: 30 70")
|
||||
lines.append(" :header-rows: 1")
|
||||
lines.append("")
|
||||
lines.append(" * - Target")
|
||||
lines.append(" - Description")
|
||||
for name, desc in sorted(groups[prefix]):
|
||||
lines.append(f" * - ``{name}``")
|
||||
for name, desc in present:
|
||||
lines.append(f" * - ``make {name}``")
|
||||
lines.append(f" - {desc}")
|
||||
lines.append("")
|
||||
|
||||
# Surface anything we forgot to categorize so it shows up in review.
|
||||
leftover = [(t, d) for t, d in all_targets.items() if t not in used]
|
||||
if leftover:
|
||||
lines.append("Uncategorized")
|
||||
lines.append("-------------")
|
||||
lines.append("")
|
||||
lines.append(
|
||||
"Targets not yet placed in a workflow phase. If you see one here, "
|
||||
"add it to ``docs/_source/_ext/makefile_targets.py`` ``PHASES``."
|
||||
)
|
||||
lines.append("")
|
||||
lines.append(".. list-table::")
|
||||
lines.append(" :widths: 30 70")
|
||||
lines.append(" :header-rows: 1")
|
||||
lines.append("")
|
||||
lines.append(" * - Target")
|
||||
lines.append(" - Description")
|
||||
for name, desc in sorted(leftover):
|
||||
lines.append(f" * - ``make {name}``")
|
||||
lines.append(f" - {desc}")
|
||||
lines.append("")
|
||||
|
||||
output_path.write_text("\n".join(lines), encoding="utf-8")
|
||||
|
||||
|
||||
|
|
@ -71,4 +210,4 @@ def setup(app):
|
|||
if makefile.exists():
|
||||
targets = parse_makefile(makefile)
|
||||
generate_rst(targets, output)
|
||||
return {"version": "1.0", "parallel_read_safe": True}
|
||||
return {"version": "1.1", "parallel_read_safe": True}
|
||||
|
|
|
|||
180
docs/_source/concepts.rst
Normal file
180
docs/_source/concepts.rst
Normal file
|
|
@ -0,0 +1,180 @@
|
|||
Concepts
|
||||
========
|
||||
|
||||
Aborist is a content-addressed document store that gives every cached
|
||||
LLM answer a verifiable Merkle proof tying it back to its source
|
||||
documents. This page is the orientation: what the system is, the
|
||||
core abstractions you'll see in code and docs, and how they compose.
|
||||
|
||||
What aborist is
|
||||
---------------
|
||||
|
||||
A reference implementation of two papers stacked:
|
||||
|
||||
1. **Merkle Providence Reverse RAG** (whitepaper, April 2026) — the
|
||||
runtime: question → cache → retrieval → LLM → verifier → cache write.
|
||||
2. **Merkle-AGI v9.8** — the admissibility ledger that makes cached
|
||||
answers cryptographically verifiable across peers.
|
||||
|
||||
Three layers stacked on one SQLite file (per shard):
|
||||
|
||||
- **Surface** — ingested documents (Wikipedia, HTML pages, anything
|
||||
with a URI). Chunked, Merkle-rooted, FTS5-indexed.
|
||||
- **Core** — distilled documents Merkle-bound back to surfaces via
|
||||
per-chunk inclusion proofs. Recursive — a depth-N core can compress
|
||||
into a depth-N+1 core.
|
||||
- **Providence cache** — Q&A records keyed on the v9.8 8-dimension
|
||||
invariant; every record carries an audit_mode and a Merkle proof.
|
||||
|
||||
.. figure:: diagrams/aborist-modules.svg
|
||||
:alt: Aborist module graph
|
||||
:width: 100%
|
||||
|
||||
Top-level module graph. Substrate (merkle, document) at the bottom;
|
||||
pipelines (ingest, distill, qa) in the middle; CLI on top.
|
||||
|
||||
The Merkle commitment
|
||||
---------------------
|
||||
|
||||
Two peers that ingest the same source + run the same chunker +
|
||||
canonicalization compute **bit-identical** ``document_root`` hashes.
|
||||
That is the v9.8 admissibility property: identity by content, not by
|
||||
location.
|
||||
|
||||
Aborist uses fox's existing Go Merkle conventions verbatim
|
||||
(``proxy.unturf.com/pkg/verified/merkle.go``):
|
||||
|
||||
- **Leaf hash:** ``sha256(0x00 || canonical_chunk_bytes)``.
|
||||
- **Internal hash:** ``sha256(0x03 || left || right)``. Non-commutative —
|
||||
``H(L,R) ≠ H(R,L)``.
|
||||
- **Odd layers self-duplicate** the trailing element. Not zero-pad.
|
||||
- **Proof carries explicit IsLeft flag** per sibling. Not lexical sort.
|
||||
|
||||
See :doc:`api/substrate` for the Python port.
|
||||
|
||||
The 8-dim cache key
|
||||
-------------------
|
||||
|
||||
Every providence record is keyed on:
|
||||
|
||||
.. code-block::
|
||||
|
||||
cache_key = sha256(
|
||||
source_root | question_hash | model_profile_hash |
|
||||
conversation_hash | governance_policy_hash |
|
||||
schema_version | canonicalization_version | chunking_version
|
||||
)
|
||||
|
||||
Bumping any of the eight dimensions invalidates prior records on
|
||||
lookup. This is how the system stays honest across model changes,
|
||||
schema migrations, prompt edits, etc. — old answers don't silently
|
||||
serve under new conditions.
|
||||
|
||||
The audit chain
|
||||
---------------
|
||||
|
||||
Every state-changing operation writes one row in ``audit_events``
|
||||
with ``event_hash = sha256(prev_event_hash || canonical(body))``.
|
||||
Linear chain per shard, verified by ``make chain-check-shards`` (any
|
||||
break is the loudest possible signal).
|
||||
|
||||
Always write via ``aborist.store.append_audit`` — never insert into
|
||||
``audit_events`` directly.
|
||||
|
||||
The trichotomy and the four-rung ladder
|
||||
----------------------------------------
|
||||
|
||||
Every answer carries two stacked labels.
|
||||
|
||||
**Schema layer** — v9.8 trichotomy, persisted, drives cache lookups
|
||||
and the audit chain:
|
||||
|
||||
============= =============================================================
|
||||
``audit_mode`` meaning
|
||||
============= =============================================================
|
||||
STRICT every evidence unit verifies against context
|
||||
HYBRID some claims source-grounded, some emerged from training
|
||||
UNGROUNDED no evidence, or none verifies — purely emergent
|
||||
============= =============================================================
|
||||
|
||||
**Display layer** — four-rung ladder for claim-lattice modes only;
|
||||
renderer-only transformation, schema unchanged:
|
||||
|
||||
==================== ==========================================================
|
||||
rung what's actually proved
|
||||
==================== ==========================================================
|
||||
EVIDENCE-WARRANTED pointer verified + warrant ran & passed + no soft demotes
|
||||
ANCHOR-WARRANTED pointer-linked + warrant passed; soft-demote violations
|
||||
POINTER-LINKED pointer/source/chunk verified, but warrant didn't apply
|
||||
UNGROUNDED no verified pairs
|
||||
==================== ==========================================================
|
||||
|
||||
The point of the display ladder: ``STRICT`` in claim-lattice mode
|
||||
is *not* "the answer is correct" — it's "every pointer resolved to a
|
||||
valid evidence object, source_role allowed, citation coverage passed."
|
||||
The display label spells out the actual property so users don't read
|
||||
``STRICT`` as full semantic entailment.
|
||||
|
||||
.. figure:: diagrams/verifier-ladder.svg
|
||||
:alt: Verifier ladder
|
||||
:width: 80%
|
||||
|
||||
How ``(audit_mode, violations)`` maps to a display rung at render time.
|
||||
|
||||
The verifier (lexical, layered)
|
||||
-------------------------------
|
||||
|
||||
Five strategies run in sequence; first to find evidence classifies.
|
||||
Each is **lexical** (substring or token coverage), never embeddings —
|
||||
soft signals stay out of the proof path.
|
||||
|
||||
1. **quote** — model wraps claims in ``"..."``; verbatim substring tested
|
||||
2. **span** — bullet/sentence units substring-tested
|
||||
3. **entity** — multi-word proper nouns with proximity-cluster gating
|
||||
4. **paraphrase** — ≥85% token coverage on prose-shaped spans
|
||||
5. **claim_lattice** — pointer-line ``[E1,E2]`` or JSON; runs **seven
|
||||
deterministic hard checks**:
|
||||
parser succeeded, evidence_id resolves, source_role allowed, claim
|
||||
text non-empty, citation coverage threshold, pointer count cap,
|
||||
anchor-class warrant.
|
||||
|
||||
See :doc:`api/qa` for the implementation; the whitepaper §13.8
|
||||
covers the layered design.
|
||||
|
||||
Falsification state
|
||||
-------------------
|
||||
|
||||
Every providence record carries a ``falsification_state`` in
|
||||
``{live, failed, stale, quarantined}``. Cache lookups filter on
|
||||
``state='live'``. Drift detection (re-ingest produces a different
|
||||
``document_root``) flips the record to ``stale``.
|
||||
|
||||
Two ways to remove a wrong answer:
|
||||
|
||||
- ``make falsify KEY=...`` — record stays in DB, state flips to
|
||||
``failed``. Audit-preserving. Use when downstream consumers might
|
||||
reference it.
|
||||
- ``make burn KEY=...`` — actually deletes the row. Refuses if the
|
||||
record has children unless ``FORCE=1``. Use during scratch corpus
|
||||
building.
|
||||
|
||||
Sidecar diagnostics
|
||||
-------------------
|
||||
|
||||
The verifier stays binary; soft signals proliferate as **sidecars** —
|
||||
read-only diagnostic functions that pull the same source chunks the
|
||||
verifier saw, classify spans, detect topic shifts, surface metaphor
|
||||
framing, and **never write to providence_cache or extend the audit
|
||||
chain**. That invariant is what keeps ``audit_mode`` a binary
|
||||
classification rather than a soft score.
|
||||
|
||||
See :doc:`api/qa` (``aborist.qa.inspect``).
|
||||
|
||||
Where to go next
|
||||
----------------
|
||||
|
||||
* :doc:`quickstart` — install and run the canonical paths
|
||||
* :doc:`api/makefile` — every workflow as a ``make`` target
|
||||
* :doc:`api/substrate` — Merkle tree + document primitives
|
||||
* :doc:`api/qa` — Q&A pipeline, verifier, evidence map, run-DAG
|
||||
* :doc:`license` — full AGPL + Permacomputer Preamble
|
||||
143
docs/_source/cookbook.rst
Normal file
143
docs/_source/cookbook.rst
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
Cookbook
|
||||
========
|
||||
|
||||
Recipes for common workflows beyond the quickstart. Each starts from
|
||||
a working aborist install (``make bootstrap`` already run) and a
|
||||
populated shards directory under ``~/.aborist/shards/``.
|
||||
|
||||
Re-crawl a website to detect changes
|
||||
-------------------------------------
|
||||
|
||||
After ``make crawl-ingest`` lands a site into a shard, the per-page
|
||||
ETag and Last-Modified headers are kept in ``document_http_meta``. A
|
||||
re-crawl can ask "did anything change?" without downloading bodies.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
make crawl-ingest URL=https://example.com DEPTH=2 # initial crawl
|
||||
# ...later...
|
||||
make recrawl-check DOMAIN=example.com # conditional HEAD per page
|
||||
|
||||
Each URL is classified ``fresh`` (304), ``stale`` (200 with new body),
|
||||
``gone`` (404/410), or ``unreachable``. One tiny round-trip per URL,
|
||||
no body transfer when content is unchanged.
|
||||
|
||||
Falsify a wrong answer (audit-preserving)
|
||||
------------------------------------------
|
||||
|
||||
The verifier called something STRICT but you know it's wrong. Mark
|
||||
the record falsified — it stays in the DB so downstream consumers
|
||||
that referenced it can still trace history.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
make query Q="When did X happen?" # see the answer + cache_key
|
||||
make inspect KEY=<cache_key> # diagnose unverified spans
|
||||
make falsify KEY=<cache_key> REASON='wrong year — sources cited 1942 not 1944'
|
||||
|
||||
Future lookups skip records whose ``falsification_state != 'live'``.
|
||||
A ``falsify`` audit event records the act; the chain stays intact.
|
||||
|
||||
Promote your own past answers into the corpus
|
||||
----------------------------------------------
|
||||
|
||||
After enough STRICT answers accumulate, treat them as a derived
|
||||
source. The ``providence`` ingest path promotes mature STRICT records
|
||||
into the document corpus where retrieval can pick them up.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
make ingest-self-providence KG_SECONDS=86400 # only records ≥1 day old
|
||||
|
||||
The ``KG_SECONDS`` (kindergarten window) prevents the system from
|
||||
trusting freshly-cached answers as ground truth before they've had
|
||||
time to fail. See :doc:`api/qa` for the ``ProvidenceSource`` impl.
|
||||
|
||||
Query across mixed corpora
|
||||
--------------------------
|
||||
|
||||
Every shard under ``~/.aborist/shards/`` is queried automatically.
|
||||
Mix Wikipedia, your Grok export, a crawled site, and your own git
|
||||
repos in one query — retrieval ranks across all of them.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
make ingest-cur-attached # Wikipedia
|
||||
make ingest-grok-attached GROK_EXPORT=$HOME/Downloads/<uuid> # Grok chats
|
||||
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 # personal site
|
||||
make ingest-git GIT_REPO=$HOME/git/myproject # source code
|
||||
|
||||
make query Q="how does my project handle authentication?"
|
||||
|
||||
Each shard contributes hits; the query path's title-relevance + body
|
||||
coverage rerank lets neologisms in your private corpus outrank generic
|
||||
Wikipedia matches.
|
||||
|
||||
Override the LLM endpoint
|
||||
-------------------------
|
||||
|
||||
Default points at ``https://hermes.ai.unturf.com/v1`` (Hermes-3-8B,
|
||||
no auth). Point at any OpenAI-compatible endpoint via env:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
export ABORIST_LLM_ENDPOINT="https://your-vllm.example.com/v1"
|
||||
export ABORIST_LLM_MODEL="meta-llama/Llama-3.1-70B-Instruct"
|
||||
export ABORIST_LLM_API_KEY="..." # optional; many vLLM deploys are open
|
||||
make query Q="..."
|
||||
|
||||
The model id folds into ``model_profile_hash`` (one of the 8 cache
|
||||
key dimensions), so swapping models invalidates prior cache hits on
|
||||
lookup — no risk of serving an answer one model produced under
|
||||
another model's identity.
|
||||
|
||||
Verify shard integrity after a bulk operation
|
||||
----------------------------------------------
|
||||
|
||||
Any state-changing op (mass falsify, hash bump, schema migration)
|
||||
should be followed by:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
make chain-check-shards # 0 chain breaks per shard = intact
|
||||
make analyze-shards # compression spectrum + audit integrity
|
||||
make verify-shards # round-trip Merkle proofs on a sample
|
||||
|
||||
Chain breaks are the loudest possible signal. Run these before
|
||||
declaring an op successful.
|
||||
|
||||
Run the QA bench and read the results
|
||||
--------------------------------------
|
||||
|
||||
The QA bench measures how often the verifier says STRICT vs HYBRID
|
||||
vs UNGROUNDED across a fixed question set, per answer mode.
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
make bench-qa-smoke # ~30s, 5 questions × 3 modes
|
||||
make bench-qa BENCH_QA_N=3 # full sweep, 3 samples each
|
||||
|
||||
Output lands in ``bench/qa_results/<utc-stamp>.{jsonl,md}``. The
|
||||
markdown file has the summary table; the JSONL has every per-question
|
||||
record for drill-down.
|
||||
|
||||
Resume an interrupted bench:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
make bench-qa --resume bench/qa_results/<previous-utc-stamp>.jsonl
|
||||
|
||||
Same ``--seed`` is required for shuffled-task-order alignment.
|
||||
|
||||
Tune retrieval per-question
|
||||
---------------------------
|
||||
|
||||
When a query returns the wrong sources, ``K=`` injects extra retrieval
|
||||
keywords without changing what the LLM sees as the question:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
make query Q="What did Orwell mean by always at war?" K="1984 Oceania Eastasia"
|
||||
|
||||
Provenance gap on this is tracked in
|
||||
:doc:`api/qa` (``aborist.qa.query``).
|
||||
1
docs/_source/diagrams
Symbolic link
1
docs/_source/diagrams
Symbolic link
|
|
@ -0,0 +1 @@
|
|||
../diagrams
|
||||
|
|
@ -10,6 +10,8 @@ Contents:
|
|||
:caption: Getting started
|
||||
|
||||
quickstart
|
||||
concepts
|
||||
cookbook
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 3
|
||||
|
|
|
|||
|
|
@ -72,6 +72,10 @@ After the answer
|
|||
The query path
|
||||
--------------
|
||||
|
||||
.. figure:: diagrams/query-pipeline.svg
|
||||
:alt: Query pipeline
|
||||
:width: 100%
|
||||
|
||||
Every query runs through the same pipeline:
|
||||
|
||||
1. **Search** — FTS5 (body) + SQL ``LIKE`` (title) + ``JOIN`` over
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue