A new π* kernel that canonicalizes pure-integer counting
expressions and FAILS CLOSED on any input whose result isn't a
non-negative sp.Integer. Tighter domain than algebra-symbolic@v1,
which already accepts the same input surface but happily returns
symbolic / negative / non-integer outputs.
Distinguishing feature versus algebra-symbolic@v1:
algebra-symbolic@v1: binomial(n, k) → "binomial(n, k)" (symbolic
passthrough)
combinatorics@v1: binomial(n, k) → PiStarError (fail-closed
on free-symbol output)
algebra-symbolic@v1: binomial(Rational(1,2), 3) → 1/16 (rational)
combinatorics@v1: binomial(Rational(1,2), 3) → PiStarError
(output not Integer)
Boundary kept explicit: binomial(-3, 2) = 6 IS accepted because the
output is an integer 6. The fail-closed rule is on output shape
(Integer ≥ 0), not input range. Documented as
test_generalized_binomial_negative_args_accepted_when_integer.
Output format: plain decimal literal (b"10", b"5040"). Composes
with arithmetic@v1 for byte-identical agreement with the rational
route (b"10/1") so the multi-modality witness (#000028) can pin
equivalence-class agreement when both routes fire on the same
question.
Allowed surface (via SymPy primitives): binomial, factorial, ff /
rf (falling/rising), catalan, bell, partition, stirling, plus
arithmetic compositions over those primitives
(3*binomial(5,2) + factorial(4) = 54).
Coverage:
- 43 unit tests including binomial symmetry C(n,k)=C(n,n-k),
Pascal's rule C(n,k)=C(n-1,k-1)+C(n-1,k), the C(n,k) =
factorial(n)/(factorial(k)·factorial(n-k)) identity,
fail-closed paths (symbolic/negative/non-integer/relational/
parse), round-trip idempotence, composition with arithmetic@v1.
- 10 syntax + 12 semantics bench fixtures, 100% pass.
- bench/batteries/base.py PHASE_1_CARRIERS gains "combinatorics".
- Makefile bench-5s-combinatorics target.
All gate on pytest.importorskip("sympy") so a sympy-less suite
stays green. Full make test: 1537 passed / 28 skipped.
Sequencing rationale honored: this kernel lands FIRST so that
#000033 (claim-pack pillar VII for combinatorics) can bind its
records to the tighter integer kernel from day one — avoids
rebind churn on pi_star_ref fields.
Three small streams in one commit:
#000028 follow-up — witness divergence → 5F fixtures
=====================================================
Witness fan-out now writes a `providence_canonical_witness` audit
event when it fires (next to the capital-ledger record landed in
708aa45). Body carries pi_star_ref, question_text, agreement_label,
canonical_answer_text, llm_raw_text, llm_canonical_bytes,
cache_status. Best-effort write — chain failure never fails the
query.
New extractor `bench/scripts/witness_to_5f.py` reads those events
from a qa.db and writes them out as 5F-Falsification fixtures
matching the existing `falsification-live-v1` schema. Filtering
includes only divergence labels (LLM-DIVERGED / KERNEL-LLM-DIVERGED
/ CACHE-DRIFT); skips KERNEL-LLM-AGREE / STRICT-WITNESSED (no
calibration signal) and KERNEL-ONLY (LLM unparseable, not a
supervised-correction sample).
Idempotent: sorted by audit-event seq, so re-running against the
same qa.db produces byte-equal fixture files. The existing
fixture-digest discipline stays valid.
Makefile: `make bench-witness-divergence` (override default
qa.db / output path via WITNESS_QA_DB / WITNESS_OUT env-vars).
Closes the divergence → calibration data loop the witness ticket
imagined: every LLM hallucination on a canonical-shape question
becomes a supervised-correction fixture downstream prompt
improvements can grade against.
#000030 Phase 7 demo — function-sampled@v1 end-to-end
======================================================
`bench/scripts/demo_plot.py` — closes the loop on opencompletion's
activity24-math-plot.yaml. SymPy expression → quantized
integer-vector signature (canonical bytes) → optional matplotlib
PNG. Canonical bytes are the proof; PNG is just a downstream view
of the same evidence.
$ make demo-plot Q='sin(x)' PNG=/tmp/sin.png
Output JSON contains canonical_bytes_sha256 + canonical_bytes_preview
+ canonical_bytes_total_chars + grid metadata + the optional png_path.
matplotlib is gated — when absent, --png prints a warning to stderr
and skips the render; the canonical bytes still print. Tests skip
the PNG-presence assertion via `pytest.importorskip("matplotlib")`.
Public docs polish (#7)
========================
- docs/_source/bench.rst: updated fixture-count narrative (~660 →
662 default tasks + ~110 math π* fixtures); `make` quick-reference
now lists all per-π* 5S targets (tabular, calculus-limit/series,
linear-algebra, function-sampled) plus bench-real-shard,
bench-fork-baseline/score, bench-witness-divergence.
- docs/_source/v8-fork-score.rst: CLI section gained --out flag
documentation + a Make-harness sub-section covering
bench-fork-baseline / bench-fork-score / FORK_PARENT/CHILD/REPORT
env-vars.
Tests
=====
- tests/test_witness_to_5f.py — 8 new tests covering the audit-event
write (3) + extractor logic (5).
- tests/test_demo_plot.py — 6 new tests covering canonical-bytes
determinism + equivalence-class collapse + matplotlib gating.
Full suite: 1624 passed, 37 skipped (was 1568; +56).
tabular-pinned@v1 + calculus-limit@v1 + calculus-series@v1 +
linear-algebra@v1 + function-sampled@v1 — all reserved stubs
graduated; the π* registry is now 15 concrete kernels with no
remaining reserved-stub entries.
#000030 Phase 4 — calculus-limit@v1
====================================
sp.limit with thread-timeout. One-sided dir support (+/-/+-).
Pinned spelling for infinity cases: b"+oo" / b"-oo" / b"zoo"
(complex infinity) — bypasses sp.expand since Infinity isn't
algebraic. Finite results re-canonicalize through algebra-symbolic
recipe (sp.expand + sp.srepr). Unevaluated cases / timeouts emit
b"unevaluated:" + sp.srepr(<Limit>) sentinel, mirroring
calculus-integral's pattern.
#000030 Phase 5 — calculus-series@v1
=====================================
sp.series(f, x, x0, n).removeO() → sp.expand → sp.srepr. Drops
O(x**n) remainder explicitly so the canonical form is finite-byte.
Sentinel format mirrors limit/integral: b"unevaluated:Series(...)"
on timeout. n must be a positive int; 0 / float / negative rejected.
#000030 Phase 6 — linear-algebra@v1
====================================
Single π* covers the whole linear-algebra surface via {op, matrix}
JSON. Ops: rref / det / eigenvalues / inverse. Matrix cells go
through Fraction(Decimal(str(...))) for floats so 1, 1.0, "1.0"
all collapse to Rational(1, 1) — matching arithmetic@v1's
discipline. Without this fold, sp.sympify keeps floats as Float
(separate type) and downstream det/inverse return Float-shaped
bytes. Eigenvalues are sorted by srepr for determinism.
Output formats:
rref / inverse: rows/cols header + cells joined by | (rows by ||)
det: det:<num/den-or-srepr>
eigenvalues: eigenvalues:<value-1>x<mult-1>|...
#000030 Phase 7 — function-sampled@v1
======================================
Bridge to time-series-quantized@v1. SymPy expression + linspace
grid → quantized integer-vector signature in time-series's exact
output format (dt=...;dv=...;n=...;t0=0:v0|v1|...). Two functions
that render identically (within sample-grid tolerance) collapse
to the same canonical bytes. This is what plotting CAN become
in π* terms — the PNG render is a downstream view of the same
canonical evidence.
Math-only sampler (no numpy in the dep surface); Python's round()
is banker's-rounding so the bytes are interchangeable with
time-series-quantized@v1's output. Complex / non-finite samples
raise PiStarError rather than silently dropping imaginary parts.
tabular-pinned@v1 — last reserved stub graduates
=================================================
JSON-rows input ({schema, key_columns, rows}); declared
key_columns sort policy (stable sort by primary-key tuple);
type-fold per column (int/rational/bool through arithmetic@v1
discipline; str verbatim; bool normalized). Header case is
PINNED EXACT — Excel and PostgreSQL both care about case;
defaulting to lowercase-fold would break operator expectations.
Output: header (schema + key + n) + rows joined by \n + cells by |.
The π* registry has no remaining reserved stubs. Every modality
the substrate paper reserved is now real.
Test suite: 1568 passed (was 1467; +101). New closure-criterion
test (test_no_stub_pi_stars_remain) replaces the old reserved-stub
parametrize — adding a future stub re-opens this list.
110/110 fixtures pass across the 5 new bench-5s-* targets.
PHASE_1_CARRIERS gained calculus / linear-algebra / function-sampled
/ tabular.
Two new π* canonicalizers extend the math substrate above
arithmetic@v1 (closed-form rationals) and logic-kernel@v1
(propositional Boolean → CNF):
algebra-symbolic@v1 (Phase 1) — symbolic-algebra domain.
sp.expand → sp.srepr canonical bytes. Polynomial identity collapses
((x+1)**2 ≡ x**2 + 2*x + 1); exponential identity collapses
(exp(a+b) ≡ exp(a)*exp(b), inherited from sp.expand's default
behavior); trigonometric identity does NOT collapse
(sin²+cos² ≢ 1). The trig surface is reserved for a future
algebra-symbolic-simplified@v1 variant that wraps sp.simplify at
unbounded CPU cost. Rejects relationals (`x > 0`) and
BooleanFunction shapes (`x & y`) via `isinstance(expr, sp.Expr)` —
sp.Symbol confusingly inherits from Boolean so the right rejection
filter is "not Expr" rather than "Boolean".
calculus-derivative@v1 (Phase 2) — calculus domain. JSON-shaped
{f, x, n} input → sp.diff → sp.expand → srepr bytes. Output is
itself a valid algebra-symbolic@v1 input so the two compose
naturally under arborist.pi_star.compose. n defaults to 1; bools
explicitly rejected (Python isinstance(True, int) is True so we
filter that explicitly).
Optional dependency: sympy ships in the new [math] extra
(pyproject.toml). Folded into [dev] so make bootstrap pulls it
transitively. An explicit `bootstrap-math` Makefile target documents
the opt-in for minimal-install users. Both modules self-guard
via `try: import sympy as sp / except ImportError: sp = None` and
only register(...) when sympy is present, so a fresh checkout
without [math] still loads arborist.pi_star without raising.
Preflight algebra route lands in
arborist.qa.query._canonical_projection_preflight between the
arithmetic and logic routes. Charset regex (_CANONICAL_ALGEBRA_RE)
allows lowercase letters + math chars; requires at least one
letter (else arithmetic wins); rejects natural-language leading
verbs via _CANONICAL_ALGEBRA_NL_LEAD_RE (4-letter minimum so
single-/two-/three-char identifiers like x, xy, sin, cos, pi
survive while "simplify (...)", "factor x...", "expand (a+b)..."
fall through). PiStarError + KeyError both fall through cleanly
so a sympy-less install just routes everything past algebra.
Bench substrate:
- bench/batteries/base.py PHASE_1_CARRIERS gains "symbolic_algebra"
- bench/fixtures/5s/syntax-algebra-symbolic-v1.jsonl (10 fixtures)
- bench/fixtures/5s/semantics-algebra-symbolic-v1.jsonl (13 fixtures
including the documented trig non-collapse + exp collapse)
- Makefile bench-5s-algebra target → 100% pass
Tests: 18 algebra-symbolic + 38 calculus-derivative unit tests +
~10 new preflight-route tests in test_canonical_projection.py. All
gate on pytest.importorskip("sympy") so a sympy-less suite stays
green. Full suite: 1369 passed / 27 skipped.
Phases 3-7 (integral, limit, series, linear-algebra,
function-sampled) remain open as future work; each lands as its
own ticket when an actual consumer surfaces.
Two fan-out streams. Both bear directly on the ticket's "search
latency on real shards" headline finding.
## Lazy concept_relations loading
Phase 1 (migration memoization) cut SQLite executes 65% but warm-
cache wall barely moved. cProfile pinned the next hotspot:
synonym_expand 2.8 s × 2 calls + _load_token_idf 0.3 s. The eager
loader dumped all ~290 K concept_relations rows on first call —
the price of being able to answer ANY future question without
re-querying. Wrong tradeoff for single-query CLI use.
Refactored arborist/concepts/query.py:
- New _load_neighbors_for(shards_dir, tokens) — targeted
WHERE token IN (...) OR target IN (...) query. Returns just the
direct synonym neighborhood for the given tokens (~300 rows for
a typical 5-token question, vs 290 K for the full table).
- New _load_rivalry_rows(shards_dir) — process-wide cache of the
~2-row rivalry-relation set; near-zero cost.
- New _load_idf_for(shards_dir, tokens) — IDF only fetched when
expansion exceeds max_total (the cap). Most queries never reach
the truncation branch and skip IDF entirely.
- Per-token process-wide neighbor cache so multi-query bench scripts
don't re-query tokens already seen.
- synonym_expand and rivalry_excluded refactored to use the lazy
loaders. Eager _load_indices / _get_indices kept for any
back-compat caller; not used by hot paths.
- invalidate_cache() clears all three caches.
All 14 concept tests pass unchanged — the contract is preserved.
Re-profile of `who wrote virt-back?` against ~38 GB of real shards
(warm cache):
metric pre-fix post-Phase-1 post-lazy-concepts
wall_ms 14,500 13,400 9,300 (-36%)
search_ms 9,900 10,600 5,000 (-49%)
SQLite executes 10,623 3,687 3,708 ~same
synonym_expand 2,966 2,840 ~0 (lazy hit)
Search target was <5 s; we hit 5.0 s on the warm path. Cold cache
should drop further (the 290 K-row dump was disk-bound).
## Real-shard baseline artifact (Phase 2)
bench/scripts/real_shard_baseline.py — runs an 8-question fixture
through the full query() pipeline and emits:
- bench/results/real-shard-baseline.json (durable; commit_sha,
shards_fingerprint, per-query rows, summary)
- bench/results/real-shard-baseline.md (human-readable summary)
Question set in bench/fixtures/real-shard-baseline-v1.jsonl:
virt-back, France, Mac OS X, Linux, Microsoft, AMD/Intel
(rivalry path), and two canonical-projection cases (math + logic
preflight short-circuit).
First baseline run (commit e78814c plus this fan-out, BURN=1):
audit_mode n notes
STRICT 4 (virt-back, France, Mac OS X, Linux)
HYBRID 2 (Microsoft founder, AMD/Intel)
CANONICAL 2 (0.1+0.2, A IMPL B; <1 ms each)
wall median 4.1 s (range 0.6 ms – 7.2 s)
primary used 4 / 8
`who wrote virt-back?` lands at 6.3 s wall, audit STRICT, primary
source #1, cited evidence still includes a copyright footer
(reviewer's warrant-quality finding — deferred to follow-up ticket
since the latency fix was the gating concern).
Hard constraint preserved: baselines NEVER gate CI. The artifact
is for confirming a fix moved the needle, seeding ForkScore
comparisons, and noting findings worth tickets.
`make bench-real-shard` wires it. Honors ARBORIST_SHARDS_DIR.
## Test status
Full suite: 1306 passed, 36 skipped (no regressions from the
concept refactor; 14 concept tests cover the lazy/eager
equivalence).
Closes the math half of fox's 2026-05-08 roadmap question ("did we
implement the math and logic stuff?"). Until now arborist had logic
KERNELS (Syllogism, Truthtables, Transitivity sub-batteries with
deterministic evaluators) but no math π*'s. SQD whitepaper §14
framed math as "operations over π*-canonical invariant objects:
integers, algebraic expressions, proof states, constraint graphs"
— that framing is now actually implementable.
arithmetic@v1 — exact rational arithmetic (SQD §14.1)
-----------------------------------------------------
Parse arithmetic expression → fractions.Fraction → canonical
"<num>/<den>" in lowest terms. Decimals via Decimal(str(...)) for
exact rational interpretation:
"0.1" → "1/10" (not 0.1±ε)
"0.1+0.2" → "3/10" (the SQD-canonical floating-point question)
"1+2" → "3/1"
"(1+2)*3" → "9/1"
"6/4" → "3/2" (lowest terms via Fraction invariant)
"2**3" → "8/1"
"-(1+2)" → "-3/1"
Rejects: identifiers, function calls, division by zero, non-integer
exponents, boolean literals. PiStarError with explanatory message.
logic-kernel@v1 — propositional CNF canonicalizer (SQD §14.3)
-------------------------------------------------------------
Parse Boolean expression (AND/OR/NOT/XOR/IMPL/IFF + named atoms +
parens) → eliminate IMPL/IFF/XOR → push NOT inward (NNF) →
distribute OR over AND (CNF) → dedupe + sort literals + sort
clauses + drop tautological clauses → serialize.
Equivalence classes preserved:
A AND B ≡ B AND A (commutativity)
(A AND B) AND C ≡ A AND (B AND C) (associativity)
A IMPL B ≡ NOT A OR B (IMPL rewrite)
A IMPL B ≡ NOT B IMPL NOT A (contrapositive)
NOT (A AND B) ≡ NOT A OR NOT B (De Morgan)
NOT NOT A ≡ A (double negation)
A OR (B AND C) ≡ (A OR B) AND (A OR C) (distribution)
A OR NOT A → TRUE (tautology)
A AND A ≡ A (idempotence)
Atom cap: N=8 (256 max clauses). Larger inputs raise PiStarError;
CNF blow-up is exponential in atom count, the cap keeps
canonicalization deterministic in bounded time.
Surface
-------
- arborist/pi_star/arithmetic.py — full implementation (new file)
- arborist/pi_star/logic.py — full implementation (graduates from
stub; preserves the LogicKernelV1 dataclass shape so the registry
key arithmetic@v1 / logic-kernel@v1 stays stable)
- arborist/pi_star/__init__.py — imports arithmetic to auto-register
- bench/batteries/base.py — PHASE_1_CARRIERS adds "arithmetic" + "logic"
- bench/fixtures/5s/syntax-arithmetic-v1.jsonl (12 fixtures)
- bench/fixtures/5s/semantics-arithmetic-v1.jsonl (15 fixtures)
- bench/fixtures/5s/syntax-logic-v1.jsonl (12 fixtures)
- bench/fixtures/5s/semantics-logic-v1.jsonl (18 fixtures)
- Makefile: bench-5s-arithmetic, bench-5s-logic-kernel,
bench-5s-math aggregate
- tests/test_pi_star.py: 30 new tests
- 13 for arithmetic@v1 (decimals, equivalence, rejects, idempotency,
integer exponents, negative results, the SQD 0.1+0.2 case)
- 17 for logic-kernel@v1 (commutativity, associativity, all rewrite
rules, De Morgan, double-neg, distribution, tautology, idempotence,
contrapositive, atom cap, syntax errors, idempotency)
Cross-modality discipline now spans:
text + claim_lattice + memory + code + arithmetic + logic
— six carrier domains, all with real π* implementations.
Two stubs remain (time-series-quantized@v1, tabular-pinned@v1);
neither is needed for math/logic coverage.
Full suite: 1256 passed, 36 skipped (+30 tests, +57 fixtures).
Completes the bridge: every 5F sub-battery now has an embedded
(Phase 1a) AND a live (Phase 1b.2) path. Per-task detail.source
distinguishes synthetic from live signal.
| Sub-battery | Live surface | Live fixtures |
|---|---|---|
| Formulate | qa.parse_claims.parse_pointer_claims | 15 (prior) |
| Feedback Loop | store.append_audit + memory.snapshot (temp shard) | 12 (prior) |
| Function | parse_pointer_claims + shape evaluators | 12 |
| Finetuning | selfmodel.store_snapshot + claims_for round-trip | 10 |
| Falsification | qa.verify.verify_quotes (real verifier) | 12 |
Function live: input_text routes through real parse_pointer_claims;
the same Phase-1a shape evaluators (shape_match / pointer_set_match
/ threshold_on_metric) run on the live-parsed output. Tests
parser→shape pipeline against actual organism behavior.
Finetuning live: gated via fixture's "live": true flag. Runner
writes parent + child SelfModel + capability_claim to a fresh temp
shard via real arborist.selfmodel.store_snapshot, reads back via
claims_for, runs the same improvement check. Tests the SelfModel
persistence + claim-attach surface, not just static fixture data.
Falsification live: answer_text + context routes through real
arborist.qa.verify.verify_quotes. Live signals translate the
verifier's audit_mode + verifier_method + unverified_quotes into
flat tags (UNGROUNDED, STRICT_<method>, HYBRID_<method>,
UNVERIFIED_QUOTE) the runner matches against expected_reason. One
fixture (5f-fal-live-003) deliberately documents a known soft-
signal gap — the entity strategy treats Insulin/Penicillin claims
as equivalent because both share "Alexander Fleming". Production
catches it via title-relevance + claim-lattice verifier; the
fixture pins the soft-path limit so future verifier changes
re-trigger review.
Surface:
- bench/batteries/b_5f.py — _live_function_produced,
_live_finetuning_measure, _live_falsification_violations
helpers + two-mode dispatch in run_function / run_finetuning /
run_falsification.
- bench/fixtures/5f/{function,finetuning,falsification}-live-v1.jsonl
(12 + 10 + 12 fixtures).
- Makefile: bench-5f-{function,finetuning,falsification}-live
targets; bench-5f-live aggregate now covers all five sub-batteries.
- Tests: 9 new in tests/test_bench_batteries.py
- live + embedded paths for each sub-battery (6 tests)
- direct helper tests verifying the real arborist surfaces are
invoked, not stubs (3 tests)
Phase 1a fixture digests unchanged. _DEFAULT_FIXTURES still points
at synthetic Phase-1a fixtures so `runner --all` behavior is
identical and deterministic; live mode invoked via explicit
--fixtures path or make targets.
Full suite: 1210 passed, 36 skipped.
5F battery is now the first to bridge synthetic→live across every
sub-battery. The pattern + live fixture format are reusable for
5R Phase 1b.2 (when SelfModel-backed workspace ops want live
verification).
Second live wire-up (after Formulate). run_feedback_loop now supports
both modes:
- Embedded (Phase 1a): chain of (operation, observation) string pairs;
runner aggregates observations and string-matches expected_delta.
- Live (Phase 1b.2): live_chain of typed ops applied to a fresh temp
arborist shard via real append_audit + memory.snapshot +
selfmodel.snapshot. expected_delta is a dict of predicates against
the resulting audit_events / memory_branch_summaries.
The live helper _live_feedback_chain creates a tempfile-backed shard,
runs the chain through real arborist surfaces, and queries the final
state. Every audit event chains via the production append_audit, so
the audit chain is re-verifiable after live execution (see new test
test_5f_live_feedback_chain_audit_chain_intact).
Three predicate types in expected_delta:
- audit_event_type_present: named event_type appears in audit chain
- memory_branch_present: named branch_id in memory_branch_summaries
- body_substring_present: substring appears in any audit body JSON
Surface:
- bench/batteries/b_5f.py — _live_feedback_chain helper +
_live_delta_satisfied predicate checker; two-mode dispatch in
run_feedback_loop
- bench/fixtures/5f/feedback-loop-live-v1.jsonl — 12 live fixtures
exercising providence_write, providence_repair, memory_snapshot,
selfmodel_snapshot ops. Includes 2 negative fixtures testing the
predicate checker (expected_delta absent → expected:fail).
- Makefile: bench-5f-feedback-loop-live + bench-5f-live aggregate
for all 5F Phase-1b.2 live wire-ups.
- Tests: 5 new in tests/test_bench_batteries.py
- live path runs all 12 fixtures
- embedded path still works (10 Phase-1a fixtures)
- helper directly tests audit-event write
- rejects unknown live op type
- audit chain re-verifies after live ops
Two of the five 5F sub-batteries now bridge synthetic → live
(Formulate + Feedback Loop). Function/Finetuning/Falsification
follow in subsequent commits.
Full suite: 1201 passed, 36 skipped.
First sub-battery to bridge from synthetic gold output to actual
organism behavior. run_formulate now supports two fixture modes
selected per-task:
- Embedded (Phase 1a): produced_lattice in the fixture. 10 seed
fixtures continue to pass via this path.
- Live (Phase 1b.2): only input_text in the fixture; runner calls
arborist.qa.parse_claims.parse_pointer_claims(input_text) and
matches the live output against expected_lattice.
Embedded takes precedence if both fields are present. Per-task
detail.source ("embedded" | "live") surfaces in bench output so
synthetic vs live signal is distinguishable.
Surface:
- bench/batteries/b_5f.py — _live_produced_lattice helper +
two-mode dispatch in run_formulate
- bench/fixtures/5f/formulate-live-v1.jsonl — 15 live-mode fixtures
with input_text + expected_lattice (no produced_lattice)
- Makefile: bench-5f-formulate-live target
- Tests: 4 new in tests/test_bench_batteries.py
- live path routes through real parser, all 15 pass
- embedded path still works (10 Phase-1a fixtures)
- _live_produced_lattice helper directly verifies parse output
- fixture missing both fields fails cleanly with explanatory reason
Phase 1a fixture digests unchanged. _DEFAULT_FIXTURES still points
at formulate-v1.jsonl so `runner --all` behavior is identical;
live-mode fixtures invoked via explicit --fixtures path.
Full suite: 1196 passed, 36 skipped.
Pattern set. Function/Finetuning/Falsification/Feedback Loop
follow in subsequent commits.
Phase 2 of #000021. React/Rearrange/Restore/Replicate/Resonate over
the workspace surface — selfmodel_records (#000014) + memory_records
(#000017), both landed earlier today. Closes the gap that gated 5R
since the substrate work shipped.
Sub-battery semantics (per SQD whitepaper §9.3 + ticket #000021 §4.2):
- React: incorporate new fact/constraint. Workspace = (snapshot_t0,
snapshot_t1, expected_delta). Pass = added_facts present + removed_facts
absent in t+1.
- Rearrange: restructure without semantic shift. Re-canonicalize
different surface forms through a named π*; pass = bytes match
expected_equivalent flag. Tests the order-invariance contracts in
SelfModel (capability_claim_hashes sorted) and Memory (branches
sorted by branch_id).
- Restore: retrieve prior fact. Workspace = (history[], current_facts[]).
Pass = fact in current OR any historical snapshot.
- Replicate: independent canonical encodings via π*. Same input run
N times must yield byte-equal output. Tests determinism contract.
- Resonate: variance across N runs. Deterministic π*'s yield
distinct=1; expected_max_distinct=1 enforces zero-variance contract.
Surface:
- bench/batteries/b_5r.py (5 deterministic runners; no LLM-as-judge)
- bench/fixtures/5r/{react,rearrange,restore,replicate,resonate}-v1.jsonl
(30 each = 150 new fixtures)
- runner.py registers 5r in _BATTERIES + _DEFAULT_FIXTURES
- Makefile: bench-5r + bench-suite (5S+5T+5F+5R aggregate)
Final tally:
5S syntax/semantics/syllogism/synthesis/semiotics 108
5T transfer/transfer-learning/triangulation/... 154
5F function/finetuning/falsification/... 50
5R react/rearrange/restore/replicate/resonate 150
TOTAL: 462 fixtures across 21 sub-batteries — 100% pass.
Tests: 6 new in tests/test_bench_batteries.py + adjustment to
test_session_integration.py for the 312→462 count + 5R sub-battery
presence assertion. Full suite: 1192 passed, 36 skipped.
Closes#000021. Phase 3 (external-corpus expansion) remains open
under the ticket but does not gate closure — the complete
Dav1DPrometheus surface is now executable infrastructure.
Per fox's "partial punt on larger ones" — ships the bench/ skeleton +
small seed fixture sets so future v8/v7-W/SelfModel work can cite a
real fitness target. Full Phase 1 (50-200 fixtures per sub-battery)
and Phases 2-3 stay open in the ticket.
Phase 1a delivers:
- bench/batteries/{base,b_5s,b_5t,runner}.py — Battery protocol,
BatteryResult, fixture-digest helpers, CLI runner.
- Seed fixtures:
- bench/fixtures/5s/syntax-v1.jsonl — 10 tasks against
wikitext-base@v1 and claim-lattice@v1
- bench/fixtures/5s/semantics-v1.jsonl — 8 equivalence tasks
- bench/fixtures/5t/transfer-v1.jsonl — 4 paraphrase-invariance
tasks
- Runners for 5S Syntax, 5S Semantics, 5T Transfer. Other 5S/5T
sub-batteries are stubs returning zero-task results.
- Makefile targets: bench-5s, bench-5t, bench-5s5t.
- runtime_digest field captures the active π* registry fingerprint
so a registry change surfaces in bench results.
Tests: tests/test_bench_batteries.py (17 cases). Full suite:
1076 passed, 36 skipped. `make bench-5s5t` runs end-to-end and
emits JSON results.
Ticket #000021 status: in progress · Phase 1a landed; Phase 1b/2/3
remain open.
Makefile cuts (64 → 58 documented targets):
- ingest-cur-parallel, ingest-old-parallel: parallel-shared mode
superseded by attached (no WAL contention)
- distill-shards: sequential never preferred over parallel variant
- bench-qa-quick: bench-qa-smoke covers same use case (~30s vs ~10s)
- ingest-grok, ingest-grok-media: single-DB grok rare; -attached is
canonical path
All cuts land in code that the underlying CLI still exposes — operators
who need the dropped variant call '.venv/bin/aborist ingest --shard ...'
directly. No behavior loss, just shortcut removal.
Docs improvements:
- New Concepts page (docs/_source/concepts.rst): orientation on what
aborist is, three layers (surface/core/providence), Merkle commitment,
8-dim cache key, audit chain, trichotomy + four-rung ladder, layered
verifier, falsification state, sidecars. Embeds module-graph and
verifier-ladder SVG diagrams.
- New Cookbook page (docs/_source/cookbook.rst): 8 recipes — recrawl,
falsify, ingest-self-providence, mixed-corpus query, LLM endpoint
override, integrity after bulk ops, bench, retrieval tuning.
- Quickstart embeds query-pipeline SVG diagram.
- docs/_source/diagrams symlinks to docs/diagrams so Sphinx can include
the SVGs (was orphaned, only README referenced them).
Better Makefile RTD page (docs/_source/_ext/makefile_targets.py):
- Group by workflow phase (Setup → Fetch → Ingest → Distill → Query →
Verify → Operations → Tests → Docs → Clean) instead of alphabetical
prefix. Tells a new operator the order they'd actually run things.
- Phase descriptions added; targets prefixed with 'make ' for copy-paste.
- Uncategorized leftover surfaces missing entries in PHASES list.
Implements Read the Docs infrastructure to generate API documentation
directly from code docstrings. Replaces static modules.md (1200+ lines).
New structure:
- docs/_source/conf.py — Sphinx configuration (furo theme)
- docs/_source/index.rst — Main TOC
- docs/_source/api/*.rst — Module groups (substrate, storage, retrieval,
qa, distill, mesh, cli)
- docs/_source/Makefile — Local build targets
- docs/_source/README.md — Documentation on building and extending
Makefile integration:
- make docs-api — generate HTML (output: docs/_source/_build/html/)
- make docs-api-clean — remove build artifacts
Build output (40 HTML files):
- API module reference with docstrings
- Source code links (:viewcode: extension)
- Full-text search index
- Module index (genindex, py-modindex)
Sphinx installed in venv as dev dependency. HTML is browseable at
docs/_source/_build/html/index.html (open in browser after build).
This justifies the deletion of modules.md: code docstrings + Sphinx
autodoc = automatically-generated, always-current API reference.
Closes#000008 with status `closed · landed in 4f2b5a6` per the
docs/TICKETS.md convention. The preflight guard mechanism + bench
cycle + default flip all shipped 2026-05-03; the design log stays
in place.
Opens #000009 — Quantifier preflight run-DAG node binding. Splits
the Phase 5 follow-up out of #000008 §11.11 into its own ticket.
Scope: bind the classifier output + policy decision into
`run_dag_root` so audit replay can distinguish guard-on vs guard-off,
cap-applied vs not, reminder-injected vs skipped. Currently those
appear on the result dict but are NOT in the run-DAG hash. Required
to close the audit-replay gap that blocks the §9.5 Merkle-AGI-DAG
framing from fully holding. Estimated 3-4h. Awaiting go/no-go.
Makefile shortcuts for the #000008 CLI flags (operator ergonomics):
BROAD=1 → --apply-quantifier-caps (flip cap apply-gate)
REJECT_BROAD=1 → --reject-broad (preflight rejection)
ALLOW_BROAD=1 → --allow-broad (emergent search)
Available on both `make query` and `make query-dry`. Default
behavior unchanged: ANSWER_MODE=claim_lattice (JSON), reminder ON
for lattice modes, cap operator-opt-in.
Smoke-tested:
$ make query-dry Q="winners of all major sports?" BROAD=1
→ cap applies on JSON; classifier reports ALL/unbounded
$ make query-dry Q="winners of all major sports?" REJECT_BROAD=1
→ preflight rejection, exit-1 (consistent with UNGROUNDED)
TICKETS.md index:
#000008 closed · landed in `4f2b5a6`
#000009 open · awaiting go/no-go (D3, D4)
Next ID bumped 000009 → 000010
scripts/bench_emergent.py + make bench-emergent + design doc.
Random word triangulation surfaces failure modes the curated
bench/qa_questions.txt doesn't reach.
Loop:
/usr/share/dict/words → random.sample(3) →
Hermes @ temp=0.8 weaves a creative question →
aborist student answers via query() →
append journey to bench/emergent_log.jsonl
(teacher review = separate manual step, fox brings entries to
Opus & gets judgment to append)
Word filter: ^[a-z]{5,12}$ after lowercasing. Skips short words
(too vague) + very long words (Hermes can't weave them).
Cadence: NOT every commit. ~20s per cycle (Hermes generator +
aborist student); N=10 ≈ 4 min, N=50 ≈ 17 min. Most cycles land
UNGROUNDED-by-corpus-design (random triplets rarely overlap with
2010-11 Wikipedia coverage); the interesting cases are STRICT/
HYBRID surprises and the verifier-disagreement cases the teacher
catches.
Teacher review is intentionally out of the bench script:
- separation of concerns: generation is automated, judgment is
contextual & needs the corpus-knowledge frame ("is this a
2010 Wikipedia gap or a substrate failure?")
- future flexibility: today the teacher is Claude Opus 4.7
in this conversation; tomorrow GPT-5 or a review committee.
Swapping teachers is a workflow change, not a code change.
Teacher output schema (appended to the same JSONL line):
teacher.match bool
teacher.audit_agreement agree|disagree|unsure
teacher.novelty_class known_truth_grounding | emergent_synthesis
| novel_claim | no_signal
teacher.score_0_5 0..5
teacher.bench_max_signal retrieval | warrant | prompt | nil
teacher.reasoning one sentence
teacher.reviewed_by model id
teacher.reviewed_ts unix ts
Smoke verified (N=2, seed=42): 41s wall-clock, both UNGROUNDED
(expected — random triplets rarely overlap 2010 Wikipedia).
Append-only log seeded with the smoke entries.
Future flag (not yet wired): --generator-endpoint &
--student-endpoint to swap LLM upstreams per role.
Full design + teacher protocol: docs/bench-emergent-design.md.
Three dev-loop speedups:
(1) `make test` already on -n auto via pytest-xdist (was implicit
serial); 38s → 11s wall-clock = 3.4× faster on the 641-test
suite. Big inner-loop win.
(2) `make test-live` now also uses -n auto (live tests are
independent against the Hermes endpoint; concurrency=4 doesn't
overload it on the 17-test fixture set).
(3) `make backfill-concepts` (new) replaces the ad-hoc
`python -c "from aborist.concepts.extract import …"` invocations
fox was running by hand for the post-2026-05-02 concept-layer
backfills. Parallelizes per-shard work via multiprocessing.Pool
with CONCEPTS_WORKERS=4 (env-tunable).
Driven by scripts/backfill_concepts.py — runs every registered
extractor in EXTRACTORS (link_reciprocity, token_idf,
documents_fts) across every numeric-stem shard. Skips qa.db /
snapshots.db / crawl_*.db by default; --include-non-numeric
opts in. Wall-clock 189s for 4 wiki shards × 3 extractors vs.
~260s serial estimate; modest 1.4× speedup because SQLite WAL
+ FTS5 vocab queries are I/O-bound on a single SSD (4 workers
contend), but the unified UX & structured progress output are
the real wins.
(4) `make bench-qa-quick` (new) — 5-question smoke fixture × all
3 modes × 1 sample × concurrency 4. ~10s wall-clock. Sits
between bench-qa-smoke (n=1, ~30s) and full bench-qa
(~70min). Use as the inner-loop pre-commit signal.
Also: docs/concept-relations-design.md updated to point at the
new make target instead of the inline `python -c` block.
No behavior change in the test suite or LLM pipeline; pure tooling.
Bench-max sprint 1a + sprint 3 + speed audit. Five wins, none of
them traded calibration.
UTF-16 surrogate fix (sprint 1a)
================================
Hermes occasionally emits text with lone UTF-16 surrogates. Bare
.encode('utf-8') raises UnicodeEncodeError on those, which aborted
the run with no Merkle root. Two errors per lattice mode in the
2026-05-02 bench were this exact path on the 'tell me about the
roman empire' question.
Fix: errors='surrogatepass' on the four sha256 helpers that hash
model-derived text, plus the two audit-chain encode sites in
store.py for defense-in-depth (audit body could carry user text in
some flows). The hash stays deterministic because WTF-8 bytes are
reversible & unique per input.
Touched:
aborist/qa/dag.py:_sha256_hex (the loud one)
aborist/qa/keys.py:_sha256
aborist/qa/evidence.py:_sha256_hex
aborist/store.py: chain_audit_events + append_audit
Predicted Δ on next bench: +1pp on lattice modes (the 2 errors
become valid runs).
Smoke fixture (sprint 3)
========================
bench/qa_questions_smoke.txt — 5 questions, all anchor classes,
each currently failing pointer mode 100% while JSON aces 100% per
the 2026-05-02 bench. Wired as 'make bench-qa-smoke', --n 1
--concurrency 4, ~30-90s wall-clock depending on vLLM warmth. The
inner loop for prompt iteration; the full 71-question sweep stays
the scoreboard.
Smoke verified: pointer=0/5 STRICT, JSON=5/5, quote=2/5. Confirms
the gap pattern from the journal.
Concurrency default
===================
Makefile bench-qa now defaults to BENCH_QA_CONCURRENCY=4 (was
sequential). Override via BENCH_QA_CONCURRENCY=N. Combined with
the --concurrency landing in 0870af6, full sweep drops from ~107
min projected to ~51 min actual.
pytest-xdist (test-speed)
=========================
Added pytest-xdist>=3.5 to dev extras. 'make test' now uses
-n auto (= one worker per logical CPU). Measured: 36s → 10s on
the 641-test suite. 3.6× speedup, no test changes required.
Bench-max scoreboard (predicted lift from this commit alone):
+1pp lattice modes (UTF fix)
+cycle-time enabler (smoke fixture, xdist)
no calibration cost — none of the verifier checks moved.
Empirically observed 2026-05-01: long discursive questions like
'what technology is currently or soon available which may enable
one person to reconstruct another person's thoughts...' under-
retrieve because their content tokens get diluted by template
phrasing. AND-mode FTS5 returns zero hits; OR-fallback ranks
unrelated articles by raw BM25.
Fox's discovery: appending domain keywords ('transcranial knowledge
acquisition') to the question lifts the verdict from HYBRID 6/10
to STRICT 1/1 by narrowing OR-mode retrieval to the topical
article (Neurotechnology). The keywords act as an operator hint
about what the question is really asking.
This commit exposes that pattern as a first-class flag without
polluting the question text:
--retrieval-keywords TEXT on `aborist query`
K="..." on `make query`
Plumbing: the keywords are concatenated with the question for
FTS5 search and title-filter token construction only. The LLM
still sees the original question; the verifier still checks against
the original question; cache_key still computes from the original
question. Keywords are session-only — successive calls with
different keywords on the same question can cache-hit each other.
Pair with BURN=1 for fresh inference per call.
Live verification:
make query Q="...thoughts...without speaking or sign language."
K="transcranial knowledge acquisition" BURN=1
-> STRICT 2/2 via claim_lattice, 37.5s, Neurotechnology article
cited for fMRI and DBS claims.
Without K: HYBRID 6/10 with Videoconferencing/Telepathy as top
sources — model hallucinated structure across irrelevant chunks.
Closes the recursive-grounding gap fox surfaced today: aborist
tends Wikipedia trees but never grafts its own past Q&A records
into the forest. Each query starts from cold corpus retrieval;
prior providence_cache records sit unread until the same question
is re-asked (cache_key match). The "kindergarten thought chains"
framing names the gap — the system has a kindergarten of thoughts
(early STRICT records) that should mature into citable substrate
as they cool.
This commit lands the MVP — STRICT live providence records past
the kindergarten window become substrate via a new Source
subclass. Trust model per fox: "we trust strict statements as
fact unless a verifier falsifies it."
NEW
---
- docs/self-reference-thought-chains-design.md — full architecture
doc covering the four iteration-time gates, the recursive Merkle
proof story (Q2 → Q1 → Wikipedia chunk per v9.8's recursive-
cores insight), the falsification trust model (state=live as the
fail-closed gate), risks (lazy-anchor compounding, echo chambers,
storage bloat), and bench-impact estimate.
- aborist/sources/providence.py — ProvidenceSource(Source) with
four hard gates:
1. audit_mode == 'STRICT' (HYBRID/UNGROUNDED stay opaque)
2. falsification_state == 'live' (failed/stale/quarantined
excluded — verifier-falsification mechanism per fox)
3. now - created_at >= kindergarten_seconds (default 1h —
fresh thoughts cool first; kills tight echo loops)
4. anti-recursion: records whose answer text contains a
self-reference URI are skipped — first-generation only
- tests/test_providence_source.py — 10 unit tests covering each
gate plus the URI-scheme source-role classifier
- Makefile target `ingest-self-providence` (KG_SECONDS=3600
default; iterates each shard and self-promotes its STRICT live
records — cross-shard sharing happens via the existing
shards-dir UNION at retrieval time)
WIRE-UP
-------
- aborist/qa/query.py
- SOURCE_ROLE_BUDGET_WEIGHTS: self_reference_source = 1.0
(same as background — Wikipedia stays canonical primary;
self-reference is supplementary anchoring)
- SOURCE_ROLE_RANK_WEIGHTS: self_reference_source = 0.9
- _classify_source_role: short-circuits on aborist://providence/
URI prefix → self_reference_source regardless of title shape
- DEFAULT_QUERY_POLICY['claim_lattice_allowed_source_roles']
+= 'self_reference_source'
- aborist/qa/runner.py — same allowlist update for the
per-document `ask` path
- aborist/cli.py — `aborist ingest --source providence` reads the
providence_cache from the same shard it writes into;
--kindergarten-seconds flag plumbed through
NOT IN THIS COMMIT
------------------
- Aggregation of multiple Q&A records into synthesized summary
records (follow-on)
- Self-reference for HYBRID records (only STRICT is substrate
today; HYBRID could land later as a soft-anchor role with
lower trust)
- Live virtual sourcing (the design discusses it; MVP uses
snapshot ingestion so existing FTS / chunker / Merkle apply
with zero schema change)
- A live bench validating actual lift on self-reference questions
(requires running ingest-self-providence then bench; deferred
to follow-on commit on real data)
10 new unit tests pass; full suite at 482 passed / 21 skipped
(live fixtures gated).
Post-retry / post-trim-and-verify bench showed the picture flipped:
JSON mode now leads on strict-rate (50%) and ties grounded count
(54) with zero errors, all at parity-or-better latency. Switching
the `make query` ANSWER_MODE default from `claim_lattice_pointer`
to `claim_lattice` so the human-facing CLI uses the strongest mode.
Library-level DEFAULT_ANSWER_MODE stays "quote" so unit tests using
StubClient aren't disrupted.
Doc updated with post-retry bench table + revised conclusion.
Live test harness:
- new test_homer_simpson_boss_is_mr_burns fixture (pinned to
pointer mode — JSON mode hallucinates evidence_ids on
cross-document relationships, pointer mode's short numeric
tags can't be fabricated). Documents the mode trade-off.
- test_laura_croft marker net broadened to absorb Hermes
single-sample variance (added "magazine", "video game",
"character", "fictional", "british" markers) plus a hard
"croft" anchor check. Both real entities still gate cleanly.
11/11 live fixtures pass at the new JSON default + pointer pin
on Homer. 460 unit tests + 11 live = 471 green; 10 skipped is
just the live tests in the default-skip path.
New tests/test_qa_quality_live.py with 9 fixtures gated by
ABORIST_LIVE_TESTS=1 (the `make test-live` target sets it). Each test
runs ONE live query against Hermes + the configured shard set and
asserts:
* audit_mode is at least HYBRID (or UNGROUNDED for the honest-
refusal case)
* key entity tokens appear in the rendered answer
(e.g. "Torvalds" for the linux-kernel question, all four turtle
names + "Splinter" for the TMNT multi-part)
* for the Mars PM no-such-thing case, either UNGROUNDED OR a
refutation phrase ("no prime minister", "does not have", etc.)
— bare affirmative claims of a Mars PM fail the test as
hallucination
Why functional tests matter alongside unit + bench:
- unit tests (test_qa, test_query, test_claim_lattice) validate
plumbing with StubClient — can't tell you "did the model
actually answer correctly?"
- bench (`make bench-qa`) measures aggregate strict-rate /
grounded counts across many questions but doesn't assert
specific content
- these live fixtures sit between: each test is a named gate
around one known-good answer. When a future change improves
things, the bench number climbs AND every fixture passes (or
gets stricter assertions). When something regresses, the bench
number falls AND specific fixtures fail by name, telling you
where the regression landed. "Benchmax" rationale: the bench
is the scoreboard, the fixtures are the gates that translate
quality drift into named test failures.
Default `make test` is unaffected — 460 passed + 10 skipped (was 1).
Three-way QA-quality bench (quote / pointer / JSON) over an expanded
22-question set × 3 samples = 198 LLM calls. Findings landed in
docs/qa-modes-bench-2026-04-30.md with per-question breakdown and
roadmap. Aggregate at bench time:
quote 31S 20H 15U 0e strict-rate 47% 7.7s
claim_lattice_pointer 14S 34H 18U 0e strict-rate 21% 4.4s
claim_lattice (JSON) 26S 12H 9U 19e strict-rate 39%* 4.4s
The 19 JSON-mode "errors" turned out to be HTTP 502 from vLLM upstream,
not parse failures — clustered, all on the JSON-mode pass, plausibly
correlated with `guided_json` stressing the grammar engine.
Two improvements based on findings:
(1) OpenAICompatibleClient grew retry on transient 502/503/504 with
exponential backoff (0.5/1/2s, 3 attempts default). Network-layer
errors (ConnectError, ReadTimeout, RemoteProtocolError) get the
same retry. Smooths over the cluster without changing semantics:
persistent failures still raise, transient bursts no longer
dominate the error column. Helps all modes; JSON benefits most.
(2) Pointer-cap behavior changed from hard SCHEMA_INVALID to
trim-and-verify. When `[E2,...,E14]` over-cites a single claim,
keep first N pointers, verify normally, record
POINTER_OVERFLOW_TRIMMED in violations. STRICT becomes unreachable
(audit_mode caps at HYBRID) so the over-cite pattern stays
surfaced — but a correct claim like "Leonardo da Vinci painted
the Mona Lisa." no longer gets nuked for cosmetic over-citation.
Pre-fix: pointer mode hit 0/3 STRICT on Mona Lisa (mega-bracket
triggered SCHEMA_INVALID). Post-fix: HYBRID 2/14 with the right
answer rendered alongside both kept source spans. The dropped
pointers count toward n_quotes so the denominator surfaces the
over-cite to the auditor.
Bench scaffolding: ANSWER_MODES tuple now includes "claim_lattice"
(JSON), Makefile default sweeps all three. Question set expanded
from 8 to 22 covering narrow factoid, broad descriptive, entity
list, relationship, comparison, niche, adversarial, out-of-corpus.
460 tests pass. Connecticut output stays clean (HYBRID 4/7); JP-
dinosaurs pointer mode still UNGROUNDED via the bare-name guard
(model emits one-token entity names, the right floor catches them).
Adds bench/qa_sweep.py + bench/qa_questions.txt + `make bench-qa` target.
Sweeps a fixed question set through quote and claim_lattice_pointer modes,
N samples per cell (default 3), each sample burns the cached record so
Hermes nondeterminism becomes the variance source.
Outputs:
bench/qa_results/<utc>.jsonl one row per (mode, question, sample)
bench/qa_results/<utc>.md summary + per-question vote counts
Question set spans the failure-mode shapes the verifier needs to handle:
narrow factoid, broad descriptive ("tell me about X"), entity list,
relationship, and out-of-corpus (should land UNGROUNDED honestly).
Methodology gap this closes: at n=1 the strict-rate swings ±20pp on a
fixed prompt purely from sampling noise. n=3 makes 5pp deltas legible.
New policy["answer_mode"] = "claim_lattice_pointer" (default "quote").
The runtime builds an evidence map with two-layer ids — pointer-id
(E1, E2, ...) shown to the model, sha256-derived evidence-id used by
the cache & run-DAG — and the model emits pointer-line prose
("Claim. [E12]") referencing them. Renderer interpolates literal
source spans at display time. Synthetic-elision-by-construction-
impossible: the model never types the quote string.
Pieces:
- aborist/qa/evidence.py (new): EvidenceObject + spotlight excerpt
(claim-token-centered window into the cited span, falls back to
leading window when no token matches).
- aborist/qa/parse_claims.py (new): pointer-line parser walks
lines, pulls [E\d+] / [E\d+,E\d+,...] tags, returns ParsedClaim
with PARSED / NO_EVIDENCE_POINTER status. Strict regex refuses
fuzzy alternatives so honest UNGROUNDED beats lax acceptance.
- aborist/qa/verify.py: verify_claim_lattice. Hard checks only —
parser succeeded, evidence_id resolves, source_role allowed, no
manual quotes (any " char violates), claim text non-empty.
Returns evidence_id_pairs (content-addressed, run-stable) for the
run-DAG. Soft signals (entailment, completeness, predicate
compatibility) stay sidecar.
- aborist/qa/dag.py: 9-stage CTI shape when evidence_map_root is
supplied — question / retrieval / evidence_map / prompt /
raw_answer / parsed_claim_lattice / verify / render /
final_label. Quote mode keeps the original 7-stage shape so
pre-G0 run_dag_root values stay valid.
- aborist/qa/query.py: per-chunk evidence-map build with role-
weighted budget AND query-relevance ordering. Within each source,
chunks are ranked by (distinct_query_tokens_present,
total_mentions, doc_order_asc) so the chunk most likely to
textually support the question gets the lowest pointer id. Without
this re-rank Hermes-3-8B lazy-anchored on doc-order-first chunks
regardless of relevance.
- aborist/qa/runner.py: same answer_mode branch for the single-doc
ask() path. No iterative repair in pointer mode (one-shot
benchmark discipline).
- aborist/store.py: verifier_method CHECK extended with
'claim_lattice'. New _rebuild_providence_cache_claim_lattice
migration preserves run_dag_root / run_dag_blob across the
rebuild — older rebuilds dropped them.
- aborist/cli.py: --answer-mode {quote,claim_lattice_pointer} on
query and ask.
- Makefile: ANSWER_MODE knob; defaults to claim_lattice_pointer for
`make query` so the testing harness exercises the new path.
Library DEFAULT_POLICY / DEFAULT_QUERY_POLICY stay "quote" so
Python callers and unit tests aren't surprised.
Prompt: one-shot worked example (Apple founders) plus strict
no-quotes / no-JSON / no-markdown / plain-prose-only rules. Without
the worked example Hermes-3-8B drops the bracket protocol on roughly
half of runs; with it, JP dinosaurs benchmark went 0/13 -> 10/11 ->
16/17 verified across the iterations that hardened the pipeline.
Tests: +48 covering parser, two-layer ids, verifier failure modes
(UNKNOWN_EVIDENCE_ID / SOURCE_ROLE_BLOCKED / MANUAL_QUOTE_VIOLATION
/ NO_EVIDENCE_POINTER / SCHEMA_INVALID), spotlight rendering with
buried-term fixture, per-chunk evidence map, query-relevance
ordering, schema migration round-trip, 9-stage DAG shape divergence.
Total 438 passed, all 7 production shards report 0 chain breaks.
Known limitation: chunk boundaries can cut wikitext mid-template, so
mwparserfromhell-backed to_base() leaves orphan </ref> tags and
leading list markers in the visible spotlight excerpts. Verifier and
CTI architecture are unaffected; the leak is cosmetic. Proper fixes
are template-aware chunking (chunker bump invalidates prior records)
or an orphan-marker post-strip in aborist/wikitext.py — both out of
G0 scope.
burn-kindergarten flake fix:
test_burn_kindergarten_zero_seconds_burns_everything failed under
full-suite ordering when the wall-clock second rolled over between
seeding the row & running burn-kindergarten. Old impl computed
`cutoff = now - 0 = now` and required `created_at >= now`, so a row
seeded at second T became invisible to a burn check that ran at T+1.
The verb's docstring is "0-second window = burn every live row".
Fix the impl to match: when kindergarten_seconds <= 0, drop the time
gate entirely (burn all live records). Non-zero windows keep the
existing cutoff-based filter unchanged.
Repair CLI:
`aborist query --repair` enables the mechanical repair pass after
first verify. `--repair-reprompts N` adds the optional re-prompt
tier (default 0 = off; mechanical-only when --repair is set without
--repair-reprompts). Both flags are off by default so `make query`
stays single-shot unless the operator opts in.
Makefile: `make query REPAIR=1` flips on mechanical repair;
`make query REPAIR=1 REPROMPTS=1` adds one re-prompt iteration.
Both knobs off by default to preserve existing behavior.
485 tests pass under full-suite ordering (was 484 + 1 flake).
Per fox: while iterating on retrieval/verifier knobs, a flag on the
query itself is more useful than a separate verb. One step reset:
change a knob, re-query, see fresh result.
Surface:
aborist query --burn ...
make query Q="..." BURN=1
make query-dry Q="..." BURN=1 (works on dry-run too)
Behavior in `aborist.qa.query.query()`:
- New `burn_existing: bool = False` parameter on query().
- After computing primary cache_key (per the active dedup mode) but
BEFORE the cache lookup, if burn_existing: DELETE the live row that
matches the primary cache_key & write one providence_burn audit
event with reason "query --burn (test-ergonomic mid-query bust)".
- The lookup then misses → fresh inference runs. Result reports
`burned_existing: 0|1` so the caller sees whether anything got
busted.
- The equivalence-class fallback cache_key is deliberately NOT
touched: prior alt-mode records stay as historic witnesses.
Tests (3): cache populates → cache hits → BURN=1 forces fresh
inference; --burn writes a providence_burn audit event; first-time
query with --burn is a clean no-op (burned_existing=0). 356 passed,
1 skipped.
Complementary to `aborist burn-kindergarten` (mass reset) — this is
the surgical version. Both are local-only by design; for cross-peer
invalidation use `make falsify` and let mesh sync broadcast.
Per fox: useful while iterating on retrieval/verifier knobs to wipe
recent test runs without finding each cache_key. Mirrors the mesh-sync
kindergarten window so what's still un-broadcast (private to this peer)
is exactly what's safe to bust without confusing peers.
Surface:
aborist burn-kindergarten [--kindergarten-seconds N]
[--reason '...'] [--by-actor X]
[--force] [--dry-run] [--verbose N]
make burn-kindergarten [SECONDS=3600] [FORCE=1] [DRY_RUN=1] [REASON='why']
Behavior:
- Walks every shard, finds providence_cache rows with
created_at >= now - SECONDS and falsification_state='live'.
- Each row goes through the existing _burn_cache_key — children gate
honored unless --force, audit event written per successful burn,
chain integrity preserved.
- Result JSON reports examined / burned / refused_has_children /
not_found counts plus a verbose tail of per-row results.
- --dry-run reports without writing or auditing.
- --kindergarten-seconds 0 = burn every live row (test reset).
Tests (4): selective burn (old kept, fresh burned), dry-run writes
nothing, 0-second window burns everything, children-gate refusal
without --force. 353 passed, 1 skipped.
Operational note: this command does NOT propagate to peers (burn is
local kindergarten cleanup by design, see docs/mesh.md). If you want
the remote effect, falsify each record individually & let mesh sync
broadcast the falsifications instead.
Per fox: the verifier reports n/N verified but doesn't tell you WHY a
span didn't ground. Some are paraphrase, some are model-added
trailing artifacts (e.g. `(Source: https://...)` citations the model
appended to verbatim prose), some are full inventions. Operator
needs to see which is which to triage.
Read-only sidecar — no audit events, no providence_cache mutations,
no v9.8 field changes. Per the verifier-no-diagnostics memory rule:
hard chain stays binary; soft signals live in sidecar verbs that
never feed back.
Five diagnosis labels:
verbatim_in_base span IS in base context — verifier or
canonicalization bug (worth flagging loudly).
verbatim_in_raw_only raw wikitext match but base form differs —
wikitext-strip edge case.
trailing_artifact a >=60-char prefix matches; the tail (often a
model-added citation) doesn't. Surfaces the
tail explicitly.
paraphrase >=85% query tokens (>4 chars) present in
base context but not in this sequence.
Model rewrote source content.
partial_paraphrase 40-85% coverage — mixed sourced/emergent.
no_overlap <40% coverage — likely full invention.
Verified live against the real Pikachu cache record (5 sources,
71KB raw -> 34KB base): two spans flagged paraphrase (token_coverage
1.0 for both, just rewritten sequence), one flagged trailing_artifact
(100-char prefix matches; tail is `(Source: https://...)` citation).
That's the actual reason HYBRID — model paraphrased + appended
citations that aren't in the corpus.
Surface:
aborist inspect --cache-key <hex> [--qa-db ...] [--json]
make inspect KEY=<hex> [JSON=1]
Tests: 9 (6 classifier on synthetic contexts pinning each label,
3 end-to-end with seeded fixture record + verification that inspect
writes nothing). 318 passed, 1 skipped overall.
Default `make query Q="..."` now uses the human render. JSON=1 flips
to the raw record (same as `aborist query --json` directly):
make query Q="who is pikachu?" # human render
make query Q="who is pikachu?" JSON=1 # raw JSON record
Same flag works on query-dry. Help line updated to advertise it.
Defect found by fox: `make crawl-ingest URL=...` wrote to
~/.aborist/aborist.db (the legacy single-DB path) but `make query Q=...`
searches ~/.aborist/shards/ — two different corpora, query never saw
the crawled content.
Fix mirrors the existing shard-aware ingest pattern (ingest-self,
ingest-git, ingest-grok-attached): derive a shard filename from the
seed URL's hostname and write into $(SHARDS_DIR). Override with
CRAWL_SHARD=path for custom layout.
https://russell.ballestrini.net
-> $(SHARDS_DIR)/crawl_russell_ballestrini_net.db
After this, `make query Q=...` (cross-shard search) automatically
picks up newly-crawled domains. Each domain stays in its own shard
file so falsifications, evictions, and deletions are scoped naturally.
recrawl-check gets the same treatment: defaults to walking every shard
under SHARDS_DIR (skipping qa.db and snapshots.db); CRAWL_SHARD=path
scopes to one when you want.
Note for fox: existing pages already at ~/.aborist/aborist.db from
the prior crawl are still there (harmless). To get them under query,
either re-crawl into the new path or migrate via sqlite ATTACH +
INSERT — a follow-up if you want it.
The lifted AsyncWebFetcher already had a fast_mode constructor knob;
just unbroken plumbing was missing. fast_mode trades politeness for
throughput:
- timeouts: 5s vs 15-60s
- parallel page workers: cpu_count() * 3 vs 1
- ignores robots.txt 'crawl-delay' directive
- robots.txt 'Disallow' is STILL honored (only the delay drops)
Surface:
aborist crawl --seed-url ... --fast
make crawl-ingest URL=... FAST=1
Tests: two regressions pin that fast=True / fast=False each translate
to the right AsyncWebFetcher(fast_mode=...) construction. Capturing
fetcher fixture to avoid network. 21 bridge tests + 273 default
suite, all passing.
Per fox: 20 was a wrong default — typical sites have many more pages
within depth 2 than that, and the cap was silently truncating real
crawls. New behavior:
aborist crawl --depth 2 # no page cap
aborist crawl --depth 2 --max-pages 50 # opt-in cap when needed
make crawl-ingest URL=... DEPTH=2 # no page cap
make crawl-ingest URL=... DEPTH=2 MAX=50 # explicit cap
Implementation: bridge's BFS loop treats max_pages=0 as "unbounded"
(while-condition becomes `queue and (max_pages == 0 or len(...) < max_pages)`).
Default flows from CLI argparse default=0 down to the bridge.
Tests: pinned max_pages=0 → no cap with a 30-page chain regression
test (19 passed in tests/crawler/test_bridge.py). Default suite still
273 passed, 1 skipped.
Glues the verbatim crawler lift to aborist's existing ingest pipeline,
plus adds a freshness probe that asks "does this need recrawling?"
without paying for a body fetch when the answer is no.
Bridge (aborist/sources/crawler/bridge.py):
- crawl_seed(seed, depth, max_pages): BFS-discovers same-domain URLs
via AsyncWebFetcher. Exact-hostname filter so subdomain crossover is
off by default. Caps at max_pages. URL dedup, fragment-strip
canonicalization. Sync wrapper around an async core so the CLI
remains a one-liner.
- ingest_crawled(conn, urls): runs the URL list through the standard
ingest_source() path so chunking_version / canonicalization_version
/ Merkle commitment all stay consistent. Captures ETag +
Last-Modified per response and writes one row per document_root
into the new document_http_meta table.
Schema (idempotent migration, mirrors _migrate_mesh_peer_chains):
document_http_meta (
document_root TEXT PRIMARY KEY,
etag TEXT,
last_modified TEXT,
last_fetched_at INTEGER NOT NULL,
last_status INTEGER,
last_checked_at INTEGER,
FK(document_root) -> documents(document_root) ON DELETE CASCADE
)
Recrawl-check (recrawl_check):
- Sends conditional HEAD with If-None-Match + If-Modified-Since.
304 -> fresh (no body transfer needed). 200 -> stale (server says
the body changed). 404/410 -> gone. Other -> unreachable.
- Updates last_status + last_checked_at so consecutive runs visit
oldest-first (NULLS FIRST in ORDER BY).
- Optional --domain filter so operators can probe one site at a time.
CLI:
aborist crawl --seed-url URL [--depth 2] [--max-pages 20] [--ingest]
aborist crawler recrawl-check [--domain DOMAIN] [--limit 100]
Makefile:
crawl-ingest URL=https://example.com [DEPTH=2 MAX=20]
recrawl-check [DOMAIN=example.com LIMIT=100]
Both targets gate on bootstrap-crawler so [crawler] extras land first.
Tests: 17 new in tests/crawler/test_bridge.py — same-domain filter,
BFS bounds (depth + max_pages), dedup, fragment normalization, ETag
capture, content-type filter, HEAD classification (304/200/404),
domain filter, last_checked_at update. All stub the network via a
fake httpx.Client; default suite still 273 passed, 1 skipped.
End-to-end test path:
make bootstrap-crawler
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 MAX=20
make query Q="who is russell ballestrini?"
make recrawl-check DOMAIN=russell.ballestrini.net
Lifts the async web fetcher into aborist as an opt-in source. The
implementation comes directly from ~/git/agents.ai.unturf.com/core
(rev 2026-04-28); aborist's adaptations are minimal and documented in
aborist/sources/crawler/__init__.py:
core/async_web_fetcher.py -> aborist/sources/crawler/async_web_fetcher.py
core/web_fetch.py -> aborist/sources/crawler/web_fetch.py
Two source-side changes during the lift:
1. Heavy deps (aiohttp, bs4, miniuri) wrapped in try/except so a bare
`import aborist.sources.crawler` raises ImportError with the install
hint instead of leaking AttributeErrors deep in user code.
2. Chat-bot fetch triggers (`has_fresh_fetch_trigger`,
`has_web_fetch_trigger` from agents.ai.unturf.com/core/keywords)
replaced with NotImplementedError stubs. Aborist has no chat
surface — fetch intent is detected at the application layer. The
two test classes that exercised these triggers are
`@pytest.mark.skip`'d with the same rationale.
Not lifted: web_cache_manager.py — it backs page caching with
SQLAlchemy. Aborist has its own content-addressed cache via
providence_cache; no need to carry SQLAlchemy as a dep just for
crawled-page memoization.
Off by default:
- `[crawler]` extras section in pyproject.toml carries the heavy
deps. `[dev]` pulls them in so the crawler tests can run.
- `make test` ignores tests/crawler/ entirely.
- `make bootstrap-crawler` installs the extras into the venv.
- `make test-crawler` runs only the lifted tests after extras land.
Tests: 74 passed, 9 skipped (the chat-bot trigger tests deliberately
dropped). Default `make test` stays at 273 passed, 1 skipped.
burn --kind {providence,document,core}; default 'providence' preserves
the prior --cache-key surface (no breaking change).
Children gates for document/core leaves:
derivations.src_root downstream cores derived from this doc
edges.dst_root other docs link to this doc
providence_cache.source_root Q&A grounded in this doc
derivations has ON DELETE CASCADE on BOTH core_root and src_root, so a
naive DELETE FROM documents would silently cascade-prune downstream
derivations and orphan deeper cores. Gate before delete; --force to
override with the count recorded in the audit body.
Burn manually clears chunks_fts (FTS5 has no FK) and outbound edges
(no FK either) before deleting the document. chunks + merkle_nodes +
inbound derivations cascade via FK.
Audit event types are document_burn / core_burn — distinct from
providence_burn so chain consumers can tell leaf kinds apart.
CLAUDE.md "Cores never evict" remains intact: burn is operator-driven
removal, evict is the automated cold-tier compression that touches
only kind='surface'.
Makefile: KIND=providence|document|core selects the leaf type;
KEY for providence, ROOT for document/core. Single 'burn' target keeps
the surface DRY.
Tests (tests/test_burn_doc.py, 13 new):
- happy path: chunks + merkle_nodes vanish via cascade
- refused on derivation downstream / providence ref / incoming edge
- --force overrides and records counts
- core burn happy path (no children) preserves parent surface
- core burn refused with downstream derivation
- audit chain integrity preserved (no dangling prev_event_hash)
- CLI invocations for document, core, default-providence, refused
Full suite: 247 passed, 1 skipped.
Burn deletes a providence_cache row outright; falsify (the existing
verb) flips state and keeps the row. Burn is for early/scratch use
during a tree's genesis when downstream consumers haven't ingested
the answer yet.
Children gate: refuses if any falsifications reference the cache_key.
Caller can override with --force; the audit event records forced=True
plus the child count at burn time so the chain can reconstruct what
was orphaned. Refusal is non-destructive AND skips the audit event so
scripts can retry without polluting history.
Always writes a 'providence_burn' audit event regardless of force.
Body fields: cache_key, burned_audit_mode, burned_n_verified,
burned_state, question_text, reason, by_actor, child_falsifications_at_burn,
forced. The chain-break query (`make chain-check`) shows 0 after burn —
the new event extends the prior chain cleanly.
Surface:
aborist burn --cache-key <hex> [--reason '...'] [--force]
make burn KEY=<hex> REASON='...' [FORCE=1]
Tests (7): happy-path delete + audit event, refusal on children,
--force override, unknown-key not-found, chain-integrity preserved,
CLI happy path, CLI non-zero exit on refusal.
Adds aborist/wikitext.py with to_base() — deterministic wikitext →
prose conversion via mwparserfromhell. Drops <ref>...</ref>,
[[File:...]], [[Image:...]], [[Category:...]] entirely; resolves
piped wikilinks to display text; collapses templates, formatting,
and HTML markup. extract_wikilinks() preserves the link graph for
the definition-cloud artifact (recoverable from any page on demand
without re-parsing wikitext at query time).
Wires to_base() into verify_quotes(). Without the strip, the
verifier compared the model's clean prose against [[Cloud Strife]],
[[Shinra Electric Power Company|Shinra]], etc. and falsely flagged
real source quotes as VISUAL. Concrete case from a make-query run:
'Cloud Strife, an unsociable mercenary who claims to be a former
1st Class member of Shinra's SOLDIER unit;' is verbatim in the
Final_Fantasy_VII article wikitext (modulo markup). With the strip
that quote now verifies; the genuine model hallucinations in the
same answer still flag honestly. Side-effect: 43% smaller context
size on average so less LLM token waste.
BASE_VERSION = 'wikitext-base-v1' is the algorithm pin. Bump when
the strip rules change. Soft-imported in verify.py so environments
without mwparserfromhell installed degrade gracefully (no strip,
same behavior as before this commit).
Build:
- pyproject.toml: new [wikitext] extras (mwparserfromhell>=0.6),
pulled in by [dev]
- Makefile: chain-check / chain-check-shards targets — fast
audit-chain integrity probe, counts dangling prev_event_hash
references; 0 = chain intact
Tests:
- tests/test_wikitext.py: 31 tests (rules, idempotence, real-corpus
fixture)
- tests/test_verify.py: 2 regression tests pinning the FF7 flip
(Cloud Strife quote: VISUAL → STRICT after strip; genuine
hallucination: stays VISUAL)
- tests/fixtures/ff7_characters_chunk0.wikitext: real chunk from a
shard, used to validate the strip on actual Wikipedia content
Adds docs/mesh.md and five graphviz diagrams covering the federation
layer: identity stack, epoch state machine, per-member secret envelope,
gossip wire contract, and operator decision tree. Pins the protocol
contract for the upcoming HTTP wire (mesh sync, mesh serve).
Makefile gets a 'docs' target with pattern rule so PNG renders are
incremental from .dot sources.
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`,
yielding one Document per text file. URI shape `git://<repo>/file/<path>`
intentionally omits the commit hash — re-ingest after new commits produces
fresh document_roots that aborist's prior-doc detection chains via
`supersedes` edges, so the audit trail and Merkle tree grow as the repo
grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files
filtered out by default. Commit hash + timestamp + subject ride along in
`extra` (informational only; not part of the Merkle commitment).
MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes
semantics, same shape.
Makefile adds:
make ingest-self (this repo -> aborist-self.db)
make ingest-git GIT_REPO=/path/to/repo (arbitrary git clone)
make ingest-hg HG_REPO=/path/to/repo (mercurial)
Each lands in its own shard file alongside existing shards/grok.db,
keeping per-shard write paths independent of the wikipedia 4-way ingest's
WAL writer lock.
Drops in for any dated snapshot in dumps.wikimedia.org/archive — pages-articles.xml.bz2
and pages-meta-current.xml.bz2 (cur snapshot, single-revision-per-page) plus
pages-meta-history.xml.bz2 (multi_revision=True). abstract.xml feed yields
pre-distilled summaries at ~1/100th the chunk volume. Streams .bz2/.gz directly
via iterparse with bounded memory; same shard / resume contract as the SQL
source. Falls back to title-prefix namespace filtering when older export
schemas omit the per-page <ns> element (e.g. enwiki 20101011).
Makefile defaults target enwiki 20101011 (6.2 GB). Override with WP_XML_YEAR /
WP_XML_MONTH / WP_XML_DATE / WP_XML_LANG to fetch any other archived snapshot.
The "permacomputer" query fell into title-relevance's empty-fallback
because "permacomputer" appears nowhere in any title. Fallback was
top-1 by raw BM25, getting one tangential source. Same problem for
any user-coined term: title-LIKE alone can't find it.
_filter_by_title_relevance now grows three accept paths:
1. Title-token overlap (existing).
2. TF-IDF core keyword overlap. A doc's distinctive low-frequency
terms get distilled into its tfidf-keywords-v1 core. A query for a
rare term ("permacomputer", "unturf") hits docs whose core lists
it as a keyword even when no title does.
3. Body density. Docs that mention a query token >= 3 times pass
regardless of title or core. Cheap proxy for "actually about it."
Empty-all-three fallback widens from top-1 to top-N (default 5) — the
LLM gets enough context to honestly say "I don't know" rather than
fabricating from a single tangential source.
Implementation:
- _docs_with_core_keyword_match: SQL join chunks -> documents -> derivations
finds source docs whose TF-IDF core content matches a token. Folded
into the per-shard search loop alongside FTS5 + title-LIKE.
- _body_density_passes: counts case-insensitive occurrences in
concatenated chunk content. Threshold 3.
- _search_corpus stashes core_match_roots + root_to_shard on the
returned hits list as side-channel info for _rerank.
Makefile: distill-shards-tfidf-parallel for the per-shard TF-IDF run.
Demo on the live cluster (after TF-IDF distill of 95k cur surface docs):
Q "tell me about permacomputer ?"
sources: MOAD Cheat Sheet + Unturf AI Infrastructure Expansion
answer: synthesized from both, bounds coverage honestly
Q "what is unturf about ?"
sources: Unturf Trademark + Unturf AI Infrastructure Expansion
answer: "grassroots, community-driven initiative ... ethical AI
hardware, software, and networks"
Both queries now find user-coined terms via TF-IDF cores even though
the terms never appear in any title.
79 tests passing.
Two issues surfaced by "tell me about permacomputer ?":
1. Stopword leak. _FTS5_STOPWORDS got "tell" added earlier but the
parallel _TITLE_STOPWORDS in qa/query.py wasn't synced. Title-LIKE
search picked up "tell" and returned every Tell_* article in the
corpus (Tell_(poker), Tell-Tale_Heart, Tell_Township, Tell_City,
etc.) all uniformly scored at 70.0. Synced the lists — title
tokenizer now drops the same question-shaping verbs as FTS5.
2. Grok docs lived only in --db (default ~/.aborist/aborist.db),
not in --shards-dir. `make query` reads the shard cluster, so
Grok was invisible to it. Added two targets:
ingest-grok-attached conversations -> $(SHARDS_DIR)/grok.db
ingest-grok-media-attached media prompts -> $(SHARDS_DIR)/grok.db
Both use --resume so re-runs are cheap.
Demo: "tell me about permacomputer ?" now answers from the
"MOAD Cheat Sheet: Common Software Defects" Grok conversation —
fox's own prior chat surfaced as corpus.
Note: 000.db already had Grok docs from an earlier ingest variant.
The cross-shard UNION currently sees them twice (376 rows, 188
unique document_roots). Retrieval is unaffected because
_search_corpus dedupes by root, but stats counts are inflated.
Cleanup is destructive (drop one copy) — fox decides which.
79 tests passing.
