Two related changes that tighten the dedup + grounding signals
without lowering quality bars.
1. question_hash drops standalone English articles (the/a/an).
Fox 2026-04-29: `who is the batman?` and `who is batman?`
produced different cache records; same question, different hash.
Articles are filler at the question-equivalence layer. Add a 4th
canonicalization step to question_hash: after lowercase + trailing
punctuation strip, split on whitespace & drop tokens equal to
"the" / "a" / "an", then rejoin.
Equivalence class now includes:
"who is X" ┐
"who is the X" │
"who is a X" │ -> same question_hash
"who is an X" │
"Who Is X？" ┘ (CJK question mark)
"thesis" stays untouched — exact-match standalone tokens only,
not substring. Conservative on i18n: ASCII English articles only;
"el / la / le / der / die / das" etc. await demand.
2. _token_coverage filters stopwords and per-token punctuation.
Fox asked: should we lower the 0.85 paraphrase threshold? Honest
answer: no — that would promote the Q1 Batman fabrication
("wealthy/businessman/resides" missing from corpus) to STRICT.
Tighten the signal instead so 0.85 means more.
- Per-token punctuation strip (.,;:!?\"()[]{}) so `wayne,` lines
up with bare `wayne` in context. Apostrophes deliberately stay
so `batman's` is distinct from `batman`.
- English stopword filter on length-≥4 fillers (from / with /
have / been / would / which / where / their / etc., curated set
in _ENGLISH_STOPWORDS). These match almost any English text &
inflate coverage scores when topical content is missing.
Net effect on Q1 Batman case: missing tokens are
`wealthy/businessman/resides` — all topical. Coverage stays well
below 0.85, span stays UNGROUNDED. Net effect on a stylistic
paraphrase (model wrote `from` instead of `with`): coverage
computed only over topical tokens, both copies match → 1.0 →
correctly promoted.
Tests: 8 new in tests/test_keys.py (article-strip equivalence
classes, substring preservation, distinct-topic non-collapse) +
2 in tests/test_verify.py (stopword filter doesn't inflate; 0.85
threshold still rejects fabrication). 335 passed, 1 skipped.
Note for fox: the underlying retrieval issue surfaced in Q1/Q2 is
independent — Batman main article IS in shard 002.db but FTS5 +
title rerank ranked List_of_Batman_comics higher. Different fix,
separate commit.
Two changes that make the verifier accept more honest grounding without
relaxing what STRICT means.
1. Strip trailing citation parentheticals at extraction time.
`extract_quotes` and `extract_claim_spans` now drop a single trailing
`(Source: ...)` / `(citing X)` / `(https://...)` parenthetical
before returning. The Pikachu real-corpus case: model wrote the
verbatim source sentence and appended `(Source: https://...)`. The
prose IS in the corpus — only the inserted citation broke substring
match. Now substring fires; STRICT recovered.
Conservative regex: only strips when the parenthetical content
starts with a citation cue word OR contains a URL. A genuine prose
parenthetical like "Pikachu (a Pokémon species) lives in forests"
stays.
2. Paraphrase strategy as a 4th verifier method.
In the span path, items that fail substring match get a
token-coverage probe. Tokens >=4 chars from the span; if >= 85% of
them appear in the normalized base context, count the span as
paraphrase-verified. Paraphrase items contribute to n_verified;
verifier_method flips to 'paraphrase' when any soft-verified items
are present so an auditor can tell.
Gated by `_is_prose_span`: a list of proper nouns ("Keanu Reeves,
Laurence Fishburne") has no lowercase content tokens >=4 chars and
falls through to the entity strategy where proximity policy can
tight-cluster check. Real prose spans ("Pikachu is a species of
Pokémon creatures...") have multiple lowercase content tokens and
qualify for paraphrase.
Soft-signal note (from CLAUDE.md soft/hard rule): paraphrase is
heuristic, not byte-equivalence. The hard chain still records the
classification — verifier_method='paraphrase' is the explicit marker
that an audit reader can use to distinguish lexical-verbatim from
paraphrase-overlap. Quote strategy stays verbatim-only — quotes ARE
quotes; paraphrase-in-quotes is the model's mistake to flag, not
auto-promote.
Schema: verifier_method CHECK constraint expanded to include
'paraphrase'. New idempotent migration `_rebuild_providence_cache_paraphrase`
mirrors `_rebuild_providence_cache_ungrounded` (table rebuild via
BEGIN IMMEDIATE; no value translation needed). Probes existing CHECK
from sqlite_master and no-ops if already expanded.
Tests: 9 new in tests/test_verify.py — 4 trailing-citation strip
(quote+span variants, URL-only parenthetical, end-to-end STRICT
recovery), 4 paraphrase strategy (high-coverage promote, low-coverage
reject, mixed-method label, quote-strategy preservation). Default
suite still green: 326 passed, 1 skipped.
Burned the live Pikachu cache record (audit chain extended) so a
fresh `make query Q="who is pikachu?"` exercises the new verifier
end-to-end.
Per fox: the verifier reports n/N verified but doesn't tell you WHY a
span didn't ground. Some are paraphrase, some are model-added
trailing artifacts (e.g. `(Source: https://...)` citations the model
appended to verbatim prose), some are full inventions. Operator
needs to see which is which to triage.
Read-only sidecar — no audit events, no providence_cache mutations,
no v9.8 field changes. Per the verifier-no-diagnostics memory rule:
hard chain stays binary; soft signals live in sidecar verbs that
never feed back.
Five diagnosis labels:
verbatim_in_base span IS in base context — verifier or
canonicalization bug (worth flagging loudly).
verbatim_in_raw_only raw wikitext match but base form differs —
wikitext-strip edge case.
trailing_artifact a >=60-char prefix matches; the tail (often a
model-added citation) doesn't. Surfaces the
tail explicitly.
paraphrase >=85% query tokens (>4 chars) present in
base context but not in this sequence.
Model rewrote source content.
partial_paraphrase 40-85% coverage — mixed sourced/emergent.
no_overlap <40% coverage — likely full invention.
Verified live against the real Pikachu cache record (5 sources,
71KB raw -> 34KB base): two spans flagged paraphrase (token_coverage
1.0 for both, just rewritten sequence), one flagged trailing_artifact
(100-char prefix matches; tail is `(Source: https://...)` citation).
That's the actual reason HYBRID — model paraphrased + appended
citations that aren't in the corpus.
Surface:
aborist inspect --cache-key <hex> [--qa-db ...] [--json]
make inspect KEY=<hex> [JSON=1]
Tests: 9 (6 classifier on synthetic contexts pinning each label,
3 end-to-end with seeded fixture record + verification that inspect
writes nothing). 318 passed, 1 skipped overall.
Two things fox surfaced from a "who is pikachu?" run:
1. Output emitted "Pok\\u00e9mon" instead of "Pokémon" — json.dumps
defaulted to ensure_ascii=True. Switched to ensure_ascii=False on
every user-facing dump in cli.py (41 sites). The one canonical-JSON
call (`separators=(",", ":")` for storing in providence_cache as
a JSON column, not for hashing) was deliberately left alone.
2. The actual answer was buried under cache_key / context_root / per-
source metadata / timings JSON. Default now renders human-readable:
who is pikachu?
HYBRID 1/2 verified via quote 9.2s (fresh)
Pikachu is a species of Pokémon creatures...
sources (3):
[1] Pikachu — en.wikipedia.org/wiki/Pikachu (002.db)
[2] List_of_Pokémon — en.wikipedia.org/wiki/List_of_Pokémon
...
unverified (1):
- "spans the model couldn't ground..."
cache_key: 35ab7d33… <run with --json for full record>
Pass `--json` to get the prior raw record (still ensure_ascii=False
so unicode renders cleanly there too — scripts parsing the output
see real chars; the JSON spec accepts either form).
Implementation:
- `_render_query_human(result, question)` — pure function, easy to
unit-test, no I/O. Truncates unverified spans over 100 chars,
omits empty sections, distinguishes cached vs fresh in the summary.
- `_strip_scheme` / `_short_path` — small helpers for source display.
- `--json` flag on the `query` subparser; default is human render.
Tests: 12 new in tests/test_cli_render.py — question position, summary
fields (audit/n_verified/method/elapsed/cache-status), unicode
literals (Pokémon not \\u00e9), source line shape, long-quote
truncation, empty-section omission, error fallback, short cache_key
with --json hint. 309 passed, 1 skipped overall.
Per fox: ASCII-only stripping leaves CJK full-width forms behind, so
"who is X？" and "who is X" still produce different cache_keys
despite being the same question. Expanded strip set:
ASCII: . ? ! , ; :
CJK: ？(U+FF1F) ！(U+FF01) 。(U+3002) 、(U+3001)
ellipsis: …(U+2026)
Pairs deliberately stay out of the set:
" ' ) ] }
Stripping one side of a pair breaks balance. `who said "X"?` after
stripping `?` is balanced; further stripping `"` would yield
`who said "X` — different equivalence class than the original. And
apostrophes carry meaning: `X's` is a different question from `X`.
Lifted the strip set into a module constant `_QUESTION_TRAILING_STRIP`
so anyone considering an addition has a documented anchor.
Tests: 11 new in tests/test_keys.py — 6 CJK / ellipsis equivalences,
5 pair-preservation cases (double-quote, single-quote, paren, bracket,
brace). 24 keys tests + 297 default suite, all passing.
Whitepaper sibling change in ~/git/unfirehose-nextjs-logger/whitepaper/
merkle-providence-reverse-rag-whitepaper.rst — expanded the
question_hash bullet to document the equivalence class. Not committed
in this commit (different repo).
Today `question_hash` canonicalizes (NFC, ws-collapse, lowercase) but
preserves trailing punctuation. So "who is X?" and "who is X" produce
different hashes, different cache_keys, different providence_cache
records — the cache misses on what's semantically the same question.
Add `rstrip(".?!,;:")` as the third canonicalization step inside
question_hash specifically. Local to question hashing — chunk leaf
hashes go through `canonicalize()` directly and are unaffected.
Equivalence class after this:
"who is X" \
"who is X?" |
"who is X." | -> all same question_hash, all same cache_key
"who is X!" |
"Who Is X" /
Internal punctuation is preserved on purpose: "X, then Y" carries
meaning that "X then Y" doesn't, even though both have the same
content tokens.
One-time impact: prior cached records whose canonical question ended
in punctuation become orphans on lookup (re-derive on next ask).
History stays on disk; nothing burned automatically.
New test file tests/test_keys.py with 13 tests pinning the question_hash
equivalence class plus light coverage of the other dim hashes
(model_profile, conversation, governance, cache_key 8-dim invariant).
286 passed, 1 skipped.
Per fox: a query for "who is Russell Ballestrini" classified STRICT by
grounding 6/6 quotes against `feeds/all.atom.xml` — a 230-chunk dump
of post metadata. The verifier was technically correct (every quoted
string IS in the feed) but the result was hollow: feeds list URLs,
they don't carry knowledge. The ACTUAL bios live at the linked posts.
Two-pass filter in _CrawledHtmlSource.iter_documents:
1. Pre-fetch URL-pattern check (_looks_like_feed_url):
- suffix matches: .atom, .atom.xml, .rss, .rss.xml, .rdf,
/feed.xml, /atom.xml, /rss.xml, /rss2.xml, wp-rss2.xml,
wp-atom.xml, wp-rdf.xml, wp-rss.xml
- substring matches in path: /feed/, /feeds/, /atom, /rss, /sitemap
2. Post-fetch Content-Type check (_looks_like_feed_response):
- rejects: application/atom+xml, application/rss+xml,
application/rdf+xml, application/xml, text/xml
- keeps: application/xhtml+xml (xhtml IS html, just stricter syntax)
Defense in depth — a feed served at a non-feed path (e.g.
/index.html returning text/xml) still gets dropped on Content-Type.
What stays out of the corpus:
/feeds/all.atom.xml (Atom feeds)
/sitemap.xml (XML sitemaps)
/wp-rss2.xml (WordPress RSS)
/feed/ (any feed alias)
What's still allowed:
/post-name/ (real prose pages)
/index.html (HTML)
/page.xhtml (xhtml)
Tests: 12 new parametric cases (8 path patterns + 6 Content-Types +
xhtml positive case). 36 bridge tests + 273 default suite, all
passing.
For fox: the existing feed entry already in the crawl shard was burned
manually via `aborist burn --kind document --root eaf7c8d5...` — 230
chunks gone. Re-running `make crawl-ingest` won't re-introduce it.
The lifted AsyncWebFetcher already had a fast_mode constructor knob;
just unbroken plumbing was missing. fast_mode trades politeness for
throughput:
- timeouts: 5s vs 15-60s
- parallel page workers: cpu_count() * 3 vs 1
- ignores robots.txt 'crawl-delay' directive
- robots.txt 'Disallow' is STILL honored (only the delay drops)
Surface:
aborist crawl --seed-url ... --fast
make crawl-ingest URL=... FAST=1
Tests: two regressions pin that fast=True / fast=False each translate
to the right AsyncWebFetcher(fast_mode=...) construction. Capturing
fetcher fixture to avoid network. 21 bridge tests + 273 default
suite, all passing.
Per fox: 20 was a wrong default — typical sites have many more pages
within depth 2 than that, and the cap was silently truncating real
crawls. New behavior:
aborist crawl --depth 2 # no page cap
aborist crawl --depth 2 --max-pages 50 # opt-in cap when needed
make crawl-ingest URL=... DEPTH=2 # no page cap
make crawl-ingest URL=... DEPTH=2 MAX=50 # explicit cap
Implementation: bridge's BFS loop treats max_pages=0 as "unbounded"
(while-condition becomes `queue and (max_pages == 0 or len(...) < max_pages)`).
Default flows from CLI argparse default=0 down to the bridge.
Tests: pinned max_pages=0 → no cap with a 30-page chain regression
test (19 passed in tests/crawler/test_bridge.py). Default suite still
273 passed, 1 skipped.
The verbatim AsyncWebFetcher.fetch_webpage defaults extract_links=False
and returns (html, []) when omitted. Bridge's BFS therefore terminated
at the seed — a real-world crawl of russell.ballestrini.net discovered
exactly 1 URL (the seed). One-line fix; the kwarg was always there in
the lifted API.
Added test_crawl_seed_passes_extract_links_true that records every
fetch_webpage call and asserts the kwarg is True. Pin so this can't
silently regress when bridge internals get reorganized.
Updated the stub fetcher signature in the existing tests to accept
the new kwarg without behavior change.
Glues the verbatim crawler lift to aborist's existing ingest pipeline,
plus adds a freshness probe that asks "does this need recrawling?"
without paying for a body fetch when the answer is no.
Bridge (aborist/sources/crawler/bridge.py):
- crawl_seed(seed, depth, max_pages): BFS-discovers same-domain URLs
via AsyncWebFetcher. Exact-hostname filter so subdomain crossover is
off by default. Caps at max_pages. URL dedup, fragment-strip
canonicalization. Sync wrapper around an async core so the CLI
remains a one-liner.
- ingest_crawled(conn, urls): runs the URL list through the standard
ingest_source() path so chunking_version / canonicalization_version
/ Merkle commitment all stay consistent. Captures ETag +
Last-Modified per response and writes one row per document_root
into the new document_http_meta table.
Schema (idempotent migration, mirrors _migrate_mesh_peer_chains):
document_http_meta (
document_root TEXT PRIMARY KEY,
etag TEXT,
last_modified TEXT,
last_fetched_at INTEGER NOT NULL,
last_status INTEGER,
last_checked_at INTEGER,
FK(document_root) -> documents(document_root) ON DELETE CASCADE
)
Recrawl-check (recrawl_check):
- Sends conditional HEAD with If-None-Match + If-Modified-Since.
304 -> fresh (no body transfer needed). 200 -> stale (server says
the body changed). 404/410 -> gone. Other -> unreachable.
- Updates last_status + last_checked_at so consecutive runs visit
oldest-first (NULLS FIRST in ORDER BY).
- Optional --domain filter so operators can probe one site at a time.
CLI:
aborist crawl --seed-url URL [--depth 2] [--max-pages 20] [--ingest]
aborist crawler recrawl-check [--domain DOMAIN] [--limit 100]
Makefile:
crawl-ingest URL=https://example.com [DEPTH=2 MAX=20]
recrawl-check [DOMAIN=example.com LIMIT=100]
Both targets gate on bootstrap-crawler so [crawler] extras land first.
Tests: 17 new in tests/crawler/test_bridge.py — same-domain filter,
BFS bounds (depth + max_pages), dedup, fragment normalization, ETag
capture, content-type filter, HEAD classification (304/200/404),
domain filter, last_checked_at update. All stub the network via a
fake httpx.Client; default suite still 273 passed, 1 skipped.
End-to-end test path:
make bootstrap-crawler
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 MAX=20
make query Q="who is russell ballestrini?"
make recrawl-check DOMAIN=russell.ballestrini.net
Lifts the async web fetcher into aborist as an opt-in source. The
implementation comes directly from ~/git/agents.ai.unturf.com/core
(rev 2026-04-28); aborist's adaptations are minimal and documented in
aborist/sources/crawler/__init__.py:
core/async_web_fetcher.py -> aborist/sources/crawler/async_web_fetcher.py
core/web_fetch.py -> aborist/sources/crawler/web_fetch.py
Two source-side changes during the lift:
1. Heavy deps (aiohttp, bs4, miniuri) wrapped in try/except so a bare
`import aborist.sources.crawler` raises ImportError with the install
hint instead of leaking AttributeErrors deep in user code.
2. Chat-bot fetch triggers (`has_fresh_fetch_trigger`,
`has_web_fetch_trigger` from agents.ai.unturf.com/core/keywords)
replaced with NotImplementedError stubs. Aborist has no chat
surface — fetch intent is detected at the application layer. The
two test classes that exercised these triggers are
`@pytest.mark.skip`'d with the same rationale.
Not lifted: web_cache_manager.py — it backs page caching with
SQLAlchemy. Aborist has its own content-addressed cache via
providence_cache; no need to carry SQLAlchemy as a dep just for
crawled-page memoization.
Off by default:
- `[crawler]` extras section in pyproject.toml carries the heavy
deps. `[dev]` pulls them in so the crawler tests can run.
- `make test` ignores tests/crawler/ entirely.
- `make bootstrap-crawler` installs the extras into the venv.
- `make test-crawler` runs only the lifted tests after extras land.
Tests: 74 passed, 9 skipped (the chat-bot trigger tests deliberately
dropped). Default `make test` stays at 273 passed, 1 skipped.
Receiver now rejects gossip envelopes whose prev_event_hash doesn't
extend the sender's last-seen event_hash — fork detection lives at the
wire layer, not just in docs/mesh.md.
Schema: new mesh_peer_chains table (peer_member_id PK, last_event_hash,
last_seq, last_seen_at) with idempotent _migrate_mesh_peer_chains
matching the _migrate_audit_mode pattern.
Envelope: WireEnvelope gains prev_event_hash + event_hash, both
optional for back-compat. canonical_bytes() excludes event_hash to
avoid the circular dependency (the hash is computed FROM the canonical
bytes). New WireEnvelope.with_chain(prev_event_hash=...) builds a
linked envelope with event_hash = sha256(prev || canonical_bytes).
Sender (MeshWireClient._signed_envelope, MeshWireServer.handle_request
DELIVER_BODY): reads our latest_event_hash, builds the linked
envelope, then appends a 'mesh_sent' audit event whose body is the
envelope's canonical (event-hash-excluded) dict. By construction the
audit event_hash equals the envelope event_hash — wire chain-of-claims
and local audit chain stay in lockstep, so back-to-back sends advance
the chain naturally.
Receiver (handle_announce): when an envelope carries chain fields,
recompute event_hash and 400 on mismatch; look up
mesh_peer_chains[sender] and 409 on prev mismatch; on accept update
the row + write the existing mesh_received event.
Backward compat: legacy envelopes (event_hash=None) bypass chain
enforcement — kept the existing test_mesh_wire.py fixtures verbatim
since they construct WireEnvelope directly without with_chain(). All
production traffic goes through MeshWireClient and is always tracked.
v1 deferred (documented in module docstring): no multi-event catchup;
operator retries on 409. No cross-peer reconciliation.
Tests: 12 new in tests/test_mesh_chain.py covering the 7 required
cases plus canonical-bytes exclusion, migration idempotence, and
legacy-envelope fallback. 246 passed, 1 skipped overall.
Adds an opt-in confidentiality layer to gossip. Senders pass
encrypt=True on announce_root / announce_falsification; bodies travel
AEAD-encrypted (ChaCha20-Poly1305) under the per-epoch symmetric secret
recovered via the existing X25519 wrap. Cleartext stays the default —
all existing wire tests pass without modification.
WireEnvelope grows an optional encrypted_body field. canonical_bytes
omits the field when None (byte-identical to v1 cleartext format) and
includes it when set, so the Ed25519 signature binds the ciphertext —
a MITM cannot swap payloads. AAD is type|sender|epoch|ts so a captured
(nonce, ct) cannot be replayed under different envelope metadata.
state.recover_epoch_secret resolves the rotator from the audit chain
and re-unwraps this peer's slot. Sender + receiver share one path; no
new sender-side cache is required.
Closes the request half of the gossip loop. MeshWireClient already
verifies signature + Merkle root on delivered bodies; this verb adds
the CLI surface, gating, local re-ingest through the standard chunker,
and a 'mesh_pulled' audit event. Re-ingest is rejected if the local
document_root differs from the requested root.
burn --kind {providence,document,core}; default 'providence' preserves
the prior --cache-key surface (no breaking change).
Children gates for document/core leaves:
derivations.src_root downstream cores derived from this doc
edges.dst_root other docs link to this doc
providence_cache.source_root Q&A grounded in this doc
derivations has ON DELETE CASCADE on BOTH core_root and src_root, so a
naive DELETE FROM documents would silently cascade-prune downstream
derivations and orphan deeper cores. Gate before delete; --force to
override with the count recorded in the audit body.
Burn manually clears chunks_fts (FTS5 has no FK) and outbound edges
(no FK either) before deleting the document. chunks + merkle_nodes +
inbound derivations cascade via FK.
Audit event types are document_burn / core_burn — distinct from
providence_burn so chain consumers can tell leaf kinds apart.
CLAUDE.md "Cores never evict" remains intact: burn is operator-driven
removal, evict is the automated cold-tier compression that touches
only kind='surface'.
Makefile: KIND=providence|document|core selects the leaf type;
KEY for providence, ROOT for document/core. Single 'burn' target keeps
the surface DRY.
Tests (tests/test_burn_doc.py, 13 new):
- happy path: chunks + merkle_nodes vanish via cascade
- refused on derivation downstream / providence ref / incoming edge
- --force overrides and records counts
- core burn happy path (no children) preserves parent surface
- core burn refused with downstream derivation
- audit chain integrity preserved (no dangling prev_event_hash)
- CLI invocations for document, core, default-providence, refused
Full suite: 247 passed, 1 skipped.
Both ask() (single-doc) and query() (multi-source) now run the
assembled context through aborist.wikitext.to_base() before message
construction. Hermes sees clean prose instead of raw wikitext, which:
- lets the model quote verbatim (raw [[Cloud Strife]] was unquotable
in fluent prose; the model paraphrased to bare "Cloud Strife" and
the verifier flagged that as UNGROUNDED)
- shrinks token bills ~43% on Wikipedia chunks (measured: 125,152 →
71,771 chars on the FF7 main-character query)
- aligns LLM-input with what the verifier already strips, so the
prose-vs-prose substring check is end-to-end consistent
Gated on policy["base_version"] (= wikitext-base-v1) so:
- governance_policy_hash flows the choice — prior answers cached
under raw-wikitext policy retain distinct cache_keys, no migration
- operators can opt out (set base_version=None) for debugging
- environments without mwparserfromhell soft-fall to raw wikitext
with no other behavior change
Tests (4): default policy carries base_version (tripwire), ask() user
messages have wikitext markers stripped, opt-out reverts to raw
wikitext, governance_hash differs with vs without base_version.
Wires the gossip-wire core (commit f141bab) into the user-facing CLI:
aborist mesh serve --host 127.0.0.1 --port 8400
Spawns MeshWireServer, blocks until SIGINT. Refuses if mesh isn't
initialized or .enabled is off — same gate every other mesh verb
enforces. Stdout is unbuffered JSON status lines.
aborist mesh sync --peer http://other.example:8400 [--limit N]
Enumerates local documents (most-recent N, default 100), fires one
ANNOUNCE_ROOT per document via MeshWireClient, reports counts of
acknowledged vs. errored. Pulling missing roots back from the peer
is a v2 addition — this verb pushes only.
Tests (6): gating refusals on uninitialized + disabled mesh; sync
round-trip with two peers proving Alice's audit chain gains one
'mesh_received' per Bob announce; unreachable-peer reporting; serve
smoke-test via start_in_thread + GET /mesh/info.
Burn deletes a providence_cache row outright; falsify (the existing
verb) flips state and keeps the row. Burn is for early/scratch use
during a tree's genesis when downstream consumers haven't ingested
the answer yet.
Children gate: refuses if any falsifications reference the cache_key.
Caller can override with --force; the audit event records forced=True
plus the child count at burn time so the chain can reconstruct what
was orphaned. Refusal is non-destructive AND skips the audit event so
scripts can retry without polluting history.
Always writes a 'providence_burn' audit event regardless of force.
Body fields: cache_key, burned_audit_mode, burned_n_verified,
burned_state, question_text, reason, by_actor, child_falsifications_at_burn,
forced. The chain-break query (`make chain-check`) shows 0 after burn —
the new event extends the prior chain cleanly.
Surface:
aborist burn --cache-key <hex> [--reason '...'] [--force]
make burn KEY=<hex> REASON='...' [FORCE=1]
Tests (7): happy-path delete + audit event, refusal on children,
--force override, unknown-key not-found, chain-integrity preserved,
CLI happy path, CLI non-zero exit on refusal.
- aborist/qa/verify.py: three strategies tried in sequence. quote uses
sequential pairing (1st & 2nd `"`, 3rd & 4th, ...) which eliminates
the phantom inter-pair captures regex pairing produced on adjacent
quote pairs like `"title" prose "quote"`. span checks bullet/sentence
lines verbatim. entity matches multi-word proper nouns; gated by
policy ∈ {strict, hybrid, drop, proximity}. Default proximity
promotes to STRICT only when N=3 verified entities cluster within
W=300 chars in source — separates structural grounding (cast list,
infobox) from incidental mention (scattered plot summary).
- aborist/store.py: schema CHECK now `('STRICT','HYBRID','UNGROUNDED')`.
Migration helper rebuilds legacy `('STRICT','HYBRID','VISUAL')`
tables via temp-table copy, translating VISUAL → UNGROUNDED in the
SELECT. Idempotent — DDL inspection skips the rebuild on already-
migrated DBs.
- aborist/cli.py: new `aborist reclassify` re-runs the verifier
against existing live records under the current entity policy; no
LLM calls. --compare runs all four policies side-by-side, --dry-run
reports transitions without writing. Cold-source records skipped.
One providence_reclassify audit event per changed row.
- AuditMode.UNGROUNDED replaces VISUAL across search backend, FTS5,
test fixtures, CLAUDE.md. The substrate (Merkle-AGI) name was
about FOR-style visualization; in the RAG layer the semantic is
"no recoverable grounding," so the label now says that.
DEFAULT_QUERY_POLICY gains entity_policy + entity_proximity_n +
entity_proximity_window so any tuning folds into governance_policy_hash
and invalidates cache cleanly.
Implements the protocol contract pinned in docs/mesh.md. Each peer
runs a stdlib ThreadingHTTPServer; clients send Ed25519-signed
envelopes carrying one of:
ANNOUNCE_ROOT document_root + source_uri + versions
ANNOUNCE_DERIVATION core_root <- surface_roots (proof-blob hash)
ANNOUNCE_PROVIDENCE cache_key + audit_mode + answer_hash
ANNOUNCE_FALSIFICATION cache_key + reason
REQUEST_BODY/DELIVER_BODY pull-on-miss + Merkle-root verify
WireEnvelope canonicalizes via sorted-key separator-tight JSON; the
canonical bytes are what get signed. Receivers verify the signature
against sender_id's sign_pub looked up in mesh_roster at the
envelope's epoch_id — non-members of that epoch produce no valid
signature, so they're silently rejected (401, no audit event).
Each accepted ANNOUNCE_* writes one 'mesh_received' event into the
local audit chain whose body is the full signed envelope (the
remote's signature stays attached for non-repudiation). The
receiver's chain stays internally consistent because we append in
receive-order.
REQUEST_BODY/DELIVER_BODY: the responder ships the document text
plus per-chunk leaf hashes; the client re-derives the Merkle root
from those leaves and refuses to return any body whose leaves
don't reconstruct the requested root.
Tests: 17 unit (envelope canonicalization, sig verify against
roster, type validation, tampered-body rejection, audit-event
write) + 5 end-to-end (two real HTTP peers on ephemeral ports
exchanging announces and bodies). 22/22 green.
Deferred to later commits, per docs/mesh.md:
- CLI verbs (mesh serve, mesh sync) — pending downstream merge
- per-peer chain-of-claims tracking (catchup, replay, fork detection)
- optional AEAD encryption of message bodies (contract says optional)
Adds aborist/wikitext.py with to_base() — deterministic wikitext →
prose conversion via mwparserfromhell. Drops <ref>...</ref>,
[[File:...]], [[Image:...]], [[Category:...]] entirely; resolves
piped wikilinks to display text; collapses templates, formatting,
and HTML markup. extract_wikilinks() preserves the link graph for
the definition-cloud artifact (recoverable from any page on demand
without re-parsing wikitext at query time).
Wires to_base() into verify_quotes(). Without the strip, the
verifier compared the model's clean prose against [[Cloud Strife]],
[[Shinra Electric Power Company|Shinra]], etc. and falsely flagged
real source quotes as VISUAL. Concrete case from a make-query run:
'Cloud Strife, an unsociable mercenary who claims to be a former
1st Class member of Shinra's SOLDIER unit;' is verbatim in the
Final_Fantasy_VII article wikitext (modulo markup). With the strip
that quote now verifies; the genuine model hallucinations in the
same answer still flag honestly. Side-effect: 43% smaller context
size on average so less LLM token waste.
BASE_VERSION = 'wikitext-base-v1' is the algorithm pin. Bump when
the strip rules change. Soft-imported in verify.py so environments
without mwparserfromhell installed degrade gracefully (no strip,
same behavior as before this commit).
Build:
- pyproject.toml: new [wikitext] extras (mwparserfromhell>=0.6),
pulled in by [dev]
- Makefile: chain-check / chain-check-shards targets — fast
audit-chain integrity probe, counts dangling prev_event_hash
references; 0 = chain intact
Tests:
- tests/test_wikitext.py: 31 tests (rules, idempotence, real-corpus
fixture)
- tests/test_verify.py: 2 regression tests pinning the FF7 flip
(Cloud Strife quote: VISUAL → STRICT after strip; genuine
hallucination: stays VISUAL)
- tests/fixtures/ff7_characters_chunk0.wikitext: real chunk from a
shard, used to validate the strip on actual Wikipedia content
Adds aborist/qa/verify.py — three-strategy verifier (explicit quotes,
bullet/sentence spans, multi-word proper nouns) that lexically checks
every claim against retrieved context under norm-v1 + lowercase. The
strategy that fires is recorded as verifier_method for diagnostics.
Entity strategy gates classification via an entity_policy
(strict/hybrid/proximity/drop) so a single proper-noun match no longer
overclaims STRICT — proximity (default) requires a cluster of 3+
verified entities within 300 chars.
Wires into ask() and query(). System prompts now require verbatim
quoted spans for every factual claim, restated via a user-turn
grounding_reminder one message before sources arrive (recent
user-turn instructions outweigh decayed system-turn rules under long
context in Hermes).
providence_cache gains 5 columns (audit_mode, n_quotes, n_verified,
unverified_quotes, verifier_method) with CHECK constraints. A
connect-time _migrate_audit_mode() ALTERs legacy DBs idempotently.
Cache hits return the persisted audit_mode rather than asserting
STRICT unconditionally.
CLI:
emergent list VISUAL/HYBRID records; --aggregate ranks
unverified quotes by frequency (corpus-growth signal).
reclassify re-run the verifier against live providence records;
cold-source records skipped; --dry-run reports
transitions without writing; each change writes one
'providence_reclassify' audit event.
A snapshot_root is MerkleTree.build([sorted DISTINCT document_roots]).root
— one 32-byte hash naming the entire content-addressed forest at a
point in time. Two peers that ingested the same dump compute bit-
identical snapshot_roots, so cross-machine "are we synced?" becomes
an O(1) hash comparison; the same root doubles as the TF-IDF
scope_root we sketched in the mesh design (snapshot_root *is* a
scope), and pins Q&A answers to a verifiable corpus state.
Schema: one new table, additive over existing data.
snapshots(snapshot_root PK, taken_at, audit_event_hash, doc_count,
parent_snapshot, reason)
+ idx_snapshots_taken_at
API:
compute_snapshot_root(conn, *, document_roots=None) -> (root, count)
create_snapshot(conn, *, reason, parent_snapshot=None) -> dict
verify_snapshot(conn, snapshot_root) -> dict (matches: bool)
diff_against_current(conn, snapshot_root) -> dict (added/removed/unchanged)
list_snapshots(conn, *, limit) -> list[dict]
CLI:
aborist snapshot create [--reason "..."] [--parent <hex>]
aborist snapshot list [--limit N]
aborist snapshot verify <snapshot_root>
aborist snapshot diff <snapshot_root>
Cross-shard: with --shards-dir + --db, the snapshot is computed over
the cluster-wide UNION view but persisted into args.db (a dedicated
snapshots store, conventionally ~/.aborist/shards/snapshots.db).
parent_snapshot auto-links to the latest prior snapshot in the
writer DB, giving a chain for free.
Each snapshot creation writes an audit_event of type 'snapshot_create'
with subject_root=snapshot_root, so the corpus's pinned states are
themselves tamper-evidently logged.
Defect caught + fixed during live test on the 2010 enwiki ingest:
verify and diff disagreed on the same connection because compute used
a list (with cross-shard duplicate document_roots) while diff used a
set. Two shards can land identical document_roots when canonicalize()
maps two structurally-similar pages to the same byte stream — rare
but real. Switched the underlying SQL to SELECT DISTINCT so a
membership snapshot is always order- AND multiplicity-independent.
Live test: 2010-11 enwiki corpus snapshotted at
43797e46605de08dbab06cdcaf5be7ad78243b193c56f8580200dee6bcc7e1b9
doc_count: 3,468,134 (after dedup)
verify + diff round-trip both report identical against current state.
11 new tests covering empty corpus, single-doc degenerate, order
independence, drift detection, audit-chain pinning, parent auto-link,
and idempotent creation on unchanged corpus. 136 tests + 1 skipped.
Phase 1 of the federation/gossip layer fox sketched as the natural
extension of v9.8 admissibility's content-addressed identity. Two
peers ingesting the same dump already compute identical document_roots
and identical 8-dim cache_keys; the mesh layer is the wire-and-trust
plumbing that lets them dedup answers, exchange Merkle proofs, and
cleanly distrust an evicted member without a hard fork.
Cryptography (cryptography lib, audited):
Ed25519 — every membership mutation + (future) gossip envelope
is signed by the actor's pubkey.
X25519 ECDH — wraps each epoch's symmetric mesh secret to every
current member's DH pubkey via HKDF-derived AEAD key.
ChaCha20-P1305— AEAD for envelope payloads + per-member secret wrap.
State machine:
mesh_identity — singleton; this peer's keys + group name
mesh_roster — per-epoch (member_id, sign_pub, dh_pub, role)
mesh_epochs — epoch_id -> {started_at, started_event_hash,
secret_envelope JSON, reason}
meta:mesh.enabled flag — off by default; gates everything
Eviction works by rotating to a new epoch whose envelope omits the
kicked member. Their prior signatures stay verifiable (the older
roster row is retained), but any gossip from epoch+1 onward is
opaque to them — the secret was never shared with their pubkey.
Authority gate: only roster members with role='admin' can add or
kick. Self-kick is rejected explicitly. The last admin can't be
kicked. Schedule-rotate (refresh secret, no roster change) is open
to any current member as a session-hygiene op.
Audit-chain integration: every mesh state mutation writes an audit
event (mesh_init, mesh_enable/disable, mesh_epoch_rotate). The
epoch's started_event_hash backfills into mesh_epochs after the
audit row commits, giving each epoch a tamper-evident pin into the
ledger.
CLI subcommands: mesh init, mesh status, mesh enable, mesh disable,
mesh members, mesh add, mesh kick, mesh rotate. All read-only or
local-state-only — no network code paths in this commit.
The HTTP gossip wire (`mesh sync`, `mesh serve`) is the next phase.
Schema, cryptography, and roster state machine are all in place to
support it without further migration.
GitRepoSource walks the working tree at HEAD via `git ls-tree` + `git show`,
yielding one Document per text file. URI shape `git://<repo>/file/<path>`
intentionally omits the commit hash — re-ingest after new commits produces
fresh document_roots that aborist's prior-doc detection chains via
`supersedes` edges, so the audit trail and Merkle tree grow as the repo
grows. Binaries skipped via NUL-byte + UTF-8 decode probes; >5 MB files
filtered out by default. Commit hash + timestamp + subject ride along in
`extra` (informational only; not part of the Merkle commitment).
MercurialRepoSource mirrors via `hg manifest` + `hg cat`. Same supersedes
semantics, same shape.
Makefile adds:
make ingest-self (this repo -> aborist-self.db)
make ingest-git GIT_REPO=/path/to/repo (arbitrary git clone)
make ingest-hg HG_REPO=/path/to/repo (mercurial)
Each lands in its own shard file alongside existing shards/grok.db,
keeping per-shard write paths independent of the wikipedia 4-way ingest's
WAL writer lock.
Three cheats stack to drop on-disk store from ~21 KB to ~6.7 KB per doc on
the 2003 enwiki cur corpus (-67% measured, apples-to-apples reingest with
identical document/edge counts; Merkle proofs round-trip 30/30):
1. zstd-compressed chunks.content (level 3). Magic-byte detection on read
means legacy plaintext rows pass through unchanged. Cores stay plaintext
so qa.query._docs_with_core_keyword_match's SQL LOWER+LIKE keeps working.
2. edges WITHOUT ROWID. The composite PK (src_root, edge_type, dst_root,
dst_uri, anchor) covers every column, so a default rowid-based table
near-doubles row data in the PK index. WITHOUT ROWID makes the table
itself the B-tree. Drops idx_edges_dst_uri too — the only query that
filters on dst_uri alone is gravity_top_inbound, a one-shot analytic.
3. contentless FTS5 (content='', contentless_delete=1) eliminates the
28 MB / 1000 docs of duplicated chunk text the old chunks_fts stored.
chunks gets an explicit chunk_id INTEGER PRIMARY KEY so the FTS5
rowid maps back to chunks.chunk_id at search time. Snippets are
built in Python (search/fts5.py:_build_snippet) since SQL snippet()
returns empty in contentless mode.
TF-IDF retrieval also fixed: the prior LIKE '%intel%' substring match
let "intelligence", "intellectual", "intellivision" drown real hits like
Pentium_4 (whose TF-IDF core has "intel" as an exact keyword). Now uses
word-boundary `LIKE '%, intel, %'` patterns plus a match_count over the
distinct query tokens — multi-token coverage outranks single-token title
boosts. Pentium_4 surfaces #1 for "what is the fastest intel CPU?" with
the canonical 2003 answer (Pentium 4 3.20 GHz) instead of an empty
"insufficient sources" reply.
Schema-level changes affect new DBs only; existing v9.8 DBs keep
working at the old layout. Cross-shard UNION views explicitly list the
intersection of columns so a mixed cluster (legacy + new schema shards
in one --shards-dir) still unions cleanly.
Drops in for any dated snapshot in dumps.wikimedia.org/archive — pages-articles.xml.bz2
and pages-meta-current.xml.bz2 (cur snapshot, single-revision-per-page) plus
pages-meta-history.xml.bz2 (multi_revision=True). abstract.xml feed yields
pre-distilled summaries at ~1/100th the chunk volume. Streams .bz2/.gz directly
via iterparse with bounded memory; same shard / resume contract as the SQL
source. Falls back to title-prefix namespace filtering when older export
schemas omit the per-page <ns> element (e.g. enwiki 20101011).
Makefile defaults target enwiki 20101011 (6.2 GB). Override with WP_XML_YEAR /
WP_XML_MONTH / WP_XML_DATE / WP_XML_LANG to fetch any other archived snapshot.
aborist/sources/grok.py exposes two Source subclasses for ingesting
xAI's user data export:
GrokExportSource walks ttl/<period>/export_data/<user-id>/ and
reads prod-grok-backend.json. Yields one Document
per conversation with:
- URI: grok://conversation/<conversation-id>
- title: the conversation's auto-generated title
- content: full message text in turn order
source_type='grok_export'.
GrokMediaPostsSource same export, but yields per media-generation
post (image/video prompts) under URI
grok://media/<post-id>.
source_type='grok_media'.
Both auto-walk down from the export root so callers can pass the
top-level directory xAI delivered (e.g., ~/Downloads/<user-uuid>/).
Wired into the CLI: aborist ingest --source {grok_export,grok_media}
accepts --path <export-root>. tests/test_grok_source.py covers the
walk + parse + Document shape with a fabricated mini-export fixture.
Once ingested, conversations become normal queryable docs in the
shard cluster — your prior chats become memory the corpus can
consult during RAG.
79 tests passing.
A small knowledge-graph layer that does two distinct jobs:
1. SYNONYM_GROUPS broaden retrieval. A query mentioning "Athlon"
now also matches AMD-titled docs because Athlon IS an AMD product.
Groups currently cover AMD-family, Intel-family, HTTP family,
FTP, Mac, Windows, Linux. Easy to extend.
2. RIVALRIES narrow retrieval. The pair (AMD-group, Intel-group)
means: if the query mentions one side and not the other, drop
docs whose titles contain the OTHER side's tokens. So a "fastest
AMD CPU" question never gets Pentium_4 in the context — even if
FTS5 BM25 ranks it high — because Pentium is in the Intel group
and Intel isn't in the query.
COMPARE_WORDS ("vs", "versus", "compare", "between", ...) suppress
the exclusion. "compare AMD vs Intel" keeps both sides. "what is
the fastest AMD CPU?" does not.
Demos against the 128k Wikipedia 2003-05-16 cur shards:
Q "fastest AMD CPU" → 8 sources, all AMD/CPU titled, NO Intel
Answer: "Athlon XP 3200+" (real AMD chip,
grounded in the AMD article)
Q "compare AMD vs Intel"
→ 8 sources, mix of Intel_8028x, Intel_8048x,
AMD_Duron — both sides preserved
Q "Athlon processor" → AMD, AMD_Duron, AMD_5x86, Athlon all
surfaced via synonym expansion
Phase 1 implementation hand-curates the groups; Phase 2 idea is to
derive them from Wikipedia's link/category graph (dense bidirectional
clusters → synonym groups; same-category-without-cross-links →
rivalry candidates).
66 tests passing.
aborist/qa/query.py exposes query() — the user-facing RAG flow:
1. FTS5 search across all shards (chunks_fts can't be UNION'd as a
view, so each shard's index is queried independently and merged
by score).
2. Top-K distinct documents are selected within a max-context-chars
budget (default 60 KB so a 768-token response fits Hermes-3's
82 K context window comfortably).
3. context_root = Merkle root over the sorted source document_roots.
That's the v9.8 'source' dimension for multi-source answers —
a verifier can recompute it from the listed source roots.
4. 8-dim cache_key over (context_root, question_hash, model_profile,
conversation, governance_policy, schema, canonicalization,
chunking). Hit returns STRICT immediately; miss calls Hermes and
persists.
CLI: aborist [--shards-dir DIR] query "<question>"
Default qa_db is <shards-dir>/qa.db (or ~/.aborist/qa.db). Uses the
same OpenAICompatibleClient/StubClient as `ask`. --dry-run skips the
LLM and returns context-only.
Search escape fix: the prior FTS5 escape ANDed every token including
stopwords + punctuation, so "What is anarcho-capitalism?" required
the doc to literally contain "what" + "is" + "anarcho-capitalism?" —
zero hits. New tokenizer drops stopwords + punctuation and ORs the
remaining content tokens; BM25 ranks the multi-token matches highest.
Live demo against the 122k-doc 4-shard cluster:
Q "What is anarcho-capitalism?" 6.1 s wall miss / 0.45 s cache hit
Q "Who was George Washington?" 10.3 s wall miss
Both answers cite the source URIs Hermes was given.
57 tests passing (4 new query tests covering search → context →
cache → audit chain).
Resume (rsync-style)
Each shard DB gains a `meta` table. After every successful batch
flush, ingest_source persists `source_high_water:<source_type>` ->
the largest row id seen (cur_id or old_id). On --resume, the source
reads it back and skips rows whose id is <= the mark, so an ingest
killed at any point can be re-run cheaply: already-cached docs are
fast-forwarded past without re-hashing or DB writes.
WikipediaSqlDump now exposes `start_id` (skip threshold) and
`last_id` (running max). cur_id is surfaced in Document.extra
alongside old_id so both tables behave the same.
CLI: aborist ingest --resume
Demo on shared DB (cur):
Round 1: --limit 3000 --resume high_water = 5714
Round 2: --limit 5000 --resume skips 1..5714, picks up at 5715
Round 3: --limit 100 --resume skips 1..15362, ingests 100 more
Always idempotent on re-run; no dups, no missing docs.
Per-shard audit chain integrity for sharded analyze
Cross-shard analyze previously reported nonsense breaks counts —
each shard owns its own audit chain (genesis -> ... -> latest), and
the UNION view interleaves them so cross-shard transitions look
like break events. The fix: when --shards-dir is set, open each
shard's DB directly and run _check_audit_chain on it, then
aggregate.
Output now reads:
"audit_chain": {
"events": <total>,
"breaks": 0,
"shards": [{"shard": "000.db", "events": N, "breaks": 0}, ...]
}
53 tests passing. Tests cover: high-water write, skip-on-resume,
idempotency across two resume runs, kill-and-resume continuity.
The 2003-05-16 archive ships three files:
20030516_cur_tablesql.bz2 82 MB current snapshot (single revision/page)
old_tablesqlbz2.1 640 MiB \
old_tablesqlbz2.2 252 MiB / split halves of old (full revision
history). Concatenate before bzcat.
Generalize the parser:
WikipediaSqlDump(table='cur'|'old') — shared statement parser, single
column-position contract for the
first 4 fields (id/ns/title/text)
WikipediaCurDump — back-compat wrapper, table='cur'
WikipediaOldDump — new, table='old'; old has no is_redirect, every
revision is real
Old rows surface old_id and old_timestamp via Document.extra so a
downstream pass can sort revisions chronologically before re-ingesting
through the supersedes-edge path.
Makefile gains:
fetch-cur / fetch-old / fetch (both)
ingest-cur / ingest-old / ingest (cur default)
WP_OLD target concatenates the two split parts
CLI ingest --source now accepts wikipedia_cur or wikipedia_old.
Smoke (real dump): 5 revisions of "AtlasShrugged/Companies" yielded
correctly with old_id=2..10, timestamps from January 2002.
aborist/qa/ implements the cache-first answer flow from the providence
whitepaper, scaled up to v9.8's full 8-dim admissibility invariant.
cache_key = SHA-256 of:
source_root | question_hash | model_profile_hash | conversation_hash
| governance_policy_hash | schema_version | canonicalization_version
| chunking_version
Any drift in any dimension yields a distinct cache_key — prior records
cannot serve. Falsification states (failed/stale/quarantined) gate
every cache hit.
- qa/keys.py — pure hash functions, deterministic & testable
- qa/client.py — ChatClient Protocol + StubClient + OpenAI-compatible
HTTP client (vllm/llama.cpp/uncloseai compatible)
- qa/runner.py — ask(): lookup -> hit (no LLM call, hit_count++)
OR miss (call client, write record, audit event,
proof binds answer to source root)
- cli.py — `aborist ask` and `aborist providence` subcommands
- pyproject.toml — httpx promoted from extras to core (used by both
html and qa); selectolax stays in [html] extras
Smoke (StubClient, no network): cache miss writes record with chunk_0
Merkle proof reconstructing source_root; cache hit returns same record
without calling client; 1085 audit events chained 0 breaks across
ingest/derive/evict/rehydrate/providence_write.
When a re-ingest produces a different document_root for an already-seen
URI, both versions now coexist in the store. A 'supersedes' edge from
the new doc to the prior one keeps the lineage addressable, and the
ingest audit body records the supersedes link.
Idempotent re-ingest is unchanged: identical bytes -> identical root ->
no new doc, no supersedes edge.
Three-version chains test that v3->v2 and v2->v1 edges form a walkable
history. The old version stays queryable (its chunks may later be
evicted to cold for storage savings while remaining provable).
Pure-Python (stdlib only). Uses chunk-level corpus baseline so terms
concentrated in fewer chunks outrank common ones. Default top-K = 16.
Output cores are comma-separated keyword lists — extreme compression
toward the tweet/haiku end of the planet metaphor. Same source can
now carry both a first-sentence-v1 core AND a tfidf-keywords-v1 core,
each derived independently and Merkle-signed back to the same surface.
The 'contributing_chunk_indices' for TF-IDF is every chunk that
contains at least one of the top-K keywords — proof binding remains
honest and cryptographically tight.
Distillers now scan kind='surface' OR kind='core'. Cores generate
deeper cores with compression_depth incremented. Each round of
recursion tightens the planet toward its center.
Audit body for derive events now records src_kind and the resulting
compression_depth so the chain reflects the layer transition.
Smoke against the existing 503 surface + 478 depth=1 core corpus
produced N deeper cores with proofs still binding back to their
depth=1 sources.
