verify_claim_lattice now derives pointer_id_distribution and
lazy_anchor_ratio from the verified-or-partial claim_statuses and
returns them in the verdict. Surfaced through ask()/query() result
dicts; CLI human-render displays a one-liner when ratio >= 0.5
(threshold for "model is lazy-anchoring").
Sidecar discipline preserved: never persisted in providence_cache,
never threaded into build_run_dag's verify_payload (test pins this
invariant — sidecar values must not enter run_dag_root). The
distribution is recoverable from claim_statuses which IS persisted,
so future sidecar tools can recompute on demand without enlarging
the audit chain.
JP dinosaurs benchmark currently triggers the warning at 11/11
ratio 1.00 — Hermes-3-8B cites [E1] for every claim even with
G0.3's relevance ranking putting the dinosaur-mention chunk near
the top. The render-layer signal now makes that lazy-anchor
behavior explicit instead of leaving it for the auditor to spot
from the spotlight excerpts.
Tests: +4 covering the ratio across distributions (single-anchor /
diversified / no-verified-pairs) plus the run_dag isolation
invariant. Total 442 passing.
Architecture diagram gains aborist/qa/evidence.py and
aborist/qa/parse_claims.py; aborist/qa/dag.py callout makes the
quote-vs-pointer DAG-shape split explicit. New conventions entry
covers the full G0 contract — two-layer ids (pointer-id E1/E2
shown to the model, evidence-id E######## used by cache & DAG),
pointer-line protocol, deterministic verifier (no entailment in the
proof path), strict no-quotes rule, spotlight rendering with
literal-span interpolation, per-chunk evidence granularity with
query-relevance ordering, 9-stage CTI run-DAG, no iterative repair,
and the Makefile testing default vs library defaults split.
New policy["answer_mode"] = "claim_lattice_pointer" (default "quote").
The runtime builds an evidence map with two-layer ids — pointer-id
(E1, E2, ...) shown to the model, sha256-derived evidence-id used by
the cache & run-DAG — and the model emits pointer-line prose
("Claim. [E12]") referencing them. Renderer interpolates literal
source spans at display time. Synthetic-elision-by-construction-
impossible: the model never types the quote string.
Pieces:
- aborist/qa/evidence.py (new): EvidenceObject + spotlight excerpt
(claim-token-centered window into the cited span, falls back to
leading window when no token matches).
- aborist/qa/parse_claims.py (new): pointer-line parser walks
lines, pulls [E\d+] / [E\d+,E\d+,...] tags, returns ParsedClaim
with PARSED / NO_EVIDENCE_POINTER status. Strict regex refuses
fuzzy alternatives so honest UNGROUNDED beats lax acceptance.
- aborist/qa/verify.py: verify_claim_lattice. Hard checks only —
parser succeeded, evidence_id resolves, source_role allowed, no
manual quotes (any " char violates), claim text non-empty.
Returns evidence_id_pairs (content-addressed, run-stable) for the
run-DAG. Soft signals (entailment, completeness, predicate
compatibility) stay sidecar.
- aborist/qa/dag.py: 9-stage CTI shape when evidence_map_root is
supplied — question / retrieval / evidence_map / prompt /
raw_answer / parsed_claim_lattice / verify / render /
final_label. Quote mode keeps the original 7-stage shape so
pre-G0 run_dag_root values stay valid.
- aborist/qa/query.py: per-chunk evidence-map build with role-
weighted budget AND query-relevance ordering. Within each source,
chunks are ranked by (distinct_query_tokens_present,
total_mentions, doc_order_asc) so the chunk most likely to
textually support the question gets the lowest pointer id. Without
this re-rank Hermes-3-8B lazy-anchored on doc-order-first chunks
regardless of relevance.
- aborist/qa/runner.py: same answer_mode branch for the single-doc
ask() path. No iterative repair in pointer mode (one-shot
benchmark discipline).
- aborist/store.py: verifier_method CHECK extended with
'claim_lattice'. New _rebuild_providence_cache_claim_lattice
migration preserves run_dag_root / run_dag_blob across the
rebuild — older rebuilds dropped them.
- aborist/cli.py: --answer-mode {quote,claim_lattice_pointer} on
query and ask.
- Makefile: ANSWER_MODE knob; defaults to claim_lattice_pointer for
`make query` so the testing harness exercises the new path.
Library DEFAULT_POLICY / DEFAULT_QUERY_POLICY stay "quote" so
Python callers and unit tests aren't surprised.
Prompt: one-shot worked example (Apple founders) plus strict
no-quotes / no-JSON / no-markdown / plain-prose-only rules. Without
the worked example Hermes-3-8B drops the bracket protocol on roughly
half of runs; with it, JP dinosaurs benchmark went 0/13 -> 10/11 ->
16/17 verified across the iterations that hardened the pipeline.
Tests: +48 covering parser, two-layer ids, verifier failure modes
(UNKNOWN_EVIDENCE_ID / SOURCE_ROLE_BLOCKED / MANUAL_QUOTE_VIOLATION
/ NO_EVIDENCE_POINTER / SCHEMA_INVALID), spotlight rendering with
buried-term fixture, per-chunk evidence map, query-relevance
ordering, schema migration round-trip, 9-stage DAG shape divergence.
Total 438 passed, all 7 production shards report 0 chain breaks.
Known limitation: chunk boundaries can cut wikitext mid-template, so
mwparserfromhell-backed to_base() leaves orphan </ref> tags and
leading list markers in the visible spotlight excerpts. Verifier and
CTI architecture are unaffected; the leak is cosmetic. Proper fixes
are template-aware chunking (chunker bump invalidates prior records)
or an orphan-marker post-strip in aborist/wikitext.py — both out of
G0 scope.
burn-kindergarten flake fix:
test_burn_kindergarten_zero_seconds_burns_everything failed under
full-suite ordering when the wall-clock second rolled over between
seeding the row & running burn-kindergarten. Old impl computed
`cutoff = now - 0 = now` and required `created_at >= now`, so a row
seeded at second T became invisible to a burn check that ran at T+1.
The verb's docstring is "0-second window = burn every live row".
Fix the impl to match: when kindergarten_seconds <= 0, drop the time
gate entirely (burn all live records). Non-zero windows keep the
existing cutoff-based filter unchanged.
Repair CLI:
`aborist query --repair` enables the mechanical repair pass after
first verify. `--repair-reprompts N` adds the optional re-prompt
tier (default 0 = off; mechanical-only when --repair is set without
--repair-reprompts). Both flags are off by default so `make query`
stays single-shot unless the operator opts in.
Makefile: `make query REPAIR=1` flips on mechanical repair;
`make query REPAIR=1 REPROMPTS=1` adds one re-prompt iteration.
Both knobs off by default to preserve existing behavior.
485 tests pass under full-suite ordering (was 484 + 1 flake).
Two enhancements continuing the toy-Hermes design pass:
#1 Chain-segment failure localization
aborist/qa/dag.py: `localize_failure(audit_mode, n_sources, n_quotes,
n_verified)` maps a non-STRICT verdict to the pipeline stage that
introduced the failure:
retrieval no admitted sources (gate over-rejected, or corpus
genuinely lacks the topic) → ingest more / relax breadth
context sources retrieved but no quotes extracted (per-source
cap dropped relevant content, or model declined to cite)
→ raise cap / tighten prompt
answer quotes extracted but didn't all verify (model fabricated,
paraphrased inside quotes, appended citations) → mechanical
+ re-prompt repair targets exactly this case
`failure_stage` lands on the run_dag's verify node payload AND on the
result dict so an operator can read the reason at a glance — `failure_stage='answer'`
means stop tuning the verifier & fix the model behavior. Debugging
becomes typed instead of vague.
#2 Re-prompt repair (second tier of the hybrid loop)
aborist/qa/repair.py: `reprompt_repair(...)` builds a feedback message
naming the failed quotes & asks the model to rewrite using only
verbatim citations. Hard rule: only fires when
`policy["repair_max_reprompts"] > 0` (default 0); caller enforces the
cap by looping at most that many times.
aborist/qa/query.py + aborist/qa/runner.py: after mechanical repair,
if the answer is still HYBRID/UNGROUNDED with unverified quotes,
loop up to `repair_max_reprompts` times. Each iteration: build
feedback (assistant turn with current answer + user turn with failed
spans), call LLM, verify. Accept the new answer if `n_verified`
strictly improved; otherwise break (the model's not converging,
don't waste cycles).
The mechanical + re-prompt combination handles the cases each tier
declines individually:
mechanical alone: synthetic_elision split, trailing_artifact trim,
no_overlap remove
+ re-prompt: paraphrase, partial_paraphrase, interior_elision
needing semantic judgment, fabrications the
model can recognize when shown its own quote
`repair_max_reprompts` lives in DEFAULT_QUERY_POLICY +
DEFAULT_POLICY so it folds into governance_policy_hash. Default 0
preserves single-shot semantics for callers that don't opt in. Each
re-prompt iteration adds a `{action: reprompt_rewrite, diagnosis:
model_feedback_loop}` entry to repair_changes; audit chain captures
the full transition through the existing providence_repair event.
Tests:
- dag: localize_failure across all four cases (STRICT, retrieval,
context, answer); failure_stage embedded in run_dag verify node.
- repair: stub client with sequenced answers (failing first, clean
on re-prompt) — assert two LLM calls, STRICT verdict, reprompt_rewrite
in the change log.
484 tests pass (dag +5, repair +1). The pre-existing test_burn flake
under full-suite ordering remains; passes in isolation.
The system was observational: verifier classified, sidecar diagnosed,
repair plans were emitted — but no loop ever closed. This adds the
hybrid repair stage from the toy-Hermes design pass: mechanical first
(deterministic string substitution from sidecar suggestions), behind
a `policy["repair_enabled"]` flag (off by default). Re-prompt fallback
is TODO.
aborist/qa/repair.py: `mechanical_repair(answer, unverified_quotes,
context)` walks each unverified quote through the sidecar classifier &
applies its `repair` action by string sub:
synthetic_elision_inside_quote (both halves verbatim)
`"prefix [...] suffix"` → `"prefix" ... "suffix"`
Two verbatim spans the verifier can independently check; the
model's [...] ellipsis-marker becomes prose between them.
trailing_artifact
`"prose. (Source: ...)"` → `"prose."`
Verbatim prefix kept; model-appended tail dropped.
no_overlap
Drop the line containing the bad quote entirely.
Skips include_aside_for_verbatim (needs precise source-span extraction;
defer to re-prompt path), paraphrase / partial_paraphrase (need prose
rewriting). Idempotent.
aborist/qa/query.py + aborist/qa/runner.py: optional pass after first
verify. When `repair_enabled=True` AND `audit_mode != "STRICT"` AND
unverified quotes exist:
1. Run mechanical_repair on the answer text.
2. If repair produced any changes, re-verify the repaired text.
3. If post-repair verdict isn't worse (n_verified didn't decrease),
accept the repair: persist the REPAIRED answer text instead of
the model's original. Cache_key inputs unchanged.
4. Audit chain gets one `providence_repair` event with the change
log + pre/post verdict so the original→repaired transition is
reconstructable.
Result dict gains `repair_changes` (list of change records) and
`pre_repair_audit_mode` (what the original was classified as).
`policy["repair_enabled"]` enters governance_policy_hash so on/off
agents share no cache silos.
Tests:
- mechanical_repair on each diagnosis (synthetic_elision, trailing_artifact,
no_overlap), idempotence on clean text.
- query() integration: repair_enabled=False (default) leaves answer
text unchanged; repair_enabled=True promotes a HYBRID/quote
synthetic_elision case to STRICT/quote, persists the repaired text,
emits the providence_repair audit event.
479 tests pass (+6 repair).
Each query/ask call now emits a 7-stage Merkle-DAG fingerprint stored
alongside the providence record. F from the toy-Hermes design pass.
Stages, in order:
question hash of question_hash (the 8-dim cache_key dim)
retrieval hash of sources summary (document_roots + roles +
scores + chunk_idx) — captures which docs ranked
context context_root (the source-Merkle for the assembly)
prompt conversation_hash
answer sha256(answer_text)
verify hash of verdict (audit_mode, verifier_method,
n_quotes, n_verified, claim_statuses)
final_label hash of (audit_mode, verifier_method, lookup_path)
run_dag_root = MerkleTree over those stage hashes (aborist conventions:
non-commutative HashCombine 0x03, leaf prefix 0x00, self-dup odd rule).
run_dag_blob = canonical JSON of {root, nodes} so an auditor can
recompute & verify (`verify_run_dag(blob)` returns True/False).
The DAG is NOT in cache_key. cache_key inputs determine the answer; the
answer determines the DAG — folding it back would create a cycle.
Instead it rides alongside as a per-record computation fingerprint.
Distinct from the linear `audit_events` chain (which tracks DB-wide
state changes); this is per-run computation provenance.
Schema: ALTER TABLE providence_cache ADD COLUMN run_dag_root TEXT;
ADD COLUMN run_dag_blob TEXT;
Idempotent migration in `_migrate_audit_mode`. Both rebuild templates
(VISUAL→UNGROUNDED dance, paraphrase verifier_method dance) updated to
include the new columns. Legacy records pre-2026-04-30 carry NULL.
Result dict gains `run_dag_root` so callers can verify without a DB
round-trip.
Tests:
- test_dag.py (9 tests): determinism, reactivity to each stage's input,
fixed stage order, round-trip verify, tamper-detection, JSON-string
acceptance.
- test_query.py: persistence on record + result, verify_run_dag round-
trip on the persisted blob.
473 tests pass (DAG +9, query +1, integration unchanged).
Two paired enhancements (D + E from the toy-Hermes design pass):
D. Per-claim status taxonomy on `verify_quotes`.
New `claim_statuses` field on every verdict — a per-evidence-unit
list with three labels:
VERIFIED_QUOTE unit substring-matched in normalized context
(any of quote/span/entity strategies)
SUPPORTED_PARAPHRASE unit cleared the paraphrase token-coverage
threshold (≥85% topical tokens present)
UNSUPPORTED unit didn't match anything
Diagnostic labels (QUOTE_INTEGRITY_FAILED, SOURCE_MISMATCH,
FALSIFIED) stay in the sidecar / falsification machinery — the
binary-verifier discipline holds. Empty list when no evidence
was extracted at all (verifier_method='none'). Backward-compatible:
existing fields (audit_mode, n_quotes, n_verified, unverified_quotes,
verifier_method) unchanged; current callers ignore the new field.
E. Repair-action plans on sidecar diagnoses.
Each `_classify_span` diagnosis now carries a `repair` field with
a concrete suggestion the operator can act on:
synthetic_elision_inside_quote → split_into_two_quotes
(when both halves verbatim)
→ trim_to_verified_half
(when only one half verbatim)
→ remove_claim
interior_elision → include_aside_for_verbatim
(with the dropped aside text)
trailing_artifact → trim_trailing_artifact
(with the kept_prefix string)
paraphrase → downgrade_to_paraphrase
partial_paraphrase → split_or_remove
no_overlap → remove_claim
Read-only suggestions — sidecar still doesn't write to providence_cache
or audit_events. The repair stage is recommendation, not mutation.
Operator (or an automated repair pass) decides whether to act.
Human render in `aborist inspect` shows `repair: <action> (<reason>)`
under each diagnosis line.
Tests:
- verify: claim_statuses_quote_path_labels_each_unit (per-quote VERIFIED
/ UNSUPPORTED), claim_statuses_paraphrase_method_flagged,
claim_statuses_empty_when_no_evidence.
- inspect: repair_synthetic_elision_split_when_both_halves_verbatim,
repair_interior_elision_includes_aside, repair_trailing_artifact_trim,
repair_no_overlap_remove.
462 tests pass (verify +3, inspect +4).
Two enhancements from the toy-Hermes design pass (2026-04-30):
A. Synthetic-elision-inside-quote diagnosis (sidecar only).
Distinct from interior_elision (model dropped a `(...)` aside source
carries) — synthetic_elision is the model writing literal `[...]`
between fragments of a `"..."` span, signaling self-elision while
claiming verbatim citation. The verifier still rejects (binary
discipline holds), but `aborist inspect` now reports
`diagnosis: synthetic_elision_inside_quote` with prefix/suffix
presence flags so an operator can judge whether the elided middle
was benign. Probe runs first in the classify-span chain (more
specific than trailing_artifact / interior_elision / paraphrase).
Catches the Brachiosaurus case: `"The film centers on the fictional
Isla Nublar [...] Universal Studios..."` — both halves are in source,
but the literal `[...]` isn't, so substring match correctly fails &
the sidecar tells the operator why.
C. Source-role classification + role-weighted context budget.
`_classify_source_role(title, qtokens_stem)` tags each top-K hit:
primary_answer_source 2.0× cap strong title-stem overlap
secondary_context_source 1.0× cap "list of", "characters",
"franchise", "history of"
noisy_background_source 0.5× cap "score", "music",
"video game", "merchandise"
sequel_background_source 0.5× cap "lost world", roman numerals
background_source 1.0× cap default
Order matters: noisy/sequel/secondary markers fire before the
primary check so peripheral pages with strong title overlap (e.g.
`Jurassic Park (film score)` shares 3 stems with the JP-film query)
don't claim a primary slot.
Cap loop now applies role weight on top of the baseline
`max_context_chars / top_k`. Total context still bounded by the
running `char_budget` — weights just shift how the budget gets
divided so primary pages get more text & noisy pages less, fixing
the case where a `(film score)` page consumed a primary slot.
`source_role` is persisted on `_Hit` and surfaces on
`merkle_proof.sources[*].source_role` in the providence record so
inspect & audits can see which slot each source occupied.
Tests:
- inspect: synthetic_elision_caught (Brachiosaurus regression),
synthetic_elision_does_not_fire_when_source_has_brackets (false-
positive guard).
- query: role classifier matrix (primary / secondary / noisy / sequel /
background) on JP-film-style titles, role persistence in sources list.
455 tests pass (sidecar +2, query +2).
The retrieval contract was undocumented — `_filter_by_title_relevance`
has three accept paths, each with its own scaling rules, and the recent
breadth + stem refinements (2026-04-29 catches) lived only in commit
messages. New "Retrieval pipeline" section between Live endpoints and
Hot path / gotchas covers the eight stages in order:
1. Parallel FTS5 (body-BM25 + title-LIKE), unioned per shard
2. Body-coverage sqrt rerank — counter BM25 short-doc bias
3. Title-token boost
4. Three accept paths in _filter_by_title_relevance:
title overlap (breadth-scaled: ≤2 require ALL, 3+ require N-1),
TF-IDF core keyword match, body density (same breadth scaling
plus min_mentions=3 depth)
5. Rivalry exclusion (Intel/AMD groups)
6. Stem-aware token matching: trailing-`s` strip on >4-char tokens
(skip `ss`-enders) — 'supermans girlfriend' now collapses onto
'superman'/'girlfriend' instead of admitting `Girlfriends` TV show
7. Per-source context cap = max_context_chars / top_k — prevents
80 KB+ bibliographies from monopolizing the budget
8. Wikitext base-prose normalization before LLM (cross-ref to
the Wikitext base prose convention)
Each item includes the commit ref or the catch date so an operator
can trace why the rule exists.
Per fox: while iterating on retrieval/verifier knobs, a flag on the
query itself is more useful than a separate verb. One step reset:
change a knob, re-query, see fresh result.
Surface:
aborist query --burn ...
make query Q="..." BURN=1
make query-dry Q="..." BURN=1 (works on dry-run too)
Behavior in `aborist.qa.query.query()`:
- New `burn_existing: bool = False` parameter on query().
- After computing primary cache_key (per the active dedup mode) but
BEFORE the cache lookup, if burn_existing: DELETE the live row that
matches the primary cache_key & write one providence_burn audit
event with reason "query --burn (test-ergonomic mid-query bust)".
- The lookup then misses → fresh inference runs. Result reports
`burned_existing: 0|1` so the caller sees whether anything got
busted.
- The equivalence-class fallback cache_key is deliberately NOT
touched: prior alt-mode records stay as historic witnesses.
Tests (3): cache populates → cache hits → BURN=1 forces fresh
inference; --burn writes a providence_burn audit event; first-time
query with --burn is a clean no-op (burned_existing=0). 356 passed,
1 skipped.
Complementary to `aborist burn-kindergarten` (mass reset) — this is
the surgical version. Both are local-only by design; for cross-peer
invalidation use `make falsify` and let mesh sync broadcast.
Per fox: useful while iterating on retrieval/verifier knobs to wipe
recent test runs without finding each cache_key. Mirrors the mesh-sync
kindergarten window so what's still un-broadcast (private to this peer)
is exactly what's safe to bust without confusing peers.
Surface:
aborist burn-kindergarten [--kindergarten-seconds N]
[--reason '...'] [--by-actor X]
[--force] [--dry-run] [--verbose N]
make burn-kindergarten [SECONDS=3600] [FORCE=1] [DRY_RUN=1] [REASON='why']
Behavior:
- Walks every shard, finds providence_cache rows with
created_at >= now - SECONDS and falsification_state='live'.
- Each row goes through the existing _burn_cache_key — children gate
honored unless --force, audit event written per successful burn,
chain integrity preserved.
- Result JSON reports examined / burned / refused_has_children /
not_found counts plus a verbose tail of per-row results.
- --dry-run reports without writing or auditing.
- --kindergarten-seconds 0 = burn every live row (test reset).
Tests (4): selective burn (old kept, fresh burned), dry-run writes
nothing, 0-second window burns everything, children-gate refusal
without --force. 353 passed, 1 skipped.
Operational note: this command does NOT propagate to peers (burn is
local kindergarten cleanup by design, see docs/mesh.md). If you want
the remote effect, falsify each record individually & let mesh sync
broadcast the falsifications instead.
Fox 2026-04-29: 'who is supermans girlfriend?' returned 7-of-8 unrelated
`Girlfriends`-titled articles (TV show, movies, songs); only 1 actual
Superman-related doc made it. Two coupled defects:
1. Title-overlap accept fired on ANY single-token match. A 2-token
query was admitting docs that shared only ONE qtoken. `Girlfriends`
passed because its title matched "girlfriend" even though no
"superman" anywhere in the doc.
2. `_body_density_passes` required at least HALF the qtokens — too
lenient for the 2-token case (1 of 2 = 50% = pass).
3. Possessive plural mismatch: question_hash strips apostrophes so
"superman's" → "supermans", but the corpus has bare "Superman".
`body.count("supermans")` returns 0 even on the canonical doc.
Three coordinated fixes in `aborist/qa/query.py`:
- New `_stem_token_for_match(t)` helper: strips a trailing `s` for
tokens > 4 chars (so `supermans` → `superman`, `girlfriends` →
`girlfriend`). Conservative: skips short tokens & double-s endings
to avoid `class` / `boss` / `pass` corruption.
- `_body_density_passes` now uses `_body_count_with_stem` (literal-
first, stem-fallback) AND a tightened breadth threshold:
≤ 2 tokens require ALL of them
3+ tokens require N - 1 (allow one weak signal token to miss)
- `_filter_by_title_relevance` mirrors the same breadth threshold for
title-overlap. Synonym fallback (any-match against synonym_expand)
is preserved ONLY for 1-token queries — otherwise a stray synonym
hit (e.g. AMD synonym matching an Intel-only title) over-recalls.
End-to-end on the actual corpus (3.4M articles, 8 shards):
before: 'supermans girlfriend' → 7 unrelated `Girlfriends` titles, 1 Superman doc
after: Lois_Lane in top-K (the canonical answer), other Superman-related
docs alongside, Girlfriends-only-titled docs filtered out
Tests:
- test_query_filter_requires_breadth_for_multi_token_queries — synthetic
3-doc corpus pinning the new behavior (Lois Lane keeps; girlfriends-tv
& superman-music both drop).
- test_query_filter_one_token_query_still_synonym_expands — pin the
1-token loose path is preserved (no over-tightening).
349 passed, 1 skipped.
Burned the two stale UNGROUNDED girlfriend-query cache records so a
fresh `make query` exercises the new filter end-to-end.
Different agents have different value functions on the same providence
records. Today aborist bakes one canonicalization policy into
governance_policy_hash and calls it universal. This adds two knobs that
let agents express their preference without breaking provenance.
- aborist/qa/keys.py: `canonical_question(q, mode=...)` and
`question_hash(q, mode=...)` accept "strict" (NFC + ws-collapse only;
every variant gets its own hash) or "equivalence_class" (default —
additionally lowercase + trailing-punct strip + article strip).
New constants: QUESTION_DEDUP_MODES, FIDELITY_MODES, defaults.
- aborist/qa/query.py + aborist/qa/runner.py: each call computes the
primary cache_key under policy["question_dedup"]. New `fidelity`
parameter:
"strict" only primary cache_key checked
"equivalence_class" primary first; if miss AND alternate mode
produces a different cache_key, try alternate
Cross-silo fallback works because _ckey_for_mode rewrites
policy["question_dedup"] to the alternate mode before computing
governance_policy_hash — so the fallback ckey matches what an agent
under that mode would have written. Lookups can find each other's
records when fidelity permits. Result dict gains `lookup_path` ∈
{"strict", "equivalence_class", "strict_fallback",
"equivalence_class_fallback", "miss"}.
- aborist/cli.py: `--question-dedup` and `--fidelity` flags on the
`query` subcommand. Human render annotates cache_hit lines that
came from a fallback ckey ("cached via equivalence_class_fallback").
- tests/test_query.py: three new regressions
* strict-policy + strict-fidelity: variants get distinct cache_keys
* cross-silo fallback: strict-policy reads eq-class-policy records
via fidelity=equivalence_class
* strict-fidelity refuses fallback (audit-grade): cache miss even
when the alternate silo has a hit
- CLAUDE.md: dedup-mode and fidelity convention bullets updated.
This is V1 + V2 of the substrate move sketched in conversation: write
policy determines which silo a record lives in; read fidelity determines
how loosely an agent walks across silos. Records stay exactly-keyed
(provenance hard); routing is per-agent (preference soft). Same
verifier_no_diagnostics discipline holds — no soft signal enters the
hard chain.
442 tests pass.
Fox 2026-04-29 (Clark Kent / Superman query): the verifier flagged a
quoted span as unverified and the sidecar diagnosed it as 'paraphrase'
with token_coverage=1.0. Investigation showed the source actually said
`Clark Joseph Kent (middle name is also Jerome according to some
versions) is a fictional character...` and the model elided the
parenthetical aside, quoting `Clark Joseph Kent is a fictional
character...`. Every word in source, but the sequence has a 60-char
gap where the aside was. Distinct failure mode from real paraphrase
(token reordering) and trailing_artifact (model APPENDS).
Sidecar refinement, no verifier change. The binary verifier discipline
(memory rule `feedback_verifier_no_diagnostics`) holds — quote strategy
is still verbatim-only; soft signal stays in the inspect verb.
Algorithm walks every `(` in base. For each open paren at position P:
the longest k where base[:P] ends with span[:k] is the model's prefix;
the span tail (≥20 chars) must then match the source after the close
paren. First paren that satisfies both checks wins.
CLI human-render shows `matched: N prefix + M suffix chars
(parenthetical aside dropped)` plus the dropped_aside text so an
operator can decide at a glance whether the elision is benign.
Live record now reads:
[1] interior_elision
matched: 17 prefix + 138 suffix chars (parenthetical aside dropped)
dropped_aside: middle name is also jerome according to some versions
439 tests pass (sidecar +2, full suite still green).
Fox 2026-04-29: "part of the gossip protocol should be some default
delay that allows a user to catch and burn kindergarteners before
they are synced." Currently `mesh sync` enumerates the most-recent
N items with no age filter — a doc ingested 30 seconds ago goes
out on the next sync, & if a peer ingests it before the operator
notices a problem, burn no longer suffices (peer has its own copy).
Adds a sender-side kindergarten window:
--kindergarten-seconds N default 3600 (1 hour)
Records younger than `now - N` are held back from broadcast. Both
ANNOUNCE_ROOT (filtered on documents.ingest_ts) & ANNOUNCE_FALSIFICATION
(filtered on falsifications.at). N=0 = broadcast everything (cron-
friendly opt-out for operators preferring immediate propagation).
Result JSON now reports the held counts so the operator can see
what the window protected:
kindergarten_seconds: 3600
announced_roots: 12
kindergarten_held_roots: 3
announced_falsifications: 0
kindergarten_held_falsifications: 1
Sender-side discipline only — the receiver has no view into when
the sender created the record, so it can't enforce. Adding a
created_at on the envelope would let receivers reject too-fresh
gossip, but that's a future protocol bump (envelopes today don't
carry sender wall-clock; ts is the send time).
Tests:
- existing tests now pass `--kindergarten-seconds 0` so freshly-
ingested fixtures broadcast immediately for the test
- new test_sync_kindergarten_holds_fresh_records: a 30-second-old
doc is held; an artificially-aged doc broadcasts. announced=1,
held>=1.
- new test_sync_kindergarten_zero_broadcasts_everything: explicit
opt-out works.
342 passed, 1 skipped.
Fox caught on 2026-04-29: 'who is batman?' and 'who is the batman?' produced
different cache_keys despite question_hash collapsing both variants. The 8-dim
cache_key has TWO seams that touch question text — question_hash AND
conversation_hash (which hashes the messages list with the literal question
in the user turn). Only the former canonicalized; the latter took bytes as
written, so each variant got its own chash and missed cache.
Fix: decouple the two forms.
- aborist/qa/keys.py: extract canonical_question(text) → str; question_hash
composes _sha256(canonical_question(t)) so the equivalence class is
defined in one place.
- aborist/qa/query.py + aborist/qa/runner.py: build TWO message lists per
call. `messages` carries the verbatim question (Hermes sees the user's
natural phrasing — no grammar drift). `canonical_messages` substitutes
the canonical form, and conversation_hash hashes that. The LLM still
gets 'Who Is THE Batman?'; the cache_key collapses to 'who is batman'.
- tests/test_query.py: regression — three variants (with/without `?`,
with/without `the`) hit the same cache_key. First call populates,
later variants cache_hit. Captures stub messages to confirm the LLM
saw the verbatim question, not the canonical form.
437 tests pass.
Per fox 2026-04-29: the wire protocol has had ANNOUNCE_FALSIFICATION
since the foundation commit (and MeshWireClient.announce_falsification
since AEAD landed), but the user-facing `mesh sync` only ever fired
ANNOUNCE_ROOT. Falsifications local to one peer never reached others
unless an operator hand-rolled a Python script.
Now `mesh sync` enumerates BOTH categories:
- ANNOUNCE_ROOT most-recent --limit documents (existing path)
- ANNOUNCE_FALSIFICATION most-recent --limit falsifications (new)
Receivers verify Ed25519 sig + per-peer chain-of-claims as before,
write one mesh_received audit event per accepted envelope. Result
JSON now reports both counts:
announced_roots: N (was: announced)
announced_falsifications: N
sent_roots: [...] (was: sent)
sent_falsifications: [...]
Two opt-out flags so operators can scope the broadcast:
--no-roots only push falsifications
--no-falsifications only push roots
Burns are deliberately NOT propagated. Burn semantics are local
kindergarten cleanup ("delete a leaf I shouldn't have written") —
other peers may have legitimately ingested the doc independently.
Falsify is the audit-preserving alternative whose broadcast IS the
right cross-peer signal for "this answer is wrong."
Caveat (deferred): no per-peer dedup state yet. Re-running sync
re-broadcasts the same most-recent N falsifications; receivers get
duplicate mesh_received audit-log entries (no state corruption,
just log noise). A `mesh_sync_state` table tracking
last_falsify_announced_ts per peer URL is the natural follow-up
when the falsification volume grows.
Tests:
- existing test_sync_announces_local_roots updated for new field
names (announced_roots, announced_falsifications)
- new test_sync_announces_falsifications: bob falsifies, syncs,
alice's chain has the ANNOUNCE_FALSIFICATION envelope
- new test_sync_no_falsifications_flag_skips_them: --no-falsifications
skips the broadcast cleanly
339 passed, 1 skipped.
- Top-level providence summary now mentions 4 strategies (quote, span,
entity, paraphrase) and that verifier_method records which fired.
- audit_mode bullet expanded with strategy 4 (paraphrase): token-coverage
≥0.85 with ≥4 content tokens after stopword filter, fires only on
prose-shaped spans (entity strategy still owns proper-noun lists).
Note that quote strategy gets NO paraphrase fallback — verbatim
citation is the contract.
- Trailing-citation strip documented: peels (Source: ...)/URL tails
before substring testing, refuses genuine prose parentheticals.
- Sidecar invariant now points at aborist inspect --cache-key X:
read-only, diagnoses each unverified span (verbatim_in_base /
trailing_artifact / paraphrase / partial_paraphrase / no_overlap),
never writes to providence_cache or audit_events.
- File tree shows aborist/qa/inspect.py; CLI line lists `inspect`.
Notes that `query` human-renders by default (--json or JSON=1).
- New question_hash convention: trailing-punctuation strip + standalone
English article strip so 'who is X', 'who is X?', 'who is the X' all
hash to the same equivalence class.
Whitepaper section 13.8 already current — fox shipped 13.8.5 (citation
strip) and 13.8.6 (sidecar) plus the 4-strategy 13.8.1 rewrite.
Fox 2026-04-29: querying "who is batman" returned only ONE source
(List_of_Batman_comics — an 80 KB+ bibliography) despite top_k=8 &
the actual bio article being in the corpus. Greedy fill: hit #1
consumed the entire 60 KB budget, every subsequent doc dropped with
char_budget <= 0.
Fix in aborist/qa/query.py:
per_source_cap = max(1, max_context_chars // max(1, top_k))
for h in hits[:top_k]:
text = _load_doc_text(...)
if len(text) > per_source_cap:
text = text[:per_source_cap] # NEW: per-source cap first
if len(text) > char_budget:
text = text[:char_budget]
...
Each top_k hit gets at most max_context_chars/top_k chars (default
60K/8 = 7.5K each — plenty for a chunk or two of prose). Total
context ≤ max_context_chars by construction. top_k=1 preserves the
legacy behavior (single source can use the full budget).
End-to-end effect on Batman: the bio (Wikipedia/Batman article) lands
in context alongside List_of_Batman_comics; the model can paraphrase-
verify against the actual character introduction text instead of
fabricating from training.
Tests: 2 regressions in tests/test_query.py — multi-source delivery
when hit #1 is huge, and top_k=1 single-source still allowed full
budget. 337 passed, 1 skipped.
Burned the two stale Batman cache records (chain extended) so a
fresh `make query Q="who is batman?"` exercises the new path.
Two related changes that tighten the dedup + grounding signals
without lowering quality bars.
1. question_hash drops standalone English articles (the/a/an).
Fox 2026-04-29: `who is the batman?` and `who is batman?`
produced different cache records; same question, different hash.
Articles are filler at the question-equivalence layer. Add a 4th
canonicalization step to question_hash: after lowercase + trailing
punctuation strip, split on whitespace & drop tokens equal to
"the" / "a" / "an", then rejoin.
Equivalence class now includes:
"who is X" ┐
"who is the X" │
"who is a X" │ -> same question_hash
"who is an X" │
"Who Is X？" ┘ (CJK question mark)
"thesis" stays untouched — exact-match standalone tokens only,
not substring. Conservative on i18n: ASCII English articles only;
"el / la / le / der / die / das" etc. await demand.
2. _token_coverage filters stopwords and per-token punctuation.
Fox asked: should we lower the 0.85 paraphrase threshold? Honest
answer: no — that would promote the Q1 Batman fabrication
("wealthy/businessman/resides" missing from corpus) to STRICT.
Tighten the signal instead so 0.85 means more.
- Per-token punctuation strip (.,;:!?\"()[]{}) so `wayne,` lines
up with bare `wayne` in context. Apostrophes deliberately stay
so `batman's` is distinct from `batman`.
- English stopword filter on length-≥4 fillers (from / with /
have / been / would / which / where / their / etc., curated set
in _ENGLISH_STOPWORDS). These match almost any English text &
inflate coverage scores when topical content is missing.
Net effect on Q1 Batman case: missing tokens are
`wealthy/businessman/resides` — all topical. Coverage stays well
below 0.85, span stays UNGROUNDED. Net effect on a stylistic
paraphrase (model wrote `from` instead of `with`): coverage
computed only over topical tokens, both copies match → 1.0 →
correctly promoted.
Tests: 8 new in tests/test_keys.py (article-strip equivalence
classes, substring preservation, distinct-topic non-collapse) +
2 in tests/test_verify.py (stopword filter doesn't inflate; 0.85
threshold still rejects fabrication). 335 passed, 1 skipped.
Note for fox: the underlying retrieval issue surfaced in Q1/Q2 is
independent — Batman main article IS in shard 002.db but FTS5 +
title rerank ranked List_of_Batman_comics higher. Different fix,
separate commit.
Per fox: README was stale on most of what shipped over the past
sessions. Sweep:
- New Quickstart section at the top (replaces "What this gets you")
with two end-to-end paths: Wikipedia 2003 dump and crawling a live
site (russell.ballestrini.net as the worked example). Each is
five-ish commands and ends with a real `make query`.
- New "Data: live websites (the crawler)" section between git/hg
ingest and the OpenAI-export placeholder. Documents:
- bootstrap-crawler / crawl-ingest with all knobs (URL, DEPTH,
MAX, FAST, CRAWL_SHARD)
- per-domain shard naming so make query auto-discovers the data
- feed/sitemap skip at ingest (discovery infrastructure, not
knowledge)
- ETag/Last-Modified capture + recrawl-check freshness probe
- provenance pointer to ~/git/agents.ai.unturf.com
- Asking-the-corpus section gains JSON=1, default human render note,
question_hash equivalence (trailing punctuation stripped), and the
full query path now reflects wikitext-strip-before-LLM step.
- New "Verifying answers (audit modes)" section explains the
trichotomy + four-strategy verifier (quote/span/entity/paraphrase),
trailing-citation strip, and the corpus-growth signal flow
(emergent → reclassify).
- New "Marking and burning records" section covers falsify vs burn
(kindergarten rule, KIND= for documents and cores), chain-check.
- Mesh section: dropped "on the roadmap" claim — wire layer +
serve/sync/pull verbs are shipped. Pointers to docs/mesh.md and
docs/mesh-deploy.md.
- Architecture diagram updated to current tree (wikitext.py,
sources/crawler/, qa/inspect.py, qa/verify.py, mesh/, vcs.py).
- Whitepaper pointer updated to canonical rst path; mesh docs added.
- Test counts: 326+ default suite, separate test-crawler target,
default suite never hits the network.
Two changes that make the verifier accept more honest grounding without
relaxing what STRICT means.
1. Strip trailing citation parentheticals at extraction time.
`extract_quotes` and `extract_claim_spans` now drop a single trailing
`(Source: ...)` / `(citing X)` / `(https://...)` parenthetical
before returning. The Pikachu real-corpus case: model wrote the
verbatim source sentence and appended `(Source: https://...)`. The
prose IS in the corpus — only the inserted citation broke substring
match. Now substring fires; STRICT recovered.
Conservative regex: only strips when the parenthetical content
starts with a citation cue word OR contains a URL. A genuine prose
parenthetical like "Pikachu (a Pokémon species) lives in forests"
stays.
2. Paraphrase strategy as a 4th verifier method.
In the span path, items that fail substring match get a
token-coverage probe. Tokens >=4 chars from the span; if >= 85% of
them appear in the normalized base context, count the span as
paraphrase-verified. Paraphrase items contribute to n_verified;
verifier_method flips to 'paraphrase' when any soft-verified items
are present so an auditor can tell.
Gated by `_is_prose_span`: a list of proper nouns ("Keanu Reeves,
Laurence Fishburne") has no lowercase content tokens >=4 chars and
falls through to the entity strategy where proximity policy can
tight-cluster check. Real prose spans ("Pikachu is a species of
Pokémon creatures...") have multiple lowercase content tokens and
qualify for paraphrase.
Soft-signal note (from CLAUDE.md soft/hard rule): paraphrase is
heuristic, not byte-equivalence. The hard chain still records the
classification — verifier_method='paraphrase' is the explicit marker
that an audit reader can use to distinguish lexical-verbatim from
paraphrase-overlap. Quote strategy stays verbatim-only — quotes ARE
quotes; paraphrase-in-quotes is the model's mistake to flag, not
auto-promote.
Schema: verifier_method CHECK constraint expanded to include
'paraphrase'. New idempotent migration `_rebuild_providence_cache_paraphrase`
mirrors `_rebuild_providence_cache_ungrounded` (table rebuild via
BEGIN IMMEDIATE; no value translation needed). Probes existing CHECK
from sqlite_master and no-ops if already expanded.
Tests: 9 new in tests/test_verify.py — 4 trailing-citation strip
(quote+span variants, URL-only parenthetical, end-to-end STRICT
recovery), 4 paraphrase strategy (high-coverage promote, low-coverage
reject, mixed-method label, quote-strategy preservation). Default
suite still green: 326 passed, 1 skipped.
Burned the live Pikachu cache record (audit chain extended) so a
fresh `make query Q="who is pikachu?"` exercises the new verifier
end-to-end.
Per fox: the verifier reports n/N verified but doesn't tell you WHY a
span didn't ground. Some are paraphrase, some are model-added
trailing artifacts (e.g. `(Source: https://...)` citations the model
appended to verbatim prose), some are full inventions. Operator
needs to see which is which to triage.
Read-only sidecar — no audit events, no providence_cache mutations,
no v9.8 field changes. Per the verifier-no-diagnostics memory rule:
hard chain stays binary; soft signals live in sidecar verbs that
never feed back.
Five diagnosis labels:
verbatim_in_base span IS in base context — verifier or
canonicalization bug (worth flagging loudly).
verbatim_in_raw_only raw wikitext match but base form differs —
wikitext-strip edge case.
trailing_artifact a >=60-char prefix matches; the tail (often a
model-added citation) doesn't. Surfaces the
tail explicitly.
paraphrase >=85% query tokens (>4 chars) present in
base context but not in this sequence.
Model rewrote source content.
partial_paraphrase 40-85% coverage — mixed sourced/emergent.
no_overlap <40% coverage — likely full invention.
Verified live against the real Pikachu cache record (5 sources,
71KB raw -> 34KB base): two spans flagged paraphrase (token_coverage
1.0 for both, just rewritten sequence), one flagged trailing_artifact
(100-char prefix matches; tail is `(Source: https://...)` citation).
That's the actual reason HYBRID — model paraphrased + appended
citations that aren't in the corpus.
Surface:
aborist inspect --cache-key <hex> [--qa-db ...] [--json]
make inspect KEY=<hex> [JSON=1]
Tests: 9 (6 classifier on synthetic contexts pinning each label,
3 end-to-end with seeded fixture record + verification that inspect
writes nothing). 318 passed, 1 skipped overall.
Default `make query Q="..."` now uses the human render. JSON=1 flips
to the raw record (same as `aborist query --json` directly):
make query Q="who is pikachu?" # human render
make query Q="who is pikachu?" JSON=1 # raw JSON record
Same flag works on query-dry. Help line updated to advertise it.
Two things fox surfaced from a "who is pikachu?" run:
1. Output emitted "Pok\\u00e9mon" instead of "Pokémon" — json.dumps
defaulted to ensure_ascii=True. Switched to ensure_ascii=False on
every user-facing dump in cli.py (41 sites). The one canonical-JSON
call (`separators=(",", ":")` for storing in providence_cache as
a JSON column, not for hashing) was deliberately left alone.
2. The actual answer was buried under cache_key / context_root / per-
source metadata / timings JSON. Default now renders human-readable:
who is pikachu?
HYBRID 1/2 verified via quote 9.2s (fresh)
Pikachu is a species of Pokémon creatures...
sources (3):
[1] Pikachu — en.wikipedia.org/wiki/Pikachu (002.db)
[2] List_of_Pokémon — en.wikipedia.org/wiki/List_of_Pokémon
...
unverified (1):
- "spans the model couldn't ground..."
cache_key: 35ab7d33… <run with --json for full record>
Pass `--json` to get the prior raw record (still ensure_ascii=False
so unicode renders cleanly there too — scripts parsing the output
see real chars; the JSON spec accepts either form).
Implementation:
- `_render_query_human(result, question)` — pure function, easy to
unit-test, no I/O. Truncates unverified spans over 100 chars,
omits empty sections, distinguishes cached vs fresh in the summary.
- `_strip_scheme` / `_short_path` — small helpers for source display.
- `--json` flag on the `query` subparser; default is human render.
Tests: 12 new in tests/test_cli_render.py — question position, summary
fields (audit/n_verified/method/elapsed/cache-status), unicode
literals (Pokémon not \\u00e9), source line shape, long-quote
truncation, empty-section omission, error fallback, short cache_key
with --json hint. 309 passed, 1 skipped overall.
Per fox: ASCII-only stripping leaves CJK full-width forms behind, so
"who is X？" and "who is X" still produce different cache_keys
despite being the same question. Expanded strip set:
ASCII: . ? ! , ; :
CJK: ？(U+FF1F) ！(U+FF01) 。(U+3002) 、(U+3001)
ellipsis: …(U+2026)
Pairs deliberately stay out of the set:
" ' ) ] }
Stripping one side of a pair breaks balance. `who said "X"?` after
stripping `?` is balanced; further stripping `"` would yield
`who said "X` — different equivalence class than the original. And
apostrophes carry meaning: `X's` is a different question from `X`.
Lifted the strip set into a module constant `_QUESTION_TRAILING_STRIP`
so anyone considering an addition has a documented anchor.
Tests: 11 new in tests/test_keys.py — 6 CJK / ellipsis equivalences,
5 pair-preservation cases (double-quote, single-quote, paren, bracket,
brace). 24 keys tests + 297 default suite, all passing.
Whitepaper sibling change in ~/git/unfirehose-nextjs-logger/whitepaper/
merkle-providence-reverse-rag-whitepaper.rst — expanded the
question_hash bullet to document the equivalence class. Not committed
in this commit (different repo).
Today `question_hash` canonicalizes (NFC, ws-collapse, lowercase) but
preserves trailing punctuation. So "who is X?" and "who is X" produce
different hashes, different cache_keys, different providence_cache
records — the cache misses on what's semantically the same question.
Add `rstrip(".?!,;:")` as the third canonicalization step inside
question_hash specifically. Local to question hashing — chunk leaf
hashes go through `canonicalize()` directly and are unaffected.
Equivalence class after this:
"who is X" \
"who is X?" |
"who is X." | -> all same question_hash, all same cache_key
"who is X!" |
"Who Is X" /
Internal punctuation is preserved on purpose: "X, then Y" carries
meaning that "X then Y" doesn't, even though both have the same
content tokens.
One-time impact: prior cached records whose canonical question ended
in punctuation become orphans on lookup (re-derive on next ask).
History stays on disk; nothing burned automatically.
New test file tests/test_keys.py with 13 tests pinning the question_hash
equivalence class plus light coverage of the other dim hashes
(model_profile, conversation, governance, cache_key 8-dim invariant).
286 passed, 1 skipped.
Per fox: a query for "who is Russell Ballestrini" classified STRICT by
grounding 6/6 quotes against `feeds/all.atom.xml` — a 230-chunk dump
of post metadata. The verifier was technically correct (every quoted
string IS in the feed) but the result was hollow: feeds list URLs,
they don't carry knowledge. The ACTUAL bios live at the linked posts.
Two-pass filter in _CrawledHtmlSource.iter_documents:
1. Pre-fetch URL-pattern check (_looks_like_feed_url):
- suffix matches: .atom, .atom.xml, .rss, .rss.xml, .rdf,
/feed.xml, /atom.xml, /rss.xml, /rss2.xml, wp-rss2.xml,
wp-atom.xml, wp-rdf.xml, wp-rss.xml
- substring matches in path: /feed/, /feeds/, /atom, /rss, /sitemap
2. Post-fetch Content-Type check (_looks_like_feed_response):
- rejects: application/atom+xml, application/rss+xml,
application/rdf+xml, application/xml, text/xml
- keeps: application/xhtml+xml (xhtml IS html, just stricter syntax)
Defense in depth — a feed served at a non-feed path (e.g.
/index.html returning text/xml) still gets dropped on Content-Type.
What stays out of the corpus:
/feeds/all.atom.xml (Atom feeds)
/sitemap.xml (XML sitemaps)
/wp-rss2.xml (WordPress RSS)
/feed/ (any feed alias)
What's still allowed:
/post-name/ (real prose pages)
/index.html (HTML)
/page.xhtml (xhtml)
Tests: 12 new parametric cases (8 path patterns + 6 Content-Types +
xhtml positive case). 36 bridge tests + 273 default suite, all
passing.
For fox: the existing feed entry already in the crawl shard was burned
manually via `aborist burn --kind document --root eaf7c8d5...` — 230
chunks gone. Re-running `make crawl-ingest` won't re-introduce it.
Defect found by fox: `make crawl-ingest URL=...` wrote to
~/.aborist/aborist.db (the legacy single-DB path) but `make query Q=...`
searches ~/.aborist/shards/ — two different corpora, query never saw
the crawled content.
Fix mirrors the existing shard-aware ingest pattern (ingest-self,
ingest-git, ingest-grok-attached): derive a shard filename from the
seed URL's hostname and write into $(SHARDS_DIR). Override with
CRAWL_SHARD=path for custom layout.
https://russell.ballestrini.net
-> $(SHARDS_DIR)/crawl_russell_ballestrini_net.db
After this, `make query Q=...` (cross-shard search) automatically
picks up newly-crawled domains. Each domain stays in its own shard
file so falsifications, evictions, and deletions are scoped naturally.
recrawl-check gets the same treatment: defaults to walking every shard
under SHARDS_DIR (skipping qa.db and snapshots.db); CRAWL_SHARD=path
scopes to one when you want.
Note for fox: existing pages already at ~/.aborist/aborist.db from
the prior crawl are still there (harmless). To get them under query,
either re-crawl into the new path or migrate via sqlite ATTACH +
INSERT — a follow-up if you want it.
The lifted AsyncWebFetcher already had a fast_mode constructor knob;
just unbroken plumbing was missing. fast_mode trades politeness for
throughput:
- timeouts: 5s vs 15-60s
- parallel page workers: cpu_count() * 3 vs 1
- ignores robots.txt 'crawl-delay' directive
- robots.txt 'Disallow' is STILL honored (only the delay drops)
Surface:
aborist crawl --seed-url ... --fast
make crawl-ingest URL=... FAST=1
Tests: two regressions pin that fast=True / fast=False each translate
to the right AsyncWebFetcher(fast_mode=...) construction. Capturing
fetcher fixture to avoid network. 21 bridge tests + 273 default
suite, all passing.
A real BFS hits sitemap.xml / RSS feeds via the verbatim crawler's
generic page handler. bs4 warns "you're using an HTML parser on XML"
on each — benign (parsing still works) but spams stderr during a
deep crawl, drowning the progress heartbeat we just wired in.
Filter at aborist/sources/crawler/__init__.py so the lifted source
stays untouched. The filter applies whenever any caller imports from
this subpackage, including the CLI's `from ... bridge import ...`
(which runs __init__.py first per Python import semantics).
A no-cap crawl could take many seconds with no feedback. Wires the
existing aborist.progress.Progress into both phases so stderr shows
heartbeats every 2s (Progress's default interval).
Discovery phase: prints prefix='crawl', counts discovered URLs,
shows queue depth as the secondary number ('inserted' slot in the
Progress format — works fine, semantically "still to do").
Ingest phase: prefix='ingest', total_estimate=len(urls) so the user
sees percent + ETA. Threads through ingest_source's existing
progress= parameter.
Plus three banner lines to stderr at phase boundaries (start crawl,
end discovery, start ingest) so even sub-2s crawls show signs of
life. All flushed via Progress's flush=True path; stderr is
line-buffered by default so this works without PYTHONUNBUFFERED.
Tests stay green (Progress goes to stderr, pytest captures only the
stdout summary). 19 bridge tests + 273 default suite, all passing.
Per fox: 20 was a wrong default — typical sites have many more pages
within depth 2 than that, and the cap was silently truncating real
crawls. New behavior:
aborist crawl --depth 2 # no page cap
aborist crawl --depth 2 --max-pages 50 # opt-in cap when needed
make crawl-ingest URL=... DEPTH=2 # no page cap
make crawl-ingest URL=... DEPTH=2 MAX=50 # explicit cap
Implementation: bridge's BFS loop treats max_pages=0 as "unbounded"
(while-condition becomes `queue and (max_pages == 0 or len(...) < max_pages)`).
Default flows from CLI argparse default=0 down to the bridge.
Tests: pinned max_pages=0 → no cap with a 30-page chain regression
test (19 passed in tests/crawler/test_bridge.py). Default suite still
273 passed, 1 skipped.
The verbatim AsyncWebFetcher.fetch_webpage defaults extract_links=False
and returns (html, []) when omitted. Bridge's BFS therefore terminated
at the seed — a real-world crawl of russell.ballestrini.net discovered
exactly 1 URL (the seed). One-line fix; the kwarg was always there in
the lifted API.
Added test_crawl_seed_passes_extract_links_true that records every
fetch_webpage call and asserts the kwarg is True. Pin so this can't
silently regress when bridge internals get reorganized.
Updated the stub fetcher signature in the existing tests to accept
the new kwarg without behavior change.
Glues the verbatim crawler lift to aborist's existing ingest pipeline,
plus adds a freshness probe that asks "does this need recrawling?"
without paying for a body fetch when the answer is no.
Bridge (aborist/sources/crawler/bridge.py):
- crawl_seed(seed, depth, max_pages): BFS-discovers same-domain URLs
via AsyncWebFetcher. Exact-hostname filter so subdomain crossover is
off by default. Caps at max_pages. URL dedup, fragment-strip
canonicalization. Sync wrapper around an async core so the CLI
remains a one-liner.
- ingest_crawled(conn, urls): runs the URL list through the standard
ingest_source() path so chunking_version / canonicalization_version
/ Merkle commitment all stay consistent. Captures ETag +
Last-Modified per response and writes one row per document_root
into the new document_http_meta table.
Schema (idempotent migration, mirrors _migrate_mesh_peer_chains):
document_http_meta (
document_root TEXT PRIMARY KEY,
etag TEXT,
last_modified TEXT,
last_fetched_at INTEGER NOT NULL,
last_status INTEGER,
last_checked_at INTEGER,
FK(document_root) -> documents(document_root) ON DELETE CASCADE
)
Recrawl-check (recrawl_check):
- Sends conditional HEAD with If-None-Match + If-Modified-Since.
304 -> fresh (no body transfer needed). 200 -> stale (server says
the body changed). 404/410 -> gone. Other -> unreachable.
- Updates last_status + last_checked_at so consecutive runs visit
oldest-first (NULLS FIRST in ORDER BY).
- Optional --domain filter so operators can probe one site at a time.
CLI:
aborist crawl --seed-url URL [--depth 2] [--max-pages 20] [--ingest]
aborist crawler recrawl-check [--domain DOMAIN] [--limit 100]
Makefile:
crawl-ingest URL=https://example.com [DEPTH=2 MAX=20]
recrawl-check [DOMAIN=example.com LIMIT=100]
Both targets gate on bootstrap-crawler so [crawler] extras land first.
Tests: 17 new in tests/crawler/test_bridge.py — same-domain filter,
BFS bounds (depth + max_pages), dedup, fragment normalization, ETag
capture, content-type filter, HEAD classification (304/200/404),
domain filter, last_checked_at update. All stub the network via a
fake httpx.Client; default suite still 273 passed, 1 skipped.
End-to-end test path:
make bootstrap-crawler
make crawl-ingest URL=https://russell.ballestrini.net DEPTH=2 MAX=20
make query Q="who is russell ballestrini?"
make recrawl-check DOMAIN=russell.ballestrini.net
Lifts the async web fetcher into aborist as an opt-in source. The
implementation comes directly from ~/git/agents.ai.unturf.com/core
(rev 2026-04-28); aborist's adaptations are minimal and documented in
aborist/sources/crawler/__init__.py:
core/async_web_fetcher.py -> aborist/sources/crawler/async_web_fetcher.py
core/web_fetch.py -> aborist/sources/crawler/web_fetch.py
Two source-side changes during the lift:
1. Heavy deps (aiohttp, bs4, miniuri) wrapped in try/except so a bare
`import aborist.sources.crawler` raises ImportError with the install
hint instead of leaking AttributeErrors deep in user code.
2. Chat-bot fetch triggers (`has_fresh_fetch_trigger`,
`has_web_fetch_trigger` from agents.ai.unturf.com/core/keywords)
replaced with NotImplementedError stubs. Aborist has no chat
surface — fetch intent is detected at the application layer. The
two test classes that exercised these triggers are
`@pytest.mark.skip`'d with the same rationale.
Not lifted: web_cache_manager.py — it backs page caching with
SQLAlchemy. Aborist has its own content-addressed cache via
providence_cache; no need to carry SQLAlchemy as a dep just for
crawled-page memoization.
Off by default:
- `[crawler]` extras section in pyproject.toml carries the heavy
deps. `[dev]` pulls them in so the crawler tests can run.
- `make test` ignores tests/crawler/ entirely.
- `make bootstrap-crawler` installs the extras into the venv.
- `make test-crawler` runs only the lifted tests after extras land.
Tests: 74 passed, 9 skipped (the chat-bot trigger tests deliberately
dropped). Default `make test` stays at 273 passed, 1 skipped.
Receiver now rejects gossip envelopes whose prev_event_hash doesn't
extend the sender's last-seen event_hash — fork detection lives at the
wire layer, not just in docs/mesh.md.
Schema: new mesh_peer_chains table (peer_member_id PK, last_event_hash,
last_seq, last_seen_at) with idempotent _migrate_mesh_peer_chains
matching the _migrate_audit_mode pattern.
Envelope: WireEnvelope gains prev_event_hash + event_hash, both
optional for back-compat. canonical_bytes() excludes event_hash to
avoid the circular dependency (the hash is computed FROM the canonical
bytes). New WireEnvelope.with_chain(prev_event_hash=...) builds a
linked envelope with event_hash = sha256(prev || canonical_bytes).
Sender (MeshWireClient._signed_envelope, MeshWireServer.handle_request
DELIVER_BODY): reads our latest_event_hash, builds the linked
envelope, then appends a 'mesh_sent' audit event whose body is the
envelope's canonical (event-hash-excluded) dict. By construction the
audit event_hash equals the envelope event_hash — wire chain-of-claims
and local audit chain stay in lockstep, so back-to-back sends advance
the chain naturally.
Receiver (handle_announce): when an envelope carries chain fields,
recompute event_hash and 400 on mismatch; look up
mesh_peer_chains[sender] and 409 on prev mismatch; on accept update
the row + write the existing mesh_received event.
Backward compat: legacy envelopes (event_hash=None) bypass chain
enforcement — kept the existing test_mesh_wire.py fixtures verbatim
since they construct WireEnvelope directly without with_chain(). All
production traffic goes through MeshWireClient and is always tracked.
v1 deferred (documented in module docstring): no multi-event catchup;
operator retries on 409. No cross-peer reconciliation.
Tests: 12 new in tests/test_mesh_chain.py covering the 7 required
cases plus canonical-bytes exclusion, migration idempotence, and
legacy-envelope fallback. 246 passed, 1 skipped overall.
Adds an opt-in confidentiality layer to gossip. Senders pass
encrypt=True on announce_root / announce_falsification; bodies travel
AEAD-encrypted (ChaCha20-Poly1305) under the per-epoch symmetric secret
recovered via the existing X25519 wrap. Cleartext stays the default —
all existing wire tests pass without modification.
WireEnvelope grows an optional encrypted_body field. canonical_bytes
omits the field when None (byte-identical to v1 cleartext format) and
includes it when set, so the Ed25519 signature binds the ciphertext —
a MITM cannot swap payloads. AAD is type|sender|epoch|ts so a captured
(nonce, ct) cannot be replayed under different envelope metadata.
state.recover_epoch_secret resolves the rotator from the audit chain
and re-unwraps this peer's slot. Sender + receiver share one path; no
new sender-side cache is required.
Closes the request half of the gossip loop. MeshWireClient already
verifies signature + Merkle root on delivered bodies; this verb adds
the CLI surface, gating, local re-ingest through the standard chunker,
and a 'mesh_pulled' audit event. Re-ingest is rejected if the local
document_root differs from the requested root.
burn --kind {providence,document,core}; default 'providence' preserves
the prior --cache-key surface (no breaking change).
Children gates for document/core leaves:
derivations.src_root downstream cores derived from this doc
edges.dst_root other docs link to this doc
providence_cache.source_root Q&A grounded in this doc
derivations has ON DELETE CASCADE on BOTH core_root and src_root, so a
naive DELETE FROM documents would silently cascade-prune downstream
derivations and orphan deeper cores. Gate before delete; --force to
override with the count recorded in the audit body.
Burn manually clears chunks_fts (FTS5 has no FK) and outbound edges
(no FK either) before deleting the document. chunks + merkle_nodes +
inbound derivations cascade via FK.
Audit event types are document_burn / core_burn — distinct from
providence_burn so chain consumers can tell leaf kinds apart.
CLAUDE.md "Cores never evict" remains intact: burn is operator-driven
removal, evict is the automated cold-tier compression that touches
only kind='surface'.
Makefile: KIND=providence|document|core selects the leaf type;
KEY for providence, ROOT for document/core. Single 'burn' target keeps
the surface DRY.
Tests (tests/test_burn_doc.py, 13 new):
- happy path: chunks + merkle_nodes vanish via cascade
- refused on derivation downstream / providence ref / incoming edge
- --force overrides and records counts
- core burn happy path (no children) preserves parent surface
- core burn refused with downstream derivation
- audit chain integrity preserved (no dangling prev_event_hash)
- CLI invocations for document, core, default-providence, refused
Full suite: 247 passed, 1 skipped.
- Wikitext base prose section now reflects reality: to_base() runs in
two places (before LLM call in query.py/runner.py AND inside
verify_quotes) gated by policy["base_version"]. Both sides see prose;
Hermes can quote source paragraphs verbatim instead of escaping
[[wikilinks]], and Wikipedia chunks ship with ~43% fewer tokens.
- Build/test block now points at the real targets: verify-shards,
analyze-shards, chain-check-shards. The bare 'make analyze' target
doesn't exist — only the cross-shard variants do.
- New "Hygiene after any state-changing op" callout: chain-check-shards
for one-second per-shard break count, analyze-shards for full audit.
- Audit-chain invariant block points at the same targets.
- Orientation protocol updated: --shards-dir replaces --db (the latter
refers to a single-DB setup that's no longer the canonical path), and
chain-check-shards is added to the per-session ritual.
Step-by-step operator guide for standing up an aborist mesh between
two real hosts: pre-flight, init/enable on both peers, out-of-band
pubkey exchange, mutual enrollment, mesh serve, mesh sync, audit
chain integrity checks, eviction protocol, reset/teardown.
Calls out what's not yet wired (mesh pull, per-peer chain merge,
AEAD body encryption) so operators don't expect features that ship
later. Pairs with docs/mesh.md (protocol contract) and the e2e tests
in tests/test_mesh_wire_e2e.py.
Both ask() (single-doc) and query() (multi-source) now run the
assembled context through aborist.wikitext.to_base() before message
construction. Hermes sees clean prose instead of raw wikitext, which:
- lets the model quote verbatim (raw [[Cloud Strife]] was unquotable
in fluent prose; the model paraphrased to bare "Cloud Strife" and
the verifier flagged that as UNGROUNDED)
- shrinks token bills ~43% on Wikipedia chunks (measured: 125,152 →
71,771 chars on the FF7 main-character query)
- aligns LLM-input with what the verifier already strips, so the
prose-vs-prose substring check is end-to-end consistent
Gated on policy["base_version"] (= wikitext-base-v1) so:
- governance_policy_hash flows the choice — prior answers cached
under raw-wikitext policy retain distinct cache_keys, no migration
- operators can opt out (set base_version=None) for debugging
- environments without mwparserfromhell soft-fall to raw wikitext
with no other behavior change
Tests (4): default policy carries base_version (tripwire), ask() user
messages have wikitext markers stripped, opt-out reverts to raw
wikitext, governance_hash differs with vs without base_version.
Wires the gossip-wire core (commit f141bab) into the user-facing CLI:
aborist mesh serve --host 127.0.0.1 --port 8400
Spawns MeshWireServer, blocks until SIGINT. Refuses if mesh isn't
initialized or .enabled is off — same gate every other mesh verb
enforces. Stdout is unbuffered JSON status lines.
aborist mesh sync --peer http://other.example:8400 [--limit N]
Enumerates local documents (most-recent N, default 100), fires one
ANNOUNCE_ROOT per document via MeshWireClient, reports counts of
acknowledged vs. errored. Pulling missing roots back from the peer
is a v2 addition — this verb pushes only.
Tests (6): gating refusals on uninitialized + disabled mesh; sync
round-trip with two peers proving Alice's audit chain gains one
'mesh_received' per Bob announce; unreachable-peer reporting; serve
smoke-test via start_in_thread + GET /mesh/info.
Burn deletes a providence_cache row outright; falsify (the existing
verb) flips state and keeps the row. Burn is for early/scratch use
during a tree's genesis when downstream consumers haven't ingested
the answer yet.
Children gate: refuses if any falsifications reference the cache_key.
Caller can override with --force; the audit event records forced=True
plus the child count at burn time so the chain can reconstruct what
was orphaned. Refusal is non-destructive AND skips the audit event so
scripts can retry without polluting history.
Always writes a 'providence_burn' audit event regardless of force.
Body fields: cache_key, burned_audit_mode, burned_n_verified,
burned_state, question_text, reason, by_actor, child_falsifications_at_burn,
forced. The chain-break query (`make chain-check`) shows 0 after burn —
the new event extends the prior chain cleanly.
Surface:
aborist burn --cache-key <hex> [--reason '...'] [--force]
make burn KEY=<hex> REASON='...' [FORCE=1]
Tests (7): happy-path delete + audit event, refusal on children,
--force override, unknown-key not-found, chain-integrity preserved,
CLI happy path, CLI non-zero exit on refusal.
- aborist/qa/verify.py: three strategies tried in sequence. quote uses
sequential pairing (1st & 2nd `"`, 3rd & 4th, ...) which eliminates
the phantom inter-pair captures regex pairing produced on adjacent
quote pairs like `"title" prose "quote"`. span checks bullet/sentence
lines verbatim. entity matches multi-word proper nouns; gated by
policy ∈ {strict, hybrid, drop, proximity}. Default proximity
promotes to STRICT only when N=3 verified entities cluster within
W=300 chars in source — separates structural grounding (cast list,
infobox) from incidental mention (scattered plot summary).
- aborist/store.py: schema CHECK now `('STRICT','HYBRID','UNGROUNDED')`.
Migration helper rebuilds legacy `('STRICT','HYBRID','VISUAL')`
tables via temp-table copy, translating VISUAL → UNGROUNDED in the
SELECT. Idempotent — DDL inspection skips the rebuild on already-
migrated DBs.
- aborist/cli.py: new `aborist reclassify` re-runs the verifier
against existing live records under the current entity policy; no
LLM calls. --compare runs all four policies side-by-side, --dry-run
reports transitions without writing. Cold-source records skipped.
One providence_reclassify audit event per changed row.
- AuditMode.UNGROUNDED replaces VISUAL across search backend, FTS5,
test fixtures, CLAUDE.md. The substrate (Merkle-AGI) name was
about FOR-style visualization; in the RAG layer the semantic is
"no recoverable grounding," so the label now says that.
DEFAULT_QUERY_POLICY gains entity_policy + entity_proximity_n +
entity_proximity_window so any tuning folds into governance_policy_hash
and invalidates cache cleanly.
