Bench-max sprint 1a + sprint 3 + speed audit. Five wins, none of
them traded calibration.
UTF-16 surrogate fix (sprint 1a)
================================
Hermes occasionally emits text with lone UTF-16 surrogates. Bare
.encode('utf-8') raises UnicodeEncodeError on those, which aborted
the run with no Merkle root. Two errors per lattice mode in the
2026-05-02 bench were this exact path on the 'tell me about the
roman empire' question.
Fix: errors='surrogatepass' on the four sha256 helpers that hash
model-derived text, plus the two audit-chain encode sites in
store.py for defense-in-depth (audit body could carry user text in
some flows). The hash stays deterministic because WTF-8 bytes are
reversible & unique per input.
Touched:
aborist/qa/dag.py:_sha256_hex (the loud one)
aborist/qa/keys.py:_sha256
aborist/qa/evidence.py:_sha256_hex
aborist/store.py: chain_audit_events + append_audit
Predicted Δ on next bench: +1pp on lattice modes (the 2 errors
become valid runs).
Smoke fixture (sprint 3)
========================
bench/qa_questions_smoke.txt — 5 questions, all anchor classes,
each currently failing pointer mode 100% while JSON aces 100% per
the 2026-05-02 bench. Wired as 'make bench-qa-smoke', --n 1
--concurrency 4, ~30-90s wall-clock depending on vLLM warmth. The
inner loop for prompt iteration; the full 71-question sweep stays
the scoreboard.
Smoke verified: pointer=0/5 STRICT, JSON=5/5, quote=2/5. Confirms
the gap pattern from the journal.
Concurrency default
===================
Makefile bench-qa now defaults to BENCH_QA_CONCURRENCY=4 (was
sequential). Override via BENCH_QA_CONCURRENCY=N. Combined with
the --concurrency landing in 0870af6, full sweep drops from ~107
min projected to ~51 min actual.
pytest-xdist (test-speed)
=========================
Added pytest-xdist>=3.5 to dev extras. 'make test' now uses
-n auto (= one worker per logical CPU). Measured: 36s → 10s on
the 641-test suite. 3.6× speedup, no test changes required.
Bench-max scoreboard (predicted lift from this commit alone):
+1pp lattice modes (UTF fix)
+cycle-time enabler (smoke fixture, xdist)
no calibration cost — none of the verifier checks moved.
Lifts the async web fetcher into aborist as an opt-in source. The
implementation comes directly from ~/git/agents.ai.unturf.com/core
(rev 2026-04-28); aborist's adaptations are minimal and documented in
aborist/sources/crawler/__init__.py:
core/async_web_fetcher.py -> aborist/sources/crawler/async_web_fetcher.py
core/web_fetch.py -> aborist/sources/crawler/web_fetch.py
Two source-side changes during the lift:
1. Heavy deps (aiohttp, bs4, miniuri) wrapped in try/except so a bare
`import aborist.sources.crawler` raises ImportError with the install
hint instead of leaking AttributeErrors deep in user code.
2. Chat-bot fetch triggers (`has_fresh_fetch_trigger`,
`has_web_fetch_trigger` from agents.ai.unturf.com/core/keywords)
replaced with NotImplementedError stubs. Aborist has no chat
surface — fetch intent is detected at the application layer. The
two test classes that exercised these triggers are
`@pytest.mark.skip`'d with the same rationale.
Not lifted: web_cache_manager.py — it backs page caching with
SQLAlchemy. Aborist has its own content-addressed cache via
providence_cache; no need to carry SQLAlchemy as a dep just for
crawled-page memoization.
Off by default:
- `[crawler]` extras section in pyproject.toml carries the heavy
deps. `[dev]` pulls them in so the crawler tests can run.
- `make test` ignores tests/crawler/ entirely.
- `make bootstrap-crawler` installs the extras into the venv.
- `make test-crawler` runs only the lifted tests after extras land.
Tests: 74 passed, 9 skipped (the chat-bot trigger tests deliberately
dropped). Default `make test` stays at 273 passed, 1 skipped.
Implements the protocol contract pinned in docs/mesh.md. Each peer
runs a stdlib ThreadingHTTPServer; clients send Ed25519-signed
envelopes carrying one of:
ANNOUNCE_ROOT document_root + source_uri + versions
ANNOUNCE_DERIVATION core_root <- surface_roots (proof-blob hash)
ANNOUNCE_PROVIDENCE cache_key + audit_mode + answer_hash
ANNOUNCE_FALSIFICATION cache_key + reason
REQUEST_BODY/DELIVER_BODY pull-on-miss + Merkle-root verify
WireEnvelope canonicalizes via sorted-key separator-tight JSON; the
canonical bytes are what get signed. Receivers verify the signature
against sender_id's sign_pub looked up in mesh_roster at the
envelope's epoch_id — non-members of that epoch produce no valid
signature, so they're silently rejected (401, no audit event).
Each accepted ANNOUNCE_* writes one 'mesh_received' event into the
local audit chain whose body is the full signed envelope (the
remote's signature stays attached for non-repudiation). The
receiver's chain stays internally consistent because we append in
receive-order.
REQUEST_BODY/DELIVER_BODY: the responder ships the document text
plus per-chunk leaf hashes; the client re-derives the Merkle root
from those leaves and refuses to return any body whose leaves
don't reconstruct the requested root.
Tests: 17 unit (envelope canonicalization, sig verify against
roster, type validation, tampered-body rejection, audit-event
write) + 5 end-to-end (two real HTTP peers on ephemeral ports
exchanging announces and bodies). 22/22 green.
Deferred to later commits, per docs/mesh.md:
- CLI verbs (mesh serve, mesh sync) — pending downstream merge
- per-peer chain-of-claims tracking (catchup, replay, fork detection)
- optional AEAD encryption of message bodies (contract says optional)
Adds aborist/wikitext.py with to_base() — deterministic wikitext →
prose conversion via mwparserfromhell. Drops <ref>...</ref>,
[[File:...]], [[Image:...]], [[Category:...]] entirely; resolves
piped wikilinks to display text; collapses templates, formatting,
and HTML markup. extract_wikilinks() preserves the link graph for
the definition-cloud artifact (recoverable from any page on demand
without re-parsing wikitext at query time).
Wires to_base() into verify_quotes(). Without the strip, the
verifier compared the model's clean prose against [[Cloud Strife]],
[[Shinra Electric Power Company|Shinra]], etc. and falsely flagged
real source quotes as VISUAL. Concrete case from a make-query run:
'Cloud Strife, an unsociable mercenary who claims to be a former
1st Class member of Shinra's SOLDIER unit;' is verbatim in the
Final_Fantasy_VII article wikitext (modulo markup). With the strip
that quote now verifies; the genuine model hallucinations in the
same answer still flag honestly. Side-effect: 43% smaller context
size on average so less LLM token waste.
BASE_VERSION = 'wikitext-base-v1' is the algorithm pin. Bump when
the strip rules change. Soft-imported in verify.py so environments
without mwparserfromhell installed degrade gracefully (no strip,
same behavior as before this commit).
Build:
- pyproject.toml: new [wikitext] extras (mwparserfromhell>=0.6),
pulled in by [dev]
- Makefile: chain-check / chain-check-shards targets — fast
audit-chain integrity probe, counts dangling prev_event_hash
references; 0 = chain intact
Tests:
- tests/test_wikitext.py: 31 tests (rules, idempotence, real-corpus
fixture)
- tests/test_verify.py: 2 regression tests pinning the FF7 flip
(Cloud Strife quote: VISUAL → STRICT after strip; genuine
hallucination: stays VISUAL)
- tests/fixtures/ff7_characters_chunk0.wikitext: real chunk from a
shard, used to validate the strip on actual Wikipedia content
Phase 1 of the federation/gossip layer fox sketched as the natural
extension of v9.8 admissibility's content-addressed identity. Two
peers ingesting the same dump already compute identical document_roots
and identical 8-dim cache_keys; the mesh layer is the wire-and-trust
plumbing that lets them dedup answers, exchange Merkle proofs, and
cleanly distrust an evicted member without a hard fork.
Cryptography (cryptography lib, audited):
Ed25519 — every membership mutation + (future) gossip envelope
is signed by the actor's pubkey.
X25519 ECDH — wraps each epoch's symmetric mesh secret to every
current member's DH pubkey via HKDF-derived AEAD key.
ChaCha20-P1305— AEAD for envelope payloads + per-member secret wrap.
State machine:
mesh_identity — singleton; this peer's keys + group name
mesh_roster — per-epoch (member_id, sign_pub, dh_pub, role)
mesh_epochs — epoch_id -> {started_at, started_event_hash,
secret_envelope JSON, reason}
meta:mesh.enabled flag — off by default; gates everything
Eviction works by rotating to a new epoch whose envelope omits the
kicked member. Their prior signatures stay verifiable (the older
roster row is retained), but any gossip from epoch+1 onward is
opaque to them — the secret was never shared with their pubkey.
Authority gate: only roster members with role='admin' can add or
kick. Self-kick is rejected explicitly. The last admin can't be
kicked. Schedule-rotate (refresh secret, no roster change) is open
to any current member as a session-hygiene op.
Audit-chain integration: every mesh state mutation writes an audit
event (mesh_init, mesh_enable/disable, mesh_epoch_rotate). The
epoch's started_event_hash backfills into mesh_epochs after the
audit row commits, giving each epoch a tamper-evident pin into the
ledger.
CLI subcommands: mesh init, mesh status, mesh enable, mesh disable,
mesh members, mesh add, mesh kick, mesh rotate. All read-only or
local-state-only — no network code paths in this commit.
The HTTP gossip wire (`mesh sync`, `mesh serve`) is the next phase.
Schema, cryptography, and roster state machine are all in place to
support it without further migration.
Three cheats stack to drop on-disk store from ~21 KB to ~6.7 KB per doc on
the 2003 enwiki cur corpus (-67% measured, apples-to-apples reingest with
identical document/edge counts; Merkle proofs round-trip 30/30):
1. zstd-compressed chunks.content (level 3). Magic-byte detection on read
means legacy plaintext rows pass through unchanged. Cores stay plaintext
so qa.query._docs_with_core_keyword_match's SQL LOWER+LIKE keeps working.
2. edges WITHOUT ROWID. The composite PK (src_root, edge_type, dst_root,
dst_uri, anchor) covers every column, so a default rowid-based table
near-doubles row data in the PK index. WITHOUT ROWID makes the table
itself the B-tree. Drops idx_edges_dst_uri too — the only query that
filters on dst_uri alone is gravity_top_inbound, a one-shot analytic.
3. contentless FTS5 (content='', contentless_delete=1) eliminates the
28 MB / 1000 docs of duplicated chunk text the old chunks_fts stored.
chunks gets an explicit chunk_id INTEGER PRIMARY KEY so the FTS5
rowid maps back to chunks.chunk_id at search time. Snippets are
built in Python (search/fts5.py:_build_snippet) since SQL snippet()
returns empty in contentless mode.
TF-IDF retrieval also fixed: the prior LIKE '%intel%' substring match
let "intelligence", "intellectual", "intellivision" drown real hits like
Pentium_4 (whose TF-IDF core has "intel" as an exact keyword). Now uses
word-boundary `LIKE '%, intel, %'` patterns plus a match_count over the
distinct query tokens — multi-token coverage outranks single-token title
boosts. Pentium_4 surfaces #1 for "what is the fastest intel CPU?" with
the canonical 2003 answer (Pentium 4 3.20 GHz) instead of an empty
"insufficient sources" reply.
Schema-level changes affect new DBs only; existing v9.8 DBs keep
working at the old layout. Cross-shard UNION views explicitly list the
intersection of columns so a mixed cluster (legacy + new schema shards
in one --shards-dir) still unions cleanly.
aborist/qa/ implements the cache-first answer flow from the providence
whitepaper, scaled up to v9.8's full 8-dim admissibility invariant.
cache_key = SHA-256 of:
source_root | question_hash | model_profile_hash | conversation_hash
| governance_policy_hash | schema_version | canonicalization_version
| chunking_version
Any drift in any dimension yields a distinct cache_key — prior records
cannot serve. Falsification states (failed/stale/quarantined) gate
every cache hit.
- qa/keys.py — pure hash functions, deterministic & testable
- qa/client.py — ChatClient Protocol + StubClient + OpenAI-compatible
HTTP client (vllm/llama.cpp/uncloseai compatible)
- qa/runner.py — ask(): lookup -> hit (no LLM call, hit_count++)
OR miss (call client, write record, audit event,
proof binds answer to source root)
- cli.py — `aborist ask` and `aborist providence` subcommands
- pyproject.toml — httpx promoted from extras to core (used by both
html and qa); selectolax stays in [html] extras
Smoke (StubClient, no network): cache miss writes record with chunk_0
Merkle proof reconstructing source_root; cache hit returns same record
without calling client; 1085 audit events chained 0 breaks across
ingest/derive/evict/rehydrate/providence_write.
