Phase 2 of the concepts/ layer. The Apr 27 commit (c6182ae) shipped
hand-curated frozensets in aborist/qa/concepts.py with a TODO to
"derive from Wikipedia's category graph or 'See also' sections" —
that's this commit. Fox's 2026-05-01 critique landed it: a 7-entry
list of arbitrary frozensets won't scale to a 3.47M-doc corpus, &
adding domains shouldn't require a Python edit + commit + redeploy.
Architecture:
(1) Per-shard concept_relations SQLite table. Append-only, with
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
so re-derivation is idempotent. Lives next to documents in each
shard so mesh sync moves relations alongside the docs that
derived them. A SECONDARY index — writes here NEVER affect
document_root / chunk_root / cache_key, so backfilling is safe
across the entire corpus.
(2) aborist/concepts/ package:
- store.py: add_concept_relation, concept_relations_for_token,
purge_by_evidence_kind, list_evidence_kinds
- query.py: cross-shard synonym_expand, rivalry_excluded;
union-find collapse on synonym edges so partial
pairs build full equivalence classes; mtime-keyed
per-process LRU so retrieval doesn't re-walk
shards on hot loops
- seed.py: one-shot migration of legacy frozensets (8 groups
incl. brain-tech) to evidence_kind='manual_legacy'
rows under source_root='__legacy__concepts__'
- extract.py: pluggable extractor registry. Built-in:
link_reciprocity_synonym — for any reciprocal
edge pair (A→B AND B→A) in the existing edges
table, emit synonym edges between the docs'
title-tokens. Works for Wikipedia (See-also
bidirectional), HTML site internal-link clusters
(russell.ballestrini.net pattern), or any link
graph the corpus already encodes — no new
crawler needed; the html_page parser already
populates `edges` rows on ingest.
(3) aborist/qa/concepts.py rewritten as a backwards-compat shim —
same public API (synonym_expand, rivalry_excluded,
has_compare_phrasing) so query.py call sites unchanged.
shards_dir threaded through _filter_by_title_relevance &
_search_corpus' synonym_expand calls. Without shards_dir
(legacy 2-arg call shape), helpers degenerate to no-op —
matches the behavior the frozenset code had when no group hit.
(4) Cross-shard UNION view: concept_relations added to
_SHARDABLE_TABLES in store.py so connect_query() exposes a
unified view across all shards (same pattern as documents,
chunks, providence_cache, etc.).
Live-verified on the 4-shard 3.47M-doc corpus + the
crawl_russell_ballestrini_net.db shard:
Q: what technology are currently or soon available which may
enable one person to reconstruct and understand some or a
portion of another persons thoughts or ideas without speaking
or sign language?
Result: same as the bde1bd6 in-memory frozenset version —
Telepathy E4 cited alongside Videoconferencing E1/E2,
POINTER-LINKED-PARTIAL 6/10. The DB-backed lookup reproduces
the frozenset behavior byte-for-byte.
Tests: 633 passed (was 624). 9 new concept tests covering
DB-backed synonym/rivalry lookup, shards_dir=None degenerate
behavior, brain-tech group seed, cache invalidation. Old tests
that called helpers directly without shards_dir kept as no-op
assertions (synonym_expand({"athlon"}) without shards_dir returns
{"athlon"} unchanged).
Backfill mechanics: existing live shards needed a one-shot
`connect()` to auto-create the new concept_relations table
(SCHEMA_SQL has CREATE TABLE IF NOT EXISTS). Re-derivation never
mutates the Merkle tree — concept_relations is fully orthogonal
to document_root / chunk_root. Cache_key dimensions are
unaffected.
Deferred (follow-on):
- CLI commands: aborist concepts {seed,list,add,derive,purge}
(currently fox runs the helpers via python -c)
- Wikipedia See-also extractor (requires parsing wikitext sections
beyond what's already in edges)
- Wikipedia category extractor (requires reading Category: links
from chunked wikitext)
Promotes the title-relevance sidecar (diagnose_title_relevance,
qa/inspect.py) to a hard check inside both verify_claim_lattice
and verify_claim_lattice_json. New violation kind TITLE_MISMATCH;
demote-to-HYBRID semantics matching the existing WARRANT_MISSING
pattern.
Catches the 2026-05-02 fox-surfaced retrieval-driven hallucination
class:
Q: "explain spin glass modeling & tensors?"
Pre-Rule-8: EVIDENCE-WARRANTED 1/1, claim cited to Quantum
chromodynamics chunk (single-line "See Also: spin
glass" reference). Token-coverage check passed
accidentally on shared physics vocabulary.
Post-Rule-8: POINTER-LINKED-PARTIAL · title mismatch 1/1.
Cited source title 'Quantum chromodynamics' shares
zero stemmed tokens with claim's {spin, glass,
modeling, tensor, ...} → demote.
Implementation:
_claim_title_overlap(claim_text, source_title) returns True iff
the source title shares ≥1 stemmed content token with the claim.
Uses qa.evidence._content_tokens (≥4-char, post-stopword) and
inline minimal stem (s-strip on tokens >4 chars, skip ss-enders)
to avoid an import cycle.
Per-claim loop in both verifiers checks every cited source's
title; ANY-match suffices (only TITLE_MISMATCH when ALL cited
titles miss). Vacuous-pass when claim or title has no
extractable tokens.
Renderer (_ladder_rung_for_lattice) treats TITLE_MISMATCH
alongside WARRANT_MISSING as the POINTER-LINKED-triggering
signal — both indicate citation/claim structural misalignment.
_render_warrant_tail surfaces "· title mismatch" alongside
"· warrant missing" so an operator sees the specific failure
mode at the audit-line.
Test fixtures updated where the synthetic claims were too minimal
(e.g. "Velociraptors are shown attacking workers" cited to
"Jurassic Park (film)" — claim had no topic anchor). Real model
output naturally references the topic (the model sees the title
in the evidence map and uses it); the fixture revisions reflect
that. 4 new tests in test_verify_json.py covering the helper +
end-to-end TITLE_MISMATCH demote.
Live verification:
Spin-glass query: POINTER-LINKED-PARTIAL · title mismatch ✓
Homer/Mr. Burns: EVIDENCE-WARRANTED ✓ (no regression)
CLAUDE.md updated with the Rule 8 convention; existing
diagnose_title_relevance sidecar marked legacy / dict-form for
per-cache-key inspect use.
Full suite: 738 passed (was 734, +4).
The phrase-pattern retrieval route (commit 1b8677d) closed the
RETRIEVAL side of reference-frame failure; this ticket closes
the ANSWER side.
New module aborist/qa/frame.py:
FrameDetection dataclass (frame_kind, reference_title,
reference_uri, confidence). Sidecar — never enters cache_key
or governance_policy_hash.
detect_frame(question, sources, phrase_match_roots) — heuristic
detector. Reference-frame classification fires when:
(a) phrase route surfaced at least one source, AND
(b) at least one phrase-matched source is a reference work,
determined by:
- title parenthetical disambig (`(novel)` / `(film)` /
`(play)` / `(franchise)` / etc.), OR
- body sample contains ≥3 DISTINCT fiction markers
(novel / published / protagonist / plot / ...).
Distinct-marker count keeps the heuristic robust against a
history article saying "novel approach" twice.
aborist/qa/query.py:
Calls detect_frame for lattice modes only. Body sample uses the
ARTICLE LEAD (chunk_idx=0, post-wikitext-strip) — fiction
markers cluster in the lead on Wikipedia, not in plot chunks
that may have been query-relevance-ranked higher.
New policy field claim_lattice_polarity_preamble injected as a
user-role message before the grounding_reminder when
frame_kind == "reference". Format-string with
{reference_title} placeholder.
Result dict carries frame_detection (kind / title / uri /
confidence) for renderer + bench consumption.
aborist/cli.py:
Renderer adds a `reference frame: <title>` line when
frame_detection.kind == "reference". Skipped for literal /
no-phrase-route / ambiguous rows.
Live verification — Orwell case:
PRE : "The text does not directly state that Oceania has always
been at war with East Asia."
POST : "In George Orwell's dystopian novel Nineteen Eighty-Four,
the nation of Oceania is always at war with Eastasia,
but this is a result of propaganda and doublethink, not
actual historical continuity. The war with Eastasia is
a fabricated conflict to maintain control..."
Multi-frame compilation: distinguishes propaganda claim from
fictional-actual continuity, exactly the polarity contract from
the ticket §2 abstraction.
Literal queries (capital of france) unchanged — polarity preamble
only injects when frame_kind classifies as "reference".
22 new tests (19 in test_frame.py + 3 in test_cli_render.py).
Full suite: 734 passed (was 712, +22).
Directive D3 status: ½ → ✓. Ticket #000002 closed.
All seven structural directives now ✓:
D1, D2, D5 (were ✓);
D3 #000002, D4 #000001, D6 #000003, D7 #000005 (closed in this
series).
New module aborist/qa/retrieval_plan.py:
RetrievalPlan dataclass — frozen, captures the operator-
influenceable retrieval inputs that determine source
selection (retrieval_keywords, top_k, over_fetch,
max_context_chars, shard_ids).
retrieval_plan_hash() — SHA-256 over canonical-JSON.
Deterministic per call; folds into the run-DAG retrieval
stage as a bound input alongside the existing
sources_summary output.
build_run_dag (aborist/qa/dag.py) accepts retrieval_plan_hash:
When provided, the retrieval stage hash binds BOTH plan
(input) and result (output): H({"retrieval_plan_hash":...,
"sources_summary_hash":...}).
When omitted (e.g. legacy / quote-mode callers that haven't
plumbed it yet), the retrieval stage falls back to the
historical sources-summary-only hash so pre-#000001
run_dag_root values stay stable.
query.py constructs the plan per call and passes it through.
Question text intentionally NOT in the plan — already covered
by question_hash. Shard ids included so audit can reproduce
which shards the search ran against.
Two runs with identical sources but different retrieval keywords
now produce different run_dag_root values — the provenance gap
on operator hints (auditor recovers "these were the keywords
that pulled in those sources") closes for the run-DAG path.
Marker test in tests/test_directives.py flipped from "absent"
assertion to "present": test_d4_retrieval_plan_binding_landed.
Plus test_d4_retrieval_plan_hash_module_exists pinning the
module shape. Full suite: 712 passed.
Deferred per ticket §6:
- audit events (retrieval_plan_built / retrieval_result_selected)
- providence_cache.retrieval_keywords SQL column
- optional strict cache_key mode (retrieval_plan_affects_cache_key)
These are ergonomic affordances atop the run-DAG binding; earn
their own tickets when bench evidence shows the blob path is
too friction-heavy for real workflows.
Directive D4 status: ½ → ✓. Ticket #000001 closed.
Replaces the two-rung EVIDENCE-LINKED / EVIDENCE-LINKED-PARTIAL
display label for claim-lattice methods with a four-rung ladder
that names a strictly stronger property at each rung:
POINTER-LINKED pointer/source/chunk verified;
warrant either didn't apply or failed
ANCHOR-WARRANTED pointer-linked + warrant passed where it ran;
other soft demotes may apply
EVIDENCE-WARRANTED anchor-warranted + no soft demotes
UNGROUNDED no verified pairs
HYBRID gets a -PARTIAL suffix on whichever rung applies.
Implementation: _render_audit_label gains a violations parameter
(defaults to None for backward-compat). _ladder_rung_for_lattice
discriminates rungs from the existing violations list:
- WARRANT_MISSING in violations → POINTER-LINKED
- any of {LAZY_ANCHOR_DEMOTED, POINTER_OVERFLOW_TRIMMED,
TOO_MANY_CLAIMS, BARE_NAME_CLAIM} → ANCHOR-WARRANTED
- else → EVIDENCE-WARRANTED
Design simplification vs the ticket's §3 sketch: the proposed
verifier_steps_ran field on the verdict dict was NOT needed. The
existing violations list carries enough signal to discriminate
all rungs. Per the five-step algorithm step 2: don't add fields
you don't need.
Quote / span / entity / paraphrase methods stay unchanged (their
STRICT verifies pinned spans, not synthesis).
Schema column audit_mode enum stays {STRICT, HYBRID, UNGROUNDED}
— pure renderer transformation, no governance_policy_hash bump,
no cache invalidation, no mesh-wire-format change. Existing
providence records render under the new ladder on next read.
5 new renderer tests in tests/test_cli_render.py covering each
rung mapping. D7 anti-regression test in tests/test_directives.py
updated to gate on the ladder labels. Bench helper docstring
follows. Full suite: 711 passed.
Directive D7 stays at ✓; ticket #000005 closed.
Three new question-shape classes dispatched through warrant_check
alongside the existing relation + date anchors:
(1) Entity-list shape — `name X`, `list X`, `who are the members
of X`. List-aware extractor `extract_entity_list_anchors`
(multi-word phrases ∪ solo-cap individual names) so comma-
separated entities each contribute. ANY-match semantics:
demote-don't-reject when an extra entity from training-prior
appears alongside grounded ones.
(2) Count shape — `how many X`, `how much X`. Digit ↔ word
equivalence (claim says "six", span says "6", or vice versa)
with ordinal collapse (`sixth → 6`). Year-shaped digits
filter out (those belong to the existing date anchor class).
ALL-match semantics: every count token in the claim must
appear in some cited span as digit or word.
(3) Why-cause shape — `why X`. Cause-anchor pool widens to
≥5-char lowercase common nouns (post a generic stopword set
that filters quantifier-adjective fillers like "various",
"factors", "situation") PLUS proper-noun anchors from the
existing extractor. Gated on why-shape only: lowercase
common-noun extraction has higher false-positive risk
elsewhere.
Per-class policy gate (proposed `claim_lattice_warrant_classes`
dict) deferred per the five-step algorithm step 2: single
`warrant_check_enabled: bool` is the minimum viable gate; per-
class flags earn their slot when bench evidence shows over-firing
on a specific class.
17 new warrant tests (detector + extractor + integration).
Marker test in test_directives.py flipped from "absent" to
"present" assertion: test_d6_warrant_generalization_landed.
Full suite: 709 passed (was 692, +17).
Directive D6 status flipped to ✓ in seven-point-program.md.
Ticket #000003 closed.
Each directive gets a structural pin in tests/test_directives.py.
A future PR that silently weakens a directive fails by name.
D1 (no LLM in hard verifier path):
- verify_quotes / verify_claim_lattice / verify_claim_lattice_json
signatures must NOT carry chat_client / llm / judge / model
parameters.
- SCHEMA_SQL verifier_method CHECK constraint enum excludes
'llm', 'model', 'judge', 'nli', 'hermes', 'ai'.
D2 (Hermes emits pointer clauses):
- ANSWER_MODES contains both lattice variants.
- DEFAULT_ANSWER_MODE in ANSWER_MODES.
- parse_pointer_claims returns structured (claim_text, pointer_ids)
nodes from pointer-line input.
D3 (build CTI internally):
- Runtime parses pointer-line prose into typed claim nodes
(text + pointer_ids list); model never produces structured
output directly in pointer mode.
- Evidence map built by runtime from retrieved chunks; pointer
ids minted sequentially; evidence ids content-addressed.
D4 (bind retrieval AND evidence map):
- build_run_dag accepts evidence_map_root parameter (✓).
- Marker test: build_run_dag does NOT yet accept retrieval_plan_hash
parameter; flip when ticket #000001 lands.
D5 (deterministic pointer verify):
- verify_claim_lattice is deterministic: same inputs → same
verdict byte-for-byte.
- _claim_textually_overlaps_evidence (the per-pair coverage
check) is a pure function with no LLM-side parameters.
D6 (anchor-class warrant before NLI):
- warrant_check signature is lexical-only (no chat_client / llm
/ judge / nli params).
- Date-anchor failure case fires (year missing from cited spans).
- Relation-shape failure case fires when ALL proper-noun anchors
are missing from cited spans.
- Marker test: per-shape detectors for entity-list / count /
why-cause shapes are absent today; flip when ticket #000003 lands.
D7 (rename labels honestly):
- Renderer maps STRICT → EVIDENCE-LINKED for claim_lattice* methods.
- Renderer keeps STRICT for quote / span / entity / paraphrase
methods (those verify against pinned spans, not synthesis).
- SCHEMA_SQL audit_mode CHECK constraint enum stays
{STRICT, HYBRID, UNGROUNDED} regardless of renderer relabel.
D8 (test-pinning before automation):
- docs/seven-point-program.md exists.
- bench/qa_sweep.py exposes _directive_compliance helper.
- docs/TICKETS.md indexes the partial directives (D3, D4, D6) so
the design log doesn't drift from the program doc.
23 new tests; full suite 692 passed (was 669, +23).
The "marker" tests for D4 and D6 document open work: today they
assert that a parameter / detector is ABSENT; when the corresponding
ticket lands, the assertion flips to PRESENT. The test failing
during ticket implementation is the expected signal, not a
regression — the test message names the ticket so the work-in-
progress is legible.
Wires the seven-point program (docs/seven-point-program.md) into
the bench harness as a per-mode coverage table.
Per-row computation in _directive_compliance(answer_mode, result, err):
D2_pointer_clauses — answer_mode is lattice variant
D3_cti_substrate_ready — lattice mode + run_dag_root populated
(full coverage pending ticket #000002)
D4_evidence_map_bound — run_dag_root populated
(retrieval-plan binding pending #000001)
D6_warrant_fired — warrant ran (lattice mode today;
per-shape gating pending #000003)
D7_honest_label — audit_mode in canonical enum
(renderer transformation pinned in
test_cli_render.py)
D1 (no LLM in verifier), D5 (verifier_method enum), D8 (test-pinning
discipline) are global properties of the substrate and don't appear
per-row — they get tracked once in CLAUDE.md / the program doc.
_summarize aggregates per-row booleans into per-mode pass counts.
_render_markdown emits a 'directive coverage (seven-point program)'
section with per-mode per-directive coverage as 'count/N (pct%)'.
Discipline rationale per CLAUDE.md's five-step algorithm step 5:
the bench harness IS the automation substrate, so it must enforce
the directives before any feature gets layered on top. Bench scores
that climb without directive coverage climbing are graveyard-digging
(step 4: don't dig the grave faster).
5 new bench tests cover: per-mode pass aggregation, the markdown
section presence and content, the helper's behavior on quote mode
(D2 fail by design), lattice mode (all directives pass), and error
rows (empty dict — no signal). Full suite 669 passed.
Generalizes warrant-lite from relation-only to anchor-class warrant.
Two anchor classes now compose:
(1) Proper-noun anchors — gated on relation-question shape; any-match
semantics (existing behavior).
(2) Date anchors — always-on, gated on the CLAIM containing a year or
month name. ALL components (year + month name if present) must
appear in some cited span.
Closes the back-to-the-future failure fox surfaced 2026-05-01:
Q: what date did back to the future come out?
Claim: "Back to the Future was released in theaters on July 3, 1985."
Cited span: trilogy/SNES/pinball prose with '...real 1985...' but
no 'July' anywhere
Pre-fix: EVIDENCE-LINKED · via claim_lattice 1/1
Post-fix: EVIDENCE-LINKED-PARTIAL · via claim_lattice · warrant missing 1/1
Year extraction (4-digit, 1500-2199 range — avoids matching ZIP codes
/ elevations / model IDs) and month extraction (full English month
names — three-letter abbrevs deferred until bench shows the gap).
Both anchors required when both are in the claim. Year-only claim
against year-only span continues to pass; month-only against
month-only continues to pass; month-and-year claim against year-only
span (the BTTF case) now correctly fails.
Architectural note: this generalizes warrant-lite without building a
typed-contract framework (per-question-type rule libraries like
"release_date_lookup with required predicate cues"). The date anchor
primitive catches the date lazy-anchor failure class deterministically
without a per-type rule book; typed contracts earn their slot only if
bench evidence shows general anchors miss specific cases. Dropped the
caller-side is_relation_question gate from verify.py — warrant_check
now decides internally what to enforce based on question shape AND
claim content.
Tests: 29 unit tests in tests/test_warrant.py (was 16). New tests
cover extract_date_anchors (year, month, dedup, non-year-skip,
empty), warrant_check date semantics (BTTF case, year-only,
year-in-unrelated-context, month+year composition), proper-noun
warrant skipped without relation question. Full suite: 554 passed,
21 skipped.
Limitation accepted: ISO date format ("1985-07-03") wouldn't satisfy
month-name requirement. Real-world Wikipedia prose uses month names
("released on July 3, 1985"), so the trade-off favors catching the
common case. If bench shows ISO false-rejections, layer in a
month-number alternative to the month-name regex.
Two follow-ups to the phrase-pattern retrieval fix (commit 1b8677d)
covering items 6 and 10-11 of fox's 2026-05-01 architectural review:
(1) Non-regression tests for the phrase route:
- test_phrase_route_skipped_when_question_shorter_than_min_n
pins the structural false-positive guard: the n=5/n=6 minimum
means a 4-token literal-geography query lacks enough tokens to
trigger the route at all.
- test_phrase_route_does_not_hijack_literal_geography_query
end-to-end: a 4-token "oceania east asia geography" query on
a synthetic 2-doc corpus surfaces only the geography-stub doc;
the orwell-stub doc (whose body has the diagnostic 5-gram) is
correctly NOT pulled in by the phrase route on a literal query.
(2) docs/ticket-000002-reference-frame-polarity-contract.md
Captures fox's Module L proposal verbatim as Appendix A and
extracts the implementation sketch into the standard ticket
body (problem statement, abstraction, CTI interpretation, three
pieces of code to write, test list, scope boundaries).
The phrase route closed the RETRIEVAL side of reference-frame
failure. Module L addresses the ANSWER side: today's substrate
answers Orwell queries as "the text does not directly state..."
when it should produce multi-frame answers distinguishing
Party propaganda from fictional-actual continuity. Forecast
cost ~3-4 hours; risk medium (prompt augmentation interaction
with claim_lattice prompt).
Module M = ticket #000001 (route provenance binding); not
duplicated. Module N (FP guards) partially landed via the
tests above; remaining tests folded into ticket #000002's
test list. Module H (relation warrant lite) lacks scope
detail; deferred without a ticket.
(3) docs/TICKETS.md updated: index gains #000002 row, Next ID
bumped to 000003.
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
Two UX fixes addressing the 2026-05-01 Orwell run feedback:
(1) Evidence pointer format. Old form `[E5: "<excerpt>"]` looked
visually like a 1-indexed source rank — paired with a "sources (8)"
list whose `[5]` slot was a different document, operators easily
mis-attributed citations. New form:
[E5 | Nineteen Eighty-Four | 682f0a11: "<excerpt>"]
Title comes from EvidenceObject.title (URI-tail fallback when None);
chunk_root prefix is the first 8 hex chars — enough to disambiguate
chunks from the same source while staying compact. Renderer change
in render_claim_lattice; pre-existing claim-lattice tests updated;
3 new unit tests pin the format (title-inline, URI-tail fallback,
no `[E1:` collision-shape).
(2) Audit-label discipline. _render_audit_label already maps STRICT
→ EVIDENCE-LINKED for claim_lattice* modes (verifier output stays
binary, schema CHECK constraint stays); 4 new tests in
test_cli_render.py pin the relabel for STRICT / HYBRID / UNGROUNDED
in claim_lattice modes and the no-relabel for quote-mode where the
verifier verifies pinned spans not synthesis.
Net: 8 new tests, full suite 649 passed.
Closes the lazy-anchor failure class fox surfaced in feedback-3
(2026-05-01) on "who is homer simpson's boss?":
Claim: "Homer Simpson's boss is Mr. Burns."
Cited span: voice-actor / Castellaneta bio prose
Pre-warrant: pointer-linked, source-role allowed, coverage passes
Post-warrant: WARRANT_MISSING violation, audit_mode capped at HYBRID
Pointer verification != warrant verification. A pointer resolves to
a span; a warrant requires the span to actually contain the claim's
named answer entity.
Adds aborist/qa/warrant.py:
- is_relation_question(q) — regex shape detector for "who is X's Y?",
"who is supermans girlfriend?", "who founded Z?", "boss of X" etc.
- extract_answer_anchors(claim) — multi-word proper nouns ("Mr.
Burns", "Lois Lane") with possessive 's stripping; solo-cap
fallback that skips sentence-starter false positives
- warrant_check(claim, spans) — at-least-one-anchor lexical match;
vacuous-pass when claim has no proper-noun anchor
Wired into verify_claim_lattice via new params (question,
warrant_check_enabled). Per-claim WARRANT_MISSING violations cap
audit_mode at HYBRID using the same demote pattern as
lazy_anchor_demoted. CLI renders " · warrant missing" tail when
violations present so the failure mode is legible without scanning
the violations array.
Policy field claim_lattice_warrant_check_enabled added to
DEFAULT_POLICY / DEFAULT_QUERY_POLICY and the verifier-policy hash
field set so changes bump verifier_policy_hash.
Hard or soft? Per CLAUDE.md "Soft hash vs hard hash":
substring/lexical = hard. Output is binary (warrant present or not),
violations enter the proof path through violations[] alongside
SOURCE_ROLE_BLOCKED and CITATION_MISMATCH.
Deferred (follow-on commits):
- full RELATION_PATTERNS table with relation-noun aliases
- two-hop bridge detection ("X founded Y" + "Y is Z" -> "X is Z")
- retrieval expansion for relation questions
Tests: 16 new unit tests in tests/test_warrant.py covering
is_relation_question, extract_answer_anchors, warrant_check
(happy path, lazy-anchor case, vacuous-pass, case-insensitive,
empty inputs). Full suite: 523 passed, 21 skipped.
Implements 6 of fox's feedback-2/3 items (2026-05-01) targeting the
red-fish-blue-fish output. Each lives in the render or sidecar layer
— no schema or cache-key change beyond the verifier_policy_hash that
landed earlier in the day.
(1) Honest audit-label rendering. Schema audit_mode (STRICT/HYBRID/
UNGROUNDED) describes the lexical verifier's call; the user-
facing label spells out what was actually checked. Mapping for
claim_lattice modes:
STRICT → EVIDENCE-LINKED · via claim_lattice
HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice
UNGROUNDED → UNGROUNDED · via claim_lattice
Quote / span / entity / paraphrase modes keep the audit_mode
token plus a `· via {method}` tail. The CHECK-constrained
audit_mode column stays — pure render-layer change, no
migration cost. Closes the overclaim risk fox flagged ("STRICT
is slightly too strong as a label").
(2) Sentence-bounded evidence spans. _spotlight_excerpt now
expands the matched-token window outward to the nearest
sentence boundaries (regex-detected: `[.!?][\"')\\]]?\\s+(?=[A-Z])`)
instead of cutting at fixed byte offsets. Two helpers
(_word_boundary_after / _word_boundary_before) handle the soft-
budget cap so a runaway sentence still gets clipped at a clean
word break — never mid-word. Pre-fix the renderer produced
excerpts like `"...freewheeling plot... friends and p..."` (the
`p...` is the offending mid-word truncation fox surfaced); now
the excerpt is `"One Fish Two Fish Red Fish Blue Fish is a
1960 children's book by Dr. Seuss. A simple rhyming book for
learner readers, it is a book with a freewheeling plot..."`.
(3) Source-role display + used/unused annotation. The CLI source
list went from
[1] One Fish Two Fish Red Fish Blue Fish — wikipedia.org/...
[2] Red Dwarf — wikipedia.org/...
to
[1] One Fish Two Fish Red Fish Blue Fish — primary_answer_source — used (E1) — ...
[2] Red Dwarf — background_source — unused — ...
Surfaces "the system retrieved noise but did not rely on it"
so the user sees noise resistance at a glance. Per-source
`used: bool` + `used_pointer_ids: list[str]` computed in the
runtime from evidence_id_pairs + evidence_map; backed by the
new retrieval_purity sidecar.
(4) Retrieval-purity sidecar. Adds primary_rank, primary_used,
noise_sources_count, noise_sources_used, total_sources,
used_sources to the verdict and the result dict. Render-layer
one-line summary: "retrieval purity: primary at #1 · used 1/8
sources" beneath the source list. Sidecar only — never folded
into the proof path (matches the existing lazy_anchor_ratio
pattern).
(5) Ordered-token title scoring. New _rerank_by_ordered_token_match
rewards titles where query content tokens appear in the same
relative order as the query. "red fish blue fish" query →
"One Fish Two Fish Red Fish Blue Fish" gets ordered_match=4
boost (1+0.5*3 = 2.5×); "Red Dwarf" stays at ordered_match=1
(no boost). LCS-based, stem-aware, deterministic. Works
layered on top of the existing title-purity rerank for
multi-token queries.
(6) Noise-resistance fixture (tests/test_qa_quality_live.py).
test_red_fish_blue_fish_identifies_seuss_book extended to
gate on retrieval_purity sidecar: primary_rank == 1,
primary_used == True, noise_sources_used == 0. Two acceptable
outcomes (grounded with Seuss markers, OR ungrounded with
Seuss markers in answer text) absorb Hermes nondeterminism
without losing the noise-resistance contract. Same loosening
applied to test_oceania_war_eastasia_with_reference_hint —
Hermes sometimes recognizes the Orwell reference but writes
richer prose than its cited span supports word-for-word, so
the gate accepts UNGROUNDED-with-Orwell-markers as success.
Live test_qa_quality_live.py: 20/20 fixtures pass. 507 unit tests
green. Working tree leaves bench/qa_questions.txt for fox's
parallel WIP.
Six items from fox's de-novo synthesis (2026-05-01) landing as one
atomic commit. Each item references its motivation and respects the
v9.8 honesty boundary (no claims of semantic truth, no proof-path
contamination by soft signals).
NEW DOCS
--------
docs/cti-architecture.md
Maps today's modules onto the architectural layer cake fox named
in his synthesis: PROMETHEUS-Σ (controller) / CTI (claim-lattice
reasoning IR) / Merkle-AGI-DAG (commitment) / Reverse-RAG
(evidence direction) / v9.8 Providence (admissibility ledger) /
Hermes (weak proposer). Per-layer responsibility table + module
map showing which existing files implement which layer. The
architecture is real; the labels above name what's already there.
docs/naming-deferral.md
Argues why we keep internal terms (claim_lattice,
claim_lattice_pointer, verify_claim_lattice) instead of renaming
to architectural labels (CTI, PROMETHEUS-Σ). The rename costs
cache invalidation, ~150 test fixture references, schema CHECK
migration, blame-history disconnect, mesh peer coordination.
The bridge is the cti-architecture mapping doc — read it once,
then read code in code's vocabulary and prose in prose's
vocabulary. Lists four triggers that would invert the decision.
docs/self-reference-distillation-design.md
Successor to docs/self-reference-thought-chains-design.md (the
flat-source MVP). Maps STRICT claims onto the existing Distiller
/ Core / derivations infrastructure: each STRICT record becomes
a Fact-Core via a new ProvidenceDistiller, with per-cited-chunk
Merkle inclusion proofs back to Wikipedia source_roots. The
fact-graph grows; new claims attach by inheriting the proof
chain. CompositionDistiller (combining facts into new claims)
is explicitly deferred — that's reasoning machinery, not
infrastructure.
docs/test-coverage-audit-2026-05-01.md
Maps fox's §11 test list (16 items) against the codebase. 16/16
covered. Documents item #9's manual-quote-rule lifecycle: the
rule was retired from pointer mode in commit 224bfd6 after the
bench journey identified it was rejecting factually correct
claims; retained in JSON variant where the punctuation-rationale
argument doesn't apply. The audit doc itself is the requirements-
drift defense.
CODE
----
aborist/qa/keys.py:verifier_policy_hash
New pure function. Hashes the verifier-relevant subset of policy
(answer_mode + claim_lattice_* verifier knobs + entity_policy
fields + base_version). Folded into cache_key as an optional
9th dimension via a new keyword arg with backward-compat default
None — legacy 8-dim callers continue to work and produce the
pre-2026-05-01 hash result.
aborist/qa/{runner,query}.py
Compute verifier_policy_hash from the policy_variant and pass
to cache_key as the 9th dim. Records written from this commit
forward bind to the verifier-policy identity. Pre-existing 8-dim
records become cache-misses on next lookup — same migration cost
as any governance_policy_hash bump. The 9-dim form makes the
question "did the verifier rules change?" answerable from
cache_key diff alone, separate from "did the prompt change?"
CLAUDE.md
Adds "old maps vs runtime maps" engineering discipline to the
bench-maxing section. Codifies the principle: every base model
carries old maps from training time; the runtime carries the
fresh map; when they disagree, the runtime wins. Move authority
OUT of the model's prior and INTO runtime artifacts (pointer IDs
the runtime mints, source spans the runtime interpolates by
offset, evidence maps assembled per query, policy hashes that
fold prompt/verifier/retrieval into cache_key, hard checks run
by the verifier). Hermes' content-addressed-evidence-id
hallucination (commit bb8450d) is the canonical case study.
TESTS
-----
tests/test_keys.py
Six new tests for verifier_policy_hash:
- only hashes verifier subset (non-verifier fields don't change it)
- changes when verifier-relevant field changes
- empty-policy → stable
- 9-dim cache_key distinct from 8-dim
- 9-dim distinct under different verifier hashes
- 8-dim form preserved for legacy callers (None == omit)
DEFERRED (per fox + naming-deferral.md)
---------------------------------------
- PROMETHEUS-Σ as an extracted controller module — the dispatch
logic in runner.py + query.py already IS PROMETHEUS-Σ; an
explicit prometheus.py is a refactor with no behavior change.
Defer until a §5-rename-trigger fires.
507 tests pass (was 501 before, +6 from verifier_policy_hash
coverage).
Three layers, 14 new tests, full suite 611 passed (was 597):
UNIT — tests/test_query.py
test_query_returns_prompt_chars_breakdown
Asserts the result dict's prompt_chars carries exactly the five
expected keys and messages_total equals sum of message contents
the StubClient saw.
test_query_answer_chars_matches_answer_text
answer_chars == len(answer_text) — drift check.
test_query_cache_hit_also_returns_capacity_metrics
Cache-hit path populates prompt_chars + answer_chars (operators
inspecting cached records still want the breakdown).
test_query_evidence_chars_grows_with_topk
Sanity: more sources / larger budget → more evidence chars
(the metric tracks actual context build, not a stale constant).
INTEGRATION — tests/test_query.py (retrieval_keywords)
test_retrieval_keywords_does_not_alter_question_to_llm
Keywords don't appear in the LLM-facing question segment;
system prompt unchanged across runs. Pins the substrate
contract: keywords are FTS5/title-filter-only.
test_retrieval_keywords_changes_retrieved_sources
Different keyword sets surface different docs (the actual
user-visible behavior).
UNIT — tests/test_cli_render.py
test_render_shows_capacity_line_when_prompt_chars_present
Capacity line appears with messages_total + breakdown when
prompt_chars is in the result dict.
test_render_omits_capacity_line_on_legacy_results_without_prompt_chars
Backwards-compat: legacy results render cleanly without the
capacity line — no KeyError, no '0 chars' noise.
test_render_capacity_thousand_separators
61,550 not 61550 — operator legibility on daily renders.
UNIT/INTEGRATION — tests/test_bench_qa_sweep.py (NEW FILE)
Imports bench/qa_sweep.py via importlib.util so the module's
not in the Python path doesn't matter. Five tests:
- _summarize counts verdicts by mode
- deflections counted only on STRICT/HYBRID rows (not UNGROUNDED)
- rendered markdown has the headline summary + size buckets
- size buckets correctly stratify strict-rate by prompt_chars_total
- empty buckets are skipped (no '0 runs' noise)
FUNCTIONAL — live verification (no automated test, manual)
`make query Q="what is the capital of france?" BURN=1` confirmed
in commit f927298 to render the capacity one-liner under the
source list. Documented in that commit's body.
Also corrected the docstring on query()'s `retrieval_keywords` to
reflect that keywords don't enter cache_key DIRECTLY but do change
context_root + conversation_hash via source selection — so the same
question with different keywords lands under different cache_keys
(legitimately, since the LLM saw different contexts).
Empirical finding 2026-05-01 on 'explain spin glass modeling, tensors?':
returned STRICT 1/1 with claim 'Spin glass modeling involves...
mathematical tools such as tensors' cited to a chunk from the
*Quantum chromodynamics* article. Token-coverage check inside the
chunk passed accidentally on shared physics vocabulary
(spin/model/etc.), but the article isn't about spin glasses — the
claim is a retrieval-driven hallucination.
Root retrieval cause: AND-mode FTS5 returned zero hits (Spin_glass
article has 0 occurrences of 'modeling' or 'tensors'); OR-mode
fallback ranked by raw BM25, so 'Glass' (glass*274) outranked
'Spin glass' (spin*29 + glass*15) despite Spin_glass matching 2
distinct query tokens vs Glass's 1. Verifier then accepted the
hallucinated claim because the cited QCD chunk has incidental
physics vocabulary overlap.
New ``diagnose_title_relevance(claim_text, cited_source_titles)``
sidecar: claim's content tokens (stemmed) must overlap the union
of cited source titles. Zero overlap → ``title_mismatch``. Catches
the spin-tensors case (claim about spin glass, cited title 'Quantum
chromodynamics' → zero stem overlap) without false-positives on
legitimate cross-document grounding (Lois_Lane article cited for a
Lois-Lane claim still has lois+lane overlap).
Sidecar-only for now — staying out of the binary verifier output
until bench evidence justifies hard-promotion. Not yet wired into
inspect_cache_key (per-claim citation source mapping isn't in
providence_cache yet) or the bench. Available as a utility for
operators auditing specific (claim, source_title) pairs.
5 new unit tests covering match/mismatch/no-titles/no-claim-tokens
and underscore-normalization for Wikipedia titles.
Closes the recursive-grounding gap fox surfaced today: aborist
tends Wikipedia trees but never grafts its own past Q&A records
into the forest. Each query starts from cold corpus retrieval;
prior providence_cache records sit unread until the same question
is re-asked (cache_key match). The "kindergarten thought chains"
framing names the gap — the system has a kindergarten of thoughts
(early STRICT records) that should mature into citable substrate
as they cool.
This commit lands the MVP — STRICT live providence records past
the kindergarten window become substrate via a new Source
subclass. Trust model per fox: "we trust strict statements as
fact unless a verifier falsifies it."
NEW
---
- docs/self-reference-thought-chains-design.md — full architecture
doc covering the four iteration-time gates, the recursive Merkle
proof story (Q2 → Q1 → Wikipedia chunk per v9.8's recursive-
cores insight), the falsification trust model (state=live as the
fail-closed gate), risks (lazy-anchor compounding, echo chambers,
storage bloat), and bench-impact estimate.
- aborist/sources/providence.py — ProvidenceSource(Source) with
four hard gates:
1. audit_mode == 'STRICT' (HYBRID/UNGROUNDED stay opaque)
2. falsification_state == 'live' (failed/stale/quarantined
excluded — verifier-falsification mechanism per fox)
3. now - created_at >= kindergarten_seconds (default 1h —
fresh thoughts cool first; kills tight echo loops)
4. anti-recursion: records whose answer text contains a
self-reference URI are skipped — first-generation only
- tests/test_providence_source.py — 10 unit tests covering each
gate plus the URI-scheme source-role classifier
- Makefile target `ingest-self-providence` (KG_SECONDS=3600
default; iterates each shard and self-promotes its STRICT live
records — cross-shard sharing happens via the existing
shards-dir UNION at retrieval time)
WIRE-UP
-------
- aborist/qa/query.py
- SOURCE_ROLE_BUDGET_WEIGHTS: self_reference_source = 1.0
(same as background — Wikipedia stays canonical primary;
self-reference is supplementary anchoring)
- SOURCE_ROLE_RANK_WEIGHTS: self_reference_source = 0.9
- _classify_source_role: short-circuits on aborist://providence/
URI prefix → self_reference_source regardless of title shape
- DEFAULT_QUERY_POLICY['claim_lattice_allowed_source_roles']
+= 'self_reference_source'
- aborist/qa/runner.py — same allowlist update for the
per-document `ask` path
- aborist/cli.py — `aborist ingest --source providence` reads the
providence_cache from the same shard it writes into;
--kindergarten-seconds flag plumbed through
NOT IN THIS COMMIT
------------------
- Aggregation of multiple Q&A records into synthesized summary
records (follow-on)
- Self-reference for HYBRID records (only STRICT is substrate
today; HYBRID could land later as a soft-anchor role with
lower trust)
- Live virtual sourcing (the design discusses it; MVP uses
snapshot ingestion so existing FTS / chunker / Merkle apply
with zero schema change)
- A live bench validating actual lift on self-reference questions
(requires running ingest-self-providence then bench; deferred
to follow-on commit on real data)
10 new unit tests pass; full suite at 482 passed / 21 skipped
(live fixtures gated).
Bench dig (2026-05-01, 65-question sweep): 4 of 7 deflecting
questions were false positives from verb-shape subject anchors.
what year did the berlin wall fall? → answer "1989" (anchor "fall")
when did the soviet union dissolve? → answer "1991" (anchor "dissolve")
how many wives did henry the eighth? → answer "six" (anchor "eighth")
what year does our cold fusion ...? → answer "1989" (anchor "happen")
why did the titanic sink? → "...sank in 1912" (anchor "sink" vs "sank")
why did the dinosaurs go extinct? → "asteroid impact" (anchor "extinct" vs "died")
These shapes ask for a year/count/cause; the answer is structurally
not expected to echo the question's subject. Pre-suppression, all
three modes landed deflection on these — the apparent
"pointer-only" deflection rate (10.3% vs 6.2%) was almost entirely
heuristic noise, not a real substrate failure.
New ``shape_suppressed`` flag returned by diagnose_deflection. When
the question starts with one of:
when, what year, what date, what time,
how many, how much, how long, how old, why
the subject-anchor check is skipped — overlap-ratio takes over.
Mars-BDFL (who-shape) still fires correctly.
Field exposed so the operator can filter the signal: a
shape_suppressed=True row that still classifies as deflection on
overlap-ratio means "answer was a bare year/count/cause with no
topic words" — known noisy, not real topic shift.
5 new unit tests; full suite 582 passed (was 578).
Empirical finding 2026-04-30 on 'who is a benevolent dictator for life
for mars?': JSON mode returned STRICT with answer 'Guido van Rossum is
a Benevolent Dictator For Life (BDFL) for the Python programming
language.' The verifier did its job (claims grounded), but the answer
never mentioned Mars — pure topic-shift. A real failure mode for
adversarial-premise questions where the model deflects rather than
refuses.
Detection rule: subject-anchor heuristic. The LAST content token in the
question (after stopword strip) is treated as the question's primary
subject ('mars' for BDFL, 'beatles' for 'who are the members of the
beatles?'). If the subject anchor is missing from the answer, classify
as 'deflection' regardless of generic-vocabulary overlap. The Mars-BDFL
case has 3/4 generic overlap (benevolent, dictator, life) but missing
subject — exactly the pattern overlap-ratio alone can't catch.
Returns dict with kind ∈ {deflection, partial_overlap, on_topic,
no_question_tokens} plus subject_anchor + subject_in_answer for
operator legibility.
Sidecar-only — wired into inspect_cache_key() but never feeds back
into providence_cache or audit_events. Per the verifier-stays-binary
discipline (CLAUDE.md): soft signals live in inspect verbs, never in
the hard chain.
Bench finding (2026-04-30, n=3 across 24 questions): JSON mode hit
S:3 H:0 U:0 on three questions where pointer mode landed S:0 H:3 U:0
or quote mode landed S:0 H:0 U:3 — clean modal divergence. Pinning
these as live fixtures gives us regression detection: a JSON-mode
quality drop on these questions reverts the STRICT signal first.
test_mona_lisa_strict_and_names_leonardo — was _grounded, tightened
to STRICT (bench: quote 0S, pointer 0S, JSON 3S)
test_supermans_girlfriend_strict_and_names_lois_lane (NEW)
— relationship-shape, JSON's strong suit (quote 0S, pointer 0S,
JSON 3S)
test_cold_fusion_breakthrough_year_strict_and_names_1989 (NEW)
— out-of-corpus rephrasing, atomic-claim rule unlocked grounding
(quote 0S 3U, pointer 3S, JSON 3S)
Per fox's "scoreboard + gates" docstring: bench gives aggregate
counts; these fixtures fail by name when the scoreboard moves.
Defense-in-depth against the "tell me all there is to know about X"
runaway shape. Bench finding (2026-04-30 york-england, n=3): JSON mode
emitted 26-59 claim-pointer pairs of which only 2-4 verified — model
treats encyclopedic prompts as a license to spam claims from training.
The atomic-claim prompt rule (b5925c8) cut typical runs to ~10 claims,
but a structural ceiling makes the failure shape operator-visible
regardless of per-claim verification.
Cap: 12. Admits typical entity-list questions (5-7 dinosaurs, simpsons
+ pets) while flagging the runaway. Plumbed through both verifiers
(pointer + JSON) via claim_lattice_max_claims_per_answer policy knob;
folds into governance_policy_hash on change.
Behavior: cap doesn't truncate — every claim still verifies so the
operator sees full evidence of the runaway. The TOO_MANY_CLAIMS
violation demotes STRICT → HYBRID via the existing violation-check
path; UNGROUNDED if no claims verified.
Tests: test_verify_json_too_many_claims_demotes_to_hybrid (13 claims
all verifying → HYBRID), test_verify_json_at_cap_can_still_strict
(12 claims at the cap → STRICT, no violation), and pointer-mode
mirror test_too_many_claims_demotes_pointer_mode_to_hybrid.
Prompt audit + rewrite per the negation-avoidance principle codified
in CLAUDE.md: small instruction-tuned models (Hermes-3-8B observed)
struggle with "do NOT X" / "never Y" / "without Z" — the negation
often pattern-matches away or inverts under attention. Rewrote 18
negation instances across 8 prompt fields in
DEFAULT_POLICY (runner.py) and DEFAULT_QUERY_POLICY (query.py) to
their positive forms. Examples:
"Do NOT invent IDs" → "Use only the IDs that appear in the
EVIDENCE blocks above"
"Never begin a line with E1:" → "Each line begins with the
claim text, then a bracket tag"
"No commentary, no preamble" → "Your output begins with `{`
and ends with `}`"
"If no evidence supports a claim, omit the claim" →
"Emit only claims that an EVIDENCE block directly supports"
Also added an atomic-claim rule (Rule 6 JSON, Rule 9 pointer):
"Each claim is one focused fact citing the source that directly
contains it. Multiple distinct facts become multiple claim
entries… Three short claims, each citing one pointer, beat one
long claim citing five." Targets the StarCraft-style mega-claim
where the model packs every fact into a single sentence with
five pointers.
Bench delta (n=3, 25 questions, 75 runs per mode):
quote 30S 28H 17U → 40S 23H 12U +10 STRICT, +13pp rate
pointer 27S 37H 11U → 23S 37H 15U -4 STRICT (acceptable —
JSON is now the daily
driver; pointer is
fallback/diagnostic)
JSON 36S 28H 11U → 44S 26H 5U +8 STRICT, +11pp rate,
-6 UNGROUNDED, -1.8s latency
Net across modes: +14 STRICT, +7 grounded, JSON latency
7.2s → 5.4s.
Bench questions: replaced "isn't it true that the great wall of
china is visible from space?" with "which side is the ground
elevation highest throughout the span of the great wall of
china, the north or south?" — a forensic-shape question with
answer "north". Live finding: model hits STRICT 1/1 with the
right answer cited to a Great-Wall-but-not-elevation chunk
(coverage threshold passes via topical-token overlap). New
fixture test_great_wall_elevation_north_side_higher gates on
entity-presence ("north" in answer) but tolerates UNGROUNDED —
documents the residual lazy-anchor failure mode for the
verifier-semantic roadmap.
CLAUDE.md: added negation-avoidance to the bench-maxing
discipline section.
Bench surfaced a real test bug: the prior assertion treated STRICT as
'model asserted a Mars BDFL,' but STRICT only means 'claims are
grounded.' On 'who is a benevolent dictator for life for mars?',
claim_lattice mode deflected to Guido van Rossum / Python (no Mars
mention) and the verifier returned STRICT correctly — claims grounded,
false premise sidestepped.
Three accept paths now:
1. UNGROUNDED (refused outright)
2. STRICT/HYBRID + mentions 'mars' + includes refutation marker
3. STRICT/HYBRID + does not mention 'mars' (deflection — answered
an adjacent grounded question instead of the false-premise one)
Fail case is unchanged: answer mentions Mars AND affirmatively names
a Mars BDFL with no refutation language.
Refutation markers expanded with 'no recognized', 'no known',
'fictional' to cover sci-fi-framing answers (Barsoom / Mars trilogy
characters as fictional dictators is also non-hallucinatory).
Question-set churn (bench/qa_questions.txt):
- narrow factoid: drop 'who founded apple computer?'; add
'a bridge between new london & groton?' (single-answer factoid
with named-entity grounding via Gold Star Memorial Bridge)
- broad descriptive: drop 'tell me about the apollo program'
- entity list: drop 'list the founders of microsoft'; add
'name simpsons family members including pets?',
'list of obelisk in connecticut'
- leading/adversarial: add 'whales are endangered from over hunting'
(declarative-shape, tests verifier under premise-seeking phrasing)
- out-of-corpus: rephrase Mars no-such-entity question to
'who is a benevolent dictator for life for mars?' — sharper
adversarial shape since BDFL is a real OSS-governance term
the model is tempted to project onto Mars
- minor edits: 'GNU linux', 'C programming language', 'method man',
'cold fusion breakthrough' phrasing
tests/test_qa_quality_live.py: rename test_mars_prime_minister_*
to test_mars_benevolent_dictator_*; refresh refutation markers
to match the new BDFL phrasing; expand to cover 'no one' /
'no person' negations the model uses on this shape.
Removed:
- test_apple_founders_grounded_and_names_steve_jobs
- test_microsoft_founders_names_gates_and_allen
Added:
- test_veronica_ballestrini_singer_october_birth
The veronica fixture is a stronger probe than the two it replaces:
- entity-disambiguation among 8 same-first-name candidates
(Landers, Guerin, Belmont, Wadley, Gamba, Doran, the
disambiguation page, plus Ballestrini herself in the corpus)
- multi-part question (who is she + when was she born)
- niche personal-bio source (small article, harder to anchor
than well-known commercial-tech bios)
Asserts: grounded + answer mentions Ballestrini + at least one
bio marker (country/singer/songwriter/Connecticut/Italian) + at
least one birth marker (October or 1991). Lands STRICT 1/1 in
JSON mode at the live test.
10/10 fixtures pass.
JSON mode previously presented content-addressed evidence_ids
(``Eed1b6e396``-style) in the EVIDENCE block headers and expected
the same in the model's claim output. Hermes-3-8B was fabricating
plausible near-miss IDs (e.g. ``E1b6e396`` when the runtime had
``Eed1b6e396``) on cross-document relationship questions →
UNKNOWN_EVIDENCE_ID → UNGROUNDED, even when the answer text was
factually correct (e.g. "Homer Simpson's boss is Mr. Burns.").
Switching the prompt-facing surface to short pointer IDs (``E1``,
``E2``, …) — same as claim_lattice_pointer mode — closes the
hallucination loop:
- Pointer IDs are short, enumerable, and fabrication-obvious.
The model can't invent ``E27`` if only ``E1``-``E10`` were shown;
out-of-range IDs read as schema violations at first glance.
- The runtime still resolves each pointer_id to its content-
addressed evidence_id internally and stores THAT in
``evidence_id_pairs`` for the cache & run-DAG. Cache_keys stay
run-stable; only the prompt-facing string changes.
- JSON schema unchanged (``evidence_ids: [str, ...]``), so vLLM
guided_json continues to constrain output shape.
Live-test verification: ``who is homer simpson's boss?`` went from
JSON-mode UNGROUNDED 0/1 (hallucinated ``E1b6e396``) to STRICT 1/1
(model emits ``E1``, resolves cleanly). Homer fixture pin removed —
runs against the JSON default now.
Files touched:
- aborist/qa/evidence.py: render_evidence_block_for_json uses
e.pointer_id instead of e.evidence_id.
- aborist/qa/verify.py: verify_claim_lattice_json switched from
evidence_map_by_evidence_id to evidence_map_by_pointer_id;
captures both pointer_ids (model-emitted) and evidence_ids
(run-stable) in claim_statuses + evidence_id_pairs.
- aborist/qa/{runner,query}.py: claim_lattice_json_system_prompt
+ grounding_reminder describe pointer IDs; example shifts from
``E........`` placeholder to ``E1``.
- tests/test_verify_json.py: stub _ev() takes pointer_id; four
fixtures updated to set it.
- tests/test_qa_quality_live.py: Homer fixture unpinned (now
runs default JSON mode and grounds). Red-fish-blue-fish
fixture pinned to pointer mode — JSON mode hits a separate
token-budget runaway on "plot of X" prose-summary shapes
(~2/3 of samples blow max_tokens with whitespace spam after
the closing brace). Different failure mode, addressed in a
later commit.
460 unit tests + 11 live fixtures pass.
Post-retry / post-trim-and-verify bench showed the picture flipped:
JSON mode now leads on strict-rate (50%) and ties grounded count
(54) with zero errors, all at parity-or-better latency. Switching
the `make query` ANSWER_MODE default from `claim_lattice_pointer`
to `claim_lattice` so the human-facing CLI uses the strongest mode.
Library-level DEFAULT_ANSWER_MODE stays "quote" so unit tests using
StubClient aren't disrupted.
Doc updated with post-retry bench table + revised conclusion.
Live test harness:
- new test_homer_simpson_boss_is_mr_burns fixture (pinned to
pointer mode — JSON mode hallucinates evidence_ids on
cross-document relationships, pointer mode's short numeric
tags can't be fabricated). Documents the mode trade-off.
- test_laura_croft marker net broadened to absorb Hermes
single-sample variance (added "magazine", "video game",
"character", "fictional", "british" markers) plus a hard
"croft" anchor check. Both real entities still gate cleanly.
11/11 live fixtures pass at the new JSON default + pointer pin
on Homer. 460 unit tests + 11 live = 471 green; 10 skipped is
just the live tests in the default-skip path.
Adds test_red_fish_blue_fish_identifies_seuss_book under the new
"retrieval disambiguation" section. The query partially matches
many noise titles (Red Dwarf, Blue whale, Toronto Blue Jays, Red
Sea, Detroit Red Wings, marine aquarium fish list) so it stress-
tests the retrieval+rerank pipeline's ability to land on the right
article — `One Fish Two Fish Red Fish Blue Fish` by Dr. Seuss —
despite the surface noise. Asserts the answer is grounded and
contains a Seuss-specific marker (seuss, rhyming, children's book,
or creatures). 10 tests collected total.
New tests/test_qa_quality_live.py with 9 fixtures gated by
ABORIST_LIVE_TESTS=1 (the `make test-live` target sets it). Each test
runs ONE live query against Hermes + the configured shard set and
asserts:
* audit_mode is at least HYBRID (or UNGROUNDED for the honest-
refusal case)
* key entity tokens appear in the rendered answer
(e.g. "Torvalds" for the linux-kernel question, all four turtle
names + "Splinter" for the TMNT multi-part)
* for the Mars PM no-such-thing case, either UNGROUNDED OR a
refutation phrase ("no prime minister", "does not have", etc.)
— bare affirmative claims of a Mars PM fail the test as
hallucination
Why functional tests matter alongside unit + bench:
- unit tests (test_qa, test_query, test_claim_lattice) validate
plumbing with StubClient — can't tell you "did the model
actually answer correctly?"
- bench (`make bench-qa`) measures aggregate strict-rate /
grounded counts across many questions but doesn't assert
specific content
- these live fixtures sit between: each test is a named gate
around one known-good answer. When a future change improves
things, the bench number climbs AND every fixture passes (or
gets stricter assertions). When something regresses, the bench
number falls AND specific fixtures fail by name, telling you
where the regression landed. "Benchmax" rationale: the bench
is the scoreboard, the fixtures are the gates that translate
quality drift into named test failures.
Default `make test` is unaffected — 460 passed + 10 skipped (was 1).
Three layered fixes for the JP-dinosaurs lazy-anchor failure mode where
"what dinosaurs were in the first jurassic park film?" returned HYBRID
5/8 with citations to a video-game tie-in chunk (Operation Genesis):
1. Bare-name claim guard. New policy field
`claim_lattice_min_claim_content_tokens` (default 2). Claims with
fewer content tokens (>=4 chars, post-spotlight-stopword) classify
as SCHEMA_INVALID. "Triceratops. [E16]" had 1 content token; lexical
overlap on a single token is too thin to verify "X is in the FIRST
film" semantics. Forcing sentence shape raises the coverage bar so
off-topic chunks can no longer satisfy the citation. Threshold of 2
keeps narrow factoids viable (e.g. "Steve Jobs co-founded Apple"
tokenizes to 3 content tokens after stopword strip).
2. Smell → demote. The lazy-anchor smell sidecar was advisory only;
the verdict could be STRICT while every claim cited the same
magnet chunk. New policy fields
`claim_lattice_lazy_anchor_demote_threshold` (0.5) and
`claim_lattice_lazy_anchor_demote_min_pairs` (3) cap audit_mode at
HYBRID when ratio + pair-count both met. STRICT now requires
diverse anchoring across pointers; emits a LAZY_ANCHOR_DEMOTE
violation record so the auditor can see what fired.
3. Noisy title markers extended. `_NOISY_TITLE_MARKERS` now includes
"operation genesis", "the game", "video games" so spinoff/tie-in
articles (Jurassic Park: Operation Genesis, X (NES game), etc.)
classify as noisy_background_source. Default
claim_lattice_allowed_source_roles excludes that role, so noisy
chunks never enter the evidence map for pointer-mode queries.
Brittle (case-by-case markers) but addresses the JP failure
directly.
Live JP-dinosaurs after fix: UNGROUNDED 0/11 (was HYBRID 5/8 with
bogus citations). Honest refusal beats false confidence — model could
recover by writing sentence-shape claims; bare-name shortcut blocked.
Test fixtures using minimal placeholder claims ("Trex appears. [E1]")
opt out of the bare-name check via `min_claim_content_tokens=0` since
those tests exercise orthogonal behaviors. 460 tests pass.
JSON mode (answer_mode="claim_lattice") is now reachable end-to-end via
runner.ask: message build, optional vLLM guided_json extra_body, verifier
dispatch, and providence_cache persistence. Verifier returns
verifier_method="claim_lattice" (same as pointer variant) so both modes
share the existing CHECK constraint; answer_mode on the run-DAG and
json_fixups on the verdict disambiguate.
Pointer verifier now splits EVIDENCE_LINKED_PARTIAL into its own
partially_verified_quotes bucket so a claim never renders as both a
verified bullet and an unverified footer. CLI shows a "partially
grounded" section between the verified body and the unverified list.
Removes the strict no-double-quote check (`MANUAL_QUOTE_VIOLATION`) from
verify_claim_lattice. The rule was rejecting factually correct,
source-grounded claims for cosmetic punctuation: Hermes-3-8B paraphrases
prose but copies named-quoted phrases verbatim from source (e.g.
`"Constitution State"` lifted from a Connecticut chunk). Pre-fix,
`make query Q="tell me about connecticut"` reported `0/13 verified` on
a paragraph where every claim was correct and source-supported, just
because the model preserved source quote marks.
The two checks that remain handle the cases the manual_quote rule was
nominally meant to catch:
- claim_lattice_min_citation_coverage (Rule 5, default 0.30)
catches lazy-anchor citations whose only overlap with the cited
span is a single topical token
- claim_lattice_max_pointers_per_claim (Rule 6, default 2)
catches the encyclopedic-mega-claim where the model emits
`[E1,E2,...,E26]` after one sentence
Bench n=3 across 8 fixed questions vs pre-G0-hardening baseline (also
n=3): pointer-mode grounded count (STRICT+HYBRID) goes 15 → 21,
UNGROUNDED 9 → 3, broad-descriptive failures (connecticut, python)
fully cured. STRICT count goes 12 → 9 because previously-bogus STRICTs
(microsoft cited transit chunks that just shared one token, supermans
lazy-anchor magnet at 32/32) are now honestly reclassified to HYBRID.
Side changes:
- Ports max_pointers / min_citation_coverage policy fields and
verify_claim_lattice call into runner.py so single-document `ask`
matches multi-source `query` semantics (query.py already had them
via b39e79b).
- Worked Example 2 (broad-descriptive) added to pointer-mode prompt
so "tell me about X" has a template, plus Rules 7 (anti-echo) and
8 (max-2-pointers).
- Two manual_quote tests in tests/test_claim_lattice.py inverted to
document the new behavior (quotes-in-claim no longer block).
- `_has_manual_quote` retained — still used by verify_claim_lattice_json.
459 tests pass.
Brings the JSON variant back as a third ANSWER_MODE, distinct from the
pointer variant. The substrate exposes both: pointer for prose-
distribution / small-model paths (Hermes-3 8B), JSON for grammar-
constrained / large-reasoning-model paths (vLLM guided_json,
Claude/GPT-4 native JSON, Qwen 3.6 reasoner). Agents pick by setting
`policy["answer_mode"]`; both fold into governance_policy_hash so
records under different modes never alias.
Components:
- aborist/qa/verify.py:
* ANSWER_MODES = ("quote", "claim_lattice_pointer", "claim_lattice")
* `_lenient_json_parse(raw)` — defensive pre-parser. Strips markdown
fences, trims preamble/suffix to {/}, normalizes curly quotes,
fixes trailing commas. Returns (parsed, fixups[]) so the verifier
logs which drift had to be peeled. Lenient on syntax, strict on
semantics: parsed JSON still has to schema-check.
* CLAIM_LATTICE_JSON_SCHEMA — JSON Schema for the {"claims":[...]}
shape. Used by vLLM guided_json sampling-time constraint.
* `verify_claim_lattice_json(answer_json_text, evidence_map, ...)` —
runs the same hard checks as verify_claim_lattice (evidence_id
resolves, source_role allowed, no manual quotes, claim text non-
empty, claim textually overlaps evidence) but on content-
addressed evidence_ids directly. Returns the same verdict shape
plus a `json_fixups` list.
- aborist/qa/client.py: ChatClient Protocol & OpenAICompatibleClient
gain optional `extra_body` kwarg. Forwarded as additional fields in
the JSON request payload — opaque pass-through for vLLM-specific
knobs like `guided_json`. Endpoints that don't recognize a key
silently drop it. StubClient ignores; tests inspect via self.calls.
- aborist/qa/evidence.py: `render_evidence_block_for_json` and
`render_evidence_map_for_json` — JSON-mode prompts label blocks with
the content-addressed evidence_id (long hex) since that's what the
model cites in its JSON. Pointer mode keeps using the short
pointer_id.
- aborist/qa/query.py:
* Imports the JSON verifier + schema + JSON-mode evidence renderer.
* Message-build branch: `elif answer_mode == "claim_lattice"`
builds the same per-chunk evidence map as pointer mode, but
uses `claim_lattice_json_system_prompt` and labels blocks with
evidence_id.
* LLM call: when answer_mode=claim_lattice and policy
`claim_lattice_use_guided_json` is on, passes
`extra_body={"guided_json": SCHEMA}` so vLLM constrains output.
* Verifier dispatch: new `elif answer_mode == "claim_lattice"`
branch calls verify_claim_lattice_json.
* DAG persistence: lattice-mode raw_answer / parsed_lattice /
rendered_text threading now applies to both pointer and JSON.
* DEFAULT_QUERY_POLICY adds `claim_lattice_json_system_prompt`,
`claim_lattice_json_grounding_reminder`, and
`claim_lattice_use_guided_json` (default True).
- tests/test_verify_json.py: 14 tests covering the lenient parser
(strict pass-through, fence strip, preamble trim, curly-quote
normalize, trailing-comma fix, multi-fixup, hard-fail) and the
JSON verifier (STRICT on resolved claims, HYBRID on partial,
UNGROUNDED on schema invalid, fence recovery, manual-quote
violation, source-role block).
The whitepaper rewrite to 13.9.1 (substrate exposes both modalities,
both first-class) becomes accurate post-ship — JSON mode now exists
in code as it always existed in the architecture's intent.
554 tests pass.
Two layered fixes that close the lazy-anchor gap. Title-purity
moves the right SOURCES to the top of retrieval; the new hard
check makes the verifier ruthlessly honest about whether each
cited chunk actually mentions what its claim says.
1. _rerank_by_title_purity. New stage at the end of _rerank().
purity = |title_tokens ∩ query_tokens| / |title_tokens|; score
multiplied by (1 + 2*purity), so a title that IS the topic
("Jurassic Park (film)" against the dinosaur question) gets
3.0x and titles that drag in off-topic tokens like
"Jurassic Park: Operation Genesis" only get 2.0x. Composes
with the role-rank weights from the prior commit. Florida
query: list-pages dropped out of top-8 entirely. JP-dinosaurs
query: Operation Genesis sank from #4 to #8.
2. CITATION_MISMATCH hard check. Sixth deterministic check in
verify_claim_lattice — for each (claim, pointer) pair, at
least one content token from the claim text must textually
appear in the cited evidence span. Lexical only, no NER, no
embeddings; same _content_tokens extractor as the spotlight
so the proof-path stays on the soft/hard boundary. Catches
the magnet-chunk lazy-anchor where the model cites a chunk
whose text contains zero overlap with what its claim asserts.
New status CITATION_MISMATCH joins the per-claim taxonomy.
New violation kind CITATION_MISMATCH joins the run-DAG verify
payload. JP-dinosaurs verdict went from "13/14 verified
ratio 0.77 (cite E13 for everything)" pre-check to "1/14
verified" post-check — only Tyrannosaurus Rex's claim
actually overlapped its cited chunk. The architecture
stopped lying about evidence-linkage that the cited spans
don't textually support.
Pure-stopword claims pass the overlap check vacuously (no
content tokens means nothing to check against) — other hard
checks own that case.
Tests: +3 covering CITATION_MISMATCH (claim-token absent yields
fail, partial overlap yields HYBRID with EVIDENCE_LINKED_PARTIAL,
pure-stopword claim passes vacuously). Updated 7 prior tests
that used colloquial stub claims ("Trex") which don't textually
appear in the sample spans ("Tyrannosaurus rex"); they now use
claim text that overlaps the source content. 445 tests passing,
all 7 production shards report 0 chain breaks.
verify_claim_lattice now derives pointer_id_distribution and
lazy_anchor_ratio from the verified-or-partial claim_statuses and
returns them in the verdict. Surfaced through ask()/query() result
dicts; CLI human-render displays a one-liner when ratio >= 0.5
(threshold for "model is lazy-anchoring").
Sidecar discipline preserved: never persisted in providence_cache,
never threaded into build_run_dag's verify_payload (test pins this
invariant — sidecar values must not enter run_dag_root). The
distribution is recoverable from claim_statuses which IS persisted,
so future sidecar tools can recompute on demand without enlarging
the audit chain.
JP dinosaurs benchmark currently triggers the warning at 11/11
ratio 1.00 — Hermes-3-8B cites [E1] for every claim even with
G0.3's relevance ranking putting the dinosaur-mention chunk near
the top. The render-layer signal now makes that lazy-anchor
behavior explicit instead of leaving it for the auditor to spot
from the spotlight excerpts.
Tests: +4 covering the ratio across distributions (single-anchor /
diversified / no-verified-pairs) plus the run_dag isolation
invariant. Total 442 passing.
New policy["answer_mode"] = "claim_lattice_pointer" (default "quote").
The runtime builds an evidence map with two-layer ids — pointer-id
(E1, E2, ...) shown to the model, sha256-derived evidence-id used by
the cache & run-DAG — and the model emits pointer-line prose
("Claim. [E12]") referencing them. Renderer interpolates literal
source spans at display time. Synthetic-elision-by-construction-
impossible: the model never types the quote string.
Pieces:
- aborist/qa/evidence.py (new): EvidenceObject + spotlight excerpt
(claim-token-centered window into the cited span, falls back to
leading window when no token matches).
- aborist/qa/parse_claims.py (new): pointer-line parser walks
lines, pulls [E\d+] / [E\d+,E\d+,...] tags, returns ParsedClaim
with PARSED / NO_EVIDENCE_POINTER status. Strict regex refuses
fuzzy alternatives so honest UNGROUNDED beats lax acceptance.
- aborist/qa/verify.py: verify_claim_lattice. Hard checks only —
parser succeeded, evidence_id resolves, source_role allowed, no
manual quotes (any " char violates), claim text non-empty.
Returns evidence_id_pairs (content-addressed, run-stable) for the
run-DAG. Soft signals (entailment, completeness, predicate
compatibility) stay sidecar.
- aborist/qa/dag.py: 9-stage CTI shape when evidence_map_root is
supplied — question / retrieval / evidence_map / prompt /
raw_answer / parsed_claim_lattice / verify / render /
final_label. Quote mode keeps the original 7-stage shape so
pre-G0 run_dag_root values stay valid.
- aborist/qa/query.py: per-chunk evidence-map build with role-
weighted budget AND query-relevance ordering. Within each source,
chunks are ranked by (distinct_query_tokens_present,
total_mentions, doc_order_asc) so the chunk most likely to
textually support the question gets the lowest pointer id. Without
this re-rank Hermes-3-8B lazy-anchored on doc-order-first chunks
regardless of relevance.
- aborist/qa/runner.py: same answer_mode branch for the single-doc
ask() path. No iterative repair in pointer mode (one-shot
benchmark discipline).
- aborist/store.py: verifier_method CHECK extended with
'claim_lattice'. New _rebuild_providence_cache_claim_lattice
migration preserves run_dag_root / run_dag_blob across the
rebuild — older rebuilds dropped them.
- aborist/cli.py: --answer-mode {quote,claim_lattice_pointer} on
query and ask.
- Makefile: ANSWER_MODE knob; defaults to claim_lattice_pointer for
`make query` so the testing harness exercises the new path.
Library DEFAULT_POLICY / DEFAULT_QUERY_POLICY stay "quote" so
Python callers and unit tests aren't surprised.
Prompt: one-shot worked example (Apple founders) plus strict
no-quotes / no-JSON / no-markdown / plain-prose-only rules. Without
the worked example Hermes-3-8B drops the bracket protocol on roughly
half of runs; with it, JP dinosaurs benchmark went 0/13 -> 10/11 ->
16/17 verified across the iterations that hardened the pipeline.
Tests: +48 covering parser, two-layer ids, verifier failure modes
(UNKNOWN_EVIDENCE_ID / SOURCE_ROLE_BLOCKED / MANUAL_QUOTE_VIOLATION
/ NO_EVIDENCE_POINTER / SCHEMA_INVALID), spotlight rendering with
buried-term fixture, per-chunk evidence map, query-relevance
ordering, schema migration round-trip, 9-stage DAG shape divergence.
Total 438 passed, all 7 production shards report 0 chain breaks.
Known limitation: chunk boundaries can cut wikitext mid-template, so
mwparserfromhell-backed to_base() leaves orphan </ref> tags and
leading list markers in the visible spotlight excerpts. Verifier and
CTI architecture are unaffected; the leak is cosmetic. Proper fixes
are template-aware chunking (chunker bump invalidates prior records)
or an orphan-marker post-strip in aborist/wikitext.py — both out of
G0 scope.
Two enhancements continuing the toy-Hermes design pass:
#1 Chain-segment failure localization
aborist/qa/dag.py: `localize_failure(audit_mode, n_sources, n_quotes,
n_verified)` maps a non-STRICT verdict to the pipeline stage that
introduced the failure:
retrieval no admitted sources (gate over-rejected, or corpus
genuinely lacks the topic) → ingest more / relax breadth
context sources retrieved but no quotes extracted (per-source
cap dropped relevant content, or model declined to cite)
→ raise cap / tighten prompt
answer quotes extracted but didn't all verify (model fabricated,
paraphrased inside quotes, appended citations) → mechanical
+ re-prompt repair targets exactly this case
`failure_stage` lands on the run_dag's verify node payload AND on the
result dict so an operator can read the reason at a glance — `failure_stage='answer'`
means stop tuning the verifier & fix the model behavior. Debugging
becomes typed instead of vague.
#2 Re-prompt repair (second tier of the hybrid loop)
aborist/qa/repair.py: `reprompt_repair(...)` builds a feedback message
naming the failed quotes & asks the model to rewrite using only
verbatim citations. Hard rule: only fires when
`policy["repair_max_reprompts"] > 0` (default 0); caller enforces the
cap by looping at most that many times.
aborist/qa/query.py + aborist/qa/runner.py: after mechanical repair,
if the answer is still HYBRID/UNGROUNDED with unverified quotes,
loop up to `repair_max_reprompts` times. Each iteration: build
feedback (assistant turn with current answer + user turn with failed
spans), call LLM, verify. Accept the new answer if `n_verified`
strictly improved; otherwise break (the model's not converging,
don't waste cycles).
The mechanical + re-prompt combination handles the cases each tier
declines individually:
mechanical alone: synthetic_elision split, trailing_artifact trim,
no_overlap remove
+ re-prompt: paraphrase, partial_paraphrase, interior_elision
needing semantic judgment, fabrications the
model can recognize when shown its own quote
`repair_max_reprompts` lives in DEFAULT_QUERY_POLICY +
DEFAULT_POLICY so it folds into governance_policy_hash. Default 0
preserves single-shot semantics for callers that don't opt in. Each
re-prompt iteration adds a `{action: reprompt_rewrite, diagnosis:
model_feedback_loop}` entry to repair_changes; audit chain captures
the full transition through the existing providence_repair event.
Tests:
- dag: localize_failure across all four cases (STRICT, retrieval,
context, answer); failure_stage embedded in run_dag verify node.
- repair: stub client with sequenced answers (failing first, clean
on re-prompt) — assert two LLM calls, STRICT verdict, reprompt_rewrite
in the change log.
484 tests pass (dag +5, repair +1). The pre-existing test_burn flake
under full-suite ordering remains; passes in isolation.
The system was observational: verifier classified, sidecar diagnosed,
repair plans were emitted — but no loop ever closed. This adds the
hybrid repair stage from the toy-Hermes design pass: mechanical first
(deterministic string substitution from sidecar suggestions), behind
a `policy["repair_enabled"]` flag (off by default). Re-prompt fallback
is TODO.
aborist/qa/repair.py: `mechanical_repair(answer, unverified_quotes,
context)` walks each unverified quote through the sidecar classifier &
applies its `repair` action by string sub:
synthetic_elision_inside_quote (both halves verbatim)
`"prefix [...] suffix"` → `"prefix" ... "suffix"`
Two verbatim spans the verifier can independently check; the
model's [...] ellipsis-marker becomes prose between them.
trailing_artifact
`"prose. (Source: ...)"` → `"prose."`
Verbatim prefix kept; model-appended tail dropped.
no_overlap
Drop the line containing the bad quote entirely.
Skips include_aside_for_verbatim (needs precise source-span extraction;
defer to re-prompt path), paraphrase / partial_paraphrase (need prose
rewriting). Idempotent.
aborist/qa/query.py + aborist/qa/runner.py: optional pass after first
verify. When `repair_enabled=True` AND `audit_mode != "STRICT"` AND
unverified quotes exist:
1. Run mechanical_repair on the answer text.
2. If repair produced any changes, re-verify the repaired text.
3. If post-repair verdict isn't worse (n_verified didn't decrease),
accept the repair: persist the REPAIRED answer text instead of
the model's original. Cache_key inputs unchanged.
4. Audit chain gets one `providence_repair` event with the change
log + pre/post verdict so the original→repaired transition is
reconstructable.
Result dict gains `repair_changes` (list of change records) and
`pre_repair_audit_mode` (what the original was classified as).
`policy["repair_enabled"]` enters governance_policy_hash so on/off
agents share no cache silos.
Tests:
- mechanical_repair on each diagnosis (synthetic_elision, trailing_artifact,
no_overlap), idempotence on clean text.
- query() integration: repair_enabled=False (default) leaves answer
text unchanged; repair_enabled=True promotes a HYBRID/quote
synthetic_elision case to STRICT/quote, persists the repaired text,
emits the providence_repair audit event.
479 tests pass (+6 repair).
Each query/ask call now emits a 7-stage Merkle-DAG fingerprint stored
alongside the providence record. F from the toy-Hermes design pass.
Stages, in order:
question hash of question_hash (the 8-dim cache_key dim)
retrieval hash of sources summary (document_roots + roles +
scores + chunk_idx) — captures which docs ranked
context context_root (the source-Merkle for the assembly)
prompt conversation_hash
answer sha256(answer_text)
verify hash of verdict (audit_mode, verifier_method,
n_quotes, n_verified, claim_statuses)
final_label hash of (audit_mode, verifier_method, lookup_path)
run_dag_root = MerkleTree over those stage hashes (aborist conventions:
non-commutative HashCombine 0x03, leaf prefix 0x00, self-dup odd rule).
run_dag_blob = canonical JSON of {root, nodes} so an auditor can
recompute & verify (`verify_run_dag(blob)` returns True/False).
The DAG is NOT in cache_key. cache_key inputs determine the answer; the
answer determines the DAG — folding it back would create a cycle.
Instead it rides alongside as a per-record computation fingerprint.
Distinct from the linear `audit_events` chain (which tracks DB-wide
state changes); this is per-run computation provenance.
Schema: ALTER TABLE providence_cache ADD COLUMN run_dag_root TEXT;
ADD COLUMN run_dag_blob TEXT;
Idempotent migration in `_migrate_audit_mode`. Both rebuild templates
(VISUAL→UNGROUNDED dance, paraphrase verifier_method dance) updated to
include the new columns. Legacy records pre-2026-04-30 carry NULL.
Result dict gains `run_dag_root` so callers can verify without a DB
round-trip.
Tests:
- test_dag.py (9 tests): determinism, reactivity to each stage's input,
fixed stage order, round-trip verify, tamper-detection, JSON-string
acceptance.
- test_query.py: persistence on record + result, verify_run_dag round-
trip on the persisted blob.
473 tests pass (DAG +9, query +1, integration unchanged).
Two paired enhancements (D + E from the toy-Hermes design pass):
D. Per-claim status taxonomy on `verify_quotes`.
New `claim_statuses` field on every verdict — a per-evidence-unit
list with three labels:
VERIFIED_QUOTE unit substring-matched in normalized context
(any of quote/span/entity strategies)
SUPPORTED_PARAPHRASE unit cleared the paraphrase token-coverage
threshold (≥85% topical tokens present)
UNSUPPORTED unit didn't match anything
Diagnostic labels (QUOTE_INTEGRITY_FAILED, SOURCE_MISMATCH,
FALSIFIED) stay in the sidecar / falsification machinery — the
binary-verifier discipline holds. Empty list when no evidence
was extracted at all (verifier_method='none'). Backward-compatible:
existing fields (audit_mode, n_quotes, n_verified, unverified_quotes,
verifier_method) unchanged; current callers ignore the new field.
E. Repair-action plans on sidecar diagnoses.
Each `_classify_span` diagnosis now carries a `repair` field with
a concrete suggestion the operator can act on:
synthetic_elision_inside_quote → split_into_two_quotes
(when both halves verbatim)
→ trim_to_verified_half
(when only one half verbatim)
→ remove_claim
interior_elision → include_aside_for_verbatim
(with the dropped aside text)
trailing_artifact → trim_trailing_artifact
(with the kept_prefix string)
paraphrase → downgrade_to_paraphrase
partial_paraphrase → split_or_remove
no_overlap → remove_claim
Read-only suggestions — sidecar still doesn't write to providence_cache
or audit_events. The repair stage is recommendation, not mutation.
Operator (or an automated repair pass) decides whether to act.
Human render in `aborist inspect` shows `repair: <action> (<reason>)`
under each diagnosis line.
Tests:
- verify: claim_statuses_quote_path_labels_each_unit (per-quote VERIFIED
/ UNSUPPORTED), claim_statuses_paraphrase_method_flagged,
claim_statuses_empty_when_no_evidence.
- inspect: repair_synthetic_elision_split_when_both_halves_verbatim,
repair_interior_elision_includes_aside, repair_trailing_artifact_trim,
repair_no_overlap_remove.
462 tests pass (verify +3, inspect +4).
Two enhancements from the toy-Hermes design pass (2026-04-30):
A. Synthetic-elision-inside-quote diagnosis (sidecar only).
Distinct from interior_elision (model dropped a `(...)` aside source
carries) — synthetic_elision is the model writing literal `[...]`
between fragments of a `"..."` span, signaling self-elision while
claiming verbatim citation. The verifier still rejects (binary
discipline holds), but `aborist inspect` now reports
`diagnosis: synthetic_elision_inside_quote` with prefix/suffix
presence flags so an operator can judge whether the elided middle
was benign. Probe runs first in the classify-span chain (more
specific than trailing_artifact / interior_elision / paraphrase).
Catches the Brachiosaurus case: `"The film centers on the fictional
Isla Nublar [...] Universal Studios..."` — both halves are in source,
but the literal `[...]` isn't, so substring match correctly fails &
the sidecar tells the operator why.
C. Source-role classification + role-weighted context budget.
`_classify_source_role(title, qtokens_stem)` tags each top-K hit:
primary_answer_source 2.0× cap strong title-stem overlap
secondary_context_source 1.0× cap "list of", "characters",
"franchise", "history of"
noisy_background_source 0.5× cap "score", "music",
"video game", "merchandise"
sequel_background_source 0.5× cap "lost world", roman numerals
background_source 1.0× cap default
Order matters: noisy/sequel/secondary markers fire before the
primary check so peripheral pages with strong title overlap (e.g.
`Jurassic Park (film score)` shares 3 stems with the JP-film query)
don't claim a primary slot.
Cap loop now applies role weight on top of the baseline
`max_context_chars / top_k`. Total context still bounded by the
running `char_budget` — weights just shift how the budget gets
divided so primary pages get more text & noisy pages less, fixing
the case where a `(film score)` page consumed a primary slot.
`source_role` is persisted on `_Hit` and surfaces on
`merkle_proof.sources[*].source_role` in the providence record so
inspect & audits can see which slot each source occupied.
Tests:
- inspect: synthetic_elision_caught (Brachiosaurus regression),
synthetic_elision_does_not_fire_when_source_has_brackets (false-
positive guard).
- query: role classifier matrix (primary / secondary / noisy / sequel /
background) on JP-film-style titles, role persistence in sources list.
455 tests pass (sidecar +2, query +2).
Per fox: while iterating on retrieval/verifier knobs, a flag on the
query itself is more useful than a separate verb. One step reset:
change a knob, re-query, see fresh result.
Surface:
aborist query --burn ...
make query Q="..." BURN=1
make query-dry Q="..." BURN=1 (works on dry-run too)
Behavior in `aborist.qa.query.query()`:
- New `burn_existing: bool = False` parameter on query().
- After computing primary cache_key (per the active dedup mode) but
BEFORE the cache lookup, if burn_existing: DELETE the live row that
matches the primary cache_key & write one providence_burn audit
event with reason "query --burn (test-ergonomic mid-query bust)".
- The lookup then misses → fresh inference runs. Result reports
`burned_existing: 0|1` so the caller sees whether anything got
busted.
- The equivalence-class fallback cache_key is deliberately NOT
touched: prior alt-mode records stay as historic witnesses.
Tests (3): cache populates → cache hits → BURN=1 forces fresh
inference; --burn writes a providence_burn audit event; first-time
query with --burn is a clean no-op (burned_existing=0). 356 passed,
1 skipped.
Complementary to `aborist burn-kindergarten` (mass reset) — this is
the surgical version. Both are local-only by design; for cross-peer
invalidation use `make falsify` and let mesh sync broadcast.
Per fox: useful while iterating on retrieval/verifier knobs to wipe
recent test runs without finding each cache_key. Mirrors the mesh-sync
kindergarten window so what's still un-broadcast (private to this peer)
is exactly what's safe to bust without confusing peers.
Surface:
aborist burn-kindergarten [--kindergarten-seconds N]
[--reason '...'] [--by-actor X]
[--force] [--dry-run] [--verbose N]
make burn-kindergarten [SECONDS=3600] [FORCE=1] [DRY_RUN=1] [REASON='why']
Behavior:
- Walks every shard, finds providence_cache rows with
created_at >= now - SECONDS and falsification_state='live'.
- Each row goes through the existing _burn_cache_key — children gate
honored unless --force, audit event written per successful burn,
chain integrity preserved.
- Result JSON reports examined / burned / refused_has_children /
not_found counts plus a verbose tail of per-row results.
- --dry-run reports without writing or auditing.
- --kindergarten-seconds 0 = burn every live row (test reset).
Tests (4): selective burn (old kept, fresh burned), dry-run writes
nothing, 0-second window burns everything, children-gate refusal
without --force. 353 passed, 1 skipped.
Operational note: this command does NOT propagate to peers (burn is
local kindergarten cleanup by design, see docs/mesh.md). If you want
the remote effect, falsify each record individually & let mesh sync
broadcast the falsifications instead.
Fox 2026-04-29: 'who is supermans girlfriend?' returned 7-of-8 unrelated
`Girlfriends`-titled articles (TV show, movies, songs); only 1 actual
Superman-related doc made it. Two coupled defects:
1. Title-overlap accept fired on ANY single-token match. A 2-token
query was admitting docs that shared only ONE qtoken. `Girlfriends`
passed because its title matched "girlfriend" even though no
"superman" anywhere in the doc.
2. `_body_density_passes` required at least HALF the qtokens — too
lenient for the 2-token case (1 of 2 = 50% = pass).
3. Possessive plural mismatch: question_hash strips apostrophes so
"superman's" → "supermans", but the corpus has bare "Superman".
`body.count("supermans")` returns 0 even on the canonical doc.
Three coordinated fixes in `aborist/qa/query.py`:
- New `_stem_token_for_match(t)` helper: strips a trailing `s` for
tokens > 4 chars (so `supermans` → `superman`, `girlfriends` →
`girlfriend`). Conservative: skips short tokens & double-s endings
to avoid `class` / `boss` / `pass` corruption.
- `_body_density_passes` now uses `_body_count_with_stem` (literal-
first, stem-fallback) AND a tightened breadth threshold:
≤ 2 tokens require ALL of them
3+ tokens require N - 1 (allow one weak signal token to miss)
- `_filter_by_title_relevance` mirrors the same breadth threshold for
title-overlap. Synonym fallback (any-match against synonym_expand)
is preserved ONLY for 1-token queries — otherwise a stray synonym
hit (e.g. AMD synonym matching an Intel-only title) over-recalls.
End-to-end on the actual corpus (3.4M articles, 8 shards):
before: 'supermans girlfriend' → 7 unrelated `Girlfriends` titles, 1 Superman doc
after: Lois_Lane in top-K (the canonical answer), other Superman-related
docs alongside, Girlfriends-only-titled docs filtered out
Tests:
- test_query_filter_requires_breadth_for_multi_token_queries — synthetic
3-doc corpus pinning the new behavior (Lois Lane keeps; girlfriends-tv
& superman-music both drop).
- test_query_filter_one_token_query_still_synonym_expands — pin the
1-token loose path is preserved (no over-tightening).
349 passed, 1 skipped.
Burned the two stale UNGROUNDED girlfriend-query cache records so a
fresh `make query` exercises the new filter end-to-end.
Different agents have different value functions on the same providence
records. Today aborist bakes one canonicalization policy into
governance_policy_hash and calls it universal. This adds two knobs that
let agents express their preference without breaking provenance.
- aborist/qa/keys.py: `canonical_question(q, mode=...)` and
`question_hash(q, mode=...)` accept "strict" (NFC + ws-collapse only;
every variant gets its own hash) or "equivalence_class" (default —
additionally lowercase + trailing-punct strip + article strip).
New constants: QUESTION_DEDUP_MODES, FIDELITY_MODES, defaults.
- aborist/qa/query.py + aborist/qa/runner.py: each call computes the
primary cache_key under policy["question_dedup"]. New `fidelity`
parameter:
"strict" only primary cache_key checked
"equivalence_class" primary first; if miss AND alternate mode
produces a different cache_key, try alternate
Cross-silo fallback works because _ckey_for_mode rewrites
policy["question_dedup"] to the alternate mode before computing
governance_policy_hash — so the fallback ckey matches what an agent
under that mode would have written. Lookups can find each other's
records when fidelity permits. Result dict gains `lookup_path` ∈
{"strict", "equivalence_class", "strict_fallback",
"equivalence_class_fallback", "miss"}.
- aborist/cli.py: `--question-dedup` and `--fidelity` flags on the
`query` subcommand. Human render annotates cache_hit lines that
came from a fallback ckey ("cached via equivalence_class_fallback").
- tests/test_query.py: three new regressions
* strict-policy + strict-fidelity: variants get distinct cache_keys
* cross-silo fallback: strict-policy reads eq-class-policy records
via fidelity=equivalence_class
* strict-fidelity refuses fallback (audit-grade): cache miss even
when the alternate silo has a hit
- CLAUDE.md: dedup-mode and fidelity convention bullets updated.
This is V1 + V2 of the substrate move sketched in conversation: write
policy determines which silo a record lives in; read fidelity determines
how loosely an agent walks across silos. Records stay exactly-keyed
(provenance hard); routing is per-agent (preference soft). Same
verifier_no_diagnostics discipline holds — no soft signal enters the
hard chain.
442 tests pass.
Fox 2026-04-29 (Clark Kent / Superman query): the verifier flagged a
quoted span as unverified and the sidecar diagnosed it as 'paraphrase'
with token_coverage=1.0. Investigation showed the source actually said
`Clark Joseph Kent (middle name is also Jerome according to some
versions) is a fictional character...` and the model elided the
parenthetical aside, quoting `Clark Joseph Kent is a fictional
character...`. Every word in source, but the sequence has a 60-char
gap where the aside was. Distinct failure mode from real paraphrase
(token reordering) and trailing_artifact (model APPENDS).
Sidecar refinement, no verifier change. The binary verifier discipline
(memory rule `feedback_verifier_no_diagnostics`) holds — quote strategy
is still verbatim-only; soft signal stays in the inspect verb.
Algorithm walks every `(` in base. For each open paren at position P:
the longest k where base[:P] ends with span[:k] is the model's prefix;
the span tail (≥20 chars) must then match the source after the close
paren. First paren that satisfies both checks wins.
CLI human-render shows `matched: N prefix + M suffix chars
(parenthetical aside dropped)` plus the dropped_aside text so an
operator can decide at a glance whether the elision is benign.
Live record now reads:
[1] interior_elision
matched: 17 prefix + 138 suffix chars (parenthetical aside dropped)
dropped_aside: middle name is also jerome according to some versions
439 tests pass (sidecar +2, full suite still green).
Fox 2026-04-29: "part of the gossip protocol should be some default
delay that allows a user to catch and burn kindergarteners before
they are synced." Currently `mesh sync` enumerates the most-recent
N items with no age filter — a doc ingested 30 seconds ago goes
out on the next sync, & if a peer ingests it before the operator
notices a problem, burn no longer suffices (peer has its own copy).
Adds a sender-side kindergarten window:
--kindergarten-seconds N default 3600 (1 hour)
Records younger than `now - N` are held back from broadcast. Both
ANNOUNCE_ROOT (filtered on documents.ingest_ts) & ANNOUNCE_FALSIFICATION
(filtered on falsifications.at). N=0 = broadcast everything (cron-
friendly opt-out for operators preferring immediate propagation).
Result JSON now reports the held counts so the operator can see
what the window protected:
kindergarten_seconds: 3600
announced_roots: 12
kindergarten_held_roots: 3
announced_falsifications: 0
kindergarten_held_falsifications: 1
Sender-side discipline only — the receiver has no view into when
the sender created the record, so it can't enforce. Adding a
created_at on the envelope would let receivers reject too-fresh
gossip, but that's a future protocol bump (envelopes today don't
carry sender wall-clock; ts is the send time).
Tests:
- existing tests now pass `--kindergarten-seconds 0` so freshly-
ingested fixtures broadcast immediately for the test
- new test_sync_kindergarten_holds_fresh_records: a 30-second-old
doc is held; an artificially-aged doc broadcasts. announced=1,
held>=1.
- new test_sync_kindergarten_zero_broadcasts_everything: explicit
opt-out works.
342 passed, 1 skipped.
Fox caught on 2026-04-29: 'who is batman?' and 'who is the batman?' produced
different cache_keys despite question_hash collapsing both variants. The 8-dim
cache_key has TWO seams that touch question text — question_hash AND
conversation_hash (which hashes the messages list with the literal question
in the user turn). Only the former canonicalized; the latter took bytes as
written, so each variant got its own chash and missed cache.
Fix: decouple the two forms.
- aborist/qa/keys.py: extract canonical_question(text) → str; question_hash
composes _sha256(canonical_question(t)) so the equivalence class is
defined in one place.
- aborist/qa/query.py + aborist/qa/runner.py: build TWO message lists per
call. `messages` carries the verbatim question (Hermes sees the user's
natural phrasing — no grammar drift). `canonical_messages` substitutes
the canonical form, and conversation_hash hashes that. The LLM still
gets 'Who Is THE Batman?'; the cache_key collapses to 'who is batman'.
- tests/test_query.py: regression — three variants (with/without `?`,
with/without `the`) hit the same cache_key. First call populates,
later variants cache_hit. Captures stub messages to confirm the LLM
saw the verbatim question, not the canonical form.
437 tests pass.
Per fox 2026-04-29: the wire protocol has had ANNOUNCE_FALSIFICATION
since the foundation commit (and MeshWireClient.announce_falsification
since AEAD landed), but the user-facing `mesh sync` only ever fired
ANNOUNCE_ROOT. Falsifications local to one peer never reached others
unless an operator hand-rolled a Python script.
Now `mesh sync` enumerates BOTH categories:
- ANNOUNCE_ROOT most-recent --limit documents (existing path)
- ANNOUNCE_FALSIFICATION most-recent --limit falsifications (new)
Receivers verify Ed25519 sig + per-peer chain-of-claims as before,
write one mesh_received audit event per accepted envelope. Result
JSON now reports both counts:
announced_roots: N (was: announced)
announced_falsifications: N
sent_roots: [...] (was: sent)
sent_falsifications: [...]
Two opt-out flags so operators can scope the broadcast:
--no-roots only push falsifications
--no-falsifications only push roots
Burns are deliberately NOT propagated. Burn semantics are local
kindergarten cleanup ("delete a leaf I shouldn't have written") —
other peers may have legitimately ingested the doc independently.
Falsify is the audit-preserving alternative whose broadcast IS the
right cross-peer signal for "this answer is wrong."
Caveat (deferred): no per-peer dedup state yet. Re-running sync
re-broadcasts the same most-recent N falsifications; receivers get
duplicate mesh_received audit-log entries (no state corruption,
just log noise). A `mesh_sync_state` table tracking
last_falsify_announced_ts per peer URL is the natural follow-up
when the falsification volume grows.
Tests:
- existing test_sync_announces_local_roots updated for new field
names (announced_roots, announced_falsifications)
- new test_sync_announces_falsifications: bob falsifies, syncs,
alice's chain has the ANNOUNCE_FALSIFICATION envelope
- new test_sync_no_falsifications_flag_skips_them: --no-falsifications
skips the broadcast cleanly
339 passed, 1 skipped.
