Closes the lazy-anchor failure class fox surfaced in feedback-3
(2026-05-01) on "who is homer simpson's boss?":
Claim: "Homer Simpson's boss is Mr. Burns."
Cited span: voice-actor / Castellaneta bio prose
Pre-warrant: pointer-linked, source-role allowed, coverage passes
Post-warrant: WARRANT_MISSING violation, audit_mode capped at HYBRID
Pointer verification != warrant verification. A pointer resolves to
a span; a warrant requires the span to actually contain the claim's
named answer entity.
Adds aborist/qa/warrant.py:
- is_relation_question(q) — regex shape detector for "who is X's Y?",
"who is supermans girlfriend?", "who founded Z?", "boss of X" etc.
- extract_answer_anchors(claim) — multi-word proper nouns ("Mr.
Burns", "Lois Lane") with possessive 's stripping; solo-cap
fallback that skips sentence-starter false positives
- warrant_check(claim, spans) — at-least-one-anchor lexical match;
vacuous-pass when claim has no proper-noun anchor
Wired into verify_claim_lattice via new params (question,
warrant_check_enabled). Per-claim WARRANT_MISSING violations cap
audit_mode at HYBRID using the same demote pattern as
lazy_anchor_demoted. CLI renders " · warrant missing" tail when
violations present so the failure mode is legible without scanning
the violations array.
Policy field claim_lattice_warrant_check_enabled added to
DEFAULT_POLICY / DEFAULT_QUERY_POLICY and the verifier-policy hash
field set so changes bump verifier_policy_hash.
Hard or soft? Per CLAUDE.md "Soft hash vs hard hash":
substring/lexical = hard. Output is binary (warrant present or not),
violations enter the proof path through violations[] alongside
SOURCE_ROLE_BLOCKED and CITATION_MISMATCH.
Deferred (follow-on commits):
- full RELATION_PATTERNS table with relation-noun aliases
- two-hop bridge detection ("X founded Y" + "Y is Z" -> "X is Z")
- retrieval expansion for relation questions
Tests: 16 new unit tests in tests/test_warrant.py covering
is_relation_question, extract_answer_anchors, warrant_check
(happy path, lazy-anchor case, vacuous-pass, case-insensitive,
empty inputs). Full suite: 523 passed, 21 skipped.
Implements 6 of fox's feedback-2/3 items (2026-05-01) targeting the
red-fish-blue-fish output. Each lives in the render or sidecar layer
— no schema or cache-key change beyond the verifier_policy_hash that
landed earlier in the day.
(1) Honest audit-label rendering. Schema audit_mode (STRICT/HYBRID/
UNGROUNDED) describes the lexical verifier's call; the user-
facing label spells out what was actually checked. Mapping for
claim_lattice modes:
STRICT → EVIDENCE-LINKED · via claim_lattice
HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice
UNGROUNDED → UNGROUNDED · via claim_lattice
Quote / span / entity / paraphrase modes keep the audit_mode
token plus a `· via {method}` tail. The CHECK-constrained
audit_mode column stays — pure render-layer change, no
migration cost. Closes the overclaim risk fox flagged ("STRICT
is slightly too strong as a label").
(2) Sentence-bounded evidence spans. _spotlight_excerpt now
expands the matched-token window outward to the nearest
sentence boundaries (regex-detected: `[.!?][\"')\\]]?\\s+(?=[A-Z])`)
instead of cutting at fixed byte offsets. Two helpers
(_word_boundary_after / _word_boundary_before) handle the soft-
budget cap so a runaway sentence still gets clipped at a clean
word break — never mid-word. Pre-fix the renderer produced
excerpts like `"...freewheeling plot... friends and p..."` (the
`p...` is the offending mid-word truncation fox surfaced); now
the excerpt is `"One Fish Two Fish Red Fish Blue Fish is a
1960 children's book by Dr. Seuss. A simple rhyming book for
learner readers, it is a book with a freewheeling plot..."`.
(3) Source-role display + used/unused annotation. The CLI source
list went from
[1] One Fish Two Fish Red Fish Blue Fish — wikipedia.org/...
[2] Red Dwarf — wikipedia.org/...
to
[1] One Fish Two Fish Red Fish Blue Fish — primary_answer_source — used (E1) — ...
[2] Red Dwarf — background_source — unused — ...
Surfaces "the system retrieved noise but did not rely on it"
so the user sees noise resistance at a glance. Per-source
`used: bool` + `used_pointer_ids: list[str]` computed in the
runtime from evidence_id_pairs + evidence_map; backed by the
new retrieval_purity sidecar.
(4) Retrieval-purity sidecar. Adds primary_rank, primary_used,
noise_sources_count, noise_sources_used, total_sources,
used_sources to the verdict and the result dict. Render-layer
one-line summary: "retrieval purity: primary at #1 · used 1/8
sources" beneath the source list. Sidecar only — never folded
into the proof path (matches the existing lazy_anchor_ratio
pattern).
(5) Ordered-token title scoring. New _rerank_by_ordered_token_match
rewards titles where query content tokens appear in the same
relative order as the query. "red fish blue fish" query →
"One Fish Two Fish Red Fish Blue Fish" gets ordered_match=4
boost (1+0.5*3 = 2.5×); "Red Dwarf" stays at ordered_match=1
(no boost). LCS-based, stem-aware, deterministic. Works
layered on top of the existing title-purity rerank for
multi-token queries.
(6) Noise-resistance fixture (tests/test_qa_quality_live.py).
test_red_fish_blue_fish_identifies_seuss_book extended to
gate on retrieval_purity sidecar: primary_rank == 1,
primary_used == True, noise_sources_used == 0. Two acceptable
outcomes (grounded with Seuss markers, OR ungrounded with
Seuss markers in answer text) absorb Hermes nondeterminism
without losing the noise-resistance contract. Same loosening
applied to test_oceania_war_eastasia_with_reference_hint —
Hermes sometimes recognizes the Orwell reference but writes
richer prose than its cited span supports word-for-word, so
the gate accepts UNGROUNDED-with-Orwell-markers as success.
Live test_qa_quality_live.py: 20/20 fixtures pass. 507 unit tests
green. Working tree leaves bench/qa_questions.txt for fox's
parallel WIP.
Fox-supplied review expanding §2-§4 with axiomatic framing, CTI /
Merkle-AGI / PROMETHEUS-Σ interpretations, a concrete implementation
diff, and a strengthened test list. Captured verbatim as Appendix A.
Key refinements over the §2-§4 sketch:
- Splits retrieval-stage hash into retrieval_plan_hash +
retrieval_result_hash + retrieval_stage_hash (plan vs result
separation, axiomatically cleaner than embedding plan in the
retrieval-stage payload directly).
- Adds two new audit events: retrieval_plan_built and
retrieval_result_selected. Surfaces the retrieval inputs in the
audit chain (not just the run_dag), so cache-hits also commit
the plan via cache_hit_with_retrieval_plan.
- Distinguishes three keyword-handling cases for cache identity
(Case A/B/C) and lays out a clean boundary rule: keywords stay
in retrieval_plan unless they reach the model (then prompt_hash)
or affect routing policy (then retrieval_policy_hash).
- Promotes 'the map must be committed, not only the territory
reached' as a core principle. Operator-supplied maps (keywords,
routing modes, ranking policies) become first-class clauses
upstream of retrieval_result.
- Lists 12 concrete tests including the critical 'same sources,
different keywords → different run_dag_root' assertion that
proves the gap is closed.
Header note: Appendix A is authoritative when it disagrees with
§2-§4 above. The original §2-§4 sketch stays in place for
chronological context but is superseded by the appendix.
Six items from fox's de-novo synthesis (2026-05-01) landing as one
atomic commit. Each item references its motivation and respects the
v9.8 honesty boundary (no claims of semantic truth, no proof-path
contamination by soft signals).
NEW DOCS
--------
docs/cti-architecture.md
Maps today's modules onto the architectural layer cake fox named
in his synthesis: PROMETHEUS-Σ (controller) / CTI (claim-lattice
reasoning IR) / Merkle-AGI-DAG (commitment) / Reverse-RAG
(evidence direction) / v9.8 Providence (admissibility ledger) /
Hermes (weak proposer). Per-layer responsibility table + module
map showing which existing files implement which layer. The
architecture is real; the labels above name what's already there.
docs/naming-deferral.md
Argues why we keep internal terms (claim_lattice,
claim_lattice_pointer, verify_claim_lattice) instead of renaming
to architectural labels (CTI, PROMETHEUS-Σ). The rename costs
cache invalidation, ~150 test fixture references, schema CHECK
migration, blame-history disconnect, mesh peer coordination.
The bridge is the cti-architecture mapping doc — read it once,
then read code in code's vocabulary and prose in prose's
vocabulary. Lists four triggers that would invert the decision.
docs/self-reference-distillation-design.md
Successor to docs/self-reference-thought-chains-design.md (the
flat-source MVP). Maps STRICT claims onto the existing Distiller
/ Core / derivations infrastructure: each STRICT record becomes
a Fact-Core via a new ProvidenceDistiller, with per-cited-chunk
Merkle inclusion proofs back to Wikipedia source_roots. The
fact-graph grows; new claims attach by inheriting the proof
chain. CompositionDistiller (combining facts into new claims)
is explicitly deferred — that's reasoning machinery, not
infrastructure.
docs/test-coverage-audit-2026-05-01.md
Maps fox's §11 test list (16 items) against the codebase. 16/16
covered. Documents item #9's manual-quote-rule lifecycle: the
rule was retired from pointer mode in commit 224bfd6 after the
bench journey identified it was rejecting factually correct
claims; retained in JSON variant where the punctuation-rationale
argument doesn't apply. The audit doc itself is the requirements-
drift defense.
CODE
----
aborist/qa/keys.py:verifier_policy_hash
New pure function. Hashes the verifier-relevant subset of policy
(answer_mode + claim_lattice_* verifier knobs + entity_policy
fields + base_version). Folded into cache_key as an optional
9th dimension via a new keyword arg with backward-compat default
None — legacy 8-dim callers continue to work and produce the
pre-2026-05-01 hash result.
aborist/qa/{runner,query}.py
Compute verifier_policy_hash from the policy_variant and pass
to cache_key as the 9th dim. Records written from this commit
forward bind to the verifier-policy identity. Pre-existing 8-dim
records become cache-misses on next lookup — same migration cost
as any governance_policy_hash bump. The 9-dim form makes the
question "did the verifier rules change?" answerable from
cache_key diff alone, separate from "did the prompt change?"
CLAUDE.md
Adds "old maps vs runtime maps" engineering discipline to the
bench-maxing section. Codifies the principle: every base model
carries old maps from training time; the runtime carries the
fresh map; when they disagree, the runtime wins. Move authority
OUT of the model's prior and INTO runtime artifacts (pointer IDs
the runtime mints, source spans the runtime interpolates by
offset, evidence maps assembled per query, policy hashes that
fold prompt/verifier/retrieval into cache_key, hard checks run
by the verifier). Hermes' content-addressed-evidence-id
hallucination (commit bb8450d) is the canonical case study.
TESTS
-----
tests/test_keys.py
Six new tests for verifier_policy_hash:
- only hashes verifier subset (non-verifier fields don't change it)
- changes when verifier-relevant field changes
- empty-policy → stable
- 9-dim cache_key distinct from 8-dim
- 9-dim distinct under different verifier hashes
- 8-dim form preserved for legacy callers (None == omit)
DEFERRED (per fox + naming-deferral.md)
---------------------------------------
- PROMETHEUS-Σ as an extracted controller module — the dispatch
logic in runner.py + query.py already IS PROMETHEUS-Σ; an
explicit prometheus.py is a refactor with no behavior change.
Defer until a §5-rename-trigger fires.
507 tests pass (was 501 before, +6 from verifier_policy_hash
coverage).
Tickets are flat docs/ticket-NNNNNN-<slug>.md — actionable proposals
awaiting go/no-go, scoped defects, or queued enhancements.
Architecture references and bench journals stay un-numbered.
docs/TICKETS.md: index + convention + Next ID line so future shifts
know where the next number comes from. Renumbering protocol: every
new ticket commit bumps the Next ID line atomically with the new
ticket file. Status field tracks open / in progress / closed; closed
tickets keep the file in place as a design log.
docs/ticket-000001-retrieval-keywords-audit-gap.md: the ticket
opened in 73058b0, renamed and re-headered to fit the convention.
Per fox's '--retrieval-keywords' commit (2d6a86b), the keyword string
is nowhere in the audit chain — it influences cache_key only
indirectly via context_root + conversation_hash. Two runs with the
same question and different keywords that surface identical sources
are Merkle-indistinguishable; an audit replay can recover the chosen
documents but not the inputs that pulled them.
Doc-only ticket: design proposal for capturing keywords in the
run_dag retrieval stage and on the providence_cache row (Option A +
B), preserving the 8-dim cache_key invariant. Out of scope: folding
keywords into question_hash (Option C) — keywords are operator
metadata, not user intent.
Includes failure cases, three design options with trade-offs,
implementation sketch, scope boundaries, and a recommended cost/value
forecast (~1-2 hours, low risk, additive schema). Awaiting fox's
go/no-go before landing.
Three layers, 14 new tests, full suite 611 passed (was 597):
UNIT — tests/test_query.py
test_query_returns_prompt_chars_breakdown
Asserts the result dict's prompt_chars carries exactly the five
expected keys and messages_total equals sum of message contents
the StubClient saw.
test_query_answer_chars_matches_answer_text
answer_chars == len(answer_text) — drift check.
test_query_cache_hit_also_returns_capacity_metrics
Cache-hit path populates prompt_chars + answer_chars (operators
inspecting cached records still want the breakdown).
test_query_evidence_chars_grows_with_topk
Sanity: more sources / larger budget → more evidence chars
(the metric tracks actual context build, not a stale constant).
INTEGRATION — tests/test_query.py (retrieval_keywords)
test_retrieval_keywords_does_not_alter_question_to_llm
Keywords don't appear in the LLM-facing question segment;
system prompt unchanged across runs. Pins the substrate
contract: keywords are FTS5/title-filter-only.
test_retrieval_keywords_changes_retrieved_sources
Different keyword sets surface different docs (the actual
user-visible behavior).
UNIT — tests/test_cli_render.py
test_render_shows_capacity_line_when_prompt_chars_present
Capacity line appears with messages_total + breakdown when
prompt_chars is in the result dict.
test_render_omits_capacity_line_on_legacy_results_without_prompt_chars
Backwards-compat: legacy results render cleanly without the
capacity line — no KeyError, no '0 chars' noise.
test_render_capacity_thousand_separators
61,550 not 61550 — operator legibility on daily renders.
UNIT/INTEGRATION — tests/test_bench_qa_sweep.py (NEW FILE)
Imports bench/qa_sweep.py via importlib.util so the module's
not in the Python path doesn't matter. Five tests:
- _summarize counts verdicts by mode
- deflections counted only on STRICT/HYBRID rows (not UNGROUNDED)
- rendered markdown has the headline summary + size buckets
- size buckets correctly stratify strict-rate by prompt_chars_total
- empty buckets are skipped (no '0 runs' noise)
FUNCTIONAL — live verification (no automated test, manual)
`make query Q="what is the capital of france?" BURN=1` confirmed
in commit f927298 to render the capacity one-liner under the
source list. Documented in that commit's body.
Also corrected the docstring on query()'s `retrieval_keywords` to
reflect that keywords don't enter cache_key DIRECTLY but do change
context_root + conversation_hash via source selection — so the same
question with different keywords lands under different cache_keys
(legitimately, since the LLM saw different contexts).
Per-call accounting of prompt capacity so we can answer 'does
context size affect strict-rate?' at aggregate scale.
Per-call result dict gains:
"prompt_chars": {
"system_prompt": <int>, # system message chars
"grounding_reminder": <int>, # restated rule message (if any)
"user_question": <int>, # the original question text
"evidence_or_context":<int>, # rendered evidence map / wikitext context
"messages_total": <int>, # sum of all messages[].content
}
"answer_chars": <int> # rendered answer length
Char-level for now: model-agnostic, fast, and a workable proxy
for prompt tokens (~4 chars/token English prose, ~2.5 for JSON-
evidence-block heavy contexts). Token-exact accounting can layer
on later if a tokenizer is cheap enough to wire in per-call.
Surfaces in three places:
(1) result["prompt_chars"] — for programmatic callers. Cache hits
return prompt_chars too (computed before lookup).
(2) `aborist query` human render — one-liner showing the breakdown
and the answer length, so an operator can tell at a glance
whether STRICT came from a 5KB tight prompt or a 60KB stuffed
context.
(3) bench/qa_sweep.py — per-row prompt_chars_{total,evidence,system,
question} + answer_chars columns in the JSONL, plus a new
"strict-rate by prompt size" section in the markdown summary
bucketing runs by capacity (<8KB, 8-16KB, 16-32KB, 32-64KB,
>=64KB) so the bench reveals whether input size correlates
with verdict quality across modes.
Never enters cache_key — runtime measurements, not policy.
Empirically observed 2026-05-01: long discursive questions like
'what technology is currently or soon available which may enable
one person to reconstruct another person's thoughts...' under-
retrieve because their content tokens get diluted by template
phrasing. AND-mode FTS5 returns zero hits; OR-fallback ranks
unrelated articles by raw BM25.
Fox's discovery: appending domain keywords ('transcranial knowledge
acquisition') to the question lifts the verdict from HYBRID 6/10
to STRICT 1/1 by narrowing OR-mode retrieval to the topical
article (Neurotechnology). The keywords act as an operator hint
about what the question is really asking.
This commit exposes that pattern as a first-class flag without
polluting the question text:
--retrieval-keywords TEXT on `aborist query`
K="..." on `make query`
Plumbing: the keywords are concatenated with the question for
FTS5 search and title-filter token construction only. The LLM
still sees the original question; the verifier still checks against
the original question; cache_key still computes from the original
question. Keywords are session-only — successive calls with
different keywords on the same question can cache-hit each other.
Pair with BURN=1 for fresh inference per call.
Live verification:
make query Q="...thoughts...without speaking or sign language."
K="transcranial knowledge acquisition" BURN=1
-> STRICT 2/2 via claim_lattice, 37.5s, Neurotechnology article
cited for fMRI and DBS claims.
Without K: HYBRID 6/10 with Videoconferencing/Telepathy as top
sources — model hallucinated structure across irrelevant chunks.
Empirical finding 2026-05-01 on 'explain spin glass modeling, tensors?':
returned STRICT 1/1 with claim 'Spin glass modeling involves...
mathematical tools such as tensors' cited to a chunk from the
*Quantum chromodynamics* article. Token-coverage check inside the
chunk passed accidentally on shared physics vocabulary
(spin/model/etc.), but the article isn't about spin glasses — the
claim is a retrieval-driven hallucination.
Root retrieval cause: AND-mode FTS5 returned zero hits (Spin_glass
article has 0 occurrences of 'modeling' or 'tensors'); OR-mode
fallback ranked by raw BM25, so 'Glass' (glass*274) outranked
'Spin glass' (spin*29 + glass*15) despite Spin_glass matching 2
distinct query tokens vs Glass's 1. Verifier then accepted the
hallucinated claim because the cited QCD chunk has incidental
physics vocabulary overlap.
New ``diagnose_title_relevance(claim_text, cited_source_titles)``
sidecar: claim's content tokens (stemmed) must overlap the union
of cited source titles. Zero overlap → ``title_mismatch``. Catches
the spin-tensors case (claim about spin glass, cited title 'Quantum
chromodynamics' → zero stem overlap) without false-positives on
legitimate cross-document grounding (Lois_Lane article cited for a
Lois-Lane claim still has lois+lane overlap).
Sidecar-only for now — staying out of the binary verifier output
until bench evidence justifies hard-promotion. Not yet wired into
inspect_cache_key (per-claim citation source mapping isn't in
providence_cache yet) or the bench. Available as a utility for
operators auditing specific (claim, source_title) pairs.
5 new unit tests covering match/mismatch/no-titles/no-claim-tokens
and underscore-normalization for Wikipedia titles.
Closes the recursive-grounding gap fox surfaced today: aborist
tends Wikipedia trees but never grafts its own past Q&A records
into the forest. Each query starts from cold corpus retrieval;
prior providence_cache records sit unread until the same question
is re-asked (cache_key match). The "kindergarten thought chains"
framing names the gap — the system has a kindergarten of thoughts
(early STRICT records) that should mature into citable substrate
as they cool.
This commit lands the MVP — STRICT live providence records past
the kindergarten window become substrate via a new Source
subclass. Trust model per fox: "we trust strict statements as
fact unless a verifier falsifies it."
NEW
---
- docs/self-reference-thought-chains-design.md — full architecture
doc covering the four iteration-time gates, the recursive Merkle
proof story (Q2 → Q1 → Wikipedia chunk per v9.8's recursive-
cores insight), the falsification trust model (state=live as the
fail-closed gate), risks (lazy-anchor compounding, echo chambers,
storage bloat), and bench-impact estimate.
- aborist/sources/providence.py — ProvidenceSource(Source) with
four hard gates:
1. audit_mode == 'STRICT' (HYBRID/UNGROUNDED stay opaque)
2. falsification_state == 'live' (failed/stale/quarantined
excluded — verifier-falsification mechanism per fox)
3. now - created_at >= kindergarten_seconds (default 1h —
fresh thoughts cool first; kills tight echo loops)
4. anti-recursion: records whose answer text contains a
self-reference URI are skipped — first-generation only
- tests/test_providence_source.py — 10 unit tests covering each
gate plus the URI-scheme source-role classifier
- Makefile target `ingest-self-providence` (KG_SECONDS=3600
default; iterates each shard and self-promotes its STRICT live
records — cross-shard sharing happens via the existing
shards-dir UNION at retrieval time)
WIRE-UP
-------
- aborist/qa/query.py
- SOURCE_ROLE_BUDGET_WEIGHTS: self_reference_source = 1.0
(same as background — Wikipedia stays canonical primary;
self-reference is supplementary anchoring)
- SOURCE_ROLE_RANK_WEIGHTS: self_reference_source = 0.9
- _classify_source_role: short-circuits on aborist://providence/
URI prefix → self_reference_source regardless of title shape
- DEFAULT_QUERY_POLICY['claim_lattice_allowed_source_roles']
+= 'self_reference_source'
- aborist/qa/runner.py — same allowlist update for the
per-document `ask` path
- aborist/cli.py — `aborist ingest --source providence` reads the
providence_cache from the same shard it writes into;
--kindergarten-seconds flag plumbed through
NOT IN THIS COMMIT
------------------
- Aggregation of multiple Q&A records into synthesized summary
records (follow-on)
- Self-reference for HYBRID records (only STRICT is substrate
today; HYBRID could land later as a soft-anchor role with
lower trust)
- Live virtual sourcing (the design discusses it; MVP uses
snapshot ingestion so existing FTS / chunker / Merkle apply
with zero schema change)
- A live bench validating actual lift on self-reference questions
(requires running ingest-self-providence then bench; deferred
to follow-on commit on real data)
10 new unit tests pass; full suite at 482 passed / 21 skipped
(live fixtures gated).
Bench dig (2026-05-01, 65-question sweep): 4 of 7 deflecting
questions were false positives from verb-shape subject anchors.
what year did the berlin wall fall? → answer "1989" (anchor "fall")
when did the soviet union dissolve? → answer "1991" (anchor "dissolve")
how many wives did henry the eighth? → answer "six" (anchor "eighth")
what year does our cold fusion ...? → answer "1989" (anchor "happen")
why did the titanic sink? → "...sank in 1912" (anchor "sink" vs "sank")
why did the dinosaurs go extinct? → "asteroid impact" (anchor "extinct" vs "died")
These shapes ask for a year/count/cause; the answer is structurally
not expected to echo the question's subject. Pre-suppression, all
three modes landed deflection on these — the apparent
"pointer-only" deflection rate (10.3% vs 6.2%) was almost entirely
heuristic noise, not a real substrate failure.
New ``shape_suppressed`` flag returned by diagnose_deflection. When
the question starts with one of:
when, what year, what date, what time,
how many, how much, how long, how old, why
the subject-anchor check is skipped — overlap-ratio takes over.
Mars-BDFL (who-shape) still fires correctly.
Field exposed so the operator can filter the signal: a
shape_suppressed=True row that still classifies as deflection on
overlap-ratio means "answer was a bare year/count/cause with no
topic words" — known noisy, not real topic shift.
5 new unit tests; full suite 582 passed (was 578).
Catch-up after a productive afternoon. Three new conventions section
items + retrieval-pipeline addendum:
- Three answer modes documented: quote, claim_lattice_pointer (G0/CTI),
claim_lattice (JSON). Bench evidence (2026-04-30) on Hermes-3-8B:
quote 0.47 strict-rate, pointer 0.36, JSON 0.49. Both lattice modes
share verifier_method='claim_lattice' (CHECK constraint
compatibility); answer_mode + json_fixups disambiguate downstream.
- Claim-count ceiling (TOO_MANY_CLAIMS): default 12/answer, demote
STRICT→HYBRID without truncating. Bench finding context
(york-england runaway) preserved so future-me knows why the cap
exists.
- Deflection sidecar (qa.inspect.diagnose_deflection): subject-anchor
heuristic catches Mars-BDFL pattern. Sidecar-only — never feeds
back into providence. Wired into bench/qa_sweep.py as a per-row
signal + summary-table column.
- Retrieval pipeline stage 9: template-phrase stopwords. The
'all there know' additions closing york-england, with the AND-mode
FTS5 mechanism documented so a future change to MATCH semantics
doesn't accidentally re-open the miss.
Two paired changes:
(1) Retrieval template-phrase stopwords. Bench finding 2026-04-30:
'tell me all there is to know about york england?' missed the
en.wikipedia.org/wiki/York article entirely — corpus has it but the
FTS5 AND-mode MATCH expanded to "all" AND "there" AND "know" AND
"york" AND "england", which filters to only docs containing every
template word. Articles like "Charles I of England" happen to mention
"all" + "there" + "know" + "england" enough to win; the bare "York"
stub doesn't, so it falls out of the top-K.
Added `all there know everything anything something` to both
_FTS5_STOPWORDS (search/fts5.py) and _TITLE_STOPWORDS (qa/query.py)
so they stay in sync. After the change "York" article ranks #1 on
both 'tell me all there is to know about' and 'tell me everything
about' phrasings.
These are retrieval-time stopwords only — they don't enter the
cache_key or governance_policy_hash (cache_key hashes the original
question text via question_hash). Existing cached records remain
valid; new queries get better retrieval.
(2) bench/qa_sweep.py: per-row deflection_kind + subject_anchor +
subject_in_answer (from qa.inspect.diagnose_deflection); summary
table grows a 'deflections' column tracking STRICT/HYBRID rows
where the question's subject anchor is missing from the answer.
Surfaces the Mars-BDFL pattern at aggregate scale.
New docs/verifier-semantic-gap-design.md captures the deep
roadmap item from docs/qa-modes-bench-2026-04-30.md ("Verifier
semantic check (soft signal) — did the claim's predicate match
the cited span's frame?").
Problem: the claim-lattice verifier's lexical coverage check
(≥30% claim-token overlap with cited span) passes any (claim,
span) pair that shares enough surface tokens, even when the
span never asserts the claim's predicate. Three concrete cases
captured from live bench data:
- Great Wall elevation: STRICT 1/1 cited to a chunk that
discusses "Outer China beyond the Great Wall" but contains
no elevation information at all.
- JP-dinos Triceratops/Operation-Genesis: model cites the
2003 video-game article for a claim about the 1993 film.
- Boltzmann constant value: model cites a chunk that names
the constant and its unit but never states the numerical
value.
Three candidate designs evaluated against §1.1 cases + the
hard architectural constraints (proof-path purity, determinism,
no external endpoints beyond Hermes, latency budget):
3a NLI cross-encoder sidecar (recommended)
3b TF-IDF predicate matching (cheap but high false-negative)
3c Per-claim re-prompt to Hermes (latency + self-eval bias)
Recommendation: 3a with cross-encoder/nli-MiniLM2-L6-H768
(~80M params, ~100MB weights, CPU-runnable, deterministic at
fp32). Default off, opt-in via `claim_lattice_semantic_check`
policy field. Demote-only sidecar — moves STRICT → HYBRID
when claim entailment fails, never invents grounding. Soft
signal stays out of the proof path, mirrors the existing
`lazy_anchor_demoted` pattern.
Estimated impact: ~5-8 STRICT demotes per 66-run bench
(false-positives removed from the ledger), +0.4s typical
latency. Six open questions for fox at end of doc.
Doc-only commit. No code changes. Implementation deferred.
Empirical finding 2026-04-30 on 'who is a benevolent dictator for life
for mars?': JSON mode returned STRICT with answer 'Guido van Rossum is
a Benevolent Dictator For Life (BDFL) for the Python programming
language.' The verifier did its job (claims grounded), but the answer
never mentioned Mars — pure topic-shift. A real failure mode for
adversarial-premise questions where the model deflects rather than
refuses.
Detection rule: subject-anchor heuristic. The LAST content token in the
question (after stopword strip) is treated as the question's primary
subject ('mars' for BDFL, 'beatles' for 'who are the members of the
beatles?'). If the subject anchor is missing from the answer, classify
as 'deflection' regardless of generic-vocabulary overlap. The Mars-BDFL
case has 3/4 generic overlap (benevolent, dictator, life) but missing
subject — exactly the pattern overlap-ratio alone can't catch.
Returns dict with kind ∈ {deflection, partial_overlap, on_topic,
no_question_tokens} plus subject_anchor + subject_in_answer for
operator legibility.
Sidecar-only — wired into inspect_cache_key() but never feeds back
into providence_cache or audit_events. Per the verifier-stays-binary
discipline (CLAUDE.md): soft signals live in inspect verbs, never in
the hard chain.
Bench finding (2026-04-30, n=3 across 24 questions): JSON mode hit
S:3 H:0 U:0 on three questions where pointer mode landed S:0 H:3 U:0
or quote mode landed S:0 H:0 U:3 — clean modal divergence. Pinning
these as live fixtures gives us regression detection: a JSON-mode
quality drop on these questions reverts the STRICT signal first.
test_mona_lisa_strict_and_names_leonardo — was _grounded, tightened
to STRICT (bench: quote 0S, pointer 0S, JSON 3S)
test_supermans_girlfriend_strict_and_names_lois_lane (NEW)
— relationship-shape, JSON's strong suit (quote 0S, pointer 0S,
JSON 3S)
test_cold_fusion_breakthrough_year_strict_and_names_1989 (NEW)
— out-of-corpus rephrasing, atomic-claim rule unlocked grounding
(quote 0S 3U, pointer 3S, JSON 3S)
Per fox's "scoreboard + gates" docstring: bench gives aggregate
counts; these fixtures fail by name when the scoreboard moves.
Defense-in-depth against the "tell me all there is to know about X"
runaway shape. Bench finding (2026-04-30 york-england, n=3): JSON mode
emitted 26-59 claim-pointer pairs of which only 2-4 verified — model
treats encyclopedic prompts as a license to spam claims from training.
The atomic-claim prompt rule (b5925c8) cut typical runs to ~10 claims,
but a structural ceiling makes the failure shape operator-visible
regardless of per-claim verification.
Cap: 12. Admits typical entity-list questions (5-7 dinosaurs, simpsons
+ pets) while flagging the runaway. Plumbed through both verifiers
(pointer + JSON) via claim_lattice_max_claims_per_answer policy knob;
folds into governance_policy_hash on change.
Behavior: cap doesn't truncate — every claim still verifies so the
operator sees full evidence of the runaway. The TOO_MANY_CLAIMS
violation demotes STRICT → HYBRID via the existing violation-check
path; UNGROUNDED if no claims verified.
Tests: test_verify_json_too_many_claims_demotes_to_hybrid (13 claims
all verifying → HYBRID), test_verify_json_at_cap_can_still_strict
(12 claims at the cap → STRICT, no violation), and pointer-mode
mirror test_too_many_claims_demotes_pointer_mode_to_hybrid.
Prompt audit + rewrite per the negation-avoidance principle codified
in CLAUDE.md: small instruction-tuned models (Hermes-3-8B observed)
struggle with "do NOT X" / "never Y" / "without Z" — the negation
often pattern-matches away or inverts under attention. Rewrote 18
negation instances across 8 prompt fields in
DEFAULT_POLICY (runner.py) and DEFAULT_QUERY_POLICY (query.py) to
their positive forms. Examples:
"Do NOT invent IDs" → "Use only the IDs that appear in the
EVIDENCE blocks above"
"Never begin a line with E1:" → "Each line begins with the
claim text, then a bracket tag"
"No commentary, no preamble" → "Your output begins with `{`
and ends with `}`"
"If no evidence supports a claim, omit the claim" →
"Emit only claims that an EVIDENCE block directly supports"
Also added an atomic-claim rule (Rule 6 JSON, Rule 9 pointer):
"Each claim is one focused fact citing the source that directly
contains it. Multiple distinct facts become multiple claim
entries… Three short claims, each citing one pointer, beat one
long claim citing five." Targets the StarCraft-style mega-claim
where the model packs every fact into a single sentence with
five pointers.
Bench delta (n=3, 25 questions, 75 runs per mode):
quote 30S 28H 17U → 40S 23H 12U +10 STRICT, +13pp rate
pointer 27S 37H 11U → 23S 37H 15U -4 STRICT (acceptable —
JSON is now the daily
driver; pointer is
fallback/diagnostic)
JSON 36S 28H 11U → 44S 26H 5U +8 STRICT, +11pp rate,
-6 UNGROUNDED, -1.8s latency
Net across modes: +14 STRICT, +7 grounded, JSON latency
7.2s → 5.4s.
Bench questions: replaced "isn't it true that the great wall of
china is visible from space?" with "which side is the ground
elevation highest throughout the span of the great wall of
china, the north or south?" — a forensic-shape question with
answer "north". Live finding: model hits STRICT 1/1 with the
right answer cited to a Great-Wall-but-not-elevation chunk
(coverage threshold passes via topical-token overlap). New
fixture test_great_wall_elevation_north_side_higher gates on
entity-presence ("north" in answer) but tolerates UNGROUNDED —
documents the residual lazy-anchor failure mode for the
verifier-semantic roadmap.
CLAUDE.md: added negation-avoidance to the bench-maxing
discipline section.
Bench surfaced a real test bug: the prior assertion treated STRICT as
'model asserted a Mars BDFL,' but STRICT only means 'claims are
grounded.' On 'who is a benevolent dictator for life for mars?',
claim_lattice mode deflected to Guido van Rossum / Python (no Mars
mention) and the verifier returned STRICT correctly — claims grounded,
false premise sidestepped.
Three accept paths now:
1. UNGROUNDED (refused outright)
2. STRICT/HYBRID + mentions 'mars' + includes refutation marker
3. STRICT/HYBRID + does not mention 'mars' (deflection — answered
an adjacent grounded question instead of the false-premise one)
Fail case is unchanged: answer mentions Mars AND affirmatively names
a Mars BDFL with no refutation language.
Refutation markers expanded with 'no recognized', 'no known',
'fictional' to cover sci-fi-framing answers (Barsoom / Mars trilogy
characters as fictional dictators is also non-hallucinatory).
Adds "Bench-maxing — measure deltas, not opinions" section to
CLAUDE.md capturing the patterns established during the
2026-04-30 JSON-mode hardening journey (retry → trim-and-verify
→ pointer-IDs → stop-sequence → self-heal):
- Bench before AND after every change. n=3 single-sample
noise is ±20pp; signal under 5pp is unreliable.
- Slight prompt adjustments are fine — local-minima are not.
One-thing-per-bench-cycle so deltas are attributable.
- Bench is the scoreboard; live fixtures
(tests/test_qa_quality_live.py) are the gates.
- Self-heal beats retry — repair malformed output rather
than re-running the LLM. See _repair_truncated_json.
- Name the failure → fix → re-bench → confirm. Each commit
references the specific failure mode it targets.
- Honest HYBRID beats false-positive STRICT every time.
Verifier improvements drop STRICT count when the count was
inflated by token-coincidence false-positives — this is
a WIN, not a regression.
- Per-question fixtures use entity-presence not byte-match;
Hermes is non-deterministic and intermittent failures are
quality-drift signals to investigate upstream, not test-
flake to suppress.
- Doc captures the journey not just end state — future-you
reads docs/qa-modes-bench-*.md and sees WHY each commit
landed.
Question-set churn (bench/qa_questions.txt):
- narrow factoid: drop 'who founded apple computer?'; add
'a bridge between new london & groton?' (single-answer factoid
with named-entity grounding via Gold Star Memorial Bridge)
- broad descriptive: drop 'tell me about the apollo program'
- entity list: drop 'list the founders of microsoft'; add
'name simpsons family members including pets?',
'list of obelisk in connecticut'
- leading/adversarial: add 'whales are endangered from over hunting'
(declarative-shape, tests verifier under premise-seeking phrasing)
- out-of-corpus: rephrase Mars no-such-entity question to
'who is a benevolent dictator for life for mars?' — sharper
adversarial shape since BDFL is a real OSS-governance term
the model is tempted to project onto Mars
- minor edits: 'GNU linux', 'C programming language', 'method man',
'cold fusion breakthrough' phrasing
tests/test_qa_quality_live.py: rename test_mars_prime_minister_*
to test_mars_benevolent_dictator_*; refresh refutation markers
to match the new BDFL phrasing; expand to cover 'no one' /
'no person' negations the model uses on this shape.
Removed:
- test_apple_founders_grounded_and_names_steve_jobs
- test_microsoft_founders_names_gates_and_allen
Added:
- test_veronica_ballestrini_singer_october_birth
The veronica fixture is a stronger probe than the two it replaces:
- entity-disambiguation among 8 same-first-name candidates
(Landers, Guerin, Belmont, Wadley, Gamba, Doran, the
disambiguation page, plus Ballestrini herself in the corpus)
- multi-part question (who is she + when was she born)
- niche personal-bio source (small article, harder to anchor
than well-known commercial-tech bios)
Asserts: grounded + answer mentions Ballestrini + at least one
bio marker (country/singer/songwriter/Connecticut/Italian) + at
least one birth marker (October or 1991). Lands STRICT 1/1 in
JSON mode at the live test.
10/10 fixtures pass.
Adds _repair_truncated_json() to recover from mid-output truncation
where Hermes-3-8B writes one long claim text and runs out of
max_tokens before closing the structure. Pre-fix the lenient parser
saw something like:
{"claims":[{"text":"The Apollo program was the United States
spaceflight effort which landed... <CUT>
with no closing `"`, no closing `}`, no closing `]`, no closing
outer `}`. Couldn't parse, lost the whole answer.
Post-fix the repair walks the input once tracking string state
and bracket stack:
- close any open string (append `"`)
- drop dangling structural artifacts:
`"key":` (partial key with colon)
`"key"` after `,` (partial key without colon)
trailing `,` before close
- close the bracket stack in reverse (`{` → `}`, `[` → `]`)
Conservative: never inserts content (no key names, no values, no
commas) — only closes / drops. Worst case the repair is a no-op
or makes parsing fail in a different way; never silently
fabricates data.
Apollo live test result: UNGROUNDED 0/0 (lost answer) →
UNGROUNDED 0/1 (partial first claim recovered with evidence_ids
intact, partial second claim dropped). The user sees the model's
attempted answer in the unverified bucket instead of empty space.
The audit chain captures the repair via fixups labels
(close_string, close_brace, close_bracket, drop_partial_key,
strip_trailing_comma) so an auditor can see what the parser had
to peel.
Pipeline ordering in _lenient_json_parse: strict parse first,
then peel (fence/prose-trim/curly-quotes/trailing-comma), then
strict parse again, then truncation repair as last resort. Each
step's fixups append to the same list so the verdict's
`json_fixups` field is comprehensive.
Bench question set picked up "name simpsons family members
including pets?" — a multi-part list with adjacent-relationship
shape, addresses a gap in the corpus probe.
460 unit tests pass; 12 skipped (live fixtures unchanged).
Adds two architectural-fix sections to the QA-modes bench doc:
1. Pointer-IDs in JSON mode (commit bb8450d) — closes the
content-addressed evidence_id hallucination loop where
Hermes-3-8B emitted near-miss IDs (E1b6e396 vs the runtime's
Eed1b6e396) on cross-document relationships, landing
UNGROUNDED on factually correct answers. Switching to short
pointer IDs (E1, E2, …) made fabrication obvious and the
Homer Simpson fixture went UNGROUNDED 0/1 → STRICT 1/1.
2. JSON-mode stop-sequence (commit f23d3a3) — guards against
post-brace token runaway where Hermes spammed whitespace
until max_tokens exhausted on broad-descriptive shapes
(apollo program 3/3 runaway in the post-pointer-ID bench).
stop=["\n\n"] cuts the runaway since well-formed JSON-mode
output never contains a blank line.
Also adds a bench progression table showing the journey from
the morning baseline (JSON 19 errors, 26 STRICT) to the post-
pointer-ID evening run (0 errors, 31 STRICT, 56 grounded).
Each row was a named-failure → fix → re-bench cycle.
Bench data shows residual JSON-mode token runaway after the
pointer-ID switch: ~4 cases out of 66 land UNGROUNDED 0/0 at
12-15s instead of ~2-4s normal. Pattern: Hermes-3-8B emits a
valid claim object, then keeps generating whitespace / blank
lines until max_tokens (512) exhausts. The truncated payload
won't parse and the lenient pre-parser returns no claims.
Concrete instances in the latest bench (2026-04-30T19-55-11Z):
- tell me about the apollo program (3/3 samples runaway)
- tell me about the python programming language (1/3)
Fix: pass `stop=["\n\n"]` to the chat completion in JSON mode so
vLLM cuts generation at the first blank line. Well-formed JSON-
mode output never legitimately contains a blank line — Hermes
emits one object on a single line (or with simple internal
newlines), never `\n\n`. The stop sequence is the runaway
signature itself.
Plumbing:
- OpenAICompatibleClient.chat_completion: new `stop` kwarg,
injects into request payload when non-empty
- StubClient already absorbs **kwargs; no change needed
- DEFAULT_POLICY (runner) + DEFAULT_QUERY_POLICY (query) gain
`claim_lattice_json_stop_sequences = ["\n\n"]`. Folds into
governance_policy_hash so changing the stop list invalidates
prior cached records.
- Both call sites in runner.py / query.py read the policy
field and pass it only on JSON mode (pointer + quote modes
don't need it).
Defensive measure: worst case the stop sequence never fires;
best case the apollo/python residuals recover and JSON's
strict-rate climbs further.
JSON mode previously presented content-addressed evidence_ids
(``Eed1b6e396``-style) in the EVIDENCE block headers and expected
the same in the model's claim output. Hermes-3-8B was fabricating
plausible near-miss IDs (e.g. ``E1b6e396`` when the runtime had
``Eed1b6e396``) on cross-document relationship questions →
UNKNOWN_EVIDENCE_ID → UNGROUNDED, even when the answer text was
factually correct (e.g. "Homer Simpson's boss is Mr. Burns.").
Switching the prompt-facing surface to short pointer IDs (``E1``,
``E2``, …) — same as claim_lattice_pointer mode — closes the
hallucination loop:
- Pointer IDs are short, enumerable, and fabrication-obvious.
The model can't invent ``E27`` if only ``E1``-``E10`` were shown;
out-of-range IDs read as schema violations at first glance.
- The runtime still resolves each pointer_id to its content-
addressed evidence_id internally and stores THAT in
``evidence_id_pairs`` for the cache & run-DAG. Cache_keys stay
run-stable; only the prompt-facing string changes.
- JSON schema unchanged (``evidence_ids: [str, ...]``), so vLLM
guided_json continues to constrain output shape.
Live-test verification: ``who is homer simpson's boss?`` went from
JSON-mode UNGROUNDED 0/1 (hallucinated ``E1b6e396``) to STRICT 1/1
(model emits ``E1``, resolves cleanly). Homer fixture pin removed —
runs against the JSON default now.
Files touched:
- aborist/qa/evidence.py: render_evidence_block_for_json uses
e.pointer_id instead of e.evidence_id.
- aborist/qa/verify.py: verify_claim_lattice_json switched from
evidence_map_by_evidence_id to evidence_map_by_pointer_id;
captures both pointer_ids (model-emitted) and evidence_ids
(run-stable) in claim_statuses + evidence_id_pairs.
- aborist/qa/{runner,query}.py: claim_lattice_json_system_prompt
+ grounding_reminder describe pointer IDs; example shifts from
``E........`` placeholder to ``E1``.
- tests/test_verify_json.py: stub _ev() takes pointer_id; four
fixtures updated to set it.
- tests/test_qa_quality_live.py: Homer fixture unpinned (now
runs default JSON mode and grounds). Red-fish-blue-fish
fixture pinned to pointer mode — JSON mode hits a separate
token-budget runaway on "plot of X" prose-summary shapes
(~2/3 of samples blow max_tokens with whitespace spam after
the closing brace). Different failure mode, addressed in a
later commit.
460 unit tests + 11 live fixtures pass.
The previous commit (588aa45) accidentally git-added four
.claude/worktrees/agent-* gitlinks via `git add -A`. Those are
session-local Claude Code worktree pointers (a tooling artifact),
not source. Removing them from the index and adding `.claude/` to
.gitignore so future sessions don't trip the same way.
Post-retry / post-trim-and-verify bench showed the picture flipped:
JSON mode now leads on strict-rate (50%) and ties grounded count
(54) with zero errors, all at parity-or-better latency. Switching
the `make query` ANSWER_MODE default from `claim_lattice_pointer`
to `claim_lattice` so the human-facing CLI uses the strongest mode.
Library-level DEFAULT_ANSWER_MODE stays "quote" so unit tests using
StubClient aren't disrupted.
Doc updated with post-retry bench table + revised conclusion.
Live test harness:
- new test_homer_simpson_boss_is_mr_burns fixture (pinned to
pointer mode — JSON mode hallucinates evidence_ids on
cross-document relationships, pointer mode's short numeric
tags can't be fabricated). Documents the mode trade-off.
- test_laura_croft marker net broadened to absorb Hermes
single-sample variance (added "magazine", "video game",
"character", "fictional", "british" markers) plus a hard
"croft" anchor check. Both real entities still gate cleanly.
11/11 live fixtures pass at the new JSON default + pointer pin
on Homer. 460 unit tests + 11 live = 471 green; 10 skipped is
just the live tests in the default-skip path.
Adds test_red_fish_blue_fish_identifies_seuss_book under the new
"retrieval disambiguation" section. The query partially matches
many noise titles (Red Dwarf, Blue whale, Toronto Blue Jays, Red
Sea, Detroit Red Wings, marine aquarium fish list) so it stress-
tests the retrieval+rerank pipeline's ability to land on the right
article — `One Fish Two Fish Red Fish Blue Fish` by Dr. Seuss —
despite the surface noise. Asserts the answer is grounded and
contains a Seuss-specific marker (seuss, rhyming, children's book,
or creatures). 10 tests collected total.
New tests/test_qa_quality_live.py with 9 fixtures gated by
ABORIST_LIVE_TESTS=1 (the `make test-live` target sets it). Each test
runs ONE live query against Hermes + the configured shard set and
asserts:
* audit_mode is at least HYBRID (or UNGROUNDED for the honest-
refusal case)
* key entity tokens appear in the rendered answer
(e.g. "Torvalds" for the linux-kernel question, all four turtle
names + "Splinter" for the TMNT multi-part)
* for the Mars PM no-such-thing case, either UNGROUNDED OR a
refutation phrase ("no prime minister", "does not have", etc.)
— bare affirmative claims of a Mars PM fail the test as
hallucination
Why functional tests matter alongside unit + bench:
- unit tests (test_qa, test_query, test_claim_lattice) validate
plumbing with StubClient — can't tell you "did the model
actually answer correctly?"
- bench (`make bench-qa`) measures aggregate strict-rate /
grounded counts across many questions but doesn't assert
specific content
- these live fixtures sit between: each test is a named gate
around one known-good answer. When a future change improves
things, the bench number climbs AND every fixture passes (or
gets stricter assertions). When something regresses, the bench
number falls AND specific fixtures fail by name, telling you
where the regression landed. "Benchmax" rationale: the bench
is the scoreboard, the fixtures are the gates that translate
quality drift into named test failures.
Default `make test` is unaffected — 460 passed + 10 skipped (was 1).
Three-way QA-quality bench (quote / pointer / JSON) over an expanded
22-question set × 3 samples = 198 LLM calls. Findings landed in
docs/qa-modes-bench-2026-04-30.md with per-question breakdown and
roadmap. Aggregate at bench time:
quote 31S 20H 15U 0e strict-rate 47% 7.7s
claim_lattice_pointer 14S 34H 18U 0e strict-rate 21% 4.4s
claim_lattice (JSON) 26S 12H 9U 19e strict-rate 39%* 4.4s
The 19 JSON-mode "errors" turned out to be HTTP 502 from vLLM upstream,
not parse failures — clustered, all on the JSON-mode pass, plausibly
correlated with `guided_json` stressing the grammar engine.
Two improvements based on findings:
(1) OpenAICompatibleClient grew retry on transient 502/503/504 with
exponential backoff (0.5/1/2s, 3 attempts default). Network-layer
errors (ConnectError, ReadTimeout, RemoteProtocolError) get the
same retry. Smooths over the cluster without changing semantics:
persistent failures still raise, transient bursts no longer
dominate the error column. Helps all modes; JSON benefits most.
(2) Pointer-cap behavior changed from hard SCHEMA_INVALID to
trim-and-verify. When `[E2,...,E14]` over-cites a single claim,
keep first N pointers, verify normally, record
POINTER_OVERFLOW_TRIMMED in violations. STRICT becomes unreachable
(audit_mode caps at HYBRID) so the over-cite pattern stays
surfaced — but a correct claim like "Leonardo da Vinci painted
the Mona Lisa." no longer gets nuked for cosmetic over-citation.
Pre-fix: pointer mode hit 0/3 STRICT on Mona Lisa (mega-bracket
triggered SCHEMA_INVALID). Post-fix: HYBRID 2/14 with the right
answer rendered alongside both kept source spans. The dropped
pointers count toward n_quotes so the denominator surfaces the
over-cite to the auditor.
Bench scaffolding: ANSWER_MODES tuple now includes "claim_lattice"
(JSON), Makefile default sweeps all three. Question set expanded
from 8 to 22 covering narrow factoid, broad descriptive, entity
list, relationship, comparison, niche, adversarial, out-of-corpus.
460 tests pass. Connecticut output stays clean (HYBRID 4/7); JP-
dinosaurs pointer mode still UNGROUNDED via the bare-name guard
(model emits one-token entity names, the right floor catches them).
Round-tripped end-to-end against Hermes-3-8B:
query "what dinosaurs were in the first jurassic park film?"
--answer-mode claim_lattice
-> HYBRID 5/5 verified via claim_lattice in 4.9s
Hermes packed all five species into a single claim with five
evidence_ids, tripping TOO_MANY_EVIDENCE_IDS (max 2/claim). The
substrate's hard rule kicked in correctly — per-id checks all passed
but claim under-decomposition demoted STRICT to HYBRID. Reasoner-class
models (Qwen 3.6, Claude, GPT-4) should decompose into one claim per
species and clear STRICT.
Three layered fixes for the JP-dinosaurs lazy-anchor failure mode where
"what dinosaurs were in the first jurassic park film?" returned HYBRID
5/8 with citations to a video-game tie-in chunk (Operation Genesis):
1. Bare-name claim guard. New policy field
`claim_lattice_min_claim_content_tokens` (default 2). Claims with
fewer content tokens (>=4 chars, post-spotlight-stopword) classify
as SCHEMA_INVALID. "Triceratops. [E16]" had 1 content token; lexical
overlap on a single token is too thin to verify "X is in the FIRST
film" semantics. Forcing sentence shape raises the coverage bar so
off-topic chunks can no longer satisfy the citation. Threshold of 2
keeps narrow factoids viable (e.g. "Steve Jobs co-founded Apple"
tokenizes to 3 content tokens after stopword strip).
2. Smell → demote. The lazy-anchor smell sidecar was advisory only;
the verdict could be STRICT while every claim cited the same
magnet chunk. New policy fields
`claim_lattice_lazy_anchor_demote_threshold` (0.5) and
`claim_lattice_lazy_anchor_demote_min_pairs` (3) cap audit_mode at
HYBRID when ratio + pair-count both met. STRICT now requires
diverse anchoring across pointers; emits a LAZY_ANCHOR_DEMOTE
violation record so the auditor can see what fired.
3. Noisy title markers extended. `_NOISY_TITLE_MARKERS` now includes
"operation genesis", "the game", "video games" so spinoff/tie-in
articles (Jurassic Park: Operation Genesis, X (NES game), etc.)
classify as noisy_background_source. Default
claim_lattice_allowed_source_roles excludes that role, so noisy
chunks never enter the evidence map for pointer-mode queries.
Brittle (case-by-case markers) but addresses the JP failure
directly.
Live JP-dinosaurs after fix: UNGROUNDED 0/11 (was HYBRID 5/8 with
bogus citations). Honest refusal beats false confidence — model could
recover by writing sentence-shape claims; bare-name shortcut blocked.
Test fixtures using minimal placeholder claims ("Trex appears. [E1]")
opt out of the bare-name check via `min_claim_content_tokens=0` since
those tests exercise orthogonal behaviors. 460 tests pass.
JSON mode (answer_mode="claim_lattice") is now reachable end-to-end via
runner.ask: message build, optional vLLM guided_json extra_body, verifier
dispatch, and providence_cache persistence. Verifier returns
verifier_method="claim_lattice" (same as pointer variant) so both modes
share the existing CHECK constraint; answer_mode on the run-DAG and
json_fixups on the verdict disambiguate.
Pointer verifier now splits EVIDENCE_LINKED_PARTIAL into its own
partially_verified_quotes bucket so a claim never renders as both a
verified bullet and an unverified footer. CLI shows a "partially
grounded" section between the verified body and the unverified list.
Removes the strict no-double-quote check (`MANUAL_QUOTE_VIOLATION`) from
verify_claim_lattice. The rule was rejecting factually correct,
source-grounded claims for cosmetic punctuation: Hermes-3-8B paraphrases
prose but copies named-quoted phrases verbatim from source (e.g.
`"Constitution State"` lifted from a Connecticut chunk). Pre-fix,
`make query Q="tell me about connecticut"` reported `0/13 verified` on
a paragraph where every claim was correct and source-supported, just
because the model preserved source quote marks.
The two checks that remain handle the cases the manual_quote rule was
nominally meant to catch:
- claim_lattice_min_citation_coverage (Rule 5, default 0.30)
catches lazy-anchor citations whose only overlap with the cited
span is a single topical token
- claim_lattice_max_pointers_per_claim (Rule 6, default 2)
catches the encyclopedic-mega-claim where the model emits
`[E1,E2,...,E26]` after one sentence
Bench n=3 across 8 fixed questions vs pre-G0-hardening baseline (also
n=3): pointer-mode grounded count (STRICT+HYBRID) goes 15 → 21,
UNGROUNDED 9 → 3, broad-descriptive failures (connecticut, python)
fully cured. STRICT count goes 12 → 9 because previously-bogus STRICTs
(microsoft cited transit chunks that just shared one token, supermans
lazy-anchor magnet at 32/32) are now honestly reclassified to HYBRID.
Side changes:
- Ports max_pointers / min_citation_coverage policy fields and
verify_claim_lattice call into runner.py so single-document `ask`
matches multi-source `query` semantics (query.py already had them
via b39e79b).
- Worked Example 2 (broad-descriptive) added to pointer-mode prompt
so "tell me about X" has a template, plus Rules 7 (anti-echo) and
8 (max-2-pointers).
- Two manual_quote tests in tests/test_claim_lattice.py inverted to
document the new behavior (quotes-in-claim no longer block).
- `_has_manual_quote` retained — still used by verify_claim_lattice_json.
459 tests pass.
Adds bench/qa_sweep.py + bench/qa_questions.txt + `make bench-qa` target.
Sweeps a fixed question set through quote and claim_lattice_pointer modes,
N samples per cell (default 3), each sample burns the cached record so
Hermes nondeterminism becomes the variance source.
Outputs:
bench/qa_results/<utc>.jsonl one row per (mode, question, sample)
bench/qa_results/<utc>.md summary + per-question vote counts
Question set spans the failure-mode shapes the verifier needs to handle:
narrow factoid, broad descriptive ("tell me about X"), entity list,
relationship, and out-of-corpus (should land UNGROUNDED honestly).
Methodology gap this closes: at n=1 the strict-rate swings ±20pp on a
fixed prompt purely from sampling noise. n=3 makes 5pp deltas legible.
Brings the JSON variant back as a third ANSWER_MODE, distinct from the
pointer variant. The substrate exposes both: pointer for prose-
distribution / small-model paths (Hermes-3 8B), JSON for grammar-
constrained / large-reasoning-model paths (vLLM guided_json,
Claude/GPT-4 native JSON, Qwen 3.6 reasoner). Agents pick by setting
`policy["answer_mode"]`; both fold into governance_policy_hash so
records under different modes never alias.
Components:
- aborist/qa/verify.py:
* ANSWER_MODES = ("quote", "claim_lattice_pointer", "claim_lattice")
* `_lenient_json_parse(raw)` — defensive pre-parser. Strips markdown
fences, trims preamble/suffix to {/}, normalizes curly quotes,
fixes trailing commas. Returns (parsed, fixups[]) so the verifier
logs which drift had to be peeled. Lenient on syntax, strict on
semantics: parsed JSON still has to schema-check.
* CLAIM_LATTICE_JSON_SCHEMA — JSON Schema for the {"claims":[...]}
shape. Used by vLLM guided_json sampling-time constraint.
* `verify_claim_lattice_json(answer_json_text, evidence_map, ...)` —
runs the same hard checks as verify_claim_lattice (evidence_id
resolves, source_role allowed, no manual quotes, claim text non-
empty, claim textually overlaps evidence) but on content-
addressed evidence_ids directly. Returns the same verdict shape
plus a `json_fixups` list.
- aborist/qa/client.py: ChatClient Protocol & OpenAICompatibleClient
gain optional `extra_body` kwarg. Forwarded as additional fields in
the JSON request payload — opaque pass-through for vLLM-specific
knobs like `guided_json`. Endpoints that don't recognize a key
silently drop it. StubClient ignores; tests inspect via self.calls.
- aborist/qa/evidence.py: `render_evidence_block_for_json` and
`render_evidence_map_for_json` — JSON-mode prompts label blocks with
the content-addressed evidence_id (long hex) since that's what the
model cites in its JSON. Pointer mode keeps using the short
pointer_id.
- aborist/qa/query.py:
* Imports the JSON verifier + schema + JSON-mode evidence renderer.
* Message-build branch: `elif answer_mode == "claim_lattice"`
builds the same per-chunk evidence map as pointer mode, but
uses `claim_lattice_json_system_prompt` and labels blocks with
evidence_id.
* LLM call: when answer_mode=claim_lattice and policy
`claim_lattice_use_guided_json` is on, passes
`extra_body={"guided_json": SCHEMA}` so vLLM constrains output.
* Verifier dispatch: new `elif answer_mode == "claim_lattice"`
branch calls verify_claim_lattice_json.
* DAG persistence: lattice-mode raw_answer / parsed_lattice /
rendered_text threading now applies to both pointer and JSON.
* DEFAULT_QUERY_POLICY adds `claim_lattice_json_system_prompt`,
`claim_lattice_json_grounding_reminder`, and
`claim_lattice_use_guided_json` (default True).
- tests/test_verify_json.py: 14 tests covering the lenient parser
(strict pass-through, fence strip, preamble trim, curly-quote
normalize, trailing-comma fix, multi-fixup, hard-fail) and the
JSON verifier (STRICT on resolved claims, HYBRID on partial,
UNGROUNDED on schema invalid, fence recovery, manual-quote
violation, source-role block).
The whitepaper rewrite to 13.9.1 (substrate exposes both modalities,
both first-class) becomes accurate post-ship — JSON mode now exists
in code as it always existed in the architecture's intent.
554 tests pass.
Two layered fixes that close the lazy-anchor gap. Title-purity
moves the right SOURCES to the top of retrieval; the new hard
check makes the verifier ruthlessly honest about whether each
cited chunk actually mentions what its claim says.
1. _rerank_by_title_purity. New stage at the end of _rerank().
purity = |title_tokens ∩ query_tokens| / |title_tokens|; score
multiplied by (1 + 2*purity), so a title that IS the topic
("Jurassic Park (film)" against the dinosaur question) gets
3.0x and titles that drag in off-topic tokens like
"Jurassic Park: Operation Genesis" only get 2.0x. Composes
with the role-rank weights from the prior commit. Florida
query: list-pages dropped out of top-8 entirely. JP-dinosaurs
query: Operation Genesis sank from #4 to #8.
2. CITATION_MISMATCH hard check. Sixth deterministic check in
verify_claim_lattice — for each (claim, pointer) pair, at
least one content token from the claim text must textually
appear in the cited evidence span. Lexical only, no NER, no
embeddings; same _content_tokens extractor as the spotlight
so the proof-path stays on the soft/hard boundary. Catches
the magnet-chunk lazy-anchor where the model cites a chunk
whose text contains zero overlap with what its claim asserts.
New status CITATION_MISMATCH joins the per-claim taxonomy.
New violation kind CITATION_MISMATCH joins the run-DAG verify
payload. JP-dinosaurs verdict went from "13/14 verified
ratio 0.77 (cite E13 for everything)" pre-check to "1/14
verified" post-check — only Tyrannosaurus Rex's claim
actually overlapped its cited chunk. The architecture
stopped lying about evidence-linkage that the cited spans
don't textually support.
Pure-stopword claims pass the overlap check vacuously (no
content tokens means nothing to check against) — other hard
checks own that case.
Tests: +3 covering CITATION_MISMATCH (claim-token absent yields
fail, partial overlap yields HYBRID with EVIDENCE_LINKED_PARTIAL,
pure-stopword claim passes vacuously). Updated 7 prior tests
that used colloquial stub claims ("Trex") which don't textually
appear in the sample spans ("Tyrannosaurus rex"); they now use
claim text that overlaps the source content. 445 tests passing,
all 7 production shards report 0 chain breaks.
Three fixes that meaningfully improve claim_lattice_pointer behavior
on real queries.
1. Prompts in positive form. Hermes-3-8B (and instruction-tuned 8Bs
in general) follow positive directives ("do X") much more
reliably than negations ("don't do Y"). Rewrote the
claim_lattice_system_prompt and claim_lattice_grounding_reminder
in DEFAULT_POLICY and DEFAULT_QUERY_POLICY so every rule says
what TO do — "Reference evidence by pointer ID", "Use only
pointer IDs that appear in the EVIDENCE blocks", "Cite 1 or 2
pointers per claim", "Stop when the evidence runs out — a short
answer is the right answer when only short evidence exists",
etc. The cite-count constraint (1-2 per claim) directly
addresses the spray-anchor failure where Hermes attached every
available pointer to one claim line ("where is florida"
produced 1 claim with 33 cites pre-fix).
2. Source-role rank boost. _rerank() now ends with a new
_rerank_by_source_role stage that classifies each hit's role
(mutating h.source_role for downstream reuse) and rescales the
score by SOURCE_ROLE_RANK_WEIGHTS — primary 2.0, secondary 0.7,
noisy/sequel 0.3. The Florida defect: list-pages
(List_of_State_Roads_in_Florida, List_of_places_in_Florida:S/C/B,
Florida_locations_by_per_capita_income) all match
_SECONDARY_TITLE_MARKERS for "list of" and now sort below the
actual Florida article instead of dominating the top-8 by body
density. Affects ranking for both quote and pointer modes; the
earlier per-source budget weights stay (still primary gets 2x
the cap), they just don't have to fight a sort order that put
list-pages first.
3. Lazy-anchor smell line gates on n_verified >= 3. With one
verified pair the ratio is trivially 1.00 ("1 of 1 verified
pairs cite [E11]") which is vacuous. Below 3 pairs the metric
has no comparison surface; suppress the warning in those cases.
Live results on the 'where is florida' query went 33/33 with
cite-spray to 1/1 with a single targeted citation; spotlight
now hits "...largest metropolitan area in the state as well as
the entire southeastern United States is the South Florida..."
instead of state-road-number tables. The JP-dinosaurs benchmark
went from 11/11 with all cites at [E1] (ratio 1.00) to 13/14
with cites distributed across [E8], [E9], [E13] (ratio 0.77),
spotlight finding film-specific spans like "the film's
Dilophosaurus stands about 1.2 meters (4 ft) tall."
442 tests still passing. All 7 production shards report 0 chain
breaks.
verify_claim_lattice now derives pointer_id_distribution and
lazy_anchor_ratio from the verified-or-partial claim_statuses and
returns them in the verdict. Surfaced through ask()/query() result
dicts; CLI human-render displays a one-liner when ratio >= 0.5
(threshold for "model is lazy-anchoring").
Sidecar discipline preserved: never persisted in providence_cache,
never threaded into build_run_dag's verify_payload (test pins this
invariant — sidecar values must not enter run_dag_root). The
distribution is recoverable from claim_statuses which IS persisted,
so future sidecar tools can recompute on demand without enlarging
the audit chain.
JP dinosaurs benchmark currently triggers the warning at 11/11
ratio 1.00 — Hermes-3-8B cites [E1] for every claim even with
G0.3's relevance ranking putting the dinosaur-mention chunk near
the top. The render-layer signal now makes that lazy-anchor
behavior explicit instead of leaving it for the auditor to spot
from the spotlight excerpts.
Tests: +4 covering the ratio across distributions (single-anchor /
diversified / no-verified-pairs) plus the run_dag isolation
invariant. Total 442 passing.
Architecture diagram gains aborist/qa/evidence.py and
aborist/qa/parse_claims.py; aborist/qa/dag.py callout makes the
quote-vs-pointer DAG-shape split explicit. New conventions entry
covers the full G0 contract — two-layer ids (pointer-id E1/E2
shown to the model, evidence-id E######## used by cache & DAG),
pointer-line protocol, deterministic verifier (no entailment in the
proof path), strict no-quotes rule, spotlight rendering with
literal-span interpolation, per-chunk evidence granularity with
query-relevance ordering, 9-stage CTI run-DAG, no iterative repair,
and the Makefile testing default vs library defaults split.
New policy["answer_mode"] = "claim_lattice_pointer" (default "quote").
The runtime builds an evidence map with two-layer ids — pointer-id
(E1, E2, ...) shown to the model, sha256-derived evidence-id used by
the cache & run-DAG — and the model emits pointer-line prose
("Claim. [E12]") referencing them. Renderer interpolates literal
source spans at display time. Synthetic-elision-by-construction-
impossible: the model never types the quote string.
Pieces:
- aborist/qa/evidence.py (new): EvidenceObject + spotlight excerpt
(claim-token-centered window into the cited span, falls back to
leading window when no token matches).
- aborist/qa/parse_claims.py (new): pointer-line parser walks
lines, pulls [E\d+] / [E\d+,E\d+,...] tags, returns ParsedClaim
with PARSED / NO_EVIDENCE_POINTER status. Strict regex refuses
fuzzy alternatives so honest UNGROUNDED beats lax acceptance.
- aborist/qa/verify.py: verify_claim_lattice. Hard checks only —
parser succeeded, evidence_id resolves, source_role allowed, no
manual quotes (any " char violates), claim text non-empty.
Returns evidence_id_pairs (content-addressed, run-stable) for the
run-DAG. Soft signals (entailment, completeness, predicate
compatibility) stay sidecar.
- aborist/qa/dag.py: 9-stage CTI shape when evidence_map_root is
supplied — question / retrieval / evidence_map / prompt /
raw_answer / parsed_claim_lattice / verify / render /
final_label. Quote mode keeps the original 7-stage shape so
pre-G0 run_dag_root values stay valid.
- aborist/qa/query.py: per-chunk evidence-map build with role-
weighted budget AND query-relevance ordering. Within each source,
chunks are ranked by (distinct_query_tokens_present,
total_mentions, doc_order_asc) so the chunk most likely to
textually support the question gets the lowest pointer id. Without
this re-rank Hermes-3-8B lazy-anchored on doc-order-first chunks
regardless of relevance.
- aborist/qa/runner.py: same answer_mode branch for the single-doc
ask() path. No iterative repair in pointer mode (one-shot
benchmark discipline).
- aborist/store.py: verifier_method CHECK extended with
'claim_lattice'. New _rebuild_providence_cache_claim_lattice
migration preserves run_dag_root / run_dag_blob across the
rebuild — older rebuilds dropped them.
- aborist/cli.py: --answer-mode {quote,claim_lattice_pointer} on
query and ask.
- Makefile: ANSWER_MODE knob; defaults to claim_lattice_pointer for
`make query` so the testing harness exercises the new path.
Library DEFAULT_POLICY / DEFAULT_QUERY_POLICY stay "quote" so
Python callers and unit tests aren't surprised.
Prompt: one-shot worked example (Apple founders) plus strict
no-quotes / no-JSON / no-markdown / plain-prose-only rules. Without
the worked example Hermes-3-8B drops the bracket protocol on roughly
half of runs; with it, JP dinosaurs benchmark went 0/13 -> 10/11 ->
16/17 verified across the iterations that hardened the pipeline.
Tests: +48 covering parser, two-layer ids, verifier failure modes
(UNKNOWN_EVIDENCE_ID / SOURCE_ROLE_BLOCKED / MANUAL_QUOTE_VIOLATION
/ NO_EVIDENCE_POINTER / SCHEMA_INVALID), spotlight rendering with
buried-term fixture, per-chunk evidence map, query-relevance
ordering, schema migration round-trip, 9-stage DAG shape divergence.
Total 438 passed, all 7 production shards report 0 chain breaks.
Known limitation: chunk boundaries can cut wikitext mid-template, so
mwparserfromhell-backed to_base() leaves orphan </ref> tags and
leading list markers in the visible spotlight excerpts. Verifier and
CTI architecture are unaffected; the leak is cosmetic. Proper fixes
are template-aware chunking (chunker bump invalidates prior records)
or an orphan-marker post-strip in aborist/wikitext.py — both out of
G0 scope.
burn-kindergarten flake fix:
test_burn_kindergarten_zero_seconds_burns_everything failed under
full-suite ordering when the wall-clock second rolled over between
seeding the row & running burn-kindergarten. Old impl computed
`cutoff = now - 0 = now` and required `created_at >= now`, so a row
seeded at second T became invisible to a burn check that ran at T+1.
The verb's docstring is "0-second window = burn every live row".
Fix the impl to match: when kindergarten_seconds <= 0, drop the time
gate entirely (burn all live records). Non-zero windows keep the
existing cutoff-based filter unchanged.
Repair CLI:
`aborist query --repair` enables the mechanical repair pass after
first verify. `--repair-reprompts N` adds the optional re-prompt
tier (default 0 = off; mechanical-only when --repair is set without
--repair-reprompts). Both flags are off by default so `make query`
stays single-shot unless the operator opts in.
Makefile: `make query REPAIR=1` flips on mechanical repair;
`make query REPAIR=1 REPROMPTS=1` adds one re-prompt iteration.
Both knobs off by default to preserve existing behavior.
485 tests pass under full-suite ordering (was 484 + 1 flake).
Two enhancements continuing the toy-Hermes design pass:
#1 Chain-segment failure localization
aborist/qa/dag.py: `localize_failure(audit_mode, n_sources, n_quotes,
n_verified)` maps a non-STRICT verdict to the pipeline stage that
introduced the failure:
retrieval no admitted sources (gate over-rejected, or corpus
genuinely lacks the topic) → ingest more / relax breadth
context sources retrieved but no quotes extracted (per-source
cap dropped relevant content, or model declined to cite)
→ raise cap / tighten prompt
answer quotes extracted but didn't all verify (model fabricated,
paraphrased inside quotes, appended citations) → mechanical
+ re-prompt repair targets exactly this case
`failure_stage` lands on the run_dag's verify node payload AND on the
result dict so an operator can read the reason at a glance — `failure_stage='answer'`
means stop tuning the verifier & fix the model behavior. Debugging
becomes typed instead of vague.
#2 Re-prompt repair (second tier of the hybrid loop)
aborist/qa/repair.py: `reprompt_repair(...)` builds a feedback message
naming the failed quotes & asks the model to rewrite using only
verbatim citations. Hard rule: only fires when
`policy["repair_max_reprompts"] > 0` (default 0); caller enforces the
cap by looping at most that many times.
aborist/qa/query.py + aborist/qa/runner.py: after mechanical repair,
if the answer is still HYBRID/UNGROUNDED with unverified quotes,
loop up to `repair_max_reprompts` times. Each iteration: build
feedback (assistant turn with current answer + user turn with failed
spans), call LLM, verify. Accept the new answer if `n_verified`
strictly improved; otherwise break (the model's not converging,
don't waste cycles).
The mechanical + re-prompt combination handles the cases each tier
declines individually:
mechanical alone: synthetic_elision split, trailing_artifact trim,
no_overlap remove
+ re-prompt: paraphrase, partial_paraphrase, interior_elision
needing semantic judgment, fabrications the
model can recognize when shown its own quote
`repair_max_reprompts` lives in DEFAULT_QUERY_POLICY +
DEFAULT_POLICY so it folds into governance_policy_hash. Default 0
preserves single-shot semantics for callers that don't opt in. Each
re-prompt iteration adds a `{action: reprompt_rewrite, diagnosis:
model_feedback_loop}` entry to repair_changes; audit chain captures
the full transition through the existing providence_repair event.
Tests:
- dag: localize_failure across all four cases (STRICT, retrieval,
context, answer); failure_stage embedded in run_dag verify node.
- repair: stub client with sequenced answers (failing first, clean
on re-prompt) — assert two LLM calls, STRICT verdict, reprompt_rewrite
in the change log.
484 tests pass (dag +5, repair +1). The pre-existing test_burn flake
under full-suite ordering remains; passes in isolation.
The system was observational: verifier classified, sidecar diagnosed,
repair plans were emitted — but no loop ever closed. This adds the
hybrid repair stage from the toy-Hermes design pass: mechanical first
(deterministic string substitution from sidecar suggestions), behind
a `policy["repair_enabled"]` flag (off by default). Re-prompt fallback
is TODO.
aborist/qa/repair.py: `mechanical_repair(answer, unverified_quotes,
context)` walks each unverified quote through the sidecar classifier &
applies its `repair` action by string sub:
synthetic_elision_inside_quote (both halves verbatim)
`"prefix [...] suffix"` → `"prefix" ... "suffix"`
Two verbatim spans the verifier can independently check; the
model's [...] ellipsis-marker becomes prose between them.
trailing_artifact
`"prose. (Source: ...)"` → `"prose."`
Verbatim prefix kept; model-appended tail dropped.
no_overlap
Drop the line containing the bad quote entirely.
Skips include_aside_for_verbatim (needs precise source-span extraction;
defer to re-prompt path), paraphrase / partial_paraphrase (need prose
rewriting). Idempotent.
aborist/qa/query.py + aborist/qa/runner.py: optional pass after first
verify. When `repair_enabled=True` AND `audit_mode != "STRICT"` AND
unverified quotes exist:
1. Run mechanical_repair on the answer text.
2. If repair produced any changes, re-verify the repaired text.
3. If post-repair verdict isn't worse (n_verified didn't decrease),
accept the repair: persist the REPAIRED answer text instead of
the model's original. Cache_key inputs unchanged.
4. Audit chain gets one `providence_repair` event with the change
log + pre/post verdict so the original→repaired transition is
reconstructable.
Result dict gains `repair_changes` (list of change records) and
`pre_repair_audit_mode` (what the original was classified as).
`policy["repair_enabled"]` enters governance_policy_hash so on/off
agents share no cache silos.
Tests:
- mechanical_repair on each diagnosis (synthetic_elision, trailing_artifact,
no_overlap), idempotence on clean text.
- query() integration: repair_enabled=False (default) leaves answer
text unchanged; repair_enabled=True promotes a HYBRID/quote
synthetic_elision case to STRICT/quote, persists the repaired text,
emits the providence_repair audit event.
479 tests pass (+6 repair).
Each query/ask call now emits a 7-stage Merkle-DAG fingerprint stored
alongside the providence record. F from the toy-Hermes design pass.
Stages, in order:
question hash of question_hash (the 8-dim cache_key dim)
retrieval hash of sources summary (document_roots + roles +
scores + chunk_idx) — captures which docs ranked
context context_root (the source-Merkle for the assembly)
prompt conversation_hash
answer sha256(answer_text)
verify hash of verdict (audit_mode, verifier_method,
n_quotes, n_verified, claim_statuses)
final_label hash of (audit_mode, verifier_method, lookup_path)
run_dag_root = MerkleTree over those stage hashes (aborist conventions:
non-commutative HashCombine 0x03, leaf prefix 0x00, self-dup odd rule).
run_dag_blob = canonical JSON of {root, nodes} so an auditor can
recompute & verify (`verify_run_dag(blob)` returns True/False).
The DAG is NOT in cache_key. cache_key inputs determine the answer; the
answer determines the DAG — folding it back would create a cycle.
Instead it rides alongside as a per-record computation fingerprint.
Distinct from the linear `audit_events` chain (which tracks DB-wide
state changes); this is per-run computation provenance.
Schema: ALTER TABLE providence_cache ADD COLUMN run_dag_root TEXT;
ADD COLUMN run_dag_blob TEXT;
Idempotent migration in `_migrate_audit_mode`. Both rebuild templates
(VISUAL→UNGROUNDED dance, paraphrase verifier_method dance) updated to
include the new columns. Legacy records pre-2026-04-30 carry NULL.
Result dict gains `run_dag_root` so callers can verify without a DB
round-trip.
Tests:
- test_dag.py (9 tests): determinism, reactivity to each stage's input,
fixed stage order, round-trip verify, tamper-detection, JSON-string
acceptance.
- test_query.py: persistence on record + result, verify_run_dag round-
trip on the persisted blob.
473 tests pass (DAG +9, query +1, integration unchanged).
Two paired enhancements (D + E from the toy-Hermes design pass):
D. Per-claim status taxonomy on `verify_quotes`.
New `claim_statuses` field on every verdict — a per-evidence-unit
list with three labels:
VERIFIED_QUOTE unit substring-matched in normalized context
(any of quote/span/entity strategies)
SUPPORTED_PARAPHRASE unit cleared the paraphrase token-coverage
threshold (≥85% topical tokens present)
UNSUPPORTED unit didn't match anything
Diagnostic labels (QUOTE_INTEGRITY_FAILED, SOURCE_MISMATCH,
FALSIFIED) stay in the sidecar / falsification machinery — the
binary-verifier discipline holds. Empty list when no evidence
was extracted at all (verifier_method='none'). Backward-compatible:
existing fields (audit_mode, n_quotes, n_verified, unverified_quotes,
verifier_method) unchanged; current callers ignore the new field.
E. Repair-action plans on sidecar diagnoses.
Each `_classify_span` diagnosis now carries a `repair` field with
a concrete suggestion the operator can act on:
synthetic_elision_inside_quote → split_into_two_quotes
(when both halves verbatim)
→ trim_to_verified_half
(when only one half verbatim)
→ remove_claim
interior_elision → include_aside_for_verbatim
(with the dropped aside text)
trailing_artifact → trim_trailing_artifact
(with the kept_prefix string)
paraphrase → downgrade_to_paraphrase
partial_paraphrase → split_or_remove
no_overlap → remove_claim
Read-only suggestions — sidecar still doesn't write to providence_cache
or audit_events. The repair stage is recommendation, not mutation.
Operator (or an automated repair pass) decides whether to act.
Human render in `aborist inspect` shows `repair: <action> (<reason>)`
under each diagnosis line.
Tests:
- verify: claim_statuses_quote_path_labels_each_unit (per-quote VERIFIED
/ UNSUPPORTED), claim_statuses_paraphrase_method_flagged,
claim_statuses_empty_when_no_evidence.
- inspect: repair_synthetic_elision_split_when_both_halves_verbatim,
repair_interior_elision_includes_aside, repair_trailing_artifact_trim,
repair_no_overlap_remove.
462 tests pass (verify +3, inspect +4).
Two enhancements from the toy-Hermes design pass (2026-04-30):
A. Synthetic-elision-inside-quote diagnosis (sidecar only).
Distinct from interior_elision (model dropped a `(...)` aside source
carries) — synthetic_elision is the model writing literal `[...]`
between fragments of a `"..."` span, signaling self-elision while
claiming verbatim citation. The verifier still rejects (binary
discipline holds), but `aborist inspect` now reports
`diagnosis: synthetic_elision_inside_quote` with prefix/suffix
presence flags so an operator can judge whether the elided middle
was benign. Probe runs first in the classify-span chain (more
specific than trailing_artifact / interior_elision / paraphrase).
Catches the Brachiosaurus case: `"The film centers on the fictional
Isla Nublar [...] Universal Studios..."` — both halves are in source,
but the literal `[...]` isn't, so substring match correctly fails &
the sidecar tells the operator why.
C. Source-role classification + role-weighted context budget.
`_classify_source_role(title, qtokens_stem)` tags each top-K hit:
primary_answer_source 2.0× cap strong title-stem overlap
secondary_context_source 1.0× cap "list of", "characters",
"franchise", "history of"
noisy_background_source 0.5× cap "score", "music",
"video game", "merchandise"
sequel_background_source 0.5× cap "lost world", roman numerals
background_source 1.0× cap default
Order matters: noisy/sequel/secondary markers fire before the
primary check so peripheral pages with strong title overlap (e.g.
`Jurassic Park (film score)` shares 3 stems with the JP-film query)
don't claim a primary slot.
Cap loop now applies role weight on top of the baseline
`max_context_chars / top_k`. Total context still bounded by the
running `char_budget` — weights just shift how the budget gets
divided so primary pages get more text & noisy pages less, fixing
the case where a `(film score)` page consumed a primary slot.
`source_role` is persisted on `_Hit` and surfaces on
`merkle_proof.sources[*].source_role` in the providence record so
inspect & audits can see which slot each source occupied.
Tests:
- inspect: synthetic_elision_caught (Brachiosaurus regression),
synthetic_elision_does_not_fire_when_source_has_brackets (false-
positive guard).
- query: role classifier matrix (primary / secondary / noisy / sequel /
background) on JP-film-style titles, role persistence in sources list.
455 tests pass (sidecar +2, query +2).
