Sprint 1b — replace the flat max_context_chars=60000 default with
per-mode budgets matching the peak bucket each mode achieved on
the 2026-05-02 bench:
quote → 24000 (peak 16-32KB)
claim_lattice_pointer → 24000 (peak 16-32KB)
claim_lattice (JSON) → 48000 (peak 32-64KB)
The flat 60K was past peak for quote/pointer (which degrade after
32 KB) and only marginally above peak for JSON. The bench's
recommended-context-budget table now flows back into the policy
defaults — operator-driven landing per the five-step algorithm
step 5: surface, don't auto-apply. The mapping lives in
DEFAULT_QUERY_POLICY so any tuning bumps governance_policy_hash
and partitions the cache namespace cleanly.
API boundary stays backward-compatible: an explicit
max_context_chars= caller value still wins. The per-mode default
fires only when the caller passes None (or omits the arg).
CLI: --max-context-chars default flips to None; help text spells
out the per-mode fallback.
Smoke (5-question fixture) confirmed no regressions vs the
pre-change baseline:
quote 2/5 → 5/5 (+3)
pointer 0/5 → 1/5 (+1)
JSON 5/5 → 5/5 (flat — already at ceiling on this fixture)
Quote's +3 is the unexpected win: the smaller 24K budget surfaces
tighter-relevance top-K instead of drowning the model in filler.
The full bench (~50 min) is the real scoreboard but smoke shape
matches the bench's predicted lift direction.
Bench-max sprint 1a + sprint 3 + speed audit. Five wins, none of
them traded calibration.
UTF-16 surrogate fix (sprint 1a)
================================
Hermes occasionally emits text with lone UTF-16 surrogates. Bare
.encode('utf-8') raises UnicodeEncodeError on those, which aborted
the run with no Merkle root. Two errors per lattice mode in the
2026-05-02 bench were this exact path on the 'tell me about the
roman empire' question.
Fix: errors='surrogatepass' on the four sha256 helpers that hash
model-derived text, plus the two audit-chain encode sites in
store.py for defense-in-depth (audit body could carry user text in
some flows). The hash stays deterministic because WTF-8 bytes are
reversible & unique per input.
Touched:
aborist/qa/dag.py:_sha256_hex (the loud one)
aborist/qa/keys.py:_sha256
aborist/qa/evidence.py:_sha256_hex
aborist/store.py: chain_audit_events + append_audit
Predicted Δ on next bench: +1pp on lattice modes (the 2 errors
become valid runs).
Smoke fixture (sprint 3)
========================
bench/qa_questions_smoke.txt — 5 questions, all anchor classes,
each currently failing pointer mode 100% while JSON aces 100% per
the 2026-05-02 bench. Wired as 'make bench-qa-smoke', --n 1
--concurrency 4, ~30-90s wall-clock depending on vLLM warmth. The
inner loop for prompt iteration; the full 71-question sweep stays
the scoreboard.
Smoke verified: pointer=0/5 STRICT, JSON=5/5, quote=2/5. Confirms
the gap pattern from the journal.
Concurrency default
===================
Makefile bench-qa now defaults to BENCH_QA_CONCURRENCY=4 (was
sequential). Override via BENCH_QA_CONCURRENCY=N. Combined with
the --concurrency landing in 0870af6, full sweep drops from ~107
min projected to ~51 min actual.
pytest-xdist (test-speed)
=========================
Added pytest-xdist>=3.5 to dev extras. 'make test' now uses
-n auto (= one worker per logical CPU). Measured: 36s → 10s on
the 641-test suite. 3.6× speedup, no test changes required.
Bench-max scoreboard (predicted lift from this commit alone):
+1pp lattice modes (UTF fix)
+cycle-time enabler (smoke fixture, xdist)
no calibration cost — none of the verifier checks moved.
zstandard.ZstdCompressor and ZstdDecompressor instances each carry an
internal libzstd context that is NOT thread-safe. Calling .compress()
or .decompress() on a single shared instance from multiple threads
corrupts the context and raises:
ZstdError: decompression error: Data corruption detected
Surfaced when bench/qa_sweep.py learned a --concurrency flag and ran
4 (question, mode) cells in parallel. ~19% of retrievals failed on
zstd corruption before the fix. The prior comment claiming the
singletons were 'stateless across calls — safe to share across
threads' was wrong.
Replace the module-level singletons with threading.local() caches.
Each thread reuses its own ZstdCompressor / ZstdDecompressor; no
contention across threads. Init cost is negligible vs decompression.
bench/qa_sweep.py:
--concurrency N (default 1) parallelizes (question, mode) CELLS
using a ThreadPoolExecutor. Samples within a cell stay sequential
so burn-then-insert against a single cache_key never races itself.
Lock-protected JSONL writes & progress prints. Exception in any
worker surfaces via fut.result().
vLLM handles concurrent requests well; 4-8 is a reasonable starting
point. With --n 2 --concurrency 4, expect ~10 min wall-clock for the
full 426-run sweep against hermes.ai.unturf.com (vs ~2h sequential).
Phase 2 of the holistic-tuning pair. Replaces the un-indexable
LOWER(title) LIKE '%tok%' title-search with an FTS5 MATCH-based
lookup. The structural fix that was deferred when the >5-token
bypass landed (commit 1d70c4f).
(1) Schema: new `documents_fts` virtual table over the title column.
Contentless mode (same trick as chunks_fts) — stores only the
inverted index, not a copy of the title. Joins back to documents
via rowid for the post-filter caller.
(2) Extractor: backfill_documents_fts (registered as evidence_kind
"documents_fts"). Reset-and-rebuild the index from documents in
one INSERT...SELECT. Idempotent. Cost: ~2.5s per 870k-doc shard.
(3) `_search_titles` rewritten: try FTS5 MATCH first, fall back to
LOWER(title) LIKE only on shards lacking documents_fts data
(legacy ingest pre-this-commit). MATCH expression OR-joins
quoted query tokens; falls through to LIKE form on tokenizer
edge cases.
(4) Removed the `>5 accept_tokens` bypass in `_search_corpus`
that commit 1d70c4f added as a workaround for the LIKE
full-scan cost. With FTS5 the title search is sub-second
regardless of token count, so synonym-expanded title search
is affordable at any query length.
Live verified on the 19-token brain-tech query with 50 accept_tokens
(post-IDF expansion):
Pre-FTS5: ~50s (LIKE '%tok%' × 870k docs × 4 shards)
Post-FTS5: ~0.04s ← 1250× speedup
End-to-end query: 10.4s (was 11.6-13s; the saved title-search
time partially absorbed by Hermes inference variance).
Backfill cost: 10.5s wall-clock across 4 wiki shards + 1 crawl
shard. Storage: ~30 MB per wiki shard for documents_fts (well
inside the 90 MB/shard concept-layer budget). Re-running is
idempotent (DELETE FROM ... INSERT INTO ...).
Tests: 641 passed (no regression).
Phase 1 of the holistic-tuning pair fox requested 2026-05-02. Replaces
the dumb alphabetical-truncation heuristic in synonym_expand's cap
path with corpus-derived IDF ranking — rare topical synonyms win the
truncation, common-corpus noise drops.
(1) New per-shard table: concept_token_idf (token, doc_freq, total_docs,
derived_at). Indexed on doc_freq for ORDER BY ranking. Folds into
the cross-shard UNION view via _SHARDABLE_TABLES.
(2) Extractor: backfill_token_idf (registered as evidence_kind
"token_idf"). Reads chunk-frequency from FTS5's fts5vocab virtual
table, bounded by the union of token + target columns in
concept_relations (only synonym tokens get an IDF row, not the 6M
full corpus vocabulary). Idempotent INSERT OR REPLACE.
(3) Query layer: _load_token_idf sums per-token doc_freq across all
shards (cross-shard ranking). _get_indices return tuple grew from
(manual, derived, rivalry) → (manual, derived, rivalry, idf). Cache
LRU keyed on shards_dir mtime so the IDF lookup is per-process-once.
(4) synonym_expand cap-time truncation: when expanded set exceeds
MAX_TOTAL_TOKENS, sort neighbors by (doc_freq ASC, token ASC) so
rare tokens come first. Tokens missing from concept_token_idf get
a sentinel high-rarity score (treats unknown as hapax — fail-safe
when the IDF backfill hasn't run yet).
Live verified on the brain-tech 19-token query:
Pre-IDF expansion (alphabetical 50): bumper, buster, channel,
convention, dave, duck, esp, field, glen, mind, minimum, ...
Post-IDF expansion (rare-first 50): neuroimaging, neurons,
neuroscience, neurotechnology, psychokinesis, telepathic,
telepathy, thinking, tms, transcranial — all the brain-tech
terms that lost the alphabetical race in pre-IDF now win.
Backfill cost: ~50s wall-clock across 4 wiki shards (~22k tokens
indexed per shard). Fts5vocab over chunks_fts is the right
infrastructure — no full corpus scan needed since we filter by the
concept_relations token set. Storage: ~1 MB per shard for the IDF
table. Stays well inside the ~90 MB/shard concept-layer budget fox
set earlier.
Tests: 641 passed (no regression). Live fixtures (boss baby,
amazon river/rainforest) all pass — ranking doesn't break gating.
Two compounding things in one commit because they're tightly coupled
(the new violation is what makes the new fixtures meaningful):
(1) DEFLECTION_DETECTED — new soft-demote violation kind.
`aborist.qa.inspect.diagnose_deflection` was a read-only sidecar:
when the question's subject anchor (last content token after
stopword strip) is missing from the answer, it returned
`kind="deflection"`. Useful telemetry but never affected the
verdict. 2026-05-02 fox surfaced two cases where the missing
signal mattered:
Q: "who burns the amazon river?"
A: "Deforestation in the Amazon Rainforest is primarily caused
by human activities..." [E10 | Deforestation of the Amazon
Rainforest]
Pre: EVIDENCE-WARRANTED 1/1 ← silent river→rainforest topic shift
Post: ANCHOR-WARRANTED-PARTIAL 1/2 with DEFLECTION_DETECTED
The verifier now runs diagnose_deflection() after all hard checks.
On `kind == "deflection"` it appends a DEFLECTION_DETECTED
violation (rationale + subject_anchor + overlap_ratio) and caps
audit_mode at HYBRID. Renderer's four-rung ladder picks it up via
`_SOFT_DEMOTE_VIOLATION_KINDS` (cli.py) and demotes EVIDENCE-
WARRANTED → ANCHOR-WARRANTED automatically.
Folds into verifier_policy_hash via new
`claim_lattice_deflection_check_enabled: True` policy field
(defaults on). Bumps the 9-dim cache_key when toggled.
Wired into both verify_claim_lattice (pointer) and
verify_claim_lattice_json (JSON). Deferred import keeps verify.py
module-load clean.
(2) Three live fixtures gating the river/rainforest/deflection family:
- test_amazon_river_burning_who_deflects_or_grounds_river
Gates the silent topic-shift pattern: if model deflects, the
verifier MUST catch it (DEFLECTION_DETECTED present OR
UNGROUNDED OR "river" actually in answer). Failure mode this
catches: EVIDENCE-WARRANTED with rainforest answer to a river
question and no soft-demote.
- test_amazon_river_burning_culture_refused_or_ungrounded
Adversarial-impossible mirror of Mars-BDFL: rivers don't burn,
so model must refuse / deflect / refute. Failure mode:
STRICT/HYBRID with affirmative culture-burns-river claim.
- test_amazon_rainforest_burning_culture_grounds_or_deflection_caught
Feasible-but-framing-specific: corpus has deforestation
content but no "culture" framing. Either grounds with rainforest
content OR a soft-demote/UNGROUNDED catches the framing gap.
Live verified (2026-05-02): all 3 pass against the real Hermes
endpoint. 641 unit tests still pass (deflection check skipped on
fixtures lacking question or rendered_text).
The previous fix capped OR-mode at top-5 longest tokens which made
the long brain-tech query fast (118s → 13s) but lost the synonym
retrieval that surfaced Telepathy / Neurotechnology — those titles
weren't reachable via the original query tokens alone.
Fix: pass synonym-expanded tokens to the OR-mode fallback as
``extra_or_tokens``. The merged pool keeps the top-5-longest cap so
retrieval cost is unchanged, but long synonym tokens like
"neurotechnology" (15 chars) and "neuroimaging" (12 chars) now beat
shorter query tokens like "thoughts" (8) by length & surface the
right titles.
OR pool example for the brain-tech query:
Pre-synonym: reconstruct, technology, understand, available, thoughts
Post-synonym: consciousness, neuroimaging, clairvoyance, compensation, reconstruct
→ 3/5 brain-tech terms surface naturally in OR-mode.
AND mode stays unchanged (synonyms in AND would relax the strict-
relevance constraint & pull in noise — wrong tradeoff). The
synonym signal flows ONLY into OR-mode-fallback.
Live verified on the 19-token brain-tech query:
- 13s total (Hermes 6-8s + retrieval ~1s + verify ~2s)
- 2/2 verified pairs
- Cited Functional neuroimaging E5 + FreeSurfer E9 (both real
brain-tech corpus sources, not Universal-pragmatics nonsense)
`_search_corpus` recomputes synonym_expand(qtokens) once into
`or_synonym_pool` (caches in concepts.query._CACHE after first
shard); each shard's FTS5Backend.search() receives the same set
through `extra_or_tokens=`. Cost stays sub-second per shard.
Tests: 641 passed (no regression).
Both retrieval phases were doing O(corpus × tokens) full-scan work
on long queries — synonym work surfaced the latent issues but they
predate it. Hermes is 6-8s; retrieval needs to be sub-second.
(1) `_search_titles` (LIKE-based title backup search) is fundamentally
O(corpus × |accept_tokens|) — `LOWER(title) LIKE '%tok%'` can't
use any index. 50 patterns × 870k docs × 4 shards = ~56s pure
full-scan. Skipped for queries with > 5 accept_tokens, where
FTS5 BM25 already returns better candidates. Title-LIKE remains
a backup for short focused queries (1-5 tokens).
(2) FTS5 OR-mode fallback was OR-joining ALL post-stopword query
tokens. A 19-token query OR-clause matched millions of docs;
BM25 then ranked them all. 13s/shard observed.
Capped to top-5 LONGEST tokens (proxy for rarity / topical
specificity). Long words like "neurotechnology" matter; short
words like "soon" don't. OR-mode now 0.25s/shard.
(3) Stopword set extended with 6 connector words (one, some,
another, without, soon, currently). Conservative addition —
these have zero topical signal. Reduces the AND-mode token
count, raising the chance AND succeeds before falling back to
OR.
Verified live: the 19-token brain-tech query that hung indefinitely
now returns in 18s (Hermes + verify + render dominate). Source
`_search_corpus` no longer expands synonyms for the title-LIKE call
either — synonym expansion stays in `_filter_by_title_relevance`
(post-retrieval, in-memory, cheap regardless of accept-set size)
where it actually does useful work.
Tests: 641 passed (no regression).
Two compounding crashes/hangs in the corpus-derived synonym layer
that the bench surfaced 2026-05-02:
(1) Union-find chained reciprocal-link clusters into one giant
connected component (54,538 tokens for any seed in the 4-shard
Wikipedia corpus). One query token expanded to the entire
synonym alphabet, tripping SQLite's expression-tree-depth=1000
limit on the OR-clause in `_search_titles`. Replaced with
direct-neighbor adjacency only.
(2) Even direct-neighbor expansion was too noisy on generic tokens:
a 19-token query expanded to 419 tokens (every token had ~22
reciprocal-link neighbors averaging out to topic-adjacency
noise). Title-LIKE on 419 patterns × 3.47M docs × 4 shards
hung indefinitely.
Fix: split index by evidence_kind, cap derived expansion only.
- ``manual_index`` — manual_legacy + manual rows. Curated; ALWAYS
expand regardless of per-token degree. The brain-tech / AMD-
family / Mac / Linux / etc. seed groups have legitimately many
members per token after seed.py started writing clique edges.
- ``derived_index`` — link_reciprocity & corpus-extracted edges.
Subject to MAX_NEIGHBORS_PER_TOKEN=8 cap. Generic tokens
("person", "thoughts") have huge degree from Wikipedia link
noise; specific named entities have small focused neighborhoods
that pass the cap.
- ``MAX_TOTAL_TOKENS=50`` overall cap on expanded set. Bounds
the SQL clause count so title-LIKE search stays tractable.
seed.py: write CLIQUE edges within each legacy group (every
member-pair, not just anchor→member). Preserves the legacy
frozenset semantic where any member retrieves every other
member. Quadratic in group size but groups stay small (largest
is the 30-member brain-tech → 435 pairs).
Live shard 000 re-seeded: 80 → 747 manual_legacy rows. Idempotent
re-seed via INSERT OR IGNORE — re-running adds nothing new.
Verified:
- 19-token query: 419 → 50 expanded (cap saturated)
- thoughts: 30+ tokens incl. brain-tech members preserved
- athlon: full AMD-family clique (amd, duron, opteron, ryzen, …)
Tests: 14/14 concept tests pass.
Why: union-find over the full Wikipedia reciprocal-link synonym
graph collapses everything into one giant connected component
(54k+ tokens for any seed in a 4-shard corpus). That made one
query token expand to the entire synonym alphabet, which:
(a) overwhelmed retrieval relevance — every query pulled in
thousands of off-topic candidates via synonym chaining;
(b) tripped SQLite's expression-tree-depth limit when
_search_titles built a per-token OR clause (fox-surfaced
2026-05-02 long-question regression — the qa/query.py side
of that fix landed in faed58d via a separate OR-chain cap;
this is the concepts-side fix to the same underlying
explosion).
Direct-neighbor adjacency: each token maps to its immediate
synonym partners only, no transitive chaining. Symmetric — a
(A,B) edge registers both A→B and B→A. Preserves the legacy
frozenset semantics (every member of a group expanded to the
others in that group, but groups didn't chain).
Test impact: three test_concepts.py tests assert the old
transitive-closure behavior:
- test_synonym_expand_amd_pulls_athlon_and_back (expects
athlon → {amd, duron, thunderbird})
- test_mac_windows_rivalry
- test_brain_tech_synonym_group_includes_telepathy_and_thoughts
They fail under direct-neighbor expansion. Either update the test
fixtures to seed direct edges (athlon, duron) etc., or update the
test expectations to match the new direct-neighbor contract.
Deferred — design choice, fox to decide which path matches intent.
Regression fox surfaced 2026-05-02:
Q: "what technology are currently or soon available which may
enable one person to reconstruct and understand some or a
portion of another persons thoughts or ideas without
speaking or sign language."
→ sqlite3.OperationalError: Expression tree is too large
(maximum depth 1000)
Root cause: the v1 of _search_titles (commit 0052845) chained N
``CASE WHEN ... THEN 1 ELSE 0 END + ...`` expressions for the
title_score column. Each CASE WHEN is multiple tree nodes; +-
chained N times exceeded SQLite's default 1000-depth bound on
question texts with ~30+ content tokens.
Fix: simplified SQL — OR-chain WHERE + ORDER BY LENGTH(title) ASC
+ LIMIT bumped 4x to compensate for the lost smart sorting. The
caller's post-filter (word-boundary stem-aware token-set
intersect) does the actual title-relevance ranking; SQL just
needs to surface enough candidates for the post-filter to grade.
Also: cap the OR-chain at MAX_TITLE_LIKE_TOKENS=24 so pathological
200-token queries don't cascade SQL expression growth even
defensively. Beyond ~24 tokens the post-filter is doing all the
work; extra LIKEs just inflate candidate sets without signal.
5 new tests in tests/test_query.py covering the regression at
unit / integration / functional layers:
- test_unit_search_titles_handles_long_question_without_crash
50-token query through _search_titles directly. Pre-fix raised
sqlite3.OperationalError; post-fix returns row list.
- test_unit_search_titles_handles_zero_tokens
Defensive: empty token list → empty result, no SQL executed.
- test_unit_search_titles_caps_or_chain_at_max_tokens
200-token pathological query — bounded by MAX_TITLE_LIKE_TOKENS,
doesn't crash.
- test_integration_query_completes_on_long_question
End-to-end query() with StubClient + long question completes
without the SQLite error. Pre-fix raised before reaching the
LLM call.
- test_functional_long_question_returns_sources
The neurotech doc (richest body match) appears in top-K despite
the long-question retrieval path.
Test_query.py: 35 → 40 passing. Full suite (excluding parallel
test_concepts churn from concepts/query.py rewrite): no other
regressions.
Phase 2 of the concepts/ layer. The Apr 27 commit (c6182ae) shipped
hand-curated frozensets in aborist/qa/concepts.py with a TODO to
"derive from Wikipedia's category graph or 'See also' sections" —
that's this commit. Fox's 2026-05-01 critique landed it: a 7-entry
list of arbitrary frozensets won't scale to a 3.47M-doc corpus, &
adding domains shouldn't require a Python edit + commit + redeploy.
Architecture:
(1) Per-shard concept_relations SQLite table. Append-only, with
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
so re-derivation is idempotent. Lives next to documents in each
shard so mesh sync moves relations alongside the docs that
derived them. A SECONDARY index — writes here NEVER affect
document_root / chunk_root / cache_key, so backfilling is safe
across the entire corpus.
(2) aborist/concepts/ package:
- store.py: add_concept_relation, concept_relations_for_token,
purge_by_evidence_kind, list_evidence_kinds
- query.py: cross-shard synonym_expand, rivalry_excluded;
union-find collapse on synonym edges so partial
pairs build full equivalence classes; mtime-keyed
per-process LRU so retrieval doesn't re-walk
shards on hot loops
- seed.py: one-shot migration of legacy frozensets (8 groups
incl. brain-tech) to evidence_kind='manual_legacy'
rows under source_root='__legacy__concepts__'
- extract.py: pluggable extractor registry. Built-in:
link_reciprocity_synonym — for any reciprocal
edge pair (A→B AND B→A) in the existing edges
table, emit synonym edges between the docs'
title-tokens. Works for Wikipedia (See-also
bidirectional), HTML site internal-link clusters
(russell.ballestrini.net pattern), or any link
graph the corpus already encodes — no new
crawler needed; the html_page parser already
populates `edges` rows on ingest.
(3) aborist/qa/concepts.py rewritten as a backwards-compat shim —
same public API (synonym_expand, rivalry_excluded,
has_compare_phrasing) so query.py call sites unchanged.
shards_dir threaded through _filter_by_title_relevance &
_search_corpus' synonym_expand calls. Without shards_dir
(legacy 2-arg call shape), helpers degenerate to no-op —
matches the behavior the frozenset code had when no group hit.
(4) Cross-shard UNION view: concept_relations added to
_SHARDABLE_TABLES in store.py so connect_query() exposes a
unified view across all shards (same pattern as documents,
chunks, providence_cache, etc.).
Live-verified on the 4-shard 3.47M-doc corpus + the
crawl_russell_ballestrini_net.db shard:
Q: what technology are currently or soon available which may
enable one person to reconstruct and understand some or a
portion of another persons thoughts or ideas without speaking
or sign language?
Result: same as the bde1bd6 in-memory frozenset version —
Telepathy E4 cited alongside Videoconferencing E1/E2,
POINTER-LINKED-PARTIAL 6/10. The DB-backed lookup reproduces
the frozenset behavior byte-for-byte.
Tests: 633 passed (was 624). 9 new concept tests covering
DB-backed synonym/rivalry lookup, shards_dir=None degenerate
behavior, brain-tech group seed, cache invalidation. Old tests
that called helpers directly without shards_dir kept as no-op
assertions (synonym_expand({"athlon"}) without shards_dir returns
{"athlon"} unchanged).
Backfill mechanics: existing live shards needed a one-shot
`connect()` to auto-create the new concept_relations table
(SCHEMA_SQL has CREATE TABLE IF NOT EXISTS). Re-derivation never
mutates the Merkle tree — concept_relations is fully orthogonal
to document_root / chunk_root. Cache_key dimensions are
unaffected.
Deferred (follow-on):
- CLI commands: aborist concepts {seed,list,add,derive,purge}
(currently fox runs the helpers via python -c)
- Wikipedia See-also extractor (requires parsing wikitext sections
beyond what's already in edges)
- Wikipedia category extractor (requires reading Category: links
from chunked wikitext)
Promotes the title-relevance sidecar (diagnose_title_relevance,
qa/inspect.py) to a hard check inside both verify_claim_lattice
and verify_claim_lattice_json. New violation kind TITLE_MISMATCH;
demote-to-HYBRID semantics matching the existing WARRANT_MISSING
pattern.
Catches the 2026-05-02 fox-surfaced retrieval-driven hallucination
class:
Q: "explain spin glass modeling & tensors?"
Pre-Rule-8: EVIDENCE-WARRANTED 1/1, claim cited to Quantum
chromodynamics chunk (single-line "See Also: spin
glass" reference). Token-coverage check passed
accidentally on shared physics vocabulary.
Post-Rule-8: POINTER-LINKED-PARTIAL · title mismatch 1/1.
Cited source title 'Quantum chromodynamics' shares
zero stemmed tokens with claim's {spin, glass,
modeling, tensor, ...} → demote.
Implementation:
_claim_title_overlap(claim_text, source_title) returns True iff
the source title shares ≥1 stemmed content token with the claim.
Uses qa.evidence._content_tokens (≥4-char, post-stopword) and
inline minimal stem (s-strip on tokens >4 chars, skip ss-enders)
to avoid an import cycle.
Per-claim loop in both verifiers checks every cited source's
title; ANY-match suffices (only TITLE_MISMATCH when ALL cited
titles miss). Vacuous-pass when claim or title has no
extractable tokens.
Renderer (_ladder_rung_for_lattice) treats TITLE_MISMATCH
alongside WARRANT_MISSING as the POINTER-LINKED-triggering
signal — both indicate citation/claim structural misalignment.
_render_warrant_tail surfaces "· title mismatch" alongside
"· warrant missing" so an operator sees the specific failure
mode at the audit-line.
Test fixtures updated where the synthetic claims were too minimal
(e.g. "Velociraptors are shown attacking workers" cited to
"Jurassic Park (film)" — claim had no topic anchor). Real model
output naturally references the topic (the model sees the title
in the evidence map and uses it); the fixture revisions reflect
that. 4 new tests in test_verify_json.py covering the helper +
end-to-end TITLE_MISMATCH demote.
Live verification:
Spin-glass query: POINTER-LINKED-PARTIAL · title mismatch ✓
Homer/Mr. Burns: EVIDENCE-WARRANTED ✓ (no regression)
CLAUDE.md updated with the Rule 8 convention; existing
diagnose_title_relevance sidecar marked legacy / dict-form for
per-cache-key inspect use.
Full suite: 738 passed (was 734, +4).
Closes the 2026-05-01 retrieval gap fox surfaced on:
Q: what technology are currently or soon available which may enable
one person to reconstruct and understand some or a portion of
another persons thoughts or ideas without speaking or sign language?
Pre-fix: top-K = [Videoconferencing, Telepathy, Identity, PKM,
Intrapersonal]; only Videoconferencing used (BM25 won on
"person/language/speak" token density). Verifier returned
EVIDENCE-WARRANTED-PARTIAL 2/5 with Videoconferencing
cited as evidence for "currently no technology...". Telepathy
was retrieved but unused — title-relevance filter dropped
it to body-density accept path with a low score.
Post-fix: same top-K but Telepathy now SURVIVES title-relevance
(its title token "telepathy" is in the accept set via
synonym expansion). Title-token-boost lifts it.
EVIDENCE-WARRANTED-PARTIAL 6/10 with Telepathy E4 now
cited alongside Videoconferencing — sources used 5/5.
Adds a 7th SYNONYM_GROUPS entry mixing concept tokens (thoughts,
mind, cognition, consciousness) with specialized entity tokens
(telepathy, neurotechnology, BCI, fMRI, EEG, TMS, transcranial).
A query naming either side expands to the whole family, so the
title-relevance accept path admits brain-tech pages even when the
question is phrased in lay vocabulary.
Limitation accepted: this only affects title-relevance filtering &
title-token rerank, not the FTS5 BM25 retrieval that produces the
top-K. Pages whose BM25 score is below the top-K cutoff (e.g.
Neurotechnology at #4 in fox's manual-hint Run 2) still won't
surface unless FTS5-side synonym expansion is added too. Tracked
as a follow-on ticket — for now, query-time retrieval hints
("transcranial knowledge acquisition") continue to be the manual
escape hatch when intent vocabulary doesn't lexically anchor.
Tests: 624 passed (no regression).
Fox-surfaced 2026-05-02 case:
Q: "what date did back to the future come out?"
→ top-8 sources: trilogy, ride, animated-series, The Future,
Future Husband, The Day You Come, Ratchet & Clank Future,
Future plc — film article entirely absent.
Two compounding defects in the title-LIKE retrieval path:
(1) `_search_titles` had no ORDER BY. SQL `LIKE '%token%' OR ...`
returned rows in arbitrary internal order; LIMIT 32 truncated
before genuine matches. The film article `Back to the Future`
(title contains both `back` AND `future`) lost the LIMIT race to
substring-match junk like `Out (poker)` (single-token match on
`out`), `Aberdeen, South Dakota` (`south` contains `out`),
`Backplane` (`back` substring), etc.
Fix: SQL builds a token-hit score via CASE WHEN per token and
ORDERs BY (title_score DESC, LENGTH(title) ASC). The film
article's score=2 (back+future) ranks above garbage's score=1.
Tie-break on title length favors title purity (shorter title +
same overlap = higher per-token signal).
(2) Caller's post-filter check used substring match
(`if t in title_lower`) — same defect as the SQL. "south"
contains "out", "outline" contains "out", "backbone" contains
"back", etc. all passed the no-op overlap check.
Fix: tokenize the title via `_title_query_tokens` (already
drops stopwords + small tokens) and stem-aware-intersect with
the query stems. Word-boundary semantics; only genuine title
tokens count.
Verified live:
Pre-fix: trilogy at #1, film article absent from top-8.
Post-fix: film article at #1, trilogy at #2, junk gone.
Verdict on the BTTF query stays POINTER-LINKED-PARTIAL · warrant
missing because the model picked the box-office chunk (mentions
"1985" but not "July") instead of the infobox/release-date chunk.
That's correct warrant behavior: claim asserts both anchors;
cited span only has one. The retrieval-side defect was a separate
gap that this fix closed; chunk-relevance ranking for cited
content is downstream work.
The phrase-pattern retrieval route (commit 1b8677d) closed the
RETRIEVAL side of reference-frame failure; this ticket closes
the ANSWER side.
New module aborist/qa/frame.py:
FrameDetection dataclass (frame_kind, reference_title,
reference_uri, confidence). Sidecar — never enters cache_key
or governance_policy_hash.
detect_frame(question, sources, phrase_match_roots) — heuristic
detector. Reference-frame classification fires when:
(a) phrase route surfaced at least one source, AND
(b) at least one phrase-matched source is a reference work,
determined by:
- title parenthetical disambig (`(novel)` / `(film)` /
`(play)` / `(franchise)` / etc.), OR
- body sample contains ≥3 DISTINCT fiction markers
(novel / published / protagonist / plot / ...).
Distinct-marker count keeps the heuristic robust against a
history article saying "novel approach" twice.
aborist/qa/query.py:
Calls detect_frame for lattice modes only. Body sample uses the
ARTICLE LEAD (chunk_idx=0, post-wikitext-strip) — fiction
markers cluster in the lead on Wikipedia, not in plot chunks
that may have been query-relevance-ranked higher.
New policy field claim_lattice_polarity_preamble injected as a
user-role message before the grounding_reminder when
frame_kind == "reference". Format-string with
{reference_title} placeholder.
Result dict carries frame_detection (kind / title / uri /
confidence) for renderer + bench consumption.
aborist/cli.py:
Renderer adds a `reference frame: <title>` line when
frame_detection.kind == "reference". Skipped for literal /
no-phrase-route / ambiguous rows.
Live verification — Orwell case:
PRE : "The text does not directly state that Oceania has always
been at war with East Asia."
POST : "In George Orwell's dystopian novel Nineteen Eighty-Four,
the nation of Oceania is always at war with Eastasia,
but this is a result of propaganda and doublethink, not
actual historical continuity. The war with Eastasia is
a fabricated conflict to maintain control..."
Multi-frame compilation: distinguishes propaganda claim from
fictional-actual continuity, exactly the polarity contract from
the ticket §2 abstraction.
Literal queries (capital of france) unchanged — polarity preamble
only injects when frame_kind classifies as "reference".
22 new tests (19 in test_frame.py + 3 in test_cli_render.py).
Full suite: 734 passed (was 712, +22).
Directive D3 status: ½ → ✓. Ticket #000002 closed.
All seven structural directives now ✓:
D1, D2, D5 (were ✓);
D3 #000002, D4 #000001, D6 #000003, D7 #000005 (closed in this
series).
New module aborist/qa/retrieval_plan.py:
RetrievalPlan dataclass — frozen, captures the operator-
influenceable retrieval inputs that determine source
selection (retrieval_keywords, top_k, over_fetch,
max_context_chars, shard_ids).
retrieval_plan_hash() — SHA-256 over canonical-JSON.
Deterministic per call; folds into the run-DAG retrieval
stage as a bound input alongside the existing
sources_summary output.
build_run_dag (aborist/qa/dag.py) accepts retrieval_plan_hash:
When provided, the retrieval stage hash binds BOTH plan
(input) and result (output): H({"retrieval_plan_hash":...,
"sources_summary_hash":...}).
When omitted (e.g. legacy / quote-mode callers that haven't
plumbed it yet), the retrieval stage falls back to the
historical sources-summary-only hash so pre-#000001
run_dag_root values stay stable.
query.py constructs the plan per call and passes it through.
Question text intentionally NOT in the plan — already covered
by question_hash. Shard ids included so audit can reproduce
which shards the search ran against.
Two runs with identical sources but different retrieval keywords
now produce different run_dag_root values — the provenance gap
on operator hints (auditor recovers "these were the keywords
that pulled in those sources") closes for the run-DAG path.
Marker test in tests/test_directives.py flipped from "absent"
assertion to "present": test_d4_retrieval_plan_binding_landed.
Plus test_d4_retrieval_plan_hash_module_exists pinning the
module shape. Full suite: 712 passed.
Deferred per ticket §6:
- audit events (retrieval_plan_built / retrieval_result_selected)
- providence_cache.retrieval_keywords SQL column
- optional strict cache_key mode (retrieval_plan_affects_cache_key)
These are ergonomic affordances atop the run-DAG binding; earn
their own tickets when bench evidence shows the blob path is
too friction-heavy for real workflows.
Directive D4 status: ½ → ✓. Ticket #000001 closed.
Replaces the two-rung EVIDENCE-LINKED / EVIDENCE-LINKED-PARTIAL
display label for claim-lattice methods with a four-rung ladder
that names a strictly stronger property at each rung:
POINTER-LINKED pointer/source/chunk verified;
warrant either didn't apply or failed
ANCHOR-WARRANTED pointer-linked + warrant passed where it ran;
other soft demotes may apply
EVIDENCE-WARRANTED anchor-warranted + no soft demotes
UNGROUNDED no verified pairs
HYBRID gets a -PARTIAL suffix on whichever rung applies.
Implementation: _render_audit_label gains a violations parameter
(defaults to None for backward-compat). _ladder_rung_for_lattice
discriminates rungs from the existing violations list:
- WARRANT_MISSING in violations → POINTER-LINKED
- any of {LAZY_ANCHOR_DEMOTED, POINTER_OVERFLOW_TRIMMED,
TOO_MANY_CLAIMS, BARE_NAME_CLAIM} → ANCHOR-WARRANTED
- else → EVIDENCE-WARRANTED
Design simplification vs the ticket's §3 sketch: the proposed
verifier_steps_ran field on the verdict dict was NOT needed. The
existing violations list carries enough signal to discriminate
all rungs. Per the five-step algorithm step 2: don't add fields
you don't need.
Quote / span / entity / paraphrase methods stay unchanged (their
STRICT verifies pinned spans, not synthesis).
Schema column audit_mode enum stays {STRICT, HYBRID, UNGROUNDED}
— pure renderer transformation, no governance_policy_hash bump,
no cache invalidation, no mesh-wire-format change. Existing
providence records render under the new ladder on next read.
5 new renderer tests in tests/test_cli_render.py covering each
rung mapping. D7 anti-regression test in tests/test_directives.py
updated to gate on the ladder labels. Bench helper docstring
follows. Full suite: 711 passed.
Directive D7 stays at ✓; ticket #000005 closed.
Three new question-shape classes dispatched through warrant_check
alongside the existing relation + date anchors:
(1) Entity-list shape — `name X`, `list X`, `who are the members
of X`. List-aware extractor `extract_entity_list_anchors`
(multi-word phrases ∪ solo-cap individual names) so comma-
separated entities each contribute. ANY-match semantics:
demote-don't-reject when an extra entity from training-prior
appears alongside grounded ones.
(2) Count shape — `how many X`, `how much X`. Digit ↔ word
equivalence (claim says "six", span says "6", or vice versa)
with ordinal collapse (`sixth → 6`). Year-shaped digits
filter out (those belong to the existing date anchor class).
ALL-match semantics: every count token in the claim must
appear in some cited span as digit or word.
(3) Why-cause shape — `why X`. Cause-anchor pool widens to
≥5-char lowercase common nouns (post a generic stopword set
that filters quantifier-adjective fillers like "various",
"factors", "situation") PLUS proper-noun anchors from the
existing extractor. Gated on why-shape only: lowercase
common-noun extraction has higher false-positive risk
elsewhere.
Per-class policy gate (proposed `claim_lattice_warrant_classes`
dict) deferred per the five-step algorithm step 2: single
`warrant_check_enabled: bool` is the minimum viable gate; per-
class flags earn their slot when bench evidence shows over-firing
on a specific class.
17 new warrant tests (detector + extractor + integration).
Marker test in test_directives.py flipped from "absent" to
"present" assertion: test_d6_warrant_generalization_landed.
Full suite: 709 passed (was 692, +17).
Directive D6 status flipped to ✓ in seven-point-program.md.
Ticket #000003 closed.
Generalizes warrant-lite from relation-only to anchor-class warrant.
Two anchor classes now compose:
(1) Proper-noun anchors — gated on relation-question shape; any-match
semantics (existing behavior).
(2) Date anchors — always-on, gated on the CLAIM containing a year or
month name. ALL components (year + month name if present) must
appear in some cited span.
Closes the back-to-the-future failure fox surfaced 2026-05-01:
Q: what date did back to the future come out?
Claim: "Back to the Future was released in theaters on July 3, 1985."
Cited span: trilogy/SNES/pinball prose with '...real 1985...' but
no 'July' anywhere
Pre-fix: EVIDENCE-LINKED · via claim_lattice 1/1
Post-fix: EVIDENCE-LINKED-PARTIAL · via claim_lattice · warrant missing 1/1
Year extraction (4-digit, 1500-2199 range — avoids matching ZIP codes
/ elevations / model IDs) and month extraction (full English month
names — three-letter abbrevs deferred until bench shows the gap).
Both anchors required when both are in the claim. Year-only claim
against year-only span continues to pass; month-only against
month-only continues to pass; month-and-year claim against year-only
span (the BTTF case) now correctly fails.
Architectural note: this generalizes warrant-lite without building a
typed-contract framework (per-question-type rule libraries like
"release_date_lookup with required predicate cues"). The date anchor
primitive catches the date lazy-anchor failure class deterministically
without a per-type rule book; typed contracts earn their slot only if
bench evidence shows general anchors miss specific cases. Dropped the
caller-side is_relation_question gate from verify.py — warrant_check
now decides internally what to enforce based on question shape AND
claim content.
Tests: 29 unit tests in tests/test_warrant.py (was 16). New tests
cover extract_date_anchors (year, month, dedup, non-year-skip,
empty), warrant_check date semantics (BTTF case, year-only,
year-in-unrelated-context, month+year composition), proper-noun
warrant skipped without relation question. Full suite: 554 passed,
21 skipped.
Limitation accepted: ISO date format ("1985-07-03") wouldn't satisfy
month-name requirement. Real-world Wikipedia prose uses month names
("released on July 3, 1985"), so the trade-off favors catching the
common case. If bench shows ISO false-rejections, layer in a
month-number alternative to the month-name regex.
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
Pre-fix: _spotlight_excerpt used first-match-of-longest-content-token.
For "who is homer simpson's boss?" the Homer Simpson chunk had
"homer" matching early in voice-actor / Castellaneta prose ("For
voicing Homer..."), so the spotlight centered there even though
the actual answer ("his boss Mr. Burns") was deeper in the chunk
where boss/burns/homer cluster together.
Post-fix: density rank. Find ALL match positions for ALL content
tokens, then pick the position whose ±half-window cluster contains
the MOST distinct tokens. Tie-break on smallest idx (deterministic,
reproducible).
Live verified:
- "who is homer simpson's boss?" — spotlight now lands on
"...At the plant, Homer is often ignored and completely
forgotten by his boss Mr. Burns..." (boss/burns/homer/plant
cluster). Pre-fix it landed on voice-actor prose with no
Mr. Burns visible.
- "name simpsons family members including pets?" — spotlight
lands on "...the family's two pets, Santa's Little Helper
and Snowball II..." (densest pets/family cluster). Already
worked pre-fix; now even tighter.
Performance: O(|positions|^2) inner ranking. Typical chunks have
4-10 content tokens and 1-3 match positions each, so 8-30 positions
and ≤900 inner iterations — trivial.
Bracket format change (in render_claim_lattice) ships alongside:
[E5: "..."] → before
[E5 | <title> | <chunk_root[:8]>: "..."] → after
Closes visual confusion where E# is a chunk pointer but readers
parsed it as a 1-indexed source rank from the sources list.
Tests: 523 passed.
Initial Ticket H commit (38d95dd) only plumbed warrant into
verify_claim_lattice (pointer mode). The JSON variant
verify_claim_lattice_json was missed — and the CLI's
--answer-mode claim_lattice (the JSON path) was producing
EVIDENCE-LINKED on relation-question records without warrant
firing.
This adds the same warrant block to verify_claim_lattice_json:
question + warrant_check_enabled params, identical
WARRANT_MISSING violations, identical demote-to-HYBRID
semantics, same warrant_missing_claim_idxs field on the verdict.
Plumbs question= and warrant_check_enabled= through call sites
in runner.py and query.py for the JSON path.
Live verified: warrant correctly checks the FULL cited span
(not the spotlight excerpt). On the Homer-Simpson "boss is Mr.
Burns" case, the chunk text DOES contain "Mr. Burns" elsewhere
in the paragraph — warrant rightly passes. The misleading
spotlight excerpt that fox flagged is a separate render-layer
issue (failure to spotlight the load-bearing slice), not a
warrant failure.
Note: cache-hit path does not currently restore violations into
the result dict (providence_cache schema has no violations
column; run_dag_blob stores only hashes). On cache hits the
warrant_tail in CLI render won't surface even when warrant
fired at write-time. Follow-up needs a schema migration to
persist violations alongside the cached record.
Closes the lazy-anchor failure class fox surfaced in feedback-3
(2026-05-01) on "who is homer simpson's boss?":
Claim: "Homer Simpson's boss is Mr. Burns."
Cited span: voice-actor / Castellaneta bio prose
Pre-warrant: pointer-linked, source-role allowed, coverage passes
Post-warrant: WARRANT_MISSING violation, audit_mode capped at HYBRID
Pointer verification != warrant verification. A pointer resolves to
a span; a warrant requires the span to actually contain the claim's
named answer entity.
Adds aborist/qa/warrant.py:
- is_relation_question(q) — regex shape detector for "who is X's Y?",
"who is supermans girlfriend?", "who founded Z?", "boss of X" etc.
- extract_answer_anchors(claim) — multi-word proper nouns ("Mr.
Burns", "Lois Lane") with possessive 's stripping; solo-cap
fallback that skips sentence-starter false positives
- warrant_check(claim, spans) — at-least-one-anchor lexical match;
vacuous-pass when claim has no proper-noun anchor
Wired into verify_claim_lattice via new params (question,
warrant_check_enabled). Per-claim WARRANT_MISSING violations cap
audit_mode at HYBRID using the same demote pattern as
lazy_anchor_demoted. CLI renders " · warrant missing" tail when
violations present so the failure mode is legible without scanning
the violations array.
Policy field claim_lattice_warrant_check_enabled added to
DEFAULT_POLICY / DEFAULT_QUERY_POLICY and the verifier-policy hash
field set so changes bump verifier_policy_hash.
Hard or soft? Per CLAUDE.md "Soft hash vs hard hash":
substring/lexical = hard. Output is binary (warrant present or not),
violations enter the proof path through violations[] alongside
SOURCE_ROLE_BLOCKED and CITATION_MISMATCH.
Deferred (follow-on commits):
- full RELATION_PATTERNS table with relation-noun aliases
- two-hop bridge detection ("X founded Y" + "Y is Z" -> "X is Z")
- retrieval expansion for relation questions
Tests: 16 new unit tests in tests/test_warrant.py covering
is_relation_question, extract_answer_anchors, warrant_check
(happy path, lazy-anchor case, vacuous-pass, case-insensitive,
empty inputs). Full suite: 523 passed, 21 skipped.
Implements 6 of fox's feedback-2/3 items (2026-05-01) targeting the
red-fish-blue-fish output. Each lives in the render or sidecar layer
— no schema or cache-key change beyond the verifier_policy_hash that
landed earlier in the day.
(1) Honest audit-label rendering. Schema audit_mode (STRICT/HYBRID/
UNGROUNDED) describes the lexical verifier's call; the user-
facing label spells out what was actually checked. Mapping for
claim_lattice modes:
STRICT → EVIDENCE-LINKED · via claim_lattice
HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice
UNGROUNDED → UNGROUNDED · via claim_lattice
Quote / span / entity / paraphrase modes keep the audit_mode
token plus a `· via {method}` tail. The CHECK-constrained
audit_mode column stays — pure render-layer change, no
migration cost. Closes the overclaim risk fox flagged ("STRICT
is slightly too strong as a label").
(2) Sentence-bounded evidence spans. _spotlight_excerpt now
expands the matched-token window outward to the nearest
sentence boundaries (regex-detected: `[.!?][\"')\\]]?\\s+(?=[A-Z])`)
instead of cutting at fixed byte offsets. Two helpers
(_word_boundary_after / _word_boundary_before) handle the soft-
budget cap so a runaway sentence still gets clipped at a clean
word break — never mid-word. Pre-fix the renderer produced
excerpts like `"...freewheeling plot... friends and p..."` (the
`p...` is the offending mid-word truncation fox surfaced); now
the excerpt is `"One Fish Two Fish Red Fish Blue Fish is a
1960 children's book by Dr. Seuss. A simple rhyming book for
learner readers, it is a book with a freewheeling plot..."`.
(3) Source-role display + used/unused annotation. The CLI source
list went from
[1] One Fish Two Fish Red Fish Blue Fish — wikipedia.org/...
[2] Red Dwarf — wikipedia.org/...
to
[1] One Fish Two Fish Red Fish Blue Fish — primary_answer_source — used (E1) — ...
[2] Red Dwarf — background_source — unused — ...
Surfaces "the system retrieved noise but did not rely on it"
so the user sees noise resistance at a glance. Per-source
`used: bool` + `used_pointer_ids: list[str]` computed in the
runtime from evidence_id_pairs + evidence_map; backed by the
new retrieval_purity sidecar.
(4) Retrieval-purity sidecar. Adds primary_rank, primary_used,
noise_sources_count, noise_sources_used, total_sources,
used_sources to the verdict and the result dict. Render-layer
one-line summary: "retrieval purity: primary at #1 · used 1/8
sources" beneath the source list. Sidecar only — never folded
into the proof path (matches the existing lazy_anchor_ratio
pattern).
(5) Ordered-token title scoring. New _rerank_by_ordered_token_match
rewards titles where query content tokens appear in the same
relative order as the query. "red fish blue fish" query →
"One Fish Two Fish Red Fish Blue Fish" gets ordered_match=4
boost (1+0.5*3 = 2.5×); "Red Dwarf" stays at ordered_match=1
(no boost). LCS-based, stem-aware, deterministic. Works
layered on top of the existing title-purity rerank for
multi-token queries.
(6) Noise-resistance fixture (tests/test_qa_quality_live.py).
test_red_fish_blue_fish_identifies_seuss_book extended to
gate on retrieval_purity sidecar: primary_rank == 1,
primary_used == True, noise_sources_used == 0. Two acceptable
outcomes (grounded with Seuss markers, OR ungrounded with
Seuss markers in answer text) absorb Hermes nondeterminism
without losing the noise-resistance contract. Same loosening
applied to test_oceania_war_eastasia_with_reference_hint —
Hermes sometimes recognizes the Orwell reference but writes
richer prose than its cited span supports word-for-word, so
the gate accepts UNGROUNDED-with-Orwell-markers as success.
Live test_qa_quality_live.py: 20/20 fixtures pass. 507 unit tests
green. Working tree leaves bench/qa_questions.txt for fox's
parallel WIP.
Six items from fox's de-novo synthesis (2026-05-01) landing as one
atomic commit. Each item references its motivation and respects the
v9.8 honesty boundary (no claims of semantic truth, no proof-path
contamination by soft signals).
NEW DOCS
--------
docs/cti-architecture.md
Maps today's modules onto the architectural layer cake fox named
in his synthesis: PROMETHEUS-Σ (controller) / CTI (claim-lattice
reasoning IR) / Merkle-AGI-DAG (commitment) / Reverse-RAG
(evidence direction) / v9.8 Providence (admissibility ledger) /
Hermes (weak proposer). Per-layer responsibility table + module
map showing which existing files implement which layer. The
architecture is real; the labels above name what's already there.
docs/naming-deferral.md
Argues why we keep internal terms (claim_lattice,
claim_lattice_pointer, verify_claim_lattice) instead of renaming
to architectural labels (CTI, PROMETHEUS-Σ). The rename costs
cache invalidation, ~150 test fixture references, schema CHECK
migration, blame-history disconnect, mesh peer coordination.
The bridge is the cti-architecture mapping doc — read it once,
then read code in code's vocabulary and prose in prose's
vocabulary. Lists four triggers that would invert the decision.
docs/self-reference-distillation-design.md
Successor to docs/self-reference-thought-chains-design.md (the
flat-source MVP). Maps STRICT claims onto the existing Distiller
/ Core / derivations infrastructure: each STRICT record becomes
a Fact-Core via a new ProvidenceDistiller, with per-cited-chunk
Merkle inclusion proofs back to Wikipedia source_roots. The
fact-graph grows; new claims attach by inheriting the proof
chain. CompositionDistiller (combining facts into new claims)
is explicitly deferred — that's reasoning machinery, not
infrastructure.
docs/test-coverage-audit-2026-05-01.md
Maps fox's §11 test list (16 items) against the codebase. 16/16
covered. Documents item #9's manual-quote-rule lifecycle: the
rule was retired from pointer mode in commit 224bfd6 after the
bench journey identified it was rejecting factually correct
claims; retained in JSON variant where the punctuation-rationale
argument doesn't apply. The audit doc itself is the requirements-
drift defense.
CODE
----
aborist/qa/keys.py:verifier_policy_hash
New pure function. Hashes the verifier-relevant subset of policy
(answer_mode + claim_lattice_* verifier knobs + entity_policy
fields + base_version). Folded into cache_key as an optional
9th dimension via a new keyword arg with backward-compat default
None — legacy 8-dim callers continue to work and produce the
pre-2026-05-01 hash result.
aborist/qa/{runner,query}.py
Compute verifier_policy_hash from the policy_variant and pass
to cache_key as the 9th dim. Records written from this commit
forward bind to the verifier-policy identity. Pre-existing 8-dim
records become cache-misses on next lookup — same migration cost
as any governance_policy_hash bump. The 9-dim form makes the
question "did the verifier rules change?" answerable from
cache_key diff alone, separate from "did the prompt change?"
CLAUDE.md
Adds "old maps vs runtime maps" engineering discipline to the
bench-maxing section. Codifies the principle: every base model
carries old maps from training time; the runtime carries the
fresh map; when they disagree, the runtime wins. Move authority
OUT of the model's prior and INTO runtime artifacts (pointer IDs
the runtime mints, source spans the runtime interpolates by
offset, evidence maps assembled per query, policy hashes that
fold prompt/verifier/retrieval into cache_key, hard checks run
by the verifier). Hermes' content-addressed-evidence-id
hallucination (commit bb8450d) is the canonical case study.
TESTS
-----
tests/test_keys.py
Six new tests for verifier_policy_hash:
- only hashes verifier subset (non-verifier fields don't change it)
- changes when verifier-relevant field changes
- empty-policy → stable
- 9-dim cache_key distinct from 8-dim
- 9-dim distinct under different verifier hashes
- 8-dim form preserved for legacy callers (None == omit)
DEFERRED (per fox + naming-deferral.md)
---------------------------------------
- PROMETHEUS-Σ as an extracted controller module — the dispatch
logic in runner.py + query.py already IS PROMETHEUS-Σ; an
explicit prometheus.py is a refactor with no behavior change.
Defer until a §5-rename-trigger fires.
507 tests pass (was 501 before, +6 from verifier_policy_hash
coverage).
Per-call accounting of prompt capacity so we can answer 'does
context size affect strict-rate?' at aggregate scale.
Per-call result dict gains:
"prompt_chars": {
"system_prompt": <int>, # system message chars
"grounding_reminder": <int>, # restated rule message (if any)
"user_question": <int>, # the original question text
"evidence_or_context":<int>, # rendered evidence map / wikitext context
"messages_total": <int>, # sum of all messages[].content
}
"answer_chars": <int> # rendered answer length
Char-level for now: model-agnostic, fast, and a workable proxy
for prompt tokens (~4 chars/token English prose, ~2.5 for JSON-
evidence-block heavy contexts). Token-exact accounting can layer
on later if a tokenizer is cheap enough to wire in per-call.
Surfaces in three places:
(1) result["prompt_chars"] — for programmatic callers. Cache hits
return prompt_chars too (computed before lookup).
(2) `aborist query` human render — one-liner showing the breakdown
and the answer length, so an operator can tell at a glance
whether STRICT came from a 5KB tight prompt or a 60KB stuffed
context.
(3) bench/qa_sweep.py — per-row prompt_chars_{total,evidence,system,
question} + answer_chars columns in the JSONL, plus a new
"strict-rate by prompt size" section in the markdown summary
bucketing runs by capacity (<8KB, 8-16KB, 16-32KB, 32-64KB,
>=64KB) so the bench reveals whether input size correlates
with verdict quality across modes.
Never enters cache_key — runtime measurements, not policy.
Empirically observed 2026-05-01: long discursive questions like
'what technology is currently or soon available which may enable
one person to reconstruct another person's thoughts...' under-
retrieve because their content tokens get diluted by template
phrasing. AND-mode FTS5 returns zero hits; OR-fallback ranks
unrelated articles by raw BM25.
Fox's discovery: appending domain keywords ('transcranial knowledge
acquisition') to the question lifts the verdict from HYBRID 6/10
to STRICT 1/1 by narrowing OR-mode retrieval to the topical
article (Neurotechnology). The keywords act as an operator hint
about what the question is really asking.
This commit exposes that pattern as a first-class flag without
polluting the question text:
--retrieval-keywords TEXT on `aborist query`
K="..." on `make query`
Plumbing: the keywords are concatenated with the question for
FTS5 search and title-filter token construction only. The LLM
still sees the original question; the verifier still checks against
the original question; cache_key still computes from the original
question. Keywords are session-only — successive calls with
different keywords on the same question can cache-hit each other.
Pair with BURN=1 for fresh inference per call.
Live verification:
make query Q="...thoughts...without speaking or sign language."
K="transcranial knowledge acquisition" BURN=1
-> STRICT 2/2 via claim_lattice, 37.5s, Neurotechnology article
cited for fMRI and DBS claims.
Without K: HYBRID 6/10 with Videoconferencing/Telepathy as top
sources — model hallucinated structure across irrelevant chunks.
Empirical finding 2026-05-01 on 'explain spin glass modeling, tensors?':
returned STRICT 1/1 with claim 'Spin glass modeling involves...
mathematical tools such as tensors' cited to a chunk from the
*Quantum chromodynamics* article. Token-coverage check inside the
chunk passed accidentally on shared physics vocabulary
(spin/model/etc.), but the article isn't about spin glasses — the
claim is a retrieval-driven hallucination.
Root retrieval cause: AND-mode FTS5 returned zero hits (Spin_glass
article has 0 occurrences of 'modeling' or 'tensors'); OR-mode
fallback ranked by raw BM25, so 'Glass' (glass*274) outranked
'Spin glass' (spin*29 + glass*15) despite Spin_glass matching 2
distinct query tokens vs Glass's 1. Verifier then accepted the
hallucinated claim because the cited QCD chunk has incidental
physics vocabulary overlap.
New ``diagnose_title_relevance(claim_text, cited_source_titles)``
sidecar: claim's content tokens (stemmed) must overlap the union
of cited source titles. Zero overlap → ``title_mismatch``. Catches
the spin-tensors case (claim about spin glass, cited title 'Quantum
chromodynamics' → zero stem overlap) without false-positives on
legitimate cross-document grounding (Lois_Lane article cited for a
Lois-Lane claim still has lois+lane overlap).
Sidecar-only for now — staying out of the binary verifier output
until bench evidence justifies hard-promotion. Not yet wired into
inspect_cache_key (per-claim citation source mapping isn't in
providence_cache yet) or the bench. Available as a utility for
operators auditing specific (claim, source_title) pairs.
5 new unit tests covering match/mismatch/no-titles/no-claim-tokens
and underscore-normalization for Wikipedia titles.
Closes the recursive-grounding gap fox surfaced today: aborist
tends Wikipedia trees but never grafts its own past Q&A records
into the forest. Each query starts from cold corpus retrieval;
prior providence_cache records sit unread until the same question
is re-asked (cache_key match). The "kindergarten thought chains"
framing names the gap — the system has a kindergarten of thoughts
(early STRICT records) that should mature into citable substrate
as they cool.
This commit lands the MVP — STRICT live providence records past
the kindergarten window become substrate via a new Source
subclass. Trust model per fox: "we trust strict statements as
fact unless a verifier falsifies it."
NEW
---
- docs/self-reference-thought-chains-design.md — full architecture
doc covering the four iteration-time gates, the recursive Merkle
proof story (Q2 → Q1 → Wikipedia chunk per v9.8's recursive-
cores insight), the falsification trust model (state=live as the
fail-closed gate), risks (lazy-anchor compounding, echo chambers,
storage bloat), and bench-impact estimate.
- aborist/sources/providence.py — ProvidenceSource(Source) with
four hard gates:
1. audit_mode == 'STRICT' (HYBRID/UNGROUNDED stay opaque)
2. falsification_state == 'live' (failed/stale/quarantined
excluded — verifier-falsification mechanism per fox)
3. now - created_at >= kindergarten_seconds (default 1h —
fresh thoughts cool first; kills tight echo loops)
4. anti-recursion: records whose answer text contains a
self-reference URI are skipped — first-generation only
- tests/test_providence_source.py — 10 unit tests covering each
gate plus the URI-scheme source-role classifier
- Makefile target `ingest-self-providence` (KG_SECONDS=3600
default; iterates each shard and self-promotes its STRICT live
records — cross-shard sharing happens via the existing
shards-dir UNION at retrieval time)
WIRE-UP
-------
- aborist/qa/query.py
- SOURCE_ROLE_BUDGET_WEIGHTS: self_reference_source = 1.0
(same as background — Wikipedia stays canonical primary;
self-reference is supplementary anchoring)
- SOURCE_ROLE_RANK_WEIGHTS: self_reference_source = 0.9
- _classify_source_role: short-circuits on aborist://providence/
URI prefix → self_reference_source regardless of title shape
- DEFAULT_QUERY_POLICY['claim_lattice_allowed_source_roles']
+= 'self_reference_source'
- aborist/qa/runner.py — same allowlist update for the
per-document `ask` path
- aborist/cli.py — `aborist ingest --source providence` reads the
providence_cache from the same shard it writes into;
--kindergarten-seconds flag plumbed through
NOT IN THIS COMMIT
------------------
- Aggregation of multiple Q&A records into synthesized summary
records (follow-on)
- Self-reference for HYBRID records (only STRICT is substrate
today; HYBRID could land later as a soft-anchor role with
lower trust)
- Live virtual sourcing (the design discusses it; MVP uses
snapshot ingestion so existing FTS / chunker / Merkle apply
with zero schema change)
- A live bench validating actual lift on self-reference questions
(requires running ingest-self-providence then bench; deferred
to follow-on commit on real data)
10 new unit tests pass; full suite at 482 passed / 21 skipped
(live fixtures gated).
Bench dig (2026-05-01, 65-question sweep): 4 of 7 deflecting
questions were false positives from verb-shape subject anchors.
what year did the berlin wall fall? → answer "1989" (anchor "fall")
when did the soviet union dissolve? → answer "1991" (anchor "dissolve")
how many wives did henry the eighth? → answer "six" (anchor "eighth")
what year does our cold fusion ...? → answer "1989" (anchor "happen")
why did the titanic sink? → "...sank in 1912" (anchor "sink" vs "sank")
why did the dinosaurs go extinct? → "asteroid impact" (anchor "extinct" vs "died")
These shapes ask for a year/count/cause; the answer is structurally
not expected to echo the question's subject. Pre-suppression, all
three modes landed deflection on these — the apparent
"pointer-only" deflection rate (10.3% vs 6.2%) was almost entirely
heuristic noise, not a real substrate failure.
New ``shape_suppressed`` flag returned by diagnose_deflection. When
the question starts with one of:
when, what year, what date, what time,
how many, how much, how long, how old, why
the subject-anchor check is skipped — overlap-ratio takes over.
Mars-BDFL (who-shape) still fires correctly.
Field exposed so the operator can filter the signal: a
shape_suppressed=True row that still classifies as deflection on
overlap-ratio means "answer was a bare year/count/cause with no
topic words" — known noisy, not real topic shift.
5 new unit tests; full suite 582 passed (was 578).
Two paired changes:
(1) Retrieval template-phrase stopwords. Bench finding 2026-04-30:
'tell me all there is to know about york england?' missed the
en.wikipedia.org/wiki/York article entirely — corpus has it but the
FTS5 AND-mode MATCH expanded to "all" AND "there" AND "know" AND
"york" AND "england", which filters to only docs containing every
template word. Articles like "Charles I of England" happen to mention
"all" + "there" + "know" + "england" enough to win; the bare "York"
stub doesn't, so it falls out of the top-K.
Added `all there know everything anything something` to both
_FTS5_STOPWORDS (search/fts5.py) and _TITLE_STOPWORDS (qa/query.py)
so they stay in sync. After the change "York" article ranks #1 on
both 'tell me all there is to know about' and 'tell me everything
about' phrasings.
These are retrieval-time stopwords only — they don't enter the
cache_key or governance_policy_hash (cache_key hashes the original
question text via question_hash). Existing cached records remain
valid; new queries get better retrieval.
(2) bench/qa_sweep.py: per-row deflection_kind + subject_anchor +
subject_in_answer (from qa.inspect.diagnose_deflection); summary
table grows a 'deflections' column tracking STRICT/HYBRID rows
where the question's subject anchor is missing from the answer.
Surfaces the Mars-BDFL pattern at aggregate scale.
Empirical finding 2026-04-30 on 'who is a benevolent dictator for life
for mars?': JSON mode returned STRICT with answer 'Guido van Rossum is
a Benevolent Dictator For Life (BDFL) for the Python programming
language.' The verifier did its job (claims grounded), but the answer
never mentioned Mars — pure topic-shift. A real failure mode for
adversarial-premise questions where the model deflects rather than
refuses.
Detection rule: subject-anchor heuristic. The LAST content token in the
question (after stopword strip) is treated as the question's primary
subject ('mars' for BDFL, 'beatles' for 'who are the members of the
beatles?'). If the subject anchor is missing from the answer, classify
as 'deflection' regardless of generic-vocabulary overlap. The Mars-BDFL
case has 3/4 generic overlap (benevolent, dictator, life) but missing
subject — exactly the pattern overlap-ratio alone can't catch.
Returns dict with kind ∈ {deflection, partial_overlap, on_topic,
no_question_tokens} plus subject_anchor + subject_in_answer for
operator legibility.
Sidecar-only — wired into inspect_cache_key() but never feeds back
into providence_cache or audit_events. Per the verifier-stays-binary
discipline (CLAUDE.md): soft signals live in inspect verbs, never in
the hard chain.
Defense-in-depth against the "tell me all there is to know about X"
runaway shape. Bench finding (2026-04-30 york-england, n=3): JSON mode
emitted 26-59 claim-pointer pairs of which only 2-4 verified — model
treats encyclopedic prompts as a license to spam claims from training.
The atomic-claim prompt rule (b5925c8) cut typical runs to ~10 claims,
but a structural ceiling makes the failure shape operator-visible
regardless of per-claim verification.
Cap: 12. Admits typical entity-list questions (5-7 dinosaurs, simpsons
+ pets) while flagging the runaway. Plumbed through both verifiers
(pointer + JSON) via claim_lattice_max_claims_per_answer policy knob;
folds into governance_policy_hash on change.
Behavior: cap doesn't truncate — every claim still verifies so the
operator sees full evidence of the runaway. The TOO_MANY_CLAIMS
violation demotes STRICT → HYBRID via the existing violation-check
path; UNGROUNDED if no claims verified.
Tests: test_verify_json_too_many_claims_demotes_to_hybrid (13 claims
all verifying → HYBRID), test_verify_json_at_cap_can_still_strict
(12 claims at the cap → STRICT, no violation), and pointer-mode
mirror test_too_many_claims_demotes_pointer_mode_to_hybrid.
Prompt audit + rewrite per the negation-avoidance principle codified
in CLAUDE.md: small instruction-tuned models (Hermes-3-8B observed)
struggle with "do NOT X" / "never Y" / "without Z" — the negation
often pattern-matches away or inverts under attention. Rewrote 18
negation instances across 8 prompt fields in
DEFAULT_POLICY (runner.py) and DEFAULT_QUERY_POLICY (query.py) to
their positive forms. Examples:
"Do NOT invent IDs" → "Use only the IDs that appear in the
EVIDENCE blocks above"
"Never begin a line with E1:" → "Each line begins with the
claim text, then a bracket tag"
"No commentary, no preamble" → "Your output begins with `{`
and ends with `}`"
"If no evidence supports a claim, omit the claim" →
"Emit only claims that an EVIDENCE block directly supports"
Also added an atomic-claim rule (Rule 6 JSON, Rule 9 pointer):
"Each claim is one focused fact citing the source that directly
contains it. Multiple distinct facts become multiple claim
entries… Three short claims, each citing one pointer, beat one
long claim citing five." Targets the StarCraft-style mega-claim
where the model packs every fact into a single sentence with
five pointers.
Bench delta (n=3, 25 questions, 75 runs per mode):
quote 30S 28H 17U → 40S 23H 12U +10 STRICT, +13pp rate
pointer 27S 37H 11U → 23S 37H 15U -4 STRICT (acceptable —
JSON is now the daily
driver; pointer is
fallback/diagnostic)
JSON 36S 28H 11U → 44S 26H 5U +8 STRICT, +11pp rate,
-6 UNGROUNDED, -1.8s latency
Net across modes: +14 STRICT, +7 grounded, JSON latency
7.2s → 5.4s.
Bench questions: replaced "isn't it true that the great wall of
china is visible from space?" with "which side is the ground
elevation highest throughout the span of the great wall of
china, the north or south?" — a forensic-shape question with
answer "north". Live finding: model hits STRICT 1/1 with the
right answer cited to a Great-Wall-but-not-elevation chunk
(coverage threshold passes via topical-token overlap). New
fixture test_great_wall_elevation_north_side_higher gates on
entity-presence ("north" in answer) but tolerates UNGROUNDED —
documents the residual lazy-anchor failure mode for the
verifier-semantic roadmap.
CLAUDE.md: added negation-avoidance to the bench-maxing
discipline section.
Adds _repair_truncated_json() to recover from mid-output truncation
where Hermes-3-8B writes one long claim text and runs out of
max_tokens before closing the structure. Pre-fix the lenient parser
saw something like:
{"claims":[{"text":"The Apollo program was the United States
spaceflight effort which landed... <CUT>
with no closing `"`, no closing `}`, no closing `]`, no closing
outer `}`. Couldn't parse, lost the whole answer.
Post-fix the repair walks the input once tracking string state
and bracket stack:
- close any open string (append `"`)
- drop dangling structural artifacts:
`"key":` (partial key with colon)
`"key"` after `,` (partial key without colon)
trailing `,` before close
- close the bracket stack in reverse (`{` → `}`, `[` → `]`)
Conservative: never inserts content (no key names, no values, no
commas) — only closes / drops. Worst case the repair is a no-op
or makes parsing fail in a different way; never silently
fabricates data.
Apollo live test result: UNGROUNDED 0/0 (lost answer) →
UNGROUNDED 0/1 (partial first claim recovered with evidence_ids
intact, partial second claim dropped). The user sees the model's
attempted answer in the unverified bucket instead of empty space.
The audit chain captures the repair via fixups labels
(close_string, close_brace, close_bracket, drop_partial_key,
strip_trailing_comma) so an auditor can see what the parser had
to peel.
Pipeline ordering in _lenient_json_parse: strict parse first,
then peel (fence/prose-trim/curly-quotes/trailing-comma), then
strict parse again, then truncation repair as last resort. Each
step's fixups append to the same list so the verdict's
`json_fixups` field is comprehensive.
Bench question set picked up "name simpsons family members
including pets?" — a multi-part list with adjacent-relationship
shape, addresses a gap in the corpus probe.
460 unit tests pass; 12 skipped (live fixtures unchanged).
Bench data shows residual JSON-mode token runaway after the
pointer-ID switch: ~4 cases out of 66 land UNGROUNDED 0/0 at
12-15s instead of ~2-4s normal. Pattern: Hermes-3-8B emits a
valid claim object, then keeps generating whitespace / blank
lines until max_tokens (512) exhausts. The truncated payload
won't parse and the lenient pre-parser returns no claims.
Concrete instances in the latest bench (2026-04-30T19-55-11Z):
- tell me about the apollo program (3/3 samples runaway)
- tell me about the python programming language (1/3)
Fix: pass `stop=["\n\n"]` to the chat completion in JSON mode so
vLLM cuts generation at the first blank line. Well-formed JSON-
mode output never legitimately contains a blank line — Hermes
emits one object on a single line (or with simple internal
newlines), never `\n\n`. The stop sequence is the runaway
signature itself.
Plumbing:
- OpenAICompatibleClient.chat_completion: new `stop` kwarg,
injects into request payload when non-empty
- StubClient already absorbs **kwargs; no change needed
- DEFAULT_POLICY (runner) + DEFAULT_QUERY_POLICY (query) gain
`claim_lattice_json_stop_sequences = ["\n\n"]`. Folds into
governance_policy_hash so changing the stop list invalidates
prior cached records.
- Both call sites in runner.py / query.py read the policy
field and pass it only on JSON mode (pointer + quote modes
don't need it).
Defensive measure: worst case the stop sequence never fires;
best case the apollo/python residuals recover and JSON's
strict-rate climbs further.
JSON mode previously presented content-addressed evidence_ids
(``Eed1b6e396``-style) in the EVIDENCE block headers and expected
the same in the model's claim output. Hermes-3-8B was fabricating
plausible near-miss IDs (e.g. ``E1b6e396`` when the runtime had
``Eed1b6e396``) on cross-document relationship questions →
UNKNOWN_EVIDENCE_ID → UNGROUNDED, even when the answer text was
factually correct (e.g. "Homer Simpson's boss is Mr. Burns.").
Switching the prompt-facing surface to short pointer IDs (``E1``,
``E2``, …) — same as claim_lattice_pointer mode — closes the
hallucination loop:
- Pointer IDs are short, enumerable, and fabrication-obvious.
The model can't invent ``E27`` if only ``E1``-``E10`` were shown;
out-of-range IDs read as schema violations at first glance.
- The runtime still resolves each pointer_id to its content-
addressed evidence_id internally and stores THAT in
``evidence_id_pairs`` for the cache & run-DAG. Cache_keys stay
run-stable; only the prompt-facing string changes.
- JSON schema unchanged (``evidence_ids: [str, ...]``), so vLLM
guided_json continues to constrain output shape.
Live-test verification: ``who is homer simpson's boss?`` went from
JSON-mode UNGROUNDED 0/1 (hallucinated ``E1b6e396``) to STRICT 1/1
(model emits ``E1``, resolves cleanly). Homer fixture pin removed —
runs against the JSON default now.
Files touched:
- aborist/qa/evidence.py: render_evidence_block_for_json uses
e.pointer_id instead of e.evidence_id.
- aborist/qa/verify.py: verify_claim_lattice_json switched from
evidence_map_by_evidence_id to evidence_map_by_pointer_id;
captures both pointer_ids (model-emitted) and evidence_ids
(run-stable) in claim_statuses + evidence_id_pairs.
- aborist/qa/{runner,query}.py: claim_lattice_json_system_prompt
+ grounding_reminder describe pointer IDs; example shifts from
``E........`` placeholder to ``E1``.
- tests/test_verify_json.py: stub _ev() takes pointer_id; four
fixtures updated to set it.
- tests/test_qa_quality_live.py: Homer fixture unpinned (now
runs default JSON mode and grounds). Red-fish-blue-fish
fixture pinned to pointer mode — JSON mode hits a separate
token-budget runaway on "plot of X" prose-summary shapes
(~2/3 of samples blow max_tokens with whitespace spam after
the closing brace). Different failure mode, addressed in a
later commit.
460 unit tests + 11 live fixtures pass.
Three-way QA-quality bench (quote / pointer / JSON) over an expanded
22-question set × 3 samples = 198 LLM calls. Findings landed in
docs/qa-modes-bench-2026-04-30.md with per-question breakdown and
roadmap. Aggregate at bench time:
quote 31S 20H 15U 0e strict-rate 47% 7.7s
claim_lattice_pointer 14S 34H 18U 0e strict-rate 21% 4.4s
claim_lattice (JSON) 26S 12H 9U 19e strict-rate 39%* 4.4s
The 19 JSON-mode "errors" turned out to be HTTP 502 from vLLM upstream,
not parse failures — clustered, all on the JSON-mode pass, plausibly
correlated with `guided_json` stressing the grammar engine.
Two improvements based on findings:
(1) OpenAICompatibleClient grew retry on transient 502/503/504 with
exponential backoff (0.5/1/2s, 3 attempts default). Network-layer
errors (ConnectError, ReadTimeout, RemoteProtocolError) get the
same retry. Smooths over the cluster without changing semantics:
persistent failures still raise, transient bursts no longer
dominate the error column. Helps all modes; JSON benefits most.
(2) Pointer-cap behavior changed from hard SCHEMA_INVALID to
trim-and-verify. When `[E2,...,E14]` over-cites a single claim,
keep first N pointers, verify normally, record
POINTER_OVERFLOW_TRIMMED in violations. STRICT becomes unreachable
(audit_mode caps at HYBRID) so the over-cite pattern stays
surfaced — but a correct claim like "Leonardo da Vinci painted
the Mona Lisa." no longer gets nuked for cosmetic over-citation.
Pre-fix: pointer mode hit 0/3 STRICT on Mona Lisa (mega-bracket
triggered SCHEMA_INVALID). Post-fix: HYBRID 2/14 with the right
answer rendered alongside both kept source spans. The dropped
pointers count toward n_quotes so the denominator surfaces the
over-cite to the auditor.
Bench scaffolding: ANSWER_MODES tuple now includes "claim_lattice"
(JSON), Makefile default sweeps all three. Question set expanded
from 8 to 22 covering narrow factoid, broad descriptive, entity
list, relationship, comparison, niche, adversarial, out-of-corpus.
460 tests pass. Connecticut output stays clean (HYBRID 4/7); JP-
dinosaurs pointer mode still UNGROUNDED via the bare-name guard
(model emits one-token entity names, the right floor catches them).
Round-tripped end-to-end against Hermes-3-8B:
query "what dinosaurs were in the first jurassic park film?"
--answer-mode claim_lattice
-> HYBRID 5/5 verified via claim_lattice in 4.9s
Hermes packed all five species into a single claim with five
evidence_ids, tripping TOO_MANY_EVIDENCE_IDS (max 2/claim). The
substrate's hard rule kicked in correctly — per-id checks all passed
but claim under-decomposition demoted STRICT to HYBRID. Reasoner-class
models (Qwen 3.6, Claude, GPT-4) should decompose into one claim per
species and clear STRICT.
Three layered fixes for the JP-dinosaurs lazy-anchor failure mode where
"what dinosaurs were in the first jurassic park film?" returned HYBRID
5/8 with citations to a video-game tie-in chunk (Operation Genesis):
1. Bare-name claim guard. New policy field
`claim_lattice_min_claim_content_tokens` (default 2). Claims with
fewer content tokens (>=4 chars, post-spotlight-stopword) classify
as SCHEMA_INVALID. "Triceratops. [E16]" had 1 content token; lexical
overlap on a single token is too thin to verify "X is in the FIRST
film" semantics. Forcing sentence shape raises the coverage bar so
off-topic chunks can no longer satisfy the citation. Threshold of 2
keeps narrow factoids viable (e.g. "Steve Jobs co-founded Apple"
tokenizes to 3 content tokens after stopword strip).
2. Smell → demote. The lazy-anchor smell sidecar was advisory only;
the verdict could be STRICT while every claim cited the same
magnet chunk. New policy fields
`claim_lattice_lazy_anchor_demote_threshold` (0.5) and
`claim_lattice_lazy_anchor_demote_min_pairs` (3) cap audit_mode at
HYBRID when ratio + pair-count both met. STRICT now requires
diverse anchoring across pointers; emits a LAZY_ANCHOR_DEMOTE
violation record so the auditor can see what fired.
3. Noisy title markers extended. `_NOISY_TITLE_MARKERS` now includes
"operation genesis", "the game", "video games" so spinoff/tie-in
articles (Jurassic Park: Operation Genesis, X (NES game), etc.)
classify as noisy_background_source. Default
claim_lattice_allowed_source_roles excludes that role, so noisy
chunks never enter the evidence map for pointer-mode queries.
Brittle (case-by-case markers) but addresses the JP failure
directly.
Live JP-dinosaurs after fix: UNGROUNDED 0/11 (was HYBRID 5/8 with
bogus citations). Honest refusal beats false confidence — model could
recover by writing sentence-shape claims; bare-name shortcut blocked.
Test fixtures using minimal placeholder claims ("Trex appears. [E1]")
opt out of the bare-name check via `min_claim_content_tokens=0` since
those tests exercise orthogonal behaviors. 460 tests pass.
JSON mode (answer_mode="claim_lattice") is now reachable end-to-end via
runner.ask: message build, optional vLLM guided_json extra_body, verifier
dispatch, and providence_cache persistence. Verifier returns
verifier_method="claim_lattice" (same as pointer variant) so both modes
share the existing CHECK constraint; answer_mode on the run-DAG and
json_fixups on the verdict disambiguate.
Pointer verifier now splits EVIDENCE_LINKED_PARTIAL into its own
partially_verified_quotes bucket so a claim never renders as both a
verified bullet and an unverified footer. CLI shows a "partially
grounded" section between the verified body and the unverified list.
Removes the strict no-double-quote check (`MANUAL_QUOTE_VIOLATION`) from
verify_claim_lattice. The rule was rejecting factually correct,
source-grounded claims for cosmetic punctuation: Hermes-3-8B paraphrases
prose but copies named-quoted phrases verbatim from source (e.g.
`"Constitution State"` lifted from a Connecticut chunk). Pre-fix,
`make query Q="tell me about connecticut"` reported `0/13 verified` on
a paragraph where every claim was correct and source-supported, just
because the model preserved source quote marks.
The two checks that remain handle the cases the manual_quote rule was
nominally meant to catch:
- claim_lattice_min_citation_coverage (Rule 5, default 0.30)
catches lazy-anchor citations whose only overlap with the cited
span is a single topical token
- claim_lattice_max_pointers_per_claim (Rule 6, default 2)
catches the encyclopedic-mega-claim where the model emits
`[E1,E2,...,E26]` after one sentence
Bench n=3 across 8 fixed questions vs pre-G0-hardening baseline (also
n=3): pointer-mode grounded count (STRICT+HYBRID) goes 15 → 21,
UNGROUNDED 9 → 3, broad-descriptive failures (connecticut, python)
fully cured. STRICT count goes 12 → 9 because previously-bogus STRICTs
(microsoft cited transit chunks that just shared one token, supermans
lazy-anchor magnet at 32/32) are now honestly reclassified to HYBRID.
Side changes:
- Ports max_pointers / min_citation_coverage policy fields and
verify_claim_lattice call into runner.py so single-document `ask`
matches multi-source `query` semantics (query.py already had them
via b39e79b).
- Worked Example 2 (broad-descriptive) added to pointer-mode prompt
so "tell me about X" has a template, plus Rules 7 (anti-echo) and
8 (max-2-pointers).
- Two manual_quote tests in tests/test_claim_lattice.py inverted to
document the new behavior (quotes-in-claim no longer block).
- `_has_manual_quote` retained — still used by verify_claim_lattice_json.
459 tests pass.
Brings the JSON variant back as a third ANSWER_MODE, distinct from the
pointer variant. The substrate exposes both: pointer for prose-
distribution / small-model paths (Hermes-3 8B), JSON for grammar-
constrained / large-reasoning-model paths (vLLM guided_json,
Claude/GPT-4 native JSON, Qwen 3.6 reasoner). Agents pick by setting
`policy["answer_mode"]`; both fold into governance_policy_hash so
records under different modes never alias.
Components:
- aborist/qa/verify.py:
* ANSWER_MODES = ("quote", "claim_lattice_pointer", "claim_lattice")
* `_lenient_json_parse(raw)` — defensive pre-parser. Strips markdown
fences, trims preamble/suffix to {/}, normalizes curly quotes,
fixes trailing commas. Returns (parsed, fixups[]) so the verifier
logs which drift had to be peeled. Lenient on syntax, strict on
semantics: parsed JSON still has to schema-check.
* CLAIM_LATTICE_JSON_SCHEMA — JSON Schema for the {"claims":[...]}
shape. Used by vLLM guided_json sampling-time constraint.
* `verify_claim_lattice_json(answer_json_text, evidence_map, ...)` —
runs the same hard checks as verify_claim_lattice (evidence_id
resolves, source_role allowed, no manual quotes, claim text non-
empty, claim textually overlaps evidence) but on content-
addressed evidence_ids directly. Returns the same verdict shape
plus a `json_fixups` list.
- aborist/qa/client.py: ChatClient Protocol & OpenAICompatibleClient
gain optional `extra_body` kwarg. Forwarded as additional fields in
the JSON request payload — opaque pass-through for vLLM-specific
knobs like `guided_json`. Endpoints that don't recognize a key
silently drop it. StubClient ignores; tests inspect via self.calls.
- aborist/qa/evidence.py: `render_evidence_block_for_json` and
`render_evidence_map_for_json` — JSON-mode prompts label blocks with
the content-addressed evidence_id (long hex) since that's what the
model cites in its JSON. Pointer mode keeps using the short
pointer_id.
- aborist/qa/query.py:
* Imports the JSON verifier + schema + JSON-mode evidence renderer.
* Message-build branch: `elif answer_mode == "claim_lattice"`
builds the same per-chunk evidence map as pointer mode, but
uses `claim_lattice_json_system_prompt` and labels blocks with
evidence_id.
* LLM call: when answer_mode=claim_lattice and policy
`claim_lattice_use_guided_json` is on, passes
`extra_body={"guided_json": SCHEMA}` so vLLM constrains output.
* Verifier dispatch: new `elif answer_mode == "claim_lattice"`
branch calls verify_claim_lattice_json.
* DAG persistence: lattice-mode raw_answer / parsed_lattice /
rendered_text threading now applies to both pointer and JSON.
* DEFAULT_QUERY_POLICY adds `claim_lattice_json_system_prompt`,
`claim_lattice_json_grounding_reminder`, and
`claim_lattice_use_guided_json` (default True).
- tests/test_verify_json.py: 14 tests covering the lenient parser
(strict pass-through, fence strip, preamble trim, curly-quote
normalize, trailing-comma fix, multi-fixup, hard-fail) and the
JSON verifier (STRICT on resolved claims, HYBRID on partial,
UNGROUNDED on schema invalid, fence recovery, manual-quote
violation, source-role block).
The whitepaper rewrite to 13.9.1 (substrate exposes both modalities,
both first-class) becomes accurate post-ship — JSON mode now exists
in code as it always existed in the architecture's intent.
554 tests pass.
Two layered fixes that close the lazy-anchor gap. Title-purity
moves the right SOURCES to the top of retrieval; the new hard
check makes the verifier ruthlessly honest about whether each
cited chunk actually mentions what its claim says.
1. _rerank_by_title_purity. New stage at the end of _rerank().
purity = |title_tokens ∩ query_tokens| / |title_tokens|; score
multiplied by (1 + 2*purity), so a title that IS the topic
("Jurassic Park (film)" against the dinosaur question) gets
3.0x and titles that drag in off-topic tokens like
"Jurassic Park: Operation Genesis" only get 2.0x. Composes
with the role-rank weights from the prior commit. Florida
query: list-pages dropped out of top-8 entirely. JP-dinosaurs
query: Operation Genesis sank from #4 to #8.
2. CITATION_MISMATCH hard check. Sixth deterministic check in
verify_claim_lattice — for each (claim, pointer) pair, at
least one content token from the claim text must textually
appear in the cited evidence span. Lexical only, no NER, no
embeddings; same _content_tokens extractor as the spotlight
so the proof-path stays on the soft/hard boundary. Catches
the magnet-chunk lazy-anchor where the model cites a chunk
whose text contains zero overlap with what its claim asserts.
New status CITATION_MISMATCH joins the per-claim taxonomy.
New violation kind CITATION_MISMATCH joins the run-DAG verify
payload. JP-dinosaurs verdict went from "13/14 verified
ratio 0.77 (cite E13 for everything)" pre-check to "1/14
verified" post-check — only Tyrannosaurus Rex's claim
actually overlapped its cited chunk. The architecture
stopped lying about evidence-linkage that the cited spans
don't textually support.
Pure-stopword claims pass the overlap check vacuously (no
content tokens means nothing to check against) — other hard
checks own that case.
Tests: +3 covering CITATION_MISMATCH (claim-token absent yields
fail, partial overlap yields HYBRID with EVIDENCE_LINKED_PARTIAL,
pure-stopword claim passes vacuously). Updated 7 prior tests
that used colloquial stub claims ("Trex") which don't textually
appear in the sample spans ("Tyrannosaurus rex"); they now use
claim text that overlaps the source content. 445 tests passing,
all 7 production shards report 0 chain breaks.
Three fixes that meaningfully improve claim_lattice_pointer behavior
on real queries.
1. Prompts in positive form. Hermes-3-8B (and instruction-tuned 8Bs
in general) follow positive directives ("do X") much more
reliably than negations ("don't do Y"). Rewrote the
claim_lattice_system_prompt and claim_lattice_grounding_reminder
in DEFAULT_POLICY and DEFAULT_QUERY_POLICY so every rule says
what TO do — "Reference evidence by pointer ID", "Use only
pointer IDs that appear in the EVIDENCE blocks", "Cite 1 or 2
pointers per claim", "Stop when the evidence runs out — a short
answer is the right answer when only short evidence exists",
etc. The cite-count constraint (1-2 per claim) directly
addresses the spray-anchor failure where Hermes attached every
available pointer to one claim line ("where is florida"
produced 1 claim with 33 cites pre-fix).
2. Source-role rank boost. _rerank() now ends with a new
_rerank_by_source_role stage that classifies each hit's role
(mutating h.source_role for downstream reuse) and rescales the
score by SOURCE_ROLE_RANK_WEIGHTS — primary 2.0, secondary 0.7,
noisy/sequel 0.3. The Florida defect: list-pages
(List_of_State_Roads_in_Florida, List_of_places_in_Florida:S/C/B,
Florida_locations_by_per_capita_income) all match
_SECONDARY_TITLE_MARKERS for "list of" and now sort below the
actual Florida article instead of dominating the top-8 by body
density. Affects ranking for both quote and pointer modes; the
earlier per-source budget weights stay (still primary gets 2x
the cap), they just don't have to fight a sort order that put
list-pages first.
3. Lazy-anchor smell line gates on n_verified >= 3. With one
verified pair the ratio is trivially 1.00 ("1 of 1 verified
pairs cite [E11]") which is vacuous. Below 3 pairs the metric
has no comparison surface; suppress the warning in those cases.
Live results on the 'where is florida' query went 33/33 with
cite-spray to 1/1 with a single targeted citation; spotlight
now hits "...largest metropolitan area in the state as well as
the entire southeastern United States is the South Florida..."
instead of state-road-number tables. The JP-dinosaurs benchmark
went from 11/11 with all cites at [E1] (ratio 1.00) to 13/14
with cites distributed across [E8], [E9], [E13] (ratio 0.77),
spotlight finding film-specific spans like "the film's
Dilophosaurus stands about 1.2 meters (4 ft) tall."
442 tests still passing. All 7 production shards report 0 chain
breaks.
verify_claim_lattice now derives pointer_id_distribution and
lazy_anchor_ratio from the verified-or-partial claim_statuses and
returns them in the verdict. Surfaced through ask()/query() result
dicts; CLI human-render displays a one-liner when ratio >= 0.5
(threshold for "model is lazy-anchoring").
Sidecar discipline preserved: never persisted in providence_cache,
never threaded into build_run_dag's verify_payload (test pins this
invariant — sidecar values must not enter run_dag_root). The
distribution is recoverable from claim_statuses which IS persisted,
so future sidecar tools can recompute on demand without enlarging
the audit chain.
JP dinosaurs benchmark currently triggers the warning at 11/11
ratio 1.00 — Hermes-3-8B cites [E1] for every claim even with
G0.3's relevance ranking putting the dinosaur-mention chunk near
the top. The render-layer signal now makes that lazy-anchor
behavior explicit instead of leaving it for the auditor to spot
from the spotlight excerpts.
Tests: +4 covering the ratio across distributions (single-anchor /
diversified / no-verified-pairs) plus the run_dag isolation
invariant. Total 442 passing.
New policy["answer_mode"] = "claim_lattice_pointer" (default "quote").
The runtime builds an evidence map with two-layer ids — pointer-id
(E1, E2, ...) shown to the model, sha256-derived evidence-id used by
the cache & run-DAG — and the model emits pointer-line prose
("Claim. [E12]") referencing them. Renderer interpolates literal
source spans at display time. Synthetic-elision-by-construction-
impossible: the model never types the quote string.
Pieces:
- aborist/qa/evidence.py (new): EvidenceObject + spotlight excerpt
(claim-token-centered window into the cited span, falls back to
leading window when no token matches).
- aborist/qa/parse_claims.py (new): pointer-line parser walks
lines, pulls [E\d+] / [E\d+,E\d+,...] tags, returns ParsedClaim
with PARSED / NO_EVIDENCE_POINTER status. Strict regex refuses
fuzzy alternatives so honest UNGROUNDED beats lax acceptance.
- aborist/qa/verify.py: verify_claim_lattice. Hard checks only —
parser succeeded, evidence_id resolves, source_role allowed, no
manual quotes (any " char violates), claim text non-empty.
Returns evidence_id_pairs (content-addressed, run-stable) for the
run-DAG. Soft signals (entailment, completeness, predicate
compatibility) stay sidecar.
- aborist/qa/dag.py: 9-stage CTI shape when evidence_map_root is
supplied — question / retrieval / evidence_map / prompt /
raw_answer / parsed_claim_lattice / verify / render /
final_label. Quote mode keeps the original 7-stage shape so
pre-G0 run_dag_root values stay valid.
- aborist/qa/query.py: per-chunk evidence-map build with role-
weighted budget AND query-relevance ordering. Within each source,
chunks are ranked by (distinct_query_tokens_present,
total_mentions, doc_order_asc) so the chunk most likely to
textually support the question gets the lowest pointer id. Without
this re-rank Hermes-3-8B lazy-anchored on doc-order-first chunks
regardless of relevance.
- aborist/qa/runner.py: same answer_mode branch for the single-doc
ask() path. No iterative repair in pointer mode (one-shot
benchmark discipline).
- aborist/store.py: verifier_method CHECK extended with
'claim_lattice'. New _rebuild_providence_cache_claim_lattice
migration preserves run_dag_root / run_dag_blob across the
rebuild — older rebuilds dropped them.
- aborist/cli.py: --answer-mode {quote,claim_lattice_pointer} on
query and ask.
- Makefile: ANSWER_MODE knob; defaults to claim_lattice_pointer for
`make query` so the testing harness exercises the new path.
Library DEFAULT_POLICY / DEFAULT_QUERY_POLICY stay "quote" so
Python callers and unit tests aren't surprised.
Prompt: one-shot worked example (Apple founders) plus strict
no-quotes / no-JSON / no-markdown / plain-prose-only rules. Without
the worked example Hermes-3-8B drops the bracket protocol on roughly
half of runs; with it, JP dinosaurs benchmark went 0/13 -> 10/11 ->
16/17 verified across the iterations that hardened the pipeline.
Tests: +48 covering parser, two-layer ids, verifier failure modes
(UNKNOWN_EVIDENCE_ID / SOURCE_ROLE_BLOCKED / MANUAL_QUOTE_VIOLATION
/ NO_EVIDENCE_POINTER / SCHEMA_INVALID), spotlight rendering with
buried-term fixture, per-chunk evidence map, query-relevance
ordering, schema migration round-trip, 9-stage DAG shape divergence.
Total 438 passed, all 7 production shards report 0 chain breaks.
Known limitation: chunk boundaries can cut wikitext mid-template, so
mwparserfromhell-backed to_base() leaves orphan </ref> tags and
leading list markers in the visible spotlight excerpts. Verifier and
CTI architecture are unaffected; the leak is cosmetic. Proper fixes
are template-aware chunking (chunker bump invalidates prior records)
or an orphan-marker post-strip in aborist/wikitext.py — both out of
G0 scope.
burn-kindergarten flake fix:
test_burn_kindergarten_zero_seconds_burns_everything failed under
full-suite ordering when the wall-clock second rolled over between
seeding the row & running burn-kindergarten. Old impl computed
`cutoff = now - 0 = now` and required `created_at >= now`, so a row
seeded at second T became invisible to a burn check that ran at T+1.
The verb's docstring is "0-second window = burn every live row".
Fix the impl to match: when kindergarten_seconds <= 0, drop the time
gate entirely (burn all live records). Non-zero windows keep the
existing cutoff-based filter unchanged.
Repair CLI:
`aborist query --repair` enables the mechanical repair pass after
first verify. `--repair-reprompts N` adds the optional re-prompt
tier (default 0 = off; mechanical-only when --repair is set without
--repair-reprompts). Both flags are off by default so `make query`
stays single-shot unless the operator opts in.
Makefile: `make query REPAIR=1` flips on mechanical repair;
`make query REPAIR=1 REPROMPTS=1` adds one re-prompt iteration.
Both knobs off by default to preserve existing behavior.
485 tests pass under full-suite ordering (was 484 + 1 flake).
Two enhancements continuing the toy-Hermes design pass:
#1 Chain-segment failure localization
aborist/qa/dag.py: `localize_failure(audit_mode, n_sources, n_quotes,
n_verified)` maps a non-STRICT verdict to the pipeline stage that
introduced the failure:
retrieval no admitted sources (gate over-rejected, or corpus
genuinely lacks the topic) → ingest more / relax breadth
context sources retrieved but no quotes extracted (per-source
cap dropped relevant content, or model declined to cite)
→ raise cap / tighten prompt
answer quotes extracted but didn't all verify (model fabricated,
paraphrased inside quotes, appended citations) → mechanical
+ re-prompt repair targets exactly this case
`failure_stage` lands on the run_dag's verify node payload AND on the
result dict so an operator can read the reason at a glance — `failure_stage='answer'`
means stop tuning the verifier & fix the model behavior. Debugging
becomes typed instead of vague.
#2 Re-prompt repair (second tier of the hybrid loop)
aborist/qa/repair.py: `reprompt_repair(...)` builds a feedback message
naming the failed quotes & asks the model to rewrite using only
verbatim citations. Hard rule: only fires when
`policy["repair_max_reprompts"] > 0` (default 0); caller enforces the
cap by looping at most that many times.
aborist/qa/query.py + aborist/qa/runner.py: after mechanical repair,
if the answer is still HYBRID/UNGROUNDED with unverified quotes,
loop up to `repair_max_reprompts` times. Each iteration: build
feedback (assistant turn with current answer + user turn with failed
spans), call LLM, verify. Accept the new answer if `n_verified`
strictly improved; otherwise break (the model's not converging,
don't waste cycles).
The mechanical + re-prompt combination handles the cases each tier
declines individually:
mechanical alone: synthetic_elision split, trailing_artifact trim,
no_overlap remove
+ re-prompt: paraphrase, partial_paraphrase, interior_elision
needing semantic judgment, fabrications the
model can recognize when shown its own quote
`repair_max_reprompts` lives in DEFAULT_QUERY_POLICY +
DEFAULT_POLICY so it folds into governance_policy_hash. Default 0
preserves single-shot semantics for callers that don't opt in. Each
re-prompt iteration adds a `{action: reprompt_rewrite, diagnosis:
model_feedback_loop}` entry to repair_changes; audit chain captures
the full transition through the existing providence_repair event.
Tests:
- dag: localize_failure across all four cases (STRICT, retrieval,
context, answer); failure_stage embedded in run_dag verify node.
- repair: stub client with sequenced answers (failing first, clean
on re-prompt) — assert two LLM calls, STRICT verdict, reprompt_rewrite
in the change log.
484 tests pass (dag +5, repair +1). The pre-existing test_burn flake
under full-suite ordering remains; passes in isolation.
