Captures the canonical "model knows it but corpus doesn't" case fox
surfaced 2026-05-02:
Q: what is boss baby?
Expected: UNGROUNDED (the 2010-11 wiki snapshot has no Boss Baby
coverage — the 2017 DreamWorks film didn't exist; the 2010 Marla
Frazee picture book had just been published & no article)
Model: emits training-prior content about the film/book
Verifier: 0/N verified — no evidence chunk anchors the claims
This is the failure-by-honest-admission case the verifier must
preserve. Two compounding false-positive risks the test guards:
1. Verifier accepting the claim against an unrelated chunk that
happens to share content tokens (Death of Baby P, Cake Boss,
Beanie Baby — all adjacent-token noise observed at top-K).
This is the warrant-failure class warrant-lite was built to
catch — if it ever passes here, anchor-class warrant has a hole.
2. Verifier accepting via OR-mode FTS5 noise — a token-level match
of "baby" or "boss" in unrelated content surfacing as positive
grounding.
Acceptable behaviors:
A. UNGROUNDED (verifier returns 0/N — corpus-truth over prior)
B. STRICT/HYBRID without "boss baby" anywhere in the answer
(deflection — model declined the false-corpus question and
answered an adjacent grounded fact, like Mars-BDFL → Guido
pattern).
Failure: STRICT/HYBRID + "boss baby" in answer = false-positive
grounding. Test fails by name pointing at the warrant gap.
Live verified (2026-05-02): UNGROUNDED 0/2, model emitted both the
2017-film claim AND the 2010-book claim, verifier rejected both
honestly. Test passes in 10s.
The previous fix capped OR-mode at top-5 longest tokens which made
the long brain-tech query fast (118s → 13s) but lost the synonym
retrieval that surfaced Telepathy / Neurotechnology — those titles
weren't reachable via the original query tokens alone.
Fix: pass synonym-expanded tokens to the OR-mode fallback as
``extra_or_tokens``. The merged pool keeps the top-5-longest cap so
retrieval cost is unchanged, but long synonym tokens like
"neurotechnology" (15 chars) and "neuroimaging" (12 chars) now beat
shorter query tokens like "thoughts" (8) by length & surface the
right titles.
OR pool example for the brain-tech query:
Pre-synonym: reconstruct, technology, understand, available, thoughts
Post-synonym: consciousness, neuroimaging, clairvoyance, compensation, reconstruct
→ 3/5 brain-tech terms surface naturally in OR-mode.
AND mode stays unchanged (synonyms in AND would relax the strict-
relevance constraint & pull in noise — wrong tradeoff). The
synonym signal flows ONLY into OR-mode-fallback.
Live verified on the 19-token brain-tech query:
- 13s total (Hermes 6-8s + retrieval ~1s + verify ~2s)
- 2/2 verified pairs
- Cited Functional neuroimaging E5 + FreeSurfer E9 (both real
brain-tech corpus sources, not Universal-pragmatics nonsense)
`_search_corpus` recomputes synonym_expand(qtokens) once into
`or_synonym_pool` (caches in concepts.query._CACHE after first
shard); each shard's FTS5Backend.search() receives the same set
through `extra_or_tokens=`. Cost stays sub-second per shard.
Tests: 641 passed (no regression).
Both retrieval phases were doing O(corpus × tokens) full-scan work
on long queries — synonym work surfaced the latent issues but they
predate it. Hermes is 6-8s; retrieval needs to be sub-second.
(1) `_search_titles` (LIKE-based title backup search) is fundamentally
O(corpus × |accept_tokens|) — `LOWER(title) LIKE '%tok%'` can't
use any index. 50 patterns × 870k docs × 4 shards = ~56s pure
full-scan. Skipped for queries with > 5 accept_tokens, where
FTS5 BM25 already returns better candidates. Title-LIKE remains
a backup for short focused queries (1-5 tokens).
(2) FTS5 OR-mode fallback was OR-joining ALL post-stopword query
tokens. A 19-token query OR-clause matched millions of docs;
BM25 then ranked them all. 13s/shard observed.
Capped to top-5 LONGEST tokens (proxy for rarity / topical
specificity). Long words like "neurotechnology" matter; short
words like "soon" don't. OR-mode now 0.25s/shard.
(3) Stopword set extended with 6 connector words (one, some,
another, without, soon, currently). Conservative addition —
these have zero topical signal. Reduces the AND-mode token
count, raising the chance AND succeeds before falling back to
OR.
Verified live: the 19-token brain-tech query that hung indefinitely
now returns in 18s (Hermes + verify + render dominate). Source
`_search_corpus` no longer expands synonyms for the title-LIKE call
either — synonym expansion stays in `_filter_by_title_relevance`
(post-retrieval, in-memory, cheap regardless of accept-set size)
where it actually does useful work.
Tests: 641 passed (no regression).
Two compounding crashes/hangs in the corpus-derived synonym layer
that the bench surfaced 2026-05-02:
(1) Union-find chained reciprocal-link clusters into one giant
connected component (54,538 tokens for any seed in the 4-shard
Wikipedia corpus). One query token expanded to the entire
synonym alphabet, tripping SQLite's expression-tree-depth=1000
limit on the OR-clause in `_search_titles`. Replaced with
direct-neighbor adjacency only.
(2) Even direct-neighbor expansion was too noisy on generic tokens:
a 19-token query expanded to 419 tokens (every token had ~22
reciprocal-link neighbors averaging out to topic-adjacency
noise). Title-LIKE on 419 patterns × 3.47M docs × 4 shards
hung indefinitely.
Fix: split index by evidence_kind, cap derived expansion only.
- ``manual_index`` — manual_legacy + manual rows. Curated; ALWAYS
expand regardless of per-token degree. The brain-tech / AMD-
family / Mac / Linux / etc. seed groups have legitimately many
members per token after seed.py started writing clique edges.
- ``derived_index`` — link_reciprocity & corpus-extracted edges.
Subject to MAX_NEIGHBORS_PER_TOKEN=8 cap. Generic tokens
("person", "thoughts") have huge degree from Wikipedia link
noise; specific named entities have small focused neighborhoods
that pass the cap.
- ``MAX_TOTAL_TOKENS=50`` overall cap on expanded set. Bounds
the SQL clause count so title-LIKE search stays tractable.
seed.py: write CLIQUE edges within each legacy group (every
member-pair, not just anchor→member). Preserves the legacy
frozenset semantic where any member retrieves every other
member. Quadratic in group size but groups stay small (largest
is the 30-member brain-tech → 435 pairs).
Live shard 000 re-seeded: 80 → 747 manual_legacy rows. Idempotent
re-seed via INSERT OR IGNORE — re-running adds nothing new.
Verified:
- 19-token query: 419 → 50 expanded (cap saturated)
- thoughts: 30+ tokens incl. brain-tech members preserved
- athlon: full AMD-family clique (amd, duron, opteron, ryzen, …)
Tests: 14/14 concept tests pass.
Why: union-find over the full Wikipedia reciprocal-link synonym
graph collapses everything into one giant connected component
(54k+ tokens for any seed in a 4-shard corpus). That made one
query token expand to the entire synonym alphabet, which:
(a) overwhelmed retrieval relevance — every query pulled in
thousands of off-topic candidates via synonym chaining;
(b) tripped SQLite's expression-tree-depth limit when
_search_titles built a per-token OR clause (fox-surfaced
2026-05-02 long-question regression — the qa/query.py side
of that fix landed in faed58d via a separate OR-chain cap;
this is the concepts-side fix to the same underlying
explosion).
Direct-neighbor adjacency: each token maps to its immediate
synonym partners only, no transitive chaining. Symmetric — a
(A,B) edge registers both A→B and B→A. Preserves the legacy
frozenset semantics (every member of a group expanded to the
others in that group, but groups didn't chain).
Test impact: three test_concepts.py tests assert the old
transitive-closure behavior:
- test_synonym_expand_amd_pulls_athlon_and_back (expects
athlon → {amd, duron, thunderbird})
- test_mac_windows_rivalry
- test_brain_tech_synonym_group_includes_telepathy_and_thoughts
They fail under direct-neighbor expansion. Either update the test
fixtures to seed direct edges (athlon, duron) etc., or update the
test expectations to match the new direct-neighbor contract.
Deferred — design choice, fox to decide which path matches intent.
Regression fox surfaced 2026-05-02:
Q: "what technology are currently or soon available which may
enable one person to reconstruct and understand some or a
portion of another persons thoughts or ideas without
speaking or sign language."
→ sqlite3.OperationalError: Expression tree is too large
(maximum depth 1000)
Root cause: the v1 of _search_titles (commit 0052845) chained N
``CASE WHEN ... THEN 1 ELSE 0 END + ...`` expressions for the
title_score column. Each CASE WHEN is multiple tree nodes; +-
chained N times exceeded SQLite's default 1000-depth bound on
question texts with ~30+ content tokens.
Fix: simplified SQL — OR-chain WHERE + ORDER BY LENGTH(title) ASC
+ LIMIT bumped 4x to compensate for the lost smart sorting. The
caller's post-filter (word-boundary stem-aware token-set
intersect) does the actual title-relevance ranking; SQL just
needs to surface enough candidates for the post-filter to grade.
Also: cap the OR-chain at MAX_TITLE_LIKE_TOKENS=24 so pathological
200-token queries don't cascade SQL expression growth even
defensively. Beyond ~24 tokens the post-filter is doing all the
work; extra LIKEs just inflate candidate sets without signal.
5 new tests in tests/test_query.py covering the regression at
unit / integration / functional layers:
- test_unit_search_titles_handles_long_question_without_crash
50-token query through _search_titles directly. Pre-fix raised
sqlite3.OperationalError; post-fix returns row list.
- test_unit_search_titles_handles_zero_tokens
Defensive: empty token list → empty result, no SQL executed.
- test_unit_search_titles_caps_or_chain_at_max_tokens
200-token pathological query — bounded by MAX_TITLE_LIKE_TOKENS,
doesn't crash.
- test_integration_query_completes_on_long_question
End-to-end query() with StubClient + long question completes
without the SQLite error. Pre-fix raised before reaching the
LLM call.
- test_functional_long_question_returns_sources
The neurotech doc (richest body match) appears in top-K despite
the long-question retrieval path.
Test_query.py: 35 → 40 passing. Full suite (excluding parallel
test_concepts churn from concepts/query.py rewrite): no other
regressions.
Two paired changes addressing fox's "learn this hyperparameter
from model use, not hard coding" + the 1M-context-window
caveat:
(1) Log-scale prompt-size buckets extend from 8KB through 1M+:
<8KB / 8-16KB / 16-32KB / 32-64KB / 64-128KB / 128-256KB /
256-512KB / 512K-1M / >=1M
The same bench harness now covers 8B-class models (Hermes 82K
context, max useful prompt ~32-64KB) through 1M-context models
(Gemini 1.5 Pro, Claude with extended context, Llama 4) without
code change. A model whose context window stops at 82K simply
never populates the giant buckets; a 1M-context model fills
them and finds its own sweet spot.
(2) New "recommended context budget (learned from this bench)"
section — per-mode peak-strict-rate bucket. Operator-driven
landing per the five-step algorithm step 5: surfaced, not
auto-applied. Minimum sample size of 5 runs per bucket so
statistical noise doesn't masquerade as signal. Tie-break on
smaller-bucket-wins so equivalent strict-rates favor the
cheaper choice.
The substrate is now self-tuning at the OBSERVATION layer: bench
records what budget actually grades best per model. Per-model
profile JSON (storing the recommended budget back into
~/.aborist/model_profiles/<model>.json) is the next beat once
this surface is observable in real bench runs.
3 new bench tests cover the recommended-budget section, the
sample-size floor, and the giant-context bucket coverage.
Full suite: 746 passed (was 738, +8).
Connects to D8 (automate after test-pinning): the bench tells us;
we don't guess.
New `docs/concept-relations-design.md`: architecture reference for
the per-shard concept_relations layer that replaced the legacy
frozenset module (commit 5fd458a). Covers:
- Why phase 1 (hand-curated frozensets) didn't scale.
- Append-only schema + the three by-construction properties (idempotent
re-derivation via UNIQUE, per-shard storage, Merkle-orthogonal).
- Built-in `link_reciprocity_synonym` extractor reading the existing
`edges` table — no new crawler, works for Wikipedia AND HTML sites.
- Measured storage: 95.58 MB across 4 wiki shards (3.47M docs,
10.75M resolved edges, 55,148 reciprocal pairs, 289,848 synonyms),
4m16s wall-clock backfill. 1.6% tax on the 6 GB corpus.
- Three storage compactions considered & rejected, each with the
specific trade-off it loses on (drop idx_concept_evid → painful
purge debugging; BLOB source_root → schema inconsistency; FK
normalization → JOIN in retrieval hot path).
- How-to: backfill, manual add, purge.
- Adding new extractors.
- Deferred follow-ons (CLI commands, Wikipedia See-also extractor,
category extractor, hatnote extractor).
CLAUDE.md item 5 in the retrieval-pipeline list updated to point at
the new module path (aborist/concepts/) and the design doc.
TICKETS.md reference list updated to mention the new design doc.
Phase 2 of the concepts/ layer. The Apr 27 commit (c6182ae) shipped
hand-curated frozensets in aborist/qa/concepts.py with a TODO to
"derive from Wikipedia's category graph or 'See also' sections" —
that's this commit. Fox's 2026-05-01 critique landed it: a 7-entry
list of arbitrary frozensets won't scale to a 3.47M-doc corpus, &
adding domains shouldn't require a Python edit + commit + redeploy.
Architecture:
(1) Per-shard concept_relations SQLite table. Append-only, with
UNIQUE (source_root, relation_kind, token, target, evidence_kind)
so re-derivation is idempotent. Lives next to documents in each
shard so mesh sync moves relations alongside the docs that
derived them. A SECONDARY index — writes here NEVER affect
document_root / chunk_root / cache_key, so backfilling is safe
across the entire corpus.
(2) aborist/concepts/ package:
- store.py: add_concept_relation, concept_relations_for_token,
purge_by_evidence_kind, list_evidence_kinds
- query.py: cross-shard synonym_expand, rivalry_excluded;
union-find collapse on synonym edges so partial
pairs build full equivalence classes; mtime-keyed
per-process LRU so retrieval doesn't re-walk
shards on hot loops
- seed.py: one-shot migration of legacy frozensets (8 groups
incl. brain-tech) to evidence_kind='manual_legacy'
rows under source_root='__legacy__concepts__'
- extract.py: pluggable extractor registry. Built-in:
link_reciprocity_synonym — for any reciprocal
edge pair (A→B AND B→A) in the existing edges
table, emit synonym edges between the docs'
title-tokens. Works for Wikipedia (See-also
bidirectional), HTML site internal-link clusters
(russell.ballestrini.net pattern), or any link
graph the corpus already encodes — no new
crawler needed; the html_page parser already
populates `edges` rows on ingest.
(3) aborist/qa/concepts.py rewritten as a backwards-compat shim —
same public API (synonym_expand, rivalry_excluded,
has_compare_phrasing) so query.py call sites unchanged.
shards_dir threaded through _filter_by_title_relevance &
_search_corpus' synonym_expand calls. Without shards_dir
(legacy 2-arg call shape), helpers degenerate to no-op —
matches the behavior the frozenset code had when no group hit.
(4) Cross-shard UNION view: concept_relations added to
_SHARDABLE_TABLES in store.py so connect_query() exposes a
unified view across all shards (same pattern as documents,
chunks, providence_cache, etc.).
Live-verified on the 4-shard 3.47M-doc corpus + the
crawl_russell_ballestrini_net.db shard:
Q: what technology are currently or soon available which may
enable one person to reconstruct and understand some or a
portion of another persons thoughts or ideas without speaking
or sign language?
Result: same as the bde1bd6 in-memory frozenset version —
Telepathy E4 cited alongside Videoconferencing E1/E2,
POINTER-LINKED-PARTIAL 6/10. The DB-backed lookup reproduces
the frozenset behavior byte-for-byte.
Tests: 633 passed (was 624). 9 new concept tests covering
DB-backed synonym/rivalry lookup, shards_dir=None degenerate
behavior, brain-tech group seed, cache invalidation. Old tests
that called helpers directly without shards_dir kept as no-op
assertions (synonym_expand({"athlon"}) without shards_dir returns
{"athlon"} unchanged).
Backfill mechanics: existing live shards needed a one-shot
`connect()` to auto-create the new concept_relations table
(SCHEMA_SQL has CREATE TABLE IF NOT EXISTS). Re-derivation never
mutates the Merkle tree — concept_relations is fully orthogonal
to document_root / chunk_root. Cache_key dimensions are
unaffected.
Deferred (follow-on):
- CLI commands: aborist concepts {seed,list,add,derive,purge}
(currently fox runs the helpers via python -c)
- Wikipedia See-also extractor (requires parsing wikitext sections
beyond what's already in edges)
- Wikipedia category extractor (requires reading Category: links
from chunked wikitext)
Promotes the title-relevance sidecar (diagnose_title_relevance,
qa/inspect.py) to a hard check inside both verify_claim_lattice
and verify_claim_lattice_json. New violation kind TITLE_MISMATCH;
demote-to-HYBRID semantics matching the existing WARRANT_MISSING
pattern.
Catches the 2026-05-02 fox-surfaced retrieval-driven hallucination
class:
Q: "explain spin glass modeling & tensors?"
Pre-Rule-8: EVIDENCE-WARRANTED 1/1, claim cited to Quantum
chromodynamics chunk (single-line "See Also: spin
glass" reference). Token-coverage check passed
accidentally on shared physics vocabulary.
Post-Rule-8: POINTER-LINKED-PARTIAL · title mismatch 1/1.
Cited source title 'Quantum chromodynamics' shares
zero stemmed tokens with claim's {spin, glass,
modeling, tensor, ...} → demote.
Implementation:
_claim_title_overlap(claim_text, source_title) returns True iff
the source title shares ≥1 stemmed content token with the claim.
Uses qa.evidence._content_tokens (≥4-char, post-stopword) and
inline minimal stem (s-strip on tokens >4 chars, skip ss-enders)
to avoid an import cycle.
Per-claim loop in both verifiers checks every cited source's
title; ANY-match suffices (only TITLE_MISMATCH when ALL cited
titles miss). Vacuous-pass when claim or title has no
extractable tokens.
Renderer (_ladder_rung_for_lattice) treats TITLE_MISMATCH
alongside WARRANT_MISSING as the POINTER-LINKED-triggering
signal — both indicate citation/claim structural misalignment.
_render_warrant_tail surfaces "· title mismatch" alongside
"· warrant missing" so an operator sees the specific failure
mode at the audit-line.
Test fixtures updated where the synthetic claims were too minimal
(e.g. "Velociraptors are shown attacking workers" cited to
"Jurassic Park (film)" — claim had no topic anchor). Real model
output naturally references the topic (the model sees the title
in the evidence map and uses it); the fixture revisions reflect
that. 4 new tests in test_verify_json.py covering the helper +
end-to-end TITLE_MISMATCH demote.
Live verification:
Spin-glass query: POINTER-LINKED-PARTIAL · title mismatch ✓
Homer/Mr. Burns: EVIDENCE-WARRANTED ✓ (no regression)
CLAUDE.md updated with the Rule 8 convention; existing
diagnose_title_relevance sidecar marked legacy / dict-form for
per-cache-key inspect use.
Full suite: 738 passed (was 734, +4).
Closes the 2026-05-01 retrieval gap fox surfaced on:
Q: what technology are currently or soon available which may enable
one person to reconstruct and understand some or a portion of
another persons thoughts or ideas without speaking or sign language?
Pre-fix: top-K = [Videoconferencing, Telepathy, Identity, PKM,
Intrapersonal]; only Videoconferencing used (BM25 won on
"person/language/speak" token density). Verifier returned
EVIDENCE-WARRANTED-PARTIAL 2/5 with Videoconferencing
cited as evidence for "currently no technology...". Telepathy
was retrieved but unused — title-relevance filter dropped
it to body-density accept path with a low score.
Post-fix: same top-K but Telepathy now SURVIVES title-relevance
(its title token "telepathy" is in the accept set via
synonym expansion). Title-token-boost lifts it.
EVIDENCE-WARRANTED-PARTIAL 6/10 with Telepathy E4 now
cited alongside Videoconferencing — sources used 5/5.
Adds a 7th SYNONYM_GROUPS entry mixing concept tokens (thoughts,
mind, cognition, consciousness) with specialized entity tokens
(telepathy, neurotechnology, BCI, fMRI, EEG, TMS, transcranial).
A query naming either side expands to the whole family, so the
title-relevance accept path admits brain-tech pages even when the
question is phrased in lay vocabulary.
Limitation accepted: this only affects title-relevance filtering &
title-token rerank, not the FTS5 BM25 retrieval that produces the
top-K. Pages whose BM25 score is below the top-K cutoff (e.g.
Neurotechnology at #4 in fox's manual-hint Run 2) still won't
surface unless FTS5-side synonym expansion is added too. Tracked
as a follow-on ticket — for now, query-time retrieval hints
("transcranial knowledge acquisition") continue to be the manual
escape hatch when intent vocabulary doesn't lexically anchor.
Tests: 624 passed (no regression).
Fox-surfaced 2026-05-02 case:
Q: "what date did back to the future come out?"
→ top-8 sources: trilogy, ride, animated-series, The Future,
Future Husband, The Day You Come, Ratchet & Clank Future,
Future plc — film article entirely absent.
Two compounding defects in the title-LIKE retrieval path:
(1) `_search_titles` had no ORDER BY. SQL `LIKE '%token%' OR ...`
returned rows in arbitrary internal order; LIMIT 32 truncated
before genuine matches. The film article `Back to the Future`
(title contains both `back` AND `future`) lost the LIMIT race to
substring-match junk like `Out (poker)` (single-token match on
`out`), `Aberdeen, South Dakota` (`south` contains `out`),
`Backplane` (`back` substring), etc.
Fix: SQL builds a token-hit score via CASE WHEN per token and
ORDERs BY (title_score DESC, LENGTH(title) ASC). The film
article's score=2 (back+future) ranks above garbage's score=1.
Tie-break on title length favors title purity (shorter title +
same overlap = higher per-token signal).
(2) Caller's post-filter check used substring match
(`if t in title_lower`) — same defect as the SQL. "south"
contains "out", "outline" contains "out", "backbone" contains
"back", etc. all passed the no-op overlap check.
Fix: tokenize the title via `_title_query_tokens` (already
drops stopwords + small tokens) and stem-aware-intersect with
the query stems. Word-boundary semantics; only genuine title
tokens count.
Verified live:
Pre-fix: trilogy at #1, film article absent from top-8.
Post-fix: film article at #1, trilogy at #2, junk gone.
Verdict on the BTTF query stays POINTER-LINKED-PARTIAL · warrant
missing because the model picked the box-office chunk (mentions
"1985" but not "July") instead of the infobox/release-date chunk.
That's correct warrant behavior: claim asserts both anchors;
cited span only has one. The retrieval-side defect was a separate
gap that this fix closed; chunk-relevance ranking for cited
content is downstream work.
The phrase-pattern retrieval route (commit 1b8677d) closed the
RETRIEVAL side of reference-frame failure; this ticket closes
the ANSWER side.
New module aborist/qa/frame.py:
FrameDetection dataclass (frame_kind, reference_title,
reference_uri, confidence). Sidecar — never enters cache_key
or governance_policy_hash.
detect_frame(question, sources, phrase_match_roots) — heuristic
detector. Reference-frame classification fires when:
(a) phrase route surfaced at least one source, AND
(b) at least one phrase-matched source is a reference work,
determined by:
- title parenthetical disambig (`(novel)` / `(film)` /
`(play)` / `(franchise)` / etc.), OR
- body sample contains ≥3 DISTINCT fiction markers
(novel / published / protagonist / plot / ...).
Distinct-marker count keeps the heuristic robust against a
history article saying "novel approach" twice.
aborist/qa/query.py:
Calls detect_frame for lattice modes only. Body sample uses the
ARTICLE LEAD (chunk_idx=0, post-wikitext-strip) — fiction
markers cluster in the lead on Wikipedia, not in plot chunks
that may have been query-relevance-ranked higher.
New policy field claim_lattice_polarity_preamble injected as a
user-role message before the grounding_reminder when
frame_kind == "reference". Format-string with
{reference_title} placeholder.
Result dict carries frame_detection (kind / title / uri /
confidence) for renderer + bench consumption.
aborist/cli.py:
Renderer adds a `reference frame: <title>` line when
frame_detection.kind == "reference". Skipped for literal /
no-phrase-route / ambiguous rows.
Live verification — Orwell case:
PRE : "The text does not directly state that Oceania has always
been at war with East Asia."
POST : "In George Orwell's dystopian novel Nineteen Eighty-Four,
the nation of Oceania is always at war with Eastasia,
but this is a result of propaganda and doublethink, not
actual historical continuity. The war with Eastasia is
a fabricated conflict to maintain control..."
Multi-frame compilation: distinguishes propaganda claim from
fictional-actual continuity, exactly the polarity contract from
the ticket §2 abstraction.
Literal queries (capital of france) unchanged — polarity preamble
only injects when frame_kind classifies as "reference".
22 new tests (19 in test_frame.py + 3 in test_cli_render.py).
Full suite: 734 passed (was 712, +22).
Directive D3 status: ½ → ✓. Ticket #000002 closed.
All seven structural directives now ✓:
D1, D2, D5 (were ✓);
D3 #000002, D4 #000001, D6 #000003, D7 #000005 (closed in this
series).
New module aborist/qa/retrieval_plan.py:
RetrievalPlan dataclass — frozen, captures the operator-
influenceable retrieval inputs that determine source
selection (retrieval_keywords, top_k, over_fetch,
max_context_chars, shard_ids).
retrieval_plan_hash() — SHA-256 over canonical-JSON.
Deterministic per call; folds into the run-DAG retrieval
stage as a bound input alongside the existing
sources_summary output.
build_run_dag (aborist/qa/dag.py) accepts retrieval_plan_hash:
When provided, the retrieval stage hash binds BOTH plan
(input) and result (output): H({"retrieval_plan_hash":...,
"sources_summary_hash":...}).
When omitted (e.g. legacy / quote-mode callers that haven't
plumbed it yet), the retrieval stage falls back to the
historical sources-summary-only hash so pre-#000001
run_dag_root values stay stable.
query.py constructs the plan per call and passes it through.
Question text intentionally NOT in the plan — already covered
by question_hash. Shard ids included so audit can reproduce
which shards the search ran against.
Two runs with identical sources but different retrieval keywords
now produce different run_dag_root values — the provenance gap
on operator hints (auditor recovers "these were the keywords
that pulled in those sources") closes for the run-DAG path.
Marker test in tests/test_directives.py flipped from "absent"
assertion to "present": test_d4_retrieval_plan_binding_landed.
Plus test_d4_retrieval_plan_hash_module_exists pinning the
module shape. Full suite: 712 passed.
Deferred per ticket §6:
- audit events (retrieval_plan_built / retrieval_result_selected)
- providence_cache.retrieval_keywords SQL column
- optional strict cache_key mode (retrieval_plan_affects_cache_key)
These are ergonomic affordances atop the run-DAG binding; earn
their own tickets when bench evidence shows the blob path is
too friction-heavy for real workflows.
Directive D4 status: ½ → ✓. Ticket #000001 closed.
Replaces the two-rung EVIDENCE-LINKED / EVIDENCE-LINKED-PARTIAL
display label for claim-lattice methods with a four-rung ladder
that names a strictly stronger property at each rung:
POINTER-LINKED pointer/source/chunk verified;
warrant either didn't apply or failed
ANCHOR-WARRANTED pointer-linked + warrant passed where it ran;
other soft demotes may apply
EVIDENCE-WARRANTED anchor-warranted + no soft demotes
UNGROUNDED no verified pairs
HYBRID gets a -PARTIAL suffix on whichever rung applies.
Implementation: _render_audit_label gains a violations parameter
(defaults to None for backward-compat). _ladder_rung_for_lattice
discriminates rungs from the existing violations list:
- WARRANT_MISSING in violations → POINTER-LINKED
- any of {LAZY_ANCHOR_DEMOTED, POINTER_OVERFLOW_TRIMMED,
TOO_MANY_CLAIMS, BARE_NAME_CLAIM} → ANCHOR-WARRANTED
- else → EVIDENCE-WARRANTED
Design simplification vs the ticket's §3 sketch: the proposed
verifier_steps_ran field on the verdict dict was NOT needed. The
existing violations list carries enough signal to discriminate
all rungs. Per the five-step algorithm step 2: don't add fields
you don't need.
Quote / span / entity / paraphrase methods stay unchanged (their
STRICT verifies pinned spans, not synthesis).
Schema column audit_mode enum stays {STRICT, HYBRID, UNGROUNDED}
— pure renderer transformation, no governance_policy_hash bump,
no cache invalidation, no mesh-wire-format change. Existing
providence records render under the new ladder on next read.
5 new renderer tests in tests/test_cli_render.py covering each
rung mapping. D7 anti-regression test in tests/test_directives.py
updated to gate on the ladder labels. Bench helper docstring
follows. Full suite: 711 passed.
Directive D7 stays at ✓; ticket #000005 closed.
Three new question-shape classes dispatched through warrant_check
alongside the existing relation + date anchors:
(1) Entity-list shape — `name X`, `list X`, `who are the members
of X`. List-aware extractor `extract_entity_list_anchors`
(multi-word phrases ∪ solo-cap individual names) so comma-
separated entities each contribute. ANY-match semantics:
demote-don't-reject when an extra entity from training-prior
appears alongside grounded ones.
(2) Count shape — `how many X`, `how much X`. Digit ↔ word
equivalence (claim says "six", span says "6", or vice versa)
with ordinal collapse (`sixth → 6`). Year-shaped digits
filter out (those belong to the existing date anchor class).
ALL-match semantics: every count token in the claim must
appear in some cited span as digit or word.
(3) Why-cause shape — `why X`. Cause-anchor pool widens to
≥5-char lowercase common nouns (post a generic stopword set
that filters quantifier-adjective fillers like "various",
"factors", "situation") PLUS proper-noun anchors from the
existing extractor. Gated on why-shape only: lowercase
common-noun extraction has higher false-positive risk
elsewhere.
Per-class policy gate (proposed `claim_lattice_warrant_classes`
dict) deferred per the five-step algorithm step 2: single
`warrant_check_enabled: bool` is the minimum viable gate; per-
class flags earn their slot when bench evidence shows over-firing
on a specific class.
17 new warrant tests (detector + extractor + integration).
Marker test in test_directives.py flipped from "absent" to
"present" assertion: test_d6_warrant_generalization_landed.
Full suite: 709 passed (was 692, +17).
Directive D6 status flipped to ✓ in seven-point-program.md.
Ticket #000003 closed.
Each directive gets a structural pin in tests/test_directives.py.
A future PR that silently weakens a directive fails by name.
D1 (no LLM in hard verifier path):
- verify_quotes / verify_claim_lattice / verify_claim_lattice_json
signatures must NOT carry chat_client / llm / judge / model
parameters.
- SCHEMA_SQL verifier_method CHECK constraint enum excludes
'llm', 'model', 'judge', 'nli', 'hermes', 'ai'.
D2 (Hermes emits pointer clauses):
- ANSWER_MODES contains both lattice variants.
- DEFAULT_ANSWER_MODE in ANSWER_MODES.
- parse_pointer_claims returns structured (claim_text, pointer_ids)
nodes from pointer-line input.
D3 (build CTI internally):
- Runtime parses pointer-line prose into typed claim nodes
(text + pointer_ids list); model never produces structured
output directly in pointer mode.
- Evidence map built by runtime from retrieved chunks; pointer
ids minted sequentially; evidence ids content-addressed.
D4 (bind retrieval AND evidence map):
- build_run_dag accepts evidence_map_root parameter (✓).
- Marker test: build_run_dag does NOT yet accept retrieval_plan_hash
parameter; flip when ticket #000001 lands.
D5 (deterministic pointer verify):
- verify_claim_lattice is deterministic: same inputs → same
verdict byte-for-byte.
- _claim_textually_overlaps_evidence (the per-pair coverage
check) is a pure function with no LLM-side parameters.
D6 (anchor-class warrant before NLI):
- warrant_check signature is lexical-only (no chat_client / llm
/ judge / nli params).
- Date-anchor failure case fires (year missing from cited spans).
- Relation-shape failure case fires when ALL proper-noun anchors
are missing from cited spans.
- Marker test: per-shape detectors for entity-list / count /
why-cause shapes are absent today; flip when ticket #000003 lands.
D7 (rename labels honestly):
- Renderer maps STRICT → EVIDENCE-LINKED for claim_lattice* methods.
- Renderer keeps STRICT for quote / span / entity / paraphrase
methods (those verify against pinned spans, not synthesis).
- SCHEMA_SQL audit_mode CHECK constraint enum stays
{STRICT, HYBRID, UNGROUNDED} regardless of renderer relabel.
D8 (test-pinning before automation):
- docs/seven-point-program.md exists.
- bench/qa_sweep.py exposes _directive_compliance helper.
- docs/TICKETS.md indexes the partial directives (D3, D4, D6) so
the design log doesn't drift from the program doc.
23 new tests; full suite 692 passed (was 669, +23).
The "marker" tests for D4 and D6 document open work: today they
assert that a parameter / detector is ABSENT; when the corresponding
ticket lands, the assertion flips to PRESENT. The test failing
during ticket implementation is the expected signal, not a
regression — the test message names the ticket so the work-in-
progress is legible.
#000004 — Directive coverage in bench summary. Filed as closed
(landed in commit acd1f9c) for design-log traceability. Captures
the v1 substrate: per-row directive_compliance helper, _summarize
aggregation, markdown directive-coverage section, 5 unit tests.
#000005 — Label ladder migration (POINTER-LINKED / ANCHOR-WARRANTED
/ EVIDENCE-WARRANTED / ENTAILMENT-VERIFIED). Status open. Migration
from today's two-rung EVIDENCE-LINKED to a four-rung ladder where
each rung names a strictly stronger property. Schema column stays
{STRICT, HYBRID, UNGROUNDED} so v9.8 cache_key invariants hold;
renderer-level mapping only. Reserves ENTAILMENT-VERIFIED for a
future committed entailment engine without forcing premature
implementation.
TICKETS.md index gains both rows. Next ID bumps to 000006.
seven-point-program.md status snapshot updated:
- D7 row links #000005 (ladder migration proposal).
- D8 row links #000004 (closed substrate).
- New "Anti-regression test layer" section maps each directive
to its structural pin in tests/test_directives.py.
Note on fox's roadmap item #000006 (Internal CTI Expansion / frame
lattice / multi-frame answers): scope overlaps with #000002
(Reference-Frame Polarity Contract / Module L). NOT opening as
a separate ticket — Module L already covers it.
Wires the seven-point program (docs/seven-point-program.md) into
the bench harness as a per-mode coverage table.
Per-row computation in _directive_compliance(answer_mode, result, err):
D2_pointer_clauses — answer_mode is lattice variant
D3_cti_substrate_ready — lattice mode + run_dag_root populated
(full coverage pending ticket #000002)
D4_evidence_map_bound — run_dag_root populated
(retrieval-plan binding pending #000001)
D6_warrant_fired — warrant ran (lattice mode today;
per-shape gating pending #000003)
D7_honest_label — audit_mode in canonical enum
(renderer transformation pinned in
test_cli_render.py)
D1 (no LLM in verifier), D5 (verifier_method enum), D8 (test-pinning
discipline) are global properties of the substrate and don't appear
per-row — they get tracked once in CLAUDE.md / the program doc.
_summarize aggregates per-row booleans into per-mode pass counts.
_render_markdown emits a 'directive coverage (seven-point program)'
section with per-mode per-directive coverage as 'count/N (pct%)'.
Discipline rationale per CLAUDE.md's five-step algorithm step 5:
the bench harness IS the automation substrate, so it must enforce
the directives before any feature gets layered on top. Bench scores
that climb without directive coverage climbing are graveyard-digging
(step 4: don't dig the grave faster).
5 new bench tests cover: per-mode pass aggregation, the markdown
section presence and content, the helper's behavior on quote mode
(D2 fail by design), lattice mode (all directives pass), and error
rows (empty dict — no signal). Full suite 669 passed.
Today's warrant-lite (aborist/qa/warrant.py) covers two anchor
classes: proper-noun (relation-shape questions) and date (any
claim with a 4-digit year). Three failure shapes pass the existing
seven hard checks unchecked:
- entity-list: 'name the simpsons family + pets' — claim
enumerates entities, but the warrant doesn't demand all named
entities appear in some cited span.
- count: 'how many wives did henry the eighth have?' — claim
says 'six', but the warrant doesn't demand the count token
appear in some cited span.
- why-cause: 'why did the titanic sink?' — claim names a cause
noun (iceberg, asteroid, propaganda), but the warrant doesn't
demand the cause token appear in some cited span.
Ticket proposes per-shape anchor extractors dispatched by question-
shape detector (existing _question_is_relation_shape pattern).
Conservative scope on cause anchors: gate by why-shape only since
lowercase common-noun extraction has higher false-positive risk.
Folds into governance_policy_hash via per-class policy dict so an
operator can disable individual classes per-corpus when they
over-fire.
Out of scope per docs/naming-deferral.md: typed-contract framework
(per-question-type rule libraries). General anchor-class primitive
catches the failures listed without per-type rule books; typed
contracts earn slots only when bench evidence shows the general
primitive misses cases.
Forecast cost ~3-4 hours; risk medium (false-positive risk on
why-shape cause-noun extraction). Closes the lazy-anchor gap on
the question shapes the bench question set already exercises.
The 2026-05-01 distilled architectural directive becomes the
audit lens applied before every new ticket / feature / prompt
edit. Each of the seven directives gets:
- status (✓ / ½ / ✗)
- code anchors (verifier paths, renderer hooks)
- pinning tests (per-rule coverage in tests/)
- tickets covering remaining work
- bench signal (which row column reports it)
Status snapshot at landing:
D1 (no LLM in verifier) ✓
D2 (pointer clauses) ✓
D3 (CTI internally) ½ → ticket #000002
D4 (retrieval map AND evidence map) ½ → ticket #000001
D5 (deterministic pointer verify) ✓
D6 (anchor-class warrant) ½ → ticket #000003 (this commit pair)
D7 (honest labels) ✓
D8 (test-pinning before automation) discipline
CLAUDE.md docs index gains a "North-star" section pointing at the
program; ticket entries gain a "Directive" column threading them
back to the program.
Generalizes warrant-lite from relation-only to anchor-class warrant.
Two anchor classes now compose:
(1) Proper-noun anchors — gated on relation-question shape; any-match
semantics (existing behavior).
(2) Date anchors — always-on, gated on the CLAIM containing a year or
month name. ALL components (year + month name if present) must
appear in some cited span.
Closes the back-to-the-future failure fox surfaced 2026-05-01:
Q: what date did back to the future come out?
Claim: "Back to the Future was released in theaters on July 3, 1985."
Cited span: trilogy/SNES/pinball prose with '...real 1985...' but
no 'July' anywhere
Pre-fix: EVIDENCE-LINKED · via claim_lattice 1/1
Post-fix: EVIDENCE-LINKED-PARTIAL · via claim_lattice · warrant missing 1/1
Year extraction (4-digit, 1500-2199 range — avoids matching ZIP codes
/ elevations / model IDs) and month extraction (full English month
names — three-letter abbrevs deferred until bench shows the gap).
Both anchors required when both are in the claim. Year-only claim
against year-only span continues to pass; month-only against
month-only continues to pass; month-and-year claim against year-only
span (the BTTF case) now correctly fails.
Architectural note: this generalizes warrant-lite without building a
typed-contract framework (per-question-type rule libraries like
"release_date_lookup with required predicate cues"). The date anchor
primitive catches the date lazy-anchor failure class deterministically
without a per-type rule book; typed contracts earn their slot only if
bench evidence shows general anchors miss specific cases. Dropped the
caller-side is_relation_question gate from verify.py — warrant_check
now decides internally what to enforce based on question shape AND
claim content.
Tests: 29 unit tests in tests/test_warrant.py (was 16). New tests
cover extract_date_anchors (year, month, dedup, non-year-skip,
empty), warrant_check date semantics (BTTF case, year-only,
year-in-unrelated-context, month+year composition), proper-noun
warrant skipped without relation question. Full suite: 554 passed,
21 skipped.
Limitation accepted: ISO date format ("1985-07-03") wouldn't satisfy
month-name requirement. Real-world Wikipedia prose uses month names
("released on July 3, 1985"), so the trade-off favors catching the
common case. If bench shows ISO false-rejections, layer in a
month-number alternative to the month-name regex.
Walked the five-step algorithm on CLAUDE.md itself.
Step 1 — requirements check: every section earned its place via a
specific operator-failure context. Sections that had grown into
prose essays got compressed back to load-bearing rules + pointers.
Step 2 — delete:
- Bench-maxing prose section (~100 lines) → moved verbatim to
docs/bench-maxing.md. CLAUDE.md keeps a one-block headline list
pointing at the doc.
- Conventions section: each rule tightened to one paragraph max,
with a `See aborist/qa/verify.py` (or similar) trailing pointer
so the operator can jump straight to the rationale in code.
Verbose duplications between conventions and retrieval-pipeline
sections (e.g. wikitext base prose explained twice) collapsed.
- Retrieval pipeline: each of the 9 stages now references the
relevant module (`qa/concepts.py`, `query.py`) instead of
re-narrating the failure case in prose.
- Architecture tree: minor trim, removed redundant comments
where the filename already names the role.
Step 3 — simplify: source papers section dropped one item that was
duplicate (PDF + RST point to same content); kept the canonical
source.
Steps 4 + 5 — n/a (this is a doc, not a process).
New section: Docs index. Lists every architectural / design doc
in docs/ plus a ticket sub-list (open tickets with their files).
Closes the discoverability gap where TICKETS.md existed but agents
didn't know to look at it.
Net: CLAUDE.md goes from 308 → 311 lines BUT the avg information
density is up — bench prose doesn't bloat the entry-point doc, and
each convention now points to its source. The `docs/bench-maxing.md`
extraction is the real win: separation of "rules I must not break"
(CLAUDE.md) from "discipline I should internalize" (bench-maxing.md).
Engineering discipline added between operational rules and the
orientation protocol. Walk the steps in sequence; skipping makes
the next ones expensive.
1. Make the requirements less dumb (every requirement gets a name,
not a department).
2. Delete the part or the process (10%+ put-back rate or you
aren't deleting hard enough).
3. Simplify and optimize (only after 1 + 2).
4. Accelerate cycle time (only after simplifying — don't dig the
grave faster).
5. Automate (last, not first).
Connected to existing aborist conventions:
- Step 1 traces every policy knob / schema invariant / verifier
check to a specific defect or owner. Aligns with the
"feedback memory" pattern (record the WHY).
- Step 2 reinforces verifier-stays-binary and "no soft signals
in the hard chain" — both are deletion-first rules.
- Step 5 is why bench harnesses, distill runners, and ingest CLIs
earned automation only after their underlying ops were
simplified to single-purpose stages.
Two follow-ups to the phrase-pattern retrieval fix (commit 1b8677d)
covering items 6 and 10-11 of fox's 2026-05-01 architectural review:
(1) Non-regression tests for the phrase route:
- test_phrase_route_skipped_when_question_shorter_than_min_n
pins the structural false-positive guard: the n=5/n=6 minimum
means a 4-token literal-geography query lacks enough tokens to
trigger the route at all.
- test_phrase_route_does_not_hijack_literal_geography_query
end-to-end: a 4-token "oceania east asia geography" query on
a synthetic 2-doc corpus surfaces only the geography-stub doc;
the orwell-stub doc (whose body has the diagnostic 5-gram) is
correctly NOT pulled in by the phrase route on a literal query.
(2) docs/ticket-000002-reference-frame-polarity-contract.md
Captures fox's Module L proposal verbatim as Appendix A and
extracts the implementation sketch into the standard ticket
body (problem statement, abstraction, CTI interpretation, three
pieces of code to write, test list, scope boundaries).
The phrase route closed the RETRIEVAL side of reference-frame
failure. Module L addresses the ANSWER side: today's substrate
answers Orwell queries as "the text does not directly state..."
when it should produce multi-frame answers distinguishing
Party propaganda from fictional-actual continuity. Forecast
cost ~3-4 hours; risk medium (prompt augmentation interaction
with claim_lattice prompt).
Module M = ticket #000001 (route provenance binding); not
duplicated. Module N (FP guards) partially landed via the
tests above; remaining tests folded into ticket #000002's
test list. Module H (relation warrant lite) lacks scope
detail; deferred without a ticket.
(3) docs/TICKETS.md updated: index gains #000002 row, Next ID
bumped to 000003.
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
Two UX fixes addressing the 2026-05-01 Orwell run feedback:
(1) Evidence pointer format. Old form `[E5: "<excerpt>"]` looked
visually like a 1-indexed source rank — paired with a "sources (8)"
list whose `[5]` slot was a different document, operators easily
mis-attributed citations. New form:
[E5 | Nineteen Eighty-Four | 682f0a11: "<excerpt>"]
Title comes from EvidenceObject.title (URI-tail fallback when None);
chunk_root prefix is the first 8 hex chars — enough to disambiguate
chunks from the same source while staying compact. Renderer change
in render_claim_lattice; pre-existing claim-lattice tests updated;
3 new unit tests pin the format (title-inline, URI-tail fallback,
no `[E1:` collision-shape).
(2) Audit-label discipline. _render_audit_label already maps STRICT
→ EVIDENCE-LINKED for claim_lattice* modes (verifier output stays
binary, schema CHECK constraint stays); 4 new tests in
test_cli_render.py pin the relabel for STRICT / HYBRID / UNGROUNDED
in claim_lattice modes and the no-relabel for quote-mode where the
verifier verifies pinned spans not synthesis.
Net: 8 new tests, full suite 649 passed.
Pre-fix: _spotlight_excerpt used first-match-of-longest-content-token.
For "who is homer simpson's boss?" the Homer Simpson chunk had
"homer" matching early in voice-actor / Castellaneta prose ("For
voicing Homer..."), so the spotlight centered there even though
the actual answer ("his boss Mr. Burns") was deeper in the chunk
where boss/burns/homer cluster together.
Post-fix: density rank. Find ALL match positions for ALL content
tokens, then pick the position whose ±half-window cluster contains
the MOST distinct tokens. Tie-break on smallest idx (deterministic,
reproducible).
Live verified:
- "who is homer simpson's boss?" — spotlight now lands on
"...At the plant, Homer is often ignored and completely
forgotten by his boss Mr. Burns..." (boss/burns/homer/plant
cluster). Pre-fix it landed on voice-actor prose with no
Mr. Burns visible.
- "name simpsons family members including pets?" — spotlight
lands on "...the family's two pets, Santa's Little Helper
and Snowball II..." (densest pets/family cluster). Already
worked pre-fix; now even tighter.
Performance: O(|positions|^2) inner ranking. Typical chunks have
4-10 content tokens and 1-3 match positions each, so 8-30 positions
and ≤900 inner iterations — trivial.
Bracket format change (in render_claim_lattice) ships alongside:
[E5: "..."] → before
[E5 | <title> | <chunk_root[:8]>: "..."] → after
Closes visual confusion where E# is a chunk pointer but readers
parsed it as a 1-indexed source rank from the sources list.
Tests: 523 passed.
Initial Ticket H commit (38d95dd) only plumbed warrant into
verify_claim_lattice (pointer mode). The JSON variant
verify_claim_lattice_json was missed — and the CLI's
--answer-mode claim_lattice (the JSON path) was producing
EVIDENCE-LINKED on relation-question records without warrant
firing.
This adds the same warrant block to verify_claim_lattice_json:
question + warrant_check_enabled params, identical
WARRANT_MISSING violations, identical demote-to-HYBRID
semantics, same warrant_missing_claim_idxs field on the verdict.
Plumbs question= and warrant_check_enabled= through call sites
in runner.py and query.py for the JSON path.
Live verified: warrant correctly checks the FULL cited span
(not the spotlight excerpt). On the Homer-Simpson "boss is Mr.
Burns" case, the chunk text DOES contain "Mr. Burns" elsewhere
in the paragraph — warrant rightly passes. The misleading
spotlight excerpt that fox flagged is a separate render-layer
issue (failure to spotlight the load-bearing slice), not a
warrant failure.
Note: cache-hit path does not currently restore violations into
the result dict (providence_cache schema has no violations
column; run_dag_blob stores only hashes). On cache hits the
warrant_tail in CLI render won't surface even when warrant
fired at write-time. Follow-up needs a schema migration to
persist violations alongside the cached record.
Closes the lazy-anchor failure class fox surfaced in feedback-3
(2026-05-01) on "who is homer simpson's boss?":
Claim: "Homer Simpson's boss is Mr. Burns."
Cited span: voice-actor / Castellaneta bio prose
Pre-warrant: pointer-linked, source-role allowed, coverage passes
Post-warrant: WARRANT_MISSING violation, audit_mode capped at HYBRID
Pointer verification != warrant verification. A pointer resolves to
a span; a warrant requires the span to actually contain the claim's
named answer entity.
Adds aborist/qa/warrant.py:
- is_relation_question(q) — regex shape detector for "who is X's Y?",
"who is supermans girlfriend?", "who founded Z?", "boss of X" etc.
- extract_answer_anchors(claim) — multi-word proper nouns ("Mr.
Burns", "Lois Lane") with possessive 's stripping; solo-cap
fallback that skips sentence-starter false positives
- warrant_check(claim, spans) — at-least-one-anchor lexical match;
vacuous-pass when claim has no proper-noun anchor
Wired into verify_claim_lattice via new params (question,
warrant_check_enabled). Per-claim WARRANT_MISSING violations cap
audit_mode at HYBRID using the same demote pattern as
lazy_anchor_demoted. CLI renders " · warrant missing" tail when
violations present so the failure mode is legible without scanning
the violations array.
Policy field claim_lattice_warrant_check_enabled added to
DEFAULT_POLICY / DEFAULT_QUERY_POLICY and the verifier-policy hash
field set so changes bump verifier_policy_hash.
Hard or soft? Per CLAUDE.md "Soft hash vs hard hash":
substring/lexical = hard. Output is binary (warrant present or not),
violations enter the proof path through violations[] alongside
SOURCE_ROLE_BLOCKED and CITATION_MISMATCH.
Deferred (follow-on commits):
- full RELATION_PATTERNS table with relation-noun aliases
- two-hop bridge detection ("X founded Y" + "Y is Z" -> "X is Z")
- retrieval expansion for relation questions
Tests: 16 new unit tests in tests/test_warrant.py covering
is_relation_question, extract_answer_anchors, warrant_check
(happy path, lazy-anchor case, vacuous-pass, case-insensitive,
empty inputs). Full suite: 523 passed, 21 skipped.
Implements 6 of fox's feedback-2/3 items (2026-05-01) targeting the
red-fish-blue-fish output. Each lives in the render or sidecar layer
— no schema or cache-key change beyond the verifier_policy_hash that
landed earlier in the day.
(1) Honest audit-label rendering. Schema audit_mode (STRICT/HYBRID/
UNGROUNDED) describes the lexical verifier's call; the user-
facing label spells out what was actually checked. Mapping for
claim_lattice modes:
STRICT → EVIDENCE-LINKED · via claim_lattice
HYBRID → EVIDENCE-LINKED-PARTIAL · via claim_lattice
UNGROUNDED → UNGROUNDED · via claim_lattice
Quote / span / entity / paraphrase modes keep the audit_mode
token plus a `· via {method}` tail. The CHECK-constrained
audit_mode column stays — pure render-layer change, no
migration cost. Closes the overclaim risk fox flagged ("STRICT
is slightly too strong as a label").
(2) Sentence-bounded evidence spans. _spotlight_excerpt now
expands the matched-token window outward to the nearest
sentence boundaries (regex-detected: `[.!?][\"')\\]]?\\s+(?=[A-Z])`)
instead of cutting at fixed byte offsets. Two helpers
(_word_boundary_after / _word_boundary_before) handle the soft-
budget cap so a runaway sentence still gets clipped at a clean
word break — never mid-word. Pre-fix the renderer produced
excerpts like `"...freewheeling plot... friends and p..."` (the
`p...` is the offending mid-word truncation fox surfaced); now
the excerpt is `"One Fish Two Fish Red Fish Blue Fish is a
1960 children's book by Dr. Seuss. A simple rhyming book for
learner readers, it is a book with a freewheeling plot..."`.
(3) Source-role display + used/unused annotation. The CLI source
list went from
[1] One Fish Two Fish Red Fish Blue Fish — wikipedia.org/...
[2] Red Dwarf — wikipedia.org/...
to
[1] One Fish Two Fish Red Fish Blue Fish — primary_answer_source — used (E1) — ...
[2] Red Dwarf — background_source — unused — ...
Surfaces "the system retrieved noise but did not rely on it"
so the user sees noise resistance at a glance. Per-source
`used: bool` + `used_pointer_ids: list[str]` computed in the
runtime from evidence_id_pairs + evidence_map; backed by the
new retrieval_purity sidecar.
(4) Retrieval-purity sidecar. Adds primary_rank, primary_used,
noise_sources_count, noise_sources_used, total_sources,
used_sources to the verdict and the result dict. Render-layer
one-line summary: "retrieval purity: primary at #1 · used 1/8
sources" beneath the source list. Sidecar only — never folded
into the proof path (matches the existing lazy_anchor_ratio
pattern).
(5) Ordered-token title scoring. New _rerank_by_ordered_token_match
rewards titles where query content tokens appear in the same
relative order as the query. "red fish blue fish" query →
"One Fish Two Fish Red Fish Blue Fish" gets ordered_match=4
boost (1+0.5*3 = 2.5×); "Red Dwarf" stays at ordered_match=1
(no boost). LCS-based, stem-aware, deterministic. Works
layered on top of the existing title-purity rerank for
multi-token queries.
(6) Noise-resistance fixture (tests/test_qa_quality_live.py).
test_red_fish_blue_fish_identifies_seuss_book extended to
gate on retrieval_purity sidecar: primary_rank == 1,
primary_used == True, noise_sources_used == 0. Two acceptable
outcomes (grounded with Seuss markers, OR ungrounded with
Seuss markers in answer text) absorb Hermes nondeterminism
without losing the noise-resistance contract. Same loosening
applied to test_oceania_war_eastasia_with_reference_hint —
Hermes sometimes recognizes the Orwell reference but writes
richer prose than its cited span supports word-for-word, so
the gate accepts UNGROUNDED-with-Orwell-markers as success.
Live test_qa_quality_live.py: 20/20 fixtures pass. 507 unit tests
green. Working tree leaves bench/qa_questions.txt for fox's
parallel WIP.
Fox-supplied review expanding §2-§4 with axiomatic framing, CTI /
Merkle-AGI / PROMETHEUS-Σ interpretations, a concrete implementation
diff, and a strengthened test list. Captured verbatim as Appendix A.
Key refinements over the §2-§4 sketch:
- Splits retrieval-stage hash into retrieval_plan_hash +
retrieval_result_hash + retrieval_stage_hash (plan vs result
separation, axiomatically cleaner than embedding plan in the
retrieval-stage payload directly).
- Adds two new audit events: retrieval_plan_built and
retrieval_result_selected. Surfaces the retrieval inputs in the
audit chain (not just the run_dag), so cache-hits also commit
the plan via cache_hit_with_retrieval_plan.
- Distinguishes three keyword-handling cases for cache identity
(Case A/B/C) and lays out a clean boundary rule: keywords stay
in retrieval_plan unless they reach the model (then prompt_hash)
or affect routing policy (then retrieval_policy_hash).
- Promotes 'the map must be committed, not only the territory
reached' as a core principle. Operator-supplied maps (keywords,
routing modes, ranking policies) become first-class clauses
upstream of retrieval_result.
- Lists 12 concrete tests including the critical 'same sources,
different keywords → different run_dag_root' assertion that
proves the gap is closed.
Header note: Appendix A is authoritative when it disagrees with
§2-§4 above. The original §2-§4 sketch stays in place for
chronological context but is superseded by the appendix.
Six items from fox's de-novo synthesis (2026-05-01) landing as one
atomic commit. Each item references its motivation and respects the
v9.8 honesty boundary (no claims of semantic truth, no proof-path
contamination by soft signals).
NEW DOCS
--------
docs/cti-architecture.md
Maps today's modules onto the architectural layer cake fox named
in his synthesis: PROMETHEUS-Σ (controller) / CTI (claim-lattice
reasoning IR) / Merkle-AGI-DAG (commitment) / Reverse-RAG
(evidence direction) / v9.8 Providence (admissibility ledger) /
Hermes (weak proposer). Per-layer responsibility table + module
map showing which existing files implement which layer. The
architecture is real; the labels above name what's already there.
docs/naming-deferral.md
Argues why we keep internal terms (claim_lattice,
claim_lattice_pointer, verify_claim_lattice) instead of renaming
to architectural labels (CTI, PROMETHEUS-Σ). The rename costs
cache invalidation, ~150 test fixture references, schema CHECK
migration, blame-history disconnect, mesh peer coordination.
The bridge is the cti-architecture mapping doc — read it once,
then read code in code's vocabulary and prose in prose's
vocabulary. Lists four triggers that would invert the decision.
docs/self-reference-distillation-design.md
Successor to docs/self-reference-thought-chains-design.md (the
flat-source MVP). Maps STRICT claims onto the existing Distiller
/ Core / derivations infrastructure: each STRICT record becomes
a Fact-Core via a new ProvidenceDistiller, with per-cited-chunk
Merkle inclusion proofs back to Wikipedia source_roots. The
fact-graph grows; new claims attach by inheriting the proof
chain. CompositionDistiller (combining facts into new claims)
is explicitly deferred — that's reasoning machinery, not
infrastructure.
docs/test-coverage-audit-2026-05-01.md
Maps fox's §11 test list (16 items) against the codebase. 16/16
covered. Documents item #9's manual-quote-rule lifecycle: the
rule was retired from pointer mode in commit 224bfd6 after the
bench journey identified it was rejecting factually correct
claims; retained in JSON variant where the punctuation-rationale
argument doesn't apply. The audit doc itself is the requirements-
drift defense.
CODE
----
aborist/qa/keys.py:verifier_policy_hash
New pure function. Hashes the verifier-relevant subset of policy
(answer_mode + claim_lattice_* verifier knobs + entity_policy
fields + base_version). Folded into cache_key as an optional
9th dimension via a new keyword arg with backward-compat default
None — legacy 8-dim callers continue to work and produce the
pre-2026-05-01 hash result.
aborist/qa/{runner,query}.py
Compute verifier_policy_hash from the policy_variant and pass
to cache_key as the 9th dim. Records written from this commit
forward bind to the verifier-policy identity. Pre-existing 8-dim
records become cache-misses on next lookup — same migration cost
as any governance_policy_hash bump. The 9-dim form makes the
question "did the verifier rules change?" answerable from
cache_key diff alone, separate from "did the prompt change?"
CLAUDE.md
Adds "old maps vs runtime maps" engineering discipline to the
bench-maxing section. Codifies the principle: every base model
carries old maps from training time; the runtime carries the
fresh map; when they disagree, the runtime wins. Move authority
OUT of the model's prior and INTO runtime artifacts (pointer IDs
the runtime mints, source spans the runtime interpolates by
offset, evidence maps assembled per query, policy hashes that
fold prompt/verifier/retrieval into cache_key, hard checks run
by the verifier). Hermes' content-addressed-evidence-id
hallucination (commit bb8450d) is the canonical case study.
TESTS
-----
tests/test_keys.py
Six new tests for verifier_policy_hash:
- only hashes verifier subset (non-verifier fields don't change it)
- changes when verifier-relevant field changes
- empty-policy → stable
- 9-dim cache_key distinct from 8-dim
- 9-dim distinct under different verifier hashes
- 8-dim form preserved for legacy callers (None == omit)
DEFERRED (per fox + naming-deferral.md)
---------------------------------------
- PROMETHEUS-Σ as an extracted controller module — the dispatch
logic in runner.py + query.py already IS PROMETHEUS-Σ; an
explicit prometheus.py is a refactor with no behavior change.
Defer until a §5-rename-trigger fires.
507 tests pass (was 501 before, +6 from verifier_policy_hash
coverage).
Tickets are flat docs/ticket-NNNNNN-<slug>.md — actionable proposals
awaiting go/no-go, scoped defects, or queued enhancements.
Architecture references and bench journals stay un-numbered.
docs/TICKETS.md: index + convention + Next ID line so future shifts
know where the next number comes from. Renumbering protocol: every
new ticket commit bumps the Next ID line atomically with the new
ticket file. Status field tracks open / in progress / closed; closed
tickets keep the file in place as a design log.
docs/ticket-000001-retrieval-keywords-audit-gap.md: the ticket
opened in 73058b0, renamed and re-headered to fit the convention.
Per fox's '--retrieval-keywords' commit (2d6a86b), the keyword string
is nowhere in the audit chain — it influences cache_key only
indirectly via context_root + conversation_hash. Two runs with the
same question and different keywords that surface identical sources
are Merkle-indistinguishable; an audit replay can recover the chosen
documents but not the inputs that pulled them.
Doc-only ticket: design proposal for capturing keywords in the
run_dag retrieval stage and on the providence_cache row (Option A +
B), preserving the 8-dim cache_key invariant. Out of scope: folding
keywords into question_hash (Option C) — keywords are operator
metadata, not user intent.
Includes failure cases, three design options with trade-offs,
implementation sketch, scope boundaries, and a recommended cost/value
forecast (~1-2 hours, low risk, additive schema). Awaiting fox's
go/no-go before landing.
Three layers, 14 new tests, full suite 611 passed (was 597):
UNIT — tests/test_query.py
test_query_returns_prompt_chars_breakdown
Asserts the result dict's prompt_chars carries exactly the five
expected keys and messages_total equals sum of message contents
the StubClient saw.
test_query_answer_chars_matches_answer_text
answer_chars == len(answer_text) — drift check.
test_query_cache_hit_also_returns_capacity_metrics
Cache-hit path populates prompt_chars + answer_chars (operators
inspecting cached records still want the breakdown).
test_query_evidence_chars_grows_with_topk
Sanity: more sources / larger budget → more evidence chars
(the metric tracks actual context build, not a stale constant).
INTEGRATION — tests/test_query.py (retrieval_keywords)
test_retrieval_keywords_does_not_alter_question_to_llm
Keywords don't appear in the LLM-facing question segment;
system prompt unchanged across runs. Pins the substrate
contract: keywords are FTS5/title-filter-only.
test_retrieval_keywords_changes_retrieved_sources
Different keyword sets surface different docs (the actual
user-visible behavior).
UNIT — tests/test_cli_render.py
test_render_shows_capacity_line_when_prompt_chars_present
Capacity line appears with messages_total + breakdown when
prompt_chars is in the result dict.
test_render_omits_capacity_line_on_legacy_results_without_prompt_chars
Backwards-compat: legacy results render cleanly without the
capacity line — no KeyError, no '0 chars' noise.
test_render_capacity_thousand_separators
61,550 not 61550 — operator legibility on daily renders.
UNIT/INTEGRATION — tests/test_bench_qa_sweep.py (NEW FILE)
Imports bench/qa_sweep.py via importlib.util so the module's
not in the Python path doesn't matter. Five tests:
- _summarize counts verdicts by mode
- deflections counted only on STRICT/HYBRID rows (not UNGROUNDED)
- rendered markdown has the headline summary + size buckets
- size buckets correctly stratify strict-rate by prompt_chars_total
- empty buckets are skipped (no '0 runs' noise)
FUNCTIONAL — live verification (no automated test, manual)
`make query Q="what is the capital of france?" BURN=1` confirmed
in commit f927298 to render the capacity one-liner under the
source list. Documented in that commit's body.
Also corrected the docstring on query()'s `retrieval_keywords` to
reflect that keywords don't enter cache_key DIRECTLY but do change
context_root + conversation_hash via source selection — so the same
question with different keywords lands under different cache_keys
(legitimately, since the LLM saw different contexts).
Per-call accounting of prompt capacity so we can answer 'does
context size affect strict-rate?' at aggregate scale.
Per-call result dict gains:
"prompt_chars": {
"system_prompt": <int>, # system message chars
"grounding_reminder": <int>, # restated rule message (if any)
"user_question": <int>, # the original question text
"evidence_or_context":<int>, # rendered evidence map / wikitext context
"messages_total": <int>, # sum of all messages[].content
}
"answer_chars": <int> # rendered answer length
Char-level for now: model-agnostic, fast, and a workable proxy
for prompt tokens (~4 chars/token English prose, ~2.5 for JSON-
evidence-block heavy contexts). Token-exact accounting can layer
on later if a tokenizer is cheap enough to wire in per-call.
Surfaces in three places:
(1) result["prompt_chars"] — for programmatic callers. Cache hits
return prompt_chars too (computed before lookup).
(2) `aborist query` human render — one-liner showing the breakdown
and the answer length, so an operator can tell at a glance
whether STRICT came from a 5KB tight prompt or a 60KB stuffed
context.
(3) bench/qa_sweep.py — per-row prompt_chars_{total,evidence,system,
question} + answer_chars columns in the JSONL, plus a new
"strict-rate by prompt size" section in the markdown summary
bucketing runs by capacity (<8KB, 8-16KB, 16-32KB, 32-64KB,
>=64KB) so the bench reveals whether input size correlates
with verdict quality across modes.
Never enters cache_key — runtime measurements, not policy.
Empirically observed 2026-05-01: long discursive questions like
'what technology is currently or soon available which may enable
one person to reconstruct another person's thoughts...' under-
retrieve because their content tokens get diluted by template
phrasing. AND-mode FTS5 returns zero hits; OR-fallback ranks
unrelated articles by raw BM25.
Fox's discovery: appending domain keywords ('transcranial knowledge
acquisition') to the question lifts the verdict from HYBRID 6/10
to STRICT 1/1 by narrowing OR-mode retrieval to the topical
article (Neurotechnology). The keywords act as an operator hint
about what the question is really asking.
This commit exposes that pattern as a first-class flag without
polluting the question text:
--retrieval-keywords TEXT on `aborist query`
K="..." on `make query`
Plumbing: the keywords are concatenated with the question for
FTS5 search and title-filter token construction only. The LLM
still sees the original question; the verifier still checks against
the original question; cache_key still computes from the original
question. Keywords are session-only — successive calls with
different keywords on the same question can cache-hit each other.
Pair with BURN=1 for fresh inference per call.
Live verification:
make query Q="...thoughts...without speaking or sign language."
K="transcranial knowledge acquisition" BURN=1
-> STRICT 2/2 via claim_lattice, 37.5s, Neurotechnology article
cited for fMRI and DBS claims.
Without K: HYBRID 6/10 with Videoconferencing/Telepathy as top
sources — model hallucinated structure across irrelevant chunks.
Empirical finding 2026-05-01 on 'explain spin glass modeling, tensors?':
returned STRICT 1/1 with claim 'Spin glass modeling involves...
mathematical tools such as tensors' cited to a chunk from the
*Quantum chromodynamics* article. Token-coverage check inside the
chunk passed accidentally on shared physics vocabulary
(spin/model/etc.), but the article isn't about spin glasses — the
claim is a retrieval-driven hallucination.
Root retrieval cause: AND-mode FTS5 returned zero hits (Spin_glass
article has 0 occurrences of 'modeling' or 'tensors'); OR-mode
fallback ranked by raw BM25, so 'Glass' (glass*274) outranked
'Spin glass' (spin*29 + glass*15) despite Spin_glass matching 2
distinct query tokens vs Glass's 1. Verifier then accepted the
hallucinated claim because the cited QCD chunk has incidental
physics vocabulary overlap.
New ``diagnose_title_relevance(claim_text, cited_source_titles)``
sidecar: claim's content tokens (stemmed) must overlap the union
of cited source titles. Zero overlap → ``title_mismatch``. Catches
the spin-tensors case (claim about spin glass, cited title 'Quantum
chromodynamics' → zero stem overlap) without false-positives on
legitimate cross-document grounding (Lois_Lane article cited for a
Lois-Lane claim still has lois+lane overlap).
Sidecar-only for now — staying out of the binary verifier output
until bench evidence justifies hard-promotion. Not yet wired into
inspect_cache_key (per-claim citation source mapping isn't in
providence_cache yet) or the bench. Available as a utility for
operators auditing specific (claim, source_title) pairs.
5 new unit tests covering match/mismatch/no-titles/no-claim-tokens
and underscore-normalization for Wikipedia titles.
Closes the recursive-grounding gap fox surfaced today: aborist
tends Wikipedia trees but never grafts its own past Q&A records
into the forest. Each query starts from cold corpus retrieval;
prior providence_cache records sit unread until the same question
is re-asked (cache_key match). The "kindergarten thought chains"
framing names the gap — the system has a kindergarten of thoughts
(early STRICT records) that should mature into citable substrate
as they cool.
This commit lands the MVP — STRICT live providence records past
the kindergarten window become substrate via a new Source
subclass. Trust model per fox: "we trust strict statements as
fact unless a verifier falsifies it."
NEW
---
- docs/self-reference-thought-chains-design.md — full architecture
doc covering the four iteration-time gates, the recursive Merkle
proof story (Q2 → Q1 → Wikipedia chunk per v9.8's recursive-
cores insight), the falsification trust model (state=live as the
fail-closed gate), risks (lazy-anchor compounding, echo chambers,
storage bloat), and bench-impact estimate.
- aborist/sources/providence.py — ProvidenceSource(Source) with
four hard gates:
1. audit_mode == 'STRICT' (HYBRID/UNGROUNDED stay opaque)
2. falsification_state == 'live' (failed/stale/quarantined
excluded — verifier-falsification mechanism per fox)
3. now - created_at >= kindergarten_seconds (default 1h —
fresh thoughts cool first; kills tight echo loops)
4. anti-recursion: records whose answer text contains a
self-reference URI are skipped — first-generation only
- tests/test_providence_source.py — 10 unit tests covering each
gate plus the URI-scheme source-role classifier
- Makefile target `ingest-self-providence` (KG_SECONDS=3600
default; iterates each shard and self-promotes its STRICT live
records — cross-shard sharing happens via the existing
shards-dir UNION at retrieval time)
WIRE-UP
-------
- aborist/qa/query.py
- SOURCE_ROLE_BUDGET_WEIGHTS: self_reference_source = 1.0
(same as background — Wikipedia stays canonical primary;
self-reference is supplementary anchoring)
- SOURCE_ROLE_RANK_WEIGHTS: self_reference_source = 0.9
- _classify_source_role: short-circuits on aborist://providence/
URI prefix → self_reference_source regardless of title shape
- DEFAULT_QUERY_POLICY['claim_lattice_allowed_source_roles']
+= 'self_reference_source'
- aborist/qa/runner.py — same allowlist update for the
per-document `ask` path
- aborist/cli.py — `aborist ingest --source providence` reads the
providence_cache from the same shard it writes into;
--kindergarten-seconds flag plumbed through
NOT IN THIS COMMIT
------------------
- Aggregation of multiple Q&A records into synthesized summary
records (follow-on)
- Self-reference for HYBRID records (only STRICT is substrate
today; HYBRID could land later as a soft-anchor role with
lower trust)
- Live virtual sourcing (the design discusses it; MVP uses
snapshot ingestion so existing FTS / chunker / Merkle apply
with zero schema change)
- A live bench validating actual lift on self-reference questions
(requires running ingest-self-providence then bench; deferred
to follow-on commit on real data)
10 new unit tests pass; full suite at 482 passed / 21 skipped
(live fixtures gated).
Bench dig (2026-05-01, 65-question sweep): 4 of 7 deflecting
questions were false positives from verb-shape subject anchors.
what year did the berlin wall fall? → answer "1989" (anchor "fall")
when did the soviet union dissolve? → answer "1991" (anchor "dissolve")
how many wives did henry the eighth? → answer "six" (anchor "eighth")
what year does our cold fusion ...? → answer "1989" (anchor "happen")
why did the titanic sink? → "...sank in 1912" (anchor "sink" vs "sank")
why did the dinosaurs go extinct? → "asteroid impact" (anchor "extinct" vs "died")
These shapes ask for a year/count/cause; the answer is structurally
not expected to echo the question's subject. Pre-suppression, all
three modes landed deflection on these — the apparent
"pointer-only" deflection rate (10.3% vs 6.2%) was almost entirely
heuristic noise, not a real substrate failure.
New ``shape_suppressed`` flag returned by diagnose_deflection. When
the question starts with one of:
when, what year, what date, what time,
how many, how much, how long, how old, why
the subject-anchor check is skipped — overlap-ratio takes over.
Mars-BDFL (who-shape) still fires correctly.
Field exposed so the operator can filter the signal: a
shape_suppressed=True row that still classifies as deflection on
overlap-ratio means "answer was a bare year/count/cause with no
topic words" — known noisy, not real topic shift.
5 new unit tests; full suite 582 passed (was 578).
Catch-up after a productive afternoon. Three new conventions section
items + retrieval-pipeline addendum:
- Three answer modes documented: quote, claim_lattice_pointer (G0/CTI),
claim_lattice (JSON). Bench evidence (2026-04-30) on Hermes-3-8B:
quote 0.47 strict-rate, pointer 0.36, JSON 0.49. Both lattice modes
share verifier_method='claim_lattice' (CHECK constraint
compatibility); answer_mode + json_fixups disambiguate downstream.
- Claim-count ceiling (TOO_MANY_CLAIMS): default 12/answer, demote
STRICT→HYBRID without truncating. Bench finding context
(york-england runaway) preserved so future-me knows why the cap
exists.
- Deflection sidecar (qa.inspect.diagnose_deflection): subject-anchor
heuristic catches Mars-BDFL pattern. Sidecar-only — never feeds
back into providence. Wired into bench/qa_sweep.py as a per-row
signal + summary-table column.
- Retrieval pipeline stage 9: template-phrase stopwords. The
'all there know' additions closing york-england, with the AND-mode
FTS5 mechanism documented so a future change to MATCH semantics
doesn't accidentally re-open the miss.
Two paired changes:
(1) Retrieval template-phrase stopwords. Bench finding 2026-04-30:
'tell me all there is to know about york england?' missed the
en.wikipedia.org/wiki/York article entirely — corpus has it but the
FTS5 AND-mode MATCH expanded to "all" AND "there" AND "know" AND
"york" AND "england", which filters to only docs containing every
template word. Articles like "Charles I of England" happen to mention
"all" + "there" + "know" + "england" enough to win; the bare "York"
stub doesn't, so it falls out of the top-K.
Added `all there know everything anything something` to both
_FTS5_STOPWORDS (search/fts5.py) and _TITLE_STOPWORDS (qa/query.py)
so they stay in sync. After the change "York" article ranks #1 on
both 'tell me all there is to know about' and 'tell me everything
about' phrasings.
These are retrieval-time stopwords only — they don't enter the
cache_key or governance_policy_hash (cache_key hashes the original
question text via question_hash). Existing cached records remain
valid; new queries get better retrieval.
(2) bench/qa_sweep.py: per-row deflection_kind + subject_anchor +
subject_in_answer (from qa.inspect.diagnose_deflection); summary
table grows a 'deflections' column tracking STRICT/HYBRID rows
where the question's subject anchor is missing from the answer.
Surfaces the Mars-BDFL pattern at aggregate scale.
New docs/verifier-semantic-gap-design.md captures the deep
roadmap item from docs/qa-modes-bench-2026-04-30.md ("Verifier
semantic check (soft signal) — did the claim's predicate match
the cited span's frame?").
Problem: the claim-lattice verifier's lexical coverage check
(≥30% claim-token overlap with cited span) passes any (claim,
span) pair that shares enough surface tokens, even when the
span never asserts the claim's predicate. Three concrete cases
captured from live bench data:
- Great Wall elevation: STRICT 1/1 cited to a chunk that
discusses "Outer China beyond the Great Wall" but contains
no elevation information at all.
- JP-dinos Triceratops/Operation-Genesis: model cites the
2003 video-game article for a claim about the 1993 film.
- Boltzmann constant value: model cites a chunk that names
the constant and its unit but never states the numerical
value.
Three candidate designs evaluated against §1.1 cases + the
hard architectural constraints (proof-path purity, determinism,
no external endpoints beyond Hermes, latency budget):
3a NLI cross-encoder sidecar (recommended)
3b TF-IDF predicate matching (cheap but high false-negative)
3c Per-claim re-prompt to Hermes (latency + self-eval bias)
Recommendation: 3a with cross-encoder/nli-MiniLM2-L6-H768
(~80M params, ~100MB weights, CPU-runnable, deterministic at
fp32). Default off, opt-in via `claim_lattice_semantic_check`
policy field. Demote-only sidecar — moves STRICT → HYBRID
when claim entailment fails, never invents grounding. Soft
signal stays out of the proof path, mirrors the existing
`lazy_anchor_demoted` pattern.
Estimated impact: ~5-8 STRICT demotes per 66-run bench
(false-positives removed from the ledger), +0.4s typical
latency. Six open questions for fox at end of doc.
Doc-only commit. No code changes. Implementation deferred.
Empirical finding 2026-04-30 on 'who is a benevolent dictator for life
for mars?': JSON mode returned STRICT with answer 'Guido van Rossum is
a Benevolent Dictator For Life (BDFL) for the Python programming
language.' The verifier did its job (claims grounded), but the answer
never mentioned Mars — pure topic-shift. A real failure mode for
adversarial-premise questions where the model deflects rather than
refuses.
Detection rule: subject-anchor heuristic. The LAST content token in the
question (after stopword strip) is treated as the question's primary
subject ('mars' for BDFL, 'beatles' for 'who are the members of the
beatles?'). If the subject anchor is missing from the answer, classify
as 'deflection' regardless of generic-vocabulary overlap. The Mars-BDFL
case has 3/4 generic overlap (benevolent, dictator, life) but missing
subject — exactly the pattern overlap-ratio alone can't catch.
Returns dict with kind ∈ {deflection, partial_overlap, on_topic,
no_question_tokens} plus subject_anchor + subject_in_answer for
operator legibility.
Sidecar-only — wired into inspect_cache_key() but never feeds back
into providence_cache or audit_events. Per the verifier-stays-binary
discipline (CLAUDE.md): soft signals live in inspect verbs, never in
the hard chain.
Bench finding (2026-04-30, n=3 across 24 questions): JSON mode hit
S:3 H:0 U:0 on three questions where pointer mode landed S:0 H:3 U:0
or quote mode landed S:0 H:0 U:3 — clean modal divergence. Pinning
these as live fixtures gives us regression detection: a JSON-mode
quality drop on these questions reverts the STRICT signal first.
test_mona_lisa_strict_and_names_leonardo — was _grounded, tightened
to STRICT (bench: quote 0S, pointer 0S, JSON 3S)
test_supermans_girlfriend_strict_and_names_lois_lane (NEW)
— relationship-shape, JSON's strong suit (quote 0S, pointer 0S,
JSON 3S)
test_cold_fusion_breakthrough_year_strict_and_names_1989 (NEW)
— out-of-corpus rephrasing, atomic-claim rule unlocked grounding
(quote 0S 3U, pointer 3S, JSON 3S)
Per fox's "scoreboard + gates" docstring: bench gives aggregate
counts; these fixtures fail by name when the scoreboard moves.
Defense-in-depth against the "tell me all there is to know about X"
runaway shape. Bench finding (2026-04-30 york-england, n=3): JSON mode
emitted 26-59 claim-pointer pairs of which only 2-4 verified — model
treats encyclopedic prompts as a license to spam claims from training.
The atomic-claim prompt rule (b5925c8) cut typical runs to ~10 claims,
but a structural ceiling makes the failure shape operator-visible
regardless of per-claim verification.
Cap: 12. Admits typical entity-list questions (5-7 dinosaurs, simpsons
+ pets) while flagging the runaway. Plumbed through both verifiers
(pointer + JSON) via claim_lattice_max_claims_per_answer policy knob;
folds into governance_policy_hash on change.
Behavior: cap doesn't truncate — every claim still verifies so the
operator sees full evidence of the runaway. The TOO_MANY_CLAIMS
violation demotes STRICT → HYBRID via the existing violation-check
path; UNGROUNDED if no claims verified.
Tests: test_verify_json_too_many_claims_demotes_to_hybrid (13 claims
all verifying → HYBRID), test_verify_json_at_cap_can_still_strict
(12 claims at the cap → STRICT, no violation), and pointer-mode
mirror test_too_many_claims_demotes_pointer_mode_to_hybrid.
Prompt audit + rewrite per the negation-avoidance principle codified
in CLAUDE.md: small instruction-tuned models (Hermes-3-8B observed)
struggle with "do NOT X" / "never Y" / "without Z" — the negation
often pattern-matches away or inverts under attention. Rewrote 18
negation instances across 8 prompt fields in
DEFAULT_POLICY (runner.py) and DEFAULT_QUERY_POLICY (query.py) to
their positive forms. Examples:
"Do NOT invent IDs" → "Use only the IDs that appear in the
EVIDENCE blocks above"
"Never begin a line with E1:" → "Each line begins with the
claim text, then a bracket tag"
"No commentary, no preamble" → "Your output begins with `{`
and ends with `}`"
"If no evidence supports a claim, omit the claim" →
"Emit only claims that an EVIDENCE block directly supports"
Also added an atomic-claim rule (Rule 6 JSON, Rule 9 pointer):
"Each claim is one focused fact citing the source that directly
contains it. Multiple distinct facts become multiple claim
entries… Three short claims, each citing one pointer, beat one
long claim citing five." Targets the StarCraft-style mega-claim
where the model packs every fact into a single sentence with
five pointers.
Bench delta (n=3, 25 questions, 75 runs per mode):
quote 30S 28H 17U → 40S 23H 12U +10 STRICT, +13pp rate
pointer 27S 37H 11U → 23S 37H 15U -4 STRICT (acceptable —
JSON is now the daily
driver; pointer is
fallback/diagnostic)
JSON 36S 28H 11U → 44S 26H 5U +8 STRICT, +11pp rate,
-6 UNGROUNDED, -1.8s latency
Net across modes: +14 STRICT, +7 grounded, JSON latency
7.2s → 5.4s.
Bench questions: replaced "isn't it true that the great wall of
china is visible from space?" with "which side is the ground
elevation highest throughout the span of the great wall of
china, the north or south?" — a forensic-shape question with
answer "north". Live finding: model hits STRICT 1/1 with the
right answer cited to a Great-Wall-but-not-elevation chunk
(coverage threshold passes via topical-token overlap). New
fixture test_great_wall_elevation_north_side_higher gates on
entity-presence ("north" in answer) but tolerates UNGROUNDED —
documents the residual lazy-anchor failure mode for the
verifier-semantic roadmap.
CLAUDE.md: added negation-avoidance to the bench-maxing
discipline section.
Bench surfaced a real test bug: the prior assertion treated STRICT as
'model asserted a Mars BDFL,' but STRICT only means 'claims are
grounded.' On 'who is a benevolent dictator for life for mars?',
claim_lattice mode deflected to Guido van Rossum / Python (no Mars
mention) and the verifier returned STRICT correctly — claims grounded,
false premise sidestepped.
Three accept paths now:
1. UNGROUNDED (refused outright)
2. STRICT/HYBRID + mentions 'mars' + includes refutation marker
3. STRICT/HYBRID + does not mention 'mars' (deflection — answered
an adjacent grounded question instead of the false-premise one)
Fail case is unchanged: answer mentions Mars AND affirmatively names
a Mars BDFL with no refutation language.
Refutation markers expanded with 'no recognized', 'no known',
'fictional' to cover sci-fi-framing answers (Barsoom / Mars trilogy
characters as fictional dictators is also non-hallucinatory).
