New `docs/concept-relations-design.md`: architecture reference for
the per-shard concept_relations layer that replaced the legacy
frozenset module (commit 5fd458a). Covers:
- Why phase 1 (hand-curated frozensets) didn't scale.
- Append-only schema + the three by-construction properties (idempotent
re-derivation via UNIQUE, per-shard storage, Merkle-orthogonal).
- Built-in `link_reciprocity_synonym` extractor reading the existing
`edges` table — no new crawler, works for Wikipedia AND HTML sites.
- Measured storage: 95.58 MB across 4 wiki shards (3.47M docs,
10.75M resolved edges, 55,148 reciprocal pairs, 289,848 synonyms),
4m16s wall-clock backfill. 1.6% tax on the 6 GB corpus.
- Three storage compactions considered & rejected, each with the
specific trade-off it loses on (drop idx_concept_evid → painful
purge debugging; BLOB source_root → schema inconsistency; FK
normalization → JOIN in retrieval hot path).
- How-to: backfill, manual add, purge.
- Adding new extractors.
- Deferred follow-ons (CLI commands, Wikipedia See-also extractor,
category extractor, hatnote extractor).
CLAUDE.md item 5 in the retrieval-pipeline list updated to point at
the new module path (aborist/concepts/) and the design doc.
TICKETS.md reference list updated to mention the new design doc.
The phrase-pattern retrieval route (commit 1b8677d) closed the
RETRIEVAL side of reference-frame failure; this ticket closes
the ANSWER side.
New module aborist/qa/frame.py:
FrameDetection dataclass (frame_kind, reference_title,
reference_uri, confidence). Sidecar — never enters cache_key
or governance_policy_hash.
detect_frame(question, sources, phrase_match_roots) — heuristic
detector. Reference-frame classification fires when:
(a) phrase route surfaced at least one source, AND
(b) at least one phrase-matched source is a reference work,
determined by:
- title parenthetical disambig (`(novel)` / `(film)` /
`(play)` / `(franchise)` / etc.), OR
- body sample contains ≥3 DISTINCT fiction markers
(novel / published / protagonist / plot / ...).
Distinct-marker count keeps the heuristic robust against a
history article saying "novel approach" twice.
aborist/qa/query.py:
Calls detect_frame for lattice modes only. Body sample uses the
ARTICLE LEAD (chunk_idx=0, post-wikitext-strip) — fiction
markers cluster in the lead on Wikipedia, not in plot chunks
that may have been query-relevance-ranked higher.
New policy field claim_lattice_polarity_preamble injected as a
user-role message before the grounding_reminder when
frame_kind == "reference". Format-string with
{reference_title} placeholder.
Result dict carries frame_detection (kind / title / uri /
confidence) for renderer + bench consumption.
aborist/cli.py:
Renderer adds a `reference frame: <title>` line when
frame_detection.kind == "reference". Skipped for literal /
no-phrase-route / ambiguous rows.
Live verification — Orwell case:
PRE : "The text does not directly state that Oceania has always
been at war with East Asia."
POST : "In George Orwell's dystopian novel Nineteen Eighty-Four,
the nation of Oceania is always at war with Eastasia,
but this is a result of propaganda and doublethink, not
actual historical continuity. The war with Eastasia is
a fabricated conflict to maintain control..."
Multi-frame compilation: distinguishes propaganda claim from
fictional-actual continuity, exactly the polarity contract from
the ticket §2 abstraction.
Literal queries (capital of france) unchanged — polarity preamble
only injects when frame_kind classifies as "reference".
22 new tests (19 in test_frame.py + 3 in test_cli_render.py).
Full suite: 734 passed (was 712, +22).
Directive D3 status: ½ → ✓. Ticket #000002 closed.
All seven structural directives now ✓:
D1, D2, D5 (were ✓);
D3 #000002, D4 #000001, D6 #000003, D7 #000005 (closed in this
series).
New module aborist/qa/retrieval_plan.py:
RetrievalPlan dataclass — frozen, captures the operator-
influenceable retrieval inputs that determine source
selection (retrieval_keywords, top_k, over_fetch,
max_context_chars, shard_ids).
retrieval_plan_hash() — SHA-256 over canonical-JSON.
Deterministic per call; folds into the run-DAG retrieval
stage as a bound input alongside the existing
sources_summary output.
build_run_dag (aborist/qa/dag.py) accepts retrieval_plan_hash:
When provided, the retrieval stage hash binds BOTH plan
(input) and result (output): H({"retrieval_plan_hash":...,
"sources_summary_hash":...}).
When omitted (e.g. legacy / quote-mode callers that haven't
plumbed it yet), the retrieval stage falls back to the
historical sources-summary-only hash so pre-#000001
run_dag_root values stay stable.
query.py constructs the plan per call and passes it through.
Question text intentionally NOT in the plan — already covered
by question_hash. Shard ids included so audit can reproduce
which shards the search ran against.
Two runs with identical sources but different retrieval keywords
now produce different run_dag_root values — the provenance gap
on operator hints (auditor recovers "these were the keywords
that pulled in those sources") closes for the run-DAG path.
Marker test in tests/test_directives.py flipped from "absent"
assertion to "present": test_d4_retrieval_plan_binding_landed.
Plus test_d4_retrieval_plan_hash_module_exists pinning the
module shape. Full suite: 712 passed.
Deferred per ticket §6:
- audit events (retrieval_plan_built / retrieval_result_selected)
- providence_cache.retrieval_keywords SQL column
- optional strict cache_key mode (retrieval_plan_affects_cache_key)
These are ergonomic affordances atop the run-DAG binding; earn
their own tickets when bench evidence shows the blob path is
too friction-heavy for real workflows.
Directive D4 status: ½ → ✓. Ticket #000001 closed.
Replaces the two-rung EVIDENCE-LINKED / EVIDENCE-LINKED-PARTIAL
display label for claim-lattice methods with a four-rung ladder
that names a strictly stronger property at each rung:
POINTER-LINKED pointer/source/chunk verified;
warrant either didn't apply or failed
ANCHOR-WARRANTED pointer-linked + warrant passed where it ran;
other soft demotes may apply
EVIDENCE-WARRANTED anchor-warranted + no soft demotes
UNGROUNDED no verified pairs
HYBRID gets a -PARTIAL suffix on whichever rung applies.
Implementation: _render_audit_label gains a violations parameter
(defaults to None for backward-compat). _ladder_rung_for_lattice
discriminates rungs from the existing violations list:
- WARRANT_MISSING in violations → POINTER-LINKED
- any of {LAZY_ANCHOR_DEMOTED, POINTER_OVERFLOW_TRIMMED,
TOO_MANY_CLAIMS, BARE_NAME_CLAIM} → ANCHOR-WARRANTED
- else → EVIDENCE-WARRANTED
Design simplification vs the ticket's §3 sketch: the proposed
verifier_steps_ran field on the verdict dict was NOT needed. The
existing violations list carries enough signal to discriminate
all rungs. Per the five-step algorithm step 2: don't add fields
you don't need.
Quote / span / entity / paraphrase methods stay unchanged (their
STRICT verifies pinned spans, not synthesis).
Schema column audit_mode enum stays {STRICT, HYBRID, UNGROUNDED}
— pure renderer transformation, no governance_policy_hash bump,
no cache invalidation, no mesh-wire-format change. Existing
providence records render under the new ladder on next read.
5 new renderer tests in tests/test_cli_render.py covering each
rung mapping. D7 anti-regression test in tests/test_directives.py
updated to gate on the ladder labels. Bench helper docstring
follows. Full suite: 711 passed.
Directive D7 stays at ✓; ticket #000005 closed.
Three new question-shape classes dispatched through warrant_check
alongside the existing relation + date anchors:
(1) Entity-list shape — `name X`, `list X`, `who are the members
of X`. List-aware extractor `extract_entity_list_anchors`
(multi-word phrases ∪ solo-cap individual names) so comma-
separated entities each contribute. ANY-match semantics:
demote-don't-reject when an extra entity from training-prior
appears alongside grounded ones.
(2) Count shape — `how many X`, `how much X`. Digit ↔ word
equivalence (claim says "six", span says "6", or vice versa)
with ordinal collapse (`sixth → 6`). Year-shaped digits
filter out (those belong to the existing date anchor class).
ALL-match semantics: every count token in the claim must
appear in some cited span as digit or word.
(3) Why-cause shape — `why X`. Cause-anchor pool widens to
≥5-char lowercase common nouns (post a generic stopword set
that filters quantifier-adjective fillers like "various",
"factors", "situation") PLUS proper-noun anchors from the
existing extractor. Gated on why-shape only: lowercase
common-noun extraction has higher false-positive risk
elsewhere.
Per-class policy gate (proposed `claim_lattice_warrant_classes`
dict) deferred per the five-step algorithm step 2: single
`warrant_check_enabled: bool` is the minimum viable gate; per-
class flags earn their slot when bench evidence shows over-firing
on a specific class.
17 new warrant tests (detector + extractor + integration).
Marker test in test_directives.py flipped from "absent" to
"present" assertion: test_d6_warrant_generalization_landed.
Full suite: 709 passed (was 692, +17).
Directive D6 status flipped to ✓ in seven-point-program.md.
Ticket #000003 closed.
#000004 — Directive coverage in bench summary. Filed as closed
(landed in commit acd1f9c) for design-log traceability. Captures
the v1 substrate: per-row directive_compliance helper, _summarize
aggregation, markdown directive-coverage section, 5 unit tests.
#000005 — Label ladder migration (POINTER-LINKED / ANCHOR-WARRANTED
/ EVIDENCE-WARRANTED / ENTAILMENT-VERIFIED). Status open. Migration
from today's two-rung EVIDENCE-LINKED to a four-rung ladder where
each rung names a strictly stronger property. Schema column stays
{STRICT, HYBRID, UNGROUNDED} so v9.8 cache_key invariants hold;
renderer-level mapping only. Reserves ENTAILMENT-VERIFIED for a
future committed entailment engine without forcing premature
implementation.
TICKETS.md index gains both rows. Next ID bumps to 000006.
seven-point-program.md status snapshot updated:
- D7 row links #000005 (ladder migration proposal).
- D8 row links #000004 (closed substrate).
- New "Anti-regression test layer" section maps each directive
to its structural pin in tests/test_directives.py.
Note on fox's roadmap item #000006 (Internal CTI Expansion / frame
lattice / multi-frame answers): scope overlaps with #000002
(Reference-Frame Polarity Contract / Module L). NOT opening as
a separate ticket — Module L already covers it.
Today's warrant-lite (aborist/qa/warrant.py) covers two anchor
classes: proper-noun (relation-shape questions) and date (any
claim with a 4-digit year). Three failure shapes pass the existing
seven hard checks unchecked:
- entity-list: 'name the simpsons family + pets' — claim
enumerates entities, but the warrant doesn't demand all named
entities appear in some cited span.
- count: 'how many wives did henry the eighth have?' — claim
says 'six', but the warrant doesn't demand the count token
appear in some cited span.
- why-cause: 'why did the titanic sink?' — claim names a cause
noun (iceberg, asteroid, propaganda), but the warrant doesn't
demand the cause token appear in some cited span.
Ticket proposes per-shape anchor extractors dispatched by question-
shape detector (existing _question_is_relation_shape pattern).
Conservative scope on cause anchors: gate by why-shape only since
lowercase common-noun extraction has higher false-positive risk.
Folds into governance_policy_hash via per-class policy dict so an
operator can disable individual classes per-corpus when they
over-fire.
Out of scope per docs/naming-deferral.md: typed-contract framework
(per-question-type rule libraries). General anchor-class primitive
catches the failures listed without per-type rule books; typed
contracts earn slots only when bench evidence shows the general
primitive misses cases.
Forecast cost ~3-4 hours; risk medium (false-positive risk on
why-shape cause-noun extraction). Closes the lazy-anchor gap on
the question shapes the bench question set already exercises.
The 2026-05-01 distilled architectural directive becomes the
audit lens applied before every new ticket / feature / prompt
edit. Each of the seven directives gets:
- status (✓ / ½ / ✗)
- code anchors (verifier paths, renderer hooks)
- pinning tests (per-rule coverage in tests/)
- tickets covering remaining work
- bench signal (which row column reports it)
Status snapshot at landing:
D1 (no LLM in verifier) ✓
D2 (pointer clauses) ✓
D3 (CTI internally) ½ → ticket #000002
D4 (retrieval map AND evidence map) ½ → ticket #000001
D5 (deterministic pointer verify) ✓
D6 (anchor-class warrant) ½ → ticket #000003 (this commit pair)
D7 (honest labels) ✓
D8 (test-pinning before automation) discipline
CLAUDE.md docs index gains a "North-star" section pointing at the
program; ticket entries gain a "Directive" column threading them
back to the program.
Walked the five-step algorithm on CLAUDE.md itself.
Step 1 — requirements check: every section earned its place via a
specific operator-failure context. Sections that had grown into
prose essays got compressed back to load-bearing rules + pointers.
Step 2 — delete:
- Bench-maxing prose section (~100 lines) → moved verbatim to
docs/bench-maxing.md. CLAUDE.md keeps a one-block headline list
pointing at the doc.
- Conventions section: each rule tightened to one paragraph max,
with a `See aborist/qa/verify.py` (or similar) trailing pointer
so the operator can jump straight to the rationale in code.
Verbose duplications between conventions and retrieval-pipeline
sections (e.g. wikitext base prose explained twice) collapsed.
- Retrieval pipeline: each of the 9 stages now references the
relevant module (`qa/concepts.py`, `query.py`) instead of
re-narrating the failure case in prose.
- Architecture tree: minor trim, removed redundant comments
where the filename already names the role.
Step 3 — simplify: source papers section dropped one item that was
duplicate (PDF + RST point to same content); kept the canonical
source.
Steps 4 + 5 — n/a (this is a doc, not a process).
New section: Docs index. Lists every architectural / design doc
in docs/ plus a ticket sub-list (open tickets with their files).
Closes the discoverability gap where TICKETS.md existed but agents
didn't know to look at it.
Net: CLAUDE.md goes from 308 → 311 lines BUT the avg information
density is up — bench prose doesn't bloat the entry-point doc, and
each convention now points to its source. The `docs/bench-maxing.md`
extraction is the real win: separation of "rules I must not break"
(CLAUDE.md) from "discipline I should internalize" (bench-maxing.md).
Two follow-ups to the phrase-pattern retrieval fix (commit 1b8677d)
covering items 6 and 10-11 of fox's 2026-05-01 architectural review:
(1) Non-regression tests for the phrase route:
- test_phrase_route_skipped_when_question_shorter_than_min_n
pins the structural false-positive guard: the n=5/n=6 minimum
means a 4-token literal-geography query lacks enough tokens to
trigger the route at all.
- test_phrase_route_does_not_hijack_literal_geography_query
end-to-end: a 4-token "oceania east asia geography" query on
a synthetic 2-doc corpus surfaces only the geography-stub doc;
the orwell-stub doc (whose body has the diagnostic 5-gram) is
correctly NOT pulled in by the phrase route on a literal query.
(2) docs/ticket-000002-reference-frame-polarity-contract.md
Captures fox's Module L proposal verbatim as Appendix A and
extracts the implementation sketch into the standard ticket
body (problem statement, abstraction, CTI interpretation, three
pieces of code to write, test list, scope boundaries).
The phrase route closed the RETRIEVAL side of reference-frame
failure. Module L addresses the ANSWER side: today's substrate
answers Orwell queries as "the text does not directly state..."
when it should produce multi-frame answers distinguishing
Party propaganda from fictional-actual continuity. Forecast
cost ~3-4 hours; risk medium (prompt augmentation interaction
with claim_lattice prompt).
Module M = ticket #000001 (route provenance binding); not
duplicated. Module N (FP guards) partially landed via the
tests above; remaining tests folded into ticket #000002's
test list. Module H (relation warrant lite) lacks scope
detail; deferred without a ticket.
(3) docs/TICKETS.md updated: index gains #000002 row, Next ID
bumped to 000003.
Empirical 2026-05-01: query 'has oceania always been at war with east
asia' surfaced literal-geography articles (Oceania, Asia, Far East)
because BM25 scored each token independently — the diagnostic signal
'oceania always been at war' is a verbatim 5-token sequence, not a
distinct content token. The Nineteen Eighty-Four article had zero
title-token overlap with the question, so even when reached via FTS5
phrase MATCH it would be filtered out before rerank.
Fix is two parts:
(1) New phrase route in `_search_corpus`. For each n-gram extracted
from the question (n=6 score 100, n=5 score 90), run an FTS5
quoted-phrase MATCH and add hits to the candidate pool. n=4 was
tried and rejected: 'always been at war' matches generic war-history
articles too noisily. 5+ tokens trade recall for precision; most
allusions ('may the force be with you', 'winter is coming',
'to be or not to be') survive at length 5 or higher.
(2) New accept-path 4 in `_filter_by_title_relevance`. Phrase-route
hits bypass the title-token-overlap gate via `phrase_match_roots`
(set of document_roots that matched a phrase). Without this, the
1984 article would be retrieved by phrase MATCH and immediately
filtered out because its title 'Nineteen Eighty-Four' shares no
content tokens with the question.
Latent-bug fix as a side effect: `_search_corpus` previously returned
a bare list, and the caller did `getattr(hits, "_core_match_roots",
set())` to fish out a sidecar set — but the sidecar was never
attached, so the `core_match_roots` accept-path in
_filter_by_title_relevance silently received an empty set for an
unknown duration. The function now returns a tuple
`(hits, core_match_roots, phrase_match_roots, root_to_shard)` so
both routes are correctly threaded.
Live verification: post-fix query lands EVIDENCE-LINKED 1/1 with
Nineteen Eighty-Four cited and the model recognizing the Orwell
frame ('the passage describes a change in alliances...'). No
operator augmentation needed.
Bench expansion: 6 allusion-shape questions added under a new
'# allusion / reference frame' category for prevalence tracking.
docs/reference-frame-failure-class.md: investigation log capturing
the diagnosis + why phrase-pattern boost beats a hand-rolled
'Reference Frame Router' (allusions are long-tail; per-pattern code
rots; the corpus already knows — fix retrieval not add a new stage).
9 new unit tests in test_query.py covering _question_phrases shape
(no stopword strip, all-short-token-skip, dedup), _search_phrases
defensive paths (empty input, double-quote-bearing input), end-to-
end phrase surfacing on a synthetic corpus, and the accept-path 4
filter behavior. Full suite 649 passed.
Fox-supplied review expanding §2-§4 with axiomatic framing, CTI /
Merkle-AGI / PROMETHEUS-Σ interpretations, a concrete implementation
diff, and a strengthened test list. Captured verbatim as Appendix A.
Key refinements over the §2-§4 sketch:
- Splits retrieval-stage hash into retrieval_plan_hash +
retrieval_result_hash + retrieval_stage_hash (plan vs result
separation, axiomatically cleaner than embedding plan in the
retrieval-stage payload directly).
- Adds two new audit events: retrieval_plan_built and
retrieval_result_selected. Surfaces the retrieval inputs in the
audit chain (not just the run_dag), so cache-hits also commit
the plan via cache_hit_with_retrieval_plan.
- Distinguishes three keyword-handling cases for cache identity
(Case A/B/C) and lays out a clean boundary rule: keywords stay
in retrieval_plan unless they reach the model (then prompt_hash)
or affect routing policy (then retrieval_policy_hash).
- Promotes 'the map must be committed, not only the territory
reached' as a core principle. Operator-supplied maps (keywords,
routing modes, ranking policies) become first-class clauses
upstream of retrieval_result.
- Lists 12 concrete tests including the critical 'same sources,
different keywords → different run_dag_root' assertion that
proves the gap is closed.
Header note: Appendix A is authoritative when it disagrees with
§2-§4 above. The original §2-§4 sketch stays in place for
chronological context but is superseded by the appendix.
Six items from fox's de-novo synthesis (2026-05-01) landing as one
atomic commit. Each item references its motivation and respects the
v9.8 honesty boundary (no claims of semantic truth, no proof-path
contamination by soft signals).
NEW DOCS
--------
docs/cti-architecture.md
Maps today's modules onto the architectural layer cake fox named
in his synthesis: PROMETHEUS-Σ (controller) / CTI (claim-lattice
reasoning IR) / Merkle-AGI-DAG (commitment) / Reverse-RAG
(evidence direction) / v9.8 Providence (admissibility ledger) /
Hermes (weak proposer). Per-layer responsibility table + module
map showing which existing files implement which layer. The
architecture is real; the labels above name what's already there.
docs/naming-deferral.md
Argues why we keep internal terms (claim_lattice,
claim_lattice_pointer, verify_claim_lattice) instead of renaming
to architectural labels (CTI, PROMETHEUS-Σ). The rename costs
cache invalidation, ~150 test fixture references, schema CHECK
migration, blame-history disconnect, mesh peer coordination.
The bridge is the cti-architecture mapping doc — read it once,
then read code in code's vocabulary and prose in prose's
vocabulary. Lists four triggers that would invert the decision.
docs/self-reference-distillation-design.md
Successor to docs/self-reference-thought-chains-design.md (the
flat-source MVP). Maps STRICT claims onto the existing Distiller
/ Core / derivations infrastructure: each STRICT record becomes
a Fact-Core via a new ProvidenceDistiller, with per-cited-chunk
Merkle inclusion proofs back to Wikipedia source_roots. The
fact-graph grows; new claims attach by inheriting the proof
chain. CompositionDistiller (combining facts into new claims)
is explicitly deferred — that's reasoning machinery, not
infrastructure.
docs/test-coverage-audit-2026-05-01.md
Maps fox's §11 test list (16 items) against the codebase. 16/16
covered. Documents item #9's manual-quote-rule lifecycle: the
rule was retired from pointer mode in commit 224bfd6 after the
bench journey identified it was rejecting factually correct
claims; retained in JSON variant where the punctuation-rationale
argument doesn't apply. The audit doc itself is the requirements-
drift defense.
CODE
----
aborist/qa/keys.py:verifier_policy_hash
New pure function. Hashes the verifier-relevant subset of policy
(answer_mode + claim_lattice_* verifier knobs + entity_policy
fields + base_version). Folded into cache_key as an optional
9th dimension via a new keyword arg with backward-compat default
None — legacy 8-dim callers continue to work and produce the
pre-2026-05-01 hash result.
aborist/qa/{runner,query}.py
Compute verifier_policy_hash from the policy_variant and pass
to cache_key as the 9th dim. Records written from this commit
forward bind to the verifier-policy identity. Pre-existing 8-dim
records become cache-misses on next lookup — same migration cost
as any governance_policy_hash bump. The 9-dim form makes the
question "did the verifier rules change?" answerable from
cache_key diff alone, separate from "did the prompt change?"
CLAUDE.md
Adds "old maps vs runtime maps" engineering discipline to the
bench-maxing section. Codifies the principle: every base model
carries old maps from training time; the runtime carries the
fresh map; when they disagree, the runtime wins. Move authority
OUT of the model's prior and INTO runtime artifacts (pointer IDs
the runtime mints, source spans the runtime interpolates by
offset, evidence maps assembled per query, policy hashes that
fold prompt/verifier/retrieval into cache_key, hard checks run
by the verifier). Hermes' content-addressed-evidence-id
hallucination (commit bb8450d) is the canonical case study.
TESTS
-----
tests/test_keys.py
Six new tests for verifier_policy_hash:
- only hashes verifier subset (non-verifier fields don't change it)
- changes when verifier-relevant field changes
- empty-policy → stable
- 9-dim cache_key distinct from 8-dim
- 9-dim distinct under different verifier hashes
- 8-dim form preserved for legacy callers (None == omit)
DEFERRED (per fox + naming-deferral.md)
---------------------------------------
- PROMETHEUS-Σ as an extracted controller module — the dispatch
logic in runner.py + query.py already IS PROMETHEUS-Σ; an
explicit prometheus.py is a refactor with no behavior change.
Defer until a §5-rename-trigger fires.
507 tests pass (was 501 before, +6 from verifier_policy_hash
coverage).
Tickets are flat docs/ticket-NNNNNN-<slug>.md — actionable proposals
awaiting go/no-go, scoped defects, or queued enhancements.
Architecture references and bench journals stay un-numbered.
docs/TICKETS.md: index + convention + Next ID line so future shifts
know where the next number comes from. Renumbering protocol: every
new ticket commit bumps the Next ID line atomically with the new
ticket file. Status field tracks open / in progress / closed; closed
tickets keep the file in place as a design log.
docs/ticket-000001-retrieval-keywords-audit-gap.md: the ticket
opened in 73058b0, renamed and re-headered to fit the convention.
Per fox's '--retrieval-keywords' commit (2d6a86b), the keyword string
is nowhere in the audit chain — it influences cache_key only
indirectly via context_root + conversation_hash. Two runs with the
same question and different keywords that surface identical sources
are Merkle-indistinguishable; an audit replay can recover the chosen
documents but not the inputs that pulled them.
Doc-only ticket: design proposal for capturing keywords in the
run_dag retrieval stage and on the providence_cache row (Option A +
B), preserving the 8-dim cache_key invariant. Out of scope: folding
keywords into question_hash (Option C) — keywords are operator
metadata, not user intent.
Includes failure cases, three design options with trade-offs,
implementation sketch, scope boundaries, and a recommended cost/value
forecast (~1-2 hours, low risk, additive schema). Awaiting fox's
go/no-go before landing.
Closes the recursive-grounding gap fox surfaced today: aborist
tends Wikipedia trees but never grafts its own past Q&A records
into the forest. Each query starts from cold corpus retrieval;
prior providence_cache records sit unread until the same question
is re-asked (cache_key match). The "kindergarten thought chains"
framing names the gap — the system has a kindergarten of thoughts
(early STRICT records) that should mature into citable substrate
as they cool.
This commit lands the MVP — STRICT live providence records past
the kindergarten window become substrate via a new Source
subclass. Trust model per fox: "we trust strict statements as
fact unless a verifier falsifies it."
NEW
---
- docs/self-reference-thought-chains-design.md — full architecture
doc covering the four iteration-time gates, the recursive Merkle
proof story (Q2 → Q1 → Wikipedia chunk per v9.8's recursive-
cores insight), the falsification trust model (state=live as the
fail-closed gate), risks (lazy-anchor compounding, echo chambers,
storage bloat), and bench-impact estimate.
- aborist/sources/providence.py — ProvidenceSource(Source) with
four hard gates:
1. audit_mode == 'STRICT' (HYBRID/UNGROUNDED stay opaque)
2. falsification_state == 'live' (failed/stale/quarantined
excluded — verifier-falsification mechanism per fox)
3. now - created_at >= kindergarten_seconds (default 1h —
fresh thoughts cool first; kills tight echo loops)
4. anti-recursion: records whose answer text contains a
self-reference URI are skipped — first-generation only
- tests/test_providence_source.py — 10 unit tests covering each
gate plus the URI-scheme source-role classifier
- Makefile target `ingest-self-providence` (KG_SECONDS=3600
default; iterates each shard and self-promotes its STRICT live
records — cross-shard sharing happens via the existing
shards-dir UNION at retrieval time)
WIRE-UP
-------
- aborist/qa/query.py
- SOURCE_ROLE_BUDGET_WEIGHTS: self_reference_source = 1.0
(same as background — Wikipedia stays canonical primary;
self-reference is supplementary anchoring)
- SOURCE_ROLE_RANK_WEIGHTS: self_reference_source = 0.9
- _classify_source_role: short-circuits on aborist://providence/
URI prefix → self_reference_source regardless of title shape
- DEFAULT_QUERY_POLICY['claim_lattice_allowed_source_roles']
+= 'self_reference_source'
- aborist/qa/runner.py — same allowlist update for the
per-document `ask` path
- aborist/cli.py — `aborist ingest --source providence` reads the
providence_cache from the same shard it writes into;
--kindergarten-seconds flag plumbed through
NOT IN THIS COMMIT
------------------
- Aggregation of multiple Q&A records into synthesized summary
records (follow-on)
- Self-reference for HYBRID records (only STRICT is substrate
today; HYBRID could land later as a soft-anchor role with
lower trust)
- Live virtual sourcing (the design discusses it; MVP uses
snapshot ingestion so existing FTS / chunker / Merkle apply
with zero schema change)
- A live bench validating actual lift on self-reference questions
(requires running ingest-self-providence then bench; deferred
to follow-on commit on real data)
10 new unit tests pass; full suite at 482 passed / 21 skipped
(live fixtures gated).
New docs/verifier-semantic-gap-design.md captures the deep
roadmap item from docs/qa-modes-bench-2026-04-30.md ("Verifier
semantic check (soft signal) — did the claim's predicate match
the cited span's frame?").
Problem: the claim-lattice verifier's lexical coverage check
(≥30% claim-token overlap with cited span) passes any (claim,
span) pair that shares enough surface tokens, even when the
span never asserts the claim's predicate. Three concrete cases
captured from live bench data:
- Great Wall elevation: STRICT 1/1 cited to a chunk that
discusses "Outer China beyond the Great Wall" but contains
no elevation information at all.
- JP-dinos Triceratops/Operation-Genesis: model cites the
2003 video-game article for a claim about the 1993 film.
- Boltzmann constant value: model cites a chunk that names
the constant and its unit but never states the numerical
value.
Three candidate designs evaluated against §1.1 cases + the
hard architectural constraints (proof-path purity, determinism,
no external endpoints beyond Hermes, latency budget):
3a NLI cross-encoder sidecar (recommended)
3b TF-IDF predicate matching (cheap but high false-negative)
3c Per-claim re-prompt to Hermes (latency + self-eval bias)
Recommendation: 3a with cross-encoder/nli-MiniLM2-L6-H768
(~80M params, ~100MB weights, CPU-runnable, deterministic at
fp32). Default off, opt-in via `claim_lattice_semantic_check`
policy field. Demote-only sidecar — moves STRICT → HYBRID
when claim entailment fails, never invents grounding. Soft
signal stays out of the proof path, mirrors the existing
`lazy_anchor_demoted` pattern.
Estimated impact: ~5-8 STRICT demotes per 66-run bench
(false-positives removed from the ledger), +0.4s typical
latency. Six open questions for fox at end of doc.
Doc-only commit. No code changes. Implementation deferred.
Adds two architectural-fix sections to the QA-modes bench doc:
1. Pointer-IDs in JSON mode (commit bb8450d) — closes the
content-addressed evidence_id hallucination loop where
Hermes-3-8B emitted near-miss IDs (E1b6e396 vs the runtime's
Eed1b6e396) on cross-document relationships, landing
UNGROUNDED on factually correct answers. Switching to short
pointer IDs (E1, E2, …) made fabrication obvious and the
Homer Simpson fixture went UNGROUNDED 0/1 → STRICT 1/1.
2. JSON-mode stop-sequence (commit f23d3a3) — guards against
post-brace token runaway where Hermes spammed whitespace
until max_tokens exhausted on broad-descriptive shapes
(apollo program 3/3 runaway in the post-pointer-ID bench).
stop=["\n\n"] cuts the runaway since well-formed JSON-mode
output never contains a blank line.
Also adds a bench progression table showing the journey from
the morning baseline (JSON 19 errors, 26 STRICT) to the post-
pointer-ID evening run (0 errors, 31 STRICT, 56 grounded).
Each row was a named-failure → fix → re-bench cycle.
Post-retry / post-trim-and-verify bench showed the picture flipped:
JSON mode now leads on strict-rate (50%) and ties grounded count
(54) with zero errors, all at parity-or-better latency. Switching
the `make query` ANSWER_MODE default from `claim_lattice_pointer`
to `claim_lattice` so the human-facing CLI uses the strongest mode.
Library-level DEFAULT_ANSWER_MODE stays "quote" so unit tests using
StubClient aren't disrupted.
Doc updated with post-retry bench table + revised conclusion.
Live test harness:
- new test_homer_simpson_boss_is_mr_burns fixture (pinned to
pointer mode — JSON mode hallucinates evidence_ids on
cross-document relationships, pointer mode's short numeric
tags can't be fabricated). Documents the mode trade-off.
- test_laura_croft marker net broadened to absorb Hermes
single-sample variance (added "magazine", "video game",
"character", "fictional", "british" markers) plus a hard
"croft" anchor check. Both real entities still gate cleanly.
11/11 live fixtures pass at the new JSON default + pointer pin
on Homer. 460 unit tests + 11 live = 471 green; 10 skipped is
just the live tests in the default-skip path.
Three-way QA-quality bench (quote / pointer / JSON) over an expanded
22-question set × 3 samples = 198 LLM calls. Findings landed in
docs/qa-modes-bench-2026-04-30.md with per-question breakdown and
roadmap. Aggregate at bench time:
quote 31S 20H 15U 0e strict-rate 47% 7.7s
claim_lattice_pointer 14S 34H 18U 0e strict-rate 21% 4.4s
claim_lattice (JSON) 26S 12H 9U 19e strict-rate 39%* 4.4s
The 19 JSON-mode "errors" turned out to be HTTP 502 from vLLM upstream,
not parse failures — clustered, all on the JSON-mode pass, plausibly
correlated with `guided_json` stressing the grammar engine.
Two improvements based on findings:
(1) OpenAICompatibleClient grew retry on transient 502/503/504 with
exponential backoff (0.5/1/2s, 3 attempts default). Network-layer
errors (ConnectError, ReadTimeout, RemoteProtocolError) get the
same retry. Smooths over the cluster without changing semantics:
persistent failures still raise, transient bursts no longer
dominate the error column. Helps all modes; JSON benefits most.
(2) Pointer-cap behavior changed from hard SCHEMA_INVALID to
trim-and-verify. When `[E2,...,E14]` over-cites a single claim,
keep first N pointers, verify normally, record
POINTER_OVERFLOW_TRIMMED in violations. STRICT becomes unreachable
(audit_mode caps at HYBRID) so the over-cite pattern stays
surfaced — but a correct claim like "Leonardo da Vinci painted
the Mona Lisa." no longer gets nuked for cosmetic over-citation.
Pre-fix: pointer mode hit 0/3 STRICT on Mona Lisa (mega-bracket
triggered SCHEMA_INVALID). Post-fix: HYBRID 2/14 with the right
answer rendered alongside both kept source spans. The dropped
pointers count toward n_quotes so the denominator surfaces the
over-cite to the auditor.
Bench scaffolding: ANSWER_MODES tuple now includes "claim_lattice"
(JSON), Makefile default sweeps all three. Question set expanded
from 8 to 22 covering narrow factoid, broad descriptive, entity
list, relationship, comparison, niche, adversarial, out-of-corpus.
460 tests pass. Connecticut output stays clean (HYBRID 4/7); JP-
dinosaurs pointer mode still UNGROUNDED via the bare-name guard
(model emits one-token entity names, the right floor catches them).
Step-by-step operator guide for standing up an aborist mesh between
two real hosts: pre-flight, init/enable on both peers, out-of-band
pubkey exchange, mutual enrollment, mesh serve, mesh sync, audit
chain integrity checks, eviction protocol, reset/teardown.
Calls out what's not yet wired (mesh pull, per-peer chain merge,
AEAD body encryption) so operators don't expect features that ship
later. Pairs with docs/mesh.md (protocol contract) and the e2e tests
in tests/test_mesh_wire_e2e.py.
Adds docs/mesh.md and five graphviz diagrams covering the federation
layer: identity stack, epoch state machine, per-member secret envelope,
gossip wire contract, and operator decision tree. Pins the protocol
contract for the upcoming HTTP wire (mesh sync, mesh serve).
Makefile gets a 'docs' target with pattern rule so PNG renders are
incremental from .dot sources.
