Five rule tightenings, each targeting a specific bench-qa STRICT
false-positive shape (the xfail regressions from the previous commit):
1. claim-lattice bracket-artifact skip — sentences matching
`\[E\d+\s*\|` (pointer markup) or `..."\]` (truncation tail)
are no longer parsed as natural-language assertions; `Such a
thesis was..."]` no longer fires vacuous.
2. Circular-rule differentia cap — circular now requires the
predicate to be (a) entirely vacuous OR (b) leads with a subject
token AND has ≤ 2 non-subject non-filler differentia tokens.
"Michael Jordan's Restaurant was a restaurant in Chicago,
Illinois, named after the basketball player Michael Jordan"
has 6 differentia → no longer fires. Pre-existing positive test
"The entity is the entity referring to the State of Israel"
has 2 differentia (state, israel) → still fires (under threshold).
3. phrase_component_reuse translation-chain exception — when the
predicate ALSO contains a quoted phrase (translation /
definition / etymology context), token reuse with the subject's
quoted phrase is legitimate, not circular. "The name 'Rosebud
River' is a translation … 'the river of the roses'" no longer
fires.
4. Vacuous-rule short-acronym escape — `_coherence_predicate_has_short_acronym_content`
recognizes title-cased element-symbols (Au, Fe, Pb…) and all-caps
2-5-char acronyms (DNA, FBI, USB, NASA…) as content even though
they're below the ≥3-char content-token filter. "The chemical
symbol for gold is Au." no longer fires; "Iron has the chemical
symbol Fe." also clean; tautology "DNA stands for DNA." still
correctly flagged circular.
5. (the 'term <X>' idiom xfail stays xfail — borderline, no clean
lexical fix.)
5 xfail → passing (the regression coverage is now executable proof
of fix); 1 xfail remains. Pooled bench-qa STRICT FP rate test ceiling
tightened from 7% to 2%. All 7 pre-existing positive coherence tests
still fire correctly. 110 total tests pass; 1 xfailed; no regressions
on doc-counts / nli / relevance.
§3.1 diagnose_coherence (now 19 tests, +10 from the parallel session's 9):
- 3 more positive shapes (multi-sentence vacuous, named-entity circular,
grammar-term phrase_component_reuse).
- 5 xfail regression tests for SHAPES THAT FALSE-FIRE on real bench-qa
STRICT data (44/808 = 5.4% FP rate measured on the pooled n=1+3+5
STRICT answers). Each xfail names the exact shape + why it should
ideally be 'ok' + which rule needs tightening:
* 'The chemical symbol for gold is Au.' → vacuous (short predicate)
* 'Michael Jordan's Restaurant was a restaurant ... named after
Michael Jordan.' → circular (named-after re-use)
* 'The Western X was the western half of the X' → circular
* 'The name <Phrase> is a translation ... of the <derivative>' →
phrase_component_reuse (translation/etymology)
* claim-lattice [E1 | … …"] tails → vacuous (truncated bracket
fragment)
* 'The term <X>' → phrase_component_reuse (idiomatic English)
- 1 load-bearing real-traffic test: FP rate on 808-cell pooled STRICT
must stay ≤ 7% (current 5.4%) — fires loud if a future change
regresses it. Skips on fresh-checkout (bench/qa_results/ gitignored).
§3.2 ShadowRelevance (now 20 tests, +7 from the round-1 scaffold):
- Manifest tests for round-2 primary (bge-reranker-large), the size
spectrum coverage (50-560MB), the candidate-bench findings block
(biggest-within-family / not-across-families / deeper-not-better /
capacity-floor).
- Pair-kind distinction (question_answer vs claim_source recorded
separately for downstream telemetry / governance hashing).
- Batch-order preservation (_score_batch must return scores in input
order — load-bearing for downstream zip-back).
- Empty-input handling (Q empty, D empty, whitespace-only).
- Zionist-entity discriminator sanity (on-topic > off-topic logit).
- demote_below_score-stays-null invariant (the §7 #18→#27 discipline:
no hardcoded threshold; must come from a real-traffic shadow sweep).
Total: 101 passed + 6 xfailed (5 §3.1 regressions documented + 1 from
parallel session). The 5 xfails are the bench-maxing receipts — they
document EXACTLY which shapes §3.1 false-fires on, with the rule that
needs tightening named in each reason.
Per-sentence shape check (no model) emitting kind ∈
{phrase_component_reuse, circular, vacuous, ok, empty}:
- circular: subject content-tokens ⊆ predicate's and the predicate
leads with a subject token ("Water is water").
- phrase_component_reuse: subject quotes a phrase, predicate reuses
one of that phrase's own tokens as a bare "the/a/an <token>"
referent — the 2026-05-12 field case ("the phrase 'Zionist entity'
is used as the entity"), a token collision the verifier +
deflection + title-relevance all pass and NLI returns neutral on.
Copulas inside a quoted span are skipped so 'war is peace' doesn't
break the subject/predicate split.
- vacuous: predicate is only placeholder hypernyms + filler ("X is
a thing").
Conservative — no full token-salad parsing; legit definitions pass ok.
Surfaced in inspect_cache_key + the `arborist inspect` human view
(· incoherent: <kind>). Advisory only — never writes providence_cache
/ audit_events / run_dag_root; demote-only verifier hook deliberately
not wired. 9 tests; full suite green (2500 passed).
Three small streams in one commit; each closes / expands a
recently-landed ticket without changing its hard contract.
#000026 Phase 3 wiring — authorship warrant ladder visible
============================================================
Phase 3 sidecar (arborist/qa/warrant_authorship.py landed in 60b5748)
exposed the classifier but didn't surface it. Two wirings:
- arborist/qa/inspect.py — diagnose_authorship_warrant runs against
the cached row's question + answer + per-source raw chunks +
URIs + titles; result lands as `authorship` field alongside the
other sidecars.
- arborist/cli.py _render_warrant_tail — appends ` · warrant:
<readable-tier>` when result['authorship'] is populated with a
non-quiet tier. AUTHOR_COPYRIGHT_FOOTER → "copyright-footer", etc.
NO_AUTHORSHIP_SIGNAL stays silent. Backward-compat: results
without an `authorship` key render unchanged.
Tests: 3 inspect-path tests (no-signal, copyright-footer,
repository-owner) + 4 render-tail tests (presence, no-signal
silence, missing-key silence, all-six-tiers readable mapping).
#000028 follow-ups — capital ledger + sample-rate
==================================================
Two policy fields layered on top of canonical_witness_enabled:
- canonical_witness_sample_rate (0.0..1.0; default 1.0). Operators
wanting passive calibration set 0.05 to fire witness on 5% of
canonical questions while paying 5% of LLM cost. 0.0 effectively
off; 1.0 = current always-on behavior. Gating uses random.random()
so distribution is uniform; clamped to [0, 1].
- Capital ledger row written for each FIRED witness (not skipped
ones). op_type='canonical_witness'; estimator inputs include
prompt_chars + answer_chars + llm_seconds + agreement_label +
pi_star_ref. Best-effort: ledger-write failure must never fail
the query (sidecar discipline).
Tests: 4 new — sample_rate=0.0 skips (no LLM call, no ledger row);
sample_rate=1.0 always fires; capital_ledger row written under
op_type='canonical_witness' with full input blob; sampled-out
witness records zero ledger rows.
Both fields fold into governance_policy_hash naturally via the
existing policy-hash machinery — flipping witness mode invalidates
prior records as expected.
#000025 Phase 1d — 5F fixture catalog 30 → 50
==============================================
Both synthetic and live sides of all 5 sub-batteries expanded
30 → 50 (+200 fixtures total: 5 × 20 synthetic, 5 × 20 live).
function — claim_count cycles 2..7 across new fixtures
falsification — 10-violation palette across new ids
feedback-loop — fact-N learning chains
finetuning — capability transitions across canonical π*
(math/logic/algebra/calculus pool)
formulate — multi-pointer claim shapes
500/500 pass through respective runners. test_session_integration
total bumped 562 → 662. Pinned test_5f_*_runs counts updated 30 →
50 (synthetic main + embedded + live).
Tests
=====
Full suite: 1467 passed, 36 skipped (was 1388; +79 across warrant
render + witness sample/ledger + 5F implicit coverage).
Three changes that shape the same lever:
(1) The metaphor-cue wordlist now unions /usr/share/dict/words +
/usr/share/dict/american-english + /usr/share/dict/british-english.
The Debian split made the prior 'just symlink to american-english'
miss British spellings (colour, organisation, realise) which
silently became false negatives on British-speaker questions.
Union: 102,485 → 104,305 entries on this machine. ~1,820 added
British-specific entries.
(2) Supplemental dictionary support: operators can layer
domain-specific vocabulary into the morphological substrate.
Two paths:
- Env var: ABORIST_METAPHOR_DICTS=/path/a:/path/b
- Programmatic: register_metaphor_dictionary(path)
Each supplemental dict is one word per line. The cue suffix
tests (-ly stem, -ing stem, -est stem) then resolve domain
stems automatically — adding 'aerodynamic' to a custom dict
makes 'aerodynamically' classify as adverbial without code
changes.
Use case: 'a tree with its own vocabulary' — an aviation
forest, a medical corpus, a legal-domain shard each carries
jargon the standard wordlist doesn't cover. Register once,
suffix tests pick up domain stems forever.
(3) README gains a 'Sidecar diagnostics' section with a table of
the three sidecars (deflection, title-relevance, metaphor-
deflection) plus a 'Metaphor-deflection cue dictionary' subsection
explaining the derivation rule, the load order, and the per-
forest vocabulary configurability. Architecturally documents
why the rule is *derived* from the union (Phase-2 lesson) and
not hand-curated.
3 new tests in tests/test_inspect.py:
- register_metaphor_dictionary unions a custom path's words
- ABORIST_METAPHOR_DICTS env var supplements with two paths
- re-registering same path is idempotent
763/34 tests pass.
Empirically motivated by the 2026-05-02 emergent log:
Q: 'How can a swallowtail butterfly, gracefully fluttering amidst
the rockiest terrain, remain undeterred by the upbraiding
winds...'
A: 'The Macleay's Swallowtail butterfly is found in Eastern
Australia including the ACT, New South Wales, Queensland...'
The model traded the metaphor for literal Macleay's-Swallowtail
taxonomic facts. Warrant passed (the literal anchor IS in cited
spans), DEFLECTION_DETECTED didn't fire (the last content token
'flight' did echo somewhere), the bench landed HYBRID 3/3 — but
the user's metaphorical question was never engaged.
Honest gap: catching this structurally requires NLI-grade
semantics, which is the verifier-semantic-gap design proposal.
Until that lands, ship a SMELL SIDECAR — purely lexical, sidecar
only, never enters the binary verifier output.
Detection rule:
1. Extract metaphor cues from the question:
- -ly adverbs (gracefully, defiantly), excluding common
-ly nouns (butterfly, italy, july) via blocklist
- -ing present participles >=6 chars (upbraiding,
fluttering, brooding), excluding common verb -ing forms
- -est superlatives >=6 chars (rockiest, harshest)
- prepositional cues (amidst, despite, against, beneath)
2. Count overlap with answer's content tokens.
3. Fire metaphor_deflection when:
cue_count >= 3 AND answer_overlap_count == 0
The threshold is conservative; the smell only triggers on
STRONGLY poetic questions with PURELY literal answers.
Wire-up:
- aborist/qa/inspect.py:diagnose_metaphor_deflection
- bench/qa_sweep.py: rows gain metaphor_deflection_kind +
metaphor_cue_count + metaphor_overlap_count
- scripts/bench_emergent.py: same fields on emergent log rows
5 new tests in tests/test_inspect.py:
- swallowtail canary case fires metaphor_deflection
- literal questions (mona lisa) return no_signal
- questions whose answer engages cues return no_signal
- common -ly nouns (butterfly, italy, july, family) filtered
- sub-threshold cue counts return no_signal
756/34 tests pass (5 new + 751 prior).
Empirical finding 2026-05-01 on 'explain spin glass modeling, tensors?':
returned STRICT 1/1 with claim 'Spin glass modeling involves...
mathematical tools such as tensors' cited to a chunk from the
*Quantum chromodynamics* article. Token-coverage check inside the
chunk passed accidentally on shared physics vocabulary
(spin/model/etc.), but the article isn't about spin glasses — the
claim is a retrieval-driven hallucination.
Root retrieval cause: AND-mode FTS5 returned zero hits (Spin_glass
article has 0 occurrences of 'modeling' or 'tensors'); OR-mode
fallback ranked by raw BM25, so 'Glass' (glass*274) outranked
'Spin glass' (spin*29 + glass*15) despite Spin_glass matching 2
distinct query tokens vs Glass's 1. Verifier then accepted the
hallucinated claim because the cited QCD chunk has incidental
physics vocabulary overlap.
New ``diagnose_title_relevance(claim_text, cited_source_titles)``
sidecar: claim's content tokens (stemmed) must overlap the union
of cited source titles. Zero overlap → ``title_mismatch``. Catches
the spin-tensors case (claim about spin glass, cited title 'Quantum
chromodynamics' → zero stem overlap) without false-positives on
legitimate cross-document grounding (Lois_Lane article cited for a
Lois-Lane claim still has lois+lane overlap).
Sidecar-only for now — staying out of the binary verifier output
until bench evidence justifies hard-promotion. Not yet wired into
inspect_cache_key (per-claim citation source mapping isn't in
providence_cache yet) or the bench. Available as a utility for
operators auditing specific (claim, source_title) pairs.
5 new unit tests covering match/mismatch/no-titles/no-claim-tokens
and underscore-normalization for Wikipedia titles.
Bench dig (2026-05-01, 65-question sweep): 4 of 7 deflecting
questions were false positives from verb-shape subject anchors.
what year did the berlin wall fall? → answer "1989" (anchor "fall")
when did the soviet union dissolve? → answer "1991" (anchor "dissolve")
how many wives did henry the eighth? → answer "six" (anchor "eighth")
what year does our cold fusion ...? → answer "1989" (anchor "happen")
why did the titanic sink? → "...sank in 1912" (anchor "sink" vs "sank")
why did the dinosaurs go extinct? → "asteroid impact" (anchor "extinct" vs "died")
These shapes ask for a year/count/cause; the answer is structurally
not expected to echo the question's subject. Pre-suppression, all
three modes landed deflection on these — the apparent
"pointer-only" deflection rate (10.3% vs 6.2%) was almost entirely
heuristic noise, not a real substrate failure.
New ``shape_suppressed`` flag returned by diagnose_deflection. When
the question starts with one of:
when, what year, what date, what time,
how many, how much, how long, how old, why
the subject-anchor check is skipped — overlap-ratio takes over.
Mars-BDFL (who-shape) still fires correctly.
Field exposed so the operator can filter the signal: a
shape_suppressed=True row that still classifies as deflection on
overlap-ratio means "answer was a bare year/count/cause with no
topic words" — known noisy, not real topic shift.
5 new unit tests; full suite 582 passed (was 578).
Empirical finding 2026-04-30 on 'who is a benevolent dictator for life
for mars?': JSON mode returned STRICT with answer 'Guido van Rossum is
a Benevolent Dictator For Life (BDFL) for the Python programming
language.' The verifier did its job (claims grounded), but the answer
never mentioned Mars — pure topic-shift. A real failure mode for
adversarial-premise questions where the model deflects rather than
refuses.
Detection rule: subject-anchor heuristic. The LAST content token in the
question (after stopword strip) is treated as the question's primary
subject ('mars' for BDFL, 'beatles' for 'who are the members of the
beatles?'). If the subject anchor is missing from the answer, classify
as 'deflection' regardless of generic-vocabulary overlap. The Mars-BDFL
case has 3/4 generic overlap (benevolent, dictator, life) but missing
subject — exactly the pattern overlap-ratio alone can't catch.
Returns dict with kind ∈ {deflection, partial_overlap, on_topic,
no_question_tokens} plus subject_anchor + subject_in_answer for
operator legibility.
Sidecar-only — wired into inspect_cache_key() but never feeds back
into providence_cache or audit_events. Per the verifier-stays-binary
discipline (CLAUDE.md): soft signals live in inspect verbs, never in
the hard chain.
Two paired enhancements (D + E from the toy-Hermes design pass):
D. Per-claim status taxonomy on `verify_quotes`.
New `claim_statuses` field on every verdict — a per-evidence-unit
list with three labels:
VERIFIED_QUOTE unit substring-matched in normalized context
(any of quote/span/entity strategies)
SUPPORTED_PARAPHRASE unit cleared the paraphrase token-coverage
threshold (≥85% topical tokens present)
UNSUPPORTED unit didn't match anything
Diagnostic labels (QUOTE_INTEGRITY_FAILED, SOURCE_MISMATCH,
FALSIFIED) stay in the sidecar / falsification machinery — the
binary-verifier discipline holds. Empty list when no evidence
was extracted at all (verifier_method='none'). Backward-compatible:
existing fields (audit_mode, n_quotes, n_verified, unverified_quotes,
verifier_method) unchanged; current callers ignore the new field.
E. Repair-action plans on sidecar diagnoses.
Each `_classify_span` diagnosis now carries a `repair` field with
a concrete suggestion the operator can act on:
synthetic_elision_inside_quote → split_into_two_quotes
(when both halves verbatim)
→ trim_to_verified_half
(when only one half verbatim)
→ remove_claim
interior_elision → include_aside_for_verbatim
(with the dropped aside text)
trailing_artifact → trim_trailing_artifact
(with the kept_prefix string)
paraphrase → downgrade_to_paraphrase
partial_paraphrase → split_or_remove
no_overlap → remove_claim
Read-only suggestions — sidecar still doesn't write to providence_cache
or audit_events. The repair stage is recommendation, not mutation.
Operator (or an automated repair pass) decides whether to act.
Human render in `aborist inspect` shows `repair: <action> (<reason>)`
under each diagnosis line.
Tests:
- verify: claim_statuses_quote_path_labels_each_unit (per-quote VERIFIED
/ UNSUPPORTED), claim_statuses_paraphrase_method_flagged,
claim_statuses_empty_when_no_evidence.
- inspect: repair_synthetic_elision_split_when_both_halves_verbatim,
repair_interior_elision_includes_aside, repair_trailing_artifact_trim,
repair_no_overlap_remove.
462 tests pass (verify +3, inspect +4).
Two enhancements from the toy-Hermes design pass (2026-04-30):
A. Synthetic-elision-inside-quote diagnosis (sidecar only).
Distinct from interior_elision (model dropped a `(...)` aside source
carries) — synthetic_elision is the model writing literal `[...]`
between fragments of a `"..."` span, signaling self-elision while
claiming verbatim citation. The verifier still rejects (binary
discipline holds), but `aborist inspect` now reports
`diagnosis: synthetic_elision_inside_quote` with prefix/suffix
presence flags so an operator can judge whether the elided middle
was benign. Probe runs first in the classify-span chain (more
specific than trailing_artifact / interior_elision / paraphrase).
Catches the Brachiosaurus case: `"The film centers on the fictional
Isla Nublar [...] Universal Studios..."` — both halves are in source,
but the literal `[...]` isn't, so substring match correctly fails &
the sidecar tells the operator why.
C. Source-role classification + role-weighted context budget.
`_classify_source_role(title, qtokens_stem)` tags each top-K hit:
primary_answer_source 2.0× cap strong title-stem overlap
secondary_context_source 1.0× cap "list of", "characters",
"franchise", "history of"
noisy_background_source 0.5× cap "score", "music",
"video game", "merchandise"
sequel_background_source 0.5× cap "lost world", roman numerals
background_source 1.0× cap default
Order matters: noisy/sequel/secondary markers fire before the
primary check so peripheral pages with strong title overlap (e.g.
`Jurassic Park (film score)` shares 3 stems with the JP-film query)
don't claim a primary slot.
Cap loop now applies role weight on top of the baseline
`max_context_chars / top_k`. Total context still bounded by the
running `char_budget` — weights just shift how the budget gets
divided so primary pages get more text & noisy pages less, fixing
the case where a `(film score)` page consumed a primary slot.
`source_role` is persisted on `_Hit` and surfaces on
`merkle_proof.sources[*].source_role` in the providence record so
inspect & audits can see which slot each source occupied.
Tests:
- inspect: synthetic_elision_caught (Brachiosaurus regression),
synthetic_elision_does_not_fire_when_source_has_brackets (false-
positive guard).
- query: role classifier matrix (primary / secondary / noisy / sequel /
background) on JP-film-style titles, role persistence in sources list.
455 tests pass (sidecar +2, query +2).
Fox 2026-04-29 (Clark Kent / Superman query): the verifier flagged a
quoted span as unverified and the sidecar diagnosed it as 'paraphrase'
with token_coverage=1.0. Investigation showed the source actually said
`Clark Joseph Kent (middle name is also Jerome according to some
versions) is a fictional character...` and the model elided the
parenthetical aside, quoting `Clark Joseph Kent is a fictional
character...`. Every word in source, but the sequence has a 60-char
gap where the aside was. Distinct failure mode from real paraphrase
(token reordering) and trailing_artifact (model APPENDS).
Sidecar refinement, no verifier change. The binary verifier discipline
(memory rule `feedback_verifier_no_diagnostics`) holds — quote strategy
is still verbatim-only; soft signal stays in the inspect verb.
Algorithm walks every `(` in base. For each open paren at position P:
the longest k where base[:P] ends with span[:k] is the model's prefix;
the span tail (≥20 chars) must then match the source after the close
paren. First paren that satisfies both checks wins.
CLI human-render shows `matched: N prefix + M suffix chars
(parenthetical aside dropped)` plus the dropped_aside text so an
operator can decide at a glance whether the elision is benign.
Live record now reads:
[1] interior_elision
matched: 17 prefix + 138 suffix chars (parenthetical aside dropped)
dropped_aside: middle name is also jerome according to some versions
439 tests pass (sidecar +2, full suite still green).
Per fox: the verifier reports n/N verified but doesn't tell you WHY a
span didn't ground. Some are paraphrase, some are model-added
trailing artifacts (e.g. `(Source: https://...)` citations the model
appended to verbatim prose), some are full inventions. Operator
needs to see which is which to triage.
Read-only sidecar — no audit events, no providence_cache mutations,
no v9.8 field changes. Per the verifier-no-diagnostics memory rule:
hard chain stays binary; soft signals live in sidecar verbs that
never feed back.
Five diagnosis labels:
verbatim_in_base span IS in base context — verifier or
canonicalization bug (worth flagging loudly).
verbatim_in_raw_only raw wikitext match but base form differs —
wikitext-strip edge case.
trailing_artifact a >=60-char prefix matches; the tail (often a
model-added citation) doesn't. Surfaces the
tail explicitly.
paraphrase >=85% query tokens (>4 chars) present in
base context but not in this sequence.
Model rewrote source content.
partial_paraphrase 40-85% coverage — mixed sourced/emergent.
no_overlap <40% coverage — likely full invention.
Verified live against the real Pikachu cache record (5 sources,
71KB raw -> 34KB base): two spans flagged paraphrase (token_coverage
1.0 for both, just rewritten sequence), one flagged trailing_artifact
(100-char prefix matches; tail is `(Source: https://...)` citation).
That's the actual reason HYBRID — model paraphrased + appended
citations that aren't in the corpus.
Surface:
aborist inspect --cache-key <hex> [--qa-db ...] [--json]
make inspect KEY=<hex> [JSON=1]
Tests: 9 (6 classifier on synthetic contexts pinning each label,
3 end-to-end with seeded fixture record + verification that inspect
writes nothing). 318 passed, 1 skipped overall.
