qa(inspect): metaphor-deflection sidecar (METAPHORICAL_DEFLECTION smell)

Empirically motivated by the 2026-05-02 emergent log:

  Q: 'How can a swallowtail butterfly, gracefully fluttering amidst
      the rockiest terrain, remain undeterred by the upbraiding
      winds...'
  A: 'The Macleay's Swallowtail butterfly is found in Eastern
      Australia including the ACT, New South Wales, Queensland...'

The model traded the metaphor for literal Macleay's-Swallowtail
taxonomic facts. Warrant passed (the literal anchor IS in cited
spans), DEFLECTION_DETECTED didn't fire (the last content token
'flight' did echo somewhere), the bench landed HYBRID 3/3 — but
the user's metaphorical question was never engaged.

Honest gap: catching this structurally requires NLI-grade
semantics, which is the verifier-semantic-gap design proposal.
Until that lands, ship a SMELL SIDECAR — purely lexical, sidecar
only, never enters the binary verifier output.

Detection rule:
  1. Extract metaphor cues from the question:
     - -ly adverbs (gracefully, defiantly), excluding common
       -ly nouns (butterfly, italy, july) via blocklist
     - -ing present participles >=6 chars (upbraiding,
       fluttering, brooding), excluding common verb -ing forms
     - -est superlatives >=6 chars (rockiest, harshest)
     - prepositional cues (amidst, despite, against, beneath)
  2. Count overlap with answer's content tokens.
  3. Fire metaphor_deflection when:
       cue_count >= 3  AND  answer_overlap_count == 0
     The threshold is conservative; the smell only triggers on
     STRONGLY poetic questions with PURELY literal answers.

Wire-up:
  - aborist/qa/inspect.py:diagnose_metaphor_deflection
  - bench/qa_sweep.py: rows gain metaphor_deflection_kind +
    metaphor_cue_count + metaphor_overlap_count
  - scripts/bench_emergent.py: same fields on emergent log rows

5 new tests in tests/test_inspect.py:
  - swallowtail canary case fires metaphor_deflection
  - literal questions (mona lisa) return no_signal
  - questions whose answer engages cues return no_signal
  - common -ly nouns (butterfly, italy, july, family) filtered
  - sub-threshold cue counts return no_signal

756/34 tests pass (5 new + 751 prior).
This commit is contained in:
russell@unturf.com 2026-05-02 13:40:41 -04:00
parent a1dd330455
commit 8fec3a5d56
No known key found for this signature in database
4 changed files with 240 additions and 1 deletions

View file

@ -22,6 +22,7 @@ from aborist.qa.inspect import (
_classify_span,
_normalize,
diagnose_deflection,
diagnose_metaphor_deflection,
diagnose_title_relevance,
inspect_cache_key,
)
@ -604,3 +605,82 @@ def test_title_mismatch_no_claim_tokens():
"""Defensive: empty/stopword-only claim returns no_claim_tokens."""
d = diagnose_title_relevance("the a an", ["Anything"])
assert d["kind"] == "no_claim_tokens"
# ─────────────────────────────────────────────────────────────────────
# Metaphor-deflection sidecar
# ─────────────────────────────────────────────────────────────────────
def test_metaphor_deflection_swallowtail_canary():
"""The original 2026-05-02 emergent log case: swallowtail butterfly
question framed metaphorically (gracefully fluttering amidst the
rockiest terrain undeterred by upbraiding winds), answer was
purely literal taxonomic (Macleay's Swallowtail found in Eastern
Australia ...). Sidecar should fire."""
q = (
"How can a swallowtail butterfly, gracefully fluttering amidst "
"the rockiest terrain, remain undeterred by the upbraiding "
"winds that seem to challenge its delicate flight?"
)
a = (
"The Macleay's Swallowtail butterfly is found in Eastern "
"Australia including the ACT, New South Wales, Queensland, "
"Victoria and Tasmania."
)
d = diagnose_metaphor_deflection(q, a)
assert d["kind"] == "metaphor_deflection"
# cue_count should pick up at least: amidst, fluttering, gracefully,
# rockiest, upbraiding (≥5 real cues; butterfly filtered).
assert d["cue_count"] >= 4
assert "amidst" in d["cue_tokens"]
assert "rockiest" in d["cue_tokens"]
assert "upbraiding" in d["cue_tokens"]
assert "gracefully" in d["cue_tokens"]
assert "butterfly" not in d["cue_tokens"] # filtered noun
assert d["answer_overlap_count"] == 0
def test_metaphor_deflection_no_signal_on_literal_question():
"""Plain factual questions don't have metaphor cues; sidecar
returns no_signal."""
d = diagnose_metaphor_deflection(
"who painted the mona lisa?",
"Leonardo da Vinci painted the Mona Lisa around 1503.",
)
assert d["kind"] == "no_signal"
assert d["cue_count"] == 0
def test_metaphor_deflection_no_signal_when_answer_engages_cues():
"""If the answer echoes any of the question's metaphor cues, the
sidecar does NOT fire the model engaged with the framing."""
q = "gracefully amidst rockiest terrain undeterred by upbraiding winds"
a = "The bird flies gracefully amidst the rockiest terrain undeterred."
d = diagnose_metaphor_deflection(q, a)
assert d["kind"] == "no_signal"
assert d["answer_overlap_count"] >= 1
def test_metaphor_deflection_filters_common_ly_nouns():
"""Naive .endswith('ly') would pick up 'butterfly', 'family',
'italy', 'july' etc. as adverbs. The block-list filters them so
they don't inflate cue count."""
from aborist.qa.inspect import _extract_metaphor_cues
cues = _extract_metaphor_cues(
"the butterfly flew over italy in july with the family"
)
# None of the -ly-suffix nouns should appear as cues.
for noun in ("butterfly", "italy", "july", "family"):
assert noun not in cues, f"{noun} leaked through as a cue"
def test_metaphor_deflection_under_threshold_returns_no_signal():
"""Sidecar requires >=3 cue tokens to fire — a single -ly word
isn't enough signal to flag metaphor framing."""
d = diagnose_metaphor_deflection(
"What gracefully describes a circle?",
"A circle is the set of points equidistant from a center.",
)
assert d["kind"] == "no_signal"
assert d["cue_count"] < 3