qa(inspect): metaphor-deflection sidecar (METAPHORICAL_DEFLECTION smell)
Empirically motivated by the 2026-05-02 emergent log:
Q: 'How can a swallowtail butterfly, gracefully fluttering amidst
the rockiest terrain, remain undeterred by the upbraiding
winds...'
A: 'The Macleay's Swallowtail butterfly is found in Eastern
Australia including the ACT, New South Wales, Queensland...'
The model traded the metaphor for literal Macleay's-Swallowtail
taxonomic facts. Warrant passed (the literal anchor IS in cited
spans), DEFLECTION_DETECTED didn't fire (the last content token
'flight' did echo somewhere), the bench landed HYBRID 3/3 — but
the user's metaphorical question was never engaged.
Honest gap: catching this structurally requires NLI-grade
semantics, which is the verifier-semantic-gap design proposal.
Until that lands, ship a SMELL SIDECAR — purely lexical, sidecar
only, never enters the binary verifier output.
Detection rule:
1. Extract metaphor cues from the question:
- -ly adverbs (gracefully, defiantly), excluding common
-ly nouns (butterfly, italy, july) via blocklist
- -ing present participles >=6 chars (upbraiding,
fluttering, brooding), excluding common verb -ing forms
- -est superlatives >=6 chars (rockiest, harshest)
- prepositional cues (amidst, despite, against, beneath)
2. Count overlap with answer's content tokens.
3. Fire metaphor_deflection when:
cue_count >= 3 AND answer_overlap_count == 0
The threshold is conservative; the smell only triggers on
STRONGLY poetic questions with PURELY literal answers.
Wire-up:
- aborist/qa/inspect.py:diagnose_metaphor_deflection
- bench/qa_sweep.py: rows gain metaphor_deflection_kind +
metaphor_cue_count + metaphor_overlap_count
- scripts/bench_emergent.py: same fields on emergent log rows
5 new tests in tests/test_inspect.py:
- swallowtail canary case fires metaphor_deflection
- literal questions (mona lisa) return no_signal
- questions whose answer engages cues return no_signal
- common -ly nouns (butterfly, italy, july, family) filtered
- sub-threshold cue counts return no_signal
756/34 tests pass (5 new + 751 prior).
This commit is contained in:
parent
a1dd330455
commit
8fec3a5d56
4 changed files with 240 additions and 1 deletions
|
|
@ -22,6 +22,7 @@ from aborist.qa.inspect import (
|
|||
_classify_span,
|
||||
_normalize,
|
||||
diagnose_deflection,
|
||||
diagnose_metaphor_deflection,
|
||||
diagnose_title_relevance,
|
||||
inspect_cache_key,
|
||||
)
|
||||
|
|
@ -604,3 +605,82 @@ def test_title_mismatch_no_claim_tokens():
|
|||
"""Defensive: empty/stopword-only claim returns no_claim_tokens."""
|
||||
d = diagnose_title_relevance("the a an", ["Anything"])
|
||||
assert d["kind"] == "no_claim_tokens"
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
# Metaphor-deflection sidecar
|
||||
# ─────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_metaphor_deflection_swallowtail_canary():
|
||||
"""The original 2026-05-02 emergent log case: swallowtail butterfly
|
||||
question framed metaphorically (gracefully fluttering amidst the
|
||||
rockiest terrain undeterred by upbraiding winds), answer was
|
||||
purely literal taxonomic (Macleay's Swallowtail found in Eastern
|
||||
Australia ...). Sidecar should fire."""
|
||||
q = (
|
||||
"How can a swallowtail butterfly, gracefully fluttering amidst "
|
||||
"the rockiest terrain, remain undeterred by the upbraiding "
|
||||
"winds that seem to challenge its delicate flight?"
|
||||
)
|
||||
a = (
|
||||
"The Macleay's Swallowtail butterfly is found in Eastern "
|
||||
"Australia including the ACT, New South Wales, Queensland, "
|
||||
"Victoria and Tasmania."
|
||||
)
|
||||
d = diagnose_metaphor_deflection(q, a)
|
||||
assert d["kind"] == "metaphor_deflection"
|
||||
# cue_count should pick up at least: amidst, fluttering, gracefully,
|
||||
# rockiest, upbraiding (≥5 real cues; butterfly filtered).
|
||||
assert d["cue_count"] >= 4
|
||||
assert "amidst" in d["cue_tokens"]
|
||||
assert "rockiest" in d["cue_tokens"]
|
||||
assert "upbraiding" in d["cue_tokens"]
|
||||
assert "gracefully" in d["cue_tokens"]
|
||||
assert "butterfly" not in d["cue_tokens"] # filtered noun
|
||||
assert d["answer_overlap_count"] == 0
|
||||
|
||||
|
||||
def test_metaphor_deflection_no_signal_on_literal_question():
|
||||
"""Plain factual questions don't have metaphor cues; sidecar
|
||||
returns no_signal."""
|
||||
d = diagnose_metaphor_deflection(
|
||||
"who painted the mona lisa?",
|
||||
"Leonardo da Vinci painted the Mona Lisa around 1503.",
|
||||
)
|
||||
assert d["kind"] == "no_signal"
|
||||
assert d["cue_count"] == 0
|
||||
|
||||
|
||||
def test_metaphor_deflection_no_signal_when_answer_engages_cues():
|
||||
"""If the answer echoes any of the question's metaphor cues, the
|
||||
sidecar does NOT fire — the model engaged with the framing."""
|
||||
q = "gracefully amidst rockiest terrain undeterred by upbraiding winds"
|
||||
a = "The bird flies gracefully amidst the rockiest terrain undeterred."
|
||||
d = diagnose_metaphor_deflection(q, a)
|
||||
assert d["kind"] == "no_signal"
|
||||
assert d["answer_overlap_count"] >= 1
|
||||
|
||||
|
||||
def test_metaphor_deflection_filters_common_ly_nouns():
|
||||
"""Naive .endswith('ly') would pick up 'butterfly', 'family',
|
||||
'italy', 'july' etc. as adverbs. The block-list filters them so
|
||||
they don't inflate cue count."""
|
||||
from aborist.qa.inspect import _extract_metaphor_cues
|
||||
cues = _extract_metaphor_cues(
|
||||
"the butterfly flew over italy in july with the family"
|
||||
)
|
||||
# None of the -ly-suffix nouns should appear as cues.
|
||||
for noun in ("butterfly", "italy", "july", "family"):
|
||||
assert noun not in cues, f"{noun} leaked through as a cue"
|
||||
|
||||
|
||||
def test_metaphor_deflection_under_threshold_returns_no_signal():
|
||||
"""Sidecar requires >=3 cue tokens to fire — a single -ly word
|
||||
isn't enough signal to flag metaphor framing."""
|
||||
d = diagnose_metaphor_deflection(
|
||||
"What gracefully describes a circle?",
|
||||
"A circle is the set of points equidistant from a center.",
|
||||
)
|
||||
assert d["kind"] == "no_signal"
|
||||
assert d["cue_count"] < 3
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue