engineering/arborist
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

qa(inspect): metaphor-deflection sidecar (METAPHORICAL_DEFLECTION smell)

Browse source 
Empirically motivated by the 2026-05-02 emergent log:

  Q: 'How can a swallowtail butterfly, gracefully fluttering amidst
      the rockiest terrain, remain undeterred by the upbraiding
      winds...'
  A: 'The Macleay's Swallowtail butterfly is found in Eastern
      Australia including the ACT, New South Wales, Queensland...'

The model traded the metaphor for literal Macleay's-Swallowtail
taxonomic facts. Warrant passed (the literal anchor IS in cited
spans), DEFLECTION_DETECTED didn't fire (the last content token
'flight' did echo somewhere), the bench landed HYBRID 3/3 — but
the user's metaphorical question was never engaged.

Honest gap: catching this structurally requires NLI-grade
semantics, which is the verifier-semantic-gap design proposal.
Until that lands, ship a SMELL SIDECAR — purely lexical, sidecar
only, never enters the binary verifier output.

Detection rule:
  1. Extract metaphor cues from the question:
     - -ly adverbs (gracefully, defiantly), excluding common
       -ly nouns (butterfly, italy, july) via blocklist
     - -ing present participles >=6 chars (upbraiding,
       fluttering, brooding), excluding common verb -ing forms
     - -est superlatives >=6 chars (rockiest, harshest)
     - prepositional cues (amidst, despite, against, beneath)
  2. Count overlap with answer's content tokens.
  3. Fire metaphor_deflection when:
       cue_count >= 3  AND  answer_overlap_count == 0
     The threshold is conservative; the smell only triggers on
     STRONGLY poetic questions with PURELY literal answers.

Wire-up:
  - aborist/qa/inspect.py:diagnose_metaphor_deflection
  - bench/qa_sweep.py: rows gain metaphor_deflection_kind +
    metaphor_cue_count + metaphor_overlap_count
  - scripts/bench_emergent.py: same fields on emergent log rows

5 new tests in tests/test_inspect.py:
  - swallowtail canary case fires metaphor_deflection
  - literal questions (mona lisa) return no_signal
  - questions whose answer engages cues return no_signal
  - common -ly nouns (butterfly, italy, july, family) filtered
  - sub-threshold cue counts return no_signal

756/34 tests pass (5 new + 751 prior).
This commit is contained in:
russell@unturf.com 2026-05-02 13:40:41 -04:00
parent a1dd330455
commit 8fec3a5d56
No known key found for this signature in database
4 changed files with 240 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

12
scripts/bench_emergent.py
View file

@ -155,6 +155,15 @@ def run_one_cycle(
t_answer = time.time()
# Sidecar smell signal: metaphorical question framing the answer
# ignored. Surfaced 2026-05-02 by the swallowtail/upbraided/rockiest
# emergent log entry. Off-the-binary-chain — gives the teacher
# reviewer a flag for poetic-question-vs-literal-answer mismatches.
from aborist.qa.inspect import diagnose_metaphor_deflection
metaphor = diagnose_metaphor_deflection(
question or "", result.get("answer_text") or ""
)
return {
"ts": int(t_pick),
"iso_ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(t_pick)),
@ -169,6 +178,9 @@ def run_one_cycle(
"violation_kinds": sorted(
{v.get("kind") for v in (result.get("violations") or []) if v.get("kind")}
),
"metaphor_deflection_kind": metaphor["kind"],
"metaphor_cue_count": metaphor["cue_count"],
"metaphor_overlap_count": metaphor["answer_overlap_count"],
"sources": [
{"title": s.get("title"), "uri": s.get("document_uri"), "used": s.get("used")}
for s in (result.get("sources") or [])