qa: live fixture — red fish blue fish → Dr. Seuss book

Adds test_red_fish_blue_fish_identifies_seuss_book under the new
"retrieval disambiguation" section. The query partially matches
many noise titles (Red Dwarf, Blue whale, Toronto Blue Jays, Red
Sea, Detroit Red Wings, marine aquarium fish list) so it stress-
tests the retrieval+rerank pipeline's ability to land on the right
article — `One Fish Two Fish Red Fish Blue Fish` by Dr. Seuss —
despite the surface noise. Asserts the answer is grounded and
contains a Seuss-specific marker (seuss, rhyming, children's book,
or creatures). 10 tests collected total.
This commit is contained in:
russell@unturf.com 2026-04-30 14:55:39 -04:00
parent 1ef93f4d2f
commit 1d7a9ee38d
No known key found for this signature in database

View file

@ -141,6 +141,24 @@ def test_mona_lisa_grounded_and_names_leonardo():
f"answer missing 'Leonardo': {_answer_lower(r)[:200]}"
# ---------------------------------------------------------------- retrieval disambiguation
def test_red_fish_blue_fish_identifies_seuss_book():
"""Stress-tests retrieval disambiguation: "red fish blue fish" partially
matches many noise titles (Red Dwarf, Blue whale, Toronto Blue Jays,
Red Sea, Detroit Red Wings, marine aquarium fish list...). The right
answer is the Dr. Seuss book ``One Fish Two Fish Red Fish Blue Fish``.
The model should anchor on the Seuss article and the answer should
contain a Seuss-specific token."""
r = _ask("plot of red fish blue fish?")
assert _grounded(r), f"audit_mode={r.get('audit_mode')!r}"
txt = _answer_lower(r)
seuss_markers = ("seuss", "rhyming", "children's book", "creatures")
assert any(m in txt for m in seuss_markers), \
f"answer missing Seuss-book markers: {txt[:300]}"
# ---------------------------------------------------------------- entity disambiguation