qa: live fixture — red fish blue fish → Dr. Seuss book
Adds test_red_fish_blue_fish_identifies_seuss_book under the new "retrieval disambiguation" section. The query partially matches many noise titles (Red Dwarf, Blue whale, Toronto Blue Jays, Red Sea, Detroit Red Wings, marine aquarium fish list) so it stress- tests the retrieval+rerank pipeline's ability to land on the right article — `One Fish Two Fish Red Fish Blue Fish` by Dr. Seuss — despite the surface noise. Asserts the answer is grounded and contains a Seuss-specific marker (seuss, rhyming, children's book, or creatures). 10 tests collected total.
This commit is contained in:
parent
1ef93f4d2f
commit
1d7a9ee38d
1 changed files with 18 additions and 0 deletions
|
|
@ -141,6 +141,24 @@ def test_mona_lisa_grounded_and_names_leonardo():
|
|||
f"answer missing 'Leonardo': {_answer_lower(r)[:200]}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- retrieval disambiguation
|
||||
|
||||
|
||||
def test_red_fish_blue_fish_identifies_seuss_book():
|
||||
"""Stress-tests retrieval disambiguation: "red fish blue fish" partially
|
||||
matches many noise titles (Red Dwarf, Blue whale, Toronto Blue Jays,
|
||||
Red Sea, Detroit Red Wings, marine aquarium fish list...). The right
|
||||
answer is the Dr. Seuss book ``One Fish Two Fish Red Fish Blue Fish``.
|
||||
The model should anchor on the Seuss article and the answer should
|
||||
contain a Seuss-specific token."""
|
||||
r = _ask("plot of red fish blue fish?")
|
||||
assert _grounded(r), f"audit_mode={r.get('audit_mode')!r}"
|
||||
txt = _answer_lower(r)
|
||||
seuss_markers = ("seuss", "rhyming", "children's book", "creatures")
|
||||
assert any(m in txt for m in seuss_markers), \
|
||||
f"answer missing Seuss-book markers: {txt[:300]}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- entity disambiguation
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue