russell@unturf.com
39c040cacc
Root cause of 'arborist abstains on everything with qwen' (fox 2026-05-21): Qwen3 thinking-on default burns the entire token budget on hidden <think> reasoning over a 20K RAG context and returns EMPTY message.content (measured: 768/768 completion tokens, content '') -> every arborist answer UNGROUNDED. Bench harnesses passed enable_thinking=False via the MODELS dict, but the CLI + control_ab did not, so the quality bench was measuring a thinking-budget-exhaustion artifact, not abstention. OpenAICompatibleClient now defaults Qwen3 to enable_thinking=False unless a caller set it explicitly (reasoning-variant path passes True, preserved). Verified: same France query goes empty/UNGROUNDED -> STRICT 'Nicolas Sarkozy' with the flag. Fixes every caller (CLI, control_ab). 5 tests; full suite 2547 passed. Today's qwen QUALITY numbers are void and need re-running; energy numbers stand (real inference happened regardless).
34 lines
1.2 KiB
Python
34 lines
1.2 KiB
Python
"""Qwen3 enable_thinking=False default (2026-05-21).
|
|
|
|
Qwen3 thinking-on default burns the token budget on hidden <think> over a
|
|
large RAG context and returns empty content → every answer UNGROUNDED.
|
|
The client forces enable_thinking=False for Qwen3 unless a caller set it.
|
|
"""
|
|
from arborist.qa.client import _qwen3_no_think_default as f
|
|
|
|
|
|
def test_qwen3_gets_no_think_by_default():
|
|
eb = f("Qwen3.6-27B-UD-Q4_K_XL.gguf", None)
|
|
assert eb["chat_template_kwargs"]["enable_thinking"] is False
|
|
|
|
|
|
def test_qwen3_merges_into_existing_extra_body():
|
|
eb = f("Qwen3.6-27B", {"guided_json": {"x": 1}})
|
|
assert eb["guided_json"] == {"x": 1}
|
|
assert eb["chat_template_kwargs"]["enable_thinking"] is False
|
|
|
|
|
|
def test_explicit_thinking_true_is_preserved():
|
|
# the reasoning-variant path must keep thinking ON.
|
|
eb = f("Qwen3.6-27B", {"chat_template_kwargs": {"enable_thinking": True}})
|
|
assert eb["chat_template_kwargs"]["enable_thinking"] is True
|
|
|
|
|
|
def test_non_qwen_untouched():
|
|
assert f("adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic", None) is None
|
|
eb = {"foo": 1}
|
|
assert f("Hermes-3", eb) is eb # unchanged object
|
|
|
|
|
|
def test_empty_model_untouched():
|
|
assert f("", None) is None
|