fix: Qwen3 defaults to enable_thinking=False — was returning empty answers
Root cause of 'arborist abstains on everything with qwen' (fox 2026-05-21): Qwen3 thinking-on default burns the entire token budget on hidden <think> reasoning over a 20K RAG context and returns EMPTY message.content (measured: 768/768 completion tokens, content '') -> every arborist answer UNGROUNDED. Bench harnesses passed enable_thinking=False via the MODELS dict, but the CLI + control_ab did not, so the quality bench was measuring a thinking-budget-exhaustion artifact, not abstention. OpenAICompatibleClient now defaults Qwen3 to enable_thinking=False unless a caller set it explicitly (reasoning-variant path passes True, preserved). Verified: same France query goes empty/UNGROUNDED -> STRICT 'Nicolas Sarkozy' with the flag. Fixes every caller (CLI, control_ab). 5 tests; full suite 2547 passed. Today's qwen QUALITY numbers are void and need re-running; energy numbers stand (real inference happened regardless).
This commit is contained in:
parent
2fd3523777
commit
39c040cacc
2 changed files with 56 additions and 0 deletions
|
|
@ -58,6 +58,27 @@ class StubClient:
|
|||
# want to assert extra_body was passed should inspect `self.calls`.
|
||||
|
||||
|
||||
def _qwen3_no_think_default(model: str, extra_body: dict | None) -> dict | None:
|
||||
"""Default Qwen3 to enable_thinking=False unless the caller set it.
|
||||
|
||||
Qwen3 thinking models default to enable_thinking=True. On a large RAG
|
||||
context that burns the entire token budget on hidden <think> reasoning
|
||||
and returns EMPTY ``message.content`` (measured 2026-05-21: 768/768
|
||||
completion tokens, content '') — every arborist answer comes back
|
||||
UNGROUNDED. arborist's QA path wants the answer, not the trace, so we
|
||||
force enable_thinking=False for Qwen3 by default. The reasoning-variant
|
||||
path passes enable_thinking=True explicitly and is preserved untouched.
|
||||
"""
|
||||
if not model or "qwen3" not in model.lower():
|
||||
return extra_body
|
||||
eb = dict(extra_body or {})
|
||||
ctk = dict(eb.get("chat_template_kwargs") or {})
|
||||
if "enable_thinking" not in ctk:
|
||||
ctk["enable_thinking"] = False
|
||||
eb["chat_template_kwargs"] = ctk
|
||||
return eb
|
||||
|
||||
|
||||
class OpenAICompatibleClient:
|
||||
"""OpenAI-compatible chat completion over HTTP.
|
||||
|
||||
|
|
@ -177,6 +198,7 @@ class OpenAICompatibleClient:
|
|||
# extra_body merges into the payload root — vLLM accepts knobs
|
||||
# like {"guided_json": {...schema...}} or {"guided_grammar": "..."}.
|
||||
# Endpoints that don't recognize a key silently drop it.
|
||||
extra_body = _qwen3_no_think_default(model, extra_body)
|
||||
if extra_body:
|
||||
for k, v in extra_body.items():
|
||||
payload[k] = v
|
||||
|
|
|
|||
34
tests/test_qwen_thinking_default.py
Normal file
34
tests/test_qwen_thinking_default.py
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
"""Qwen3 enable_thinking=False default (2026-05-21).
|
||||
|
||||
Qwen3 thinking-on default burns the token budget on hidden <think> over a
|
||||
large RAG context and returns empty content → every answer UNGROUNDED.
|
||||
The client forces enable_thinking=False for Qwen3 unless a caller set it.
|
||||
"""
|
||||
from arborist.qa.client import _qwen3_no_think_default as f
|
||||
|
||||
|
||||
def test_qwen3_gets_no_think_by_default():
|
||||
eb = f("Qwen3.6-27B-UD-Q4_K_XL.gguf", None)
|
||||
assert eb["chat_template_kwargs"]["enable_thinking"] is False
|
||||
|
||||
|
||||
def test_qwen3_merges_into_existing_extra_body():
|
||||
eb = f("Qwen3.6-27B", {"guided_json": {"x": 1}})
|
||||
assert eb["guided_json"] == {"x": 1}
|
||||
assert eb["chat_template_kwargs"]["enable_thinking"] is False
|
||||
|
||||
|
||||
def test_explicit_thinking_true_is_preserved():
|
||||
# the reasoning-variant path must keep thinking ON.
|
||||
eb = f("Qwen3.6-27B", {"chat_template_kwargs": {"enable_thinking": True}})
|
||||
assert eb["chat_template_kwargs"]["enable_thinking"] is True
|
||||
|
||||
|
||||
def test_non_qwen_untouched():
|
||||
assert f("adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic", None) is None
|
||||
eb = {"foo": 1}
|
||||
assert f("Hermes-3", eb) is eb # unchanged object
|
||||
|
||||
|
||||
def test_empty_model_untouched():
|
||||
assert f("", None) is None
|
||||
Loading…
Add table
Add a link
Reference in a new issue