fix: Qwen3 defaults to enable_thinking=False — was returning empty answers

Root cause of 'arborist abstains on everything with qwen' (fox 2026-05-21):
Qwen3 thinking-on default burns the entire token budget on hidden <think>
reasoning over a 20K RAG context and returns EMPTY message.content
(measured: 768/768 completion tokens, content '') -> every arborist answer
UNGROUNDED. Bench harnesses passed enable_thinking=False via the MODELS
dict, but the CLI + control_ab did not, so the quality bench was measuring
a thinking-budget-exhaustion artifact, not abstention.

OpenAICompatibleClient now defaults Qwen3 to enable_thinking=False unless a
caller set it explicitly (reasoning-variant path passes True, preserved).
Verified: same France query goes empty/UNGROUNDED -> STRICT 'Nicolas
Sarkozy' with the flag. Fixes every caller (CLI, control_ab). 5 tests;
full suite 2547 passed. Today's qwen QUALITY numbers are void and need
re-running; energy numbers stand (real inference happened regardless).
This commit is contained in:
russell@unturf.com 2026-05-21 12:50:05 -04:00
parent 2fd3523777
commit 39c040cacc
No known key found for this signature in database
2 changed files with 56 additions and 0 deletions

View file

@ -58,6 +58,27 @@ class StubClient:
# want to assert extra_body was passed should inspect `self.calls`.
def _qwen3_no_think_default(model: str, extra_body: dict | None) -> dict | None:
"""Default Qwen3 to enable_thinking=False unless the caller set it.
Qwen3 thinking models default to enable_thinking=True. On a large RAG
context that burns the entire token budget on hidden <think> reasoning
and returns EMPTY ``message.content`` (measured 2026-05-21: 768/768
completion tokens, content '') every arborist answer comes back
UNGROUNDED. arborist's QA path wants the answer, not the trace, so we
force enable_thinking=False for Qwen3 by default. The reasoning-variant
path passes enable_thinking=True explicitly and is preserved untouched.
"""
if not model or "qwen3" not in model.lower():
return extra_body
eb = dict(extra_body or {})
ctk = dict(eb.get("chat_template_kwargs") or {})
if "enable_thinking" not in ctk:
ctk["enable_thinking"] = False
eb["chat_template_kwargs"] = ctk
return eb
class OpenAICompatibleClient:
"""OpenAI-compatible chat completion over HTTP.
@ -177,6 +198,7 @@ class OpenAICompatibleClient:
# extra_body merges into the payload root — vLLM accepts knobs
# like {"guided_json": {...schema...}} or {"guided_grammar": "..."}.
# Endpoints that don't recognize a key silently drop it.
extra_body = _qwen3_no_think_default(model, extra_body)
if extra_body:
for k, v in extra_body.items():
payload[k] = v

View file

@ -0,0 +1,34 @@
"""Qwen3 enable_thinking=False default (2026-05-21).
Qwen3 thinking-on default burns the token budget on hidden <think> over a
large RAG context and returns empty content every answer UNGROUNDED.
The client forces enable_thinking=False for Qwen3 unless a caller set it.
"""
from arborist.qa.client import _qwen3_no_think_default as f
def test_qwen3_gets_no_think_by_default():
eb = f("Qwen3.6-27B-UD-Q4_K_XL.gguf", None)
assert eb["chat_template_kwargs"]["enable_thinking"] is False
def test_qwen3_merges_into_existing_extra_body():
eb = f("Qwen3.6-27B", {"guided_json": {"x": 1}})
assert eb["guided_json"] == {"x": 1}
assert eb["chat_template_kwargs"]["enable_thinking"] is False
def test_explicit_thinking_true_is_preserved():
# the reasoning-variant path must keep thinking ON.
eb = f("Qwen3.6-27B", {"chat_template_kwargs": {"enable_thinking": True}})
assert eb["chat_template_kwargs"]["enable_thinking"] is True
def test_non_qwen_untouched():
assert f("adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic", None) is None
eb = {"foo": 1}
assert f("Hermes-3", eb) is eb # unchanged object
def test_empty_model_untouched():
assert f("", None) is None