feat(#000057): arborist+qwen enablement — multi-engine JSON-schema + per-model extras pass-through

Two surgical fixes unblock 'arborist with synthesis LLM = Qwen-on-
llama.cpp' as a viable arm in the control sweep. Pre-existing
docstring said 'Arborist×Qwen needs proof-path guided_json+extra_body
surgery — coupled follow-up'; this is that follow-up.

Fix 1 — multi-engine structured-output extras

The runner / query JSON-mode paths previously sent only vLLM's
'guided_json' key for the claim_lattice schema. llama.cpp silently
drops it, leaving Qwen un-enforced (the parse-tolerant fallback did
all the work). Helper

    claim_lattice_structured_output_extras() in arborist/qa/verify.py

now returns a dict carrying the schema under all three engine
conventions:

  - guided_json     (vLLM grammar-constrained sampling)
  - json_schema     (llama.cpp native shorthand)
  - response_format (OpenAI-spec, honoured by llama.cpp and newer vLLM)

Each engine recognises its own key and silently drops the others.
Used at both inference call sites (runner.py:740, query.py:3324).
Hermes/vLLM path is unchanged — it picks up 'guided_json' and
ignores the other two.

Fix 2 — query() accepts user-supplied extra_body, merges with defaults

query() grew a keyword-only extra_body parameter (default None).
Per-model knobs (Qwen's {'chat_template_kwargs': {'enable_thinking':
False}} toggle, future template knobs) can flow from the caller to
the synthesis chat-completion call. Schema-enforcement extras are
added inside query() and merge under user keys — common case is
disjoint namespaces, but if a caller wants to override 'guided_json'
they can.

bench/control_sweep.py now passes MODELS[arborist_ref]['extra']
through to query() in the arborist branch, so --arborist-ref
qwen-nothink runs with reasoning disabled and --arborist-ref
qwen-think runs with reasoning enabled. Phase 1's arborist arm with
--arborist-ref=hermes is unaffected (MODELS['hermes']['extra'] is
None, merges to no-op).

Tests
  + 3 new in tests/test_verify_json.py covering helper default shape,
    alternate-schema reuse, and query()'s new extra_body parameter
  220 affected tests still green (verify / claim_lattice / judge /
    runner suite)
  pytest test_verify_json: 27/27

Next: small smoke run --arborist-ref qwen-nothink against 4-8 items
to confirm end-to-end before any full sweep. Phase 2 (qwen-think solo)
still running in background, unaffected — it doesn't touch the
arborist arm.
This commit is contained in:
russell@unturf.com 2026-05-19 19:53:00 -04:00
parent 3450e8a281
commit c6621ee700
No known key found for this signature in database
5 changed files with 141 additions and 11 deletions

View file

@ -189,10 +189,21 @@ def _process_item(idx: int, it: dict, variants: list[str],
"judge_raw": (v.raw or "")[:700]})
if arborist_on:
try:
# Pass through MODELS[arborist_ref]["extra"] so
# per-model knobs (e.g. Qwen's enable_thinking
# toggle under chat_template_kwargs) reach the
# synthesis call in the proof-path. Without this,
# an arborist-ref=qwen-nothink sweep would still
# emit reasoning tokens because the toggle never
# crosses query()'s boundary. JSON-schema extras
# are added inside query() via
# claim_lattice_structured_output_extras() and
# merge with this per-model extras dict.
r = query(question=q_asked, qa_db=qa_db,
chat_client=cl(arborist_ref),
model_id=MODELS[arborist_ref]["model"],
shards_dir=shards_dir,
extra_body=MODELS[arborist_ref]["extra"],
policy=dict(DEFAULT_QUERY_POLICY,
answer_mode="claim_lattice"))
arb_raw = (r.get("raw_answer")