feat(#000057): arborist+qwen enablement — multi-engine JSON-schema + per-model extras pass-through
Two surgical fixes unblock 'arborist with synthesis LLM = Qwen-on-
llama.cpp' as a viable arm in the control sweep. Pre-existing
docstring said 'Arborist×Qwen needs proof-path guided_json+extra_body
surgery — coupled follow-up'; this is that follow-up.
Fix 1 — multi-engine structured-output extras
The runner / query JSON-mode paths previously sent only vLLM's
'guided_json' key for the claim_lattice schema. llama.cpp silently
drops it, leaving Qwen un-enforced (the parse-tolerant fallback did
all the work). Helper
claim_lattice_structured_output_extras() in arborist/qa/verify.py
now returns a dict carrying the schema under all three engine
conventions:
- guided_json (vLLM grammar-constrained sampling)
- json_schema (llama.cpp native shorthand)
- response_format (OpenAI-spec, honoured by llama.cpp and newer vLLM)
Each engine recognises its own key and silently drops the others.
Used at both inference call sites (runner.py:740, query.py:3324).
Hermes/vLLM path is unchanged — it picks up 'guided_json' and
ignores the other two.
Fix 2 — query() accepts user-supplied extra_body, merges with defaults
query() grew a keyword-only extra_body parameter (default None).
Per-model knobs (Qwen's {'chat_template_kwargs': {'enable_thinking':
False}} toggle, future template knobs) can flow from the caller to
the synthesis chat-completion call. Schema-enforcement extras are
added inside query() and merge under user keys — common case is
disjoint namespaces, but if a caller wants to override 'guided_json'
they can.
bench/control_sweep.py now passes MODELS[arborist_ref]['extra']
through to query() in the arborist branch, so --arborist-ref
qwen-nothink runs with reasoning disabled and --arborist-ref
qwen-think runs with reasoning enabled. Phase 1's arborist arm with
--arborist-ref=hermes is unaffected (MODELS['hermes']['extra'] is
None, merges to no-op).
Tests
+ 3 new in tests/test_verify_json.py covering helper default shape,
alternate-schema reuse, and query()'s new extra_body parameter
220 affected tests still green (verify / claim_lattice / judge /
runner suite)
pytest test_verify_json: 27/27
Next: small smoke run --arborist-ref qwen-nothink against 4-8 items
to confirm end-to-end before any full sweep. Phase 2 (qwen-think solo)
still running in background, unaffected — it doesn't touch the
arborist arm.
This commit is contained in:
parent
3450e8a281
commit
c6621ee700
5 changed files with 141 additions and 11 deletions
|
|
@ -523,3 +523,64 @@ def test_verify_json_subject_tokens_absent_below_threshold_passes():
|
|||
f"expected STRICT (below threshold); got {v['audit_mode']} "
|
||||
f"with violations {[vio['kind'] for vio in v['violations']]}"
|
||||
)
|
||||
|
||||
|
||||
# --- multi-engine structured-output extras (claim_lattice JSON mode) -------
|
||||
|
||||
def test_claim_lattice_structured_output_extras_default_shape():
|
||||
"""The helper returns extras under all three engine conventions so
|
||||
the same call site enforces the JSON schema across vLLM, llama.cpp,
|
||||
and OpenAI-spec endpoints. Engines silently drop unknown keys, so
|
||||
sending all three is safe."""
|
||||
from arborist.qa.verify import (
|
||||
CLAIM_LATTICE_JSON_SCHEMA,
|
||||
claim_lattice_structured_output_extras,
|
||||
)
|
||||
|
||||
e = claim_lattice_structured_output_extras()
|
||||
assert set(e.keys()) == {"guided_json", "json_schema", "response_format"}
|
||||
|
||||
# vLLM convention — the schema directly under `guided_json`.
|
||||
assert e["guided_json"] is CLAIM_LATTICE_JSON_SCHEMA
|
||||
|
||||
# llama.cpp shorthand — the schema directly under `json_schema`.
|
||||
assert e["json_schema"] is CLAIM_LATTICE_JSON_SCHEMA
|
||||
|
||||
# OpenAI-spec — wrapped under response_format with the type tag.
|
||||
rf = e["response_format"]
|
||||
assert rf["type"] == "json_schema"
|
||||
inner = rf["json_schema"]
|
||||
assert inner["name"] == "claim_lattice"
|
||||
assert inner["schema"] is CLAIM_LATTICE_JSON_SCHEMA
|
||||
assert inner["strict"] is True
|
||||
|
||||
|
||||
def test_claim_lattice_structured_output_extras_accepts_alternate_schema():
|
||||
"""The helper isn't claim-lattice-specific in mechanism — pass a
|
||||
different schema and an alternate name to reuse the multi-engine
|
||||
bridge for any structured-output feature."""
|
||||
from arborist.qa.verify import claim_lattice_structured_output_extras
|
||||
|
||||
other = {"type": "object", "properties": {"x": {"type": "string"}},
|
||||
"required": ["x"], "additionalProperties": False}
|
||||
e = claim_lattice_structured_output_extras(other, name="my_thing")
|
||||
assert e["guided_json"] is other
|
||||
assert e["json_schema"] is other
|
||||
assert e["response_format"]["json_schema"]["name"] == "my_thing"
|
||||
assert e["response_format"]["json_schema"]["schema"] is other
|
||||
|
||||
|
||||
def test_query_accepts_extra_body_parameter():
|
||||
"""query() grew an extra_body keyword so per-model knobs (Qwen's
|
||||
enable_thinking, etc.) can flow through to the synthesis call.
|
||||
Schema-enforcement extras are added inside query() and merge with
|
||||
the user-supplied dict (user keys override defaults on collision —
|
||||
common case is disjoint namespaces)."""
|
||||
import inspect
|
||||
from arborist.qa.query import query
|
||||
|
||||
sig = inspect.signature(query)
|
||||
assert "extra_body" in sig.parameters
|
||||
p = sig.parameters["extra_body"]
|
||||
# keyword-only with default None — accepts any caller, optional.
|
||||
assert p.default is None
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue