russell@unturf.com
f23d3a3067
Bench data shows residual JSON-mode token runaway after the
pointer-ID switch: ~4 cases out of 66 land UNGROUNDED 0/0 at
12-15s instead of ~2-4s normal. Pattern: Hermes-3-8B emits a
valid claim object, then keeps generating whitespace / blank
lines until max_tokens (512) exhausts. The truncated payload
won't parse and the lenient pre-parser returns no claims.
Concrete instances in the latest bench (2026-04-30T19-55-11Z):
- tell me about the apollo program (3/3 samples runaway)
- tell me about the python programming language (1/3)
Fix: pass `stop=["\n\n"]` to the chat completion in JSON mode so
vLLM cuts generation at the first blank line. Well-formed JSON-
mode output never legitimately contains a blank line — Hermes
emits one object on a single line (or with simple internal
newlines), never `\n\n`. The stop sequence is the runaway
signature itself.
Plumbing:
- OpenAICompatibleClient.chat_completion: new `stop` kwarg,
injects into request payload when non-empty
- StubClient already absorbs **kwargs; no change needed
- DEFAULT_POLICY (runner) + DEFAULT_QUERY_POLICY (query) gain
`claim_lattice_json_stop_sequences = ["\n\n"]`. Folds into
governance_policy_hash so changing the stop list invalidates
prior cached records.
- Both call sites in runner.py / query.py read the policy
field and pass it only on JSON mode (pointer + quote modes
don't need it).
Defensive measure: worst case the stop sequence never fires;
best case the apollo/python residuals recover and JSON's
strict-rate climbs further.
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| client.py | ||
| concepts.py | ||
| dag.py | ||
| evidence.py | ||
| inspect.py | ||
| keys.py | ||
| parse_claims.py | ||
| query.py | ||
| repair.py | ||
| runner.py | ||
| verify.py | ||