OpenAICompatibleClient.chat_completion previously constructed a
fresh httpx.Client per request inside a 'with' block:
with httpx.Client(timeout=self.timeout) as client:
resp = client.post(url, headers=headers, json=payload)
Each call paid a TLS handshake (~100-300ms) — wall-clock cost
that bench --concurrency surfaces sharply. With 426 calls per
bench sweep, the throwaway-client pattern was burning roughly
1-2 minutes of pure connection setup per sweep.
Move the httpx.Client to __init__ so it lives across calls.
HTTP/1.1 keep-alive holds the TCP+TLS connection open between
chat_completion invocations from the same client instance;
httpx's internal connection pool is thread-safe so the bench's
ThreadPoolExecutor can share one client across worker threads.
Add close() / __enter__ / __exit__ for clean shutdown — the
client now has connection pool state worth releasing
explicitly. Existing callers that don't use the context-manager
form work unchanged because httpx.Client cleanup runs on GC.
Smoke at --concurrency 6 against hermes.ai.unturf.com: 15 cells
in 2m24s, no errors. vLLM continuous-batching observed: first 6
cells completed together at ~76s, suggesting the batch filled
and processed as a unit. Subsequent waves at 14-50s as the queue
drained.
Predicted bench savings: 1-2 min off the ~51 min full sweep.
Combined with --concurrency 6, predicted total: ~30 min full
bench (vs 51 min at concurrency=4).
russell@unturf.com
c49e1beb1f