qa(client): persistent httpx.Client; HTTP/1.1 keep-alive across calls

OpenAICompatibleClient.chat_completion previously constructed a
fresh httpx.Client per request inside a 'with' block:

    with httpx.Client(timeout=self.timeout) as client:
        resp = client.post(url, headers=headers, json=payload)

Each call paid a TLS handshake (~100-300ms) — wall-clock cost
that bench --concurrency surfaces sharply. With 426 calls per
bench sweep, the throwaway-client pattern was burning roughly
1-2 minutes of pure connection setup per sweep.

Move the httpx.Client to __init__ so it lives across calls.
HTTP/1.1 keep-alive holds the TCP+TLS connection open between
chat_completion invocations from the same client instance;
httpx's internal connection pool is thread-safe so the bench's
ThreadPoolExecutor can share one client across worker threads.

Add close() / __enter__ / __exit__ for clean shutdown — the
client now has connection pool state worth releasing
explicitly. Existing callers that don't use the context-manager
form work unchanged because httpx.Client cleanup runs on GC.

Smoke at --concurrency 6 against hermes.ai.unturf.com: 15 cells
in 2m24s, no errors. vLLM continuous-batching observed: first 6
cells completed together at ~76s, suggesting the batch filled
and processed as a unit. Subsequent waves at 14-50s as the queue
drained.

Predicted bench savings: 1-2 min off the ~51 min full sweep.
Combined with --concurrency 6, predicted total: ~30 min full
bench (vs 51 min at concurrency=4).
This commit is contained in:
russell@unturf.com 2026-05-02 10:03:17 -04:00
parent 4b356030fb
commit c49e1beb1f
No known key found for this signature in database

View file

@ -91,6 +91,28 @@ class OpenAICompatibleClient:
self.timeout = timeout
self.max_retries = max(1, max_retries)
self.retry_backoff_base_s = retry_backoff_base_s
# Persistent httpx.Client for HTTP/1.1 keep-alive + connection
# reuse. Constructing a fresh Client per chat_completion paid a
# TLS handshake every call (~100-300ms vs free for keep-alive),
# which adds up fast under bench --concurrency. httpx.Client is
# thread-safe for sequential or concurrent use; its internal
# connection pool serializes pool access. Closed via close().
import httpx
self._http = httpx.Client(timeout=self.timeout)
def close(self) -> None:
"""Release the underlying connection pool."""
try:
self._http.close()
except Exception:
pass
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
self.close()
return False
def chat_completion(
self,
@ -106,6 +128,7 @@ class OpenAICompatibleClient:
import httpx
import time as _time
client = self._http # persistent connection pool from __init__
headers = {"Content-Type": "application/json"}
if self.api_key:
headers["Authorization"] = f"Bearer {self.api_key}"
@ -128,8 +151,7 @@ class OpenAICompatibleClient:
last_exc: Exception | None = None
for attempt in range(self.max_retries):
try:
with httpx.Client(timeout=self.timeout) as client:
resp = client.post(url, headers=headers, json=payload)
resp = client.post(url, headers=headers, json=payload)
if resp.status_code in self._RETRY_STATUS and attempt < self.max_retries - 1:
# Exponential backoff: 0.5s, 1.0s, 2.0s with the
# default base. Last attempt raises through.