qa(client): persistent httpx.Client; HTTP/1.1 keep-alive across calls
OpenAICompatibleClient.chat_completion previously constructed a
fresh httpx.Client per request inside a 'with' block:
with httpx.Client(timeout=self.timeout) as client:
resp = client.post(url, headers=headers, json=payload)
Each call paid a TLS handshake (~100-300ms) — wall-clock cost
that bench --concurrency surfaces sharply. With 426 calls per
bench sweep, the throwaway-client pattern was burning roughly
1-2 minutes of pure connection setup per sweep.
Move the httpx.Client to __init__ so it lives across calls.
HTTP/1.1 keep-alive holds the TCP+TLS connection open between
chat_completion invocations from the same client instance;
httpx's internal connection pool is thread-safe so the bench's
ThreadPoolExecutor can share one client across worker threads.
Add close() / __enter__ / __exit__ for clean shutdown — the
client now has connection pool state worth releasing
explicitly. Existing callers that don't use the context-manager
form work unchanged because httpx.Client cleanup runs on GC.
Smoke at --concurrency 6 against hermes.ai.unturf.com: 15 cells
in 2m24s, no errors. vLLM continuous-batching observed: first 6
cells completed together at ~76s, suggesting the batch filled
and processed as a unit. Subsequent waves at 14-50s as the queue
drained.
Predicted bench savings: 1-2 min off the ~51 min full sweep.
Combined with --concurrency 6, predicted total: ~30 min full
bench (vs 51 min at concurrency=4).
This commit is contained in:
parent
4b356030fb
commit
c49e1beb1f
1 changed files with 24 additions and 2 deletions
|
|
@ -91,6 +91,28 @@ class OpenAICompatibleClient:
|
|||
self.timeout = timeout
|
||||
self.max_retries = max(1, max_retries)
|
||||
self.retry_backoff_base_s = retry_backoff_base_s
|
||||
# Persistent httpx.Client for HTTP/1.1 keep-alive + connection
|
||||
# reuse. Constructing a fresh Client per chat_completion paid a
|
||||
# TLS handshake every call (~100-300ms vs free for keep-alive),
|
||||
# which adds up fast under bench --concurrency. httpx.Client is
|
||||
# thread-safe for sequential or concurrent use; its internal
|
||||
# connection pool serializes pool access. Closed via close().
|
||||
import httpx
|
||||
self._http = httpx.Client(timeout=self.timeout)
|
||||
|
||||
def close(self) -> None:
|
||||
"""Release the underlying connection pool."""
|
||||
try:
|
||||
self._http.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
self.close()
|
||||
return False
|
||||
|
||||
def chat_completion(
|
||||
self,
|
||||
|
|
@ -106,6 +128,7 @@ class OpenAICompatibleClient:
|
|||
import httpx
|
||||
import time as _time
|
||||
|
||||
client = self._http # persistent connection pool from __init__
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if self.api_key:
|
||||
headers["Authorization"] = f"Bearer {self.api_key}"
|
||||
|
|
@ -128,8 +151,7 @@ class OpenAICompatibleClient:
|
|||
last_exc: Exception | None = None
|
||||
for attempt in range(self.max_retries):
|
||||
try:
|
||||
with httpx.Client(timeout=self.timeout) as client:
|
||||
resp = client.post(url, headers=headers, json=payload)
|
||||
resp = client.post(url, headers=headers, json=payload)
|
||||
if resp.status_code in self._RETRY_STATUS and attempt < self.max_retries - 1:
|
||||
# Exponential backoff: 0.5s, 1.0s, 2.0s with the
|
||||
# default base. Last attempt raises through.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue