From baa0a1fd7c32112a3f09c3c557258f52f5fd6c1a Mon Sep 17 00:00:00 2001 From: Russell Ballestrini Date: Thu, 27 Nov 2025 09:08:53 -0500 Subject: [PATCH] docs: Add tool calling flags for Hermes 3 vLLM setup Add --enable-auto-tool-choice and --tool-call-parser hermes flags to the vLLM server command. These flags are required for Hermes 3 to support function calling via the OpenAI-compatible API. --- public/inference.html | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/public/inference.html b/public/inference.html index adc0b15..42fa278 100644 --- a/public/inference.html +++ b/public/inference.html @@ -92,6 +92,7 @@

vLLM Setup

We use vLLM to run models, generally with full f16 safetensors. We make sure to use a virtualenv to hold the dependencies.

Note: For Hermes, we use an FP8 quant by adamo1139 (adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic), which is optimized for 4090 and 3090 GPUs.

+

Tool Calling: The --enable-auto-tool-choice and --tool-call-parser hermes flags are required for Hermes 3 to support function calling via the OpenAI-compatible API. Without these flags, tool calling requests will fail.

We are considering supporting ollama for better quant support.

Stand up a replica cluster on a new domain.

sudo apt-get install gcc python3.12-dev python3.12-venv
@@ -99,7 +100,7 @@ cd ~
 python3 -m venv env
 source env/bin/activate
 pip install vllm
-python -m vllm.entrypoints.openai.api_server --model  adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic --host 0.0.0.0 --port 18888 --max-model-len 82000
+python -m vllm.entrypoints.openai.api_server --model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic --host 0.0.0.0 --port 18888 --max-model-len 82000 --enable-auto-tool-choice --tool-call-parser hermes

TTS Setup