engineering/uncloseai.com
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: Add tool calling flags for Hermes 3 vLLM setup

Browse source 
Add --enable-auto-tool-choice and --tool-call-parser hermes flags to the
vLLM server command. These flags are required for Hermes 3 to support
function calling via the OpenAI-compatible API.
This commit is contained in:
Russell Ballestrini 2025-11-27 09:08:53 -05:00
parent 911c4281da
commit baa0a1fd7c
1 changed files with 2 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

3
public/inference.html
View file

@ -92,6 +92,7 @@
<h3 id="vllm-setup">vLLM Setup</h3>
<p>We use vLLM to run models, generally with full f16 safetensors. We make sure to use a virtualenv to hold the dependencies.</p>
<p><strong>Note:</strong> For Hermes, we use an FP8 quant by adamo1139 (adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic), which is optimized for 4090 and 3090 GPUs.</p>
<p><strong>Tool Calling:</strong> The <code>--enable-auto-tool-choice</code> and <code>--tool-call-parser hermes</code> flags are required for Hermes 3 to support function calling via the OpenAI-compatible API. Without these flags, tool calling requests will fail.</p>
<p>We are considering supporting ollama for better quant support.</p>
<p>Stand up a replica cluster on a new domain.</p>
<pre><code>sudo apt-get install gcc python3.12-dev python3.12-venv
@ -99,7 +100,7 @@ cd ~
python3 -m venv env
source env/bin/activate
pip install vllm
python -m vllm.entrypoints.openai.api_server --model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic --host 0.0.0.0 --port 18888 --max-model-len 82000
python -m vllm.entrypoints.openai.api_server --model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic --host 0.0.0.0 --port 18888 --max-model-len 82000 --enable-auto-tool-choice --tool-call-parser hermes
</code></pre>
<h3 id="tts-setup">TTS Setup</h3>