diff --git a/public/inference.html b/public/inference.html index adc0b15..42fa278 100644 --- a/public/inference.html +++ b/public/inference.html @@ -92,6 +92,7 @@
We use vLLM to run models, generally with full f16 safetensors. We make sure to use a virtualenv to hold the dependencies.
Note: For Hermes, we use an FP8 quant by adamo1139 (adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic), which is optimized for 4090 and 3090 GPUs.
+Tool Calling: The --enable-auto-tool-choice and --tool-call-parser hermes flags are required for Hermes 3 to support function calling via the OpenAI-compatible API. Without these flags, tool calling requests will fail.
We are considering supporting ollama for better quant support.
Stand up a replica cluster on a new domain.
sudo apt-get install gcc python3.12-dev python3.12-venv
@@ -99,7 +100,7 @@ cd ~
python3 -m venv env
source env/bin/activate
pip install vllm
-python -m vllm.entrypoints.openai.api_server --model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic --host 0.0.0.0 --port 18888 --max-model-len 82000
+python -m vllm.entrypoints.openai.api_server --model adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic --host 0.0.0.0 --port 18888 --max-model-len 82000 --enable-auto-tool-choice --tool-call-parser hermes