engineering/uncloseai.com
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

clarify vLLM setup: f16 safetensors generally, FP8 quant for Hermes (optimized for 4090/3090)

Browse source
This commit is contained in:
Russell Ballestrini 2025-10-15 12:23:15 -04:00
parent f7c5d6d553
commit 4c5960a6be
1 changed files with 2 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

3
inference.html
View file

@ -90,7 +90,8 @@
<p>This section is optional. This is only if you wanted to try to contribute idle GPU time to the project or if you wanted to reproduce everything in your own cluster.</p>
<h3 id="vllm-setup">vLLM Setup</h3>
<p>We use vLLM to run models, currently full f16 safetensors. We make sure to use a virtualenv to hold the dependencies.</p>
<p>We use vLLM to run models, generally with full f16 safetensors. We make sure to use a virtualenv to hold the dependencies.</p>
<p><strong>Note:</strong> For Hermes, we use an FP8 quant by adamo1139 (adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic), which is optimized for 4090 and 3090 GPUs.</p>
<p>We are considering supporting ollama for better quant support.</p>
<p>Stand up a replica cluster on a new domain.</p>
<pre><code>sudo apt-get install gcc python3.12-dev python3.12-venv