Using the Hermes AI Model
+At ai.unturf.com, we offer a free AI service powered by the model NousResearch/Hermes-3-Llama-3.1-8B. Our mission is to provide accessible AI tools for everyone, embodying the principles of both free as in beer & free as in freedom. You can interact with our model without any cost, and you are encouraged to contribute and build upon the open-source code & models that we use.
-Installing the OpenAI Client
-Python
-To install the OpenAI package for Python, use pip:
pip install openaiInstalling the OpenAI Client
+Python
+To install the OpenAI package for Python, use pip:
pip install openaiNode.js
-To install the OpenAI package for Node.js, you can use npm in your package.json:
{
+ Node.js
+ To install the OpenAI package for Node.js, you can use npm in your package.json:
+ {
"dependencies": {
"openai": "^v4.67.3" // Use the latest version
}
}
-
- Run the following command to install it:
- npm install
+ Run the following command to install it:
+npm installPython Example
-Non-Streaming
-
+ Python Example
+ Non-Streaming
+ # Python Fizzbuzz Example
from openai import OpenAI
client = OpenAI(base_url="https://hermes.ai.unturf.com/v1", api_key="none")
-# Set your model
MODEL = "NousResearch/Hermes-3-Llama-3.1-8B"
-# Define your messages
messages = [{"role": "user", "content": "Give a Python Fizzbuzz solution in one line of code?"}]
-# Make the request
response = client.chat.completions.create(
model=MODEL,
messages=messages,
@@ -122,25 +92,21 @@ response = client.chat.completions.create(
max_tokens=150
)
-# Print the response
print(response.choices[0].message.content)
-
+ Streaming
-
+ Streaming
+ # Streaming response in Python
from openai import OpenAI
client = OpenAI(base_url="https://hermes.ai.unturf.com/v1", api_key="none")
-# Set your model
MODEL = "NousResearch/Hermes-3-Llama-3.1-8B"
-# Define your messages
messages = [
{"role": "user", "content": "Give a Python Fizzbuzz solution in one line of code?"}
]
-# Make the streaming request
response = client.chat.completions.create(
model=MODEL,
messages=messages,
@@ -149,16 +115,14 @@ response = client.chat.completions.create(
stream=True, # Enable streaming
)
-# Print each message in the streaming response
for chunk in response:
- # The 'ChatCompletionChunk' object exposes the 'choices' attribute directly
if hasattr(chunk.choices[0].delta, "content"):
print(chunk.choices[0].delta.content, end="")
-
+ Node.js Example
-Non-Streaming
-const OpenAI = require('openai');
+ Node.js Example
+ Non-Streaming
+ const OpenAI = require('openai');
const client = new OpenAI({
baseURL: "https://hermes.ai.unturf.com/v1",
@@ -185,10 +149,10 @@ async function getResponse() {
}
getResponse();
-
+ Streaming
-
+ Streaming
+
const OpenAI = require('openai');
const client = new OpenAI({
@@ -221,60 +185,68 @@ async function streamResponse() {
}
streamResponse();
-
+ How we run inference if you wanted to try to contribute
+How we run inference if you wanted to try to contribute
-We use vLLM to run models, currently full f16 safetensors. We make sure to use a virtualenv to hold the dependencies.
-We are considering supporting ollama for better quant support.
-
+ We use vLLM to run models, currently full f16 safetensors. We make sure to use a virtualenv to hold the dependencies.
+ We are considering supporting ollama for better quant support.
+
cd ~
python3 -m venv env
source env/bin/activate
pip install vllm
python -m vllm.entrypoints.openai.api_server --model NousResearch/Hermes-3-Llama-3.1-8B --host 0.0.0.0 --port 18888 --max-model-len 16000
-
+