engineering/opencompletion.com
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

dynamic models for all the platforms! #10

Merged
russellballestrini merged 2 commits from dynamic-models into main 2025-03-11 22:02:53 -04:00
Conversation 14 Commits 2 Files changed 7 +244 -1010
7 changed files with 244 additions and 1010 deletions
Download patch file Download diff file Expand all files Collapse all files

1
.gitignore vendored
View file

@ -6,3 +6,4 @@ __pycache__/
.flaskenv-exported
.aws-sam/
samconfig.toml
vars.sh

66
README.rst
View file

@ -39,9 +39,6 @@ Requirements
- eventlet or gevent
- boto3 (for interacting with AWS Bedrock currently Claude, and S3 access)
- openai (for interacting with OpenAI's language models)
- mistralai (for interacting with MistralAI's language models)
- together (for interacting with together.ai language models)
- groq (for interacting with very fast groq language models)
Installation
------------
@ -72,27 +69,21 @@ To set up the project, follow these steps:
Usage
-----
Set up environment variables for your AWS, OpenAI, MistralAI, together.ai, grok, groq, google, API keys.
* make a copy of ``vars.sh.sample`` and fill in your API keys!
Set up optional environment variables for your AWS, OpenAI, MistralAI, or together.ai API keys::
Other env vars::
export AWS_ACCESS_KEY_ID="your_access_key"
export AWS_SECRET_ACCESS_KEY="your_secret_key"
export S3_BUCKET_NAME="your_s3_bucket_name"
export OPENAI_API_KEY="your_openai_api_key"
export MISTRAL_API_KEY="your_mistralai_api_key"
export TOGETHER_API_KEY="your_togetherai_api_key"
export GROQ_API_KEY="your_groq_api_key"
export XAI_API_KEY="your_twitter_x_ai_api_key_for_grok"
export GOOGLE_API_KEY="your_google_gemini_api_key"
export VLLM_API_KEY="not-needed"
export VLLM_ENDPOINT="http://localhost:18888/v1"
Here are some free endpoint for research only!::
export VLLM_ENDPOINT_1=https://hermes.ai.unturf.com/v1
export VLLM_ENDPOINT_2=https://node2.naptha.ai/inference
export VLLM_ENDPOINT_3=https://node3.naptha.ai/inference
export MODEL_ENDPOINT_1=https://hermes.ai.unturf.com/v1
export MODEL_ENDPOINT_2=https://node2.naptha.ai/inference
export MODEL_ENDPOINT_3=https://node3.naptha.ai/inference
To start the application with socket.io run::
@ -114,44 +105,7 @@ The application will be available at ``http://127.0.0.1:5001`` by default.
Interacting with Language Models
--------------------------------
To interact with the various language models, you can use the following commands within the chat:
- For GPT-3, send a message with ``gpt-3`` and include your prompt.
- For GPT-4o, send a message with ``gpt-4`` and include your prompt.
- For GPT-4o cheapest version, send a message with ``gpt-4o-2024-08-06`` and include your prompt.
- For GPT-4o-mini, send a message with ``gpt-mini`` and include your prompt.
- For GPT-o1-mini, send a message with ``gpt-o1-mini`` and include your prompt.
- For GPT-o1-preview, send a message with ``gpt-o1-preview`` and include your prompt.
- For GPT-o1, send a message with ``gpt-o1`` and include your prompt.
- For GPT-o3-mini, send a message with ``gpt-o3-mini`` and include your prompt.
..
- For GPT-o3-mini-medium, send a message with ``gpt-o3-mini-medium`` and include your prompt.
- For GPT-o3-mini-high, send a message with ``gpt-o3-mini-high`` and include your prompt
- For Claude-haiku, send a message with ``claude-haiku`` and include your prompt.
- For Claude-sonnet, send a message with ``claude-sonnet`` and include your prompt.
- For Claude-opus, send a message with ``claude-opus`` and include your prompt.
- For Mistral-tiny, send a message with ``mistral-tiny`` and include your prompt.
- For Mistral-small, send a message with ``mistral-small`` and include your prompt.
- For Mistral-medium, send a message with ``mistral-medium`` and include your prompt.
- For Mistral-large, send a message with ``mistral-large`` and include your prompt.
- For Mistral-nemo, send a message with ``mistral-nemo`` and include your prompt.
- For Mistral-codestral, send a message with ``mistral-codestral`` and include your prompt.
- For Together OpenChat, send a message with ``together/openchat`` and include your prompt.
- For Together Mistral, send a message with ``together/mistral`` and include your prompt.
- For Together Mixtral, send a message with ``together/mixtral`` and include your prompt.
- For Together Solar, send a message with ``together/solar`` and include your prompt.
- For Groq Mixtral, send a message with ``groq/mixtral`` and include your prompt.
- For Groq Llama-2, send a message with ``groq/llama2`` and include your prompt.
- For Groq Llama-3, send a message with ``groq/llama3`` and include your prompt.
- For Groq Gemma, send a message with ``groq/gemma`` and include your prompt.
- For Google Gemini Flash, send a message with ``gemini-flash`` and include your prompt.
- For Google Gemini Flash 8B, send a message with ``gemini-flash-8b`` and include your prompt.
- For Google Gemini Pro, send a message with ``gemini-pro`` and include your prompt.
- For Twitter/X AI Grok, send a message with ``grok-beta`` and include your prompt.
- For vLLM Hermes, send a message with ``vllm/hermes`` and include your prompt.
- For vLLM Deepseek R1 32B, send a message with ``vllm/r1`` and include your prompt.
- For Ollama Hermes, send a message with ``ollama/hermes`` and include your prompt.
- For Dall-e-3, send a message with ``dall-e-3`` and include your prompt.
To interact with the various language models, choose from the drop down and send a message!
The system will process your message and provide a response from the selected language model.
@ -230,7 +184,11 @@ I prefer the ``vllm`` inference server but lot of people like to use ``ollama``
ollama run hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0
Then in the app you should be able to talk to `ollama/hermes`
then::
export MODEL_ENDPOINT_1=https://localhost:11434/v1
Then in the app you should be able to talk to ``NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0``
Contributing

973
app.py
View file

File diff suppressed because it is too large Load diff

5
requirements.txt
View file

@ -5,11 +5,9 @@ flask-socketio
gevent
gevent-websocket
mistralai
together
openai
openai[datalib]
groq
tiktoken
#llama-cpp-python[server]
@ -18,6 +16,7 @@ tiktoken
Flask-SQLAlchemy
Flask-Migrate
# used for s3/spaces or aws bedrock (claude)
boto3
pyyaml

72
templates/base.html
View file

@ -283,36 +283,6 @@
<label for="model-select-mobile">Choose Model:</label>
<select id="model-select-mobile">
<option value="None">None</option>
<option value="vllm/r1">vllm/r1</option>
<option value="vllm/hermes">vllm/hermes</option>
<option value="vllm/qwq">vllm/qwq</option>
<option value="ollama/qwen-coder">ollama/qwen-coder</option>
<option value="ollama/deepseek-coder">ollama/deepseek-coder</option>
<option value="mistral-tiny">mistral-tiny</option>
<option value="mistral-small">mistral-small</option>
<option value="mistral-medium">mistral-medium</option>
<option value="mistral-large">mistral-large</option>
<option value="mistral-codestral">mistral-codestral</option>
<option value="gemini-flash">gemini-flash</option>
<option value="gemini-flash-8b">gemini-flash-8b</option>
<option value="gemini-pro">gemini-pro</option>
<option value="grok-beta">grok-beta</option>
<option value="gpt-4">gpt-4</option>
<option value="gpt-4o-2024-08-06">gpt-4o-2024-08-06</option>
<option value="gpt-mini">gpt-mini</option>
<option value="gpt-o1-mini">gpt-o1-mini</option>
<option value="gpt-o1-preview">gpt-o1-preview</option>
<option value="gpt-o1">gpt-o1</option>
<option value="gpt-o3-mini">gpt-o3-mini</option>
<!--
<option value="gpt-o3-mini-medium">gpt-o3-mini-medium</option>
<option value="gpt-o3-mini-high">gpt-o3-mini-high</option>
-->
<option value="claude-haiku">claude-haiku</option>
<option value="claude-sonnet">claude-sonnet</option>
<option value="claude-opus">claude-opus</option>
</select>
</div>
<div id="voice-chooser-mobile">
@ -348,7 +318,6 @@
</div>
</div>
<!-- Chatroom list -->
<div class="main-container">
<div id="rooms-list">
@ -421,6 +390,47 @@
// Add event listener to the hamburger button
document.getElementById("hamburger-button").addEventListener("click", openModal);
// Populate the mobile model dropdown dynamically
document.addEventListener('DOMContentLoaded', (event) => {
const modelSelectMobile = document.getElementById("model-select-mobile");
// Function to populate the dropdown
function populateModelDropdown(models) {
while (modelSelectMobile.options.length > 1) {
modelSelectMobile.remove(1);
}
models.forEach(modelId => {
const option = document.createElement('option');
option.value = modelId;
option.textContent = modelId;
modelSelectMobile.appendChild(option);
});
}
// Memoization with localStorage (1-minute cache)
const cacheKey = 'modelList';
const cacheExpirationKey = 'modelListExpiration';
const cacheDuration = 60 * 1000; // 1 minute in milliseconds
const cachedData = localStorage.getItem(cacheKey);
const cachedExpiration = localStorage.getItem(cacheExpirationKey);
if (cachedData && cachedExpiration && Date.now() < parseInt(cachedExpiration)) {
const models = JSON.parse(cachedData);
populateModelDropdown(models);
} else {
fetch('/models')
.then(response => response.json())
.then(data => {
const models = data.models;
populateModelDropdown(models);
localStorage.setItem(cacheKey, JSON.stringify(models));
localStorage.setItem(cacheExpirationKey, Date.now() + cacheDuration);
})
.catch(error => console.error("Error fetching models:", error));
}
});
</script>
</body>
</html>

99
templates/chat.html
View file

@ -24,37 +24,6 @@
<label for="model-select">Model</label>
<select id="model-select">
<option value="None">None</option>
<option value="vllm/hermes">vllm/hermes</option>
<option value="vllm/r1">vllm/r1</option>
<option value="vllm/qwq">vllm/qwq</option>
<option value="ollama/qwen-coder">ollama/qwen-coder</option>
<option value="ollama/deepseek-coder">ollama/deepseek-coder</option>
<option value="mistral-tiny">mistral-tiny</option>
<option value="mistral-small">mistral-small</option>
<option value="mistral-medium">mistral-medium</option>
<option value="mistral-large">mistral-large</option>
<option value="mistral-codestral">mistral-codestral</option>
<option value="gemini-flash">gemini-flash</option>
<option value="gemini-flash-8b">gemini-flash-8b</option>
<option value="gemini-pro">gemini-pro</option>
<option value="grok-beta">grok-beta</option>
<option value="gpt-4">gpt-4</option>
<option value="gpt-4o-2024-08-06">gpt-4o-2024-08-06</option>
<option value="gpt-mini">gpt-mini</option>
<option value="gpt-o1-mini">gpt-o1-mini</option>
<option value="gpt-o1-preview">gpt-o1-preview</option>
<option value="gpt-o1">gpt-o1</option>
<option value="gpt-o3-mini">gpt-o3-mini</option>
<!--
<option value="gpt-o3-mini-medium">gpt-o3-mini-medium</option>
<option value="gpt-o3-mini-high">gpt-o3-mini-high</option>
-->
<option value="claude-haiku">claude-haiku</option>
<option value="claude-sonnet">claude-sonnet</option>
<option value="claude-opus">claude-opus</option>
<option value="dall-e-3">dall-e-3</option>
</select>
</div>
<div>
@ -93,16 +62,8 @@ const urlParams = new URLSearchParams(window.location.search);
const username = urlParams.get("username");
const room_name = "{{ room_name }}";
// Global constants for valid voices and models
// Global constants for valid voices
const VALID_VOICES = ['alloy', 'echo', 'fable', 'onyx', 'nova', 'shimmer'];
const VALID_MODELS = [
'None', 'gemini-flash', 'gemini-flash-8b', 'gemini-pro', 'grok-beta',
'vllm/hermes', 'vllm/r1', 'vllm/qwq', 'ollama/qwen-coder', 'ollama/deepseek-coder',
'mistral-tiny', 'mistral-small', 'mistral-medium', 'mistral-large', 'mistral-codestral',
'gpt-4', 'gpt-4o-2024-08-06', 'gpt-mini', 'gpt-o1-mini', 'gpt-o1-preview', 'gpt-o1',
'gpt-o3-mini', 'gpt-o3-mini-medium', 'gpt-o3-mini-high',
'claude-haiku', 'claude-sonnet', 'claude-opus', 'dall-e-3'
];
// Configuration for DOMPurify to specify which tags and attributes are allowed
const dompurify_config = {
@ -122,7 +83,6 @@ let audioCache = {}; // Cache to store audio blobs
// Flag to prevent mutual updates on desktop/mobile
let isSyncingDropdowns = false;
// Function to sanitize the username
function sanitizeUsername(username) {
// Split the username on commas and take the first part.
@ -139,7 +99,7 @@ function syncDropdownsAndQueryString() {
const voiceSelectMobile = document.getElementById("voice-select-mobile");
// Determine the current model and voice from any dropdown
const currentModel = VALID_MODELS.includes(modelSelectDesktop.value) ? modelSelectDesktop.value : 'None';
const currentModel = modelSelectDesktop.value;
const currentVoice = VALID_VOICES.includes(voiceSelectDesktop.value) ? voiceSelectDesktop.value : 'onyx';
// Sync both desktop and mobile dropdowns
@ -162,6 +122,51 @@ document.addEventListener('DOMContentLoaded', (event) => {
const modelSelectMobile = document.getElementById("model-select-mobile");
const voiceSelectMobile = document.getElementById("voice-select-mobile");
// Function to populate the dropdown
function populateModelDropdown(models) {
// Clear options starting from index 1 (preserve "None" at index 0)
while (modelSelectDesktop.options.length > 1) {
modelSelectDesktop.remove(1);
}
// Append new model options
models.forEach(modelId => {
const option = document.createElement('option');
option.value = modelId;
option.textContent = modelId;
modelSelectDesktop.appendChild(option);
});
// Set initial value from URL
const urlParams = new URLSearchParams(window.location.search);
const initialModel = urlParams.get("model") || "None";
modelSelectDesktop.value = initialModel;
}
// Memoization with localStorage (1-minute cache)
const cacheKey = 'modelList';
const cacheExpirationKey = 'modelListExpiration';
const cacheDuration = 60 * 1000; // 1 minute in milliseconds
const cachedData = localStorage.getItem(cacheKey);
const cachedExpiration = localStorage.getItem(cacheExpirationKey);
if (cachedData && cachedExpiration && Date.now() < parseInt(cachedExpiration)) {
// Use cached data if it exists and hasn't expired
const models = JSON.parse(cachedData);
populateModelDropdown(models);
} else {
// Fetch from backend and update cache
fetch('/models')
.then(response => response.json())
.then(data => {
const models = data.models;
populateModelDropdown(models);
// Store in localStorage with expiration
localStorage.setItem(cacheKey, JSON.stringify(models));
localStorage.setItem(cacheExpirationKey, Date.now() + cacheDuration);
})
.catch(error => console.error("Error fetching models:", error));
}
chatContainer.addEventListener('scroll', () => {
const distanceFromBottom = chatContainer.scrollHeight - chatContainer.scrollTop - chatContainer.clientHeight;
userHasScrolledUp = distanceFromBottom > 5;
@ -272,10 +277,12 @@ function sendMessage() {
let messageToSend = message.trim();
if (messageToSend !== "") { // Ensure we're not sending empty messages
if (model !== "None") {
messageToSend = `${model} \n\n ${messageToSend}`;
}
socket.emit("chat_message", {"username": username, "message": messageToSend, "room_name": room_name});
socket.emit("chat_message", {
"username": username,
"message": messageToSend,
"model": model, // Pass model as a separate attribute
"room_name": room_name
});
document.getElementById("message").value = "";
}
}

38
vars.sh.sample Normal file
View file

@ -0,0 +1,38 @@
#!/bin/bash
# vars.sh: Example configuration for dynamic endpoints
# Official OpenAI (hermes) endpoint.
export MODEL_ENDPOINT_0="https://hermes.ai.unturf.com/v1"
export MODEL_API_KEY_0="your-hermes-api-key" # optional; if omitted, "not-needed" is used
# Naptha endpoints.
export MODEL_ENDPOINT_1="https://node2.naptha.ai/inference"
export MODEL_API_KEY_1="your-node2-api-key"
export MODEL_ENDPOINT_2="https://node3.naptha.ai/inference"
export MODEL_API_KEY_2="your-node3-api-key"
# Google Gemini endpoint.
export MODEL_ENDPOINT_3="https://generativelanguage.googleapis.com/v1beta/openai"
export MODEL_API_KEY_3="your-google-api-key"
# Grok endpoint.
export MODEL_ENDPOINT_4="https://api.x.ai/v1"
export MODEL_API_KEY_4=""
# Groq endpoint.
export MODEL_ENDPOINT_5="https://api.groq.com/openai/v1"
export MODEL_API_KEY_5="gone"
# Together endpoint.
export MODEL_ENDPOINT_6="https://api.together.xyz/v1"
export MODEL_API_KEY_6="gone"
# OpenAI endpoint.
export MODEL_ENDPOINT_7="https://api.openai.com/v1"
export MODEL_API_KEY_7="gone"
export OPENAI_API_KEY="gone"
# MistralAI La Platform endpoint.
export MODEL_ENDPOINT_8="https://api.mistral.ai/v1"
export MODEL_API_KEY_8="gone"