Merge pull request #10 from russellballestrini/dynamic-models
dynamic models for all the platforms!
This commit is contained in:
GitHub
commit
54223fc4bb
7 changed files with 244 additions and 1010 deletions
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -6,3 +6,4 @@ __pycache__/
|
|||
.flaskenv-exported
|
||||
.aws-sam/
|
||||
samconfig.toml
|
||||
vars.sh
|
||||
|
|
|
|||
66
README.rst
66
README.rst
|
|
@ -39,9 +39,6 @@ Requirements
|
|||
- eventlet or gevent
|
||||
- boto3 (for interacting with AWS Bedrock currently Claude, and S3 access)
|
||||
- openai (for interacting with OpenAI's language models)
|
||||
- mistralai (for interacting with MistralAI's language models)
|
||||
- together (for interacting with together.ai language models)
|
||||
- groq (for interacting with very fast groq language models)
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
|
@ -72,27 +69,21 @@ To set up the project, follow these steps:
|
|||
Usage
|
||||
-----
|
||||
|
||||
Set up environment variables for your AWS, OpenAI, MistralAI, together.ai, grok, groq, google, API keys.
|
||||
|
||||
* make a copy of ``vars.sh.sample`` and fill in your API keys!
|
||||
|
||||
Set up optional environment variables for your AWS, OpenAI, MistralAI, or together.ai API keys::
|
||||
Other env vars::
|
||||
|
||||
export AWS_ACCESS_KEY_ID="your_access_key"
|
||||
export AWS_SECRET_ACCESS_KEY="your_secret_key"
|
||||
export S3_BUCKET_NAME="your_s3_bucket_name"
|
||||
export OPENAI_API_KEY="your_openai_api_key"
|
||||
export MISTRAL_API_KEY="your_mistralai_api_key"
|
||||
export TOGETHER_API_KEY="your_togetherai_api_key"
|
||||
export GROQ_API_KEY="your_groq_api_key"
|
||||
export XAI_API_KEY="your_twitter_x_ai_api_key_for_grok"
|
||||
export GOOGLE_API_KEY="your_google_gemini_api_key"
|
||||
export VLLM_API_KEY="not-needed"
|
||||
export VLLM_ENDPOINT="http://localhost:18888/v1"
|
||||
|
||||
Here are some free endpoint for research only!::
|
||||
|
||||
export VLLM_ENDPOINT_1=https://hermes.ai.unturf.com/v1
|
||||
export VLLM_ENDPOINT_2=https://node2.naptha.ai/inference
|
||||
export VLLM_ENDPOINT_3=https://node3.naptha.ai/inference
|
||||
export MODEL_ENDPOINT_1=https://hermes.ai.unturf.com/v1
|
||||
export MODEL_ENDPOINT_2=https://node2.naptha.ai/inference
|
||||
export MODEL_ENDPOINT_3=https://node3.naptha.ai/inference
|
||||
|
||||
To start the application with socket.io run::
|
||||
|
||||
|
|
@ -114,44 +105,7 @@ The application will be available at ``http://127.0.0.1:5001`` by default.
|
|||
Interacting with Language Models
|
||||
--------------------------------
|
||||
|
||||
To interact with the various language models, you can use the following commands within the chat:
|
||||
|
||||
- For GPT-3, send a message with ``gpt-3`` and include your prompt.
|
||||
- For GPT-4o, send a message with ``gpt-4`` and include your prompt.
|
||||
- For GPT-4o cheapest version, send a message with ``gpt-4o-2024-08-06`` and include your prompt.
|
||||
- For GPT-4o-mini, send a message with ``gpt-mini`` and include your prompt.
|
||||
- For GPT-o1-mini, send a message with ``gpt-o1-mini`` and include your prompt.
|
||||
- For GPT-o1-preview, send a message with ``gpt-o1-preview`` and include your prompt.
|
||||
- For GPT-o1, send a message with ``gpt-o1`` and include your prompt.
|
||||
- For GPT-o3-mini, send a message with ``gpt-o3-mini`` and include your prompt.
|
||||
..
|
||||
- For GPT-o3-mini-medium, send a message with ``gpt-o3-mini-medium`` and include your prompt.
|
||||
- For GPT-o3-mini-high, send a message with ``gpt-o3-mini-high`` and include your prompt
|
||||
- For Claude-haiku, send a message with ``claude-haiku`` and include your prompt.
|
||||
- For Claude-sonnet, send a message with ``claude-sonnet`` and include your prompt.
|
||||
- For Claude-opus, send a message with ``claude-opus`` and include your prompt.
|
||||
- For Mistral-tiny, send a message with ``mistral-tiny`` and include your prompt.
|
||||
- For Mistral-small, send a message with ``mistral-small`` and include your prompt.
|
||||
- For Mistral-medium, send a message with ``mistral-medium`` and include your prompt.
|
||||
- For Mistral-large, send a message with ``mistral-large`` and include your prompt.
|
||||
- For Mistral-nemo, send a message with ``mistral-nemo`` and include your prompt.
|
||||
- For Mistral-codestral, send a message with ``mistral-codestral`` and include your prompt.
|
||||
- For Together OpenChat, send a message with ``together/openchat`` and include your prompt.
|
||||
- For Together Mistral, send a message with ``together/mistral`` and include your prompt.
|
||||
- For Together Mixtral, send a message with ``together/mixtral`` and include your prompt.
|
||||
- For Together Solar, send a message with ``together/solar`` and include your prompt.
|
||||
- For Groq Mixtral, send a message with ``groq/mixtral`` and include your prompt.
|
||||
- For Groq Llama-2, send a message with ``groq/llama2`` and include your prompt.
|
||||
- For Groq Llama-3, send a message with ``groq/llama3`` and include your prompt.
|
||||
- For Groq Gemma, send a message with ``groq/gemma`` and include your prompt.
|
||||
- For Google Gemini Flash, send a message with ``gemini-flash`` and include your prompt.
|
||||
- For Google Gemini Flash 8B, send a message with ``gemini-flash-8b`` and include your prompt.
|
||||
- For Google Gemini Pro, send a message with ``gemini-pro`` and include your prompt.
|
||||
- For Twitter/X AI Grok, send a message with ``grok-beta`` and include your prompt.
|
||||
- For vLLM Hermes, send a message with ``vllm/hermes`` and include your prompt.
|
||||
- For vLLM Deepseek R1 32B, send a message with ``vllm/r1`` and include your prompt.
|
||||
- For Ollama Hermes, send a message with ``ollama/hermes`` and include your prompt.
|
||||
- For Dall-e-3, send a message with ``dall-e-3`` and include your prompt.
|
||||
To interact with the various language models, choose from the drop down and send a message!
|
||||
|
||||
The system will process your message and provide a response from the selected language model.
|
||||
|
||||
|
|
@ -230,7 +184,11 @@ I prefer the ``vllm`` inference server but lot of people like to use ``ollama``
|
|||
|
||||
ollama run hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0
|
||||
|
||||
Then in the app you should be able to talk to `ollama/hermes`
|
||||
then::
|
||||
|
||||
export MODEL_ENDPOINT_1=https://localhost:11434/v1
|
||||
|
||||
Then in the app you should be able to talk to ``NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0``
|
||||
|
||||
|
||||
Contributing
|
||||
|
|
|
|||
|
|
@ -5,11 +5,9 @@ flask-socketio
|
|||
gevent
|
||||
gevent-websocket
|
||||
|
||||
mistralai
|
||||
together
|
||||
openai
|
||||
openai[datalib]
|
||||
groq
|
||||
|
||||
tiktoken
|
||||
|
||||
#llama-cpp-python[server]
|
||||
|
|
@ -18,6 +16,7 @@ tiktoken
|
|||
Flask-SQLAlchemy
|
||||
Flask-Migrate
|
||||
|
||||
# used for s3/spaces or aws bedrock (claude)
|
||||
boto3
|
||||
|
||||
pyyaml
|
||||
|
|
|
|||
|
|
@ -283,36 +283,6 @@
|
|||
<label for="model-select-mobile">Choose Model:</label>
|
||||
<select id="model-select-mobile">
|
||||
<option value="None">None</option>
|
||||
<option value="vllm/r1">vllm/r1</option>
|
||||
<option value="vllm/hermes">vllm/hermes</option>
|
||||
<option value="vllm/qwq">vllm/qwq</option>
|
||||
<option value="ollama/qwen-coder">ollama/qwen-coder</option>
|
||||
<option value="ollama/deepseek-coder">ollama/deepseek-coder</option>
|
||||
|
||||
<option value="mistral-tiny">mistral-tiny</option>
|
||||
<option value="mistral-small">mistral-small</option>
|
||||
<option value="mistral-medium">mistral-medium</option>
|
||||
<option value="mistral-large">mistral-large</option>
|
||||
<option value="mistral-codestral">mistral-codestral</option>
|
||||
|
||||
<option value="gemini-flash">gemini-flash</option>
|
||||
<option value="gemini-flash-8b">gemini-flash-8b</option>
|
||||
<option value="gemini-pro">gemini-pro</option>
|
||||
<option value="grok-beta">grok-beta</option>
|
||||
<option value="gpt-4">gpt-4</option>
|
||||
<option value="gpt-4o-2024-08-06">gpt-4o-2024-08-06</option>
|
||||
<option value="gpt-mini">gpt-mini</option>
|
||||
<option value="gpt-o1-mini">gpt-o1-mini</option>
|
||||
<option value="gpt-o1-preview">gpt-o1-preview</option>
|
||||
<option value="gpt-o1">gpt-o1</option>
|
||||
<option value="gpt-o3-mini">gpt-o3-mini</option>
|
||||
<!--
|
||||
<option value="gpt-o3-mini-medium">gpt-o3-mini-medium</option>
|
||||
<option value="gpt-o3-mini-high">gpt-o3-mini-high</option>
|
||||
-->
|
||||
<option value="claude-haiku">claude-haiku</option>
|
||||
<option value="claude-sonnet">claude-sonnet</option>
|
||||
<option value="claude-opus">claude-opus</option>
|
||||
</select>
|
||||
</div>
|
||||
<div id="voice-chooser-mobile">
|
||||
|
|
@ -348,7 +318,6 @@
|
|||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<!-- Chatroom list -->
|
||||
<div class="main-container">
|
||||
<div id="rooms-list">
|
||||
|
|
@ -421,6 +390,47 @@
|
|||
|
||||
// Add event listener to the hamburger button
|
||||
document.getElementById("hamburger-button").addEventListener("click", openModal);
|
||||
|
||||
// Populate the mobile model dropdown dynamically
|
||||
document.addEventListener('DOMContentLoaded', (event) => {
|
||||
const modelSelectMobile = document.getElementById("model-select-mobile");
|
||||
|
||||
// Function to populate the dropdown
|
||||
function populateModelDropdown(models) {
|
||||
while (modelSelectMobile.options.length > 1) {
|
||||
modelSelectMobile.remove(1);
|
||||
}
|
||||
models.forEach(modelId => {
|
||||
const option = document.createElement('option');
|
||||
option.value = modelId;
|
||||
option.textContent = modelId;
|
||||
modelSelectMobile.appendChild(option);
|
||||
});
|
||||
}
|
||||
|
||||
// Memoization with localStorage (1-minute cache)
|
||||
const cacheKey = 'modelList';
|
||||
const cacheExpirationKey = 'modelListExpiration';
|
||||
const cacheDuration = 60 * 1000; // 1 minute in milliseconds
|
||||
|
||||
const cachedData = localStorage.getItem(cacheKey);
|
||||
const cachedExpiration = localStorage.getItem(cacheExpirationKey);
|
||||
|
||||
if (cachedData && cachedExpiration && Date.now() < parseInt(cachedExpiration)) {
|
||||
const models = JSON.parse(cachedData);
|
||||
populateModelDropdown(models);
|
||||
} else {
|
||||
fetch('/models')
|
||||
.then(response => response.json())
|
||||
.then(data => {
|
||||
const models = data.models;
|
||||
populateModelDropdown(models);
|
||||
localStorage.setItem(cacheKey, JSON.stringify(models));
|
||||
localStorage.setItem(cacheExpirationKey, Date.now() + cacheDuration);
|
||||
})
|
||||
.catch(error => console.error("Error fetching models:", error));
|
||||
}
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
|
|||
|
|
@ -24,37 +24,6 @@
|
|||
<label for="model-select">Model</label>
|
||||
<select id="model-select">
|
||||
<option value="None">None</option>
|
||||
<option value="vllm/hermes">vllm/hermes</option>
|
||||
<option value="vllm/r1">vllm/r1</option>
|
||||
<option value="vllm/qwq">vllm/qwq</option>
|
||||
<option value="ollama/qwen-coder">ollama/qwen-coder</option>
|
||||
<option value="ollama/deepseek-coder">ollama/deepseek-coder</option>
|
||||
|
||||
<option value="mistral-tiny">mistral-tiny</option>
|
||||
<option value="mistral-small">mistral-small</option>
|
||||
<option value="mistral-medium">mistral-medium</option>
|
||||
<option value="mistral-large">mistral-large</option>
|
||||
<option value="mistral-codestral">mistral-codestral</option>
|
||||
|
||||
<option value="gemini-flash">gemini-flash</option>
|
||||
<option value="gemini-flash-8b">gemini-flash-8b</option>
|
||||
<option value="gemini-pro">gemini-pro</option>
|
||||
<option value="grok-beta">grok-beta</option>
|
||||
<option value="gpt-4">gpt-4</option>
|
||||
<option value="gpt-4o-2024-08-06">gpt-4o-2024-08-06</option>
|
||||
<option value="gpt-mini">gpt-mini</option>
|
||||
<option value="gpt-o1-mini">gpt-o1-mini</option>
|
||||
<option value="gpt-o1-preview">gpt-o1-preview</option>
|
||||
<option value="gpt-o1">gpt-o1</option>
|
||||
<option value="gpt-o3-mini">gpt-o3-mini</option>
|
||||
<!--
|
||||
<option value="gpt-o3-mini-medium">gpt-o3-mini-medium</option>
|
||||
<option value="gpt-o3-mini-high">gpt-o3-mini-high</option>
|
||||
-->
|
||||
<option value="claude-haiku">claude-haiku</option>
|
||||
<option value="claude-sonnet">claude-sonnet</option>
|
||||
<option value="claude-opus">claude-opus</option>
|
||||
<option value="dall-e-3">dall-e-3</option>
|
||||
</select>
|
||||
</div>
|
||||
<div>
|
||||
|
|
@ -93,16 +62,8 @@ const urlParams = new URLSearchParams(window.location.search);
|
|||
const username = urlParams.get("username");
|
||||
const room_name = "{{ room_name }}";
|
||||
|
||||
// Global constants for valid voices and models
|
||||
// Global constants for valid voices
|
||||
const VALID_VOICES = ['alloy', 'echo', 'fable', 'onyx', 'nova', 'shimmer'];
|
||||
const VALID_MODELS = [
|
||||
'None', 'gemini-flash', 'gemini-flash-8b', 'gemini-pro', 'grok-beta',
|
||||
'vllm/hermes', 'vllm/r1', 'vllm/qwq', 'ollama/qwen-coder', 'ollama/deepseek-coder',
|
||||
'mistral-tiny', 'mistral-small', 'mistral-medium', 'mistral-large', 'mistral-codestral',
|
||||
'gpt-4', 'gpt-4o-2024-08-06', 'gpt-mini', 'gpt-o1-mini', 'gpt-o1-preview', 'gpt-o1',
|
||||
'gpt-o3-mini', 'gpt-o3-mini-medium', 'gpt-o3-mini-high',
|
||||
'claude-haiku', 'claude-sonnet', 'claude-opus', 'dall-e-3'
|
||||
];
|
||||
|
||||
// Configuration for DOMPurify to specify which tags and attributes are allowed
|
||||
const dompurify_config = {
|
||||
|
|
@ -122,7 +83,6 @@ let audioCache = {}; // Cache to store audio blobs
|
|||
// Flag to prevent mutual updates on desktop/mobile
|
||||
let isSyncingDropdowns = false;
|
||||
|
||||
|
||||
// Function to sanitize the username
|
||||
function sanitizeUsername(username) {
|
||||
// Split the username on commas and take the first part.
|
||||
|
|
@ -139,7 +99,7 @@ function syncDropdownsAndQueryString() {
|
|||
const voiceSelectMobile = document.getElementById("voice-select-mobile");
|
||||
|
||||
// Determine the current model and voice from any dropdown
|
||||
const currentModel = VALID_MODELS.includes(modelSelectDesktop.value) ? modelSelectDesktop.value : 'None';
|
||||
const currentModel = modelSelectDesktop.value;
|
||||
const currentVoice = VALID_VOICES.includes(voiceSelectDesktop.value) ? voiceSelectDesktop.value : 'onyx';
|
||||
|
||||
// Sync both desktop and mobile dropdowns
|
||||
|
|
@ -162,6 +122,51 @@ document.addEventListener('DOMContentLoaded', (event) => {
|
|||
const modelSelectMobile = document.getElementById("model-select-mobile");
|
||||
const voiceSelectMobile = document.getElementById("voice-select-mobile");
|
||||
|
||||
// Function to populate the dropdown
|
||||
function populateModelDropdown(models) {
|
||||
// Clear options starting from index 1 (preserve "None" at index 0)
|
||||
while (modelSelectDesktop.options.length > 1) {
|
||||
modelSelectDesktop.remove(1);
|
||||
}
|
||||
// Append new model options
|
||||
models.forEach(modelId => {
|
||||
const option = document.createElement('option');
|
||||
option.value = modelId;
|
||||
option.textContent = modelId;
|
||||
modelSelectDesktop.appendChild(option);
|
||||
});
|
||||
// Set initial value from URL
|
||||
const urlParams = new URLSearchParams(window.location.search);
|
||||
const initialModel = urlParams.get("model") || "None";
|
||||
modelSelectDesktop.value = initialModel;
|
||||
}
|
||||
|
||||
// Memoization with localStorage (1-minute cache)
|
||||
const cacheKey = 'modelList';
|
||||
const cacheExpirationKey = 'modelListExpiration';
|
||||
const cacheDuration = 60 * 1000; // 1 minute in milliseconds
|
||||
|
||||
const cachedData = localStorage.getItem(cacheKey);
|
||||
const cachedExpiration = localStorage.getItem(cacheExpirationKey);
|
||||
|
||||
if (cachedData && cachedExpiration && Date.now() < parseInt(cachedExpiration)) {
|
||||
// Use cached data if it exists and hasn't expired
|
||||
const models = JSON.parse(cachedData);
|
||||
populateModelDropdown(models);
|
||||
} else {
|
||||
// Fetch from backend and update cache
|
||||
fetch('/models')
|
||||
.then(response => response.json())
|
||||
.then(data => {
|
||||
const models = data.models;
|
||||
populateModelDropdown(models);
|
||||
// Store in localStorage with expiration
|
||||
localStorage.setItem(cacheKey, JSON.stringify(models));
|
||||
localStorage.setItem(cacheExpirationKey, Date.now() + cacheDuration);
|
||||
})
|
||||
.catch(error => console.error("Error fetching models:", error));
|
||||
}
|
||||
|
||||
chatContainer.addEventListener('scroll', () => {
|
||||
const distanceFromBottom = chatContainer.scrollHeight - chatContainer.scrollTop - chatContainer.clientHeight;
|
||||
userHasScrolledUp = distanceFromBottom > 5;
|
||||
|
|
@ -272,10 +277,12 @@ function sendMessage() {
|
|||
let messageToSend = message.trim();
|
||||
|
||||
if (messageToSend !== "") { // Ensure we're not sending empty messages
|
||||
if (model !== "None") {
|
||||
messageToSend = `${model} \n\n ${messageToSend}`;
|
||||
}
|
||||
socket.emit("chat_message", {"username": username, "message": messageToSend, "room_name": room_name});
|
||||
socket.emit("chat_message", {
|
||||
"username": username,
|
||||
"message": messageToSend,
|
||||
"model": model, // Pass model as a separate attribute
|
||||
"room_name": room_name
|
||||
});
|
||||
document.getElementById("message").value = "";
|
||||
}
|
||||
}
|
||||
|
|
|
|||
38
vars.sh.sample
Normal file
38
vars.sh.sample
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
#!/bin/bash
|
||||
# vars.sh: Example configuration for dynamic endpoints
|
||||
|
||||
# Official OpenAI (hermes) endpoint.
|
||||
export MODEL_ENDPOINT_0="https://hermes.ai.unturf.com/v1"
|
||||
export MODEL_API_KEY_0="your-hermes-api-key" # optional; if omitted, "not-needed" is used
|
||||
|
||||
# Naptha endpoints.
|
||||
export MODEL_ENDPOINT_1="https://node2.naptha.ai/inference"
|
||||
export MODEL_API_KEY_1="your-node2-api-key"
|
||||
|
||||
export MODEL_ENDPOINT_2="https://node3.naptha.ai/inference"
|
||||
export MODEL_API_KEY_2="your-node3-api-key"
|
||||
|
||||
# Google Gemini endpoint.
|
||||
export MODEL_ENDPOINT_3="https://generativelanguage.googleapis.com/v1beta/openai"
|
||||
export MODEL_API_KEY_3="your-google-api-key"
|
||||
|
||||
# Grok endpoint.
|
||||
export MODEL_ENDPOINT_4="https://api.x.ai/v1"
|
||||
export MODEL_API_KEY_4=""
|
||||
|
||||
# Groq endpoint.
|
||||
export MODEL_ENDPOINT_5="https://api.groq.com/openai/v1"
|
||||
export MODEL_API_KEY_5="gone"
|
||||
|
||||
# Together endpoint.
|
||||
export MODEL_ENDPOINT_6="https://api.together.xyz/v1"
|
||||
export MODEL_API_KEY_6="gone"
|
||||
|
||||
# OpenAI endpoint.
|
||||
export MODEL_ENDPOINT_7="https://api.openai.com/v1"
|
||||
export MODEL_API_KEY_7="gone"
|
||||
export OPENAI_API_KEY="gone"
|
||||
|
||||
# MistralAI La Platform endpoint.
|
||||
export MODEL_ENDPOINT_8="https://api.mistral.ai/v1"
|
||||
export MODEL_API_KEY_8="gone"
|
||||
Loading…
Add table
Add a link
Reference in a new issue