ollama Hermes

russellballestrini commented

2024-11-22 14:16:24 -05:00

(Migrated from git2.unturf.com)

Created by: russellballestrini

ollama run hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0

modified:   README.rst
modified:   app.py

Summary by CodeRabbit

New Features
- Updated usage instructions for language models, including new commands for vLLM and Ollama Hermes.
- Added a new model option for user interactions.
Documentation
- Enhanced clarity and comprehensiveness of the README file regarding model interactions.
Bug Fixes
- Improved handling of chat messages to accommodate new model identifiers.

*Created by: russellballestrini* ollama run hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0 modified: README.rst modified: app.py  ## Summary by CodeRabbit - **New Features** - Updated usage instructions for language models, including new commands for vLLM and Ollama Hermes. - Added a new model option for user interactions. - **Documentation** - Enhanced clarity and comprehensiveness of the README file regarding model interactions. - **Bug Fixes** - Improved handling of chat messages to accommodate new model identifiers.

russellballestrini commented

2024-11-22 14:16:33 -05:00

(Migrated from git2.unturf.com)

Created by: coderabbitai[bot]

Walkthrough

The pull request introduces updates to the README.rst and app.py files. In README.rst, the command for the vLLM Hermes model has been changed and a new command for the Ollama Hermes model has been added. Additionally, a new section comparing Ollama and vLLM has been included. In app.py, the integration of the gevent library replaces the previous eventlet implementation, and the model identifiers have been updated to reflect the changes in the README. The chat message handling logic has also been adjusted to accommodate these updates.

Changes

File	Change Summary
README.rst	- Updated command for vLLM Hermes from `vllm/hermes-llama-3` to `vllm/hermes`.
	- Added command for Ollama Hermes: `ollama/hermes`.
	- Expanded section on language model commands and added "Ollama versus vLLM" commentary.
app.py	- Integrated `gevent` for asynchronous operations, replacing `eventlet`.
	- Updated `system_users` to include new model identifier `"hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0"`.
	- Adjusted help messages to reflect new model commands.
	- Modified `get_openai_client_and_model` function to handle `ollama` models.
	- Updated chat message handling logic to accommodate new model identifiers.

#5: The changes in README.rst and app.py both involve updates to model commands and usage instructions, similar to the modifications made in the main PR regarding the vLLM and Ollama models.

Poem

🐇 In the land of code where rabbits play,
New commands have hopped in, hip-hip-hooray!
VLLM and Ollama, side by side,
With gevent's speed, we take a ride.
So send your prompts, let the models gleam,
In this coding warren, we live the dream! 🌟

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

*Created by: coderabbitai[bot]*   ## Walkthrough The pull request introduces updates to the `README.rst` and `app.py` files. In `README.rst`, the command for the vLLM Hermes model has been changed and a new command for the Ollama Hermes model has been added. Additionally, a new section comparing Ollama and vLLM has been included. In `app.py`, the integration of the `gevent` library replaces the previous `eventlet` implementation, and the model identifiers have been updated to reflect the changes in the README. The chat message handling logic has also been adjusted to accommodate these updates. ## Changes | File | Change Summary | |--------------|-----------------------------------------------------------------------------------------------------| | README.rst | - Updated command for vLLM Hermes from `vllm/hermes-llama-3` to `vllm/hermes`. | | | - Added command for Ollama Hermes: `ollama/hermes`. | | | - Expanded section on language model commands and added "Ollama versus vLLM" commentary. | | app.py | - Integrated `gevent` for asynchronous operations, replacing `eventlet`. | | | - Updated `system_users` to include new model identifier `"hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0"`. | | | - Adjusted help messages to reflect new model commands. | | | - Modified `get_openai_client_and_model` function to handle `ollama` models. | | | - Updated chat message handling logic to accommodate new model identifiers. | ## Possibly related PRs - #5: The changes in `README.rst` and `app.py` both involve updates to model commands and usage instructions, similar to the modifications made in the main PR regarding the vLLM and Ollama models. ## Poem > 🐇 In the land of code where rabbits play, > New commands have hopped in, hip-hip-hooray! > VLLM and Ollama, side by side, > With gevent's speed, we take a ride. > So send your prompts, let the models gleam, > In this coding warren, we live the dream! 🌟      --- Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? <details> <summary>❤️ Share</summary> - [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai) - [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai) - [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai) - [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code) </details> <details> <summary>🪧 Tips</summary> ### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit <commit_id>, please review it.` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (Invoked using PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. ### Other keywords and placeholders - Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. - Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description. - Add `@coderabbitai` anywhere in the PR title to generate the title automatically. ### CodeRabbit Configuration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit. - Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements. </details>

russellballestrini commented

2024-11-22 14:17:43 -05:00

(Migrated from git2.unturf.com)

Created by: coderabbitai[bot]

🛠️ Refactor suggestion

Add setup requirements for new models.

While the commands for vLLM and Ollama Hermes are documented, there's no mention of their setup requirements. Users would benefit from knowing:

How to set up and run the vLLM inference server
How to install and configure Ollama

Consider adding a new section under "Requirements" or "Installation" that covers these prerequisites.

*Created by: coderabbitai[bot]* _:hammer_and_wrench: Refactor suggestion_ **Add setup requirements for new models.** While the commands for vLLM and Ollama Hermes are documented, there's no mention of their setup requirements. Users would benefit from knowing: - How to set up and run the vLLM inference server - How to install and configure Ollama Consider adding a new section under "Requirements" or "Installation" that covers these prerequisites.

russellballestrini commented

2024-11-22 14:17:44 -05:00

(Migrated from git2.unturf.com)

mentioned in merge request !3

russellballestrini commented

2024-11-22 14:17:44 -05:00

(Migrated from git2.unturf.com)

Created by: coderabbitai[bot]

Review: Commented

Actionable comments posted: 1

🧹 Outside diff range and nitpick comments (2)

README.rst (1)

209-218: Enhance the Ollama vs vLLM comparison section.

The current section could be more helpful with the following improvements:

Provide reasoning for the vLLM preference (e.g., performance, features)

Add context around the Ollama command:

Required system resources

Expected download size

Model loading time

Clarify the app communication statement with a complete example

Consider this structure:
Ollama versus vLLM
-----------------------------

While both Ollama and vLLM are excellent choices for running local models, this project prefers the ``vllm`` inference server for its superior throughput and optimized implementation. However, Ollama offers a more user-friendly experience that many users appreciate.

Using Ollama:
~~~~~~~~~~~~

1. Install Ollama following instructions at https://ollama.ai
2. Run the Hermes model::

    ollama run hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0

   Note: This requires approximately 8GB of RAM and 4GB of disk space.

3. Once running, interact with the model in the chat using ``ollama/hermes``::

    ollama/hermes What is the capital of France?

app.py (1)

1032-1040: Consider improving Ollama integration robustness.

While the Ollama integration works, consider these improvements:

Make the Ollama endpoint URL configurable via environment variables
Add error handling for Ollama service availability

+# Add to top of file with other env vars
+OLLAMA_ENDPOINT = os.environ.get("OLLAMA_ENDPOINT", "http://127.0.0.1:11434/v1")

 def get_openai_client_and_model(model_name="NousResearch/Hermes-3-Llama-3.1-8B"):
     vllm_endpoint = os.environ.get("VLLM_ENDPOINT")
     vllm_api_key = os.environ.get("VLLM_API_KEY", "not-needed")
     xai_api_key = os.environ.get("XAI_API_KEY")
     google_api_key = os.environ.get("GOOGLE_API_KEY")

     is_openai_model = "gpt" in model_name.lower() or "o1" in model_name.lower()
     is_xai_model = "grok-" in model_name.lower()
     is_google_model = "gemini-" in model_name.lower()
     is_ollama_model = "hf.co" in model_name.lower()
     is_vllm_model = True

     if is_openai_model or is_xai_model or is_google_model or is_ollama_model:
         is_vllm_model = False

     if is_vllm_model:
         openai_client = OpenAI(base_url=vllm_endpoint, api_key=vllm_api_key)
     elif is_ollama_model:
-        openai_client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key=vllm_api_key)
+        try:
+            openai_client = OpenAI(base_url=OLLAMA_ENDPOINT, api_key=vllm_api_key)
+            # Test connection
+            openai_client.models.list()
+        except Exception as e:
+            raise RuntimeError(f"Failed to connect to Ollama service at {OLLAMA_ENDPOINT}: {e}")

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between db8749a2df and 185ebbcb59.

📒 Files selected for processing (2)

README.rst (2 hunks)
app.py (5 hunks)

🔇 Additional comments (3)

app.py (3)

84-84: LGTM: Model identifier addition is consistent.

The addition of "hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0" to the system users list aligns with the PR objectives.

130-131: LGTM: Help message updates are clear and consistent.

The help message updates appropriately document both:

The modified vLLM Hermes command
The new Ollama Hermes command

595-595: LGTM: Message handler correctly updated for Ollama support.

The addition of or "ollama/" in data["message"] properly extends message handling to support Ollama models.

*Created by: coderabbitai[bot]* **Review:** Commented **Actionable comments posted: 1** <details> <summary>🧹 Outside diff range and nitpick comments (2)</summary><blockquote> <details> <summary>README.rst (1)</summary><blockquote> `209-218`: **Enhance the Ollama vs vLLM comparison section.** The current section could be more helpful with the following improvements: 1. Provide reasoning for the vLLM preference (e.g., performance, features) 2. Add context around the Ollama command: - Required system resources - Expected download size - Model loading time 3. Clarify the app communication statement with a complete example Consider this structure: ```rst Ollama versus vLLM ----------------------------- While both Ollama and vLLM are excellent choices for running local models, this project prefers the ``vllm`` inference server for its superior throughput and optimized implementation. However, Ollama offers a more user-friendly experience that many users appreciate. Using Ollama: ~~~~~~~~~~~~ 1. Install Ollama following instructions at https://ollama.ai 2. Run the Hermes model:: ollama run hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0 Note: This requires approximately 8GB of RAM and 4GB of disk space. 3. Once running, interact with the model in the chat using ``ollama/hermes``:: ollama/hermes What is the capital of France? ``` </blockquote></details> <details> <summary>app.py (1)</summary><blockquote> `1032-1040`: **Consider improving Ollama integration robustness.** While the Ollama integration works, consider these improvements: 1. Make the Ollama endpoint URL configurable via environment variables 2. Add error handling for Ollama service availability ```diff +# Add to top of file with other env vars +OLLAMA_ENDPOINT = os.environ.get("OLLAMA_ENDPOINT", "http://127.0.0.1:11434/v1") def get_openai_client_and_model(model_name="NousResearch/Hermes-3-Llama-3.1-8B"): vllm_endpoint = os.environ.get("VLLM_ENDPOINT") vllm_api_key = os.environ.get("VLLM_API_KEY", "not-needed") xai_api_key = os.environ.get("XAI_API_KEY") google_api_key = os.environ.get("GOOGLE_API_KEY") is_openai_model = "gpt" in model_name.lower() or "o1" in model_name.lower() is_xai_model = "grok-" in model_name.lower() is_google_model = "gemini-" in model_name.lower() is_ollama_model = "hf.co" in model_name.lower() is_vllm_model = True if is_openai_model or is_xai_model or is_google_model or is_ollama_model: is_vllm_model = False if is_vllm_model: openai_client = OpenAI(base_url=vllm_endpoint, api_key=vllm_api_key) elif is_ollama_model: - openai_client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key=vllm_api_key) + try: + openai_client = OpenAI(base_url=OLLAMA_ENDPOINT, api_key=vllm_api_key) + # Test connection + openai_client.models.list() + except Exception as e: + raise RuntimeError(f"Failed to connect to Ollama service at {OLLAMA_ENDPOINT}: {e}") ``` </blockquote></details> </blockquote></details> <details> <summary>📜 Review details</summary> **Configuration used: CodeRabbit UI** **Review profile: CHILL** <details> <summary>📥 Commits</summary> Reviewing files that changed from the base of the PR and between db8749a2df36049f9241e210aacc114929df9e21 and 185ebbcb592cfae05336b0f4112d26adb16a1f6c. </details> <details> <summary>📒 Files selected for processing (2)</summary> * `README.rst` (2 hunks) * `app.py` (5 hunks) </details> <details> <summary>🔇 Additional comments (3)</summary> <details> <summary>app.py (3)</summary> `84-84`: **LGTM: Model identifier addition is consistent.** The addition of `"hf.co/NousResearch/Hermes-3-Llama-3.1-8B-GGUF:Q8_0"` to the system users list aligns with the PR objectives. --- `130-131`: **LGTM: Help message updates are clear and consistent.** The help message updates appropriately document both: - The modified vLLM Hermes command - The new Ollama Hermes command --- `595-595`: **LGTM: Message handler correctly updated for Ollama support.** The addition of `or "ollama/" in data["message"]` properly extends message handling to support Ollama models. </details> </details> </details>

russellballestrini commented

2024-11-22 14:18:56 -05:00

(Migrated from git2.unturf.com)

Merged by: russellballestrini at 2024-11-22 19:18:56 UTC

*Merged by: russellballestrini at 2024-11-22 19:18:56 UTC*

russellballestrini (Migrated from git2.unturf.com) closed this pull request

2024-11-22 14:18:56 -05:00

russellballestrini (Migrated from git2.unturf.com) merged commit into main

2024-11-22 14:18:56 -05:00

russellballestrini commented

2025-02-08 16:05:21 -05:00

(Migrated from git2.unturf.com)

mentioned in merge request !8

Rows
Columns

ollama Hermes #6

Summary by CodeRabbit

Walkthrough

Changes

Possibly related PRs

Poem

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

CodeRabbit Configuration File (`.coderabbit.yaml`)