Russell Ballestrini
edd0bffd68
new file: content/2024-08-30-comparing-elixir-and-phoenix-performance-with-openai-client.rst
189 lines
7 KiB
ReStructuredText
189 lines
7 KiB
ReStructuredText
Comparing Elixir and Phoenix Performance with a Community-Driven OpenAI Client
|
||
###############################################################################
|
||
|
||
:author: Russell Ballestrini
|
||
:slug: comparing-elixir-and-phoenix-performance-with-openai-client
|
||
:date: 2024-08-30 17:26
|
||
:tags: Code, elixir
|
||
:status: published
|
||
|
||
As a developer exploring the capabilities of Elixir and Phoenix, I recently conducted a performance experiment using a community-driven OpenAI client. The goal was to evaluate how well Elixir, with its powerful concurrency model, handles high-volume API requests. The results were impressive, demonstrating Elixir's efficiency and robustness in managing concurrent tasks across multiple CPU cores.
|
||
|
||
Background
|
||
----------
|
||
|
||
At `CourseMojo, we are scaling an AI assistant teacher for public schools <https://www.coursemojo.com/>`_ which provides real-time, intelligent responses to students' answers to open-ended questions. During our load testing, we identified that heavy CPU utilization was primarily due to calls to the OpenAI API. This prompted me to explore Elixir and Phoenix to see if they could offer a more efficient solution.
|
||
|
||
Project Setup
|
||
-------------
|
||
|
||
To set up the project, we used the following steps:
|
||
|
||
1. **Create a New Elixir Project**:
|
||
Run the following command to create a new Elixir project and navigate into the directory:
|
||
|
||
.. code-block:: bash
|
||
|
||
mix local.hex
|
||
mix archive.install hex phx_new
|
||
mix phx.new openai_experiment --no-ecto
|
||
cd openai_experiment
|
||
|
||
2. **Add Dependencies**:
|
||
Add the `ex_openai <https://github.com/dvcrn/ex_openai>`_ library to your ``mix.exs`` file:
|
||
|
||
.. code-block:: elixir
|
||
|
||
defp deps do
|
||
[
|
||
{:ex_openai, "~> 1.6"}
|
||
]
|
||
end
|
||
|
||
Run ``mix deps.get`` to fetch the dependencies.
|
||
|
||
3. **Configure the Client**:
|
||
Set up your configuration in ``config/config.exs``:
|
||
|
||
.. code-block:: elixir
|
||
|
||
import Config
|
||
|
||
config :ex_openai,
|
||
api_key: System.get_env("OPENAI_API_KEY")
|
||
|
||
# the ex_openai http client seems to use this.
|
||
config :hackney,
|
||
pool_size: 1800,
|
||
max_connections: 1800
|
||
|
||
4. **Increase File Descriptor Limit**:
|
||
To handle more than 1000 concurrent connections, increase the file descriptor limit:
|
||
|
||
.. code-block:: bash
|
||
|
||
ulimit -n 4096
|
||
|
||
5. **Implement the Concurrency Logic**:
|
||
We used ``Task.async_stream/3`` to manage concurrency, allowing us to process tasks efficiently across multiple CPU cores.
|
||
|
||
Elixir Script
|
||
-------------
|
||
|
||
Here is the Elixir script ``lib/openai_experiment.ex`` used for the experiment:
|
||
|
||
.. code-block:: elixir
|
||
|
||
defmodule OpenaiExperiment do
|
||
alias ExOpenAI.Chat
|
||
alias ExOpenAI.Components.ChatCompletionRequestUserMessage
|
||
|
||
@max_retries 4
|
||
@retry_delay 2000 # milliseconds
|
||
@concurrency_limit 500
|
||
|
||
def generate_chat_completion(index, attempt \\ 1) do
|
||
msgs = [
|
||
%ChatCompletionRequestUserMessage{role: :system, content: "You are a helpful assistant."},
|
||
%ChatCompletionRequestUserMessage{role: :user, content: "What is the number #{index}?"}
|
||
]
|
||
|
||
case Chat.create_chat_completion(msgs, "gpt-4o-mini") do
|
||
{:ok, response} ->
|
||
IO.inspect(response, label: "Response for number #{index}")
|
||
{:ok, index}
|
||
{:error, %HTTPoison.Error{reason: reason}} when attempt <= @max_retries ->
|
||
IO.puts("HTTP error for number #{index}: #{inspect(reason)}, attempt #{attempt}")
|
||
:timer.sleep(@retry_delay)
|
||
generate_chat_completion(index, attempt + 1)
|
||
{:error, reason} when attempt <= @max_retries ->
|
||
IO.puts("Retrying number #{index} due to #{inspect(reason)}, attempt #{attempt}")
|
||
:timer.sleep(@retry_delay)
|
||
generate_chat_completion(index, attempt + 1)
|
||
{:error, reason} ->
|
||
IO.inspect(reason, label: "Final error for number #{index}")
|
||
{:error, index, reason}
|
||
end
|
||
end
|
||
|
||
def execute_chats_concurrently do
|
||
{time, results} = :timer.tc(fn ->
|
||
1..1800
|
||
|> Task.async_stream(&generate_chat_completion/1, max_concurrency: @concurrency_limit, timeout: 30000)
|
||
|> Enum.to_list()
|
||
end)
|
||
|
||
summarize_results(results)
|
||
IO.puts("Total execution time: #{time / 1_000_000} seconds")
|
||
end
|
||
|
||
defp summarize_results(results) do
|
||
successes = Enum.count(results, fn
|
||
{:ok, {:ok, _index}} -> true
|
||
_ -> false
|
||
end)
|
||
|
||
failures = Enum.count(results, fn
|
||
{:ok, {:error, _index, _reason}} -> true
|
||
_ -> false
|
||
end)
|
||
|
||
IO.puts("Summary:")
|
||
IO.puts("Successful tasks: #{successes}")
|
||
IO.puts("Failed tasks: #{failures}")
|
||
end
|
||
end
|
||
|
||
# Run the test
|
||
#OpenaiExperiment.execute_chats_concurrently()
|
||
|
||
Compiling and Running the Example
|
||
---------------------------------
|
||
|
||
1. **Compile the Project**:
|
||
Ensure your project is compiled by running:
|
||
|
||
.. code-block:: bash
|
||
|
||
mix compile
|
||
|
||
2. **Start the Elixir Interactive Shell**:
|
||
Launch the interactive Elixir shell with your project loaded:
|
||
|
||
.. code-block:: bash
|
||
|
||
iex -S mix
|
||
|
||
3. **Run the Example**:
|
||
Once inside the interactive shell, execute the function to run the experiment:
|
||
|
||
.. code-block:: elixir
|
||
|
||
OpenaiExperiment.execute_chats_concurrently()
|
||
|
||
Results
|
||
-------
|
||
|
||
The results of the experiment were as follows:
|
||
|
||
- **Successful tasks**: 1800
|
||
- **Failed tasks**: 0
|
||
- **Total execution time**: 23.436011 seconds
|
||
|
||
Analysis
|
||
--------
|
||
|
||
The real time, which represents the total elapsed time, was significantly efficient for the Elixir script. Here are some key takeaways from the results:
|
||
|
||
1. **Concurrency Handling**: Elixir handled 1800 tasks efficiently, leveraging its lightweight process model with a concurrency limit of 500.
|
||
2. **Multi-Core Utilization**: Elixir utilized all 4 CPU cores, demonstrating its ability to efficiently distribute tasks across available resources.
|
||
3. **Built-in Retries**: The script included a retry mechanism to handle transient errors, ensuring robustness in task execution.
|
||
|
||
This experiment demonstrated that Elixir, with its concurrency model, is a strong contender for handling high-volume, concurrent API requests using a community-driven OpenAI client. Elixir's efficient concurrency handling makes it a superior choice for scenarios where performance and speed are critical.
|
||
|
||
If you’re working on a project that involves extensive use of the OpenAI API or any other high-volume API interactions, consider leveraging Elixir to maximize performance and efficiency. The results of this experiment highlight the potential gains in speed and responsiveness that can be achieved with the right choice of technology.
|
||
|
||
It's worth noting that the OpenAI client used in this experiment is community-driven, not official, which also impacts performance. Additionally, keep an eye on developments in Elixir and Phoenix, as they continue to evolve and improve.
|
||
|
||
This post was written with the help of GPT-4 using the Elixir `ex_openai <https://github.com/dvcrn/ex_openai>`_ library.
|
||
|