We learn as we go, we write as we learn.

About michelada.io


By joining forces with your team, we can help with your Ruby on Rails and Javascript work. Or, if it works better for you, we can even become your team! From e-Commerce to Fintech, for years we have helped turning our clients’ great ideas into successful business.

Go to our website for more info.

Tags


Do we need background jobs in Elixir?

7th August 2020

In Ruby, we are used to implement background jobs through tools such as Sidekiq, or Resque. In this post, we will see the benefits of some OTP concurrency tools.

Hold your horses! OTP to the rescue

In the face of background processing as developers, our instinct tells us "search  'Sidekiq in Elixir'". This would lead us to find some queue systems implementations but, if we knew a little of what the OTP offers us, we would not look for third-party solution and implement a solution according to our needs.

One of the highlights of Elixir features that Ruby lacks is the concurrency. Although the community has generated various solutions around parallel processing, it is mostly applied to languages where concurrency does not exist and, in the case of background jobs, it is very common to have a separate process that runs in parallel and communicates through messages that travel through Redis, RabitMQ or even the same database.  Is all of this necessary in Elixir? How would we avoid this communication to external services?

Processes and tasks

According to the Elixir documentation

Processes are isolated from each other, run concurrent to one another and communicate via message passing. Processes are not only the basis for concurrency in Elixir, but they also provide the means for building distributed and fault-tolerant programs.

The above is the base of concurrency in Elixir: in Ruby, everything is an object; in Elixir, everything is a process or we can put everything in small processes, which opens the doors to the world of concurrency.

Suppose we want to have a process which we do not care whether it fails, i.e.  metrics. In the case of Ruby, we would try to make it take as little time as possible or send it to a background job. How would we do it in Elixir? by sending it to a different process:

spawn fn() -> Metrics.Purchase.complete end

That's it? Yes! It is really all we need in this case since the only thing that matters to us is that it does not block the main process.

Now, we could also do the same using tasks:

Task.start fn() -> Metrics.Purchase.complete end

So, what is the difference? Tasks are built on top of processes and provide better error handling as they return a tuple of values {:ok, pid}. If we want, we can implement a mechanism that gives us better control over the result.

GenServers

Both, the processes and the tasks do not persist the state once they finish. What if we want to store something at runtime? Shall we write it to disk? Do we save it in the database? Although it is true that in some cases it will be the most convenient,  we have another tool at our disposal: The GenServers or generic servers, which allow us to persist the state with a base structure that we can adapt to our needs.

defmodule Example.Registry do
  use GenServer

  # Client
  def start_link(_opts) do
    GenServer.start_link(__MODULE__, %{}, name: __MODULE__)
  end

  def add(key) do
    GenServer.cast(__MODULE__, {:add, key})
  end
  
  def find(key) do
    GenServer.call(__MODULE__, {:find, key})
  end

  # Server
  @impl true
  def init(_opts) do
    {:ok, %{}}
  end

  @impl true
  def handle_cast({:add, key}, state) do
    updated_state = Map.put(state, key, LongRunningProcess.run(key))
    {:noreply, updated_state}
  end

  @impl true
  def handle_call({:find, key}, _from, state) do
    value = Map.get(state, key)
    if is_nil(value) do
        {:reply, {:error, "Not found"}, state}
    else
        {:reply, {:ok, value}, state}
    end
  end
end
v

In this example, we will notice that there is much more code than we would normally use in a Ruby worker. If we analyze it in detail, we will notice that our server has a small client-server architecture. We would use our GenServer in the following way.

iex> Example.Registry.add("something")
iex> Example.Registry.find("something")
{:ok, 21wqeds24354trgfdq2365ytgfde23465}

Now, on the server side, we have an init for initialization and handlers. The function handle_cast allow us to make async calls and it returns a tuple {:noreply, state} , while handle_call handles sync calls by returning {:reply, value, state}.

The equivalent in Ruby would be to have a worker in the background job, a state stored in an external medium and a client that gives us access to it. Whereas here, with the use of a simple GenServer, we have an asynchronous client-server application in a few lines of code.

When should I use Background Jobs?

In most cases when we do not require a background job in Elixir, we could practically build everything we need using the OTP. When we want to have more control, i.e. persistence on disk, a workers pool, retries, among other things, there are several popular solutions:

  • Oban: a robust PostgreSQL-based queue backend process system. It has a UI in its pro version
  • Exq: a Redis-based Resque and Sidekiq compliant processing library. It has an open source UI exq_ui
  • verk: same as Exq. Is is compatible with Resque and Sidekiq job definitions

There are other options, however, those are projects that are not as popular and are partially abandoned. If you venture to use them, keep in mind that this can translate into technical debt.

Conclusion

Thanks to the OTP, Elixir has a wide range of tools that facilitate concurrent processing and that could avoid us including third-party libraries. In this post I only mentioned some of them, but it would be worthwhile to review them on your own.

Whole enchilada developer

View Comments