Sheharyar Naseer

Background Job Processing in Elixir with Que

Elixir and Erlang provide a simple yet powerful set of features for data sharing, reliability and fault-tolerance, commonly known together as OTP. These consist of modules and behaviors such as GenServers and Supervisors among many others, providing an easy and efficient way to design reliable background processes without too much work.

There already are excellent background job processing libraries for Elixir, such as Exq and Toniq, that make use of them to provide an interface for working with background job workers, to developers unfamiliar with OTP. But all of these libraries are either excessively complicated, have dependence on external programs and binaries such as Redis, or both.

Que

Why would you lock yourself in with an external service, when Erlang itself ships with an extremely fast, distributed real-time datastore, called Mnesia? So, after a few months of hard work and testing it in multiple Elixir applications in production, I am proud to announce Que. Que is an extemely easy to use background job processing library for Elixir, that’s backed by Mnesia. Taking inspiration from intuitive job processing libraries in Ruby, Que’s complete setup takes less than a minute, and you can start writing job workers with almost zero boilerplate or gibberish:

defmodule App.Workers.ImageConverter do
  use Que.Worker

  def perform(image) do
    ImageTool.save_resized_copy!(image, :thumbnail)
    ImageTool.save_resized_copy!(image, :medium)
  end
end

Adding support for concurrency is also easy-as-pie, one argument and you’re done:

use Que.Worker, concurrency: 4

Que makes use of powerful Elixir and OTP features, such as:

  • Pattern Matching
  • Worker Supervision
  • Job Success Callbacks
  • Failure Recovery and Callback
  • Concurrency
  • In-Memory and On-Disk storage

By default, Que automatically configures the application to store all jobs in memory, but you can persist them across application restarts using:

mix que.setup

Check out the project on Github and Hexdocs for more details.