File Uploads in Elixir with Arc and Google Cloud Storage

October 18, 2019

Arc is one of the go-to libraries for file uploads and attachments in Elixir, most commonly used with Ecto. While there are plenty of documentation and guides on using it with local storage or AWS, there’s not enough detailed information available when you want to use Google Cloud Storage. This post is meant as a quick reference guide for using GCS with Arc.

Set up Arc for Local Storage

If you’ve already set up Arc (possibly using it with Local Storage), you can skip this section. If not, follow along to get started. First add the :arc_ecto dependency (if you don’t want Ecto support, just use :arc instead):

defp deps do
  [{:arc_ecto, "0.11.2"}]
end

Create a new Arc Definition file:

# lib/my_app/uploads/image_attachment.ex

defmodule MyApp.Uploads.ImageAttachment do
  use Arc.Definition
  use Arc.Ecto.Definition

  @versions [:original, :thumb]
  @extension_whitelist ~w[.jpg .jpeg .gif .png]

  def transform(:thumb, _scope) do
    {:convert, "-strip -thumbnail 100x100^ -gravity center -extent 100x100 -format png", :png}
  end

  def validate({file, _}) do
    file_extension = file.file_name |> Path.extname() |> String.downcase()
    Enum.member?(@extension_whitelist, file_extension)
  end

  def storage_dir(_version, {_file, scope}) do
    "uploads/images/#{scope.id}"
  end

  def filename(version, {_file, _scope}) do
    to_string(version)
  end
end

The ImageAttachment definition above is for image uploads only. For other file types, you can modify it according to your requirements. To use it with Ecto schemas, add the field (and the appropriate migration) referencing the type:

defmodule MyApp.User do
  use Ecto.Schema
  alias MyApp.Uploads.ImageAttachment

  schema "users" do
    field :name, :string
    field :avatar, ImageAttachment.Type

    # ...
  end

  def changeset(struct, attrs) do
    struct
    |> cast(attrs, [:name, ...])
    |> cast_attachments(attrs, [:avatar])
  end
end

A quick way to test uploads is to run this in IEx:

# Define a module to mock file uploads from local files
defmodule MyApp.Mock do
  def file_upload(path) do
    %Plug.Upload{
      path: Path.expand(path),
      filename: Path.basename(path),
      content_type: MIME.from_path(path),
    }
  end
end

# Try "uploading" a local file as a user avatar
avatar = Mock.file_upload("~/Pictures/some-image.png")

%User{}
|> User.changeset(%{name: "Shey", avatar: avatar})
|> Repo.insert()

See Arc.Ecto for more details. Once you have uploads correctly working for local storage, we can then move on to GCS support.

Configure Google Cloud Storage

We need to set up Google Cloud Storage before we can start uploading files to it:

Create a Project in Google Cloud Console (if you haven’t already)
Create a new Bucket in Cloud Storage.
- Using something like {myapp}-{env}-uploads for the bucket id is a good idea.
- Choose your bucket’s region and storage class. Most common and recommended settings are “Multi-region” bucket with “Standard” storage class, but this might be different depending on your use-cases (make sure you understand pricing too).
- For access control, “Fine-grained” is recommended which gives you best of both worlds, but again depends on your requirements. (You can skip advanced settings).
Chances are that you want your uploaded files to be publicly accessible, so you should update the bucket permissions to reflect that:
- After your bucket is created, click on “Permissions” tab
- Click on the “Add members” button to show the right sidebar
- In “New members”, add allUsers and under roles, select “Storage Object Viewer”, and save
We also need to give your app a way to upload files to GCS. For that we need to create a Service Account with the correct permissions:
- Go to IAM > Service Accounts in the console
- Create a new Service account and give it an easily identifiable name and id
- Under roles, add “Storage Object Creator”
- You can skip “User Access”, but before finishing select “Create Key” and download it in JSON format (you can also always download it later)

I know, that was a lot of work to just upload some files to Google Cloud, but the good part is it’s over and we can get back to our Elixir app.

Enable GCS Support in Arc

First, we need to add the dependency for GCS storage provider in mix.exs:

defp deps do
  [
    {:arc_ecto, "0.11.2"},
    {:arc_gcs, "0.1.2"},
    # ...
  ]
end

Add the previously downloaded Service Account Key to your app, preferably at config/secrets/{env}/gcp.json. Important Note: You should absolutely not commit this file to Git (or any other version control). Make sure you add it to your .gitignore.

Now, update the Arc config to use GCS for file uploads with the service account key:

config :arc,
  storage: Arc.Storage.GCS,
  bucket: "myapp-dev-uploads"

config :goth,
  json: File.read!("config/secrets/dev/gcp.json")

Finally, update your ImageAttachment definition to send MIME (and any other custom) headers to GCP when uploading to identify the file:

defmodule MyApp.Uploads.ImageAttachment do
  use Arc.Definition
  use Arc.Ecto.Definition

  @versions [:original, :thumb]
  @extension_whitelist ~w[.jpg .jpeg .gif .png]

  def transform(:thumb, _scope) do
    {:convert, "-strip -thumbnail 100x100^ -gravity center -extent 100x100 -format png", :png}
  end

  def validate({file, _scope}) do
    file_extension = file.file_name |> Path.extname() |> String.downcase()
    Enum.member?(@extension_whitelist, file_extension)
  end

  def storage_dir(_version, {_file, scope}) do
    "uploads/images/#{scope.id}"
  end

  def filename(version, {_file, _scope}) do
    to_string(version)
  end

  # Add this:

  def gcs_object_headers(_version, {file, _scope}) do
    [content_type: MIME.from_path(file.file_name)]
  end
end

Try uploading a file again:

avatar = Mock.file_upload("~/Pictures/some-image.png")

user =
  %User{}
  |> User.changeset(%{name: "Shey", avatar: avatar})
  |> Repo.insert()

ImageAttachment.url({user.avatar, user})
# => "https://storage.googleapis.com/myapp-dev-uploads/....."

The above will return a public link to your uploaded image. You can also see all uploaded files in GCP Storage Browser.

This guide doesn’t cover the frontend aspect because assets can be uploaded via a variety of methods, which have already been covered extensively in the Arc docs and other places on the internet. Also, while the ImageAttachment definition above does cover the majority of cases, for other use-cases or for fine-grained configuration, you should again refer to the official Arc Github repo.