WIP: Add file uploads with configurable storage

[wout]

Purpose

This is the first part of the Lucky::Attachment implementation (#1995). Test pass and it works inside the app I'm building, but it's not yet the complete first version of what I intend to create for the first phase. My goal is to have this ready by the end of next weekend.

Description

It's quite a lot of code already, so I want to put it out here for everyone to see and comment. I'll chop the description up in sections for easier reading.

Setting up models and operations

This is something I immediately have a question about. There are three modules for Avram:

• Avram::Attachment::Model
• Avram::Attachment::SaveOperation
• Avram::Attachment::DeleteOperation

By including the Avram::Attachment::Model in a model, the other two get included in the operations automatically. This could be either included by default, or manually if the user decides to use attachments in models.

My question is: where does this code live? It depends on the Lucky::Attachment Habitat configuration and some classes as well, so it can't be used without Lucky. Should it live in the lucky repo and be renamed? Or should it live in the Avram repo?

Setting up an attachment

class User < BaseModel
  include Avram::Attachment::Model

  table do
    # ...
    attach avatar : AvatarUploader?
    # ...
  end
end

Note

The underling column here is nilable, but it can also not be.

The in the operation:

class User::SaveAccount < User::SaveOperation
  attach avatar
  # ...
end

Important

There's no type declaration here, just the reference to the column in the model. If it does not exist on the model, there will be a compile-time error.

This will set up a file_attribute called avatar_file which can be used in the forms. If the value is nilable, there will also be a delete_avatar attribute which takes a boolean.

Setting up the uploader

struct AvatarUploader < Lucky::Attachment::Uploader
end

Here users can override a number of methods to customize behaviour. This is also where variants and other features will be configured.

Setting up the stores

Lucky::Attachment.configure do |settings|
  if LuckyEnv.production?
    settings.storages["cache"] = Lucky::Attachment::Storage::FileSystem.new(
      directory: "uploads",
      prefix: "cache"
    )
    settings.storages["store"] = Lucky::Attachment::Storage::FileSystem.new(
      directory: "uploads"
    )
  elsif LuckyEnv.development?
    settings.storages["cache"] = Lucky::Attachment::Storage::FileSystem.new(
      directory: "tmp/uploads",
      prefix: "cache"
    )
    settings.storages["store"] = Lucky::Attachment::Storage::FileSystem.new(
      directory: "tmp/uploads"
    )
  else
    settings.storages["cache"] = Lucky::Attachment::Storage::Memory.new
    settings.storages["store"] = Lucky::Attachment::Storage::Memory.new
  end
end

Currently I only implemented memory and file_system, but next weekend I'll add s3 and compatible.

Rendering attachments

In a form, so you always get a fresh version:

      file_input op.avatar_file
      if avatar = op.avatar.value
        img src: avatar.url
        checkbox op.delete_avatar
      end

Elsewhere:

      if avatar = current_user.avatar
        img src: avatar.url
      end

The flow

Attachments are uploaded to the cache first in a before_save callback. When validation passes, the file is moved from the cache to the store in an after_commit callback.

If validation fails, the user does not have to select the file again. The reference to the cache will be re-submitted and the file can be rendered because it is already in the cache store.

If a record with attachments is deleted, the delete operation will automatically trigger deletion of the files in a after_delete block. So there's no manual configuration required to clean up any files still in the store.

What does have to be managed though, are any files remaining in the cache storage. If a user submits, validation fails, and then the user closes the browser, the file will be in the cache forever. This is also the case with Shrine.rb in Ruby apps. We just set a retention period of 30 days for the cache dir on our MinIO instances.

What's next?

• Lucky::Attachment::Storage::S3
• Variants (for example images in different sizes)
• Better mime type extraction
• Dimensions extraction
• Async file promotion, processing, and deletion via a background job (needs an adapter; this may be for another release)

Checklist

• - An issue already exists detailing the issue/or feature request that this PR fixes
• - All specs are formatted with crystal tool format spec src
• - Inline documentation has been added and/or updated
• - Lucky builds on docker with ./script/setup
• - All builds and specs pass on docker with ./script/test

[jwoertink]

I just had a quick look through this, and it looks awesome.

My question is: where does this code live?

My first thought was probably inside Avram, and maybe within the Lucky extension if needed. Params sort of works that way.

Now for my questions: Does this mean that the file_attribute for operations goes away? https://github.com/luckyframework/avram/blob/be8a22f9a7f6f7cc8d94b4becc236668ec22e740/src/avram/define_attribute.cr#L90C9-L95 This is all connected into the Avram::Uploadable or is this all just added on top?

Yeah, this all looks pretty good. 🚀

[wout]

Yeah, I think having it in Avram would make it easier to test, I still have to write those for that part. I suppose we can create mock objects for the Lucky parts or something. It will probably be easier to write tests for it in the Avram repo than in the Lucky repo.

As for your question: no the file_attibute macro does not go. In fact this implementation depends on it:
https://github.com/luckyframework/lucky/pull/2016/changes#diff-6f228ffec5fade80b5c3972498465c0dfd822cddb36821aeb52249e5b6c199efR105

I think it's also good to keep file_attribute around so people can hook into that if they want to use another upload implementation.

UPDATE
I suppose it can go if you'd like to. The attach macro on the operation can also declare it directly as:

attribute {{ field_name }} : Avram::Uploadable