shrine: backgrounding plugin breaks manual call of `promote`

You may sometimes want to manually call promote, and you ought to be able to. One use case is if the ordinary promotion failed for some reason, perhaps a reason related to external environment; maybe a disk filled up. Now the model is an unpromoted state and will never change unless you promote it; maybe you want to manually call promote to do so.

Backgrounding changes the nature of Attacher#promote, so it no longer works when called without any arguments. Worse, it fails silently leaving you no way to know it did not actually save the record with promoted data (file on store, any processing or metadata extraction you have configured to happen during promotion), etc.

This confused me greatly and took me all day to figure out, so I figured I’d report it!

To verify, we can look at an existing attacher_test, and just copy it the backgrounding_test, to be tested in the context of backgrounding. If you run that “#promote uploads the cached file to store” test with backgrounding loaded – it will fail. I can supply a branch or PR with this failing test if this isn’t clear.

Why? Well, that’s what took me a while to figure out.

A) In a plain shrine uploader, on a model with an attachment, if you have configured things for refresh_metadata on :store (as is one suggested way to do things with backgrounding), calling promote will trigger metadata extraction, which will result in a mutated in-memory model. The original in-memory model now has that new extracted metadata. But then promotion goes on to call update, with, which the activerecord (or other ORM) plugin, just saves it to disk, along with that new metadata.

B) The backgrounding plugin, however, will silently refuse to save it to disk. Because it looks at the changed in-memory data, and sees that the thing stored in the db does not match, and therefore decides to abort promotion entirely, in it’s overridden swap

C) So how does backgrounding normally work at all? Well, backgrounding only breaks the no-argument use of promote – or specifically, no first argument given, so the default value is used, calling method get. But ordinarily under backgrounding, a no-arg promote is never called. Instead backgrounding calls promote with an explicit first arg, deserialized from a bg job argument. Since the explicit arg was created by deserializing some JSON, when passing it, refresh/extract_metadata during storage does not mutate the in-memory model, so it passes backgrounding’s check in overridden “swap”.

So it only breaks promote manually called with no arguments, whcih doesn’t ordinarily happen in backgrounding “happy path”.

I suppose this could be considered not a bug – why would you want to call promote manually with no args? I think I’ve given a use case, but also it’s just confusing if backgrounding changes the semantics of existing methods (to make them stop working when they used to work). And led to me spending a day figuring out why things weren’t working like I expected – in this case in some of my local test setup code, but still. Adding a plugin probably shouldn’t change the semantics of an existing method, to make something that would have worked fine before, now fail silently. Such gotchas make shrine seem a lot less predictable/reliable.

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Comments: 18 (18 by maintainers)

Commits related to this issue

Most upvoted comments

This issue should be fixed by the two commits above ☝️

OK! I got it to reproduce with: activerecord plugin (rather than sequel), and the _data column as an activerecord json type instead of string.

Somehow that is resulting in some shared data structures that wouldn’t otherwise be shared, so read in the overridden swap returns the new metadata values, and causes comparison with reloaded to fail.

Possibly could reproduce with sequel too, if sequel can support native json columns which shrine will treat speically for (de)serialization?

Here’s a reproducing script. Phew!

require "shrine"
require "shrine/storage/memory"
require "active_record"
require "action_controller/railtie"
require "stringio"
require 'byebug'

ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: "promote_failure_repro")

ActiveRecord::Schema.define do
  create_table :users, force: true do |t|
    t.json :avatar_data
  end
end

Shrine.storages = {
  cache: Shrine::Storage::Memory.new,
  store: Shrine::Storage::Memory.new,
}



class ImageUploader < Shrine
  plugin :activerecord, callbacks: false
  plugin :backgrounding
  plugin :add_metadata
  add_metadata(:value_from_store) do |io, context|
    if context[:action] == :store
      "value_from_store"
    end
  end
  plugin :refresh_metadata
  plugin :processing
  process(:store) do |io, context|
    io.refresh_metadata!(context) # extracts metadata and updates `io.metadata`
    io
  end
end

class User < ActiveRecord::Base
  include ImageUploader::Attachment.new(:avatar)
end


user = User.new
user.avatar_attacher.assign(StringIO.new("image"))
user.save!
user.avatar_attacher.promote(action: :store, for_real_dont_fail: true)
user.reload
puts user.avatar.metadata # => does not include new metadata
puts user.avatar.storage_key # => "cache"