rails: Dependency on mimemagic 0.3.x no longer valid

We (the Rails core team and friends) are currently investigating the possibility of using the 2-clause BSD–licensed libmagic, the basis of the standard file tool, or Ruby translation of its magic DBs.

Rails 5.2.5, 6.0.3.6, and 6.1.3.1 have been released, removing the mimemagic dependency.

“Someday ~ImageMagick~ mimemagic will finally break for good and we’ll have a long period of scrambling as we try to reassemble civilization from the rubble.”

https://xkcd.com/2347/

For now I decided to archive the project. The last released versions are 0.3.6 and 0.4.0 released under the GPL-2.0.

I did not understand the root cause of yanked versions, but what is the solution for this scenario? Force everyone in the world to upgrade to 0.3.6? I will ask this in mimemagic repo, but the repo was archived (that I also not understand why was archived). Crazy day!

I think so

How will this affect projects running older Rails versions? E.g Commercial legacy projects etc?

I’m not a lawyer, but in my understanding, the mimemagic change to GPL licence forces rails to be distributed in GPL also, which forces all projects that are using Rails to be open-sourced

Not only Rails depends on mimemagic. The gem is used by 576,971 Repositories and 172 Packages on Github ATM. It would be more correct for @minad to unarchive https://github.com/minad/mimemagic and move this conversation there IMO.

@sergey-alekseev I disagree. The Rails dependency is certainly the most impactful one. It is best if we find a solution which works for Rails and which is sanctioned by the Rails team.

@minad The problem with yanking the older versions unannounced means every existing project that (indirectly) depends on mimemagic can no longer be installed, unless they bump the version. The new license in turn may prevent may projects from doing so for legal reasons.

Of course the Gem should already have been using the GPL license, but it would have been much better if this was at least announced ahead of time in some way.

As an example: we are running into this issue in GitLab, effectively making it impossible to deploy new versions until we find a workaround. Bumping the version may not work either, as we disallow GPL dependencies (as these conflict with our enterprise edition).

My perspective on this (and as far as I can tell I’m the only one actively trying to resolve this issue) is that I don’t want to reach a conclusion which while technically compliant with the license goes against the spirit of that license, and the author’s intention when they released the database under that license. I’m talking to the maintainers about potential options, and will update once there’s something to say.

I’ve spoken to the freedesktop.org contributor who raised the original complaint, and they are happy with the approach being taken in my fork of the gem. I’m going to push ahead with implementing that now. See https://github.com/jellybob/mimemagic/issues/1 for more details (but please pay heed of the comment right at the top about license lawyering).

I’ve started here : https://github.com/rails/rails/pull/41751

I did not understand the root cause of yanked versions, but what is the solution for this scenario? Force everyone in the world to upgrade to 0.3.6? I will ask this in mimemagic repo, but the repo was archived (that I also not understand why was archived). Crazy day! 😆

yeah GPL definitely is not an option for us.

I don’t want to party poop but because 0.3.6 is already GPL-2 which brings everybody using rails in a sticky situation right now i guess…

👍🏽 Since @minad is a fan of yanking gems maybe they’ll yank the GPL-2 license as a non-breaking gem version since it poisons the license for anyone using rails too as far as I can tell. /s

I just wanted to drop a small hint here for people who want to build their apps right now.

mimemagic is only a dependency to activestorage and actionmailbox.

If your Rails app doesn’t use activestorage nor actionmailbox , you could cherry-pick the other rails subgems for now:

gem 'activesupport'
gem 'actionpack'
gem 'actionview'
gem 'activemodel'
gem 'activerecord'
gem 'actionmailer'
gem 'railties'
gem 'sprockets-rails'

# if you use cucumber-rails, the latest version depends on Rails
# however, cucumber-rails on master branch only depends on railties
gem 'cucumber-rails', require: false, github: 'cucumber/cucumber-rails'

Today, I am trying to build docker image of my rails app 6.0.3.5

Then got a nasty error below:

I got it works by:

bundle update --conservative mimemagic

More detail: user1678123 - stackoverflow

Updating to 0.4.x would be a substantial issue for many people, as it will result in a GPL2 dependency being pulled in, which a large proportion of companies have policies against using.

There’s no “technically” right about this. It’s a clear violation of the GPL in that its a derived work of GPL licensed code. The question here isn’t whether or not a violation has occurred, but what the resolution to that is. I’m talking to the maintainers, and I suspect an amicable solution can be found which doesn’t involve everyone lawyering up.

This is no longer a breaking issue because the maintainer of mimemagic has released a compatible version called 0.3.6, but the dependency probably still might need updating. https://github.com/minad/mimemagic/issues/98

I don’t want to party poop but because 0.3.6 is already GPL-2, deploying rails right now brings everybody using a not compatible license in a sticky situation i guess…

I’m going to try and make a PR to replace the gem. Even if it’s not ideal, it’s a good starting point for me and can be rejected/updated by others

@erran This is not a good proposal. Do you prefer to have no 0.3.x version available at all? The fastest way to sort this out is:

1. Generate tables.rb from https://pagure.io/mailcap/blob/master/f/mime.types and distribute that as 0.3.7 and 0.4.1. But @jellybob’s objection is true.
2. Load the freedesktop.org.xml at startup and don’t distribute tables.rb as part of the mimemagic gem. Then we can also create a 0.3.7 and 0.4.1 version under the MIT license.

@Pierstoval While you are technically correct, I am willing to change the license of the mimemagic.rb files back to MIT. The only files which cannot be licensed and distributed under MIT are freedesktop.org.xml and the derived tables.rb. Note that the “loading at runtime approach” has been proposed by @hadess themselves in minad/mimemagic#97. It would be best to get confirmation by @hadess as to what is considered a compliant solution, as @jellybob is attempting. I suggest you follow jellybob/mimemagic#1.

I don’t know whether that’d be “compliant”, because that would require a law degree and rubber stamps which I don’t have, but that would solve the problem as I saw it.

What I wouldn’t agree to is shipping the freedesktop.org.xml (and derived) files in an MIT-licensed project. None of the (past) versions of mimemagic labelled as “MIT” are actually “MIT” licensed, as they use a GNU GPL source, essentially making them all GPL. The changed license is a correction, not a relicense.

To anyone forking the mimemagic repo, remember that you can fork it only starting at the f46e0d38 commit, because it is the last one with MIT license. If you fork it on later commits, you’ll have a GPL-based repository, that’s not what you seem to want in your private hidden closed-source code

Also, please note that the mimemagic gem is now on GPL2 license https://github.com/minad/mimemagic/commit/c0f7b6b21a192629839db87612794d08f9ff7e88

🚀 ⬆️ Deployed as part of production 2021-03-28 23:57:06.

This could have been handled better 😕 honestly!

The best way forward is probably to create a version of the gem which is licensed under the MIT license and loads the mime database at startup. This is the approach proposed by the maintainer of shared-mime-info, see https://github.com/minad/mimemagic/issues/97.

the entire thing is actually GPL licensed

I’m not sure that’s true. At this point, we need laywers 🤷.

I’ve forked mimemagic, and intend to make changes as described in the issue opened on my fork at https://github.com/jellybob/mimemagic/issues/1

This may be a stupid question but has anyone asked the original copyright holder if they will allow that single file to be MIT licensed. I know they reported the problem originally but they may see the bind people are in and allow just that data to be MIT licensable.

Quite honestly, I think the original claim that this was a GPL violation is wrong. I think the database could be included under fair use. You’d need a lawyer to verify but I think this could have been kept under the MIT license.

For anyone else that runs into this error when trying to build their project, I updated my Rails 5 project to 5.2.5 and this resolved the issue and mimemagic is no longer in the Gemfile.lock.

I personally have not tried this for Rails 6, but I suspect that if you are on Rails 6 and upgrade to 6.1.3.1 that this will also resolve your build issues and remove mimemagic from your lock file.

Another solutions is to use: https://pagure.io/mailcap/blob/master/f/mime.types like https://github.com/elixir-plug/mime

Seems to be under Public Domain from Redhat, derives from http://www.iana.org/assignments/media-types/media-types.xhtml

Could not find gem ‘mimemagic (= 0.3.6)’ in any of the gem sources listed in your Gemfile

@Pierstoval IANAL, but that’s not how it works. It just means you are in violation of the GPLv2. The way that situation is resolved is by a lawsuit. Again, IANAL but no judge would ever say you should give all your businesses proprietary code away just because you inadvertently pulled in a GPL source file.

If you do it knowingly though, that’s a whole different story, and companies have in the past been forced to release their full source code just to make them comply.

I don’t like the release of 0.3.6, I think it was better to break people’s deployments, so that they are made aware of their violation, and they could upgrade to 0.4 to make a decision. Now people are unknowingly pulling in the new license, and their legal position is slightly diminished. Either way it was definitely the most ethical move by Daniel to pull the gems.

@Pierstoval You’ll need to roll back to a commit before the GPL dependency (freedesktop.org.xml) was added to mimemagic. The issue precipitating all of this is that the old MIT license was invalid for the project. It’s not as simple as going back to when the license file changed, unless you think the original license violation claim was invalid.

as the file is part of GNOME.

The shared MIME database is not “part of GNOME”; as you might have noticed, the source repository is on freedesktop.org. GNOME, KDE, and other projects are consumers of the shared MIME database.

The recommendation to ask a lawyer for licensing issues, though, is a good recommendation.

@regismesquita the previous version had a license file saying it was MIT, but due to including a file taken from a GPL licensed project, the entire thing is actually GPL licensed because that’s a requirement of using GPL licensed libraries. This works in a similar way to it not being sufficient for me to just slap a sticker on something saying “Property of Jon” for ownership to be transferred to me.

In case someone is looking for a solution, you need to upgrade the Rails version, in my case to 6.1.3.1.

More info here https://weblog.rubyonrails.org/2021/3/26/marcel-upgrade-releases/

Thanks IBM! You are making the world a harder place to get work done.

What does IBM have to do with licensing terms not being followed?

@Pierstoval While you are technically correct, I am willing to change the license of the mimemagic.rb files back to MIT. The only files which cannot be licensed and distributed under MIT are freedesktop.org.xml and the derived tables.rb. Note that the “loading at runtime approach” has been proposed by @hadess themselves in minad/mimemagic#97. It would be best to get confirmation by @hadess as to what is considered a compliant solution, as @jellybob is attempting. I suggest you follow https://github.com/jellybob/mimemagic/issues/1.

If you have a SasS product maybe this applies?

As long as you don't distribute your binaries, there is no problem with using GPL libraries (or other code) in an otherwise closed-source project. 

As far as the regular GPL and LGPL are concerned, providing access to use your software over a network (like in SaaS) is not considered distribution. This means that there is no problem with using (L)GPL libraries in a closed-source SaaS project.

ref: https://softwareengineering.stackexchange.com/questions/220742/using-full-gpl-libraries-in-software-as-a-service-saas-models/220743

@Deradon the distributed gem will still be GPL licensed, because you’re distributing a GPL licensed source file as part of it. All that change really does is remove the dependency from Github, however the Github repo isn’t what’s being distributed to end users.

I’m willing to help out/contribute, but will need some good explanation on what the possibilities are that can be done to remedy this problem correctly. Although I’d rather see the solution in the hands of the Rails team

Hey @georgeclaghorn, rails 5.2.5 still uses MimeMagic 😔

have u solved the issue? I have the same issue even after install shared-mime-info package. macos big sur 11.2.3, apple m1

Hey @georgeclaghorn, rails 5.2.5 still uses MimeMagic 😔

So I managed to get my PR to be green for Rails. Not sure if this solves the problem given the marcel dependency, but point for effort I guess xD

@Deradon by a strict reading of the GPL license, yes. I’m attempting to get in touch with the maintainer of the source file to discuss what the options are.

The database at https://www.nationalarchives.gov.uk/PRONOM/Default.aspx includes magic numbers signatures and was mentionned by https://github.com/zRedShift/mimemagic/issues/4#issuecomment-805774658 as an alternative to freedesktop’s database.

Using a database that doesn’t contain magic signatures isn’t a viable path. We (via Marcel) use MimeMagic specifically for this feature; if we don’t need that, we already have multiple other MIME type DBs available in Rails with varying completeness (but we do need it).

@pedantic-git I certainly haven’t, as my personal objective here is to unblock deployments of services I’m responsible for with the minimum of friction while retaining mimemagic’s runtime functionality unchanged, but it sounds like it could be a good solution longer term.

@regismesquita @mvz There’s a few different things going on. I may very well have misread what the OP was saying. All I was pointing out is that the issue isn’t that the license file changed. Whatever your ultimate solution is, it needs to address the notion that the project was never properly MIT in the first place.

@nirvdrum that would mean a license violation can never be fixed and I don’t think that’s how it’s supposed to work.

@Pierstoval I believe https://github.com/minad/mimemagic/commit/40dd02bb6b442535f97c35326c0383bc67146ac4 since https://github.com/minad/mimemagic/commit/c0f7b6b21a192629839db87612794d08f9ff7e88 is the one where GPLv2 is first used?

I just updated Rails from v6.1.3 to v6.1.3.1, this replaced the dependency in activestorage with mini_mime (~> 1.0.2).

Active Storage was introduced in Rails 5.2. Rails < 5.2 doesn’t transitively depend on mimemagic.

What about projects with rails 4 (specifically I have 4.2.5)? Is there going to be an upgrade for that version?

Your screenshot shows the installed version of carrierwave depending on it. You can check Gemfile.lock for other gems that may depend on it.

Looks good, @rmacklin. 👍

@georgeclaghorn I wish I could hit the ❤️ button multiple times

If you have a SasS product maybe this applies?

As long as you don't distribute your binaries, there is no problem with using GPL libraries (or other code) in an otherwise closed-source project. 

As far as the regular GPL and LGPL are concerned, providing access to use your software over a network (like in SaaS) is not considered distribution. This means that there is no problem with using (L)GPL libraries in a closed-source SaaS project.

ref: https://softwareengineering.stackexchange.com/questions/220742/using-full-gpl-libraries-in-software-as-a-service-saas-models/220743

There’s more information about how GPL applies to SaaS products (that are accessed over a network, but not distributed) in an article at https://resources.whitesourcesoftware.com/blog-whitesource/the-saas-loophole-in-gpl-open-source-licenses

From the looks of it, hosting a Rails app that uses GPL code doesn’t require the source code to be made available to all users.

My perspective on this (and as far as I can tell I’m the only one actively trying to resolve this issue) is that I don’t want to reach a conclusion which while technically compliant with the license goes against the spirit of that license, and the author’s intention when they released the database under that license. I’m talking to the maintainers about potential options, and will update once there’s something to say.

I generally agree and negotiation and a mutual agreement is the best solution. But look at what the file in question actually is: It describes the “magic numbers” of file types. Claiming that there’s a copyright on that is a little like saying there’s a copyright on an API. It’s not straightforward copyrightable under US law, to the best of my knowledge.

And I’m not suggesting “lawyering” up. I’m suggesting that we’re not qualified to know what the options are. I strongly, strongly, strongly recommend talking to a lawyer before doing too much more.

I really encourage a qualified lawyer be consulted as well as folks at GNOME as the file is part of GNOME.

Terming something as fair use or illegal at this scale can’t be done using random opinion from internet, I think community need to consult organisation like EFF or FSF for practical way forward. I think decision to yank gem was bit to early, But I empathize with the gem owner as it is difficult to take a decision when internet is storming about it

@nirvdrum so technically speaking before mimemagic was created.

@YorickPeterse

Or would the derived file still be GPL due to being based on a GPL XML file?

You basically answered your own question here. The file is then GPL since it is derived.

As I see it there are two options, both with drawbacks, so as others have said some input from the Rails team is probably needed:

Option 1: Download the freedesktop.org mime type descriptions at runtime. This probably results in a minimum of changes to mimemagic, and doesn’t impact how that gem behaves. However, this then means potential issues for people deploying in tightly regulated environments as additional communication with third party servers will need to be approved - in some cases there’s potential for pulling in dependencies from third parties at run time simply not being approved at all.

Option 2: Move to using Red Hat’s public domain licensed mime type descriptions. Doesn’t have issues with pulling in resources at run time, but it will mean a reduction in accuracy during mime type sniffing.

You could take a similar approach to faraday, where there is a common interface but two implementations

[Solved] Bundler::GemNotFound: Could not find mimemagic-0.3.5 in any of the sources on Rails project with Docker

All the GPL versions (<= 0.3.6 and = 0.4.0) were yanked.

The latest release of the 0.3.x line is 0.3.10 if you would like to give it a try.

You can always check https://rubygems.org/gems/mimemagic

Note that by yanking all prior versions the author has broken every Gemfile.lock which is currently being used which references, which will require a forced upgrade and release on everyone (breaking e.g. autoscaling of existing tested services).

If this is just a licensing issue, it might be useful for the author to work with rubygems directly to replace the old versions with ones that just have the license fixed.

I’ve spoken to the freedesktop.org contributor who raised the original complaint, and they are happy with the approach being taken in my fork of the gem. I’m going to push ahead with implementing that now. See jellybob/mimemagic#1 for more details (but please pay heed of the comment right at the top about license lawyering).

I mentioned there but also my only minor request would be that those of us can use GPL2+ code have a way to use the original mimemagic. In short, let’s not break it for people who don’t need it broken 😃

This may be a stupid question but has anyone asked the original copyright holder if they will allow that single file to be MIT licensed. I know they reported the problem originally but they may see the bind people are in and allow just that data to be MIT licensable.

@enebo Not a stupid question, but there are about 120 contributors.

Background info: @enebo @wwahammy etc.: it seems @hadess is technically within their rights. have a look here for example: https://gitlab.freedesktop.org/xdg/shared-mime-info/-/issues/154 and https://github.com/zRedShift/mimemagic/issues/4#issuecomment-805712232

@madejejej I’m a maintainer for cucumber rails (As is @mvz), we can try expedite something here if required. I’ll catch up on this thread tomorrow.

@Pierstoval there’s a difference between ‘should be licensed under GPL-2 or else it’s a license violation’ and ‘is automatically licensed under GPL-2’.

@pauldub The acknowledgements for PRONOM include Adobe, Borland, Corel and Microsoft amongst others - we’d need clarification that this isn’t tainted in some way as well.

@coding-bunny Please notify me when you have something ready. @jellybob Yes, that’s right. Pulling/Loading at runtime is not without impact. I think it needs a bit of effort to get this right. Maybe it is best to require the file being present at startup by requiring manual installation of the file.