rails: rails assets:precompile in production failed due to missing master key

@kaspth I understand the resistance to having some commands not load the whole environment and some load the whole environment, but assets:precompile in particular is a real pain for containers. Right now our build tooling has to provide a database connection during image build, which we do with some nonsense, and now it looks like we’re going have to do a similar workaround with SECRET_KEY_BASE. I don’t think this task making building containers much more awkward is that niche a concern, and it would be really great for Rails to address it.

Even when I set require_master_key to false I am getting this error message. A master key seems unnecessary when building assets.

For me using docker build args solved this issue

My Dockerfile:

...
ARG RAILS_MASTER_KEY
RUN RAILS_MASTER_KEY=${RAILS_MASTER_KEY} RAILS_ENV=production bundle exec rails assets:precompile

I can build it with docker build --build-arg RAILS_MASTER_KEY={{master_key}} I integrated this into github actions using github’s repository secrets and passing the env to the docker build command.

Using workaround SECRET_KEY_BASE=`bin/rake secret` bin/rake assets:precompile for now.

@kaspth rails assets:precompile is basically a build step. Should we really need the master key during precompile? I had to perform wacky workaround by injecting dummy key into my CI build (because I don’t want to expose my real key during build) to get around this issue 😢

Workaround in Dockerfile

# Precompile assets
# We use dummy master.key and credentials.yml.enc to workaround the fact that
# assets:precompile needs them but we don't want the real master.key to be built
# into the container. We will inject RAILS_MASTER_KEY env var when starting the
# container.

RUN if [[ "$RAILS_ENV" == "production" ]]; then \
      mv config/credentials.yml.enc config/credentials.yml.enc.backup; \
      mv config/credentials.yml.enc.sample config/credentials.yml.enc; \
      mv config/master.key.sample config/master.key; \
      bundle exec rails assets:precompile; \
      mv config/credentials.yml.enc.backup config/credentials.yml.enc; \
      rm config/master.key; \
    fi

This really needs an official solution and shouldn’t be a closed issue. It boggles my mind why it works this way and why I need to load my credentials to compile assets.

My issue is that a NoMethodError: undefined method '[]' for nil:NilClass is being thrown whenever the credentials are accessed. For my first workaround, I followed this suggestion and accessed credentials via the dig method.

This worked but several gems then complained about empty strings for the values that should have been set from the credentials. As a workaround, I’m now building assets in my Dockerfile by running RUN ASSETS_PRECOMPILE=1 SECRET_KEY_BASE=1 RAILS_ENV=production bundle exec rake assets:precompile. Then, in every initializer that uses a credential I added return if ENV['ASSETS_PRECOMPILE'].present?

I wasn’t able to deploy my app for a week until I got this issue taken care of using a hack. This needs more attention.

This is intentional behavior. If require_master_key is set to true, a check is made as to whether the master key exists, regardless of the executed task.

Ran into exactly the same issue. We can just use the same dummy key approach as with development and test. Apologies for the (very, very) long delay on such a simple, but annoying issue.

I still don’t know how to deal with this in Rails 6, Docker build using credentials

It used to be possible to just set: config.assets.initialize_on_precompile = false in the application.rb or wherever you enable the asset pipeline.

After all, you just want to precompile the static assets.

But that was removed, so now precompiling the assets for container deployment now requires not only the secret keys, but a database connection. I can’t fathom why this is considered a good idea, especially since it forces what should be an immutable production image to be built using a fake environment.

For me using docker build args solved this issue

My Dockerfile:

...
ARG RAILS_MASTER_KEY
RUN RAILS_MASTER_KEY=${RAILS_MASTER_KEY} RAILS_ENV=production bundle exec rails assets:precompile

I can build it with docker build --build-arg RAILS_MASTER_KEY={{master_key}} I integrated this into github actions using github’s repository secrets and passing the env to the docker build command.

Although this works, you should probably not do it this way because Docker actually stores the passed in ARG value in the image’s history. An alternative approach would be to use the new Docker BuildKit features to pass in secrets to your Dockerfile.

Your Dockerfile would look something like this:

RUN --mount=type=secret,id=master_key,dst=/path/to/rails_app/config/master.key \
  RAILS_ENV=production bin/rails assets:precompile

And then build the image like so:

DOCKER_BUILDKIT=1 docker build --secret id=master_key,src=config/master.key .

Hope this helps any other Docker users that may have ran into this issue!

This should be really fixed, it’s a pain in a containerized world nowadays, just triggered me too.

This issue should be opened at least again.

Same for me. Can’t precompile on build stage.

I’ve run into this problem as well. A solution I’ve seen in other forums is to pass a dummy value for secret_key_base SECRET_KEY_BASE=1 RAILS_ENV=production bin/rails assets:precompile. This satisfies the presence check and the precompile process can execute successfully. Would this cause any side effects?

If you’d like to use the SECRET_KEY_BASE_DUMMY approach, but you’re not going to run against edge Rails, you can reopen the Rails::Application in your config/environment.rb file and add:

class Rails::Application
  def secret_key_base
    if Rails.env.development? || Rails.env.test? || ENV["SECRET_KEY_BASE_DUMMY"]
      secrets.secret_key_base ||= generate_development_secret
    else
      validate_secret_key_base(
        ENV["SECRET_KEY_BASE"] || credentials.secret_key_base || secrets.secret_key_base
      )
    end
  end
end

Above the Rails.application.initialize! line. That’ll give you the same effect as running a version of Rails with #46760 included.

I strongly agree with the idea of a debug and a release setting for compiling assets. This supports the idea of consistent deployments up the chain.

You don’t want to compile your assets for your staging environment and validate things, only to discover when you go to production that things compiled just a wee bit different. Ideally, you could compile your asset package once and promote those exact bit-for-bit artifacts up the ladder from “qa” to “staging” to “production”, or whatever mix of environments you have.

That said, as an aside, one practice I follow is to treat all deployed environments as “production”. Our Rails app has only test, development and production environments; our qa, staging and production deployments are all “production”. Any differences (such as database credentials) are externalized to environment variables.

Why is this closed? I’m having this problem RIGHT NOW.

Same issue here. Why WAS this one closed?

For all you poor souls out there, trying to deploy to heroku. I would like to save you from suffering. If you are getting NoMethodError: undefined method '[]' for nil:NilClass I have worked around this issue. Like this.

Find all you files that contain Rails.application.credentials, like stripe.rb or twillio, or stripe-form in views etc… replace with dig method for example my twillio.rb looks like this now.

config.account_sid = Rails.application.credentials.dig(:twilio, :TWILIO_ACCOUNT_SSID)
config.account_sid = Rails.application.credentials.dig(:twilio, :TWILIO_AUTH_TOKEN)

do this for all your files. Use dig method.

open rails console in production to test if these values don’t return nil RAILS_ENV=production rails c > Rails.application.credentials.dig(:twilio, :TWILIO_ACCOUNT_SSID)

> HJ12345678944dhaaha

check all your credentials and ensure they don’t return nil.

Make sure you check production.rb credentials as well, like paypal or sendgrid, or aws etc. Ensure to check config.require_master_key = true in roduction.rb

Now if you run on your terminal RAILS_ENV=production bundle exec rake assets:precompile it will work without any issues.

if you’re using only one credential file run EDITOR='code --wait' rails credentials:edit for vscode copy your secret_key_base.

If you’re using heroku app, go to dashboard and config vars and add key value pairs secret_key_base to be the value you copied. also ensure your RAILS_MASTER_KEY is present in config vars in heroku, this is important.

I don’t know about adding all this info on config vars in heroku. But desperado.

DHH’s solution doesn’t solve the problem for me.

I have two requirements:

1. I would like to have a friendly message whenever someone tries to run rails c or rails s without a key.
2. However, I would also like to have the ability to run rails assets:precompile without a key (for us within docker).

My solution is:

config.require_master_key = true if Rake.application.top_level_tasks.exclude?('assets:precompile')

# or even if you need to run `assets:clean` without a key

config.require_master_key = true if Rake.application.top_level_tasks.none? { |task| task.start_with?('assets:') }

For all you poor souls out there. I would like to save you from sadistic suffering. If you are getting NoMethodError: undefined method '[]' for nil:NilClass I have worked around this issue. Like this.

Find all you files that contain Rails.application.credentials, like stripe.rb or twillio, or stripe-form in views etc…

replace with dig method for example my twillio.rb looks like this now.

config.account_sid = Rails.application.credentials.dig(:twilio, :TWILIO_ACCOUNT_SSID)

config.account_sid = Rails.application.credentials.dig(:twilio, :TWILIO_AUTH_TOKEN)

do this for all your files. Use dig method.

open rails console in production to test if these values don’t return nil

RAILS_ENV=production rails c

> Rails.application.credentials.dig(:twilio, :TWILIO_ACCOUNT_SSID)

> HJ12345678944dhaaha

check all your credentials and ensure they don’t return nil.

Make sure you check production.rb credentials as well, like paypal or sendgrid, or aws etc.

Ensure to check config.require_master_key = true in roduction.rb

Now if you run on your terminal RAILS_ENV=production bundle exec rake assets:precompile it will work without any issues.

if you’re using only one credential file run EDITOR='code --wait' rails credentials:edit for vscode

copy your secret_key_base.

If you’re using heroku app, go to dashboard and config vars and add key value pairs secret_key_base to be the value you copied.

also ensure your RAILS_MASTER_KEY is present in config vars in heroku, this is important.

I don’t know about adding all this info on config vars in heroku. But desperado.

You are missing the point. The issue is about creating a docker image from a rails app. To create the image the assets need to be compiled. For asset precompilation the master_key is required which, for obvious reasons, is not included in the source code. This thread discusses this issue.

However, I would also like to have the ability to run rails assets:precompile without a key (for us within docker).

At Fly.io, there’s a lot of support issues and confusion related to this requirement. Eliminating the need to pass a “dummy” key into the precompile command would make more deploys successful. I’d be interested in putting a patch together if I could get some guidance on knowing whether or not this was attempted the past.

I recall a time when there was an asset group (I think that’s what it was called) in the Gemfile that probably tried solving this problem, but that was removed. Curious what other attempts have been made at eliminating the dummy key requirement.

For anyone using GitLab Auto DevOps and facing this problem, here is how I got it working using an Auto DevOps customisation that leverages Docker BuildKit secrets:

1. Enable docker experimental features by adding this to the top of your Dockerfile: # syntax = docker/dockerfile:experimental
2. For the asset compilation stage, use this command: RUN --mount=type=secret,id=auto-devops-build-secrets . /run/secrets/auto-devops-build-secrets && RAILS_ENV=production bundle exec rails assets:precompile

Here is my Dockerfile in full:

# syntax = docker/dockerfile:experimental
FROM ruby:2.5.1

RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app

RUN apt-get update && apt-get install -y nodejs mysql-client postgresql-client sqlite3 vim --no-install-recommends && rm -rf /var/lib/apt/lists/*

ENV RAILS_ENV production
ENV RAILS_SERVE_STATIC_FILES true
ENV RAILS_LOG_TO_STDOUT true

COPY Gemfile /usr/src/app/
COPY Gemfile.lock /usr/src/app/
RUN bundle config --global frozen 1
RUN bundle install --without development test

COPY . /usr/src/app

RUN --mount=type=secret,id=auto-devops-build-secrets . /run/secrets/auto-devops-build-secrets && RAILS_ENV=production bundle exec rails assets:precompile

COPY docker-entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/docker-entrypoint.sh
ENTRYPOINT ["docker-entrypoint.sh"]

EXPOSE 3000
CMD ["rails", "server", "-b", "0.0.0.0"]

1. In GitLab add a CI/CD variable for your rails master key (mask the variable) RAILS_MASTER_KEY = <your real rails master key>
2. In GitLab add a CI/CD variable to forward the RAILS_MASTER_KEY variable to the Docker build process AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES = RAILS_MASTER_KEY

My previous attempts for others who land here:

I’m using GitLab Auto DevOps. I attempted two of the workarounds above with the following outcomes:

1. @alphabt’s workaround: worked for the build process, but attempting to run the container with the real master key failed. I’m using devise and got a devise error on application boot. Devise doesn’t seem to play well with using one key in the precompile process and a different key at runtime.

[+] Running 2/2
 ⠿ Container app_db_1    Running                                                                     0.0s
 ⠿ Container app_prod_1  Recreated                                                                   0.1s
Attaching to prod_1
prod_1  | rails aborted!
prod_1  | ActiveSupport::MessageEncryptor::InvalidMessage: ActiveSupport::MessageEncryptor::InvalidMessage
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/message_encryptor.rb:206:in `rescue in _decrypt'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/message_encryptor.rb:183:in `_decrypt'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/message_encryptor.rb:157:in `decrypt_and_verify'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/messages/rotator.rb:21:in `decrypt_and_verify'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/encrypted_file.rb:79:in `decrypt'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/encrypted_file.rb:42:in `read'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/encrypted_configuration.rb:21:in `read'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/encrypted_configuration.rb:33:in `config'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/encrypted_configuration.rb:38:in `options'
prod_1  | /usr/local/bundle/gems/activesupport-5.2.5/lib/active_support/core_ext/module/delegation.rb:271:in `method_missing'
prod_1  | /usr/src/app/config/initializers/devise.rb:284:in `block (2 levels) in <main>'
prod_1  | /usr/local/bundle/gems/devise-jwt-0.6.0/lib/devise/jwt.rb:20:in `jwt'
prod_1  | /usr/src/app/config/initializers/devise.rb:283:in `block in <main>'
prod_1  | /usr/local/bundle/gems/devise-4.4.3/lib/devise.rb:307:in `setup'
prod_1  | /usr/src/app/config/initializers/devise.rb:5:in `<main>'

1. @Oldharlem’s workaround of passing the rails master key as a build arg to Docker worked, but I was uncomfortable about it being exposed in the build process.

Here is an approach that could simplify things… and provide options in all sorts of build/deployment/execution scenarios.

RAILS_MASTER_KEY is already an ENV var, configurable in there usual ways in the shell and within Docker files.

Could another ENV var be added – RAILS_CREDENTIALS_FILE – to point to what credentials file to use?

One could then set the RAILS_MASTER_KEY to a non-sensitive key and RAILS_CREDENTIALS_FILE to a corresponding encrypted credentials file. If the latter ENV var is not set, the RAILS_ENV is used to find the proper file as is done now.

If viable, I’d be happy to post a PR.

This is really irritating that the new asset system assumes the full environment along with the master key is available just to invoke webpack…

My solution is just to add a env var to skip the config in production.rb during asset compilation.

production.rb

Rails.application.configure do
#...
end unless ENV['RAILS_BUILD']

Dockerfile

#...
RUN SECRET_KEY_BASE=1 RAILS_BUILD=1 bundle exec rake assets:precompile

It would be nice to instead just create a standalone script to invoke Webpack(er) without needing to mess with this.

Suggested workaround earlier in the issue of SECRET_KEY_BASE=`bundle exec rake secret` RAILS_ENV=production bundle exec rake assets:precompile does not work for me.

Error: Missing encryption key to decrypt file with. Ask your team for your master key and write it to /config/master.key or put it in the ENV['RAILS_MASTER_KEY'].

Setting RAILS_MASTER_KEY manually in env gives errors of improper format. I’ll probably see how Rails is generating it behind the scenes and duplicate that. Must say, it’s really weird this issue is a wontfix.

For all you poor souls out there. I would like to save you from sadistic suffering. If you are getting NoMethodError: undefined method '[]' for nil:NilClass I have worked around this issue. Like this. Find all you files that contain Rails.application.credentials, like stripe.rb or twillio, or stripe-form in views etc… replace with dig method for example my twillio.rb looks like this now.

config.account_sid = Rails.application.credentials.dig(:twilio, :TWILIO_ACCOUNT_SSID)

config.account_sid = Rails.application.credentials.dig(:twilio, :TWILIO_AUTH_TOKEN)

do this for all your files. Use dig method. open rails console in production to test if these values don’t return nil RAILS_ENV=production rails c > Rails.application.credentials.dig(:twilio, :TWILIO_ACCOUNT_SSID) > HJ12345678944dhaaha check all your credentials and ensure they don’t return nil. Make sure you check production.rb credentials as well, like paypal or sendgrid, or aws etc. Ensure to check config.require_master_key = true in roduction.rb Now if you run on your terminal RAILS_ENV=production bundle exec rake assets:precompile it will work without any issues. if you’re using only one credential file run EDITOR='code --wait' rails credentials:edit for vscode copy your secret_key_base. If you’re using heroku app, go to dashboard and config vars and add key value pairs secret_key_base to be the value you copied. also ensure your RAILS_MASTER_KEY is present in config vars in heroku, this is important. I don’t know about adding all this info on config vars in heroku. But desperado.

You are missing the point. The issue is about creating a docker image from a rails app. To create the image the assets need to be compiled. For asset precompilation the master_key is required which, for obvious reasons, is not included in the source code. This thread discusses this issue. So this isn’t about this? “Like he said, this in intentional. Right now, to compile assets you need to boot the Rails application in the same environment you need to compile those assets. This means that all information required to boot the application needs to be provided. Changing this would require a lot of changes to the framework and it is not in our plans to do so.”

all information required? So all information was provided. Regardless Docker or not. It didn’t work. No Method for nil class. Took me 2 days to to get this with dig method thanks to @jmadkins

The point you are missing is that this issue is about compiling assets WITHOUT the master key. You are requiring it.

For me using docker build args solved this issue

My Dockerfile:

…

ARG RAILS_MASTER_KEY

RUN RAILS_MASTER_KEY=${RAILS_MASTER_KEY} RAILS_ENV=production bundle exec rails assets:precompile

I can build it with docker build --build-arg RAILS_MASTER_KEY={{master_key}}

I integrated this into github actions using github’s repository secrets and passing the env to the docker build command.

Although this works, you should probably not do it this way because Docker actually stores the passed in ARG value in the image’s history. An alternative approach would be to use the new Docker BuildKit features to pass in secrets to your Dockerfile.

Your Dockerfile would look something like this:

RUN --mount=type=secret,id=master_key,dst=/path/to/rails_app/config/master.key \

  RAILS_ENV=production bin/rails assets:precompile

And then build the image like so:

DOCKER_BUILDKIT=1 docker build --secret id=master_key,src=config/master.key .

Hope this helps any other Docker users that may have ran into this issue!

Good find! Did not realize. In my case the image never sees a publicly available hub but get uploaded straight to AWS ECR which should be quite safe. When your images are publicly available this could pose a serious threat indeed.

^ Ran into this problem also in build environment - really needs to be looked at especially because currently giving an actual secret to a DockerFile is quite a pain see: https://docs.docker.com/develop/develop-images/build_enhancements/

@bradgessler @martinstreicher I’m not advocating either of you invest time and effort into this, but generally I’d expect you to get better feedback on feature requests to discuss, or in the form of a PR. 🙏

Wow… It’s been so long that I can’t even remember how I solved this! hahahaha! 😃

But if you just scroll through the issue and find my name you’ll see what my solution was. It’s still what my server uses to this day… So I guess it worked!

On Fri, Aug 6, 2021 at 5:39 PM aphel @.***> wrote:

Suggested workaround earlier in the issue of SECRET_KEY_BASE=bundle exec rake secret RAILS_ENV=production bundle exec rake assets:precompile does not work for me.

Error: Missing encryption key to decrypt file with. Ask your team for your master key and write it to /config/master.key or put it in the ENV[‘RAILS_MASTER_KEY’].

Setting RAILS_MASTER_KEY manually in env gives errors of improper format. I’ll probably see how Rails is generating it behind the scenes and duplicate that. Must say, it’s really weird this issue is a wontfix.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rails/rails/issues/32947#issuecomment-894344943, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2KI54X6JLIDXMGS32HAUTT3P64RANCNFSM4FA4GXPQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .