moby: docker cannot pass newlines from variables in `--env-file` files

I stumbled over this issue today. My use case is trying to load a pair of PEM encoded RSA keys without exposing the keys via the process table with the -e flag. This led me to --env-file which I pointed at a file I also use both with bash source foo.sh and systemd’s EnvironmentFile=foo.env directive.

It would be great if all three of these use cases could work without having to modify the file to make it compatible with the different parsing behaviors of the three. I suggest a simple double quoting like systemd does, with preservation of whitespace between the double quotes.

As an example, I’d like this file to work with docker --env-file:

RSA_PUBLIC_KEY="-----BEGIN RSA PUBLIC KEY-----
...
-----END RSA PUBLIC KEY-----"

RSA_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----"

As a short term fix, a work around is to base64 encode variables with new lines, and then decode them in your container.

cat cert.pem | base64

and then in container

echo -e "$CERT_VAR" | base64 -d > cert.pem

I don’t know if it helps all your use cases, but you can have multi-line values for environment variables in docker-compose:

environment:
    SERVER_NAME: "myserver.doma.in" 
    # Dummy key, cert
    SSL_KEY: |-
        -----BEGIN RSA PRIVATE KEY-----
        MIICXQIBAAKBgQD272jYrLm8Ph5QpMWFcWUO9Ua1EviykalP+tkMIg12yZ3GvezF
        y8aayxdztB5vu68jqMeg6mOJlscWqFUhmAxj4mDknYenVzVX2CKzCgHlGninTKxY
        61rXDaDZVpSZ+XIESJkaB0z9HHYtrSLr0coKmq4cT5TRptOnkpDlJxIRaQIDAQAB
        AoGATcTYoGTFmiN2KK+8BWrRCQT2X9C5woNdb3LxKIEQ/HhC2HS4PRMQWW/c0vPH
        IilZ30EoneUztAFochpRtWLNg4lJoLy04X/eNjEiC/imp0KSwWXmnuLhDcWcb0+M
        slddvzHAAfK1jzIk8zEvcFnhuRoR/L3yWLQp3dIkG07h5IECQQD7xdyhfYMKiYZ7
        HIq9mU0oNaC7UvxHTw3HB4rT3yvqVZXW15JvR64qCe+sOn9xJEesGDkUUxghV+dd
        0GFOODPb2trQ1NGyKzus72JvO61pVpIhT6kVQo3xC543/+1FvU+albjtmqKe1MjZ
        32B6VtNdlgA4VzMC7qrZXwU+oD83WbG6s1GKQa/rXB8wo9moOGcNPP7PmXovDr6F
        zIVbekIj0YuTQdo31aKPNmrTVqd+iOk0LGaUC5zMi0OgDaKKDuP1Ou6ANVTcBXcF
        BnJRQ7XYtjs0oSmvA7bgbTfN4IMJxJ4hYybF1pURh/o4QW4FUytLaF8zghuTc0aP
        yKIsFLzi5ZT34KLVNGAEA44FsG71hOajFkA4c2I/SwU+
        -----END RSA PRIVATE KEY-----

👎 such basic problem could not be solved for so long time! really disappointed!

+1 same here. We store private keys as env variables and this would greatly help us too.

+1 same here. We store private keys as env variables and this would greatly help us

+1 for --env-file handling quoted values; I need to handle some secrets containing newlines.

A pretty compelling use case for .env files is storing encryption keys outside of a code repository. Not being able to include new lines in values means being unable to store PEM encoded RSA keys without implementing some hackish workaround as described above. Moreover, this is inconsistent with the “environment:” section of a Docker compose YML file, where “\n” is actually replaced by a new line. I personally don’t think that having a slighly better parser for env files is unnecessary complexity.

👍 same use case here with wanting to store private keys in env vars. It’s hard to integrate work arounds with third party libraries that expect the env in a certain state. It requires additional tooling to convert vars from the format they can be specified in to the format they are expected to be in. I would rather not create that layer.

And again: storing private keys with multiple lines is needed here too

+1 Same usecase here

+1

Same use case as others: encryption keys being loaded, would prefer to be able to use --env-file for that.

+1 not having multiline support makes 12 factor REALLY hard to achieve and is a barrier to my engineering users running multiple projects with multiple envs. Sad this is closed. Yes we can workaround but that is very confusing for people. (Because it requires additional steps when creating new environment vars).

With Compose, for example, you get this behavior:

web:
 build: .
 environment: 
   TEST: "hello\nworld"

docker-compose run web ruby -e "puts ENV['TEST']"
hello
world

and this is because in YAML, if you have a quoted string with \n in there, they will be interpreted as newlines and not literally:

>>> import yaml
>>> filename = 'docker-compose.yml'
>>> with open(filename, 'r') as fh:
...   yml = yaml.safe_load(fh)
...
>>> yml['web']['environment']['TEST']
'hello\nworld'
>>> yml['web']['environment']['TEST'][5]
'\n'

My ugly hack at the moment to workaround this is to use echo -e to evaluate the line before passing it to docker https://gist.github.com/hudon/149466af21dfc52fdc70