compose: Inconsistent env var parsing from docker-compose.yml

PR added - please show it some love if you want this fixed!

This has caused me grief as well…

@dnephin I agree with your break down, the problem is that the resulting behaviour is as if compose is setting export TESTVAR="\"test\"". It’s confusing and unexpected, which is compounded by the fact things just silently don’t work as expected (because values don’t match up).

If this really is the intended behaviour, it should be documented.

I’m probably beating a dead horse here, but this has really caused some major confusion in my scenario as well. I’ve created an example to demonstrate all the possible ways you can screw up environment variables in docker-compose. I’m using it as a reference to get the correct behavior as it is unpredictable in my opinion.

Here’s my docker-compose.yml (the command just prints out the values of the env vars):

env-array:
    image: python:3
    volumes:
        - ./test.py:/test.py
    command: /test.py
    environment:
        - "TEST1=this is a normal string"
        - "TEST2=this is a $UNQUOTED string"
        - "TEST3=this is a $QUOTED string"
        - TEST4="this is a $UNQUOTED string"
        - TEST5="this is a $QUOTED string"
        - TEST6=this is a $UNQUOTED string
        - TEST7=this is a $QUOTED string

env-map:
    image: python:3
    volumes:
        - ./test.py:/test.py
    command: /test.py
    environment:
        TEST8: this is a normal string
        TEST9: this is a $UNQUOTED string
        TEST10: this is a $QUOTED string
        TEST11: "this is a $UNQUOTED string"
        TEST12: "this is a $QUOTED string"

and my .env file:

UNQUOTED=unquoted
QUOTED="quoted"

Let’s start the betting to see which tests come out with no quotes at all in the values themselves 🎲 🎲

Results:

$ docker-compose run --rm env-array
TEST1: [this is a normal string]
TEST2: [this is a unquoted string]
TEST3: [this is a "quoted" string]
TEST4: ["this is a unquoted string"]
TEST5: ["this is a "quoted" string"]
TEST6: [this is a unquoted string]
TEST7: [this is a "quoted" string]

$ docker-compose run --rm env-map
TEST8: [this is a normal string]
TEST9: [this is a unquoted string]
TEST10: [this is a "quoted" string]
TEST11: [this is a unquoted string]
TEST12: [this is a "quoted" string]

Test 1 and 8 are controls, so they don’t count. That leaves only test 2 and 6 from the array syntax, and 9 and 11 from the map syntax:

# Acceptable array syntax:
- "TEST2=this is a $UNQUOTED string"
- TEST6=this is a $UNQUOTED string

# Acceptable map syntax:
TEST9: this is a $UNQUOTED string
TEST11: "this is a $UNQUOTED string"

Particularly troubling was test 5, which ended up with a quoted word inside a quoted sentence, effectively breaking out of the quotes, ala SQL injection.

My biggest complaint is that docker-compose does not allow you to quote values in .env files. This flies in the face of all the dotEnv implementations that I’ve seen, across multiple languages, and breaks compatibility with my applications, since they can no longer share information with Docker via .env files reliably. For now I’m just going to keep trimming quotes from env vars, but I really think this should be addressed.

This is also unintuitive behavior in env_file. Suppose you want to set PIP_TRUSTED_HOSTS with more than one host. In docker-compose’s env_file syntax, you would have to write

PIP_TRUSTED_HOSTS=pip1 pip2

In shell, this would be taken as export PIP_TRUSTED_HOSTS=pip1 for the command pip2. Since the shell is already so finicky with its quotes-handling, I beg you not to diverge too much from it in env_file.

dotenv is a specific format used by more than just docker-compose. The dotenv format allows quoted values. The expected behavior of dotenv files is bash-like.

So in my opinion, docker-compose should make one of two possible changes: A) Properly handle dotenv files with quoted values just as source .env works in bash B) Use a different format that is intentionally distinct from .env, e.g. yaml, toml, ini, or similar.

The current behavior seems totally incorrect and surprising, considering what people expect. I don’t think the expectations are unreasonable either, since the env_file is used to set environment variables in a running container.

it appears i’m having problems due to docker-compose keeping quotes as part of the value when using the env_file. considering (as noted previously) most/all other env file implementation do allow quoting of the values, will this be fixed/changed at any point?

per the comments in the thread i can understand (even if i don’t agree with) the rationale for not “altering the expected behavior of yaml” , however since we have both environment and env_file, maybe have env_file allow quoting of the values? this would both keep the spirit of the yaml handling and give others an avenue for more proper dotenv handling, though i can see some additional documentation being needed for the difference between them.

this issue has been hanging open for a few years, so if this isn’t going to happen, maybe it should get an official response and be closed.

@cerw When 1.26 releases, it will have support. The feature was already merged in.

If you would support changing the behavior to clean quotes from values in .env, I’m happy to submit a PR.

I’m not a native Python speaker, but this should do the trick:

(k, v) = env.split('=', 1)
if len(v) >= 2:
    if v[0] == v[-1] and v[0] in ['"', "'"]:
        v = v[1:-1]
return (k, v)

right here: https://github.com/docker/compose/blob/7ae632a9ee7530fcf81e212baa3e588f477ea862/compose/config/environment.py#L21

A possible temporary solution is to run this which removes the quotes sed -r 's/^([^=]+=)\"(.*)\"$/\1\2/g' .env > .env.docker and just load .env.docker in env file for docker compose till this lands.

Too bad the fix didn’t make it into the cli. Same issue using --env-file with docker run.

Any updates on this?

Hello everyone.

I’m really sorry to hear that you are having problems/doubts with env_file parsing. About the topic, my concern is that the issue was opened in 2016 and since then, the entire mechanism of env_file parsing was delegated to python-dotenv and now the behavior is more “standardized”.

Also it is very hard for us to know if all of you refer to the same issue.

Given that, I invite you to update docker-compose to the latest version and retry it. If the problem persists, please open a new bug with the clearest case of reproduction that you can and we will be happy to triage and fix it when it’s a real bug.

Thank you all for the feedback!

.env files simply may need quotes due to special characters in some of the settings, and they work just fine in just about any environment. Given that docker-compose supports external .env files, they should be parsed accordingly - as external files, with their own schema. So docker-componse should be interested in the key/value pairs on those files, and be aware the values may or may not be in quotes, rather than blindly reading them, assuming the files adhere to it’s own preferred schema.

Obviously, that’s a biased view, but it feels cleaner then asking for docker-compose specific .env files, which just would end up in a mess.

So, whats the proper way if I wanna env file which I can set -a;source env;set +a work in bash still work in deploy? any idea?

just mount the file inside the container and run source directly?

I think this is working as expected.

In the docker example above, bash is interpreting the quotes for you, so the application never gets "test", it gets tests.

In the compose example using map, yaml is interpreting the quotes for you, so Compose never sees the quotes.

In the list of strings example: TESTVAR="test" is interpreted by yaml as the literal string 'TESTVAR="test"'.

I don’t think compose should alter the expected behaviour of yaml, and remove quotes from the string.