moby: `docker stack deploy` in 1.13 doesn't load `.env` file as `docker-compose up` does

I don’t think the CLI is ever going to add .env support. The docker stack deploy command was originally built with Distributed Application Bundle files in mind (hence the --bundle-file option). DAB files were generated by Docker Compose with the docker-compose bundle command. .env and other Compose features would be handled by docker-compose, before being passed to docker stack deploy. Direct support for Compose files in docker stack deploy was always more of a compromise.

The original promise of DAB files now lives on in Docker App, which also does not process .env.

I have my developers use docker-compose config to pre-process Docker Compose projects before passing them to docker stack deploy. You can do this in one line with:

docker stack deploy -c <(docker-compose config) stack-name-here

That way, all Docker Compose features including .env processing are fully applied.

@whoan I’m using this as a workaround:

env $(cat .env | grep ^[A-Z] | xargs) docker stack deploy --compose-file docker-compose.yml [STACK_NAME]

That way, variables don’t get stuck on the terminal window

The .env support is quite useful in Compose, we used it in many of our compose files. It separates dynamic parts and static parts of docker-compose.yml file. With load .env by default, we can just provides different .env file for different environments, and keeping the docker-compose.yml and related scripts static.

It’s not possible to use env_file or environment in docker-compose.yml to achieve the same envvars substitution result. .env is the most easy way to do it.

Otherwise, we have to prefix export in each line of .env file and manually source .env every time before loading the docker-compose.yml, and the extra steps sometimes are prone to mistake.

This is by design. The .env support is a feature of Compose, not of the file format.

We can discuss adding this for a future release, but I don’t know if it’s really the best option.

EDIT: I hope this post doesn’t come across aggressively. I love the Docker Stack concept and I do think the whole Docker suite is pretty well supported. It just seems that Docker (the organisation) currently doesn’t view its products the same way users do, which is unfortunate and appears to be the cause of most issues I’ve run into using Compose and Stack.

Docker never had this feature; docker compose had; docker compose and docker stack deploy share the file format but not nescessarily all behaviors of the software itself.

I feel like this statement reveals a fundamental disconnect between the way Docker’s developers and users view these products. As a user, whether I’m working with Docker Engine, Docker Compose or Docker Swarm, it’s all Docker and should behave consistently as far as possible.

My understanding of the purpose of these products is:

• Docker Engine is for building and running individual containers
• Docker Compose is for running a suite of containers in development
• Docker Stack is for deploying containers to production

One of the key selling points of Docker (and containers in general) is easy reuse of the same app in multiple environments. Therefore, Compose should be additive to Engine and Stack should be additive to Compose. Particularly as they share the same file format, Stack not supporting features of Compose leads to confusion for developers and added complexity in CI processes.

Just specify the .env file in your docker-compose.yml and it works:

env_file:
  - .env

See https://stackoverflow.com/questions/44694640/docker-swarm-with-image-versions-externalized-to-env-file

This confused me like crazy. My app would come up using docker-compose, but not with docker stack deploy. There should at least be a disclaimer prominently displayed in the docs

But IMO this is a bug, and is degrading the docker experience.

This is ridiculous. I just spent all day trying to get this to work. A disclaimer is at least needed somewhere in the docs that .env is not read by docker stack commands.

sigh

It’s been two years and we are still like that… I don’t want to come across as rude, but come on! It’s matter of reusing a bit of code from docker-compose!

@dnephin : you state that .env is not part of the docker compose file format. However, the .env file feature is documented in the Compose file version 3 reference: https://docs.docker.com/compose/compose-file/#variable-substitution . This gives the impression .env also should work with docker stack deploy. Besides that, as already mentioned by others, separating actual environment specific values from the stack definition yaml is a much wanted/needed feature. Specifically we want to use it to specify image versions and replications.

It’s incredibly dumb that docker doesn’t support this yet.

+1 docker stack deploy should really support --env-file. Myself I’m struggling with re-using the same .yml file to deploy different stacks for development/test/production environments and I don’t want to deal with some fiddly export/sourcing of environment variables prior to deploying the stack. This is very error-prone (and a pain in the a** too to remember the exact commands each time). In my particular case I e.g. want to set different index-names (elasticsearch) per environment - but the rest of the config stays the same. This has nothing to do with “secrets” as mentioned in earlier posts. There seems to be a real need for --env-file for docker stack deploy based on the comment history above but sadly the Docker developers don’t seem to see that.

I am using a bash function as a workaround.

You may adapt it to your needs until the secrets feature be released:

dsd() {
    stack=${1:-${PWD##*/}} # by default, the name of the cointaining folder
    compose_file=${2:-docker-compose.yml}

    if [ ! -f $compose_file ]; then
        echo "Misses compose file: $compose_file" >&2
        return 1
    fi

    # execute as a subcommand in order to avoid the variables remain set
    (
        # export variables excluding comments
        [ -f .env ] && export $(sed '/^#/d' .env)

        # Use dsd your_stack your_compose_file to override the defaults
        docker stack deploy --compose-file $compose_file $stack
    )
}

To summarize, for people like me, who have read this so far:

• docker stack deploy supports “Variable Substitution” but you need to set the environment variables from the .env file in some way. Several options have been explained above. One of the neatest workarounds is using docker-compose config as pre-processor. So you can create your stack with docker stack deploy -c <(docker-compose config) STACK_NAME.
• Although Docker introduced the configs and secrets capabilities, they have their use cases but won’t do the same thing as “Variable Substitution”.
• The env_file parameter in thedocker-compose.yml file will set the environment variables in the created container and not on the docker-compose.yml file itself (and hence it cannot be used as an alternative to “Variable Substitution”).

Update:

I was corrected by @thaJeztah.

Just . .env before calling docker stack deploy does not help, $variables are not substituted. Only env .. xargs trick did help. It would be nice to have --env-file=FILE option like in docker run.

BEWARE of the <(docker-compose -f CUSTOM-COMPOSE-FILENAME.yml config HACK!!! The output of docker-compose config and docker-compose -f myfile.yml config is NOT necessarily the same! The latter sometimes wraps env variables in extra quotes, so the output will be wrong!

Example:

environment:
  SERVER_URL: '"xxx"'

instead of

environment:
  SERVER_URL: xxx

Is this issue tracked somewhere already?

.env file

export MYSQL_USER=user

source .env && docker stack deploy

It will also work with docker-compose up for bash

@andyfinch Alternatively:

#!/bin/sh -eu
set -a
. .env
docker stack deploy -c docker-compose.yml STACK

or:

$ set -a && . .env && set +a
$ docker stack deploy -c docker-compose.yml STACK

set -a adds the export attribute to every variable that gets created. set +a restores the default behavior.

@akhildangore please don’t comment on issues with questions that are not directly related. The docker stack deploy feature, by design does not perform builds. Reason for this is that a build is performed on the host that the command is run from, so the image will only be available on that node. When deploying that image in a Swarm, the service cannot be started on other nodes. To deploy services, make sure the image you’re deploying is pushed to a registry (or for testing; make sure the image is available on every node in the swarm)

@kinghuang Tried your solution, but didn’t work. “docker-compose config” produces the correct output in the same directory, but “docker stack deploy -c docker-compose.yaml my_stack” will not substitute variables from the .env. What am I missing? This is Docker version 18.09.0, build 4d60db4.

docker stack deploy -c <(docker-compose -f docker-compose-frpc-swarm-traefik.yml config) traefik

I would love if .env files were read by default like in Docker Compose and --env-file (short -e) would be added for docker stack deploy to be able to override environment variables and the defaults from the .env file.

docker stack deploy can take a generated file. This works in bash for all my variables and keeps secrets secret as well:

docker stack deploy --with-registry-auth --compose-file=<(ENV1=value1 ENV2=value2 docker-compose -f docker-compose.yaml -f docker-compose.override.yaml -f third-sample-compose.yaml config) stack-name

Please note I added the in-line ENV variables as additional ones or overrides. My compose files reference .env files and this works. Hope this is helpful 👍

The functionality of docker-compose config is what ties this together.

Try this: echo "$(docker-compose -f stack.yml config 2>/dev/null)" | docker stack deploy -c- stack_name

It uses docker-compose to parse the config file, substituting env vars, strips warnings from the output and then pipes it to docker stack deploy via stdin. Replace ‘-f stack.yml’ with the name of your config file or omit it to use the default docker-compose.yml.

Hello everyone, this helped me put env_file in my docker-compose:

env_file: /path/to/env/file not

env_file:
  - /path/to/env/file

i just saw this from another post and I can only speculate why it works (same as your speculation) but we need confirmation as to why this is working as docker explicitly mentioned in the docu that env_file doesnt work.

@cjancsar They do. The documentation says:

Important: The .env file feature only works when you use the docker-compose up command and does not work with docker stack deploy

oh, wow. I joined the confusion and pain while evaluating docker swarm for a big project as well. The secrets are all fun and games, just that not all external docker images support reading secrets from files. I do not want to modify for each of them and place a custom image. Might use them for my projects, yes, but for external projects would be a no go. So that leaves env variables. But hey, they behave differently from compose to stack! Imagine my surprise when I noticed that they don’t really work. How is it possible to have docker stack without env support?best would be in my opinion --env-file=dev.env , so it wouldn’t interfere with other env files… but the support for this is horrible. I must go back to k8s unfortunately, as the stack missing this from the start, could prove to be a horrible choice while in production, who knows how many other things will be missing.

There is a workaround if you don’t want to remember custom commands to do variable substitution in docker stack deploy. Add the following function to ~/.bashrc to the very end of the file and open bash again to enact those changes:

loadenvs() {
    local envfile="${1:-.env}"
    set -a && . $envfile && set +a
}

The script does the following:

1. Declare a function loadenvs
2. Declare a local variable envfile and assign the first argument to the function to it; if there is no argument, use ‘.env’ instead
3. Temporarily set all declared functions as exported; run the envfile with the current shell; revert default exportability back to defaults

Then the usage is very simple:

1. If the env file is “.env” then run loadenvs && docker stack deploy...
2. If the env file name is different, specify it as an argument: loadenvs .custom_env && docker stack deploy...

Please note that if you are using sudo to run docker commands, you need to use -E option to preserve environmental variables in sudo session like that: loadenvs && sudo -E docker stack deploy...

I don’t think the CLI is ever going to add .env support. The docker stack deploy command was originally built with Distributed Application Bundle files in mind (hence the --bundle-file option). DAB files were generated by Docker Compose with the docker-compose bundle command. .env and other Compose features would be handled by docker-compose, before being passed to docker stack deploy. Direct support for Compose files in docker stack deploy was always more of a compromise.

The original promise of DAB files now lives on in Docker App, which also does not process .env.

I have my developers use docker-compose config to pre-process Docker Compose projects before passing them to docker stack deploy. You can do this in one line with:

docker stack deploy -c <(docker-compose config) stack-name-here

That way, all Docker Compose features including .env processing are fully applied.

This works like a charm, keep in mind if your compose file does not have the “docker-compose.yml” name your command should include the actual compose file name.

docker stack deploy -c <(docker-compose -f CUSTOM-COMPOSE-FILENAME.yml config) CUSTOM-STACK

It’s always nice to have the ability to load things from a .env file - apart from a simpler Docker secrets use-case which can now be circumvented, there are always Docker Compose values that you don’t want to have hardcoded in your repo.

Let’s assume you use docker stack deploy to deploy a complete stack and you have the following requirements:

• you want to use the exact same configuration for different environments - say staging and production
• you need to have different scale values for each environment

If you can’t configure this with a .env file, you need to manually change the docker-compose.yml file before deploying the stack each time, otherwise the service scale value will revert back to 1.

You can extend the above with things like networks, configuring the DNS that a service should listen to, etc.

I can draft a PR that will load a .env file, if one exists in the same directory as docker-compose.yml if we believe the above uses cases make sense.

Hello All,

Docker version: v1.13.0-rc4

I am getting error : Ignoring unsupported options: build, Does it mean it will not create build from Dockerfile ?

And also getting same error for network_mode: Ignoring unsupported options: network_mode.

Below is command: DOCKER_IMAGE=akhil123 docker stack deploy -c docker-compose.yml foo

Many thanks in advance.

In docker compose V2 this approach won’t work causing unsupported Compose file version: 1.0 error. (see https://github.com/docker/compose/issues/8950) This happens because now config command trimms version field.

It seems to me that the creators did it on purpose to even take away these hacks we had 🥇 😸

This should be more clear in the documentation. Although it’s called out that .env is not supported in the Variable Substitution section per @lengxuehx, it is not mentioned that env_file is ignored for stack deploys.

Another workaround is: set -a && . .env && set +a && docker stack deploy....

Just ran into the same thing as we are trying out swarm. I’ve read through all of the messages in various threads and am left extremely frustrated. IMO, some really good points and strong arguments were made by @ntwrkguru and others. I hope we can consider them to get this feature in; hacks won’t cut it.

+1 this would be really useful. Don’t see a downside really.

Having environment variable interpolation in deploy is the best solution for continuous integration.

One issue with the use of ‘.env’ is that it overlaps with similar file names used by, for example, Ruby on Rails, having the ability to specify an environment file by argument is superior.

@ntelisil’s suggestions are well taken, they could be supplemented or modified by:

https://unix.stackexchange.com/questions/294835/replace-environment-variables-in-a-file-with-their-actual-values:

You could use envsubst (part of gnu gettext): envsubst < infile

I used:

$ envsubst < docker-compose.yml-template > docker-compose.yml

And that worked fine, but I needed to add 3Mb or so to my container.

For those subscribing to this issue; secrets support for docker-compose files will be included in the 1.13.1 release, which should be not too far in the future

@vovimayhem no decision was made yet on .env file, but you may be interested in https://github.com/docker/docker/pull/30144, which adds support for secrets to the compose file

We need an answer about this

.env file

export MYSQL_USER=user

source .env && docker stack deploy

It will also work with docker-compose up

That would only work for bash and not other shells.

+1

I wouldn’t say compose is for dev and stack for prod, necessarily. I view compose as being for a multi-container application or “system” and stack for multi-host deployments using swarm-mode as the scheduler/orchestrator. Otherwise, your comment is 100% spot on. I also am frustrated (if it weren’t obvious by my other comments) that there is a disconnect between the developers and the users as to how the product(s) are being used.

This still works as an alternative;

$ echo 'BAR=worked' > ./.env

$ export $(cat .env) && docker stack deploy testing -c -<<'EOF'
version: '3'
services:
  simple:
    image: nginx:alpine
    environment:
      FOO: ${BAR}
    env_file: .env
EOF

$ docker inspect testing_simple -f '{{json .Spec.TaskTemplate.ContainerSpec.Env}}'
["BAR=worked","FOO=worked"]

You don’t seem to get it @vovimayhem. I’m not talking about passing envs to containers; that works as expected. What I mean is to use an env_file that is parsed during stack deploy as it was during compose up.

Docker does not support Variable Substitution using docker stack deploy, and non of the proposed solution do not work.

docker stack deploy does support variable substitution, but does not evaluate the .env file; in the example below, HELLO_VERSION is set to linux, and the service will use the hello-world:linux tag (instead of the default hello-world:latest)

HELLO_VERSION=linux docker stack deploy -c docker-compose.yml mystack
Creating network mystack_default
Creating service mystack_hello

docker service inspect --format '{{.Spec.TaskTemplate.ContainerSpec.Image}}' mystack_hello
hello-world:linux@sha256:d073a5775c0b99d653c413161a8ed0e9685061afe697931d30eddf6afeef40f6

That being said, you still can use environment variables using env_file parameter in your docker-compose.yml file

The env_file serves a different purpose (and is the equivalent of docker run --env-file); the env_file option specifies what env-vars to set on the container that’s created, but is not used in the compose file itself (and not used for substituting variables in the compose file before deploying the stack).

Semantics aside, thanks for the update. When I say Docker, I don’t mean necessarily the daemon or engine, but the company that builds the tools. The genesis for the comment about there being little effort really stems from the fact that swarm-mode has been out for over a year and is still WAY behind where we were with a single-node using compose.

I can appreciate that they are different, but use the same file format, but that actually leads to more confusion, and potentially disappoint, when users discover that they have to trade features for using swarm. That and the fact that the DAB files are STILL being listed as experimental. Unfortunately, we made the decision to use swarm-mode in a big project, or I wouldn’t even care. Fortunately, we won’t make that mistake again.

[edit]

If the environment variables are set, docker slack deploy should already interpolate them

Doing this in a CI environment is a royal ass pain. Essentially, I ended up installing compose, sourcing the envvars, washing the compose file through (after pulling the images) and then spitting out a new compose file for use with stack deploy. (And I won’t even start in on how painful it is when :latest doesn’t mean :latest in swarm-mode, since this issue is specifically about variable interpolation.)

Hello All, Today I tried to run docker stack deploy. My docker-compose.yml file:

version: '3'
services:
  simple:
    image: alpine
    environment:
      - FOO: ${BAR}
    env_file: .env

And .env file in the same directory:

BAR=worked

After launch docker stack deploy -c docker-compose.yml test, I inspected container and got this:

$ env
BAR=worked
FOO=
...

Seems that, .env provided in container, but not provided on host machine. It means, that I can use .env file instead of environment section, but I can’t use .env file with Variable Substitution. Docker version: v17.03

I wrote a compose-spec (compose v2) compliant docker-cli plugin replacement of docker stack deploy called deployx using compose-go that also does environment variable interpolation and handles .env files. My hope is this kind of functionality can be added to docker stack deploy, but in the mean time, I hope this plugin solves this issue for some people!

A work around in powershell to use on windows in order to parse the .env file before calling docker:

switch -regex -file “./.env” { “^\s*([^#].+?)\s*=\s*(.*)” { $name,$value = $matches[1…2]; Set-Item “env:$name” $value}}

Greatly inspired by parsing .ini files: https://stackoverflow.com/questions/417798/ini-file-parsing-in-powershell

+1

And that worked fine, but I needed to add 3Mb or so to my container.

If the environment variables are set, docker slack deploy should already interpolate them

compared to Docker being able to do this the way it used to.

Docker never had this feature; docker compose had; docker compose and docker stack deploy share the file format but not nescessarily all behaviors of the software itself.

that there is very little effort going into swarm-mode since the announcement of K8s support

Compose file handling is all done client side, and used both for k8s and swarmkit, so nothing slowing down in this area, other than “there’s only so much we can do at any given time”

For those waiting for that feature; the upcoming release will have support for multiple compose files for docker stack deploy, so if you’re currently using that feature in docker compose: the upcoming docker release will support this as well

I would add that having docker-compose installed only to be able to process these variables can also be quite dangerous actually.

If anyone deploy the stack using docker-compose up -d by mistake instead of using docker stack deploy, it would most probably induce applications errors and file corruption.

+1 It would be quite convenient to read the .env in the first place as docker-compose does. Either by default or just pass the env file as parameter in docker stack command line.

The two already mentioned workarounds (find them also below) seem to work although a little bit ugly compared to the proposed ‘fix’.

echo “$(docker-compose -f stack.yml config 2>/dev/null)” | docker stack deploy -c- stack_name

env $(cat .env | grep ^[A-Z] | xargs) docker stack deploy --compose-file docker-compose.yml [STACK_NAME]

“For other stuff, may I suggest having one compose file per deployed/deployable environment ? Most of the stuff that changes between different environments will be the available servers, different server organization, etc… and thus needing different values on the deploy key contents (placement rules, resources reservations, etc), making using a single file for everything a bit too complicated.”

@vovimayhem Sorry to be blunt, but that doesn’t make nearly as much sense as using a single compose file with variable substitution using an env_file. I’d prefer to use the env_file format as this also makes it much easier for me to provide stacks as a deliverable to my customers and train them to update a single ini-style vars files than to mess with a compose file (which I DON’T want).

Just . .env before calling docker stack deploy does not help

This should help @C-Pro - you need to have export ... lines in your .env file. Alternatively, you can do export $(cat .env), which would work for most cases.

This is my aproach until I found this thread:

deploy.sh:

#!/bin/sh
export $(cat .env) > /dev/null 2>&1; 
docker stack deploy -c docker-compose.yml ${1:-STACK_NAME}

. deploy.sh <stack_name>

(stack_name optional if previously defined in .env)

+1 @joao-fidalgo It definitely is.

I can’t believe this isn’t possible. We would like to keep a single stack file and be able to change deployment options using dynamic variables.

This would work but is ugly and a hack:

echo "$(docker-compose -f stack.yml config 2>/dev/null)" | docker stack deploy -c- stack_name

Thanks, but any word on simply supporting variable substitution from a file?

[edit] This shouldn’t be an issue (but could be) as I’m using Ansible to perform the task with become: yes, but the user is still the same user who sources the environment vars.

@ntwrkguru https://docs.docker.com/engine/swarm/configs/

I came across this issue when looking for a solution to my .env not being read in by docker stack deploy. I was a bit surprised to see it wasn’t supported (docs could be clarified), but want to add my support for .env, like compose.

My use case is that I use docker stack deploy in dev, test, staging environments and use environment variables for stack options to keep the yml file the same.

Currently using one of the workarounds posted here but using a .env file would be the preferred route.

Here is what we do, it works very well on Docker Swarm cluster.

1. Put all your environment variables in a .env file like below.

$ cat STG.env
export ENVIRONMENT=stg
export DOMAIN=stg.example.com
export NGINX_PORT=8080

1. Run below commands to create Docker Swarm stack.

$ source STG.env
$ envsubst < stack.yml | sudo docker stack deploy -c - stg

Hope it helps.

On a side note, docker compose? I don’t see such command in Docker 20.10.6. What is admirable about it?

In the docs, it’s still listed as a ‘technical preview’ but some of the talks at DockerCon gave the impression it was now more mainstream than that. They were (confusingly) calling it Compose v2, but that’s nothing to do with the version numbers in the docker-compose.yml file.

The basic idea was to rewrite docker-compose in Go and include it in the main CLI, partly for speed but partly so it always had access to the latest facilities in the engine/CLI.

There are still some differences, but I’ve been using it all day for the last couple of days on my Mac with no issues other than those I mentioned above.

More info in this talk, if it’s accessible: https://docker.events.cube365.net/dockercon-live/2021/content/Videos/WGpFvvKmhhwvY52va

Quentin

Out of interest. What do people currently do when in swarm mode, and they want to update their newly built docker image from, e.g. application:v1 to application:v2, same service (nothing else changed)?

This is the reason I came to this issue. That is why I am interested.

Currently in docker-compose, it is trivial to get automation to interact with source-control untracked file . env to update the image version, then docker-compose up -d

Often I use the commit sha as a docker image version for ‘staging’ deploys (git tags used for actual releases after sufficient testing cycles), so it isn’t really possible to commit the new image version to source control.

Is it possible to update a docker secret (even though it isn’t really a secret) as a workaround? Is that what people do, or is there something else that is done.

I am asking for future visitors of this issue, so that they actually come away with a current best practice, since the functionality they were looking for doesn’t (yet?) exist.

+1

Nice @rnickle, but still “hacky” compared to Docker being able to do this the way it used to. It seems (and please Docker dudes correct me if I’m wrong) that there is very little effort going into swarm-mode since the announcement of K8s support, so I’m doubtful that any of these regressions will be addressed.

A year later and we still have to hack this to work. As of 17.09.1, docker stack deploy -f compose.yml won’t even substitute using known shell vars.

see How to keep Environment Variables when Using SUDO

$ export PORTAINER_VER=1.14.3
$ printenv | grep PORTAINER
PORTAINER_VER=1.14.3

$ sudo printenv | grep PORTAINER

$ sudo -E printenv | grep PORTAINER
PORTAINER_VER=1.14.3


$ cat > docker-compose.yml <<'EOF'
version: '3'
services:
  portainer:
    image: "portainer/portainer:${PORTAINER_VER}"
EOF


$ sudo -E docker stack deploy -c docker-compose.yml foobar
Creating network foobar_default
Creating service foobar_portainer

$ sudo docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                        PORTS
vt4h0g8yygcg        foobar_portainer    replicated          1/1                 portainer/portainer:1.14.3   

A year later and we still have to hack this to work. As of 17.09.1, docker stack deploy -f compose.yml won’t even substitute using known shell vars.

$ printenv | grep PORTAINER
PORTAINER_VER=1.14.3

portainer:
    image: "portainer/portainer:${PORTAINER_VER}"

$ sudo docker stack deploy -c /tmp/docker/jedi-core.yml --with-registry-auth --prune --resolve-image always jedi-core
Updating service jedi-core_redis_server (id: 0csjyandro713y13ncitk6c0k)
Updating service jedi-core_api-gateway (id: gzcz2eturuxqkjojsc9e6954l)
Updating service jedi-core_auto_results_api (id: tw43c6e0x98q6f0m2g0zttwhf)
Updating service jedi-core_auto_results_api_mongo (id: qpwpypyzd9aigpa71xgxxdzcp)
invalid reference format

Any variables for tags will produce invalid reference format.

Thanks @djmaze, but that’s ultimately a ridiculous thing to need to do when there’s inherent version tracking built in to every source control system. Also, thanks for the link; I commented there as well. I don’t see why Docker cares about the version other than the one that’s current. The only reason I could see for needed to version these is for rollback, but tons of things need to be tracked there as well anyway, so what’s a couple more? (says the guy that doesn’t have to write the code) 😃

@thaJeztah , indeed it does. I have a task in my playbook to source an environments file, but that’s missing the bigger point. This used to be possible with compose and is now not possible with stack. IMO, that’s a regression. Also, in your example, you’re still passing an environment variable into the container. I can do that with the env_files today; the issue is parsing the env_file during the stack deploy -c docker-compose.yml operation.

This is especially painful since stack no longer supports multiple compose files, else one could simply use a secondary compose file per environment. To do it that way requires one to build the stack file from the compose files, introducing another step. So, either way you look at it, we went from a single-step process to a multi-step process to accomplish the same end result.

Also, FWIW, the configs is really subpar in my experience for actual configs. I was envisioning (hoping) that I could have /application/config.conf in source. Update that file, commit, push and CI would re-deploy the stack with the new config. That’s not possible without some hackery to change the file name or some other mechanism. Surely we can checksum the file, at a minimum, or scan for content changes and update based on the file itself? All in all, it just seems like we’r’e going backwards. Throw in the support for K8s now and it makes me wonder if swarm will just languish on the vine until it loses enough usage that it’s quietly dropped.

Just a friendly reminder: There might be the possibility your’e trying to use dotenv files to store API keys, passwords & other stuff considered to be “sensitive” to put into the cluster. For these cases, you should be using Docker Secrets.

For other stuff, may I suggest having one compose file per deployed/deployable environment ? Most of the stuff that changes between different environments will be the available servers, different server organization, etc… and thus needing different values on the deploy key contents (placement rules, resources reservations, etc), making using a single file for everything a bit too complicated.

I’ve just noticed this. I assumed/expected it to work in the same way it does for Compose, but is not the case for docker deploy.

In my particular case, I’m was expecting to use the .env file to store sensitive data (passwords, API keys, etc) used in the services I’ll be creating in the stack:

version: '3'

volumes:
  data:
    driver: local

networks:
  backend:
    driver: overlay

services:
  rabbitmq:
    image: rabbitmq:${EXCHANGE_RABBITMQ_TAG}
    volumes: [ "data:/var/lib/rabbitmq" ]
    logging: { driver: gelf, options: { gelf-address: "udp://0.0.0.0:12201" } }
    networks: [ "backend" ]
    ports: [ "15672:15672", "5672:5672" ]
    environment:
      RABBITMQ_DEFAULT_USER: ${EXCHANGE_RABBITMQ_USER}
      RABBITMQ_DEFAULT_PASS: ${EXCHANGE_RABBITMQ_PASS}
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.labels.queue-host == true

While this Compose file will be checked in on Git, the .env file would be ignored.

I’ve followed the whole stuff/history of the *.dab vs compose files, and I feel you guys are trying to avoid something - or proposing a better solution - but I lost track of the whole discussion…

If anyone is looking for a powershell script to replace the environment variables in a docker-compose.yaml file, here your go:

# Define the path to env file as parameter as well as the path to docker-compose.yaml
param(
    [string]$envFilePath = ".env",
    [string]$dockerComposeFilePath = "docker-compose.yaml"
) 

# Check if the .env file exists
if (Test-Path $envFilePath) {
    # Read the contents of the .env file
    $envVariables = Get-Content $envFilePath

    # Loop through each line in the .env file
    foreach ($line in $envVariables) {
        # Split the line into variable name and value
        $parts = $line -split "="

        [System.Environment]::SetEnvironmentVariable($parts[0], ($parts[1..($parts.Length - 1)] -join "="), [System.EnvironmentVariableTarget]::Process)
    }
} else {
    Write-Host "The .env file does not exist."
}

# Input string containing references to environment variables
$inputString = Get-Content $dockerComposeFilePath -Raw

# Regular expression pattern to match environment variable references
$pattern = '\${([a-zA-Z_][a-zA-Z0-9_]*)}'

# Replace environment variable references with their values
$outputString = [regex]::Replace($inputString, $pattern, {
    param($match)

    $envValue = [System.Environment]::GetEnvironmentVariable($match.Groups[1].Value)
    if ($envValue -ne $null) {
        return $envValue
    } else {
        return $match.Value
    }
})

# Output the modified string
Write-Output $outputString

You can even pass the name of the .env and docker-compose.yaml file (with -envFilePath and -dockerComposeFilePath)

For people who would like to supply a custom .env file like .env.prod you can use the following: (Combining @kinghuang and @vovimayhem answers)

env $(cat .env.prod | grep ^[A-Z] | xargs) docker-compose config | docker stack deploy -c - stackname

To do this with a local registry, don’t forget to initialize a local registry with a given port (f.e.5000) and tag your services in the docker-compose.yml file with:

image: 127.0.0.1:5000/<parent_folder>_<service_name>

Then build with docker-compose and push them to the registry as explained here: https://docs.docker.com/engine/swarm/stack-deploy/

It’s worth noting, BTW, that the new, non-hyphenated docker compose, while admirable in many ways, doesn’t quite give the same output as docker-compose when run with the config command. (I noticed that variable substitution was slightly different, and the ‘version’ line was missing.)

So, even though I’ve switched to docker compose for most other things, I’m sticking with the hyphen for this particular case.

As per many I hit this issue and found this. A solution that has worked for me is as follows: Create .env (or any other file name) with your variables:

MYVAR1=foo
MYVAR2=bar

Create startmystack.sh script containing the stack command required:

source .env
export MYVAR1 MYVAR2
docker stack deploy --compose-file my-docker-composes.yml mystack

Variable picked up using source from .env, exported and then are available for substitution in the compose file for Swarm.

This should be more clear in the documentation. Although it’s called out that .env is not supported in the Variable Substitution section per @lengxuehx, it is not mentioned that env_file is ignored for stack deploys.

I can confirm that env_file is used and working for deploy stack command, exactly as the docker compose does.

docker version
Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf5
 Built:             Thu Oct 17 23:44:48 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf5
  Built:            Thu Oct 17 23:50:38 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

@kinghuang Only the root environment will not report an error.

Another similar problem I just ran in to is that even if you use env_file, the behavior is still not exactly the same.

I use env_file to load paths specified from the project root, but my compose file is in a different subdirectory. To launch with docker-compose, I just use --project-directory . -f path/to/docker-compose.yml

docker stack does not allow me to specify the project-directory, causing all of my env_files to fail to load.

Changing env_file to “…/…/path/to/envrc” does not work with docker-compose, as these files must be inside the project root dir

I’ve been messing with this for many hours and looks like doing some background script work (I’ll be using CI to do this) to create a DAB file from these various pieces. It would be nice to be have a docker tool that would do this… 😃 Maybe add a --env-file version.env added to docker-compose bundle?

In my use case, I want to use a version.env “manifest” to track and control the versions of containers/images that comprise a platform. My process looks like this:

• Update .env file
• Source .env file to populate the shell vars
• Run docker-compose -f stack.yml config > docker-compose.yml
• Run docker-compose pull to register the image digests
• Run docker-compose bundle -o stack.version.dab DAB file

At this point, ideally I would be able to docker deploy --host manager.swarm.com --bundle-file stack.version.dab --with-registry-auth --resolve-image always stack-name, but I get that it’s not possible now. In this case, I would scp the DAB file to the manager and execute the docker deploy.

Is this the workflow Docker had in mind @thaJeztah? Is there a better way?

[edit] This does not work since a DAB ignores volume:, network:, and deploy: directives. I’ve taken to simply sourcing an env file to the shell, rebuilding a temp compose file, then deploying that.

+1

Thanks, I opened https://github.com/docker/docker.github.io/issues/3654 to fix the docs

@realcbb Yes. But for using .env file variables inside container, you must remove their in environment section. For example: docker-compose.yml

...
environment:
  - FOO=${FOO:-empty}
...

.env

FOO=filled

In container I got this:

$ env
FOO=empty

But if I remove FOO from environment section in docker-compose.yml, I got this:

$ env
FOO=filled

The design of docker stack deploy is a bit different from docker-compose commands. I don’t think it makes sense to just read a .env by default. Not even the docker-compse.yml is read by default, because in the future the default deploy source will likely be something else.

You can always source an .env file yourself before running stack deploy.