portainer: `stack.env` does not work

Bug description

Following along with the tutorial here: https://www.portainer.io/blog/using-env-files-in-stacks-with-portainer I get stack.env: no such file or directory

Expected behavior

stack.env should work as per the above blog post.

Portainer Logs

level=info msg="2022/03/28 12:47:58 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:47:57 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:52:49 http error: open /data/compose/57/stack.env: no such file or directory"
level=info msg=" (err=open /data/compose/57/stack.env: no such file or directory"
level=info msg=") (code=500)"
level=info msg=" (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:49:46 http error: open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:53:58 http error: open /data/compose/58/stack.env: no such file or directory"
level=info msg=" (err=open /data/compose/58/stack.env: no such file or directory"
level=info msg="2022/03/28 12:49:31 http error: open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:48:45 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:48:25 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"

Steps to reproduce the issue:

Be running in swarm mode.

Create a new stack with the content.

version: "3.8"

services:
  test:
    env_file:
      - stack.env

Add at least one environment variable to the environment variables section.

Try to deploy.

Technical details:

  • Portainer version: Portainer CE 2.11.0
  • Docker version (managed by Portainer): API version 1.41
  • Platform (windows/linux): linux
  • Command used to start Portainer (docker run -p 9443:9443 portainer/portainer): Uhh, it’s been running for a while as a stack…
  • Browser: Firefox
  • Use Case (delete as appropriate): Using Portainer at Home
  • Have you reviewed our technical documentation and knowledge base? Yes

Additional context

Assessing the possibility of running a dockerized mail server through portainer for my employer.

About this issue

  • Original URL
  • State: open
  • Created 5 years ago
  • Reactions: 29
  • Comments: 83 (8 by maintainers)

Commits related to this issue

Most upvoted comments

to me, the error reason was the path of docker-compose.yml(using git repository), seted on Compose path field

Compose path: /some-project/docker-compose.yml

Before:

version: '3.4'
services:
  some-project:
    env_file:
      -  stack.env
    build:
      context: .
    image: some-project
    restart: always
    ports:
      - '8080:80'

After(Working): returning to the root directory with …/stack.env

version: '3.4'
services:
  some-project:
    env_file:
      - ../stack.env
    build:
      context: .
    image: some-project
    restart: always
    ports:
      - '8080:80'

Obs: I know it doesn’t answer your question, but it could be related.

Tried using local (docker.sock) environment and Portainer agent environment. Both give errors saying stack.env can’t be found.

With portainer agent, it worked after trying the suggestion from JuniorC07. changing the path of stack.env to ../stack.env

how easy you ask. I asked for chatgpt to create that parser and it did it, without any mistakes at least for my use case. maybe this will help someone, just run ./docker_swarm_easy_fix.py docker-compose.yml stack.env should return the file with the variables pasted in it should support multiple files like ./docker_swarm_easy_fix.py docker-compose.yml stack.env stack2.env but who knows just asked chatgpt to create the parser.

docker_swarm_easy_fix.py
#!/usr/bin/env python3
import sys
import os
import yaml

def parse_env_file(filename): “”" Parse an .env file into a dictionary. Lines starting with ‘#’ or blank lines are ignored. “”" env_vars = {} with open(filename, ‘r’) as f: for line in f: line = line.strip() if not line or line.startswith(‘#’): continue if ‘=’ in line: key, value = line.split(‘=’, 1) env_vars[key.strip()] = value.strip() return env_vars

def convert_environment_to_dict(env): “”" Convert the service’s environment section into a dictionary. The environment section can be a list (e.g. [“VAR=val”, …]) or a dictionary. “”" env_dict = {} if isinstance(env, list): for item in env: if isinstance(item, str): if ‘=’ in item: key, value = item.split(‘=’, 1) env_dict[key] = value else: env_dict[item] = None elif isinstance(item, dict): env_dict.update(item) elif isinstance(env, dict): env_dict = env.copy() return env_dict

def main(): if len(sys.argv) < 3: print(“Usage: {} <yaml_file> <env_file1> [<env_file2> …]”.format(sys.argv[0])) sys.exit(1)

yaml_file = sys.argv[1]
env_files = sys.argv[2:]

# Load all provided .env files into a map using their basename.
env_files_map = {}
for env_file in env_files:
    basename = os.path.basename(env_file)
    try:
        env_vars = parse_env_file(env_file)
        env_files_map[basename] = env_vars
    except FileNotFoundError:
        print(f"Error: file {env_file} not found.")
        sys.exit(1)

# Load the YAML file.
try:
    with open(yaml_file, 'r') as f:
        data = yaml.safe_load(f)
except Exception as e:
    print("Error loading YAML file:", e)
    sys.exit(1)

# Process services if the YAML has a "services" section.
if 'services' in data:
    for service_name, service in data['services'].items():
        if 'env_file' in service:
            # Support both a single string or a list of env files.
            env_file_entry = service['env_file']
            if isinstance(env_file_entry, str):
                env_file_list = [env_file_entry]
            elif isinstance(env_file_entry, list):
                env_file_list = env_file_entry
            else:
                env_file_list = []

            # Get or create the environment section.
            current_env = service.get('environment', {})
            current_env = convert_environment_to_dict(current_env)

            # For each env file referenced in the service, add its variables.
            for env_filename in env_file_list:
                key = os.path.basename(env_filename)
                if key in env_files_map:
                    env_vars = env_files_map[key]
                    # Merge variables; env file variables may override existing keys.
                    current_env.update(env_vars)
                else:
                    print(f"Warning: env file '{env_filename}' referenced in service '{service_name}' was not provided.")
            # Save the updated environment and remove the env_file key.
            service['environment'] = current_env
            del service['env_file']
else:
    print("Warning: no 'services' key found in the YAML file. Nothing to fix.")

# Output the updated YAML.
# You can redirect this output to a file or overwrite the original YAML.
print(yaml.dump(data, default_flow_style=False))

if name == “main”: main()

I just got hit with this. The weird thing was the /data/compose/X/v1/stack.env (or v2, v3, etc) exists, but if I copy it to /data/compose/X/stack.env then I can deploy the stack. I’ve been using portainer for ages and this just happened to me today.

Same here 😦

My local /data/compose directory contains no single stack.env file. I am using the web editor which states “stack.env is created automatically”. This is obviously a lie.

Hello guys, This issue happens because you are indicating there is a .env file to include in your composer code.

It is not needed at all to indicate you have a “.env” file, if you click on the option to import variables from env file, all the variables in the file will be included so you don’t need to include the env file on the code, I’m posting this from my phone, but let me know if you need a video example

My workaround is manually insert stack.env file inside portainer’s docker volume then i updated related docker stack and now it works.

Screenshot 2023-02-13 at 10 16 36

Screenshot 2023-02-13 at 10 20 21

Seeing the same behavior on EE 2.12.1

Any updates on this? It is reproducible for me in Portainer Business Edition 2.18.3

I am also experiencing the same issue running Community Edition 2.18.1.

I have tried using “.env” & “stack.env” and it is still throwing errors.

I think there is very much confusion in this thread as there are many different constellations available:

  • Docker Standalone vs. Docker Swarm (and maybe even Kubernetes)
  • Deployment using Web editor/Upload/Custom template vs. Deployment via Repository

I’ll try to explain some things (and in turn also make it clearer for myself).

Docker Standalone

Deployment using Web editor

Defining environment variables:

  1. You can either specify as many environment variables as you’d like manually using the Add an environment variable button.

    OR

  2. Use the Load variables from .env file button to upload an existing env file from your local computer. Portainer then reads this and puts all the values in the web form.

In both cases the result is the same (expand to view screenshot).

grafik

Using the environment variables:

You can now choose between using them separately or as a whole stack.env.

  1. Using them separately: Specify the environment variables like in this example compose file:

    services:
      test-service:
        image: ubuntu
        command: bash
        stdin_open: true
        tty: true
        environment:
          - FOO=${FOO}
      service-with-user-and-password:
        image: ubuntu
        command: bash
        stdin_open: true
        tty: true
        environment:
          - USERNAME=${USERNAME}
          - PASSWORD=${PASSWORD}
    

    (command, stdin_open and tty are only needed to prevent the container to automatically exit when using the ubuntu image)

    If you inspect the containers you will see that test-service only knows about FOO=bar and service-with-user-and-password will only know about USERNAME=itsame and PASSWORD=mario.

    Inspection results:
    $ docker inspect <stackname>-test-service-1 | jq '.[].Config.Env'
    [
      "FOO=bar",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
    $ docker inspect <stackname>-service-with-user-and-password-1 | jq '.[].Config.Env'
    [
      "USERNAME=itsame",
      "PASSWORD=mario",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
    
  2. Using them as a whole stack.env:

    services:
      service-with-all-env-variables:
        image: ubuntu
        env_file: stack.env
    

    If you inspect the container you will see that it knows about all three variables.

    Inspection results:
    $ docker inspect <stackname>-service-with-all-env-variables-1 | jq '.[].Config.Env'
    [
      "PASSWORD=mario",
      "FOO=bar",
      "USERNAME=itsame",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
    

Deployment using Repository

As stated in the Environment variables section, a file named stack.env must be present in your repository. Consider the following Git repository structure (for file contents see the example repo):

$ ls -l
total 8
-rw-rw-r-- 1 kevin kevin 115 Sep  6 14:44 docker-compose.yml
-rw-rw-r-- 1 kevin kevin  39 Sep  6 14:45 stack.env

In this case you can only use the variables as a whole stack.env. Using them separately as with the Web editor example seems not to be possible.

Inspection results:
$ docker inspect <stackname>-all-together-1 | jq '.[].Config.Env'
[
  "USERNAME=itsame",
  "PASSWORD=mario",
  "FOO=bar",
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
$ docker inspect <stackname>-single-not-working-1 | jq '.[].Config.Env'
[
  "FOO=",
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]

Docker Swarm

This one is special. As far as I understood the documentation and according to my testing, this is only possible if you are deploying via Repository.

Example Repository deployment:

grafik

Inspection results:
$ docker inspect test_all-together.1.<uniqueid> | jq '.[].Config.Env'
[
  "FOO=bar",
  "PASSWORD=mario",
  "USERNAME=itsame",
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
$ docker inspect test_single-not-working.1.<uniqueid> | jq '.[].Config.Env'
[
  "FOO=notbar",
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]

The results show that defining new environment variable values in the web editor doesn’t alter the in-repo stack.env file. That’s why all-together still has the original values.

And single-not-working doesn’t use the stack.env file at all (see also the “Deploy using Web editor > Using whole stack.env” above). It is using the variables defined in the web interface, which in this case has FOO=otherfoo.

When instead trying to use stack.env in a deployment via Web editor, we get the famous error: grafik

To me, using stack.env in Swarm mode doesn’t seem feasible or to bring any benefit. I would rather stay at defining the environment variables separately in the services.

It seems Portainer itself is checking the existence of env_file entries. Since Portainer is running as a container, it can’t find the files unless you add a volume to the host file.

Example solution: Your stack:

version: '3.9'

services:
  foo:
    image: foo:latest
    env_file:
      - /opt/portainer-config/my-stack-config/foo.env

Portainer stack:

# ...

  portainer:
    image: portainer/portainer-ce:2.18.4
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - "9443:9443"
      - "9000:9000"
      - "8000:8000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - portainer_data:/data
      - /opt/portainer-config:/opt/portainer-config:ro # Paths must match!
# ...

Important:

  • The path inside portainer MUST equal the path on the host.
  • The .env file:s must exist on every single node on which the stack may run. Think about an cifs volume or something similar.

What fixed it for me is in the compose web editor in Portainer, I changed ALL instances of:

env_file:
      - .env

to

env_file:
      - stack.env

I am using Portainer Community Edition 2.18.3. You can’t just change one .env to stack.env, or it will still throw errors. You have to change all of them.

Hello, just a couple more, hopefully useful, use cases 😃

Using “stack.env” does not currently work when:

  • Using Portainer CE 2.16.1
  • Using a Docker Standalone environment
  • Trying to deploy a stack through a git repository.

It does work, however, by using “…/stack.env” as path instead.


Using “stack.env” on the other hand, works just fine when:

  • Using Portainer CE 2.16.1
  • Using a Docker Standalone environment
  • Trying to deploy a stack directly with Web editor.

same issue here

@fidelix You are right. I just tried to summarize the main things I found here, in the docs and in my testing.

I believe this issue is a bit too convoluted by now and (maybe also due to this) doesn’t seem to get much attraction by Portainer development team. Maybe this issue should be converted into a discussion and a new clean issue should be opened, with only relevant information.

Seems like a bug. Trying to deploy on CE v2.19.4, and the Stack YAML editor says To reference the .env file in your compose file, use ‘stack.env’., but doing just that causes open /data/compose/3/stack.env: no such file or directory and it is not possible to update/deploy the stack.

I can see the problem. When deploying a stack using web editor you can define Env Vars in Portainer UI and refer to a stack.env file. Portainer then creates a stack.env file along with the compose file and writes all the vars to that.

This is not the case when deploying from Git, if you refer to stack.env it is expected to be in the repo. We clone the repo but do not add any files to that location as it will get overridden at next update.

We should however clarify this in the documentation.

Tried using local (docker.sock) environment and Portainer agent environment. Both give errors saying stack.env can’t be found.

With portainer agent, it worked after trying the suggestion from JuniorC07. changing the path of stack.env to ../stack.env

I am experiencing the exact opposite on Portainer 2.16.2, Docker Standalone environment connected through portainer/agent:latest:

  • env_file: stack.env works
  • env_file: ../stack.env -> file not found

This thread and the documentation could really use some clarification on this.

I am also experiencing the same issue, referencing ‘stack.env’ as per the blog post results in an error:

open /data/compose/89/stack.env: no such file or directory

EE 2.14.0, docker swarm mode.

This is a bit confusing, as I’m fairly certain I am following the blog post instructions correctly.

Ditto on EE 2.12.1, running on a swarm

When I go into the portainer vol, the compose folder that it’s looking for doesn’t exist. If I create the stack and then add env_file to the compose, it’s never created in the compose folder created for the id.

If I create an empty file in that folder the stack is able to be updated with the env_file definition, but the file isn’t updated with the env set in the gui.

Still not working with Portainer Business Edition 2.21.4

Ah, I just saw that the documentation clearly states that using stack.env in Swarm mode is not possible:

Note: Using env_file to define a file does not work in Docker Swarm due to the lack of env_file support in docker stack deploy (used on Swarm environments to deploy your stack). On Docker Swarm, you will need to define each environment variable manually.

It’s just not very visible, as it is not highlighted like the other Notes. Additionally it’s only mentioned in the “Option 1: Web editor” section (the other Note is repeated in every section).

“you can also take note of the missing file error, and recreate the path adding +1 to the path in the portainer container files”

what do you mean add +1, can you give an example?

Yes and sorry for not including it originally.

When a new stack is deployed, it gives a number to the path:

/data/compose/3/stack.env: no such file or directory

So I noticed that if you wish to redeploy it with the env file on the server, you need to place the env file under the next one:

/data/compose/4/stack.env: no such file or directory

I hope this helps

@christianstrauch, @ecker00 and @voslucas : Did you tried this? https://github.com/portainer/portainer/issues/6701#issuecomment-1704036982

Portainer itself is running as a docker service, which means that you need to add volumes that matches the host path. Otherwise portainer can not “see” the files 🙂

volumes:
  - /opt/portainer-config:/opt/portainer-config:ro # Paths must match!

…

  • The path inside portainer MUST equal the path on the host.

Thank you! I didn’t even think about the fact that portainer itself is a container and the fact that by default it might assume the pathing provided in env_file is a portainer container path and not a host path.

I added /nfs/configs:/nfs/configs:ro as one of my portainer bind mounts and then immediately all of my .env files starting with /nfs/configs pathing were found.

On my end this is easily reproduced using the following:

Docker Standalone, using a server & a couple agents CPU Arch: x86_64 Relative path: /mnt/portainer on local filesystem

Brand new repo with only example.yml: https://github.com/portainer/portainer/assets/81395386/c988aa42-3a7f-4823-8bf1-1d165102f630

Full portainer config screen: https://github.com/portainer/portainer/assets/81395386/8dc12d24-f044-4a28-972f-103194877433

example.yml itself contains only the following:

services:
  hello_world:
    image: hello-world
    env_file:
      - stack.env

The error: https://github.com/portainer/portainer/assets/81395386/10d641c4-414a-47f1-8554-d667eba3e2e9

Either removing the env_file entry or disabling relative paths results in a successful deployment. Combining the two results in an error. This worked previously, as I had a stack configured that way which broke not too long ago. I could no longer pull and redeploy that stack, or create a new stack with the same config.

I tried rolling back the versions to test but it resulted in some instability. I haven’t had time to spin up fresh environments to play with and see if I can figure out when it broke.

For some reason I have no luck using stack.env or ../stack.env. Both result in file not found. I tried the web editor and also uploading my own compose file. Any ideas?

Portainer 2.16.1, Docker Standalone environment.

Edit:

I really need this to work as I have to run a dozen services and there are many env vars some of which are used by multiple services. I could go and list them all one by one for each service in the compose file, but maintaining them and updating them will be a nightmare.

There currently is a difference on using env_file: stack.yml between Portainer on Docker Standalone and Portainer on Docker Swarm, see also #6814.

According to this other issue it is “due to a lack of env file support in docker stack deploy”.

But I think there are some big misunderstandings between using a .env file to substitute/insert values into a docker-compose.yml and using an env_file for directly inserting environment variable into a container.

It’s perfectly valid and possible to do the latter. Assume a stack.yml (or docker-compose.yml, the file name does not matter) contains:

version: '3.8'

services:
  test:
    image: ubuntu
    command: env
    env_file: stack.env

And there is a stack.env file containing e.g.:

MYVAR1=something
FOO=bar

Running docker stack deploy -c stack.yml test succeeds and prints in the log (docker service logs test_test):

test_test.1.zh6ijswlq1q0@node3.swarm.idial.fh    | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_test.1.zh6ijswlq1q0@node3.swarm.idial.fh    | HOSTNAME=9194cb8346cc
test_test.1.zh6ijswlq1q0@node3.swarm.idial.fh    | FOO=bar
test_test.1.zh6ijswlq1q0@node3.swarm.idial.fh    | MYVAR1=something
test_test.1.zh6ijswlq1q0@node3.swarm.idial.fh    | HOME=/root

@fcaspani01 nice that it works but extremely bodgy and non-portable workaround.

@Klionheart

Thank you for the additional information. I figured out how to reproduce this.

[1] I changed the name of my stack.env to stack.env.bak in my repo [2] Create Stack with wordpress.yml in my repo using - stack.env [3] Upload my local stack.env [4] Received error [5] Updated my wordpress.yml to - ../stack.env [6] Deployed without error

TODO: Update our Blog with further instructions for Git Repositories

Thanks!

Hi again @tamarahenson, I think I see the misunderstanding here.

I’m not trying to reference an existing “stack.env” inside the git repository. In fact I absolutely do not want an .env file in the git repository since it contains secrets.

The idea is to do as documented in https://www.portainer.io/blog/using-env-files-in-stacks-with-portainer where it is explained that you can add Environment variables to the local Portainer stack, and then reference them en masse by using “stack.env” as an env_file, instead of manually referencing them one by one, in the docker-compose.yaml.

This indeed works with Web editor in Docker Standalone, but not in “Repository” mode where you need to use “…/stack.env” instead.

Hopefully this is more clear, thank you for your time 😃

@Klionheart

Is this what you are trying to do here:

Screen Shot 2022-11-17 at 4 28 27 PM

Thanks!

@tamarahenson the idea is to do:

image

where the docker-compose.yaml referenced is:

version: "3"

volumes:
    db-data:

services:
  db:
    image: postgres:12
    volumes:
      - db-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=$DB_NAME
      - POSTGRES_PASSWORD=$DB_PASSWORD
  web:
    image: healthchecks:v2.4.1
    env_file:
        - stack.env
        [...]

which should work according to https://www.portainer.io/blog/using-env-files-in-stacks-with-portainer

but results in error: error: open /data/compose/51/stack.env: no such file or directory

If I put “- …/stack.env” as the env_file in the compose.yaml, instead, it works, I guess because the stack.env file is generated as /data/compose/stack.env.

Interestingly, if I just use the same docker-compose.yaml directly in portainer (with Web Editor) it instead works with “- stack.env” as expected.

Hey, just wanted to say that I have the same problem. I’m deploying a docker-compose.yml file that sits in a subdirectory of a git repo. Trying to use stack.env as the env_file doesn’t work but …/stack.env does

PS: I’m using Portainer BE 2.16.1

Same issue for 2.13.1 on EE. I had to revert to specifying environment variables manually… If anyone needs to quickly generate required YAML, you can use the following.

Switch to “Advanced Mode” copy everything into a file somewhere (i.e. /tmp/z).

EXAMPLE
$ cat /tmp/z
VAR1="TEST"
VAR2="TEST"
awk -F= 'BEGIN{print "    environment: "}{print "      "$1": \"${"$1"}\""}' /tmp/z

Which should output

    environment:
      VAR1: "${VAR1}"
      VAR2: "${VAR2}"