portainer: `stack.env` does not work
Bug description
Following along with the tutorial here: https://www.portainer.io/blog/using-env-files-in-stacks-with-portainer I get stack.env: no such file or directory
Expected behavior
stack.env should work as per the above blog post.
Portainer Logs
level=info msg="2022/03/28 12:47:58 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:47:57 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:52:49 http error: open /data/compose/57/stack.env: no such file or directory"
level=info msg=" (err=open /data/compose/57/stack.env: no such file or directory"
level=info msg=") (code=500)"
level=info msg=" (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:49:46 http error: open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:53:58 http error: open /data/compose/58/stack.env: no such file or directory"
level=info msg=" (err=open /data/compose/58/stack.env: no such file or directory"
level=info msg="2022/03/28 12:49:31 http error: open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:48:45 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"
level=info msg="2022/03/28 12:48:25 http error: Unable to start stack (err=open /data/compose/51/stack.env: no such file or directory"
Steps to reproduce the issue:
Be running in swarm mode.
Create a new stack with the content.
version: "3.8"
services:
test:
env_file:
- stack.env
Add at least one environment variable to the environment variables section.
Try to deploy.
Technical details:
- Portainer version: Portainer CE 2.11.0
- Docker version (managed by Portainer): API version 1.41
- Platform (windows/linux): linux
- Command used to start Portainer (
docker run -p 9443:9443 portainer/portainer): Uhh, it’s been running for a while as a stack… - Browser: Firefox
- Use Case (delete as appropriate): Using Portainer at Home
- Have you reviewed our technical documentation and knowledge base? Yes
Additional context
Assessing the possibility of running a dockerized mail server through portainer for my employer.
About this issue
- Original URL
- State: open
- Created 5 years ago
- Reactions: 29
- Comments: 83 (8 by maintainers)
Links to this issue
Commits related to this issue
- stack.env seems to be generated in the root according to https://github.com/portainer/portainer/issues/6701#issuecomment-1126852560 — committed to patzm/dockerfiles by patzm 4 years ago
- [firefly] Fix path of stack.env file See: https://github.com/portainer/portainer/issues/6701#issuecomment-1126852560 — committed to JuliusFreudenberger/infrastructure by JuliusFreudenberger 3 years ago
- Remove env.stack from authentik config https://github.com/portainer/portainer/issues/6701 — committed to tsachleben/portainer-stacks by tsachleben 3 years ago
to me, the error reason was the path of docker-compose.yml(using git repository), seted on Compose path field
Compose path: /some-project/docker-compose.yml
Before:
After(Working): returning to the root directory with …/stack.env
Obs: I know it doesn’t answer your question, but it could be related.
Tried using local (docker.sock) environment and Portainer agent environment. Both give errors saying
stack.envcan’t be found.With portainer agent, it worked after trying the suggestion from JuniorC07. changing the path of
stack.envto../stack.envhow easy you ask. I asked for chatgpt to create that parser and it did it, without any mistakes at least for my use case. maybe this will help someone, just run
./docker_swarm_easy_fix.py docker-compose.yml stack.envshould return the file with the variables pasted in it should support multiple files like./docker_swarm_easy_fix.py docker-compose.yml stack.env stack2.envbut who knows just asked chatgpt to create the parser.docker_swarm_easy_fix.py
Hello guys, This issue happens because you are indicating there is a .env file to include in your composer code.
It is not needed at all to indicate you have a “.env” file, if you click on the option to import variables from env file, all the variables in the file will be included so you don’t need to include the env file on the code, I’m posting this from my phone, but let me know if you need a video example
My workaround is manually insert stack.env file inside portainer’s docker volume then i updated related docker stack and now it works.
Seeing the same behavior on EE 2.12.1
Any updates on this? It is reproducible for me in Portainer Business Edition 2.18.3
I am also experiencing the same issue running Community Edition 2.18.1.
I have tried using “.env” & “stack.env” and it is still throwing errors.
I think there is very much confusion in this thread as there are many different constellations available:
I’ll try to explain some things (and in turn also make it clearer for myself).
Docker Standalone
Deployment using Web editor
Defining environment variables:
You can either specify as many environment variables as you’d like manually using the Add an environment variable button.
OR
Use the Load variables from .env file button to upload an existing env file from your local computer. Portainer then reads this and puts all the values in the web form.
In both cases the result is the same (expand to view screenshot).
Using the environment variables:
You can now choose between using them separately or as a whole stack.env.
Using them separately: Specify the environment variables like in this example compose file:
(
command,stdin_openandttyare only needed to prevent the container to automatically exit when using theubuntuimage)If you inspect the containers you will see that
test-serviceonly knows aboutFOO=barandservice-with-user-and-passwordwill only know aboutUSERNAME=itsameandPASSWORD=mario.Inspection results:
Using them as a whole stack.env:
If you inspect the container you will see that it knows about all three variables.
Inspection results:
Deployment using Repository
As stated in the Environment variables section, a file named
stack.envmust be present in your repository. Consider the following Git repository structure (for file contents see the example repo):In this case you can only use the variables as a whole stack.env. Using them separately as with the Web editor example seems not to be possible.
Inspection results:
Docker Swarm
This one is special. As far as I understood the documentation and according to my testing, this is only possible if you are deploying via Repository.
Example Repository deployment:
Inspection results:
The results show that defining new environment variable values in the web editor doesn’t alter the in-repo stack.env file. That’s why
all-togetherstill has the original values.And
single-not-workingdoesn’t use the stack.env file at all (see also the “Deploy using Web editor > Using whole stack.env” above). It is using the variables defined in the web interface, which in this case hasFOO=otherfoo.When instead trying to use stack.env in a deployment via Web editor, we get the famous error:
To me, using stack.env in Swarm mode doesn’t seem feasible or to bring any benefit. I would rather stay at defining the environment variables separately in the services.
It seems Portainer itself is checking the existence of
env_fileentries. Since Portainer is running as a container, it can’t find the files unless you add a volume to the host file.Example solution: Your stack:
Portainer stack:
Important:
What fixed it for me is in the compose web editor in Portainer, I changed ALL instances of:
to
I am using Portainer Community Edition 2.18.3. You can’t just change one .env to stack.env, or it will still throw errors. You have to change all of them.
Hello, just a couple more, hopefully useful, use cases 😃
Using “stack.env” does not currently work when:
It does work, however, by using “…/stack.env” as path instead.
Using “stack.env” on the other hand, works just fine when:
same issue here
@fidelix You are right. I just tried to summarize the main things I found here, in the docs and in my testing.
I believe this issue is a bit too convoluted by now and (maybe also due to this) doesn’t seem to get much attraction by Portainer development team. Maybe this issue should be converted into a discussion and a new clean issue should be opened, with only relevant information.
Seems like a bug. Trying to deploy on CE v2.19.4, and the Stack YAML editor says
To reference the .env file in your compose file, use ‘stack.env’., but doing just that causesopen /data/compose/3/stack.env: no such file or directoryand it is not possible to update/deploy the stack.I can see the problem. When deploying a stack using web editor you can define Env Vars in Portainer UI and refer to a
stack.envfile. Portainer then creates astack.envfile along with the compose file and writes all the vars to that.This is not the case when deploying from Git, if you refer to
stack.envit is expected to be in the repo. We clone the repo but do not add any files to that location as it will get overridden at next update.We should however clarify this in the documentation.
I am experiencing the exact opposite on Portainer 2.16.2, Docker Standalone environment connected through
portainer/agent:latest:env_file: stack.envworksenv_file: ../stack.env-> file not foundThis thread and the documentation could really use some clarification on this.
I am also experiencing the same issue, referencing ‘stack.env’ as per the blog post results in an error:
open /data/compose/89/stack.env: no such file or directoryEE 2.14.0, docker swarm mode.
This is a bit confusing, as I’m fairly certain I am following the blog post instructions correctly.
Ditto on EE 2.12.1, running on a swarm
When I go into the portainer vol, the compose folder that it’s looking for doesn’t exist. If I create the stack and then add env_file to the compose, it’s never created in the compose folder created for the id.
If I create an empty file in that folder the stack is able to be updated with the env_file definition, but the file isn’t updated with the env set in the gui.
Still not working with Portainer Business Edition 2.21.4
Ah, I just saw that the documentation clearly states that using stack.env in Swarm mode is not possible:
It’s just not very visible, as it is not highlighted like the other Notes. Additionally it’s only mentioned in the “Option 1: Web editor” section (the other Note is repeated in every section).
Yes and sorry for not including it originally.
When a new stack is deployed, it gives a number to the path:
/data/compose/3/stack.env: no such file or directorySo I noticed that if you wish to redeploy it with the env file on the server, you need to place the env file under the next one:
/data/compose/4/stack.env: no such file or directoryI hope this helps
@christianstrauch, @ecker00 and @voslucas : Did you tried this? https://github.com/portainer/portainer/issues/6701#issuecomment-1704036982
Portainer itself is running as a docker service, which means that you need to add volumes that matches the host path. Otherwise portainer can not “see” the files 🙂
Thank you! I didn’t even think about the fact that portainer itself is a container and the fact that by default it might assume the pathing provided in env_file is a portainer container path and not a host path.
I added /nfs/configs:/nfs/configs:ro as one of my portainer bind mounts and then immediately all of my .env files starting with /nfs/configs pathing were found.
On my end this is easily reproduced using the following:
Docker Standalone, using a server & a couple agents CPU Arch: x86_64 Relative path:
/mnt/portaineron local filesystemBrand new repo with only
example.yml: https://github.com/portainer/portainer/assets/81395386/c988aa42-3a7f-4823-8bf1-1d165102f630Full portainer config screen: https://github.com/portainer/portainer/assets/81395386/8dc12d24-f044-4a28-972f-103194877433
example.ymlitself contains only the following:The error: https://github.com/portainer/portainer/assets/81395386/10d641c4-414a-47f1-8554-d667eba3e2e9
Either removing the
env_fileentry or disabling relative paths results in a successful deployment. Combining the two results in an error. This worked previously, as I had a stack configured that way which broke not too long ago. I could no longer pull and redeploy that stack, or create a new stack with the same config.I tried rolling back the versions to test but it resulted in some instability. I haven’t had time to spin up fresh environments to play with and see if I can figure out when it broke.
For some reason I have no luck using
stack.envor../stack.env. Both result infile not found. I tried the web editor and also uploading my own compose file. Any ideas?Portainer 2.16.1, Docker Standalone environment.
Edit:
I really need this to work as I have to run a dozen services and there are many env vars some of which are used by multiple services. I could go and list them all one by one for each service in the compose file, but maintaining them and updating them will be a nightmare.
There currently is a difference on using
env_file: stack.ymlbetween Portainer on Docker Standalone and Portainer on Docker Swarm, see also #6814.According to this other issue it is “due to a lack of env file support in docker stack deploy”.
But I think there are some big misunderstandings between using a .env file to substitute/insert values into a docker-compose.yml and using an
env_filefor directly inserting environment variable into a container.It’s perfectly valid and possible to do the latter. Assume a stack.yml (or docker-compose.yml, the file name does not matter) contains:
And there is a stack.env file containing e.g.:
Running
docker stack deploy -c stack.yml testsucceeds and prints in the log (docker service logs test_test):@fcaspani01 nice that it works but extremely bodgy and non-portable workaround.
@Klionheart
Thank you for the additional information. I figured out how to reproduce this.
[1] I changed the name of my
stack.envtostack.env.bakin my repo [2] Create Stack withwordpress.ymlin my repo using- stack.env[3] Upload my localstack.env[4] Received error [5] Updated mywordpress.ymlto- ../stack.env[6] Deployed without errorTODO: Update our Blog with further instructions for Git Repositories
Thanks!
Hi again @tamarahenson, I think I see the misunderstanding here.
I’m not trying to reference an existing “stack.env” inside the git repository. In fact I absolutely do not want an .env file in the git repository since it contains secrets.
The idea is to do as documented in https://www.portainer.io/blog/using-env-files-in-stacks-with-portainer where it is explained that you can add Environment variables to the local Portainer stack, and then reference them en masse by using “stack.env” as an env_file, instead of manually referencing them one by one, in the docker-compose.yaml.
This indeed works with Web editor in Docker Standalone, but not in “Repository” mode where you need to use “…/stack.env” instead.
Hopefully this is more clear, thank you for your time 😃
@tamarahenson the idea is to do:
where the docker-compose.yaml referenced is:
which should work according to https://www.portainer.io/blog/using-env-files-in-stacks-with-portainer
but results in error:
error: open /data/compose/51/stack.env: no such file or directoryIf I put “- …/stack.env” as the env_file in the compose.yaml, instead, it works, I guess because the stack.env file is generated as /data/compose/stack.env.
Interestingly, if I just use the same docker-compose.yaml directly in portainer (with Web Editor) it instead works with “- stack.env” as expected.
Hey, just wanted to say that I have the same problem. I’m deploying a docker-compose.yml file that sits in a subdirectory of a git repo. Trying to use stack.env as the env_file doesn’t work but …/stack.env does
PS: I’m using Portainer BE 2.16.1
Same issue for 2.13.1 on EE. I had to revert to specifying environment variables manually… If anyone needs to quickly generate required YAML, you can use the following.
Switch to “Advanced Mode” copy everything into a file somewhere (i.e. /tmp/z).
Which should output