portainer: New LTS - Stack.env not found

Before you start please confirm the following.

Problem Description

I upgraded to the new LTS. I’m finding that a compose file that used to work, no longer does. It is deployed via web editor and using envronment variables created in the web editor

Expected Behavior

stack created

Actual Behavior

<html> <body> Failed to deploy a stack: env file /data/compose/35/stack.env not found: stat /data/compose/35/stack.env: no such file or directory --


</body> </html>

Steps to Reproduce

create a stack with an env file.

services: immich-server: container_name: immich_server image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release} devices: - /dev/dri:/dev/dri # extends: # file: hwaccel.transcoding.yml # service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding volumes: # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - photos2:/usr/src/app/upload - /etc/localtime:/etc/localtime:ro - photos:/photos:ro env_file: - stack.env etc, etc, etc, etc

And this env (advanced view) UPLOAD_LOCATION=/xxx/immup DB_DATA_LOCATION=/xxx/immich/immich_immdb IMMICH_VERSION=release DB_PASSWORD=xxx DB_USERNAME=xxx DB_DATABASE_NAME=immich TZ=America/Los_Angeles

Portainer logs or screenshots

No response

Portainer version

2.21.0

Portainer Edition

Business Edition (BE/EE) with 5NF / 3NF license

Platform and Version

docker

OS and Architecture

ubuntu 22.04

Browser

ie

What command did you use to deploy Portainer?

docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ee:2.21.0

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 19
  • Comments: 134 (35 by maintainers)

Most upvoted comments

The issue is still present in 2.21.4.

We have identified the issue, and as you can see, it’s a regression for stacks deployed from a Git repository. However, we now understand the full context of why this issue was introduced in the first place. We’re working on a fix—stay tuned. Thanks!

Thanks for your patience on this everyone, we understand it’s a frustrating bug you’re hitting. Good news - we’ve managed to reproduce the behavior you’re seeing here which means we can now identify what’s going wrong and get a fix in.

A huge thanks to all of you that provided information on the issue - it was extremely helpful in finding the reproduction steps!

I’ll update here when I have more info on the fix and a timeline for when we’ll roll it out.

Thanks everybody that helped out with testing this fix. We’ve now rolled this fix out in our latest release - 2.21.2. If you have updated to 2.21.2 and are still hitting this issue please ensure your browser cache is cleared as we have seen instances in the past where this has masked fixes for some users.

Hello @MichaelF25 , don’t worry, the data is not lost. You can refer to this comment to address the issue manually. #12151 (comment). If you’re not in a hurry to update your stack, you can wait for the patch release next week.

wasn’t fixed in 2.21.1, will be there another patch shortly?

Great news - do you have an estimated timeframe for the fix to be shipped?

The fix will be included in our next patch of the LTS release, but we don’t have an ETA for it yet. We’ll provide a link to an image for those willing to try out the fix candidate once it’s available.

Any ETA for a fix?

I’m having the same problem here. Since I rebooted, my Immich stack won’t start, with the folowing message : Failed to deploy a stack: env file /data/compose/10/stack.env not found: stat /data/compose/10/stack.env: no such file or directory

I am running 2.21.3 and this issue is coming up now. Consider reopening.

sure 🙂 #12339

Hi all,

Thanks for your patience while we figure out a fix for this issue. Good news: we believe we have a solution. We’re still going through final testing on the fix but as some of you have put your hands up as willing to help us test, you can now do so on non-production environments with a fix candidate image.

Please note: Do not run this on a production environment! It is a pre-release build focused around fixing this particular issue and this issue only. It may not fully fix the issue. It may have other bugs. If you are at all uncomfortable with running pre-release code on your environment, do not use this build.

If you understand all that and still wish to use it, make sure you take a backup of your Portainer install beforehand. You can find instructions on how to do this in our documentation.

With that disclaimer out of the way, here is the image to use:

portainerci/portainer-ee:pr5443

Simply stop and rm your running Portainer (after taking a backup!) then start Portainer up using the above image and tag. For example:

docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainerci/portainer-ee:pr5443

Those of you that do give this a go, please give us your feedback as to whether (or not) it solved your issue. Your testing is extremely valuable here and will help us get the final build out as quickly as possible.

@oscarzhou-portainer, I’ve only used the .env file, no other variables from the UI and there is no .env at the root of my Git repo.

Great news - do you have an estimated timeframe for the fix to be shipped?

The fix will be included in our next patch of the LTS release, but we don’t have an ETA for it yet. We’ll provide a link to an image for those willing to try out the fix candidate once it’s available.

Also would like to try out a fix as I found this issue as I was updating Authentik, I’m now in a position where the containers aren’t running, I can’t pull from Git, and I can’t even delete the stack to try to run it with the web editor! Sort of a worst case scenario for me.

@juanvmtz It’s strange. If you moved /data/compose/{stack_id}/{vx}/stack.env file to /data/compose/{stack_id}/stack.env, you should be able to update the stack with stack_id. Could you verify if you have managed it in the correct path that is mounted to the Portainer server? Which command do you use to start your Portainer server?

Yes. I copy /data/compose/{stack_id}/{vx}/stack.env file to /data/compose/{stack_id}/stack.env and I can update the stack without error.

Here is the whole compose file.

https://pastebin.com/raw/WSSkd3Vx

Hey, good catch but mine is not relatée to immich actually. Where I’m with you is around the path in env file. That’s something I am using in every stacks

@jamescarppe

Regarding the path in the error message, I did understand the reason for it. It would have been nice if Portainer was able to “translate” that path to the real one outside the container. Not sure that is possible though. Please find my answers to your questions below.

  • What version of Portainer were you on prior to 2.21.0 LTS? Latest STS, don’t remember the exact version number
  • Which method did you use to update - via the CLI or via the in-app update (for BE users)? In-app updater
  • Does this issue occur for new stacks, or just existing stacks? I have not created new stacks since, so not tested
  • Does it only occur when your stacks use stack.env? I have 3 stacks, only one impacted uses stack.env
  • If this occurs intermittently on some stacks but not others, do the stacks that it does occur on have a large (greater than 10) number of stack versions? For non-Git stacks you can tell this by the Deployed version count in the Stacks list. Highest stack version in my case is v3

It seems like the issues are related to stack versioning, so I’ll do some more testing here locally to see if I can nail down exactly what’s going on. Stack versioning was actually introduced in 2.19 so I’m surprised we’re only seeing this issue now.

Hello Guys,

I’m facing the same issue and here what I found in my case: Portainer is looking at stack.en in: data/compose/370

where there’s actually 2 folders: v1 and v2; and stack.env in into these 2 folders.

So I tried to move the stack.env directly to data/compose/370 and it worked ! (and to let you know, that’s a brand new stack I created)

So that’s a regression