portainer: Compose yaml files are no longer interpolated with the contents of .env
Before you start please confirm the following.
- Yes, I’ve searched similar issues on GitHub.
- Yes, I’ve checked whether this issue is covered in the Portainer documentation or knowledge base.
Problem Description
In Portainer 2.27.0, the compose yaml files associated with my stacks seem to no longer be interpolated with the values defined in my .env file.
For example, in a service definition I have things like this:
volumes:
- ${STORAGE_BASE}/tailscale/data:/var/lib # tailscale/tailscale.state in here is where our authkey lives
which should be interpolated with the relevant value from .env:
STORAGE_BASE=/srv/ssdtank/docker
This worked prior to 2.27.0, but now the container is brought up trying to mount /tailscale/data from the host, as if the interpolation was not set, which matches up with what I see logged by portainer:
Feb 20 17:04:18 gnubert portainer[3689]: 2025/02/20 05:04PM INF github.com/portainer/portainer@v0.6.1-0.20241105012540-458d722d4754/pkg/libstack/compose/logwriter.go:55 > level=warning msg="The \"STORAGE_BASE\" variable is not set. Defaulting to a blank string." |
Expected Behavior
compose yaml interpolation happens from a .env file present in the same directory as the yaml files
Actual Behavior
no interpolation happens
Steps to Reproduce
- Create a compose yaml file that uses interpolation
- Create a .env file in the same directory with values to interpolate
- Deploy the compose yaml file from a git repo
- Observe that the interpolation didn’t happen.
Portainer logs or screenshots
No response
Portainer version
2.27.0
Portainer Edition
Business Edition (BE/EE) with Home & Student license
Platform and Version
Docker version 24.0.7, build 24.0.7-0ubuntu2~22.04.1
OS and Architecture
Ubuntu 22.04 amd64
Browser
No response
What command did you use to deploy Portainer?
Ansible:
- name: Configure Portainer
docker_container:
name: portainer
hostname: portainer
state: started
restart_policy: unless-stopped
image: portainer/portainer-ee
network_mode: host
volumes:
- /srv/ssdtank/docker/portainer/data:/data
- /srv/ssdtank/docker/letsencrypt/config/:/etc/tenshu-ssl:ro
- /var/run/docker.sock:/var/run/docker.sock
- /bin/busybox:/bin/sh:ro
log_driver: journald
log_options:
tag: "{{ '{{' }}.Name}}"
labels:
net.tenshu.ssl: "true"
com.centurylinklabs.watchtower.enable: "true"
Additional Information
No response
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 9
- Comments: 69 (7 by maintainers)
Hi all, we have identified the issue and aim to have this patched in our 2.30 release.
I’m in a similar boat to @Jikeh. In addition to having volumes with
${DOCKER_ROOT}in the path, I also have several entries that use labels for Traefik, e.g.,APP_NAMEis provided via the stack’s env vars and passes through into the label just fine, butDOMAINis expected to come from the host (in this case, the set of env vars passed to Portainer) and does not.I also have env vars in the stack env section that interpolate the host env vars. I could re-work those if needed, though.
@xAt0mZ, that’s not correct. Docker Compose allows you to reference the host environment. I tested it just now on the latest versions of Docker (v28.0.1) and Docker Compose (v2.33.1) to confirm. Note that if you’re running with
sudo, your env vars won’t be part of the env thatdocker composeis run with. So this will fail:and if you’re interpolating
DOMAINin your compose file, you’ll get a message likeWARN[0000] The "DOMAIN" variable is not set. Defaulting to a blank string.But if you runthen the references to
DOMAINin your compose file will be resolved correctly.This is in-line with the Docker docs. From Set environment variables - Use the
environmentattribute - Additional informationIt’s also referenced in several other places:
environmentorenv_fileattribute but with the value interpolated from your shell or …”I’m confused as to what sensitive information would be visible. When I run
docker inspect portainer | jq '.[0].Config.Env'the only env var that I see other than the ones that I added specifically so they would be available to my stacks isPATH. I realize that more env vars may have been added at run-time and those should of course not be available to the stacks. It’s not trivial to check what those are since Portainer doesn’t include a shell or the env command.I’m also confused as to how “the entire Portainer container environment” could be leaked. My understanding is that Docker Compose does not provide a means by which all host environment variables can be passed into a container; they have to be referenced individually.
Also, couldn’t someone with the ability to deploy stacks also set up a volume pointing to
/and get root access to the entire host file system?I’m surprised by that, but happy to hear that this is being actively discussed.
Intuitively this feels like it could be added in Portainer environments. That would be more flexible than the way I’m currently using it and would fully support my use case.
But I’m open to any alternative way to portably, across multiple stacks, reuse environment variables (such that they can be interpolated and referenced in both the compose and stack env).
I’m losing faith with Portainer. Their QA/QC has gone downhill in the last few months. I do not understand how bugs like this even get in to LTS or even STS releases without any dev noticing. Surely there are a bunch of CI regressions tests that cover basic user acceptance test cases like “can I deploy a compose stack with an .env file without error?”.
Then, when users report critical issues like this, they remain unfixed and unacknowledged in release notes for months at a time whilst new features are added in.
What’s worse is businesses pay for this critical infrastructure management tool and cannot afford any downtime whilst they investigate and rollback. I would 100% be considering alternatives if I were a business.
Only the other day a 2.29.1 STS release CVE fix broke the ability to even log in to Portainer, yet it was still somehow released without anyone in Portainer noticing. Although it was fixed quickly, unlike this one, it still should never have made it to release. Releases seem to be chucked out with minimal (if any) regression testing at the moment.
I can’t help but feel that the OP’s issue has been hijacked by a different issue here… This issue clearly states it is about “Compose yaml files are no longer interpolated with the contents of .env” NOT about the compose file not being interpolated with the portainer server (host) container’s environmental variables. So why has it been merged with #12555 ?
I appreciate the latter is still an issue, but would it not be better to keep them separate issues to avoid confusing people coming here expecting to see answers for the original issue.
I for one have the OP’s issue and would like to get an update on that in this discussion, not the other issue which is just confusing matters.
@jamescarppe any updates or proper workarounds for the matter? Almost 2 months since the report of a critical feature being broken in a LTS release, yet there are no estimates on fix complexity or timeline.
If I was a customer using Portainer for business, I’d already moved to a different solution.
Any chance we can get an update on this? Would be nice to know if it will be fixed or not.
We’ve been able to reproduce this internally and are working on a fix.
In the meantime we have a potential workaround - if you set an environment variable within the UI, interpolation from the
.envfile should work as previously. The variable you set doesn’t need to be used by anything, it just needs to exist in the UI.@velary nope, it’s still broken. To be fair, I tested it by only updating the agent, with server sitting on 2.27.4.
Here’s the stack config I’m using, with secrets amended: https://gist.github.com/hedger/b781af828336b1fa8f26800ffb098698
With agent on 2.29.2, it still fails to interpolate variables from the env file. Trying to pull & update an existing stack from the UI will result in the following error:
At the same time, I can
cdto the folder where it pulls the repo with docker-compose files and dodocker compose up -d– and it will happily bring the stack up. It will show up as running and healthy in the UI. Immediately after that, trying to redeploy it again from the UI fails the exact same way.What’s baffling is that there had been multiple releases in the LTS 2.27 branch, and not a single one even acknowledges this critical issue in the “known issues” section. The lack of communication and recognition of the issue is also just inexplainable.
@jamescarppe care to drop a line, please?
i use the env file to store my folder locations so that if I restore my docker containers on another machine, i would not have to edit all stacks. having the environment file allowed me to change those folders with simple env file change. Now it is a nightmare to recreate the stacks on another machine. instead of just taking away a feature, please either make it backward compatible or provide alternate option.
@jamescarppe , please prioritize this issue.
Hey there
I re-ran all my tests for interpolation. The detailed scripts, stacks and results can be found here
Tests description
Expand me
The repo contains scripts and files to deploy 24 variations of a basic
natsstack that is using either a.envor astack.envfile, referencing it (or not) in the compose file.The content of the file is either (no reference of the env files)
or (with a reference of the env file)
All
.envfiles containAll
stack.envfiles containTests run the following combinations
.envstack.env.envandstack.envFor each of the above, override (or not)
A=uito verify the environments are properly merged between the file and the env fields in the UI.For each of the above, deploy with and without relative paths enabled.
compose-unpacker.Testing environments
All Portainer instances are started with
-e C="portainer-container-env"so we can validate that the container environment is forwarded (or not) to the stack.Tests results
Based on the test results, here’s an analysis of the anomalies using 2.21.5 as the baseline reference:
Anomaly Analysis (Business Edition)
1. Variable C Security Fix (Intentional Change)
Note: this was a point reported by you and that we fixed in 2.30.0 by allowing the Portainer container environment variables defined with a
PORTAINER_prefix to be automatically passed to the stack deployment context.Pattern: Variable C removal was a deliberate security hardening measure
Impact: This affects 12 test cases (unpacker scenarios already didn’t forward portainer container env vars) but is the correct behavior. The previous injection of host environment variables into stacks deployment context was a security vulnerability.
2. Environment Variable Loading Failure in 2.27.9 (Actual Bug)
Pattern: Variables A and B incorrectly lose their “env” values in unpacker scenarios
Affected Test Cases:
*-unpackerscenarios where.envfile should provide valuesRoot Cause: 2.27.9 introduced a legitimate bug in
.envfile loading when unpacker is active.3. UI Override + Unpacker Interaction Bug (Actual Bug)
Pattern: Variable B incorrectly loses “env” value when UI override + unpacker are both active
Affected Test Cases:
*-ui-unpackerscenariosImplementation Timeline
2.21.5 → 2.27.9 (Security Hardening + Accidental Regressions)
Intentional Changes:
✅ Security Fix: Removed host environment variable injection (Variable C)
Unintended Regressions:
❌ Broke
.envfile loading in unpacker scenarios ❌ Broke UI override + unpacker interaction2.27.9 → 2.31.2 (Bug Fixes)
Successful Fixes:
✅ Restored
.envfile loading functionality ✅ Fixed UI override + unpacker interactionI couldn’t find any regressions on CE in 2.27.9 nor 2.31.2 apart from the
C=env var inherited from the Portainer container, which is the correct behaviour (has been fixed in 2.30.0 withPORTAINER_prefixed env vars)Same problem here. Even if I manually define the varaibles.
Community Edition 2.27.6 LTS
Thanks @pmharris77, I have raised this with the team
Correct me if I’m wrong here, but would the security issue not be the same as using any env file? The only difference being that there may be more variables than a single container or stack.
To me this comes down to user awareness/training and risk appetite. I am no expert and continually learning, but I had passwords saved in my env files until I understood the risk better, now I use secrets and an environment variable
SECRETSDIRto point at them. But things like my global Appdata, NAS & log folders, local and global domain names, timezones, puid/guid are used in almost every container anyway.I believe this is actually a quite critical issue. Any ETA on the fix?
is there any plan to fix this behavior ? it used to work earlier, but not now, so it would be a regression or breaking change.
I am using the latest version trying to specify a profile (
--profile) on a docker-compose using theCOMPOSE_PROFILESenv var, but it appears it is not being interpolated.Hey there, for those of you who still encounter issues, can you edit your above messages and specify if you are using an environment variable inherited from the Portainer container, or from a
.envfile located besides your compose file in your git repository?I created the following test suite to cover variations, and 2.27.1 works the same as 2.21.5 when interpolating environment variables from a
.envfile located besides the compose file in git.https://github.com/xAt0mZ/portainer-git-feats-test-files/tree/master/env-interpolate
For the explanation
In 2.21.5 it was possible to use the entire Portainer environment when deploying stack files, as we were calling the compose binary directly from inside the Portainer container. However starting from 2.24.0 (when we removed the compose binary and used the compose lib instead) this is not possible anymore.
In 2.27.1 we did fix the
.envissue not being read by default (compose binary default behaviour is to read the.envfile next to the compose file for interpolation when no--env-fileis specified)However we still (for now) don’t pass the Portainer container environment to compose when deploying stacks (which is also a compose binary default behaviour) as it is a security risk: any user able to deploy a stack can leak the entire Portainer container environment, which may contain sensible information. We were honestly not expecting some of you to use this behaviour and are actively discussing internally the next step (likely re-introducing the behaviour for now).
I’ll jump in with what I think is a related issue:
My Portainer compose.yaml contains these lines:
Which allows all of my Portainer-created containers to reference thusly:
So while I’m not explicitly using an env file, I’m able to reference this single set variable in all of my containers. This setup was working fine for the last 6 months or so, until yesterday after updating Portainer CE to 2.27.0. As @kaneoriley mentioned - it seems only containers I am manually editing the compose file from the Stack Editor are affected. Starting and stopping containers and stacks, as well as restarting the host machine do not affect running containers.
The .env file is stored at
/home/user/docker/portainer/docker.env, and is passed to portainer when starting via the command:The contents of
docker.env:Until this morning, this worked to allow me to use
DOCKERDIRas a variable in any stack I created using the web UI, as the environment for portainer had the value set. An example stack yaml:I use a template saved at
/home/user/docker/portainer/template.yml(which to portainer is/data/template.yml) to save repeating myself a million times in all my stacks for some basic setup stuff, but the contents aren’t really relevant to the problem.The env file isn’t in the same location as the compose file as that’s not how it’s being used. It’s a portainer env, not a stack specific env, and it just allows me to set global variables which are then available to all stacks. This has all worked perfectly until the 2.27.0 update.
@Nick-Portainer don’t know if it makes a difference, but I’m using the EE docker image version of 2.29.2 STS, not the CE version
@Nick-Portainer Confirming same on LTS 2.27.4 and on previous 2.27.3. I did also try STS 2.29.2 but like @pmharris77 - same issue persisted so I rolled back to LTS version.
Currently, it’s only possible to rollback to the last version you were using before the update. In my case, I was on version 23.0 before updating to 27.0. When I needed to rollback, I could return to 23.0 and then update again to other versions. If any errors occurred, I would rollback back to 23.0.
From what I understand, it’s not possible to rollback directly to a specific version using this method. Because of that, at work, we decided to use virtual machine snapshots. That really saved us, because once we updated from 21.x to 24, and then to 27. When we tried to rollback, the system could only return to 24, not 21.x.
Fortunately, we had a snapshot; otherwise, we would have lost that version. Another important point: several GitLab pipelines depend on .env files that store the image tags used. If we update Portainer carelessly, we might break the entire CI/CD process, since the environment variables defining the images would be affected.
You can update to 2.28.0 - 2.29.2 versions, works fine for me Fix: https://github.com/portainer/portainer/commit/dd98097897fc2dbe437aee24642d8f3a77c12194
I’m using Portainer to deploy a stack where the docker-compose.yml and .env fire are stored in GIT. Until the upgrade to 2.27.1, it was working fine.
Since the upgrade, Portainer complains about not being able to get the contents of the file ‘docker-compose.yml’ and invalid spec because of empty section.
In the portainer logs, I can see that there is a lot messages about variable not set level=warning msg=\"The \\\"EXTERNAL_NC_LIBRARY\\\" variable is not set. Defaulting to a blank string.\""
If I create manually the variables within portainer, everything works as before
This is for immich, I’m using their docker-compose file Here is an extract
The directory layout is
For my case, the 2.27.1 update has not solved the issue
My issue is not resolved by 2.27.1, problem remains the same unfortunately.
Edit as per https://github.com/portainer/portainer/issues/12546#issuecomment-2688604043:
I’ve been using environment variables inherited from the Portainer container, as it’s a convenient way to have ‘global’ variables.
I’ve seen the same issue after upgrade to 2.27.0. My docker-compose.yml and .env are stored in a repo in GitLab. Variables used were to specify the Image version and configure traefik via labels.
Example code for the traefik labels: .env
docker-compose.yml
Running ‘docker compose config’ results in everything being replaced correctly. However the container in Portainer is showing labels as: traefik.http.services…loadbalancer.server.port= traefik.http.routers…rule= traefik.http.routers…entrypoints=http"
I’ve now upgraded to 2.27.1. and I’ve moved the variables into my named env_file for the service and redeployed the stack. Still not working, the variables weren’t merged from the application.env file that is explicitly referenced in the docker compose file. Still works correctly when tested with ‘docker compose config’.