portainer: Failed Loading Environment error after rebooting manager node
Bug description After rebooting the manager node in a 1 manager, 1 worker docker swarm setup, portainer will boot the service, but attempting to load the primary environment will fail with an error:
Failed loading environment
Unable to find an agent on any manager node
This error happens both from the worker and manager node instances of the portainer web interface.
Expected behavior After a reboot, portainer boots normally and allows loading the environment.
Portainer Logs Provide the logs of your Portainer container or Service.
Steps to reproduce the issue:
- Setup a docker swarm with 1 manager, 1 worker. The manager is Ubuntu Server in my case, while the worker is Pop OS!
- Verify that portainer is working normally by visiting the web interface
- Reboot the manager node (system reboot)
- Visit web interface again, attempt clicking on primary environment
- See error
Additional context Discord thread has additional context if needed.
About this issue
- Original URL
- State: open
- Created 4 years ago
- Reactions: 5
- Comments: 18
Update:
Upon further investigation, I have found a better workaround.
Run the following on the Manager node (image name may need to be edited):
sudo docker service update --image portainer/agent:2.16.2 --force portainer_agentThanks!
I am experiencing the same problem after restarting my entire Swarm cluster. Portainer and its agents do not start properly, resulting in the following error message:
I’m in portainer v1.19.5 (the LTS version). Running a
docker service updatetemporarily fixes the problem, but the issue reoccurs after each reboot.This is an ongoing issue. Can we have a permanent fix, please?
Hi, I’m using portainer for 4 years in production swarm env. I have always seen this issue and the only workaround is to restart portainer_agent on all nodes … quite annoying. And come again few days later I think portainer doesnt support well network latency / reconnections and also I can’t use portainer in a multicloud swarm env with 5ms latency because the agent node timeout half the time… I use the latest version 2.19.4
So now I restart portainer in a cronjob at 23:00pm every day to limit this issue … Please, give us a fix for that
thanks
DOCKER SWARM REBOOT TESTING:
(1) Manager, (1+) Worker
[1] Install Swarm Portainer Server on Manager:
curl -L https://downloads.portainer.io/ee2-16/portainer-agent-stack.yml -o portainer-agent-stack.ymldocker stack deploy -c portainer-agent-stack.yml portainer[2] Reboot Manager Node [3] Navigate UI [4] Wait 2 Minutes https://user-images.githubusercontent.com/20426210/210469483-0e6c40c9-3325-4601-8fcf-3c11100d9b33.mov [5] Toast appears:
[6] Home >>
Environment:primary>> Node indicator is no longer appearingWORKAROUND: [1]
[5] Toast no longer appears
sudo systemctl restart dockeron worker node [2] Navigate UI on Manager [3] Wait 1 minute.[4] Home >> Environment:primary >> Node indicator is appearing with node counts
Note: this workaround also works on multiple worker node clusters. You only need to restart docker on one worker node.
ADDITIONAL TESTING:
(2) Manager, (1) Worker Notes: Issue does not occur.
TODO: Need to determine if this is Docker Swarm behavior or a bug. Updates to follow.
Thanks!
Happened again today after restarting two worker nodes (Azure VMs). 1st VM took a while and did not affect anything. 2nd VM had a fast reboot and throwed the same “portainer unable to find an agent on any manager node” message in Portainer web.
Fixed it with:
docker service update --image portainer/agent:2.xx.x --force portainer_agentwe are on the same boat, this happen in every environment that we have (Portainer BE customer) after we reboot any node in a swarm cluster, not necessairly a manager node. This is quite frustrating and we have it since the beginning of our experience with Portainer. We tried to configure a systemd unit that restart the portainer_agent service on one manager for each swarm cluster 40 minutes after the node booted up (that’s pretty much the time it takes to update one of our clusters plus 20 minutes of margin) but is not a real solution. For example if we have to restart any worker node for an unplanned maintenance then we have to remember also to manually restart the portainer_agent service for that swarm cluster.
@tamarahenson Any plan to fix this? The workaround is not a real solution because we have different reboot/update policies in our swarm clusters. If that’s not on roadmap we will open a support case on the customer portal because we are having many complaints from out end users that are perceiving Portainer as unstable and broken every time they have this errors. I know that the workaround is a quick fix but at that point the user has already experienced the error.
This has been an issue for years now. Has any progress been made? 😕
We have written a cron job that looks fore the “unable to redirect request to a manager node: no manager node found” then kicks off a service update, but it is not really a great solution…
Per our Discord conversation, using
sudo systemctl restart dockeron the worker node is confirmed as a workaround for the issue. After rebooting the service, the error goes away and the GUI can be accessed again - without any changes on the manager node.@joshbuker
Thank you for logging this request that was initially posted in our Discord channel. I am going to provide further updates here with my testing and forward to Product for review.
Thanks!