portainer: Warning Messages in Portainer Logs Caused by docker.sock not Available in Docker Swarm Setting

Bug description There are warning messages in the Portainer Container log.

Expected behavior There should not be such a warning message in the Portainer Container log.

Portainer Logs 2023/04/18 01:07PM WRN platform/platform.go:68 > failed to retrieve docker info | error=“Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?”

Steps to reproduce the issue:

  1. Have Portainer for a single node docker swarm using the official manual: https://docs.portainer.io/start/install-ce/server/swarm/linux
  2. Check the Logs of the Portainer Container using Portainer
  3. Notice the warning messages in the Portainer Container Log.

Technical details:

  • Portainer version: 2.18.1 (CE)
  • Docker version (managed by Portainer): 23.0.4
  • Platform (windows/linux): linux

Additional information

The changes introduced by #8095 to DetermineContainerPlatform in api/platform/platform.go (https://github.com/portainer/portainer/pull/8095/files#diff-1de261cf96cf89923c3d864d51e93cb0910700fa0f749bf85ab8167eefdee1b5R37) triggers docker info since that change. Using the recommended install manual for swarm scenario, the docker.sock will not be mounted into the Portainer container but only to the agent container. I guess this is causing the warning messages in the Portainer Container Log.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 22
  • Comments: 47 (8 by maintainers)

Most upvoted comments

@jamescarppe Sadly, the recently released 2.20.0 version is still suffering from this issue.

Even when using the standard officially documented installation method for Docker Swarm here: https://docs.portainer.io/start/install-ce/server/swarm/linux#deployment

Portainer starts to spam the logs frequently (more than once per minute) as seen below:

2024/03/19 10:56AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:56AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:56AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:56AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:59AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:59AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:59AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:59AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:59AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:01AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:02AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:02AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:02AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:02AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:02AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:03AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

The problem manifests when the Portainer server container does not have /var/run/docker.sock mounted in it, which is not necessary when using Portainer with a remote Portainer agent, as it goes via network. In fact, even when using Portainer in standalone mode (without containers) it should manifest if the user is running Portainer without Docker running locally.

The problem is in the DetermineContainerPlatform() function here: https://github.com/portainer/portainer/blob/61642b8df610712c3cd71efc0e792c53627878d2/api/platform/platform.go#L39-L82

As it can be seen, this function tries to create a client to a Docker daemon unconditionally in line 59. I believe that instead, Portainer should check if /var/run/docker.sock (or more generically, DOCKER_HOST) exists before attempting to do this, and skip the rest of the detection if the socket is not available.

I understand the Portainer team is already working on this (Internal Ref: EE-5372) for a while, but it has been a really long while as of now and the log spamming is really an important issue for us (and I think lots of users conscious of log storage space).

We haven’t been able to migrate from Portainer 2.14.x due to this (and another) issue. I hope this can be fixed once and for all sooner than later before the new LTS hits. Thanks!

Hi and I am not very familiar with Go but I looked up the code and I think the issue is that a simple docker client is created here: https://github.com/portainer/portainer/blob/develop/api/platform/platform.go#L59

In that case the DefaultDockerHost is used which is unix:///var/run/docker.sock.

image

You can override that with the DOCKER_HOST environment variable but that does not work either because the simple client refers to the DefaultDockerHost. The warning occurs also each time the Info-Endpoint is called via REST which is quite often 😃

IMHO this should not be a warning. We use a docker socket proxy in between, so we don’t mount the socket directly: https://github.com/Tecnativa/docker-socket-proxy

Alternatively it would be nice to make that overridable.

Not sure if that’s helpful and even the fix for the root cause but I came around the issue by adding into the offical portainer-agent-stack.yml yaml: volumes: - /var/run/docker.sock:/var/run/docker.sock

image

Since then the warning disappered and I was able to connect properly to the portainer environement (before just an empty portainer had been presented)

@jamescarppe @tamarahenson Is there any update on this issue or a time schedule regarding a fix? 🙂

Internal Ref: EE-5372

Yes, these changes and some further ones in this area are being worked on for our 2.22.0 Portainer release. Well, that’s the current plan.

+1 …I got this problem (continual error in my logs) too.

This issue was duplicated in #12285, where some days ago the Portainer team finally confirmed this is fixed in the upcoming 2.27.x release. See the latest comments from more information.

I can confirm that the latest released 2.20.3 version is still suffering from this issue. However, I can also confirm that it will be fixed once any of the above linked PR is merged (likely 11728). 💯

I was looking for solution for quite some time, but since there isn’t an actual solution or at least I wasn’t able to find one, I decided to mount the docker.sock directly to portainer service with read-only access. This way Portainer isn’t able to do anything that is not supposed to do to docker.sock and the errors are finally gone.

volumes:
      - portainer_data:/data
      - /var/run/docker.sock:/var/run/docker.sock:ro

same issue on docker swarm use portainer-agent-stack.yml my test doesn’t have any worker, only 1 manager

portainer_agent

portainer_agent.0.qxeee0ajedcv@pallas    | 2023/12/27 11:55:29 http: TLS handshake error from 10.0.2.3:49700: EOF

portainer_portainer

portainer_portainer.1.ql8dowwiqz2n@pallas    | 2023/12/27 12:16PM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
portainer_portainer.1.ql8dowwiqz2n@pallas    | 2023/12/27 12:16PM ERR github.com/portainer/portainer/api/http/client/client.go:92 > unexpected status code | status_code=403

portainer version 2.19.4 (CE)

docker version

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:07:41 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:41 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Hi there, everyone! I’m experiencing the same issue on version 2.19.3-alpine. Do you have any expectations on when it could be fixed?

2023/12/03 10:43AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/12/03 10:43AM ERR github.com/portainer/portainer/api/http/client/client.go:92 > unexpected status code | status_code=403

I use Docker version 24.0.7

~# docker --version
Docker version 24.0.7, build afdd53b

I was using portainer-ee:218.2 and did not realize any problem, but after upgrading to 2.19.1 log is flooded with theese two errors: 2023/09/22 03:22PM INF docker/snapshot.go:180 > unable to inspect container in other Swarm nodes | error=“Error response from daemon: No such container: d70efd1405298804dc09d3b72e4a8b0a2463799fcbe12478bca093d2dc8a9bc4” container=d70efd1405298804dc09d3b72e4a8b0a2463799fcbe12478bca093d2dc8a9bc4

2023/09/22 03:22PM WRN platform/platform.go:70 > failed to retrieve docker info | error=“Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?”

I can’t say it’s harmless, sometimes the portainer is not showing the “stats” and “exec console” icons at container. and then show after refresh

not sure how to downgrade, if I just deploy the 2.18.2 it’s not working, I see the stacks, but cant go into the stack (error was something like it can’t read compose.yml file)

2 node docker swarm, Docker version 23.0.6

The warning comes periodically as well with 2.18.3 when using Docker Standalone > API with a Docker API URL as environment.

Issue is stiil actual for 2.18.3

image

Just migrated to Portainer 2.18.2 today and started seeing those messages as well. Using with Docker Swarm clusters.

Just a heads-up that this issue is still present in the newly released 2.18.3 version as well 😦

There are periodic error messages in the console logs:

2023/05/23 12:02PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:02PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:02PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:03PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:03PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:04PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:06PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:06PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:06PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

This has the potential to fill up any logger system in place for Portainer, including the default Docker logger driver that does not rotate/limit logs by default.

I can confirm its indeed working! I tested using 2.27.0-rc3-alpine. I did not mount the docker socket on portainer itself, but only on the agent (as recommended by the docs when using Swarm Mode) and it indeed works now. No more warnings!

I just set up a fresh Docker Swarm cluster, and now had the same problem. Everything worked fine for the first few minutes, then the web interface took forever to load for some tasks, and sometimes didn’t respond at all. In the logs, the error message failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?” occurred.

I switched to version 2.17.0 and everything works perfectly, even after a longer time of using it.

+1

Sorry for the double post but it may help other people. My problems were ultimately not linked to Portainer but to Docker Swarm, I identified it by observing communication problems on other docker services. And in the end it all came from a port conflict with VMWare: https://portal.portainer.io/knowledge/known-issues-with-vmware

Any update on this?

When i add, volumes: - /var/run/docker.sock:/var/run/docker.sock to portainer-agent-stack.yml file, my system also see 2 environments, swarm and local. swarm comes from the yml file as expected but the added docker.sock creates local environment.

Hi,
probably same issue here, with a small cluster of CT Docker in swarm mode (manager, worker-01 and worker-02):

- OS Debian 12 - PRETTY_NAME=“Debian GNU/Linux 12 (bookworm)” - NAME=“Debian GNU/Linux” - VERSION_ID=“12” - VERSION=“12 (bookworm)” - VERSION_CODENAME=bookworm - ID=debian - HOME_URL=“https://www.debian.org/” - SUPPORT_URL=“https://www.debian.org/support” - BUG_REPORT_URL=“https://bugs.debian.org/”

  • Docker version 24.0.5, build ced0996
  • Portainer-CE Server and Agents version :2.19.0

In maneger:

*docker logs d3e6e3d66e4d*
2023/09/01 05:33PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:359 > encryption key file not present | filename=portainer
2023/09/01 05:33PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:382 > proceeding without encryption key |
2023/09/01 05:33PM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/09/01 05:33PM INF github.com/portainer/portainer/api/chisel/service.go:197 > Found Chisel private key file on disk | private-key=/data/chisel/private-key.pem
2023/09/01 17:33:49 server: Reverse tunnelling enabled
2023/09/01 17:33:49 server: Fingerprint OVNK17d/Zx8vtoYKk7oUDZf3QGvk0Y5rKqSwLh4suOc=
2023/09/01 17:33:49 server: Listening on http://0.0.0.0:8000
2023/09/01 05:33PM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/09/01 05:33PM INF github.com/portainer/portainer/api/cmd/portainer/main.go:634 > starting Portainer | build_number=34280 go_version=1.20.5 image_tag=linux-amd64-2.19.0 nodejs_version=18.17.1 version=2.19.0 webpack_version=5.88.1 yarn_version=1.22.19
2023/09/01 05:33PM INF github.com/portainer/portainer/api/http/server.go:356 > starting HTTPS server | bind_address=:9443
2023/09/01 05:33PM INF github.com/portainer/portainer/api/http/server.go:340 > starting HTTP server | bind_address=:9000
2023/09/01 05:38PM INF github.com/portainer/portainer/api/adminmonitor/admin_monitor.go:62 > the Portainer instance timed out for security purposes, to re-enable your Portainer instance, you will need to restart Portainer

The last version that kept working for me was 2.17.0.