portainer: Warning Messages in Portainer Logs Caused by docker.sock not Available in Docker Swarm Setting

Hi and I am not very familiar with Go but I looked up the code and I think the issue is that a simple docker client is created here: https://github.com/portainer/portainer/blob/develop/api/platform/platform.go#L59

In that case the DefaultDockerHost is used which is unix:///var/run/docker.sock.

You can override that with the DOCKER_HOST environment variable but that does not work either because the simple client refers to the DefaultDockerHost. The warning occurs also each time the Info-Endpoint is called via REST which is quite often 😃

IMHO this should not be a warning. We use a docker socket proxy in between, so we don’t mount the socket directly: https://github.com/Tecnativa/docker-socket-proxy

Alternatively it would be nice to make that overridable.

@jamescarppe Sadly, the recently released 2.20.0 version is still suffering from this issue.

Even when using the standard officially documented installation method for Docker Swarm here: https://docs.portainer.io/start/install-ce/server/swarm/linux#deployment

Portainer starts to spam the logs frequently (more than once per minute) as seen below:

2024/03/19 10:56AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:56AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:56AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:56AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:59AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:59AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:59AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:59AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 10:59AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:00AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:01AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:02AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:02AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:02AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:02AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:02AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2024/03/19 11:03AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

The problem manifests when the Portainer server container does not have /var/run/docker.sock mounted in it, which is not necessary when using Portainer with a remote Portainer agent, as it goes via network. In fact, even when using Portainer in standalone mode (without containers) it should manifest if the user is running Portainer without Docker running locally.

The problem is in the DetermineContainerPlatform() function here: https://github.com/portainer/portainer/blob/61642b8df610712c3cd71efc0e792c53627878d2/api/platform/platform.go#L39-L82

As it can be seen, this function tries to create a client to a Docker daemon unconditionally in line 59. I believe that instead, Portainer should check if /var/run/docker.sock (or more generically, DOCKER_HOST) exists before attempting to do this, and skip the rest of the detection if the socket is not available.

I understand the Portainer team is already working on this (Internal Ref: EE-5372) for a while, but it has been a really long while as of now and the log spamming is really an important issue for us (and I think lots of users conscious of log storage space).

We haven’t been able to migrate from Portainer 2.14.x due to this (and another) issue. I hope this can be fixed once and for all sooner than later before the new LTS hits. Thanks!

@jamescarppe @tamarahenson Is there any update on this issue or a time schedule regarding a fix? 🙂

Not sure if that’s helpful and even the fix for the root cause but I came around the issue by adding into the offical portainer-agent-stack.yml yaml: volumes: - /var/run/docker.sock:/var/run/docker.sock

Since then the warning disappered and I was able to connect properly to the portainer environement (before just an empty portainer had been presented)

Internal Ref: EE-5372

+1 …I got this problem (continual error in my logs) too.

+1

same issue on docker swarm use portainer-agent-stack.yml my test doesn’t have any worker, only 1 manager

portainer_agent

portainer_agent.0.qxeee0ajedcv@pallas    | 2023/12/27 11:55:29 http: TLS handshake error from 10.0.2.3:49700: EOF

portainer_portainer

portainer_portainer.1.ql8dowwiqz2n@pallas    | 2023/12/27 12:16PM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
portainer_portainer.1.ql8dowwiqz2n@pallas    | 2023/12/27 12:16PM ERR github.com/portainer/portainer/api/http/client/client.go:92 > unexpected status code | status_code=403

portainer version 2.19.4 (CE)

docker version

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:07:41 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:07:41 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Hi there, everyone! I’m experiencing the same issue on version 2.19.3-alpine. Do you have any expectations on when it could be fixed?

2023/12/03 10:43AM WRN github.com/portainer/portainer/api/platform/platform.go:70 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/12/03 10:43AM ERR github.com/portainer/portainer/api/http/client/client.go:92 > unexpected status code | status_code=403

I use Docker version 24.0.7

~# docker --version
Docker version 24.0.7, build afdd53b

I was using portainer-ee:218.2 and did not realize any problem, but after upgrading to 2.19.1 log is flooded with theese two errors: 2023/09/22 03:22PM INF docker/snapshot.go:180 > unable to inspect container in other Swarm nodes | error=“Error response from daemon: No such container: d70efd1405298804dc09d3b72e4a8b0a2463799fcbe12478bca093d2dc8a9bc4” container=d70efd1405298804dc09d3b72e4a8b0a2463799fcbe12478bca093d2dc8a9bc4

2023/09/22 03:22PM WRN platform/platform.go:70 > failed to retrieve docker info | error=“Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?”

I can’t say it’s harmless, sometimes the portainer is not showing the “stats” and “exec console” icons at container. and then show after refresh

not sure how to downgrade, if I just deploy the 2.18.2 it’s not working, I see the stacks, but cant go into the stack (error was something like it can’t read compose.yml file)

2 node docker swarm, Docker version 23.0.6

+1

Any update on this?

When i add, volumes: - /var/run/docker.sock:/var/run/docker.sock to portainer-agent-stack.yml file, my system also see 2 environments, swarm and local. swarm comes from the yml file as expected but the added docker.sock creates local environment.

The warning comes periodically as well with 2.18.3 when using Docker Standalone > API with a Docker API URL as environment.

Issue is stiil actual for 2.18.3

Just migrated to Portainer 2.18.2 today and started seeing those messages as well. Using with Docker Swarm clusters.

Just a heads-up that this issue is still present in the newly released 2.18.3 version as well 😦

There are periodic error messages in the console logs:

2023/05/23 12:02PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:02PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:02PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:03PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:03PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:04PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:06PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:06PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"
2023/05/23 12:06PM WRN github.com/portainer/portainer/api/platform/platform.go:68 > failed to retrieve docker info | error="Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

This has the potential to fill up any logger system in place for Portainer, including the default Docker logger driver that does not rotate/limit logs by default.