datadog-agent: Too many levels of symbolic links occur again in datadog-agent:6.11.0

Output of the info page (if this is a bug)

(Paste the output of the info page here)

Describe what happened:

CORE | WARN | (pkg/collector/py/datadog_agent.go:148 in LogMessage) | (disk.py:114) | Unable to get disk metrics for /host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links: '/host/proc/sys/fs/binfmt_misc'

Describe what you expected: No error

Steps to reproduce the issue:

Additional environment details (Operating System, Cloud provider, etc): docker inspect

...
"Mounts": [
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/var/run/docker.sock",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/sys/fs/cgroup",
                "Destination": "/host/sys/fs/cgroup",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/proc",
                "Destination": "/host/proc",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            },
]
...

amazonlinux - ami-0c57dafd95a102862

docker run -d -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -p 127.0.0.1:8126:8126/tcp \
              -e DD_API_KEY=<MY_API_KEY> \
              -e DD_APM_ENABLED=true \
              datadog/agent:latest

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 19
Comments: 15 (3 by maintainers)

Most upvoted comments

Thanks for sharing your tip @johnpekcan! I actually found another way to make it work:

datadog:
  # See: https://github.com/DataDog/datadog-agent/issues/1961#issuecomment-446918067
  confd:
    disk.yaml: |-
      init_config:

      instances:
        - use_mount: false
          file_system_blacklist:
            - autofs$
          mount_point_blacklist:
            - /proc/sys/fs/binfmt_misc
            - /host/proc/sys/fs/binfmt_misc

+15

arthurio on Mar 6, 2020

I’m getting this error running the agent on Kubernetes. Surely there is a simple way to stop the agent from spamming the logs with this warning?

ianwalter on Jan 14, 2020

Hello and thanks for reporting this!

I’m able to reproduce this after the container instance boot when systemd-binfmt is in an inactive/dead state.

This service unit file comes with the following conditions:

ConditionPathIsReadWrite=/proc/sys/
ConditionDirectoryNotEmpty=|/lib/binfmt.d
ConditionDirectoryNotEmpty=|/usr/lib/binfmt.d
ConditionDirectoryNotEmpty=|/usr/local/lib/binfmt.d
ConditionDirectoryNotEmpty=|/etc/binfmt.d
ConditionDirectoryNotEmpty=|/run/binfmt.d

Interestingly, by making sure one of these is met or by simply removing them, then restarting the service, the symlink recursion issue is gone.

The most viable approach is to ignore that mount point in the disk check by uncommenting these lines.

An example of passing check configuration to the agent with docker is detailed here.

Hope this helps!

xlucas on May 14, 2019

Our workaround ending up being to explicitly mount the folder. So, we originally just had the mount /proc/:/host/proc/:ro. So we then had to add /proc/sys/fs/binfmt_misc/:/host/proc/sys/fs/binfmt_misc/:ro to get around the symlink issue.

jmchuster on Jul 21, 2020

uncommenting these lines.

Is there a way to pass it as an ENV Variable? https://docs.datadoghq.com/agent/docker/?tab=standard#installation ?

minnieko on Mar 13, 2020

Ooh, much more surgical! Thank you 😇

johnpekcan on Mar 6, 2020

Hi @xlucas, I’m still seeing these warnings in my logs, and I’m not sure how to change the configuration for the docker agent. I’m using Convox, and I have this in my convox.yml (which is similar to docker-compose.yml):

datadog:
    agent:
      ports:
        - 8125/udp
        - 8126/tcp
    image: datadog/agent:latest
    environment:
      - DD_API_KEY
      - DD_APM_ENABLED=true
      - DD_PROCESS_AGENT_ENABLED=true
    privileged: true
    scale:
      cpu: 128
      memory: 128
    volumes:
      - /sys/fs/cgroup/:/host/sys/fs/cgroup/
      - /proc/:/host/proc/
      - /var/run/docker.sock:/var/run/docker.sock

How can I modify this config to remove these warnings:

WARN | (pkg/collector/python/datadog_agent.go:118 in LogMessage) | disk:e5dffb8bef24336f | (disk.py:75) | Unable to get disk metrics for /host/proc/sys/fs/binfmt_misc: [Errno 40] Too many levels of symbolic links: '/host/proc/sys/fs/binfmt_misc'

Thanks!

ndbroadbent on Nov 23, 2019