telegraf: Disk input gives error which disappears on restart

NOTE this is currently attributed to an issue with systemd: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1555760

Bug report

System info:

Telegraf version 0.13.1 CoreOS version 1010.5.0 Many plugins running, including the disk input.

Steps to reproduce:

We can reproduce the issue in our environment.

Expected behavior:

Expect the Telegraf disk input to work correctly on startup.

Actual behavior:

The Telegraf disk input reports the following errors every collection cycle.

2016/06/08 18:59:20 Error in input [disk]: error getting disk usage info: too many levels of symbolic links

The input starts working correctly when Telegraf is restarted.

About this issue

Original URL
State: closed
Created 8 years ago
Comments: 27 (13 by maintainers)

Most upvoted comments

I found the cause. My case is that the host binfmt_misc is basically an automount so it will mount at the point of use.

sudo systemctl status proc-sys-fs-binfmt_misc.automount

At first, you can see the following logs.

 proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point
   Loaded: loaded (/lib/systemd/system/proc-sys-fs-binfmt_misc.automount; static; vendor preset: enabled)
   Active: active (waiting) since Mon 2017-02-06 19:24:19 KST; 22h ago
    Where: /proc/sys/fs/binfmt_misc
     Docs: https://www.kernel.org/doc/Documentation/binfmt_misc.txt
           http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

After running telegraf, you can see the following logs.

proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point
   Loaded: loaded (/lib/systemd/system/proc-sys-fs-binfmt_misc.automount; static; vendor preset: enabled)
   Active: active (running) since Mon 2017-02-06 19:24:20 KST; 22h ago
    Where: /proc/sys/fs/binfmt_misc
     Docs: https://www.kernel.org/doc/Documentation/binfmt_misc.txt
           http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems

Feb 07 00:58:50 ip-xxx systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 28579 (telegraf)
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

I’ve never used binfmt_misc on my host before, which means it will not mount. However, since bingmt_misc is mounted to access mount information from telegraf first, since then volume mapping information is included as well, so telegraf can collect information based on disk information without problems.

Dark0096 on Feb 7, 2017

I’m seeing the same issue with all telegraf pods created by deis workflow 2.3.0 on kubernetes 1.3.4, 1.0.0-beta2-18-g755b2ec and CoreOS stable as the host os.

I think @sstarcher is on the right track. If I attach to a telegraf container and traverse the rootfs with find I get the following:

find / -type s
find: '/rootfs/proc/sys/fs/binfmt_misc': Too many levels of symbolic links

The error only occurs for /rootfs/proc/sys/fs/binfmt_misc while /proc/sys/fs/binfmt_misc can be listed just fine.

The output of mount shows:

systemd-1 on /rootfs/proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=0,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11343)

felixbuenemann on Aug 12, 2016