node_exporter: ARP collector error: rtnetlink NeighMessage has a wrong attribute data length

Host operating system: output of uname -a

Linux 5.14.21-150500.55.36-default #1 SMP PREEMPT_DYNAMIC Tue Oct 31 08:37:43 UTC 2023 (e7a2e23) x86_64 x86_64 x86_64 GNU/Linux

openSUSE Leap 15.5

node_exporter version: output of node_exporter --version

node_exporter, version 1.7.0 (branch: master, revision: 78af952e638b5e0d00640fbdeefd096df4a51dc2)
  build user:       ~~~
  build date:       ~~~
  go version:       go1.21.4.1
  platform:         linux/amd64
  tags:             netgo osusergo static_build

node_exporter command line flags

defaults

node_exporter log output

ts=2023-11-17T14:15:00.958Z caller=collector.go:169 level=error msg="collector failed" name=arp duration_seconds=0.000270392 err="could not get ARP entries: rtnetlink NeighMessage has a wrong attribute data length"

Are you running node_exporter in Docker?

no

What did you do that produced an error?

upgraded from version 1.6.0

What did you expect to see?

node_arp_entries node_scrape_collector_success{collector="arp"} 1

What did you see instead?

node_scrape_collector_success{collector="arp"} 0

About this issue

  • Original URL
  • State: open
  • Created 7 months ago
  • Reactions: 5
  • Comments: 24 (13 by maintainers)

Commits related to this issue

Most upvoted comments

This looks like it is the issue currently. It appears iproute2 silently ignores this entry.

The ll_addr_n2a function in the iproute2 source code sheds some light on how it handles various length NDA_LLADRs:

const char *ll_addr_n2a(const unsigned char *addr, int alen, int type,
            char *buf, int blen)
{
    int i;
    int l;

    if (alen == 4 &&
        (type == ARPHRD_TUNNEL || type == ARPHRD_SIT
         || type == ARPHRD_IPGRE))
        return inet_ntop(AF_INET, addr, buf, blen);

    if (alen == 16 && (type == ARPHRD_TUNNEL6 || type == ARPHRD_IP6GRE))
        return inet_ntop(AF_INET6, addr, buf, blen);
    if (alen == 7 && type == ARPHRD_AX25)
        return ax25_ntop(AF_AX25, addr, buf, blen);
    if (alen == 7 && type == ARPHRD_NETROM)
        return netrom_ntop(AF_NETROM, addr, buf, blen);
    if (alen == 5 && type == ARPHRD_ROSE)
        return rose_ntop(AF_ROSE, addr, buf, blen);

    snprintf(buf, blen, "%02x", addr[0]);
    for (i = 1, l = 2; i < alen && l < blen; i++, l += 3)
        snprintf(buf + l, blen - l, ":%02x", addr[i]);
    return buf;
}

The type is taken from the interface with which the link-local address is associated. Some of these types (ARPHRD_NETROM, ARPHRD_AX25) are quite archaic, and probably not likely to be encountered in most environments. However, several of them are still in widespread use, e.g. the various tunnel types.

cf. (abridged):

/* ARP protocol HARDWARE identifiers. */
#define ARPHRD_NETROM	0		/* from KA9Q: NET/ROM pseudo	*/
#define	ARPHRD_AX25	3		/* AX.25 Level 2		*/

/* Dummy types for non ARP hardware */
#define ARPHRD_ROSE	270
#define ARPHRD_TUNNEL	768		/* IPIP tunnel			*/
#define ARPHRD_TUNNEL6	769		/* IP6IP6 tunnel       		*/
#define ARPHRD_SIT	776		/* sit0 device - IPv6-in-IPv4	*/
#define ARPHRD_IPGRE	778		/* GRE over IP			*/
#define ARPHRD_IP6GRE	823		/* GRE over IPv6		*/

Technically, the ll_addr_n2a function will handle a NDA_LLADDR with zero-length alen:

    snprintf(buf, blen, "%02x", addr[0]);
    for (i = 1, l = 2; i < alen && l < blen; i++, l += 3)
        snprintf(buf + l, blen - l, ":%02x", addr[i]);
    return buf;

Since this code would still expect at least one byte of valid data in addr[0], I suspect that elsewhere in the iproute2 source code they avoid calling this function with zero-length alen.

You could be on to something here. Let me see if I can make a reproducer.

This looks like it is the issue currently. It appears iproute2 silently ignores this entry.

I think the correct course of action would be to allow zero-length NDA_LLADDR attributes on a neighbor entry, as clearly the kernel thinks it’s OK. This could have some effects to consumers of the module, as they would now have to manually filter out these entries.