node_exporter: netdev collector failing with 'couldn't get netstats: incorrect size' since 1.4.0
Host operating system: output of uname -a
Linux RT-AX86U-FAC0 4.1.52 #2 SMP PREEMPT Sun Jul 24 18:28:51 EDT 2022 aarch64 ASUSWRT-Merlin
node_exporter version: output of node_exporter --version
1.4.0 (TODO: Grabbing exact output momentarily…)
node_exporter command line flags
N/A
Are you running node_exporter in Docker?
No
What did you do that produced an error?
Tried starting node_exporter on my ASUS RT-AX86U router, and hitting the /metrics endpoint.
What did you expect to see?
No errors in the log output.
What did you see instead?
The following error in the log output:
ts=2022-10-06T00:48:07.144Z caller=collector.go:169 level=error msg="collector failed" name=netdev duration_seconds=0.003381482 err="couldn't get netstats: incorrect size, want: 92 or 96"
Here are the contents of net/dev:
admin@RT-AX86U-FAC0:/tmp/mnt/SANDISK/node_exporter# cat /proc/net/dev
Inter-| Receive | Transmit
face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed
spu_us_dummy: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
sit0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
imq1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
eth7: 1140948 2880 0 21 0 0 0 238 2684974 5992 0 570 0 0 0 0
lo: 771360 2822 0 0 0 0 0 0 771360 2822 0 0 0 0 0 0
eth3: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
br0: 7475066 23742 0 69 0 0 0 0 21374265 29308 0 0 0 0 0 0
bcmsw: 48632726 94292 0 95 0 0 0 6792 53283823 99930 0 0 0 0 0 0
ifb1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
imq0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
eth6: 1635313 4029 0 0 0 14646 0 0 1809752 7811 5 0 0 0 0 0
eth2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ip6tnl0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ifb0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ip6gre0: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
eth5: 6249677 22374 0 106 0 0 0 970 20705170 29611 0 0 0 0 0 0
eth1: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
bcmswlpbk0: 20705170 29611 0 0 0 0 0 1939 6239336 22279 0 0 0 0 0 0
dpsta: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
imq2: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
spu_ds_dummy: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
eth4: 756 8 0 0 0 0 0 3 995142 3460 0 0 0 0 0 0
eth0: 14370789 16652 0 0 0 0 0 128 3529271 11601 0 0 0 0 0 0
Downgrading to node_exporter 1.3.1 linux arm64 release seems to fix the problem (I also had no issue on older versions as well).
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 18 (7 by maintainers)
Commits related to this issue
- Add procfs fallback to netdev collector Some systems have broken netlink messages due to patched kernels. Since these messages can not be parsed, add a flag to fall back to parsing from `/proc/net/de... — committed to prometheus/node_exporter by SuperQ 2 years ago
- Add procfs fallback to netdev collector (#2509) Some systems have broken netlink messages due to patched kernels. Since these messages can not be parsed, add a flag to fall back to parsing from `/p... — committed to prometheus/node_exporter by SuperQ 2 years ago
Yup, here it is: https://github.com/RMerl/asuswrt-merlin.ng/blob/cc86026010d319771af97e87f7bf0025dbaebb3e/release/src-rt-5.02L.07p2axhnd/kernel/linux-4.1/include/uapi/linux/if_link.h#L18
There are additional fields added to the standard struct… thats an additional 6*4 bytes. I’ll be honest, I have no idea how to fix that, especially since they are added in an arbitrary location. 😦
Ah,
GOOS=linux GOARCH=arm64 make builddid the trick.And now getting: