vigil: `internal error` when pinging self

Hi!

Thanks for vigil, it is a great peace of software. I have however found an issue which is properly related to the ping library used by vigil.

While it may seem odd to ping an IP on the same machine that the uptime monitor is running I still think it is misleading that it says internal error. If anything you should just get a stupid monitor that would always be up (because when it is down the uptime monitor itself is not running).

I have confirmed this issue exists on several machines some with Rocky Linux others with Debian bookworm with ARM and Intel CPU architecture.

In order to eliminate possible sources of errors I did a clean install of the latest stable version of Debian and installed vigil with this service block (here the local machine has the IPv4 192.168.1.122 from DHCP):

[[probe.service.node]]

id = "router"
label = "Core main router"
mode = "poll"

replicas = [
  "icmp://192.168.1.222",
  "icmp://[::1]",
  "icmp://localhost"
]

Which results in errors such as:

Feb 03 18:19:11 vigil-test vigil[4100]: (INFO) - starting 4 workers
Feb 03 18:19:11 vigil-test vigil[4100]: (INFO) - Actix runtime found; starting in Actix runtime
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will fire for icmp host: 192.168.1.222 (1 targets)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will send icmp ping to target: 192.168.1.222 from host: 192.168.1.222
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll error for icmp target: 192.168.1.222 from host: 192.168.1.222 (error: internal error)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - will probe replica: ICMP("192.168.1.222") with retry count: 1
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will fire for icmp host: localhost (2 targets)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will send icmp ping to target: ::1 from host: localhost
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll error for icmp target: ::1 from host: localhost (error: internal error)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - will probe replica: ICMP("localhost") with retry count: 1
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will fire for icmp host: ::1 (1 targets)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will send icmp ping to target: ::1 from host: ::1
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll error for icmp target: ::1 from host: ::1 (error: internal error)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - will probe replica: ICMP("::1") with retry count: 1
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will fire for icmp host: 192.168.1.222 (1 targets)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will send icmp ping to target: 192.168.1.222 from host: 192.168.1.222
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll error for icmp target: 192.168.1.222 from host: 192.168.1.222 (error: internal error)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - will probe replica: ICMP("192.168.1.222") with retry count: 2
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will fire for icmp host: ::1 (1 targets)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will send icmp ping to target: ::1 from host: ::1
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll error for icmp target: ::1 from host: ::1 (error: internal error)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - will probe replica: ICMP("::1") with retry count: 2
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will fire for icmp host: localhost (2 targets)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll will send icmp ping to target: ::1 from host: localhost
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - prober poll error for icmp target: ::1 from host: localhost (error: internal error)
Feb 03 18:19:12 vigil-test vigil[4100]: (DEBUG) - will probe replica: ICMP("localhost") with retry count: 2
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - prober poll will fire for icmp host: 192.168.1.222 (1 targets)
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - prober poll will send icmp ping to target: 192.168.1.222 from host: 192.168.1.222
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - prober poll error for icmp target: 192.168.1.222 from host: 192.168.1.222 (error: internal error)
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - replica probe result: web:router:icmp://192.168.1.222 => Dead
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - prober poll will fire for icmp host: ::1 (1 targets)
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - prober poll will send icmp ping to target: ::1 from host: ::1
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - prober poll error for icmp target: ::1 from host: ::1 (error: internal error)
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - replica probe result: web:router:icmp://[::1] => Dead
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - prober poll will fire for icmp host: localhost (2 targets)
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - prober poll will send icmp ping to target: ::1 from host: localhost
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - prober poll error for icmp target: ::1 from host: localhost (error: internal error)
Feb 03 18:19:13 vigil-test vigil[4100]: (DEBUG) - replica probe result: web:router:icmp://localhost => Dead
Feb 03 18:19:13 vigil-test vigil[4100]: (INFO) - replicas have been probed with 3/4 threads in 1.504252723s

Environment info:

root@vigil-test:/home/ek# ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.1.222/24 brd 192.168.1.255 scope global dynamic enp1s0
       valid_lft 6368sec preferred_lft 6368sec
root@vigil-test:/home/ek# uname -a
Linux vigil-test 6.1.0-17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30) x86_64 GNU/Linux
root@vigil-test:/home/ek# vigil --version
vigil-server 1.26.3

I had some issues with vigil and I spend quite a lot of time to figure out why vigil did not work. I used local IP’s because then I would eliminate all kinds of other issues that could be with the network, not knowing that the system could not ping local IP’s.

Thanks a lot in advance.

I hope this report contains enough info to make the next step to solve this issue actionable.

Best regards, Emil Kristensen

About this issue

  • Original URL
  • State: open
  • Created 5 months ago
  • Comments: 20 (7 by maintainers)

Most upvoted comments

FYI: The issue has been fixed upstream.

+1
eKristensen on Feb 24, 2024

Thank you, leaving this open so that I can review it & change when I batch process task on Vigil (probably not in the immediate future to be fully transparent, as I have other priorities atm, so this might take quite some time).

+1
valeriansaliou on Feb 8, 2024

Also note that vigil-local uses the same library, w/ no issue pinging localhost and LAN hosts in my production environments: https://github.com/valeriansaliou/vigil-local/blob/master/src/probe/poll.rs#L147

I’ll test and see what I find.

Though doing all this on the side for to monitor it systems as a part of some volunteer work it might take some time between work to get a proper look, but that is life.

I did not expect you to respond so quickly in here 💪, keep up the good work 😃

+1
eKristensen on Feb 6, 2024

Also note that vigil-local uses the same library, w/ no issue pinging localhost and LAN hosts in my production environments: https://github.com/valeriansaliou/vigil-local/blob/master/src/probe/poll.rs#L147

+1
valeriansaliou on Feb 5, 2024

The code is here: https://github.com/valeriansaliou/vigil/blob/master/src/prober/manager.rs#L301

+1
valeriansaliou on Feb 5, 2024
Read more comments on GitHub
← vscode-ltex: LTeX checking seems to be stuck when certain text is in file with de-de as selected language
validator: link rel=apple-touch-icon no longer valid? →