lighthouse: Validator Client intermittently freezes on Linux kernel `6.14.4` -> `6.14.7`

Summary

The Validator Client on Linux with kernel versions 6.14.4 -> 6.14.7 will intermittently freeze, stopping it from performing its duties.

If you’re running a distro which closely follows Linux mainline (such as Arch Linux or Fedora) you may be affected.

Run uname -r to check if your kernel version is in the affected range.

Note: Ubuntu 24.04 and older use older kernels and thus are not affected. However, Ubuntu 25.04 is currently running the 6.14.0 kernel which is unaffected but it is possible that a future package upgrade will include one of the affected kernels so upgrade with caution.

Solutions and Workarounds

This bug, caused by changes to the eventpoll code, has already been patched in the Linux mainline kernel and will be fixed in 6.14.8+.

Once your distro allows you to update the kernel to 6.14.8 you can safely do so.

In the meantime, if you are running an affected kernel version you have a few options:

Install an LTS kernel

The procedure for this will differ depending on your distro but the below example is the instructions for Arch Linux:

sudo pacman -S linux-lts

If you are using systemd-boot, it should automatically generate the corresponding bootloader entries.

If you are using grub you will need to regenerate them:

sudo grub-mkconfig -o /boot/grub/grub.cfg

Reboot and you should see linux-lts included in the grub menu.

Downgrade your kernel

This will vary depending on your distro and for some distros it is very involved. Here are the instructions for Fedora 41:

# Find available kernels
sudo dnf list kernel --showduplicates

# Install a specific kernel. For example:
sudo dnf install kernel-6.11.4-301.fc41

grub entries will automatically be added, so reboot and select the new kernel from the list.

Run an API polling script

If you do not want to touch your kernel in case you break something there is a simple bash script you can run instead.

Due to the internals of eventpoll, when the VC receives an API call, it will wake from its freeze.

Because of this we can use a script running in the background which continuously polls the VC. Here is an example of such a script:

while sleep 5; do curl -s --fail "http://localhost:5062/lighthouse/auth" > /dev/null && echo "polled at $(date)"; done

This will keep the VC awake. Note that running a full VC metrics server with Grafana polling the VC will also keep it awake for the same reason.

Acknowledgments

A huge thank you to the users on Discord who discovered this issue and assisted in diagnosis and testing, particularly @smooth.ninja and @ChosunOne.


See https://github.com/tokio-rs/tokio/issues/7335 and #7403 for more details

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 18 (11 by maintainers)

Most upvoted comments

Hello everyone, I’ve raised an issue on the Tokio repo: https://github.com/tokio-rs/tokio/issues/7335

I managed to reproduce the issue with a very simple program: https://github.com/macladson/tokio-lock-example This is helpful because you won’t need any validator keys to test it.

This confirms it is not a Lighthouse specific issue.

In the meantime, either use the API polling trick, OR use the Arch linux-lts kernel. Both will prevent the issue.

Another thing you can try is limiting the VC to only 2 threads. I am yet to reproduce the issue when running that way. You can either use cgroups in systemd or run the VC like taskset -c 0,1 lighthouse vc .... If you still get a freeze like this, please let us know.

If anyone finds anything else, please post here

Just a PSA that 6.14.9 was released on Arch Linux today which should fix the issue (I guess they skipped 6.14.8)

I’ve unpinned this, as most users are no longer affected (have upgraded to a new kernel), and this is unlikely to be an issue for stable distros

I’ve just pinned this issue and updated the description to explain our current findings and solutions. 6.14.8 will likely be released for Arch Linux in next couple of days so this issue should become less of a problem soon. We will have to keep monitoring to see if it ends up affecting any of the major Ubuntu releases but my hunch is that they will likely skip these versions.

@michaelsproul I just got another hang on 6.14.6, started working again when I started the curl cmd up again.