containerd: pod healthchecks fail when host IPv6 is disabled

Description

Our kubernetes hosts have IPv6 disabled via the following sysctl settings:

net.ipv6.conf.all.disable_ipv6 =  1
net.ipv6.conf.default.disable_ipv6 = 1

When containerd starts the pod, it does so without honouring those sysctl values, and IPv6 is therefore enabled in the pod.

kubelet creates an /etc/hosts file in the pod with the following lines (amongst others):

127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback

Given the app reasonably expects IPv6 to be disabled on the pod (it was disabled when running under docker after all), it only tries to bind to the IPv4 interface.

The pod’s healthcheck, trying to reach http://localhost:8080, then fails because:

IPv6 being enabled in the pod causes localhost to resolve to ::1
The app isn’t bound to ::1
Healthcheck fails with a connection refused error.

Steps to reproduce the issue

Describe the results you received and expected

Results received:

DNS resolves localhost to ::1
Pod healthchecks fail with a connection refused error

Results expected:

DNS resolves localhost to 127.0.0.1
Pod healthchecks pass

What version of containerd are you using?

1.6.12

Any other relevant information

$ runc --version
runc version 1.1.4-0ubuntu1~22.04.3
spec: 1.0.2-dev
go: go1.18.1
libseccomp: 2.5.3

$ nerdctl info
Client:
 Namespace:	k8s.io
 Debug Mode:	false

Server:
 Server Version: 1.6.12-0ubuntu1~22.04.3
 Storage Driver: overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Log: fluentd journald json-file syslog
  Storage: native overlayfs
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.19.0-1028-aws
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.83GiB

$ uname -a
Linux ip-xx-xx-xx-xx 5.19.0-1028-aws #29~22.04.1-Ubuntu SMP Tue Jun 20 19:12:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Show configuration if it is related to CRI plugin.

No response

About this issue

Original URL
State: closed
Created a year ago
Comments: 18 (7 by maintainers)

Most upvoted comments

Ah, that’s an interesting line! We don’t explicitly disable IPv6 on the lo interface using the net/ipv6/conf/lo/disable_ipv6 setting but instead expect it to be disabled because net/ipv6/conf/all/disable_ipv6 is set. I’ll try setting that sysctl parameter when I’m back at work on Monday and see if that helps at all. That said, it’s still interesting to me that it worked with the existing settings under docker.

mattburgess on Aug 11, 2023