rancher: Rancher 2.5 (Single Install) not starting after nf_conntrack_max value adjustment

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible):

In this scenario, the Docker Host where the Rancher container is started is a LXC container.

root@ubuntu4:~# sysctl -a|grep conntrack_max
net.netfilter.nf_conntrack_max = 524288

root@ubuntu4:~# docker run -d --name rancher --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher:v2.5.8

Installation and setup works fine.

Now the nf_conntrack_max value is set to another value on the LXC host:

root@host ~ # sysctl -w net.netfilter.nf_conntrack_max=525000
net.netfilter.nf_conntrack_max = 525000

Inside the LXC container, the nested Rancher container is restarted:

root@ubuntu4:~# docker restart rancher

root@ubuntu4:~# docker logs --follow rancher
[...]
I0624 07:38:23.051838      54 node.go:136] Successfully retrieved node IP: 172.17.0.2
I0624 07:38:23.051867      54 server_others.go:143] kube-proxy node IP is an IPv4 address (172.17.0.2), assume IPv4 operation
I0624 07:38:23.052887      54 server_others.go:186] Using iptables Proxier.
I0624 07:38:23.053288      54 server.go:650] Version: v1.19.8+k3s1
I0624 07:38:23.053960      54 conntrack.go:103] Set sysctl 'net/netfilter/nf_conntrack_max' to 524288
F0624 07:38:23.053999      54 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied
2021/06/24 07:38:23 [FATAL] k3s exited with: exit status 1

Result: Start of Rancher container fails. At least if the current value is same or higher than the remembered value.

Expected Result: Start of Rancher container succeeds.

Other details that may be helpful:

The problem seems to be in the func realConntracker in k3s/conntrack.go: https://github.com/k3s-io/k3s/blob/master/vendor/k8s.io/kubernetes/cmd/kube-proxy/app/conntrack.go#L98

This function checks the current sysctl value and compares it to a (somewhere internally retained?) internal value:

func (realConntracker) setIntSysCtl(name string, value int) error {
	entry := "net/netfilter/" + name

	sys := sysctl.New()
	if val, _ := sys.GetSysctl(entry); val != value {
		klog.Infof("Set sysctl '%v' to %v", entry, value)
		if err := sys.SetSysctl(entry, value); err != nil {
			return err
		}
	}
	return nil
}

If the current system value doesn’t match the internal value, sys.SetSysctl is launched to set the previously known value. Yet this fails in a LXC container as the value is defined at Host level:

root@ubuntu4:~# sysctl -w net/netfilter/nf_conntrack_max=524288
root@ubuntu4:~# sysctl -a|grep conntrack_max
net.netfilter.nf_conntrack_max = 525000

Probably related: https://github.com/kubernetes-sigs/kind/pull/2241

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.5.8

• Installation option (single install/HA): Single Install

• Docker version (use docker version):

root@ubuntu4:~# docker version
Client:
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.2-0ubuntu1~20.04.2
 Built:             Tue Mar 30 21:24:57 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.2
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       20.10.2-0ubuntu1~20.04.2
  Built:            Mon Mar 29 19:10:09 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4-0ubuntu1~20.04.2
  GitCommit:        
 runc:
  Version:          1.0.0~rc95-0ubuntu1~20.04.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit: