k3s: KubeletInUserNamespace is not set in unprivileged LXD containers when k3s is run as root

Environmental Info:

K3s Version:

• k3s version v1.22.2+k3s2 (3f5774b4)
• go version go1.16.8

Node(s) CPU architecture, OS, and Version:

• Linux u2110 5.13.0-19-generic-Ubuntu SMP Thu Oct 7 21:58:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Ubuntu 21.10 LXD host

Cluster Configuration:

• 1 server (inside LXD on zfs)

Describe the bug:

• KubeletInUserNamespace is not set in unprivileged LXD containers when k3s is run as root

Steps To Reproduce:

• Installed K3s:
• started rootful k3s service inside unprivileged LXD container

Expected behavior:

• cluster comes up

Actual behavior:

• service fails

Additional context / logs:

E1019 02:54:37.458733     215 container_manager_linux.go:456] "Updating kernel flag failed (Hint: enable KubeletInUserNamespace feature flag to ignore the error)" err="open /proc/sys/kernel/panic_on_oops: permission denied" flag="kernel/panic_on_oops"
E1019 02:54:37.458850     215 container_manager_linux.go:456] "Updating kernel flag failed (Hint: enable KubeletInUserNamespace feature flag to ignore the error)" err="open /proc/sys/vm/overcommit_memory: permission denied" flag="vm/overcommit_memory"
E1019 02:54:37.459114     215 kubelet.go:1423] "Failed to start ContainerManager" err="[open /proc/sys/kernel/panic_on_oops: permission denied, open /proc/sys/vm/overcommit_memory: permission denied]"

Backporting

• Needs backporting to older releases

---

Trying to run k3s rootless inside unprivileged LXD on zfs is problematic (btrfs gives a similar error):

WARN[0000] The host root filesystem is mounted as "master:258". Setting child propagation to "" is not supported. 

(This causes sandbox creation to fail)

This error disappears when running rootful k3s inside unprivileged LXD but the service fails due to KubeletInUserNamespace feature gate not being enabled.

An easy way to check if running inside a container is to check /proc/1/environ which contains container=lxc inside LXD containers.