cilium: Cilium fails to start with "RTNETLINK answers: Operation not supported"

I think I can reproduce the issue by hacking cilium agent kernel config probe code to always return false so cilium agent think the system is missing kernel config file from both /boot/config-xxxx and /proc/config.gz

diff --git a/pkg/datapath/linux/probes/probes.go b/pkg/datapath/linux/probes/probes.go
index c0e01d69b8..28b0af79e5 100644
--- a/pkg/datapath/linux/probes/probes.go
+++ b/pkg/datapath/linux/probes/probes.go
@@ -420,5 +420,5 @@ func (p *ProbeManager) KernelConfigAvailable() bool {
                }
        }
 
-       return true
+       return false
 }

so there is no error about system config check, but later errored due to missing required kernel config option

level=info msg="Cilium 1.12.90 d8909fbda4 2022-08-11T14:08:51+00:00 go version go1.19 linux/amd64" subsys=daemon
level=info msg="cilium-envoy  version: 5739e4be8ae7134fee683d920d25c3732ac6c819/1.21.5/Distribution/RELEASE/BoringSSL" subsys=daemon
level=info msg="clang (10.0.0) and kernel (5.19.0) versions: OK!" subsys=linux-datapath
level=info msg="linking environment: OK!" subsys=linux-datapath
level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf
...SNIP..
level=warning msg="+ tc qdisc replace dev cilium_vxlan clsact" subsys=datapath-loader
level=warning msg="Error: Cannot find ingress queue for specified device." subsys=datapath-loader

then I made code change below in addition to above code hack, and rebuild the image, I can see correct log message

diff --git a/pkg/datapath/linux/requirements.go b/pkg/datapath/linux/requirements.go
index 199c135716..b12452550c 100644
--- a/pkg/datapath/linux/requirements.go
+++ b/pkg/datapath/linux/requirements.go
@@ -146,7 +146,7 @@ func CheckMinRequirements() {
                if err := probeManager.SystemConfigProbes(); err != nil {
                        errMsg := "BPF system config check: NOT OK."
                        // TODO(brb) warn after GH#14314 has been resolved
-                       if !errors.Is(err, probes.ErrKernelConfigNotFound) {
+                       if errors.Is(err, probes.ErrKernelConfigNotFound) {
                                log.WithError(err).Warn(errMsg)
                        }
                }

correct log:

level=info msg="Cilium 1.12.90 d8909fbda4 2022-08-11T14:08:51+00:00 go version go1.19 linux/amd64" subsys=daemon
level=info msg="cilium-envoy  version: 5739e4be8ae7134fee683d920d25c3732ac6c819/1.21.5/Distribution/RELEASE/BoringSSL" subsys=daemon
level=info msg="clang (10.0.0) and kernel (5.19.0) versions: OK!" subsys=linux-datapath
level=info msg="linking environment: OK!" subsys=linux-datapath
level=warning msg="BPF system config check: NOT OK." error="Kernel Config file not found" subsys=linux-datapath
...SNIP...
level=warning msg="+ tc qdisc replace dev cilium_vxlan clsact" subsys=datapath-loader
level=warning msg="Error: Cannot find ingress queue for specified device." subsys=datapath-loader

looks to be a minor log bug that I can fix 😃

@borkmann:

config.txt

@dirien I don’t see CONFIG_NET_CLS_ACT=y, probably missing kernel config ? here is kernel requirements

https://docs.cilium.io/en/stable/operations/system_requirements/

CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_NET_CLS_BPF=y
CONFIG_BPF_JIT=y
CONFIG_NET_CLS_ACT=y
CONFIG_NET_SCH_INGRESS=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_BPF=y

Many thanks @borkmann and @vincentmli for your feedback!

I switched my image from the jammy minimal to “normal” image and now it works as all kernel modules are available!

@dirien Thanks! I think given it’s an distro issue not providing the needed modules, we can close this one here. Could you open a ticket at Canonical so they get this fixed?

@borkmann: config.txt

@dirien I don’t see CONFIG_NET_CLS_ACT=y, probably missing kernel config ? here is kernel requirements

Agree, could you open an issue at Canonical that this is missing in their jammy cloud image?

Also, did the agent log on startup warn you about missing config?

https://github.com/cilium/cilium/blob/b4b750002c229d559aa83b80f7aca30a8084a0bb/pkg/datapath/linux/probes/probes.go#L211

@borkmann, I will recreate the VM to find the informations.

@dirien do you have the kernel config for the jammy cloud image?