cilium: failed to load bpf_lxc.o with ubuntu 2004 and cilium 1.10.6

Is there an existing issue for this?

• I have searched the existing issues

What happened?

I’m using the public built image directly:

cilium/cilium:v1.10.6
cilium/operator-generic:v1.10.6

And the bpf_lxc.o is failed to load.

cilium configuration and log: cilium.log

It seems like a regression introduced in 1.10.6. I can’t reproduce it using 1.10.5

Cilium Version

Client: 1.10.6 17d3d15 2021-12-09T18:34:11-08:00 go version go1.16.11 linux/amd64 Daemon: 1.10.6 17d3d15 2021-12-09T18:34:11-08:00 go version go1.16.11 linux/amd64

Kernel Version

5.4.0-42-generic

Kubernetes Version

Client Version: version.Info{Major:“1”, Minor:“23”, GitVersion:“v1.23.1”, GitCommit:“86ec240af8cbd1b60bcc4c03c20da9b98005b92e”, GitTreeState:“clean”, BuildDate:“2021-12-16T11:41:01Z”, GoVersion:“go1.17.5”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“21”, GitVersion:“v1.21.5-gke.1200”, GitCommit:“90a16981ade07f163a0233adb631b42ac1fc53ff”, GitTreeState:“clean”, BuildDate:“2021-10-04T09:25:23Z”, GoVersion:“go1.16.7b7”, Compiler:“gc”, Platform:“linux/amd64”} WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

Sysdump

No response

Relevant log output

level=error msg="Command execution failed" cmd="[tc filter replace dev lxcfd73d634716c ingress prio 1 handle 1 bpf da obj 2722_next/bpf_lxc.o sec from-container]" error="exit status 1" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="BTF debug data section '.BTF' rejected: Invalid argument (22)!" subsys=datapath-loader
level=warning msg=" - Length:       27165" subsys=datapath-loader
level=warning msg="Verifier analysis:" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="magic: 0xeb9f" subsys=datapath-loader
level=warning msg="version: 1" subsys=datapath-loader
level=warning msg="flags: 0x0" subsys=datapath-loader
level=warning msg="hdr_len: 24" subsys=datapath-loader
level=warning msg="type_off: 0" subsys=datapath-loader
level=warning msg="type_len: 2116" subsys=datapath-loader
level=warning msg="str_off: 2116" subsys=datapath-loader
level=warning msg="str_len: 25025" subsys=datapath-loader
level=warning msg="btf_total_size: 27165" subsys=datapath-loader
level=warning msg="[1] PTR (anon) type_id=2" subsys=datapath-loader
level=warning msg="[2] STRUCT __sk_buff size=184 vlen=32" subsys=datapath-loader
level=warning msg="\tlen type_id=3 bits_offset=0" subsys=datapath-loader
level=warning msg="\tpkt_type type_id=3 bits_offset=32" subsys=datapath-loader
level=warning msg="\tmark type_id=3 bits_offset=64" subsys=datapath-loader
level=warning msg="\tqueue_mapping type_id=3 bits_offset=96" subsys=datapath-loader
level=warning msg="\tprotocol type_id=3 bits_offset=128" subsys=datapath-loader
level=warning msg="\tvlan_present type_id=3 bits_offset=160" subsys=datapath-loader
level=warning msg="\tvlan_tci type_id=3 bits_offset=192" subsys=datapath-loader
level=warning msg="\tvlan_proto type_id=3 bits_offset=224" subsys=datapath-loader
level=warning msg="\tpriority type_id=3 bits_offset=256" subsys=datapath-loader
level=warning msg="\tingress_ifindex type_id=3 bits_offset=288" subsys=datapath-loader
level=warning msg="\tifindex type_id=3 bits_offset=320" subsys=datapath-loader
level=warning msg="\ttc_index type_id=3 bits_offset=352" subsys=datapath-loader
level=warning msg="\tcb type_id=5 bits_offset=384" subsys=datapath-loader
level=warning msg="\thash type_id=3 bits_offset=544" subsys=datapath-loader
level=warning msg="\ttc_classid type_id=3 bits_offset=576" subsys=datapath-loader
level=warning msg="\tdata type_id=3 bits_offset=608" subsys=datapath-loader
level=warning msg="\tdata_end type_id=3 bits_offset=640" subsys=datapath-loader
level=warning msg="\tnapi_id type_id=3 bits_offset=672" subsys=datapath-loader
level=warning msg="\tfamily type_id=3 bits_offset=704" subsys=datapath-loader
level=warning msg="\tremote_ip4 type_id=3 bits_offset=736" subsys=datapath-loader
level=warning msg="\tlocal_ip4 type_id=3 bits_offset=768" subsys=datapath-loader
level=warning msg="\tremote_ip6 type_id=7 bits_offset=800" subsys=datapath-loader
level=warning msg="\tlocal_ip6 type_id=7 bits_offset=928" subsys=datapath-loader
level=warning msg="\tremote_port type_id=3 bits_offset=1056" subsys=datapath-loader
level=warning msg="\tlocal_port type_id=3 bits_offset=1088" subsys=datapath-loader
level=warning msg="\tdata_meta type_id=3 bits_offset=1120" subsys=datapath-loader
level=warning msg="\t(anon) type_id=8 bits_offset=1152" subsys=datapath-loader
level=warning msg="\ttstamp type_id=10 bits_offset=1216" subsys=datapath-loader
level=warning msg="\twire_len type_id=3 bits_offset=1280" subsys=datapath-loader
level=warning msg="\tgso_segs type_id=3 bits_offset=1312" subsys=datapath-loader
level=warning msg="\t(anon) type_id=12 bits_offset=1344" subsys=datapath-loader
level=warning msg="\tgso_size type_id=3 bits_offset=1408" subsys=datapath-loader
level=warning msg="[3] TYPEDEF __u32 type_id=4" subsys=datapath-loader
level=warning msg="[4] INT unsigned int size=4 bits_offset=0 nr_bits=32 encoding=(none)" subsys=datapath-loader
level=warning msg="[5] ARRAY (anon) type_id=3 index_type_id=6 nr_elems=5" subsys=datapath-loader
level=warning msg="[6] INT __ARRAY_SIZE_TYPE__ size=4 bits_offset=0 nr_bits=32 encoding=(none)" subsys=datapath-loader
level=warning msg="[7] ARRAY (anon) type_id=3 index_type_id=6 nr_elems=4" subsys=datapath-loader
level=warning msg="[8] UNION (anon) size=8 vlen=1" subsys=datapath-loader
level=warning msg="\tflow_keys type_id=9 bits_offset=0" subsys=datapath-loader
level=warning msg="[9] PTR (anon) type_id=75" subsys=datapath-loader
level=warning msg="[10] TYPEDEF __u64 type_id=11" subsys=datapath-loader
level=warning msg="[11] INT long long unsigned int size=8 bits_offset=0 nr_bits=64 encoding=(none)" subsys=datapath-loader
level=warning msg="[12] UNION (anon) size=8 vlen=1" subsys=datapath-loader
level=warning msg="\tsk type_id=13 bits_offset=0" subsys=datapath-loader
level=warning msg="[13] PTR (anon) type_id=76" subsys=datapath-loader
level=warning msg="[14] FUNC_PROTO (anon) return=15 args=(1 ctx)" subsys=datapath-loader
level=warning msg="[15] INT int size=4 bits_offset=0 nr_bits=32 encoding=SIGNED" subsys=datapath-loader
level=warning msg="[16] FUNC __send_drop_notify type_id=14 vlen != 0" subsys=datapath-loader
level=warning subsys=datapath-loader
level=warning msg="Log buffer too small to dump verifier log 16777215 bytes (10 tries)!" subsys=datapath-loader
level=warning msg="Error filling program arrays!" subsys=datapath-loader
level=warning msg="Unable to load program" subsys=datapath-loader
level=warning msg="JoinEP: Failed to load program" containerID=5e049f73cf datapathPolicyRevision=0 desiredPolicyRevision=6 endpointID=2722 error="Failed to load prog with tc: exit status 1" file-path=2722_next/bpf_lxc.o identity=46365 ipv4=192.168.2.159 ipv6= k8sPodName=kube-system/csi-snapshot-controller-7469856f78-hs5z9 subsys=datapath-loader veth=lxcfd73d634716c

Anything else?

No response

Code of Conduct

• I agree to follow this project’s Code of Conduct