cilium: Failed to set sysctl needed by BPF bandwidth manager

Bug report

General Information

  • Cilium version (run cilium version) 1.9

  • Kernel version (run uname -a) Linux unknown000c29c41271 5.3.18-24.34-default #1 SMP Thu Oct 29 22:18:30 UTC 2020 (89a1d25) x86_64 x86_64 x86_64 GNU/Linux

  • Orchestration system version in use (e.g. kubectl version, …)

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2+k3s1", GitCommit:"1d4adb0301b9a63ceec8cabb11b309e061f43d5f", GitTreeState:"clean", BuildDate:"2021-01-14T23:52:37Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2+k3s1", GitCommit:"1d4adb0301b9a63ceec8cabb11b309e061f43d5f", GitTreeState:"clean", BuildDate:"2021-01-14T23:52:37Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
curl -sLO https://git.io/cilium-sysdump-latest.zip && python cilium-sysdump-latest.zip

cilium-sysdump-20210524-111215-quick-start.zip

How to reproduce the issue

  1. kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/quick-install.yaml

cilium-agent Logs

level=info msg="Skipped reading configuration file" reason="Config File \"ciliumd\" Not Found in \"[/root]\"" subsys=config
level=info msg="Started gops server" address="127.0.0.1:9890" subsys=daemon
level=info msg="Memory available for map entries (0.003% of 1514762240B): 3786905B" subsys=config
level=info msg="option bpf-ct-global-tcp-max set by dynamic sizing to 131072" subsys=config
level=info msg="option bpf-ct-global-any-max set by dynamic sizing to 65536" subsys=config
level=info msg="option bpf-nat-global-max set by dynamic sizing to 131072" subsys=config
level=info msg="option bpf-neigh-global-max set by dynamic sizing to 131072" subsys=config
level=info msg="option bpf-sock-rev-map-max set by dynamic sizing to 65536" subsys=config
level=info msg="  --agent-health-port='9876'" subsys=daemon
level=info msg="  --agent-labels=''" subsys=daemon
level=info msg="  --allow-icmp-frag-needed='true'" subsys=daemon
level=info msg="  --allow-localhost='auto'" subsys=daemon
level=info msg="  --annotate-k8s-node='true'" subsys=daemon
level=info msg="  --api-rate-limit='map[]'" subsys=daemon
level=info msg="  --arping-refresh-period='5m0s'" subsys=daemon
level=info msg="  --auto-create-cilium-node-resource='true'" subsys=daemon
level=info msg="  --auto-direct-node-routes='false'" subsys=daemon
level=info msg="  --blacklist-conflicting-routes='false'" subsys=daemon
level=info msg="  --bpf-compile-debug='false'" subsys=daemon
level=info msg="  --bpf-ct-global-any-max='262144'" subsys=daemon
level=info msg="  --bpf-ct-global-tcp-max='524288'" subsys=daemon
level=info msg="  --bpf-ct-timeout-regular-any='1m0s'" subsys=daemon
level=info msg="  --bpf-ct-timeout-regular-tcp='6h0m0s'" subsys=daemon
level=info msg="  --bpf-ct-timeout-regular-tcp-fin='10s'" subsys=daemon
level=info msg="  --bpf-ct-timeout-regular-tcp-syn='1m0s'" subsys=daemon
level=info msg="  --bpf-ct-timeout-service-any='1m0s'" subsys=daemon
level=info msg="  --bpf-ct-timeout-service-tcp='6h0m0s'" subsys=daemon
level=info msg="  --bpf-fragments-map-max='8192'" subsys=daemon
level=info msg="  --bpf-lb-acceleration='disabled'" subsys=daemon
level=info msg="  --bpf-lb-algorithm='random'" subsys=daemon
level=info msg="  --bpf-lb-maglev-hash-seed='JLfvgnHc2kaSUFaI'" subsys=daemon
level=info msg="  --bpf-lb-maglev-table-size='16381'" subsys=daemon
level=info msg="  --bpf-lb-map-max='65536'" subsys=daemon
level=info msg="  --bpf-lb-mode='snat'" subsys=daemon
level=info msg="  --bpf-map-dynamic-size-ratio='0.0025'" subsys=daemon
level=info msg="  --bpf-nat-global-max='524288'" subsys=daemon
level=info msg="  --bpf-neigh-global-max='524288'" subsys=daemon
level=info msg="  --bpf-policy-map-max='16384'" subsys=daemon
level=info msg="  --bpf-root=''" subsys=daemon
level=info msg="  --bpf-sock-rev-map-max='262144'" subsys=daemon
level=info msg="  --certificates-directory='/var/run/cilium/certs'" subsys=daemon
level=info msg="  --cgroup-root=''" subsys=daemon
level=info msg="  --cluster-id=''" subsys=daemon
level=info msg="  --cluster-name='default'" subsys=daemon
level=info msg="  --clustermesh-config='/var/lib/cilium/clustermesh/'" subsys=daemon
level=info msg="  --cmdref=''" subsys=daemon
level=info msg="  --config=''" subsys=daemon
level=info msg="  --config-dir='/tmp/cilium/config-map'" subsys=daemon
level=info msg="  --conntrack-gc-interval='0s'" subsys=daemon
level=info msg="  --crd-wait-timeout='5m0s'" subsys=daemon
level=info msg="  --datapath-mode='veth'" subsys=daemon
level=info msg="  --debug='false'" subsys=daemon
level=info msg="  --debug-verbose=''" subsys=daemon
level=info msg="  --device=''" subsys=daemon
level=info msg="  --devices=''" subsys=daemon
level=info msg="  --direct-routing-device=''" subsys=daemon
level=info msg="  --disable-cnp-status-updates='true'" subsys=daemon
level=info msg="  --disable-conntrack='false'" subsys=daemon
level=info msg="  --disable-endpoint-crd='false'" subsys=daemon
level=info msg="  --disable-envoy-version-check='false'" subsys=daemon
level=info msg="  --disable-iptables-feeder-rules=''" subsys=daemon
level=info msg="  --dns-max-ips-per-restored-rule='1000'" subsys=daemon
level=info msg="  --egress-masquerade-interfaces=''" subsys=daemon
level=info msg="  --egress-multi-home-ip-rule-compat='false'" subsys=daemon
level=info msg="  --enable-auto-protect-node-port-range='true'" subsys=daemon
level=info msg="  --enable-bandwidth-manager='false'" subsys=daemon
level=info msg="  --enable-bpf-clock-probe='true'" subsys=daemon
level=info msg="  --enable-bpf-masquerade='true'" subsys=daemon
level=info msg="  --enable-bpf-tproxy='false'" subsys=daemon
level=info msg="  --enable-endpoint-health-checking='true'" subsys=daemon
level=info msg="  --enable-endpoint-routes='false'" subsys=daemon
level=info msg="  --enable-external-ips='true'" subsys=daemon
level=info msg="  --enable-health-check-nodeport='true'" subsys=daemon
level=info msg="  --enable-health-checking='true'" subsys=daemon
level=info msg="  --enable-host-firewall='false'" subsys=daemon
level=info msg="  --enable-host-legacy-routing='false'" subsys=daemon
level=info msg="  --enable-host-port='true'" subsys=daemon
level=info msg="  --enable-host-reachable-services='false'" subsys=daemon
level=info msg="  --enable-hubble='true'" subsys=daemon
level=info msg="  --enable-identity-mark='true'" subsys=daemon
level=info msg="  --enable-ip-masq-agent='false'" subsys=daemon
level=info msg="  --enable-ipsec='false'" subsys=daemon
level=info msg="  --enable-ipv4='true'" subsys=daemon
level=info msg="  --enable-ipv4-fragment-tracking='true'" subsys=daemon
level=info msg="  --enable-ipv6='false'" subsys=daemon
level=info msg="  --enable-ipv6-ndp='false'" subsys=daemon
level=info msg="  --enable-k8s-api-discovery='false'" subsys=daemon
level=info msg="  --enable-k8s-endpoint-slice='true'" subsys=daemon
level=info msg="  --enable-k8s-event-handover='false'" subsys=daemon
level=info msg="  --enable-l7-proxy='true'" subsys=daemon
level=info msg="  --enable-local-node-route='true'" subsys=daemon
level=info msg="  --enable-local-redirect-policy='false'" subsys=daemon
level=info msg="  --enable-monitor='true'" subsys=daemon
level=info msg="  --enable-node-port='false'" subsys=daemon
level=info msg="  --enable-policy='default'" subsys=daemon
level=info msg="  --enable-remote-node-identity='true'" subsys=daemon
level=info msg="  --enable-selective-regeneration='true'" subsys=daemon
level=info msg="  --enable-session-affinity='true'" subsys=daemon
level=info msg="  --enable-svc-source-range-check='true'" subsys=daemon
level=info msg="  --enable-tracing='false'" subsys=daemon
level=info msg="  --enable-well-known-identities='false'" subsys=daemon
level=info msg="  --enable-xt-socket-fallback='true'" subsys=daemon
level=info msg="  --encrypt-interface=''" subsys=daemon
level=info msg="  --encrypt-node='false'" subsys=daemon
level=info msg="  --endpoint-interface-name-prefix='lxc+'" subsys=daemon
level=info msg="  --endpoint-queue-size='25'" subsys=daemon
level=info msg="  --endpoint-status=''" subsys=daemon
level=info msg="  --envoy-log=''" subsys=daemon
level=info msg="  --exclude-local-address=''" subsys=daemon
level=info msg="  --fixed-identity-mapping='map[]'" subsys=daemon
level=info msg="  --flannel-master-device=''" subsys=daemon
level=info msg="  --flannel-uninstall-on-exit='false'" subsys=daemon
level=info msg="  --force-local-policy-eval-at-source='true'" subsys=daemon
level=info msg="  --gops-port='9890'" subsys=daemon
level=info msg="  --host-reachable-services-protos='tcp,udp'" subsys=daemon
level=info msg="  --http-403-msg=''" subsys=daemon
level=info msg="  --http-idle-timeout='0'" subsys=daemon
level=info msg="  --http-max-grpc-timeout='0'" subsys=daemon
level=info msg="  --http-normalize-path='true'" subsys=daemon
level=info msg="  --http-request-timeout='3600'" subsys=daemon
level=info msg="  --http-retry-count='3'" subsys=daemon
level=info msg="  --http-retry-timeout='0'" subsys=daemon
level=info msg="  --hubble-disable-tls='false'" subsys=daemon
level=info msg="  --hubble-event-queue-size='0'" subsys=daemon
level=info msg="  --hubble-flow-buffer-size='4095'" subsys=daemon
level=info msg="  --hubble-listen-address=':4244'" subsys=daemon
level=info msg="  --hubble-metrics=''" subsys=daemon
level=info msg="  --hubble-metrics-server=''" subsys=daemon
level=info msg="  --hubble-socket-path='/var/run/cilium/hubble.sock'" subsys=daemon
level=info msg="  --hubble-tls-cert-file='/var/lib/cilium/tls/hubble/server.crt'" subsys=daemon
level=info msg="  --hubble-tls-client-ca-files='/var/lib/cilium/tls/hubble/client-ca.crt'" subsys=daemon
level=info msg="  --hubble-tls-key-file='/var/lib/cilium/tls/hubble/server.key'" subsys=daemon
level=info msg="  --identity-allocation-mode='crd'" subsys=daemon
level=info msg="  --identity-change-grace-period='5s'" subsys=daemon
level=info msg="  --install-iptables-rules='true'" subsys=daemon
level=info msg="  --ip-allocation-timeout='2m0s'" subsys=daemon
level=info msg="  --ip-masq-agent-config-path='/etc/config/ip-masq-agent'" subsys=daemon
level=info msg="  --ipam='cluster-pool'" subsys=daemon
level=info msg="  --ipsec-key-file=''" subsys=daemon
level=info msg="  --iptables-lock-timeout='5s'" subsys=daemon
level=info msg="  --iptables-random-fully='false'" subsys=daemon
level=info msg="  --ipv4-node='auto'" subsys=daemon
level=info msg="  --ipv4-pod-subnets=''" subsys=daemon
level=info msg="  --ipv4-range='auto'" subsys=daemon
level=info msg="  --ipv4-service-loopback-address='169.254.42.1'" subsys=daemon
level=info msg="  --ipv4-service-range='auto'" subsys=daemon
level=info msg="  --ipv6-cluster-alloc-cidr='f00d::/64'" subsys=daemon
level=info msg="  --ipv6-mcast-device=''" subsys=daemon
level=info msg="  --ipv6-node='auto'" subsys=daemon
level=info msg="  --ipv6-pod-subnets=''" subsys=daemon
level=info msg="  --ipv6-range='auto'" subsys=daemon
level=info msg="  --ipv6-service-range='auto'" subsys=daemon
level=info msg="  --ipvlan-master-device='undefined'" subsys=daemon
level=info msg="  --join-cluster='false'" subsys=daemon
level=info msg="  --k8s-api-server=''" subsys=daemon
level=info msg="  --k8s-force-json-patch='false'" subsys=daemon
level=info msg="  --k8s-heartbeat-timeout='30s'" subsys=daemon
level=info msg="  --k8s-kubeconfig-path=''" subsys=daemon
level=info msg="  --k8s-namespace='kube-system'" subsys=daemon
level=info msg="  --k8s-require-ipv4-pod-cidr='false'" subsys=daemon
level=info msg="  --k8s-require-ipv6-pod-cidr='false'" subsys=daemon
level=info msg="  --k8s-service-cache-size='128'" subsys=daemon
level=info msg="  --k8s-service-proxy-name=''" subsys=daemon
level=info msg="  --k8s-sync-timeout='3m0s'" subsys=daemon
level=info msg="  --k8s-watcher-endpoint-selector='metadata.name!=kube-scheduler,metadata.name!=kube-controller-manager,metadata.name!=etcd-operator,metadata.name!=gcp-controller-manager'" subsys=daemon
level=info msg="  --k8s-watcher-queue-size='1024'" subsys=daemon
level=info msg="  --keep-config='false'" subsys=daemon
level=info msg="  --kube-proxy-replacement='probe'" subsys=daemon
level=info msg="  --kube-proxy-replacement-healthz-bind-address=''" subsys=daemon
level=info msg="  --kvstore=''" subsys=daemon
level=info msg="  --kvstore-connectivity-timeout='2m0s'" subsys=daemon
level=info msg="  --kvstore-lease-ttl='15m0s'" subsys=daemon
level=info msg="  --kvstore-opt='map[]'" subsys=daemon
level=info msg="  --kvstore-periodic-sync='5m0s'" subsys=daemon
level=info msg="  --label-prefix-file=''" subsys=daemon
level=info msg="  --labels=''" subsys=daemon
level=info msg="  --lib-dir='/var/lib/cilium'" subsys=daemon
level=info msg="  --log-driver=''" subsys=daemon
level=info msg="  --log-opt='map[]'" subsys=daemon
level=info msg="  --log-system-load='false'" subsys=daemon
level=info msg="  --masquerade='true'" subsys=daemon
level=info msg="  --max-controller-interval='0'" subsys=daemon
level=info msg="  --metrics=''" subsys=daemon
level=info msg="  --monitor-aggregation='medium'" subsys=daemon
level=info msg="  --monitor-aggregation-flags='all'" subsys=daemon
level=info msg="  --monitor-aggregation-interval='5s'" subsys=daemon
level=info msg="  --monitor-queue-size='0'" subsys=daemon
level=info msg="  --mtu='0'" subsys=daemon
level=info msg="  --nat46-range='0:0:0:0:0:FFFF::/96'" subsys=daemon
level=info msg="  --native-routing-cidr=''" subsys=daemon
level=info msg="  --node-port-acceleration='disabled'" subsys=daemon
level=info msg="  --node-port-algorithm='random'" subsys=daemon
level=info msg="  --node-port-bind-protection='true'" subsys=daemon
level=info msg="  --node-port-mode='snat'" subsys=daemon
level=info msg="  --node-port-range='30000,32767'" subsys=daemon
level=info msg="  --policy-audit-mode='false'" subsys=daemon
level=info msg="  --policy-queue-size='100'" subsys=daemon
level=info msg="  --policy-trigger-interval='1s'" subsys=daemon
level=info msg="  --pprof='false'" subsys=daemon
level=info msg="  --preallocate-bpf-maps='false'" subsys=daemon
level=info msg="  --prefilter-device='undefined'" subsys=daemon
level=info msg="  --prefilter-mode='native'" subsys=daemon
level=info msg="  --prepend-iptables-chains='true'" subsys=daemon
level=info msg="  --prometheus-serve-addr=''" subsys=daemon
level=info msg="  --proxy-connect-timeout='1'" subsys=daemon
level=info msg="  --proxy-prometheus-port='0'" subsys=daemon
level=info msg="  --read-cni-conf=''" subsys=daemon
level=info msg="  --restore='true'" subsys=daemon
level=info msg="  --sidecar-istio-proxy-image='cilium/istio_proxy'" subsys=daemon
level=info msg="  --single-cluster-route='false'" subsys=daemon
level=info msg="  --skip-crd-creation='false'" subsys=daemon
level=info msg="  --socket-path='/var/run/cilium/cilium.sock'" subsys=daemon
level=info msg="  --sockops-enable='false'" subsys=daemon
level=info msg="  --state-dir='/var/run/cilium'" subsys=daemon
level=info msg="  --tofqdns-dns-reject-response-code='refused'" subsys=daemon
level=info msg="  --tofqdns-enable-dns-compression='true'" subsys=daemon
level=info msg="  --tofqdns-endpoint-max-ip-per-hostname='50'" subsys=daemon
level=info msg="  --tofqdns-idle-connection-grace-period='0s'" subsys=daemon
level=info msg="  --tofqdns-max-deferred-connection-deletes='10000'" subsys=daemon
level=info msg="  --tofqdns-min-ttl='0'" subsys=daemon
level=info msg="  --tofqdns-pre-cache=''" subsys=daemon
level=info msg="  --tofqdns-proxy-port='0'" subsys=daemon
level=info msg="  --tofqdns-proxy-response-max-delay='100ms'" subsys=daemon
level=info msg="  --trace-payloadlen='128'" subsys=daemon
level=info msg="  --tunnel='vxlan'" subsys=daemon
level=info msg="  --version='false'" subsys=daemon
level=info msg="  --write-cni-conf-when-ready=''" subsys=daemon
level=info msg="     _ _ _" subsys=daemon
level=info msg=" ___|_| |_|_ _ _____" subsys=daemon
level=info msg="|  _| | | | | |     |" subsys=daemon
level=info msg="|___|_|_|_|___|_|_|_|" subsys=daemon
level=info msg="Cilium 1.9.7 f993696 2021-05-12T18:21:30-07:00 go version go1.15.12 linux/amd64" subsys=daemon
level=info msg="cilium-envoy  version: 82a70d56bf324287ced3129300db609eceb21d10/1.17.3/Distribution/RELEASE/BoringSSL" subsys=daemon
level=info msg="clang (10.0.0) and kernel (5.3.18) versions: OK!" subsys=linux-datapath
level=info msg="linking environment: OK!" subsys=linux-datapath
level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf
level=info msg="Parsing base label prefixes from default label list" subsys=labels-filter
level=info msg="Parsing additional label prefixes from user inputs: []" subsys=labels-filter
level=info msg="Final label prefixes to be used for identity evaluation:" subsys=labels-filter
level=info msg=" - reserved:.*" subsys=labels-filter
level=info msg=" - :io.kubernetes.pod.namespace" subsys=labels-filter
level=info msg=" - :io.cilium.k8s.namespace.labels" subsys=labels-filter
level=info msg=" - :app.kubernetes.io" subsys=labels-filter
level=info msg=" - !:io.kubernetes" subsys=labels-filter
level=info msg=" - !:kubernetes.io" subsys=labels-filter
level=info msg=" - !:.*beta.kubernetes.io" subsys=labels-filter
level=info msg=" - !:k8s.io" subsys=labels-filter
level=info msg=" - !:pod-template-generation" subsys=labels-filter
level=info msg=" - !:pod-template-hash" subsys=labels-filter
level=info msg=" - !:controller-revision-hash" subsys=labels-filter
level=info msg=" - !:annotation.*" subsys=labels-filter
level=info msg=" - !:etcd_node" subsys=labels-filter
level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.156.0.0/16
level=info msg="Initializing daemon" subsys=daemon
level=info msg="Establishing connection to apiserver" host="https://10.43.0.1:443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
level=warning msg="Session affinity for host reachable services needs kernel 5.7.0 or newer to work properly when accessed from inside cluster: the same service endpoint will be selected from all network namespaces on the host." subsys=daemon
level=info msg="BPF host routing is only available in native routing mode. Falling back to legacy host routing (enable-host-legacy-routing=true)." subsys=daemon
level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=192.168.1.156 mtu=1500 subsys=mtu
level=info msg="Restored services from maps" failed=0 restored=6 subsys=service
level=info msg="Reading old endpoints..." subsys=daemon
level=info msg="No old endpoints found." subsys=daemon
level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
level=error msg="Command execution failed" cmd="[iptables -t mangle -n -L CILIUM_PRE_mangle]" error="exit status 1" subsys=iptables
level=warning msg="iptables: No chain/target/match by that name." subsys=iptables
level=info msg="Waiting until all Cilium CRDs are available" subsys=k8s
level=info msg="All Cilium CRDs have been found and are available" subsys=k8s
level=info msg="Creating or updating CiliumNode resource" node=unknown000c29c41271 subsys=nodediscovery
level=info msg="Retrieved node information from cilium node" nodeName=unknown000c29c41271 subsys=k8s
level=info msg="Received own node information from API server" ipAddr.ipv4=192.168.1.156 ipAddr.ipv6="<nil>" k8sNodeIP=192.168.1.156 labels="map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:k3s beta.kubernetes.io/os:linux k3s.io/hostname:unknown000c29c41271 k3s.io/internal-ip:192.168.1.156 kubernetes.io/arch:amd64 kubernetes.io/hostname:unknown000c29c41271 kubernetes.io/os:linux node-role.kubernetes.io/control-plane:true node-role.kubernetes.io/etcd:true node-role.kubernetes.io/master:true node.kubernetes.io/instance-type:k3s]" nodeName=unknown000c29c41271 subsys=k8s v4Prefix=10.0.0.0/24 v6Prefix="<nil>"
level=info msg="k8s mode: Allowing localhost to reach local endpoints" subsys=daemon
level=info msg="Using auto-derived devices for BPF node port" devices="[eth0]" directRoutingDevice=eth0 subsys=daemon
level=info msg="Enabling k8s event listener" subsys=k8s-watcher
level=info msg="Removing stale endpoint interfaces" subsys=daemon
level=info msg="Skipping kvstore configuration" subsys=daemon
level=info msg="Restored router address from node_config" file=/var/run/cilium/state/globals/node_config.h ipv4=10.0.0.115 ipv6="<nil>" subsys=node
level=info msg="Initializing node addressing" subsys=daemon
level=info msg="Initializing cluster-pool IPAM" subsys=ipam v4Prefix=10.0.0.0/24 v6Prefix="<nil>"
level=info msg="Restoring endpoints..." subsys=daemon
level=info msg="Waiting until all pre-existing resources related to policy have been received" subsys=k8s-watcher
level=info msg="Endpoints restored" failed=0 restored=0 subsys=daemon
level=info msg="Addressing information:" subsys=daemon
level=info msg="  Cluster-Name: default" subsys=daemon
level=info msg="  Cluster-ID: 0" subsys=daemon
level=info msg="  Local node-name: unknown000c29c41271" subsys=daemon
level=info msg="  Node-IPv6: <nil>" subsys=daemon
level=info msg="  External-Node IPv4: 192.168.1.156" subsys=daemon
level=info msg="  Internal-Node IPv4: 10.0.0.115" subsys=daemon
level=info msg="  IPv4 allocation prefix: 10.0.0.0/24" subsys=daemon
level=info msg="  Loopback IPv4: 169.254.42.1" subsys=daemon
level=info msg="  Local IPv4 addresses:" subsys=daemon
level=info msg="  - 192.168.1.156" subsys=daemon
level=info msg="  - 10.0.0.115" subsys=daemon
level=info msg="Creating or updating CiliumNode resource" node=unknown000c29c41271 subsys=nodediscovery
level=info msg="Adding local node to cluster" node="{unknown000c29c41271 default [{InternalIP 192.168.1.156} {CiliumInternalIP 10.0.0.115}] 10.0.0.0/24 <nil> 10.0.0.40 <nil> 0 local 0 map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:k3s beta.kubernetes.io/os:linux k3s.io/hostname:unknown000c29c41271 k3s.io/internal-ip:192.168.1.156 kubernetes.io/arch:amd64 kubernetes.io/hostname:unknown000c29c41271 kubernetes.io/os:linux node-role.kubernetes.io/control-plane:true node-role.kubernetes.io/etcd:true node-role.kubernetes.io/master:true node.kubernetes.io/instance-type:k3s] 6}" subsys=nodediscovery
level=info msg="Annotating k8s node" subsys=daemon v4CiliumHostIP.IPv4=10.0.0.115 v4Prefix=10.0.0.0/24 v4healthIP.IPv4=10.0.0.40 v6CiliumHostIP.IPv6="<nil>" v6Prefix="<nil>" v6healthIP.IPv6="<nil>"
level=info msg="Initializing identity allocator" subsys=identity-cache
level=info msg="Cluster-ID is not specified, skipping ClusterMesh initialization" subsys=daemon
level=info msg="Setting up BPF datapath" bpfClockSource=ktime bpfInsnSet=v2 subsys=datapath-loader
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.core.bpf_jit_enable sysParamValue=1
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.ipv4.conf.all.rp_filter sysParamValue=0
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.unprivileged_bpf_disabled sysParamValue=1
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.timer_migration sysParamValue=0
level=info msg="All pre-existing resources related to policy have been received; continuing" subsys=k8s-watcher
level=error msg="Command execution failed" cmd="[/var/lib/cilium/bpf/init.sh /var/lib/cilium/bpf /var/run/cilium/state 10.0.0.115 <nil> vxlan eth0 <nil> <nil> 1500 true true false /var/run/cilium/cgroupv2 /sys/fs/bpf true false v2 eth0=0x9c01a8c0 <nil> 128 false]" error="exit status 2" subsys=datapath-loader
level=warning msg="+ set -o pipefail" subsys=datapath-loader
level=warning msg="++ command -v cilium-map-migrate" subsys=datapath-loader
level=warning msg="+ [[ ! -n /usr/bin/cilium-map-migrate ]]" subsys=datapath-loader
level=warning msg="+ rm /var/run/cilium/state/encap.state" subsys=datapath-loader
level=warning msg="+ true" subsys=datapath-loader
level=warning msg="+ DIR=/run/cilium/state/globals" subsys=datapath-loader
level=warning msg="+ case \"${MODE}\" in" subsys=datapath-loader
level=warning msg="+ HOST_DEV1=cilium_host" subsys=datapath-loader
level=warning msg="+ HOST_DEV2=cilium_net" subsys=datapath-loader
level=warning msg="+ setup_veth_pair cilium_host cilium_net" subsys=datapath-loader
level=warning msg="+ local -r NAME1=cilium_host" subsys=datapath-loader
level=warning msg="+ local -r NAME2=cilium_net" subsys=datapath-loader
level=warning msg="++ cut -d ' ' -f 2" subsys=datapath-loader
level=warning msg="++ ip link show cilium_host type veth" subsys=datapath-loader
level=warning msg="+ '[' cilium_host@cilium_net: '!=' cilium_host@cilium_net: ']'" subsys=datapath-loader
level=warning msg="+ setup_dev cilium_host" subsys=datapath-loader
level=warning msg="+ local -r NAME=cilium_host" subsys=datapath-loader
level=warning msg="+ ip link set cilium_host up" subsys=datapath-loader
level=warning msg="+ '[' '<nil>' '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ '[' 10.0.0.115 '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ echo 1" subsys=datapath-loader
level=warning msg="+ echo 0" subsys=datapath-loader
level=warning msg="+ echo 1" subsys=datapath-loader
level=warning msg="+ echo 0" subsys=datapath-loader
level=warning msg="+ setup_dev cilium_net" subsys=datapath-loader
level=warning msg="+ local -r NAME=cilium_net" subsys=datapath-loader
level=warning msg="+ ip link set cilium_net up" subsys=datapath-loader
level=warning msg="+ '[' '<nil>' '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ '[' 10.0.0.115 '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ echo 1" subsys=datapath-loader
level=warning msg="+ echo 0" subsys=datapath-loader
level=warning msg="+ echo 1" subsys=datapath-loader
level=warning msg="+ echo 0" subsys=datapath-loader
level=warning msg="+ ip link set cilium_host arp off" subsys=datapath-loader
level=warning msg="+ ip link set cilium_net arp off" subsys=datapath-loader
level=warning msg="+ ip link set cilium_host mtu 1500" subsys=datapath-loader
level=warning msg="+ ip link set cilium_net mtu 1500" subsys=datapath-loader
level=warning msg="+ case \"${MODE}\" in" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*CILIUM_NET_MAC.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="++ grep ether" subsys=datapath-loader
level=warning msg="++ awk '{print $2}'" subsys=datapath-loader
level=warning msg="++ ip link show cilium_net" subsys=datapath-loader
level=warning msg="+ CILIUM_NET_MAC=d6:68:ae:82:4e:5b" subsys=datapath-loader
level=warning msg="++ mac2array d6:68:ae:82:4e:5b" subsys=datapath-loader
level=warning msg="++ echo '{0xd6,0x68,0xae,0x82,0x4e,0x5b}'" subsys=datapath-loader
level=warning msg="+ CILIUM_NET_MAC='{0xd6,0x68,0xae,0x82,0x4e,0x5b}'" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*CILIUM_NET_MAC.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="+ echo '#ifndef CILIUM_NET_MAC'" subsys=datapath-loader
level=warning msg="+ echo '#define CILIUM_NET_MAC { .addr = {0xd6,0x68,0xae,0x82,0x4e,0x5b}}'" subsys=datapath-loader
level=warning msg="+ echo '#endif /* CILIUM_NET_MAC */'" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*HOST_IFINDEX.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="++ cat /sys/class/net/cilium_net/ifindex" subsys=datapath-loader
level=warning msg="+ HOST_IDX=72" subsys=datapath-loader
level=warning msg="+ echo '#define HOST_IFINDEX 72'" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*HOST_IFINDEX_MAC.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="++ grep ether" subsys=datapath-loader
level=warning msg="++ awk '{print $2}'" subsys=datapath-loader
level=warning msg="++ ip link show cilium_host" subsys=datapath-loader
level=warning msg="+ HOST_MAC=3e:9d:06:54:3a:d0" subsys=datapath-loader
level=warning msg="++ mac2array 3e:9d:06:54:3a:d0" subsys=datapath-loader
level=warning msg="++ echo '{0x3e,0x9d,0x06,0x54,0x3a,0xd0}'" subsys=datapath-loader
level=warning msg="+ HOST_MAC='{0x3e,0x9d,0x06,0x54,0x3a,0xd0}'" subsys=datapath-loader
level=warning msg="+ echo '#define HOST_IFINDEX_MAC { .addr = {0x3e,0x9d,0x06,0x54,0x3a,0xd0}}'" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*CILIUM_IFINDEX.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="++ cat /sys/class/net/cilium_host/ifindex" subsys=datapath-loader
level=warning msg="+ CILIUM_IDX=73" subsys=datapath-loader
level=warning msg="+ echo '#define CILIUM_IFINDEX 73'" subsys=datapath-loader
level=warning msg="++ awk '{print $1}'" subsys=datapath-loader
level=warning msg="++ cat /proc/sys/net/ipv4/ip_local_port_range" subsys=datapath-loader
level=warning msg="+ CILIUM_EPHEMERAL_MIN=32768" subsys=datapath-loader
level=warning msg="+ echo '#define EPHEMERAL_MIN 32768'" subsys=datapath-loader
level=warning msg="+ '[' true = true ']'" subsys=datapath-loader
level=warning msg="+ MAC_BY_IFINDEX_MACRO='#define NATIVE_DEV_MAC_BY_IFINDEX(IFINDEX) ({ \\" subsys=datapath-loader
level=warning msg="\tunion macaddr __mac = {.addr = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}; \\" subsys=datapath-loader
level=warning msg="\tswitch (IFINDEX) { \\\\\\n'" subsys=datapath-loader
level=warning msg="+ MAC_BY_IFINDEX_MACRO_END='\t} \\" subsys=datapath-loader
level=warning msg="\t__mac; })'" subsys=datapath-loader
level=warning msg="+ for NATIVE_DEV in ${NATIVE_DEVS//;/ }" subsys=datapath-loader
level=warning msg="++ cat /sys/class/net/eth0/ifindex" subsys=datapath-loader
level=warning msg="+ IDX=2" subsys=datapath-loader
level=warning msg="++ grep ether" subsys=datapath-loader
level=warning msg="++ awk '{print $2}'" subsys=datapath-loader
level=warning msg="++ ip link show eth0" subsys=datapath-loader
level=warning msg="+ MAC=00:0c:29:c4:12:71" subsys=datapath-loader
level=warning msg="++ mac2array 00:0c:29:c4:12:71" subsys=datapath-loader
level=warning msg="++ echo '{0x00,0x0c,0x29,0xc4,0x12,0x71}'" subsys=datapath-loader
level=warning msg="+ MAC='{0x00,0x0c,0x29,0xc4,0x12,0x71}'" subsys=datapath-loader
level=warning msg="+ MAC_BY_IFINDEX_MACRO='#define NATIVE_DEV_MAC_BY_IFINDEX(IFINDEX) ({ \\" subsys=datapath-loader
level=warning msg="\tunion macaddr __mac = {.addr = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}; \\" subsys=datapath-loader
level=warning msg="\tswitch (IFINDEX) { \\\\\\n\tcase 2: {union macaddr __tmp = {.addr = {0x00,0x0c,0x29,0xc4,0x12,0x71}}; __mac=__tmp;} break; \\\\\\n'" subsys=datapath-loader
level=warning msg="+ MAC_BY_IFINDEX_MACRO='#define NATIVE_DEV_MAC_BY_IFINDEX(IFINDEX) ({ \\" subsys=datapath-loader
level=warning msg="\tunion macaddr __mac = {.addr = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}; \\" subsys=datapath-loader
level=warning msg="\tswitch (IFINDEX) { \\\\\\n\tcase 2: {union macaddr __tmp = {.addr = {0x00,0x0c,0x29,0xc4,0x12,0x71}}; __mac=__tmp;} break; \\\\\\n\t} \\" subsys=datapath-loader
level=warning msg="\t__mac; })'" subsys=datapath-loader
level=warning msg="+ echo -e '#define NATIVE_DEV_MAC_BY_IFINDEX(IFINDEX) ({ \\" subsys=datapath-loader
level=warning msg="\tunion macaddr __mac = {.addr = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}; \\" subsys=datapath-loader
level=warning msg="\tswitch (IFINDEX) { \\\\\\n\tcase 2: {union macaddr __tmp = {.addr = {0x00,0x0c,0x29,0xc4,0x12,0x71}}; __mac=__tmp;} break; \\\\\\n\t} \\" subsys=datapath-loader
level=warning msg="\t__mac; })'" subsys=datapath-loader
level=warning msg="+ case \"${MODE}\" in" subsys=datapath-loader
level=warning msg="+ '[' '<nil>' '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ '[' 10.0.0.115 '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="++ ip -4 addr show to 10.0.0.115 dev cilium_host" subsys=datapath-loader
level=warning msg="+ '[' -n '73: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000" subsys=datapath-loader
level=warning msg="    inet 10.0.0.115/32 scope link cilium_host" subsys=datapath-loader
level=warning msg="       valid_lft forever preferred_lft forever' ']'" subsys=datapath-loader
level=warning msg="+ move_local_rules" subsys=datapath-loader
level=warning msg="+ '[' 10.0.0.115 '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ move_local_rules_af 'ip -4'" subsys=datapath-loader
level=warning msg="+ IP='ip -4'" subsys=datapath-loader
level=warning msg="++ ip -4 rule list" subsys=datapath-loader
level=warning msg="+ '[' -z '9:\tfrom all fwmark 0x200/0xf00 lookup 2004 " subsys=datapath-loader
level=warning msg="10:\tfrom all fwmark 0xa00/0xf00 lookup 2005 " subsys=datapath-loader
level=warning msg="100:\tfrom all lookup local " subsys=datapath-loader
level=warning msg="32766:\tfrom all lookup main " subsys=datapath-loader
level=warning msg="32767:\tfrom all lookup default ' ']'" subsys=datapath-loader
level=warning msg="+ grep 100" subsys=datapath-loader
level=warning msg="+ grep 'lookup local'" subsys=datapath-loader
level=warning msg="+ ip -4 rule list" subsys=datapath-loader
level=warning msg="100:\tfrom all lookup local " subsys=datapath-loader
level=warning msg="+ ip -4 rule del from all lookup local pref 0" subsys=datapath-loader
level=warning msg="+ true" subsys=datapath-loader
level=warning msg="++ grep 'lookup local'" subsys=datapath-loader
level=warning msg="++ wc -l" subsys=datapath-loader
level=warning msg="++ ip -4 rule list" subsys=datapath-loader
level=warning msg="+ '[' 1 -eq 0 ']'" subsys=datapath-loader
level=warning msg="+ '[' '<nil>' '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ setup_proxy_rules" subsys=datapath-loader
level=warning msg="+ '[' vxlan = ipvlan ']'" subsys=datapath-loader
level=warning msg="+ from_ingress_rulespec='fwmark 0xA00/0xF00 pref 10 lookup 2005'" subsys=datapath-loader
level=warning msg="+ to_proxy_rulespec='fwmark 0x200/0xF00 pref 9 lookup 2004'" subsys=datapath-loader
level=warning msg="+ '[' 10.0.0.115 '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="++ ip -4 rule list" subsys=datapath-loader
level=warning msg="+ '[' -n '9:\tfrom all fwmark 0x200/0xf00 lookup 2004 " subsys=datapath-loader
level=warning msg="10:\tfrom all fwmark 0xa00/0xf00 lookup 2005 " subsys=datapath-loader
level=warning msg="100:\tfrom all lookup local " subsys=datapath-loader
level=warning msg="32766:\tfrom all lookup main " subsys=datapath-loader
level=warning msg="32767:\tfrom all lookup default ' ']'" subsys=datapath-loader
level=warning msg="++ ip -4 rule list fwmark 0x200/0xF00 pref 9 lookup 2004" subsys=datapath-loader
level=warning msg="+ '[' -z '9:\tfrom all fwmark 0x200/0xf00 lookup 2004 ' ']'" subsys=datapath-loader
level=warning msg="+ '[' false = true ']'" subsys=datapath-loader
level=warning msg="++ ip -4 rule list fwmark 0xA00/0xF00 pref 10 lookup 2005" subsys=datapath-loader
level=warning msg="+ '[' -z '10:\tfrom all fwmark 0xa00/0xf00 lookup 2005 ' ']'" subsys=datapath-loader
level=warning msg="+ ip route replace table 2004 local 0.0.0.0/0 dev lo" subsys=datapath-loader
level=warning msg="+ '[' false = true ']'" subsys=datapath-loader
level=warning msg="+ ip route replace table 2005 10.0.0.115/32 dev cilium_host" subsys=datapath-loader
level=warning msg="+ ip route replace table 2005 default via 10.0.0.115" subsys=datapath-loader
level=warning msg="+ case \"${MODE}\" in" subsys=datapath-loader
level=warning msg="+ '[' '<nil>' '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ ip -6 rule del fwmark 0x200/0xF00 pref 9 lookup 2004" subsys=datapath-loader
level=warning msg="+ true" subsys=datapath-loader
level=warning msg="+ ip -6 rule del fwmark 0xA00/0xF00 pref 10 lookup 2005" subsys=datapath-loader
level=warning msg="+ true" subsys=datapath-loader
level=warning msg="+ sed -i /ENCAP_GENEVE/d /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="+ sed -i /ENCAP_VXLAN/d /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="+ '[' vxlan = vxlan ']'" subsys=datapath-loader
level=warning msg="+ echo '#define ENCAP_VXLAN 1'" subsys=datapath-loader
level=warning msg="+ '[' vxlan = vxlan -o vxlan = geneve ']'" subsys=datapath-loader
level=warning msg="+ ENCAP_DEV=cilium_vxlan" subsys=datapath-loader
level=warning msg="+ ip link show cilium_vxlan" subsys=datapath-loader
level=warning msg="74: cilium_vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000" subsys=datapath-loader
level=warning msg="    link/ether 3a:18:41:e0:20:e3 brd ff:ff:ff:ff:ff:ff" subsys=datapath-loader
level=warning msg="+ ip link set cilium_vxlan mtu 1500" subsys=datapath-loader
level=warning msg="+ setup_dev cilium_vxlan" subsys=datapath-loader
level=warning msg="+ local -r NAME=cilium_vxlan" subsys=datapath-loader
level=warning msg="+ ip link set cilium_vxlan up" subsys=datapath-loader
level=warning msg="+ '[' '<nil>' '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ '[' 10.0.0.115 '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ echo 1" subsys=datapath-loader
level=warning msg="+ echo 0" subsys=datapath-loader
level=warning msg="+ echo 1" subsys=datapath-loader
level=warning msg="+ echo 0" subsys=datapath-loader
level=warning msg="++ cat /sys/class/net/cilium_vxlan/ifindex" subsys=datapath-loader
level=warning msg="+ ENCAP_IDX=74" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*ENCAP_IFINDEX.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="+ echo '#define ENCAP_IFINDEX 74'" subsys=datapath-loader
level=warning msg="+ CALLS_MAP=cilium_calls_overlay_2" subsys=datapath-loader
level=warning msg="+ COPTS='-DSECLABEL=2 -DFROM_ENCAP_DEV=1'" subsys=datapath-loader
level=warning msg="+ '[' true = true ']'" subsys=datapath-loader
level=warning msg="+ COPTS='-DSECLABEL=2 -DFROM_ENCAP_DEV=1 -DDISABLE_LOOPBACK_LB'" subsys=datapath-loader
level=warning msg="+ bpf_load cilium_vxlan '-DSECLABEL=2 -DFROM_ENCAP_DEV=1 -DDISABLE_LOOPBACK_LB' ingress bpf_overlay.c bpf_overlay.o from-overlay cilium_calls_overlay_2" subsys=datapath-loader
level=warning msg="+ DEV=cilium_vxlan" subsys=datapath-loader
level=warning msg="+ OPTS='-DSECLABEL=2 -DFROM_ENCAP_DEV=1 -DDISABLE_LOOPBACK_LB'" subsys=datapath-loader
level=warning msg="+ WHERE=ingress" subsys=datapath-loader
level=warning msg="+ IN=bpf_overlay.c" subsys=datapath-loader
level=warning msg="+ OUT=bpf_overlay.o" subsys=datapath-loader
level=warning msg="+ SEC=from-overlay" subsys=datapath-loader
level=warning msg="+ CALLS_MAP=cilium_calls_overlay_2" subsys=datapath-loader
level=warning msg="++ grep ether" subsys=datapath-loader
level=warning msg="++ awk '{print $2}'" subsys=datapath-loader
level=warning msg="++ ip link show cilium_vxlan" subsys=datapath-loader
level=warning msg="+ NODE_MAC=3a:18:41:e0:20:e3" subsys=datapath-loader
level=warning msg="++ mac2array 3a:18:41:e0:20:e3" subsys=datapath-loader
level=warning msg="++ echo '{0x3a,0x18,0x41,0xe0,0x20,0xe3}'" subsys=datapath-loader
level=warning msg="+ NODE_MAC='{.addr={0x3a,0x18,0x41,0xe0,0x20,0xe3}}'" subsys=datapath-loader
level=warning msg="+ OPTS='-DSECLABEL=2 -DFROM_ENCAP_DEV=1 -DDISABLE_LOOPBACK_LB -DNODE_MAC={.addr={0x3a,0x18,0x41,0xe0,0x20,0xe3}} -DCALLS_MAP=cilium_calls_overlay_2'" subsys=datapath-loader
level=warning msg="+ bpf_compile bpf_overlay.c bpf_overlay.o obj '-DSECLABEL=2 -DFROM_ENCAP_DEV=1 -DDISABLE_LOOPBACK_LB -DNODE_MAC={.addr={0x3a,0x18,0x41,0xe0,0x20,0xe3}} -DCALLS_MAP=cilium_calls_overlay_2'" subsys=datapath-loader
level=warning msg="+ IN=bpf_overlay.c" subsys=datapath-loader
level=warning msg="+ OUT=bpf_overlay.o" subsys=datapath-loader
level=warning msg="+ TYPE=obj" subsys=datapath-loader
level=warning msg="+ EXTRA_OPTS='-DSECLABEL=2 -DFROM_ENCAP_DEV=1 -DDISABLE_LOOPBACK_LB -DNODE_MAC={.addr={0x3a,0x18,0x41,0xe0,0x20,0xe3}} -DCALLS_MAP=cilium_calls_overlay_2'" subsys=datapath-loader
level=warning msg="+ clang -O2 -target bpf -std=gnu89 -nostdinc -emit-llvm -Wall -Wextra -Werror -Wshadow -Wno-address-of-packed-member -Wno-unknown-warning-option -Wno-gnu-variable-sized-type-not-at-end -Wdeclaration-after-statement -I. -I/run/cilium/state/globals -I/var/lib/cilium/bpf -I/var/lib/cilium/bpf/include -D__NR_CPUS__=128 -DENABLE_ARP_RESPONDER=1 -DSECLABEL=2 -DFROM_ENCAP_DEV=1 -DDISABLE_LOOPBACK_LB '-DNODE_MAC={.addr={0x3a,0x18,0x41,0xe0,0x20,0xe3}}' -DCALLS_MAP=cilium_calls_overlay_2 -c /var/lib/cilium/bpf/bpf_overlay.c -o -" subsys=datapath-loader
level=warning msg="+ llc -march=bpf -mcpu=v2 -mattr=dwarfris -filetype=obj -o bpf_overlay.o" subsys=datapath-loader
level=warning msg="+ tc qdisc replace dev cilium_vxlan clsact" subsys=datapath-loader
level=warning msg="Error: Specified qdisc not found." subsys=datapath-loader
level=warning msg="+ true" subsys=datapath-loader
level=warning msg="++ grep -v 'pref 1 bpf chain 0 $\\|pref 1 bpf chain 0 handle 0x1'" subsys=datapath-loader
level=warning msg="++ tc filter show dev cilium_vxlan ingress" subsys=datapath-loader
level=warning msg="+ '[' -z '' ']'" subsys=datapath-loader
level=warning msg="+ cilium-map-migrate -s bpf_overlay.o" subsys=datapath-loader
level=warning msg="+ set +e" subsys=datapath-loader
level=warning msg="+ tc filter replace dev cilium_vxlan ingress prio 1 handle 1 bpf da obj bpf_overlay.o sec from-overlay" subsys=datapath-loader
level=warning msg="Error: Parent Qdisc doesn't exists." subsys=datapath-loader
level=warning msg="We have an error talking to the kernel, -1" subsys=datapath-loader
level=warning msg="+ RETCODE=2" subsys=datapath-loader
level=warning msg="+ set -e" subsys=datapath-loader
level=warning msg="+ cilium-map-migrate -e bpf_overlay.o -r 2" subsys=datapath-loader
level=warning msg="+ return 2" subsys=datapath-loader
level=error msg="Error while initializing daemon" error="exit status 2" subsys=daemon
level=fatal msg="Error while creating daemon" error="exit status 2" subsys=daemon

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 26 (14 by maintainers)

Most upvoted comments

Hi @jamesongithub

The JeOS kernel is indeed mising fq and fq_codel modules. We didn’t notice that, because both openSUSE Kubic, and SUSE CaaSP are using the default kernel flavors (kernel-default package), not the minimal one which comes with JeOS (kernel-default-base package).

I will try to get those modules also in JeOS flavor, but it might take time.

For now, as a workaround, I would suggest installing the default kernel on all nodes with:

zypper in kernel-default -kernel-default-base

Sorry for the inconvenience and late reaction

yeah so the sysdump tool attempts to run this:

kubectl exec -n kube-system cilium-mh9kn -c cilium-agent cilium-bugtool kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] – [COMMAND] instead. error: unable to upgrade connection: container not found (“cilium-agent”)

but the cilium-agent is in crashloopbackup so it’s gone when sysdump runs. the logs posted earlier was from the cilium-agent container

trying to run the container directly doesn’t work – requires some env vars i guess crictl run ccda7ef2b0849 cilium-bugtool FATA[2021-05-26T13:29:37.128835757-07:00] running container: load podSandboxConfig: config at cilium-bugtool not found

logs from sysdump

DEBUG Fetching nodes to determine cluster size...
INFO collecting cilium gops stats ...
INFO collecting kubernetes version info ...
INFO collecting Kubernetes events JSON ...
INFO collecting Kubernetes namespaces yaml ...
INFO collecting nodes overview ...
INFO collecting pods overview ...
INFO collected kubernetes version info: k8s-version-info-20210526-130004.txt
INFO collecting pods summary ...
INFO collected kubernetes namespaces: k8s-namespaces-20210526-130004.yaml
INFO collecting services overview ...
INFO collected nodes overview: nodes-20210526-130004.json
INFO collected pods overview: pods-20210526-130004.json
INFO collected kubernetes events: k8s-events-20210526-130004.json
INFO collected pods summary: pods-20210526-130005.txt
INFO collected svc overview: services-20210526-130005.yaml
WARNING Warning: Command 'kubectl exec -n kube-system cilium-mh9kn -c cilium-agent -- /bin/gops stats 1 > ./cilium-sysdump-20210526-130004/cilium-mh9kn-cilium-agent-20210526-130007-stats.txt' returned non-zero exit status 1. Could not collect gops stats: cilium-mh9kn-cilium-agent-20210526-130007-stats.txt
WARNING Warning: Command 'kubectl exec -n kube-system cilium-mh9kn -c cilium-agent -- /bin/gops memstats 1 > ./cilium-sysdump-20210526-130004/cilium-mh9kn-cilium-agent-20210526-130008-memstats.txt' returned non-zero exit status 1. Could not collect gops memstats: cilium-mh9kn-cilium-agent-20210526-130008-memstats.txt
INFO collecting hubble gops stats ...
WARNING Warning: Command 'kubectl exec -n kube-system cilium-mh9kn -c cilium-agent -- /bin/gops stack 1 > ./cilium-sysdump-20210526-130004/cilium-mh9kn-cilium-agent-20210526-130009-stack.txt' returned non-zero exit status 1. Could not collect gops stack: cilium-mh9kn-cilium-agent-20210526-130009-stack.txt
INFO collecting hubble relay gops stats ...
INFO collecting cilium-bugtool output ...
INFO collecting kubernetes network policy ...
INFO collecting cilium network policy ...
INFO collecting cilium clusterwide network policy ...
INFO collecting cilium etcd secret ...
INFO collecting cilium endpoints ...
INFO Command 'kubectl get secret cilium-etcd-secrets -n kube-system -o json' returned non-zero exit status 1. Unable to get and redact cilium secret (this is expected in CRD mode).
INFO collecting cilium identities ...
INFO collected kubernetes network policy: netpol-20210526-130012.yaml
INFO collecting cilium nodes ...
INFO collected cilium endpoints: cep-20210526-130012.yaml
INFO collecting cilium daemonset yaml ...
INFO collected cilium clusterwide network policy: ccnp-20210526-130012.yaml
INFO collecting hubble daemonset yaml ...
INFO collected cilium network policy: cnp-20210526-130012.yaml
INFO collecting hubble-relay deployment yaml ...
INFO collected cilium identities: ciliumidentities-20210526-130014.yaml
INFO collecting hubble-ui deployment yaml ...
WARNING Command 'kubectl get deployment hubble-relay -n kube-system -oyaml > ./cilium-sysdump-20210526-130004/hubble-relay-deployment-20210526-130014.yaml' returned non-zero exit status 1. Unable to get hubble-relay deployment yaml
INFO collecting cilium-operator deployment yaml ...
INFO collecting cilium logs ...
INFO collected cilium nodes: ciliumnodes-20210526-130014.yaml
INFO collecting cilium configmap yaml ...
INFO collected cilium daemonset yaml file: cilium-ds-20210526-130014.yaml
WARNING Command 'kubectl get deployment hubble-ui -n kube-system -oyaml > ./cilium-sysdump-20210526-130004/hubble-ui-deployment-20210526-130015.yaml' returned non-zero exit status 1. Unable to get hubble-ui deployment yaml
ERROR Error: Command '['kubectl', 'exec', '-n', 'kube-system', 'cilium-mh9kn', '-c', 'cilium-agent', 'cilium-bugtool']' returned non-zero exit status 1. Could not run cilium-bugtool on cilium-mh9kn
INFO collected cilium-operator deployment yaml file: cilium-operator-deployment-20210526-130016.yaml
INFO collected cilium configmap yaml file: cilium-configmap-20210526-130016.yaml
INFO collecting cilium operator logs ...
INFO collecting hubble logs ...
INFO collected log file: cilium-mh9kn-cilium-agent-20210526-130018
INFO collected log file: cilium-operator-bcf985bf9-mgjxs-cilium-operator-20210526-130019
INFO collected log file: cilium-mh9kn-cilium-agent-20210526-130018-previous
INFO collecting hubble relay logs ...
DEBUG no previous pod logs to gather for pod/container cilium-operator-bcf985bf9-mgjxs/cilium-operator
INFO collecting hubble ui logs ...
INFO deleting directory: ./cilium-sysdump-20210526-130004
INFO the sysdump has been saved in the file /root/cilium-sysdump-20210526-130004.zip.