kubernetes: kubelet: Observed a panic: "invalid memory address or nil pointer dereference"

What happened:

Kubelet startup error

E0428 09:37:13.276141   26151 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:189
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:214
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:125
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/reflector.go:124
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54
/workspace/anago-v1.14.1-beta.0.44+b7394102d6ef77/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71
/usr/local/go/src/runtime/asm_amd64.s:1337

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

# yum install -y kubelet
# kubelet --version
Kubernetes v1.14.1
# kubelet --cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --fail-swap-on=false

Anything else we need to know?:

kubernetes 1.13.3 does not have this problem.

Environment:

Kubernetes version (use kubectl version):

Kubernetes v1.14.1

OS (e.g: cat /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 33 (18 by maintainers)

Most upvoted comments

@liggitt After git bisecting between v1.13.3 and v1.14.1, I located the root cause to PR https://github.com/kubernetes/kubernetes/pull/75308.

This pr ignores kubeclient nil for csi plugin init, but nil kubeclient can cause nil pointer dereference later.

I’m willing to fix this with guide.

cizixs on May 2, 2019

I have the same problem now!!!

[17:44:28 root@k8s-master ingress]#kubectl logs -n ingress-nginx ingress-nginx-controller-7566fdb978-qjcpt | grep -i error E1022 09:44:15.076751 9 runtime.go:78] Observed a panic: “invalid memory address or nil pointer dereference” (runtime error: invalid memory address or nil pointer dereference) panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference

As9530272755 on Oct 22, 2021

https://github.com/kubernetes/kubernetes/issues/76424 seems like a related issue --> td;lr add --feature-gates=RuntimeClass=false flag as workaround

zamog on Jun 2, 2019