kubernetes: kube-apiserver: runtime error: invalid memory address or nil pointer dereference

What happened: We see the following panics multiple times in our kube-apiserver@v1.17.12 logs:

E0112 10:06:05.718331       1 runtime.go:78] Observed a panic: &errors.errorString{s:"killing connection/stream because serving request timed out and response had been started"} (killing connection/stream because serving request timed out and response had been started)
goroutine 1199280295 [running]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3bc6ee0, 0xc000248ea0)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc01c8ddc90, 0x1, 0x1)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x3bc6ee0, 0xc000248ea0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc04b5b1200, 0xc040642960)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:257 +0x1cf
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc0101c99e0, 0x4fe6f60, 0xc0543dd960, 0xc015b14700)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:141 +0x310
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x4fe6f60, 0xc0543dd960, 0xc015b14400)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47 +0x10f
net/http.HandlerFunc.ServeHTTP(0xc0101e1e60, 0x4fe6f60, 0xc0543dd960, 0xc015b14400)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x4fe6f60, 0xc0543dd960, 0xc015b14300)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x274
net/http.HandlerFunc.ServeHTTP(0xc0101e1e90, 0x4fe6f60, 0xc0543dd960, 0xc015b14300)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x4fe6f60, 0xc0543dd960, 0xc015b14300)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc0101c9a00, 0x4fe6f60, 0xc0543dd960, 0xc015b14300)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:89 +0x2ca
net/http.HandlerFunc.ServeHTTP(0xc0101c9a20, 0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0x13e
net/http.HandlerFunc.ServeHTTP(0xc0101c9a40, 0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc0101e1ec0, 0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc008526e00, 0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/usr/local/go/src/net/http/server.go:2831 +0xa4
net/http.initNPNRequest.ServeHTTP(0x4ff3c20, 0xc01e3f4c90, 0xc01b1eaa80, 0xc008526e00, 0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/usr/local/go/src/net/http/server.go:3395 +0x8d
k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).runHandler(0xc058c07b00, 0xc06400eb88, 0xc024bb0f00, 0xc062dc3440)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2149 +0x9f
created by k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).processHeaders
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:1883 +0x4eb
E0112 10:06:05.718401       1 wrap.go:39] apiserver panic'd on GET /api/v1/nodes
I0112 10:06:05.718596       1 log.go:172] http2: panic serving 10.132.15.228:56520: killing connection/stream because serving request timed out and response had been started
goroutine 1199280295 [running]:
k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).runHandler.func1(0xc06400eb88, 0xc01c8ddf67, 0xc058c07b00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2142 +0x16b
panic(0x3bc6ee0, 0xc000248ea0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc01c8ddc90, 0x1, 0x1)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x3bc6ee0, 0xc000248ea0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc04b5b1200, 0xc040642960)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:257 +0x1cf
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc0101c99e0, 0x4fe6f60, 0xc0543dd960, 0xc015b14700)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:141 +0x310
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithWaitGroup.func1(0x4fe6f60, 0xc0543dd960, 0xc015b14400)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/waitgroup.go:47 +0x10f
net/http.HandlerFunc.ServeHTTP(0xc0101e1e60, 0x4fe6f60, 0xc0543dd960, 0xc015b14400)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1(0x4fe6f60, 0xc0543dd960, 0xc015b14300)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/requestinfo.go:39 +0x274
net/http.HandlerFunc.ServeHTTP(0xc0101e1e90, 0x4fe6f60, 0xc0543dd960, 0xc015b14300)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1(0x4fe6f60, 0xc0543dd960, 0xc015b14300)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/cachecontrol.go:31 +0xa8
net/http.HandlerFunc.ServeHTTP(0xc0101c9a00, 0x4fe6f60, 0xc0543dd960, 0xc015b14300)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.WithLogging.func1(0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog/httplog.go:89 +0x2ca
net/http.HandlerFunc.ServeHTTP(0xc0101c9a20, 0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1(0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/wrap.go:51 +0x13e
net/http.HandlerFunc.ServeHTTP(0xc0101c9a40, 0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0xc0101e1ec0, 0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:189 +0x51
net/http.serverHandler.ServeHTTP(0xc008526e00, 0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/usr/local/go/src/net/http/server.go:2831 +0xa4
net/http.initNPNRequest.ServeHTTP(0x4ff3c20, 0xc01e3f4c90, 0xc01b1eaa80, 0xc008526e00, 0x4fd9f60, 0xc06400eb88, 0xc024bb0f00)
	/usr/local/go/src/net/http/server.go:3395 +0x8d
k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).runHandler(0xc058c07b00, 0xc06400eb88, 0xc024bb0f00, 0xc062dc3440)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:2149 +0x9f
created by k8s.io/kubernetes/vendor/golang.org/x/net/http2.(*serverConn).processHeaders
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/http2/server.go:1883 +0x4eb
E0112 10:06:05.718598       1 writers.go:105] apiserver was unable to write a JSON response: http: Handler timeout
E0112 10:06:05.718628       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}
I0112 10:06:05.720062       1 trace.go:116] Trace[2081929191]: "List" url:/api/v1/nodes,user-agent:calico-typha/v0.0.0 (linux/amd64) kubernetes/$Format,client:10.132.15.228 (started: 2021-01-12 10:05:05.47897996 +0000 UTC m=+1893382.932388142) (total time: 1m0.241038715s):
Trace[2081929191]: [1m0.241038715s] [1m0.235878583s] END
E0112 10:06:05.720256       1 timeout.go:134] runtime error: invalid memory address or nil pointer dereference
goroutine 1199280268 [running]:
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc062df3e00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x107
panic(0x3c920a0, 0x7529e70)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
compress/gzip.(*Writer).Write(0xc04c9d4000, 0xc05a2aa880, 0x72, 0x72, 0x30, 0x3f4d060, 0xc021b9bc01)
	/usr/local/go/src/compress/gzip/gzip.go:168 +0x237
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.(*deferredResponseWriter).Write(0xc05c9015e0, 0xc05a2aa880, 0x72, 0x72, 0xc05a2aa880, 0x72, 0x72)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:182 +0x54e
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.SerializeObject(0x44b0e36, 0x10, 0x7f205daf9fd0, 0xc040642b40, 0x4fe6de0, 0xc05f69e570, 0xc015b14a00, 0xc8, 0x4f8ac60, 0xc0543dda40)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:117 +0x389
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters.WriteObjectNegotiated(0x4fe9ee0, 0xc000a9ae40, 0x4fe6d60, 0xc00e98cea0, 0x0, 0x0, 0x449488d, 0x2, 0x4fe6de0, 0xc05f69e570, ...)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/responsewriters/writers.go:251 +0x555
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.transformResponseObject(0x4ff3c20, 0xc054513260, 0xc00e98cea0, 0xc021b9c7e0, 0xc015b14a00, 0x4fe6de0, 0xc05f69e570, 0xc8, 0x0, 0x0, ...)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/response.go:137 +0x353
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers.ListResource.func1(0x4fe6de0, 0xc05f69e570, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/handlers/get.go:276 +0xdfc
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints.restfulListResource.func1(0xc054513140, 0xc0543dd9d0)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/installer.go:1083 +0x8f
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics.InstrumentRouteFunc.func1(0xc054513140, 0xc0543dd9d0)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/metrics/metrics.go:374 +0x254
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).dispatch(0xc000ae0120, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:288 +0xa4f
k8s.io/kubernetes/vendor/github.com/emicklei/go-restful.(*Container).Dispatch(...)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/emicklei/go-restful/container.go:199
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x44aab69, 0xe, 0xc000ae0120, 0xc000810000, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:146 +0x4d3
k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver.(*proxyHandler).ServeHTTP(0xc00c417c70, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go:116 +0x161
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*pathHandler).ServeHTTP(0xc060522bc0, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:248 +0x38a
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).ServeHTTP(0xc0101aca10, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/mux/pathrecorder.go:234 +0x84
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.director.ServeHTTP(0x44ade2f, 0xf, 0xc0101dcab0, 0xc0101aca10, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/handler.go:154 +0x6b1
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthorization.func1(0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authorization.go:64 +0x512
net/http.HandlerFunc.ServeHTTP(0xc0101a6bc0, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.WithMaxInFlightLimit.func1(0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/maxinflight.go:160 +0x5dc
net/http.HandlerFunc.ServeHTTP(0xc0101e1e30, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithImpersonation.func1(0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/impersonation.go:50 +0x229a
net/http.HandlerFunc.ServeHTTP(0xc0101a6c00, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAudit.func1(0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/audit.go:54 +0x717
net/http.HandlerFunc.ServeHTTP(0xc0101a6c40, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14a00)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters.WithAuthentication.func1(0x7f205d8bd318, 0xc05f69e540, 0xc015b14700)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/endpoints/filters/authentication.go:131 +0xa8c
net/http.HandlerFunc.ServeHTTP(0xc010182fa0, 0x7f205d8bd318, 0xc05f69e540, 0xc015b14700)
	/usr/local/go/src/net/http/server.go:2036 +0x44
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1(0xc062df3e00, 0xc0101c99e0, 0x4ff4660, 0xc05f69e540, 0xc015b14700)
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:113 +0xd0
created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP
	/workspace/anago-v1.17.12-rc.0.60+02c8616ca83844/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:99 +0x1cb

I believe at least the last panic runtime error: invalid memory address or nil pointer dereference should not happen and sounds more like possible nil pointer dereference in the kube-apiserver.

Initially we observed Pods that fail to connect to the kube-apiserver with:

Unable to connect to the server: dial tcp 10.243.0.1:443: connect: no route to host

When we checked the logs of one of the apiserver replicas, we saw the above panics.

What you expected to happen: No nil pointer dereference on kube-apiserver side.

How to reproduce it (as minimally and precisely as possible): No clear steps to reproduce for now.

Environment:

  • Kubernetes version (use kubectl version): v1.17.12
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools: https://github.com/gardener/gardener
  • Network plugin and version (if this is a network-related bug):
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (19 by maintainers)

Most upvoted comments

@sxllwx , thank you once again! Then I am closing this issue.

+1
ialidzhikov on Feb 23, 2021
Read more comments on GitHub
← kubernetes: kube-apiserver log always has TLS handshake error
kubernetes: kube-apiserver[14274]: E1015 10:17:36.804307 14274 watcher.go:214] watch chan error: etcdserver: mvcc: required revision has been compacted →