istio: istio 1.5 Pod SecurityContext break volumeMount

Bug description my pod has a volume mount which is a NFS(pv+pvc) and their are about 100 thousand files in this volume. when using istio 1.5, I find it’s very slow to init the pod :

Warning  FailedMount  39s   kubelet, cn-shanghai.10.98.5.213  Unable to mount volumes for pod "ubuntu-deploy-1-677f474ff9-bq5gd_backend(e3a8ffc4-61d3-11ea-a85d-aa30c59d7048)": timeout expired waiting for volumes to attach or mount for pod "backend"/"ubuntu-deploy-1-677f474ff9-bq5gd". list of unmounted volumes=[mypd]. list of unattached volumes=[mypd default-token-flkq5]

it takes about 30min to finish the mount.

with some debug in it, I found the reason is: istio 1.5 inject a pod SecurityContext: fsGroup: 1337, and kubelet will use this fsGroup to configure my volume, which means do chomd and chown for any files in the volume, in my case it’s 100 thousand files, so it’s very very slow.

I find https://github.com/istio/istio/pull/20428 add the pod SecurityContext but I’m a little confused why we must set a pod SecurityContext rather than a container SecurityContext.

And I think it may affects not only NFS but also any volumeMount.

Expected behavior the inject configure should NOT affect volume mount of my container.

Steps to reproduce the bug

1. create pv + pvc which use NFS with a large number of file.
2. setup a pod using this pvc as a volume.

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

$ ./istioctl version --remote
client version: 1.5.0
control plane version: 1.5.0
data plane version: 1.5.0 (575 proxies)

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6-aliyun.1", GitCommit:"a4182a8", GitTreeState:"", BuildDate:"2019-08-27T06:03:13Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed? istioctl

Environment where bug was observed (cloud vendor, OS, etc) aliyun