kubernetes: K8s volume mount race condition - Failed to create subPath directory for volumeMount {volume} of container {container}

What happened: I have observed a race condition / issue with multiple pods creating the same subpath volume mount on the same node.

My understanding is that if you use a volume with a subpath that doesn’t exist, Kubernetes will create it.

I believe what happens if that both pods observe that the subpath needs to be created as seen here: https://github.com/kubernetes/kubernetes/blob/v1.15.11/pkg/kubelet/kubelet_pods.go#L200

Then, both pods try and create the subpath: https://github.com/kubernetes/kubernetes/blob/v1.15.11/pkg/kubelet/kubelet_pods.go#L212

However the “slower” pod gets the error that

E0815 16:33:36.119849 19214 kubelet_pods.go:212] failed to create subPath directory for volumeMount "shared" of container "baf13717756c7b8a88829e6fb42022bc": cannot create directory /var/lib/kubelet/pods/1dab3c2f-0df9-4a1e-aedd-5a2a3e182774/volumes/kubernetes.io~glusterfs/XXXXXXXX-shared/XXX_XXXXX_XXXXXXXX_1_0_5_2020-08-15T16-33-30.421922_00-00: file exists

And finally:

E0815 16:33:36.119889 19214 kuberuntime_manager.go:779] container start failed: CreateContainerConfigError: failed to create subPath directory for volumeMount "shared" of container "baf13717756c7b8a88829e6fb42022bc"

What you expected to happen: Instead of failing because the subpath already exists, it would be better if the code instead stopped trying to create it and simply use the subpath.

How to reproduce it (as minimally and precisely as possible):

• Schedule two new pods to the same node which use subpaths for a directory that does not yet exist.
• The pods will both try and create the subpath, one will work, the other will report the error: E0815 16:33:36.119889 19214 kuberuntime_manager.go:779] container start failed: CreateContainerConfigError: failed to create subPath directory for volumeMount "shared" of container "baf13717756c7b8a88829e6fb42022bc" The timing of these events needs to be very close as I believe this issue may only occur if both pods observe they need to create the subpath.

Anything else we need to know?: Airflow is being used to schedule these pods. The shared directory we are mounting is supposed to be shared across tasks in a DAG.

Environment:

• Kubernetes version (use kubectl version): v1.15.11-eks-065dce
• Cloud provider or hardware configuration: EKS - r5.4xlarge
• OS (e.g: cat /etc/os-release):

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

• Kernel (e.g. uname -a): 4.14.186-146.268.amzn2.x86_64 #1 SMP Tue Jul 14 18:16:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Others Gluster is used for the backing volume but based on the code I suspect this would occur with other storage providers.