kubernetes: VolumeMount for ConfigMap fails to be remounted - Nested mountPoints

Hi.

I’m keep getting “device or resource is busy” errors when kubernetes tries to re-mount volumes which are based on configMap. I’ve seen several threads regarding volume mounting failures but they doesn’t seem to be related to the failure I’m seeing.

I’m using Kubernetes v1.4.1

The error I’m seeing is for specific configMap ‘hekad-common-config’

/var/log/messages is full of the following logs:

MountVolume operation started for volume "kubernetes.io/configmap/c4d620be-1ac3-11e7-a48d-fa163e70f679-hekad-common-config" (spec.Name: "hekad-common-config") to pod "c4d620be-1ac3-11e7-a48d-fa163e70f679" (UID: "c4d620be-1ac3-11e7-a48d-fa163e70f679"). Volume is already mounted to pod, but remount was requested.
pod lbeta/infrabridge-kube2consul-4287528991-obzn3 volume hekad-common-config: error pruning old user-visible path component: remove /var/lib/origin/openshift.local.volumes/pods/c4d620be-1ac3-11e7-a48d-fa163e70f679/volumes/kubernetes.io~configmap/hekad-common-config/component: device or resource busy
pod lbeta/infrabridge-kube2consul-4287528991-obzn3 volume hekad-common-config: error removing old visible symlinks: remove /var/lib/origin/openshift.local.volumes/pods/c4d620be-1ac3-11e7-a48d-fa163e70f679/volumes/kubernetes.io~configmap/hekad-common-config/component: device or resource busy                                                                                                                                                                                  
Error writing payload to dir: remove /var/lib/origin/openshift.local.volumes/pods/c4d620be-1ac3-11e7-a48d-fa163e70f679/volumes/kubernetes.io~configmap/hekad-common-config/component: device or resource busy                                                                      
Operation for "\"kubernetes.io/configmap/c4d620be-1ac3-11e7-a48d-fa163e70f679-hekad-common-config\" (\"c4d620be-1ac3-11e7-a48d-fa163e70f679\")" failed. No retries permitted until 2017-04-07 16:57:09.001834524 +0000 UTC (durationBeforeRetry 2m0s). Error: MountVolume.SetUp failed for volume "kubernetes.io/configmap/c4d620be-1ac3-11e7-a48d-fa163e70f679-hekad-common-config" (spec.Name: "hekad-common-config") pod "c4d620be-1ac3-11e7-a48d-fa163e70f679" (UID: "c4d620be-1ac3-11e7-a48d-fa163e70f679") with: remove /var/lib/origin/openshift.local.volumes/pods/c4d620be-1ac3-11e7-a48d-fa163e70f679/volumes/kubernetes.io~configmap/hekad-common-config/component: device or resource busy

And it just keeps re-trying re-mounting the volume every 2 minutes.

From what I’ve been able to understand, Kubernetes keeps re-mounting volumes which are based on ConfigMaps. (I guess this is to keep the mount content updated in case the configMap has been updated) There are several volumes of configMaps which are set to pods, but I think I understand the reason specific failures are seen to hekad-common-config.

Note the it tries to re-mount the volume for hekad-common-config, which is mounted to /etc/heka-config.d But the device is busy error is for hekad-common-config/component, which is from hekad-component-config configMap.

In the pod template:

      volumeMounts:
        - name: hekad-component-config
          mountPath: /etc/heka-config.d/component
        - name: hekad-common-config
          mountPath: /etc/heka-config.d

So my question here, could it be the root cause of the issue is that there are 2 VolumeMounts on the same path (Where one of them is in a sub-directory of another mountPath)

My only guess is that device busy errors for ‘/etc/heka-config.d’ may happen if it has files within this path which are still in use by the system - In this case /etc/heka-config.d/component may be mounted and causing a failure when trying to un-mount /etc/heka-config.d.

If this is the cause, then I think Kubernetes should fail creation of such pods which have multiple volumes which are being mounted to the same base paths.