kubernetes: hostPath volume with subPath volume mount does not work with containerized kubelets

Fix status:

Fix development in progress for master branch
Fix for release-1.10 branch pending completion of master branch fix
Fix for release-1.9 branch pending completion of master branch fix
Fix for release-1.8 branch pending completion of master branch fix
Fix for release-1.7 branch pending completion of master branch fix

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: With following pod:

apiVersion: v1
kind: Pod
metadata:
  name: hp
spec:
  containers:
  - image: nginx
    name: hp1
    volumeMounts:
    - mountPath: /test-pd
      subPath: m1
      name: test-volume
  volumes:
  - name: test-volume
    hostPath:
      path: /tmp

With k8s v1.10.0-beta.0, it will create ‘/tmp/m1’ on host agent, and mount to the pod container. Inside the container, if creating a file under ‘/test-pd’, the file will appear in /tmp/m1 of host agent. This looks good.

With k8s v1.10.0-beta.4, no dir will be created under ‘/tmp’ on host agent. And if create some file under ‘/test-pd’, the file will appear in ‘/var/lib/docker/overlay2/{id}/diff/tmp/m1’.

The mount output for two versions:

v1.10.0-beta.0 Inside container:

# mount |grep test-pd
/dev/sda1 on /test-pd type ext4 (rw,relatime,discard,data=ordered)

v1.10.0-beta.4 Inside container:

# mount |grep test-pd
overlay on /test-pd type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/CWJ5CIKPXWKFO3IUAXUUSKBYKW:/var/lib/docker/overlay2/l/JU4V2IT3H2PDJH2UZH6GT3QMNQ:/var/lib/docker/overlay2/l/3PAY56CXYE6E2XJ6F54DZGYN2J:/var/lib/docker/overlay2/l/LZXQK45MQSY3JXCYS4TGR5C3ZN:/var/lib/docker/overlay2/l/TKPK6O3B4JA7DFE7E56B256KNS:/var/lib/docker/overlay2/l/NQLMB3ZLZ2XKPGKWG75T54UUXP:/var/lib/docker/overlay2/l/KOZYE6745J23T5T6ZOHZPUXXHL:/var/lib/docker/overlay2/l/SBWQYRHZ57TZMSH4HJ3EVYJVR2:/var/lib/docker/overlay2/l/CHXJUW26W2TRZ4DCWGS64GN4CN,upperdir=/var/lib/docker/overlay2/96acfb05025b8aa34fb28f33ae9dc684b4cbb09f670507e2b8d176feea66a5de/diff,workdir=/var/lib/docker/overlay2/96acfb05025b8aa34fb28f33ae9dc684b4cbb09f670507e2b8d176feea66a5de/work)

What you expected to happen: The subpath to be created under volume root.

How to reproduce it (as minimally and precisely as possible): Create the pod with given spec.

Anything else we need to know?:

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-beta.4", GitCommit:"a2dacb6f846e9d64d02c5dc5ab3972287d337405", GitTreeState:"clean", BuildDate:"2018-03-14T05:45:34Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 1
Comments: 20 (19 by maintainers)

Commits related to this issue

Known Issue: Added 61446 and 61456 * Use of subPath module with hostPath volumes can cause issues during reconstruction ([#61446](https://github.com/kubernetes/kubernetes/issues/61446)) and with cont... — committed to nickchase/sig-release by deleted user 6 years ago
Merge pull request #63143 from jsafrane/containerized-subpath Automatic merge from submit-queue (batch tested with PRs 63348, 63839, 63143, 64447, 64567). If you want to cherry-pick this change to an... — committed to kubernetes/kubernetes by deleted user 6 years ago
default disable Prometheus volumeMount SubPath There will be bugs when mounting local-path pvc and defining subpath, it is recommended to close subpath by default https://github.com/kubernetes/kub... — committed to hamishliu/prometheus-helm-charts by hamishliu 2 years ago

Most upvoted comments

@msau42 If we don’t use subPath in hostPath volume in containerized kubelet, it works.

Below is containerized kubelet config on Azure, the host’s “/” is not mounted into kubelet container:

ExecStart=/usr/bin/docker run \
  --net=host \
  --pid=host \
  --privileged \
  --rm \
  --volume=/dev:/dev \
  --volume=/sys:/sys:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/var/lib/cni/:/var/lib/cni:rw \
  --volume=/sbin/d/:/sbin/d:rw \
  --volume=/var/lib/docker/:/var/lib/docker:rw \
  --volume=/var/lib/containers/:/var/lib/containers:rw \
  --volume=/var/lib/kubelet/:/var/lib/kubelet:shared \
  --volume=/var/log:/var/log:rw \
  --volume=/etc/kubernetes/:/etc/kubernetes:ro \
  --volume=/srv/kubernetes/:/srv/kubernetes:ro $DOCKER_OPTS \
  --volume=/var/lib/waagent/ManagedIdentity-Settings:/var/lib/waagent/ManagedIdentity-Settings:ro \
  --volume=/etc/kubernetes/volumeplugins:/etc/kubernetes/volumeplugins:rw \
    ${KUBELET_IMAGE} \
      /hyperkube kubelet \
        --enable-server \
        --node-labels="${KUBELET_NODE_LABELS}" \
        --v=2 \
        --non-masquerade-cidr=${KUBELET_NON_MASQUERADE_CIDR} \
        --volume-plugin-dir=/etc/kubernetes/volumeplugins \
        $KUBELET_CONFIG $KUBELET_OPTS \
        ${KUBELET_REGISTER_NODE} ${KUBELET_REGISTER_WITH_TAINTS}

andyzhangx on Mar 22, 2018

I think I got the root cause why subPath in hostPath volume does not work, this issue should only happen for containerized kubelet, SafeMakeDir in kubelet would make subpath directories, if kubelet is running directly in host, it would work, while for containerized kubelet, SafeMakeDir only makes directories in kubelet container, it’s useless. Before this subpath security fix, actually docker will make subpath directories itself, while with this subpath security fix, in Linux, the subPath dir will be “bind mount” to dir like /var/lib/kubelet/pods/..., so finally docker will create subPath dir in /var/lib/kubelet/pods/... other than /tmp in the above example.

In one word, this subpath security fix has broken subPath functionality in hostPath volume for containerized kubelet in Linux

andyzhangx on Mar 21, 2018

kubelet on Azure would run as a continer, and this issue should be related to “bind mount” code, so I don’t use “bind mount”, use original code, it works well.

andyzhangx on Mar 21, 2018