kubernetes: ConfigMaps and Secrets mounted with subPath do not update when changed

@supereagle that is not the same thing. sometimes you want to merge in a few config files into a magic config dir, like, /etc/condor/config.d, but leave the rest of the dir open for other things to drop things in. subPath is the right way to do that I think. but should be able to get updates still.

@thesandlord Maybe your use case is Add ConfigMap data to a specific path in the Volume.

I follow the documentation, and it works as expected. My configmap:

apiVersion: v1
data:
  config.json: |
    {
      "special.level": "good",
      "special.type": "charm"
    }
kind: ConfigMap
metadata:
  creationTimestamp: 2017-08-10T01:21:28Z
  name: my-config
  namespace: test
  resourceVersion: "2368592"
  selfLink: /api/v1/namespaces/test/configmaps/my-config
  uid: 3bde7d2d-7d6a-11e7-9d4a-fa163ed438e9

My pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
  namespace: test
spec:
  containers:
    - name: test-container
      image: nginx:latest
      volumeMounts:
      - name: config-volume
        mountPath: /usr/src/app/config
  volumes:
    - name: config-volume
      configMap:
        name: my-config
        items:
        - key: config.json
          path: config.json
  restartPolicy: Never

This is a known limitation of using subpath with atomic volume types and has been documented. There is no fix planned.

@mingfang you are right.

This configMap / items works great, it’s a symlink, so it get updated when the ConfigMap updates. However, it does not solve the other problem: let’s say if I want to mount nginx.conf to /etc/nginx/nginx.conf the directory /etc/nginx becomes empty (removing all other default configs), and will only contain this (finally symlinked) nginx.conf 😦

I would love to have a solution that combines symlinks (for hot reload), and keeping original directory contents…

@msau42 thanks for the clarification.

For anyone affected by this, I just learned about this https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#add-configmap-data-to-a-specific-path-in-the-volume and it seems to do the same thing as subPath but actually works.

For the original example above, the solution would look like this

        volumeMounts:
          - name: my-config
            mountPath: /usr/src/app/config
      volumes:
      - name: my-config
        configMap:
          name: my-config
          items:
          - key: config.json
            path: config.json

I think we just need to document this as a known limitation.

Because subpaths are bind mounted by docker, if it was a symlink, then it gets resolved to the actual path during the bindmount.

we do it like this:

      initContainers:
      - name: init-myservice
        image: openresty:1.13.6.1
        volumeMounts:
        - mountPath: /tmpconfig
          name: tmpconfig
        - mountPath: /configmap/usr/local/openresty/nginx/conf/
          name: nginx-conf
        command: ['sh', '-c', 'mkdir /tmpconfig/test; cp /usr/local/openresty/nginx/conf/* /tmpconfig/test;ln -sf  /configmap/usr/local/openresty/nginx/conf/* /tmpconfig/test/;']
      containers:
      - image: openresty:1.13.6.1
        name: openresty
        command: ['sleep', '100000']
        volumeMounts:
        - mountPath: /usr/local/openresty/nginx/conf/
          name: tmpconfig
          subPath: test
        - mountPath: /configmap/usr/local/openresty/nginx/conf/
          name: nginx-conf
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: openresty-nginx.conf
            path: nginx.conf
          name: nginxconf
        name: nginx-conf
      - emptyDir: {}
        name: tmpconfig

it work for us but not clean enough

I bumped into this issue because I was having the same frustraing problem. However, I think the following could be a legit workaround:

Designate a folder where you will mount your desired configmaps, for example:

/configmaps/

Make the mountPath of each configmap (or secret) a subfolder of this folder. Going further on previous examples with nginx, your mount would look like this:

- name: nginx-configmap   # contains nginx.conf
  mountPath: /configmaps/nginx

Now, to tie this all together, create a symlink from /etc/nginx/nginx.conf to /configmaps/nginx/nginx.conf. This symlink can exist before the configmap is even mounted.

Whenever you update the configmap, the /configmap/nginx folder is remounted, and your symlink is now pointing to the updated version.

The main challenge is technical. The subpath implementation uses bind-mounts for security reasons, and bind mount will stay with the original inode. I do not think we can remove the bind mount implementation, so the main alternatives to explore would be if we can get projected or configmap.items to provide single files instead of the whole directory.

Any way that this issue can be reopened? I ran into it with AWS EKS, and mounting a ConfigMap like

apiVersion: v1
kind: ConfigMap
metadata:
  name: site-config
  ...
data:
  site_config.yml: |
    ...some raw config string...

---

apiVersion: apps/v1
kind: Deployment

spec:
  ...
  template:
    ...
    spec:
      containers:
        ...
          volumeMounts:
            - name: config-volume
              mountPath: /usr/src/app/site_config.yml
              subPath: site_config.yml
      volumes:
      - name: config-volume
        configMap:
          name: site-config

You can update the ConfigMap, then wait some time (I waited 30min). Then kubectl exec -ti {pod} -- bash into the running pod, cat /usr/src/app/site_config.yml and still see the old ConfigMap value.

Deleting the pod to force recreation results in having the correct file contents mounted.

---

Edit:

OK I guess this isn’t an issue because it’s “documented” as not working for subPath 😞

https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically

I tried working off the example shown in that docs page, but that is set up to mount a directory, not a single file.

the docs also state that Caution: Like before, all previous files in the /etc/config/ directory will be deleted which is not how mounting the subPath way works. That also won’t work for me because I need to mount a file into an existing directory that has other files in it.

---

What is the correct way to mount a single key in a ConfigMap to a single file in an existing directory that has existing files? 😕

Really I don’t even care about a ConfigMap at all, it just forces me to have a key/value pair, but in reality all I want to do is generate a file that contains a JSON string and mount it into my project root, but there seems to be no obvious way to do that.

Yes, symlinks are involved. Atomic writer relies on symlinks to update configmaps, secrets and such: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/util/atomic_writer.go. On the host, the pod volume directory looks something like this:

$ pwd
/var/lib/kubelet/pods/uuid/volumes/kubernetes.io~secret/my-secret*
$ tree -a
.
├── ..8988_09_08_14_21_15.788262705
│   └── secret.json
├── ..data -> ..8988_09_08_14_21_15.788262705
└── secret.json -> ..data/secret.json

When you subPath the secret file, that secret.json file is bind mounted into the container. But secret.json is actually a symlink to another secret.json in a timestamped folder (…8988_09_08_14_21_15.788262705). When the secret gets updated, the symlinks get changed around but the file bind mounted into the container remains the same.

Not sure what the solution is. Hope this helps somebody else think of one though 😃

e: Disregarding the symlinking complexity of the atomic writer algorithm for the moment…the tool we have to atomically update the file is a rename, in that case is a solution even possible? The bind mount file will always refer to the same inode yes?

@kundralaci it seems to be now possible , look at that : https://github.com/kubernetes/kubernetes/issues/44815#issuecomment-297077509 I’ve just tested a few minutes ago and seems to be ok.

why is this close? what is the fix or is there going to be a fix?

It’s so sad that this one is closed without any improvement. So we have to use symlink to workaround this problem?

@jeff-1amstudios In the Dockerfile of the image. The destination doesn’t need to exist (yet) when creating the symlink

/sig storage /sig docs