kubernetes: pod crashes if /run/secrets is used as a mount target

/kind bug

What happened: After upgrading from 1.9.0 to 1.10.2 / 1.10.5 (tried both), we started to get multiple containers in crash loop. This turned out to be caused by the fact that we usually map Kubernetes secrets to the container path /run/secrets and use company-internal libraries to populate application config from these secrets. I don’t know what the change is from 1.9.0 to 1.10.x (the order of which the mounts are happening maybe?)

What you expected to happen: The pod should start

How to reproduce it (as minimally and precisely as possible): create a Secret, mount it into /run/secrets on a pod

Environment:

Kubernetes version (use kubectl version): v1.10.2
Cloud provider or hardware configuration: aws (custom)
OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS
Kernel (e.g. uname -a): 4.4.0-1057-aws
Install tools: custom
Others: docker://1.13.1

log output from kubectl logs:

container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/var/lib/kubelet/pods/63a4bfea-7fa3-11e8-9f7b-06589ff22a28/volumes/kubernetes.io~secret/ansiblejobservice-token-dsbdf\\\" to rootfs \\\"/dockerdata/overlay2/ece45c15fd86a5676d99ac0dae3559f7bda882d7f7b0f9862d572f2ad52949a3/merged\\\" at \\\"/dockerdata/overlay2/ece45c15fd86a5676d99ac0dae3559f7bda882d7f7b0f9862d572f2ad52949a3/merged/run/secrets/kubernetes.io/serviceaccount\\\" caused \\\"mkdir /dockerdata/overlay2/ece45c15fd86a5676d99ac0dae3559f7bda882d7f7b0f9862d572f2ad52949a3/merged/run/secrets/kubernetes.io: read-only file system\\\"\""

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 14
Comments: 27 (4 by maintainers)

Commits related to this issue

Kubernetes: mount DB secrets elsewhere Workaround https://github.com/kubernetes/kubernetes/issues/65835 — committed to BenedictAdamson/MC by BenedictAdamson 5 years ago

Most upvoted comments

@AntonFriberg would this “solution” work, if the secret had multiple keys?

Update: Nevermind, works this way:

    spec:
      containers:
        - name: abcde
          image: someimage
          volumeMounts:
            - name: login-secret
              mountPath: "/run/secrets/username"
              readOnly: true
              subPath: username
            - name: login-secret
              mountPath: "/run/secrets/password"
              readOnly: true
              subPath: password
      volumes:
        - name: login-secret
          secret:
            secretName: readonly-user
            items:
              - key: username.txt
                path: username
              - key: password.txt
                path: password

+15

gunzl1ng3r on Jan 17, 2019

I think I found the reason for the crash. Since 1.9.4 secrets and configMaps will be mounted as read-only volumes. Since kubernetes needs to place its own secrets in /run/secrets/kubernetes.io/serviceaccount/ it is unable to place them as /run/secrets is read-only. The result is that the pod crashes. This also explains why mounting my secrets in /run/secrets/test works but not /run.

I have been unable to find a workaround since readOnly: false does not work.

#60814

+15

AntonFriberg on Aug 8, 2018

An alternative workaround, which is less intrusive, is to disable the automounting of the API credentials by setting automountServiceAccountToken to false. This prevents /var/run/secrets/kubernetes.io from being mounted.

Source Docs

mmahkamov on Jan 4, 2021

Any progress on this?

mlensment on Jun 10, 2019

This feels like a fairly critical bug to me, but we obviously made a workaround in our setup using a similar technique to what @AntonFriberg describes above. Still feels like this should at least be documented somewhere, since it breaks with the “docker standard” of placing things in /run/secrets directly.

trondhindenes on Nov 7, 2018

Ok I have found a workaround using “subPath”

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
data:
  test: TS1oeUtKYkNrQWdZYlBZSGlnWF9kcWRNZUFiME9IdWlxN0xQTjFHWG9oUQo=
type: Opaque
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: test-nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: test-nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: Always
        volumeMounts:
        - name: secret-volume
          mountPath: "/run/secrets/test"
          subPath: test
      volumes:
      - name: secret-volume
        secret:
          secretName: test-secret
          items:
          - key: test
            path: "test"

AntonFriberg on Aug 9, 2018

So that means that docker secrets path is not compatible with kubernetes secrets path, doesnt it? Is it there a way to change at least docker secrets path so they can be the same?

marc1161 on Nov 8, 2019