kubernetes: Secrets with specific permissions (defaultMode or mode) not being applied in Kubernetes 1.4.0

Many others referenced in a wall of text. I think a common confusion is that the files are linked, so ls will show the link permissions by default.

Simply add -L: ls -laL /path/to/directory/ to dereference the link

I was beating my head on this to understand what happening. @thockin Explaned its correctly. TLDR; But if you, @petrokashlikov can set the permission to the configMap or secret file make use of subPath which wil create the file with the permission specified in defaultMode

---

Lets take look eg:-

.
.
volumes:
      - name: test-vol
        secret:
          defaultMode: 0400
          secretName: test-secret
.
.
 containers:
      - name: test-cont
         volumeMounts:
         - mountPath: /root/non_exiting_folder/secret_key_1_new_name
           name: test-secret
           subPath: scret_key_1
          - mountPath: /root/non_exiting_folder/secret_key_2_new_name
           name: test-secret
           subPath: scret_key_2

now if you get into the pod and check the permission

root@***:/root/non_exiting_folder/# ls -la
total 16
drwxr-xr-x    2 root     root          4096 Feb 27 07:22 .
drwx------    1 root     root          4096 Feb 27 07:23 ..
-r--------    1 root     root          3243 Feb 27 07:22 secret_key_1_new_name
-r--------    1 root     root           422 Feb 27 07:22 secret_key_2_new_name

• drwxr-xr-x 2 root root 4096 Feb 27 07:22 . ‘.’ is current folder; which is no_existing_folder created by k8s have the default permission 0755

• -r-------- 1 root root 3243 Feb 27 07:22 secret_key_1_new_name -r-------- 1 root root 422 Feb 27 07:22 secret_key_2_new_name The no_existing_folder created by k8s have the default permission 0755 The secrete file(renamed scret_key_1 -> secret_key_1_new_name) by k8s is having the defaultMode as specified in config Note:- If the no defaultMode is specifed, secret key file permission will be 0644 by default

---

Now comming back to @thockin explantion. You may have notice that …data, …2019_01_21_03_26_48.920981748 folders are missing when you use the subPath config If we remove the subPath config

eg:

.
.
volumes:
      - name: test-vol
        secret:
          defaultMode: 0400
          secretName: test-secret
.
.
 containers:
      - name: test-cont
         volumeMounts:
         - mountPath: /root/non_exiting_folder/child_folder
           name: test-secret

See the different by getting into the pod

root@***:/root/non_exiting_folder/# ls -la
total 16
total 8
drwxr-xr-x    3 root     root          4096 Feb 27 07:39 .
drwx------    1 root     root          4096 Feb 27 07:39 ..
drwxrwxrwt    3 root     root           120 Feb 27 07:38 child_folder

• ddrwxr-xr-x 3 root root 4096 Feb 27 07:39 . ‘.’ is current folder; which is no_existing_folder created by k8s. Permissions remains same default permission 0755
• drwxrwxrwt 3 root root 120 Feb 27 07:38 child_folder child_folder created by k8s. but the folder permission is 0777 not default permission ~0755~

/root/non_exiting_folder/ # cd child_folder/
/root/non_exiting_folder/child_folder/ # ls -la
total 4
drwxrwxrwt    3 root     root           120 Feb 27 07:38 .
drwxr-xr-x    3 root     root          4096 Feb 27 07:39 ..
drwxr-xr-x    2 root     root            80 Feb 27 07:38 ..2019_02_27_07_38_59.023449857
lrwxrwxrwx    1 root     root            31 Feb 27 07:38 ..data -> ..2019_02_27_07_38_59.023449857
lrwxrwxrwx    1 root     root            13 Feb 27 07:38 scret_key_1-> ..data/scret_key_1
lrwxrwxrwx    1 root     root            18 Feb 27 07:38 scret_key_2-> ..data/scret_key_2
~/.ssh/test #

Notice that when we removed the subPath config from yaml,

• new folder ‘…2019_02_27_07_38_59.023449857’ is created
• new file ‘data’ symlinked to that timestamp(…2019_02_27_07_38_59.023449857) folder created
• ‘…2019_02_27_07_38_59.023449857’ hold the mounted data and the files secrete_key_1, secrete_key_2 is symlinked with actual files through …data(there by timestamp folder) respectively

---

If we check the file permission

• lrwxrwxrwx 1 root root 31 Feb 27 07:38 …data -> …2019_02_27_07_38_59.023449857 **…data ** symlink file have a permission 0777 and is symlinked with …2019_02_27_07_38_59.023449857 folder.
• drwxr-xr-x 2 root root 80 Feb 27 07:38 …2019_02_27_07_38_59.023449857 but …2019_02_27_07_38_59.023449857 folder has default permission(no defaultMode) 0755 and it holds actula data.
• lrwxrwxrwx 1 root root 13 Feb 27 07:38 scret_key_1-> …data/scret_key_1 scret_key_1 the secret key file have a permission 0777 and is symlinked with …data/scret_key_1. As said earlier ‘…data’ is a symlink to timestamp folder ‘…2019_02_27_07_38_59.023449857’

Checking permission for actual data

~/.ssh/test # cd ..data/
~/.ssh/test/..2019_02_27_07_38_59.023449857 # ls -la
total 8
drwxr-xr-x    2 root     root            80 Feb 27 07:38 .
drwxrwxrwt    3 root     root           120 Feb 27 07:38 ..
-rw-r--r--    1 root     root          3243 Feb 27 07:38 scret_key_1
-rw-r--r--    1 root     root           422 Feb 27 07:38 scret_key_2
~/.ssh/test/..2019_02_27_07_38_59.023449857 #

• -rw-r–r-- 1 root root 3243 Feb 27 07:38 scret_key_1 The secret key file scret_key_1 has permission specified in defaultMode: 0400 and it holds actula data.

---

So as i understand if k8s created file or folder automatically(like no_existng_folder in mountPath, no the end file) they will use the default Permission(folder 0755, file 0644). For the end file like the last part of mountPath, will give 0777(It wont respect defaultMode value unless subPath config is specified), though am not sure why 0777 ??

We’re using a workaround as well:

containers:
- name: sftp
  image: atmoz/sftp:debian-jessie
  lifecycle:
    postStart:
      exec:
        command:
        - bash
        - -c
        - chmod 400 /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_ed25519_key
  volumeMounts:
  - name: keys
    mountPath: /etc/ssh/ssh_host_rsa_key
    subPath: ssh_host_rsa_key
  - name: keys
    mountPath: /etc/ssh/ssh_host_ed25519_key
    subPath: ssh_host_ed25519_key
volumes:
- name: keys
  secret:
    secretName: ssh-keys

Joshperry–you’re getting the files mounted with octal permissions “0620”…0620 is 400 in decimal. The mode/defaultMode field is treating “400” as a decimal number…and you’re getting the expected behavior for that.

In my case the lifecycle approach ran into race condition issues. If it doesn’t work for you, consider using an init container. This solved my problem with a secret git-secret and key ssh

     # FIXME: Remove this when defaultMode is working
     initContainers:
      - name: fix-perms
        image: busybox
        command:
        - sh
        - -c
        - /bin/chmod 400 /etc/git-secret/ssh
        volumeMounts:
        - name: git-secret
          mountPath: /etc/git-secret/ssh
          subPath: ssh
        securityContext:
          runAsUser: 0
      volumes:
      - name: git-secret
         secret:
           secretName: the-secret-name
           defaultMode: 0400

@thockin didn’t work for me 😢

I applied your above yaml file. Belows are output:

root@test-secret-perms-996fb7547-n7689:/test# ls -al
total 4
drwxrwsrwt 3 root  123  120 Jan 21 03:26 .
drwxr-xr-x 1 root root 4096 Jan 21 03:26 ..
drwxr-sr-x 2 root  123   80 Jan 21 03:26 ..2019_01_21_03_26_48.920981748
lrwxrwxrwx 1 root root   31 Jan 21 03:26 ..data -> ..2019_01_21_03_26_48.920981748
lrwxrwxrwx 1 root root   18 Jan 21 03:26 expect_0400 -> ..data/expect_0400
lrwxrwxrwx 1 root root   18 Jan 21 03:26 expect_0700 -> ..data/expect_0700

Which k8s version should I use?

my version:

kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T11:55:20Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

I also faced similar issue and seems it is issue with representation, but actual permissions are fine. I’ve set defaultMode: 256 for volume with ssh keys. When I look at /etc/ssh/keys folder which I mounted, permissions doesn’t look right lrwxrwxrwx 1 root root 27 Feb 15 23:29 ssh_host_rsa_key.pub -> ..data/ssh_host_rsa_key.pub lrwxrwxrwx 1 root root 23 Feb 15 23:29 ssh_host_rsa_key -> ..data/ssh_host_rsa_key lrwxrwxrwx 1 root root 31 Feb 15 23:29 ssh_host_ed25519_key.pub -> ..data/ssh_host_ed25519_key.pub lrwxrwxrwx 1 root root 27 Feb 15 23:29 ssh_host_ed25519_key -> ..data/ssh_host_ed25519_key

but you see that it is actually not files, but links to actual files and if you go to that directory, permissions are read only and from original /etc/ssh/keys dir you in fact can’t delete files, so there is no write permissions as displayed

root@sftp-5f5cb76dd4-mljww:/etc/ssh/keys/…data# ls -lrt total 16 -r-------- 1 root root 754 Feb 15 23:29 ssh_host_rsa_key.pub -r-------- 1 root root 3401 Feb 15 23:29 ssh_host_rsa_key -r-------- 1 root root 110 Feb 15 23:29 ssh_host_ed25519_key.pub -r-------- 1 root root 419 Feb 15 23:29 ssh_host_ed25519_key

I flagged this for followup, and finally had some time. My test shows it working – someone tell me otherwise?

kubectl apply this file:

apiVersion: v1
kind: Secret
metadata:
  name: test-secret-perms
stringData:
  expect_0700: "should be 0700"
  expect_0400: "should be 0400"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-secret-perms
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-secret-perms
  template:
    metadata:
      labels:
        app: test-secret-perms
    spec:
      containers:
      - name: test
        image: ubuntu
        command: [ "sh", "-c", "while true; do ls -ldH /test/*; sleep 60; done" ]                                                                                         
        volumeMounts:
        - name: secret
          mountPath: /test
      securityContext:
        fsGroup: 123 
      volumes:
      - name: secret
        secret:
          secretName: test-secret-perms
          defaultMode: 256
          items:
            - key: "expect_0700"
              path: "expect_0700"
              mode: 448
            - key: "expect_0400"
              path: "expect_0400"
              # no mode             

What I see in the logs is:

-r--r----- 1 root 123 14 Dec 21 22:44 /test/expect_0400
-rwxr----- 1 root 123 14 Dec 21 22:44 /test/expect_0700

0400 and 0700 are correct. The extra g+r is because I tested the fsGroup at the same time.

This appears fixed (by my repro test)

I’ve changed my config to use octal 0400 (also tried 256 decimal) and it is now working properly. So perhaps not an upgrade issue, at least for me, from ~1.3.