longhorn: [BUG] Longhorn 1.2.0 - wrong volume permissions inside container / broken fsGroup

Describe the bug After upgrade longhorn to 1.20, some container are unable to start corectly (e.g prometheus).

Looks like root cause is wrong Longhorn volume permisions inside container when container is not running as root.

Even with fsGroup specified, permissions are not set for volume.

To Reproduce

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: broken-longhorn
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: longhorn

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: broken-longhorn
  namespace: default
  labels:
    app.kubernetes.io/name: broken-longhorn
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: broken-longhorn
  template:
    metadata:
      labels:
        app.kubernetes.io/name: broken-longhorn
    spec:
      containers:
        - name: broken-longhorn
          image: ubuntu:focal-20210723
          command:
          - "/bin/sh"
          - "-ec"
          - |
            tail -f /dev/null
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - mountPath: /data
              name: data
      securityContext:
        runAsUser: 65534
        runAsNonRoot: true
        runAsGroup: 65534
        fsGroup: 65534
        fsGroupChangePolicy: Always
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: broken-longhorn

% kubectl get pvc broken-longhorn -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"broken-longhorn","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"longhorn"}}
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
  creationTimestamp: "2021-09-01T07:48:46Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: broken-longhorn
  namespace: default
  resourceVersion: "25249959"
  uid: 9ab5ca66-0794-4ad2-8aa5-73e96fa603fc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: longhorn
  volumeMode: Filesystem
  volumeName: pvc-9ab5ca66-0794-4ad2-8aa5-73e96fa603fc
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

% kubectl get pv pvc-9ab5ca66-0794-4ad2-8aa5-73e96fa603fc -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: driver.longhorn.io
  creationTimestamp: "2021-09-01T07:48:48Z"
  finalizers:
  - kubernetes.io/pv-protection
  - external-attacher/driver-longhorn-io
  name: pvc-9ab5ca66-0794-4ad2-8aa5-73e96fa603fc
  resourceVersion: "25250154"
  uid: d4db7690-4763-45bf-a5c7-9b7ae0b5d584
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: broken-longhorn
    namespace: default
    resourceVersion: "25249908"
    uid: 9ab5ca66-0794-4ad2-8aa5-73e96fa603fc
  csi:
    driver: driver.longhorn.io
    volumeAttributes:
      fromBackup: ""
      numberOfReplicas: "3"
      staleReplicaTimeout: "30"
      storage.kubernetes.io/csiProvisionerIdentity: 1630473362980-8081-driver.longhorn.io
    volumeHandle: pvc-9ab5ca66-0794-4ad2-8aa5-73e96fa603fc
  persistentVolumeReclaimPolicy: Delete
  storageClassName: longhorn
  volumeMode: Filesystem
status:
  phase: Bound

% kubectl get csidriver driver.longhorn.io -o yaml
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  annotations:
    driver.longhorn.io/kubernetes-version: v1.20.7+k3s1
    driver.longhorn.io/version: v1.2.0
  creationTimestamp: "2021-08-31T17:47:21Z"
  name: driver.longhorn.io
  resourceVersion: "24953648"
  uid: 274cd12a-6aca-47a9-bfd8-32261eb5033a
spec:
  attachRequired: true
  fsGroupPolicy: ReadWriteOnceWithFSType
  podInfoOnMount: true
  volumeLifecycleModes:
  - Persistent

$ kubectl exec -t -i broken-longhorn-c4ccbbb6f-79djg -- bash
nobody@broken-longhorn-c4ccbbb6f-79djg:/$ ls -la /data/
total 24
drwxr-xr-x 3 root root  4096 Sep  1 07:49 .
drwxr-xr-x 1 root root  4096 Sep  1 07:49 ..
drwx------ 2 root root 16384 Sep  1 07:49 lost+found
nobody@broken-longhorn-c4ccbbb6f-79djg:/$ touch /data/test
touch: cannot touch '/data/test': Permission denied

Expected behavior When fsGroup is provided, it should be used to chown destination mount.

Environment:

• Longhorn version: 1.20
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): helm
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3os/k3s
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): baremetal

Additional context

% kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:52:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.7+k3s1", GitCommit:"aa768cbdabdb44c95c5c1d9562ea7f5ded073bc0", GitTreeState:"clean", BuildDate:"2021-05-20T01:07:13Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}