csi-driver: CSI Mount Error on Hetzner

Hi all,

hope somebody has a hint for me as I already invested hours without a solution, in the beginning everything with my kuberone cluster on Hetzner worked as expected. auto provisioning of block storage over the csi interface is exactly as it should. Since days across the different namespaces errors occur at initial pod starting after provisioning via helm

the error message at pod startup

MountVolume.MountDevice failed for volume "pvc-a0a20263-9462-4c3e-9ebb-be45b92da7f4" : rpc error: code = Internal desc = failed to stage volume: format of disk "/dev/disk/by-id/scsi-0HC_Volume_16943979" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-a0a20263-9462-4c3e-9ebb-be45b92da7f4/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.7 (28-Jan-2021) The file /dev/disk/by-id/scsi-0HC_Volume_16943979 does not exist and no size was specified. )

Unable to attach or mount volumes: unmounted volumes=[redis-pvc], unattached volumes=[redis-pvc default-token-g7mjq]: timed out waiting for the condition

parts of the deployment file

spec:
  volumes:
    - name: redis-pvc
      persistentVolumeClaim:
        claimName: redis-pvc-chirpstack-redis-0
    - name: default-token-g7mjq
      secret:
        secretName: default-token-g7mjq
        defaultMode: 420


 volumeMounts:
        - name: redis-pvc
          mountPath: /data
        - name: default-token-g7mjq
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount

pvc is like this

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-pvc-chirpstack-redis-0
  namespace: chirpstack
  selfLink: >-
    /api/v1/namespaces/chirpstack/persistentvolumeclaims/redis-pvc-chirpstack-redis-0
  uid: 70520ae2-b99f-4d7f-a625-5e97d1748dd9
  resourceVersion: '2367972'
  creationTimestamp: '2022-02-15T17:44:04Z'
  labels:
    app: redis
    release: chirpstack
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: csi.hetzner.cloud
    volume.kubernetes.io/selected-node: oc4-pool1-779fc8f494-whkcs
  finalizers:
    - kubernetes.io/pvc-protection
  managedFields:
    - manager: kube-scheduler
      operation: Update
      apiVersion: v1
      time: '2022-02-15T17:44:04Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:volume.kubernetes.io/selected-node: {}
    - manager: kube-controller-manager
      operation: Update
      apiVersion: v1
      time: '2022-02-15T17:44:07Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            f:pv.kubernetes.io/bind-completed: {}
            f:pv.kubernetes.io/bound-by-controller: {}
            f:volume.beta.kubernetes.io/storage-provisioner: {}
          f:labels:
            .: {}
            f:app: {}
            f:release: {}
        f:spec:
          f:accessModes: {}
          f:resources:
            f:requests:
              .: {}
              f:storage: {}
          f:storageClassName: {}
          f:volumeMode: {}
          f:volumeName: {}
        f:status:
          f:accessModes: {}
          f:capacity:
            .: {}
            f:storage: {}
          f:phase: {}
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 10Gi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500M
  volumeName: pvc-70520ae2-b99f-4d7f-a625-5e97d1748dd9
  storageClassName: hcloud-volumes
  volumeMode: Filesystem

pv

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-70520ae2-b99f-4d7f-a625-5e97d1748dd9
  selfLink: /api/v1/persistentvolumes/pvc-70520ae2-b99f-4d7f-a625-5e97d1748dd9
  uid: cc6f8d63-a75f-4954-88d3-4ad5c68ffbc1
  resourceVersion: '2367987'
  creationTimestamp: '2022-02-15T17:44:07Z'
  annotations:
    pv.kubernetes.io/provisioned-by: csi.hetzner.cloud
  finalizers:
    - kubernetes.io/pv-protection
    - external-attacher/csi-hetzner-cloud
  managedFields:
    - manager: csi-provisioner
      operation: Update
      apiVersion: v1
      time: '2022-02-15T17:44:07Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:pv.kubernetes.io/provisioned-by: {}
        f:spec:
          f:accessModes: {}
          f:capacity:
            .: {}
            f:storage: {}
          f:claimRef:
            .: {}
            f:apiVersion: {}
            f:kind: {}
            f:name: {}
            f:namespace: {}
            f:resourceVersion: {}
            f:uid: {}
          f:csi:
            .: {}
            f:driver: {}
            f:fsType: {}
            f:volumeAttributes:
              .: {}
              f:storage.kubernetes.io/csiProvisionerIdentity: {}
            f:volumeHandle: {}
          f:nodeAffinity:
            .: {}
            f:required:
              .: {}
              f:nodeSelectorTerms: {}
          f:persistentVolumeReclaimPolicy: {}
          f:storageClassName: {}
          f:volumeMode: {}
    - manager: kube-controller-manager
      operation: Update
      apiVersion: v1
      time: '2022-02-15T17:44:07Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:phase: {}
    - manager: csi-attacher
      operation: Update
      apiVersion: v1
      time: '2022-02-15T17:44:08Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            v:"external-attacher/csi-hetzner-cloud": {}
status:
  phase: Bound
spec:
  capacity:
    storage: 10Gi
  csi:
    driver: csi.hetzner.cloud
    volumeHandle: '16943978'
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1644479434542-8081-csi.hetzner.cloud
  accessModes:
    - ReadWriteOnce
  claimRef:
    kind: PersistentVolumeClaim
    namespace: chirpstack
    name: redis-pvc-chirpstack-redis-0
    uid: 70520ae2-b99f-4d7f-a625-5e97d1748dd9
    apiVersion: v1
    resourceVersion: '2367918'
  persistentVolumeReclaimPolicy: Delete
  storageClassName: hcloud-volumes
  volumeMode: Filesystem
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: csi.hetzner.cloud/location
              operator: In
              values:
                - nbg1

this messages happen across different deployments with different applications / services.

thx a lot Martin