csi-driver: MapVolume.SetUp failed for volume: blockMapper.stageVolumeForBlock failed: "no mount capability"

I’m trying to set up Ceph via Rook, using pvcs with a StorageClass powered by hetznercloud/csi-driver, but Volumes get stuck between being attached and being mounted to a Pod.

NAMESPACE      NAME                                            READY   STATUS     RESTARTS   AGE
rook-hetzner   rook-ceph-osd-prepare-set1-0-data-wnvcq-mjrwc   0/1     Init:0/2   0          26m
rook-hetzner   rook-ceph-osd-prepare-set1-1-data-nc7cw-96czb   0/1     Init:0/2   0          26m
rook-hetzner   rook-ceph-osd-prepare-set1-2-data-mbzrb-ggftg   0/1     Init:0/2   0          26m

When I describe one of these pods, there are the following events:

Events:
  Type     Reason                  Age                 From                     Message
  ----     ------                  ----                ----                     -------
  Normal   Scheduled               <unknown>           default-scheduler        Successfully assigned rook-hetzner/rook-ceph-osd-prepare-set1-2-data-mbzrb-ggftg to kube3
  Warning  FailedAttachVolume      17m (x4 over 17m)   attachdetach-controller  AttachVolume.Attach failed for volume "pvc-5f2b44a7-169c-4958-a61a-4c86e8186cef" : rpc error: code = Aborted desc = failed to publish volume: server is locked
  Normal   SuccessfulAttachVolume  17m                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-5f2b44a7-169c-4958-a61a-4c86e8186cef"
  Warning  FailedMount             15m                 kubelet, kube3           Unable to attach or mount volumes: unmounted volumes=[set1-2-data-mbzrb], unattached volumes=[rook-ceph-osd-token-8rjhx rook-data rook-ceph-log udev set1-2-data-mbzrb-bridge ceph-conf-emptydir devices rook-binaries set1-2-data-mbzrb]: timed out waiting for the condition
  Warning  FailedMount             12m                 kubelet, kube3           Unable to attach or mount volumes: unmounted volumes=[set1-2-data-mbzrb], unattached volumes=[rook-data ceph-conf-emptydir set1-2-data-mbzrb rook-ceph-osd-token-8rjhx set1-2-data-mbzrb-bridge udev rook-binaries rook-ceph-log devices]: timed out waiting for the condition
  Warning  FailedMount             10m                 kubelet, kube3           Unable to attach or mount volumes: unmounted volumes=[set1-2-data-mbzrb], unattached volumes=[udev rook-binaries set1-2-data-mbzrb-bridge rook-data rook-ceph-osd-token-8rjhx ceph-conf-emptydir rook-ceph-log devices set1-2-data-mbzrb]: timed out waiting for the condition
  Warning  FailedMount             8m17s               kubelet, kube3           Unable to attach or mount volumes: unmounted volumes=[set1-2-data-mbzrb], unattached volumes=[set1-2-data-mbzrb set1-2-data-mbzrb-bridge rook-data ceph-conf-emptydir rook-ceph-log devices udev rook-ceph-osd-token-8rjhx rook-binaries]: timed out waiting for the condition
  Warning  FailedMount             6m2s                kubelet, kube3           Unable to attach or mount volumes: unmounted volumes=[set1-2-data-mbzrb], unattached volumes=[rook-ceph-log devices rook-binaries rook-ceph-osd-token-8rjhx set1-2-data-mbzrb rook-data ceph-conf-emptydir udev set1-2-data-mbzrb-bridge]: timed out waiting for the condition
  Warning  FailedMount             3m45s               kubelet, kube3           Unable to attach or mount volumes: unmounted volumes=[set1-2-data-mbzrb], unattached volumes=[rook-data ceph-conf-emptydir rook-ceph-log udev rook-ceph-osd-token-8rjhx set1-2-data-mbzrb-bridge devices rook-binaries set1-2-data-mbzrb]: timed out waiting for the condition
  Warning  FailedMount             90s                 kubelet, kube3           Unable to attach or mount volumes: unmounted volumes=[set1-2-data-mbzrb], unattached volumes=[devices rook-ceph-osd-token-8rjhx set1-2-data-mbzrb set1-2-data-mbzrb-bridge rook-data ceph-conf-emptydir rook-ceph-log udev rook-binaries]: timed out waiting for the condition
  Warning  FailedMapVolume         31s (x16 over 16m)  kubelet, kube3           MapVolume.SetUp failed for volume "pvc-5f2b44a7-169c-4958-a61a-4c86e8186cef" : kubernetes.io/csi: blockMapper.stageVolumeForBlock failed: rpc error: code = InvalidArgument desc = no mount capability

After a little digging, no mount capability comes from the file https://github.com/hetznercloud/csi-driver/blob/master/driver/node.go

My hetzner cloud k8s manifests installed are just the ones from the README, my CephCluster manifest is:

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-hetzner
  namespace: rook-hetzner
spec:
  dataDirHostPath: /var/lib/rook
  network:
    hostNetwork: false
  placement:
    all:
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Equal
        effect: NoSchedule
  cephVersion:
    image: ceph/ceph:v14.2.4-20190917
  mon:
    count: 3
    allowMultiplePerNode: false
  storage:
   storageClassDeviceSets:
    - name: set1
      count: 3
      portable: true
      volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          resources:
            requests:
              storage: 30Gi
          # IMPORTANT: Change the storage class depending on your environment (e.g. local-storage, gp2)
          storageClassName: hcloud-volumes
          volumeMode: Block
          accessModes:
            - ReadWriteOnce

Involved PVs

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                            STORAGECLASS     REASON   AGE
pvc-5f2b44a7-169c-4958-a61a-4c86e8186cef   30Gi       RWO            Delete           Bound    rook-hetzner/set1-2-data-mbzrb   hcloud-volumes            35m
pvc-ed978abc-56c7-4cfc-83f5-0201ceda53f3   30Gi       RWO            Delete           Bound    rook-hetzner/set1-0-data-wnvcq   hcloud-volumes            35m
pvc-f9433424-d211-454d-99fb-b90fa9891357   30Gi       RWO            Delete           Bound    rook-hetzner/set1-1-data-nc7cw   hcloud-volumes            35m

Involved PVCs

rook-hetzner   set1-0-data-wnvcq   Bound    pvc-ed978abc-56c7-4cfc-83f5-0201ceda53f3   30Gi       RWO            hcloud-volumes   35m
rook-hetzner   set1-1-data-nc7cw   Bound    pvc-f9433424-d211-454d-99fb-b90fa9891357   30Gi       RWO            hcloud-volumes   35m
rook-hetzner   set1-2-data-mbzrb   Bound    pvc-5f2b44a7-169c-4958-a61a-4c86e8186cef   30Gi       RWO            hcloud-volumes   35m

Any ideas why the volume would have no mount capability?