topolvm: mount fails for topolvm volume

Describe the bug Volumes created by topolvm are created, but fail to mount into the pods.

The pod has the following messages:

Events:
  Type     Reason                  Age    From                     Message
  ----     ------                  ----   ----                     -------
  Normal   Scheduled               2m43s  default-scheduler        Successfully assigned android-build/topolvm-echo-test-867799b4f9-tp2fc to rtos-k8-111-247.ghs.com
  Normal   SuccessfulAttachVolume  2m44s  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-3cf2294f-81b9-4245-8ef8-5fa8643d2cc6"
  Warning  FailedMount             41s    kubelet                  Unable to attach or mount volumes: unmounted volumes=[testvol], unattached volumes=[testvol default-token-nn7bn]: timed out waiting for the condition
  Warning  FailedMount             36s    kubelet                  MountVolume.SetUp failed for volume "pvc-3cf2294f-81b9-4245-8ef8-5fa8643d2cc6" : rpc error: code = DeadlineExceeded desc = context deadline exceeded

Running lvdisplay on the host the pod is assigned to shows that a logical volume was created with the appropriate size.

Environments

• Version: v0.8.0 (same behavior on v0.7.0)
• OS: Host is Debian 10
  • Linux hostname.example.com 4.19.0-11-amd64 #1 SMP Debian 4.19.146-1 (2020-09-17) x86_64 GNU/Linux
• Kubernetes: Server Version: version.Info{Major:“1”, Minor:“19”, GitVersion:“v1.19.3”, GitCommit:“1e11e4a2108024935ecfcb2912226cedeafd99df”, GitTreeState:“clean”, BuildDate:“2020-10-14T12:41:49Z”, GoVersion:“go1.15.2”, Compiler:“gc”, Platform:“linux/amd64”}

To Reproduce Steps to reproduce the behavior:

1. Create test.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: android-build
  name: test-lvm-pvc
spec:
  storageClassName: topolvm-provisioner
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: topolvm-echo-test
spec:
  replicas: 1
  selector:
    matchLabels:
      container: test-topolvm
  template:
    metadata:
      labels:
        container: test-topolvm
    spec:
      containers:
        - name: echoserv
          image: k8s.gcr.io/echoserver:1.4
          volumeMounts:
            - name: testvol
              mountPath: /test
      volumes:
        - name: testvol
          persistentVolumeClaim:
            claimName: test-lvm-pvc

1. kubectl apply -f test.yaml

Expected behavior The volume should mound and the pod should become ready.

Additional context The fsType in the StorageClass has been changed to “ext4”.

“lvmd” is only running on some nodes in the cluster. The pod is getting assigned to a node with lvmd. The “node” DaemonSet was modified to only run on the nodes with lvmd installed.

If additional logs or information would help, please ask.