topolvm: topolvm-node does not unmount btrfs volumes after running `parted -l`

Describe the bug

Looks like parted -l triggers udev rules that rename topolvm devices (/dev/topolvm/c168dad9-978b-40e4-a45b-b77e141bff7a -> /dev/mapper/topolvm-c168dad9--978b--40e4--a45b--b77e141bff7a). This is not an issue on the host system, but inside topolvm-node container there are no symlinks under /dev/mapper/, so the topolvm-node thinks the device is not mounted.

Environments

  • Version: 0.27.0 (installed via Helm chart).
  • OS: Ubuntu 22.04 (with 5.15 kernel).
  • Kubernetes version: 1.28.8.

To Reproduce

Steps to reproduce the behavior:

  1. Create a StorageClass with fsType: btrfs.
  2. Create a Pod with generic ephemeral volume. Something like this:
     apiVersion: v1
     kind: Pod
     metadata:
       name: test-pod
     spec:
       containers:
       - command:
         - sleep
         - infinity
         image: ubuntu:22.04
         name: test-container
         volumeMounts:
         - mountPath: /mnt/ephemeral-pv
           name: ephemeral-pv
       volumes:
       - ephemeral:
           volumeClaimTemplate:
             spec:
               accessModes:
               - ReadWriteOnce
               resources:
                 requests:
                   storage: 1Gi
               storageClassName: topolvm-btrfs
               volumeMode: Filesystem
         name: ephemeral-pv
    
  3. Wait until the Pod will be scheduled and ready.
  4. Check /proc/mounts on the node:
    # cat /proc/mounts | grep pvc-
    /dev/topolvm/c168dad9-978b-40e4-a45b-b77e141bff7a /var/lib/kubelet/pods/a50597ec-22df-4c28-9f1b-6e5f7660499d/volumes/kubernetes.io~csi/pvc-40dfde50-d302-47d3-ba97-bebc4d003857/mount btrfs rw,relatime,space_cache=v2,subvolid=5,subvol=/ 0 0
    
  5. Call parted -l on the node.
    # parted -l >/dev/null 2>&1
    
  6. Check /proc/mounts on the node again (note that the device path changed from /dev/topolvm/ to /dev/mapper/):
    # cat /proc/mounts | grep pvc-
    /dev/mapper/topolvm-c168dad9--978b--40e4--a45b--b77e141bff7a /var/lib/kubelet/pods/a50597ec-22df-4c28-9f1b-6e5f7660499d/volumes/kubernetes.io~csi/pvc-40dfde50-d302-47d3-ba97-bebc4d003857/mount btrfs rw,relatime,space_cache=v2,subvolid=5,subvol=/ 0 0
    
  7. Delete the Pod.
  8. Verify the Pod stuck in Terminating state.
  9. Check topolvm-node logs on the node. You will see something like this:
    {"level":"info","ts":"2024-04-05T14:33:23Z","logger":"driver.node","msg":"NodeUnpublishVolume called","volume_id":"c168dad9-978b-40e4-a45b-b77e141bff7a","target_path":"/var/lib/kubelet/pods/a50597ec-22df-4c28-9f1b-6e5f7660499d/volumes/kubernetes.io~csi/pvc-40dfde50-d302-47d3-ba97-bebc4d003857/mount"}
    {"level":"error","ts":"2024-04-05T14:33:23Z","msg":"error on grpc call","method":"/csi.v1.Node/NodeUnpublishVolume","error":"rpc error: code = Internal desc = remove dir failed for /var/lib/kubelet/pods/a50597ec-22df-4c28-9f1b-6e5f7660499d/volumes/kubernetes.io~csi/pvc-40dfde50-d302-47d3-ba97-bebc4d003857/mount: error=unlinkat /var/lib/kubelet/pods/a50597ec-22df-4c28-9f1b-6e5f7660499d/volumes/kubernetes.io~csi/pvc-40dfde50-d302-47d3-ba97-bebc4d003857/mount: device or resource busy","stacktrace":"github.com/topolvm/topolvm/cmd/topolvm-node/app.ErrorLoggingInterceptor\n\t/workdir/cmd/topolvm-node/app/run.go:186\ngithub.com/container-storage-interface/spec/lib/go/csi._Node_NodeUnpublishVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.6.0/lib/go/csi/csi.pb.go:6168\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1374\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1751\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:986"}
    

Expected behavior

Pod should not stuck in Terminating state.

Additional context

To recover you can do something from the following list:

  1. Create symlinks in /dev/mapper/ and /dev/topolvm/ inside topolvm-node container (same as on the host).
  2. Manually unmount PV on the node (path can be found in topolvm-node’s logs).
  3. Reboot the node.

I can’t reproduce it with ext4. Only with btrfs.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 19 (16 by maintainers)

Most upvoted comments

Actually I think this may work. Let me see if that resolves our issue.

@jakobmoellerdev How about the following as relatively simple fixes? A design proposal is not necessary for this level of detail.

  1. If /proc/mounts contains a path starting with /dev/mapper, check the major and minor numbers of the block special file to which the path points.
  2. If the major and minor numbers match the major and minor numbers used to create the block special file, it can be judged to be mounted.

I wonder why parted -l is changing the /proc/mounts here tbh. Do we have some info on that?

After running parted -l, the following message appeared in dmesg.

[91357.279111] BTRFS info: devid 1 device path /dev/mapper/ubuntu--vg-ubuntu--lv changed to /dev/dm-0 scanned by systemd-udevd (257020)
[91357.290952] BTRFS info: devid 1 device path /dev/dm-0 changed to /dev/mapper/ubuntu--vg-ubuntu--lv scanned by systemd-udevd (257020)

It appears that udev is changing the device path.

As for why this is a problem with btrfs, /proc/mounts uses show_devname if the filesystem implements it [1]. And btrfs implements it [2], so this seems to be the problem. (I think they made it so we can dynamically change it because we can attach and detach devices, but I think it’s causing the troubling behavior.)

[1] https://github.com/torvalds/linux/blob/96fca68c4fbf77a8185eb10f7557e23352732ea2/fs/proc_namespace.c#L109 [2] https://github.com/torvalds/linux/blob/96fca68c4fbf77a8185eb10f7557e23352732ea2/fs/btrfs/super.c#L2361

@jakobmoellerdev Thank you for your explanation. I understood. However, the btrfs is experimental support in topolvm. So our team is not proactive about fixing. But if the PR is created, we will review it. So could you create the PR? Note that another possible modification as an idea would be to “support the host /dev/mapper to be mounted on topolvm-node”. (Either is fine.)

No worries, once there is clarity on what needs to be done I can submit a PR @seleznev . Just want to make sure that the TopoLVM device setup is handled correctly.