topolvm: topolvm-node does not unmount btrfs volumes after running `parted -l`
Describe the bug
Looks like parted -l triggers udev rules that rename topolvm devices (/dev/topolvm/c168dad9-978b-40e4-a45b-b77e141bff7a -> /dev/mapper/topolvm-c168dad9--978b--40e4--a45b--b77e141bff7a). This is not an issue on the host system, but inside topolvm-node container there are no symlinks under /dev/mapper/, so the topolvm-node thinks the device is not mounted.
Environments
- Version: 0.27.0 (installed via Helm chart).
- OS: Ubuntu 22.04 (with 5.15 kernel).
- Kubernetes version: 1.28.8.
To Reproduce
Steps to reproduce the behavior:
- Create a StorageClass with
fsType: btrfs. - Create a Pod with generic ephemeral volume. Something like this:
apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - command: - sleep - infinity image: ubuntu:22.04 name: test-container volumeMounts: - mountPath: /mnt/ephemeral-pv name: ephemeral-pv volumes: - ephemeral: volumeClaimTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: topolvm-btrfs volumeMode: Filesystem name: ephemeral-pv - Wait until the Pod will be scheduled and ready.
- Check
/proc/mountson the node:# cat /proc/mounts | grep pvc- /dev/topolvm/c168dad9-978b-40e4-a45b-b77e141bff7a /var/lib/kubelet/pods/a50597ec-22df-4c28-9f1b-6e5f7660499d/volumes/kubernetes.io~csi/pvc-40dfde50-d302-47d3-ba97-bebc4d003857/mount btrfs rw,relatime,space_cache=v2,subvolid=5,subvol=/ 0 0 - Call
parted -lon the node.# parted -l >/dev/null 2>&1 - Check
/proc/mountson the node again (note that the device path changed from/dev/topolvm/to/dev/mapper/):# cat /proc/mounts | grep pvc- /dev/mapper/topolvm-c168dad9--978b--40e4--a45b--b77e141bff7a /var/lib/kubelet/pods/a50597ec-22df-4c28-9f1b-6e5f7660499d/volumes/kubernetes.io~csi/pvc-40dfde50-d302-47d3-ba97-bebc4d003857/mount btrfs rw,relatime,space_cache=v2,subvolid=5,subvol=/ 0 0 - Delete the Pod.
- Verify the Pod stuck in Terminating state.
- Check topolvm-node logs on the node. You will see something like this:
{"level":"info","ts":"2024-04-05T14:33:23Z","logger":"driver.node","msg":"NodeUnpublishVolume called","volume_id":"c168dad9-978b-40e4-a45b-b77e141bff7a","target_path":"/var/lib/kubelet/pods/a50597ec-22df-4c28-9f1b-6e5f7660499d/volumes/kubernetes.io~csi/pvc-40dfde50-d302-47d3-ba97-bebc4d003857/mount"} {"level":"error","ts":"2024-04-05T14:33:23Z","msg":"error on grpc call","method":"/csi.v1.Node/NodeUnpublishVolume","error":"rpc error: code = Internal desc = remove dir failed for /var/lib/kubelet/pods/a50597ec-22df-4c28-9f1b-6e5f7660499d/volumes/kubernetes.io~csi/pvc-40dfde50-d302-47d3-ba97-bebc4d003857/mount: error=unlinkat /var/lib/kubelet/pods/a50597ec-22df-4c28-9f1b-6e5f7660499d/volumes/kubernetes.io~csi/pvc-40dfde50-d302-47d3-ba97-bebc4d003857/mount: device or resource busy","stacktrace":"github.com/topolvm/topolvm/cmd/topolvm-node/app.ErrorLoggingInterceptor\n\t/workdir/cmd/topolvm-node/app/run.go:186\ngithub.com/container-storage-interface/spec/lib/go/csi._Node_NodeUnpublishVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.6.0/lib/go/csi/csi.pb.go:6168\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1374\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:1751\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.58.3/server.go:986"}
Expected behavior
Pod should not stuck in Terminating state.
Additional context
To recover you can do something from the following list:
- Create symlinks in
/dev/mapper/and/dev/topolvm/inside topolvm-node container (same as on the host). - Manually unmount PV on the node (path can be found in topolvm-node’s logs).
- Reboot the node.
I can’t reproduce it with ext4. Only with btrfs.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 19 (16 by maintainers)
Actually I think this may work. Let me see if that resolves our issue.
@jakobmoellerdev How about the following as relatively simple fixes? A design proposal is not necessary for this level of detail.
/proc/mountscontains a path starting with/dev/mapper, check the major and minor numbers of the block special file to which the path points.After running
parted -l, the following message appeared indmesg.It appears that udev is changing the device path.
As for why this is a problem with btrfs,
/proc/mountsusesshow_devnameif the filesystem implements it [1]. And btrfs implements it [2], so this seems to be the problem. (I think they made it so we can dynamically change it because we can attach and detach devices, but I think it’s causing the troubling behavior.)[1] https://github.com/torvalds/linux/blob/96fca68c4fbf77a8185eb10f7557e23352732ea2/fs/proc_namespace.c#L109 [2] https://github.com/torvalds/linux/blob/96fca68c4fbf77a8185eb10f7557e23352732ea2/fs/btrfs/super.c#L2361
@jakobmoellerdev Thank you for your explanation. I understood. However, the btrfs is experimental support in topolvm. So our team is not proactive about fixing. But if the PR is created, we will review it. So could you create the PR? Note that another possible modification as an idea would be to “support the host
/dev/mapperto be mounted on topolvm-node”. (Either is fine.)No worries, once there is clarity on what needs to be done I can submit a PR @seleznev . Just want to make sure that the TopoLVM device setup is handled correctly.