kind: Kind cluster fails to provision PV when a USB device was removed from the machine

What happened:

I’m running Kind (with export KIND_EXPERIMENTAL_PROVIDER=podman) on my laptop. When I start the cluster while a mouse is connected to the machine, I’m able to create a pod with a local volume. Once I remove that mouse, this starts to fail.

The same issue happens when I close the lid to have the laptop go to sleep, and then wake it up again.

What you expected to happen:

Setup of PVCs and PVs continues to work.

How to reproduce it (as minimally and precisely as possible):

1. export KIND_EXPERIMENTAL_PROVIDER=podman
2. lsusb returns something like

Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 003: ID 13d3:5405 IMC Networks Integrated Camera
Bus 003 Device 044: ID 06cb:00f9 Synaptics, Inc. 
Bus 003 Device 046: ID 0458:0007 KYE Systems Corp. (Mouse Systems) Trackbar Emotion
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

1. kind create cluster
2. Have a YAML file duplicating the standard storageclass under the name local-path, something like cat storageclass-local-path.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path
  namespace: kube-system
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

1. kubectl apply -f storageclass-local-path.yaml
2. kubectl apply -k 'https://github.com/rancher/local-path-provisioner/examples/pod-with-local-volume'
3. After a small while, kubectl get pods -A show volume-test in namespace default as Running.
4. kubectl delete -k 'https://github.com/rancher/local-path-provisioner/examples/pod-with-local-volume'
5. Disconnect that USB mouse.
6. Check with lsusb that the device 003/046 or whatever ids it had is no longer there.
7. kubectl apply -k 'https://github.com/rancher/local-path-provisioner/examples/pod-with-local-volume'
8. kubectl get pods -A shows

NAMESPACE            NAME                                                         READY   STATUS       RESTARTS   AGE
default              volume-test                                                  0/1     Pending      0          9s
[...]
local-path-storage   helper-pod-create-pvc-1e7e0729-1ec4-4b0e-91ef-3c41e0495783   0/1     StartError   0          9s

1. kubectl events -n local-path-storage deployment/local-path-provisioner shows

42s         Warning   Failed              Pod/helper-pod-create-pvc-1e7e0729-1ec4-4b0e-91ef-3c41e0495783   Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error creating device nodes: mount /dev/bus/usb/003/046:/run/containerd/io.containerd.runtime.v2.task/k8s.io/helper-pod/rootfs/dev/bus/usb/003/046 (via /proc/self/fd/6), flags: 0x1000: no such file or directory: unknown

Anything else we need to know?:

I actually first encountered it when I suspended the laptop and then woken it up and wanted to continue using the Kind cluster. The Bus 003 Device 044: ID 06cb:00f9 Synaptics, Inc. device gets a different device id upon wakeup.

Environment:

• kind version: (use kind version): kind v0.20.0 go1.20.4 linux/amd64
• Runtime info: (use docker info or podman info):

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 70.31
    systemPercent: 6.54
    userPercent: 23.15
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: xfce
    version: "38"
  eventLogger: journald
  freeLocks: 2038
  hostname: machine.example.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.5.6-200.fc38.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 8981233664
  memTotal: 33331113984
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc38.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc38.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.9.2-1.fc38.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.9.2
      commit: 35274d346d2e9ffeacb22cc11590b0266a23d634
      rundir: /run/user/2000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231004.gf851084-1.fc38.x86_64
    version: |
      pasta 0^20231004.gf851084-1.fc38.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/2000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.fc38.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589877248
  swapTotal: 8589930496
  uptime: 202h 32m 16.00s (Approximately 8.42 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/kind/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/kind/.local/share/containers/storage
  graphRootAllocated: 26241896448
  graphRootUsed: 11933265920
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 94
  runRoot: /tmp/containers-user-2000/containers
  transientStore: false
  volumePath: /home/kind/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.0
  Built: 1695839078
  BuiltTime: Wed Sep 27 20:24:38 2023
  GitCommit: ""
  GoVersion: go1.20.8
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.0

• OS (e.g. from /etc/os-release): CPE_NAME="cpe:/o:fedoraproject:fedora:38"
• Kubernetes version: (use kubectl version):

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.9", GitCommit:"d1483fdf7a0578c83523bc1e2212a606a44fd71d", GitTreeState:"archive", BuildDate:"2023-09-16T00:00:00Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.3", GitCommit:"25b4e43193bcda6c7328a6d147b1fb73a33f1598", GitTreeState:"clean", BuildDate:"2023-06-15T00:36:28Z", GoVersion:"go1.20.5", Compiler:"gc", Platform:"linux/amd64"}

• Any proxies or other special environment settings?: KIND_EXPERIMENTAL_PROVIDER=podman