zfs-localpv: Openebs-zfs-controller crashes on PVC request

What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.] As soon as a PVC is created, we see openebs-zfs-controller crashing and then after some time the PVC updates its status to bound.

Example of PVC

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: csi-zfspv
spec:
  storageClassName: test-class-sonarr
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi

I have multiple machines and was not able to reproduce it but a user is successfully able to reproduce this on his machine ( i am creating this issue on his behalf - https://jira.ixsystems.com/browse/NAS-111147 )

What did you expect to happen:

I expect the PVC to be bound immediately and not go in a pending state for some minutes.

The output of the following commands will help us better understand what’s going on: (Pasting long output into a GitHub gist or other Pastebin is fine.)

• kubectl logs -f openebs-zfs-controller-0 -n kube-system -c openebs-zfs-plugin https://pastebin.com/gGJK1Yza
• kubectl logs -f openebs-zfs-node-[xxxx] -n kube-system -c openebs-zfs-plugin https://pastebin.com/kzNXAY07
• kubectl get zv -A -o yaml

apiVersion: v1
items:
- apiVersion: zfs.openebs.io/v1
  kind: ZFSVolume
  metadata:
    creationTimestamp: "2021-06-24T16:09:30Z"
    finalizers:
    - zfs.openebs.io/finalizer
    generation: 2
    labels:
      kubernetes.io/nodename: ix-truenas
    name: pvc-8ecac328-dd1b-4f71-a614-6af9081ae0f9
    namespace: openebs
    resourceVersion: "866"
    uid: 0ba22f42-5f5a-4683-9e31-a2f4d01f030b
  spec:
    capacity: "107374182400"
    fsType: zfs
    ownerNodeID: ix-truenas
    poolName: storage_404/ix-applications/releases/truecommand/volumes
    shared: "yes"
    volumeType: DATASET
  status:
    state: Ready
- apiVersion: zfs.openebs.io/v1
  kind: ZFSVolume
  metadata:
    creationTimestamp: "2021-06-24T16:17:29Z"
    finalizers:
    - zfs.openebs.io/finalizer
    generation: 2
    labels:
      kubernetes.io/nodename: ix-truenas
    name: pvc-0bffbc09-5e56-4292-9a7a-2a7ecd71f69a
    namespace: openebs
    resourceVersion: "1483"
    uid: fe25f8f1-c90e-43b8-bd32-5b51cea02291
  spec:
    capacity: "107374182400"
    fsType: zfs
    ownerNodeID: ix-truenas
    poolName: storage_404/ix-applications/releases/sonarr/volumes
    shared: "yes"
    volumeType: DATASET
  status:
    state: Ready
- apiVersion: zfs.openebs.io/v1
  kind: ZFSVolume
  metadata:
    creationTimestamp: "2021-06-24T16:26:41Z"
    finalizers:
    - zfs.openebs.io/finalizer
    generation: 2
    labels:
      kubernetes.io/nodename: ix-truenas
    name: pvc-21ece2be-212d-4912-a9be-a68207c8ad16
    namespace: openebs
    resourceVersion: "2045"
    uid: c6a6c4a6-23b0-463e-abce-ba169668d937
  spec:
    capacity: "4294967296"
    fsType: zfs
    ownerNodeID: ix-truenas
    poolName: storage_404/ix-applications/releases/sonarr/volumes
    shared: "yes"
    volumeType: DATASET
  status:
    state: Ready
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Anything else you would like to add:

Environment:

• ZFS-LocalPV version
• Kubernetes version (use kubectl version):

❯ k version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0-k3s1", GitCommit:"bcdd3feac64be5b58c8f3207f0eb6f6a02ea9f4d", GitTreeState:"clean", BuildDate:"2021-04-12T18:20:57Z", GoVersion:"go1.15.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0-k3s1", GitCommit:"bcdd3feac64be5b58c8f3207f0eb6f6a02ea9f4d", GitTreeState:"clean", BuildDate:"2021-04-12T18:20:57Z", GoVersion:"go1.15.9", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:

❯ k3s -version
k3s version v1.21.0+k3s-2f068bd4-dirty (2f068bd4)
go version go1.15.9

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

❯ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"