democratic-csi: can't get it working with truenas scale

I’m trying to setup democratic-csi with my homelab k8s and truenas scale and despite trying a lot of solutions found in web I simply cannot get it working so I’m desperately asking for help.

I tried various config options but to keep it clear here I will paste just the ones I’ve tried recently.

Values used for helm install.

csiDriver:
  # should be globally unique for a given cluster
  name: "org.democratic-csi.nfs"

# add note here about volume expansion requirements
storageClasses:
- name: freenas-nfs-csi
  defaultClass: false
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
  allowVolumeExpansion: true
  parameters:
    # for block-based storage can be ext3, ext4, xfs
    # for nfs should be nfs
    fsType: nfs
      
    # if true, volumes created from other snapshots will be
    # zfs send/received instead of zfs cloned
    # detachedVolumesFromSnapshots: "false"
    
    # if true, volumes created from other volumes will be
    # zfs send/received instead of zfs cloned
    # detachedVolumesFromVolumes: "false"

  mountOptions:
  - noatime
  - nfsvers=3
  secrets:
    provisioner-secret:
    controller-publish-secret:
    node-stage-secret:
    node-publish-secret:
    controller-expand-secret:

# if your cluster supports snapshots you may enable below
volumeSnapshotClasses: []
#- name: freenas-nfs-csi
#  parameters:
#  # if true, snapshots will be created with zfs send/receive
#  # detachedSnapshots: "false"
#  secrets:
#    snapshotter-secret:

driver:
  config:
    # please see the most up-to-date example of the corresponding config here:
    # https://github.com/democratic-csi/democratic-csi/tree/master/examples
    # YOU MUST COPY THE DATA HERE INLINE!
    driver: freenas-api-nfs
    instance_id:
    httpConnection:
      protocol: http
      host: 10.10.20.100
      port: 80
      # use only 1 of apiKey or username/password
      # if both are present, apiKey is preferred
      # apiKey is only available starting in TrueNAS-12
      apiKey: 1-IvCjJtMLUhEzIRezRzZtz4rK1HKRIFWd1UFK5ay52HogLUrwC2UxjHNQWODCRGhe
      username: root
      password:
      allowInsecure: true
      # use apiVersion 2 for TrueNAS-12 and up (will work on 11.x in some scenarios as well)
      # leave unset for auto-detection
      #apiVersion: 2
    zfs:
      # can be used to override defaults if necessary
      # the example below is useful for TrueNAS 12
      #cli:
      #  sudoEnabled: true
      #
      #  leave paths unset for auto-detection
      #  paths:
      #    zfs: /usr/local/sbin/zfs
      #    zpool: /usr/local/sbin/zpool
      #    sudo: /usr/local/bin/sudo
      #    chroot: /usr/sbin/chroot
      
      # can be used to set arbitrary values on the dataset/zvol
      # can use handlebars templates with the parameters from the storage class/CO
      #datasetProperties:
      #  "org.freenas:description": "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}/{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
      #  "org.freenas:test": "{{ parameters.foo }}"
      #  "org.freenas:test2": "some value"
    
      datasetParentName: storage/k8s/nfs
      # do NOT make datasetParentName and detachedSnapshotsDatasetParentName overlap
      # they may be siblings, but neither should be nested in the other
      #      detachedSnapshotsDatasetParentName: tank/k8s/a/snaps
      datasetEnableQuotas: true
      datasetEnableReservation: false
      datasetPermissionsMode: "0777"
      datasetPermissionsUser: 0
      datasetPermissionsGroup: 0
    
      # not supported yet
      #datasetPermissionsAcls:
      #- "-m everyone@:full_set:allow"
      #- "-m u:kube:full_set:allow"
    
    nfs:
      #shareCommentTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}-{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
      shareHost: 10.10.20.100
      shareAlldirs: false
      shareAllowedHosts: []
      shareAllowedNetworks: []
      shareMaprootUser: root
      shareMaprootGroup: root
      shareMapallUser: ""
      shareMapallGroup: ""

And actual helm command:

(fenio@debian) ➜ democratic-csi helm upgrade \
–install
–values nfs.yaml
–namespace storage --create-namespace
nfs democratic-csi/democratic-csi

It creates pods but controller fails with CrashLoopBackOff and it logs it shows:

Defaulted container "external-provisioner" out of: external-provisioner, external-resizer, external-snapshotter, csi-driver, csi-proxy
I0202 18:16:49.573963       1 feature_gate.go:245] feature gates: &{map[]}
I0202 18:16:49.574180       1 csi-provisioner.go:139] Version: v3.1.0
I0202 18:16:49.574193       1 csi-provisioner.go:162] Building kube configs for running in cluster...
I0202 18:16:49.575077       1 connection.go:154] Connecting to unix:///csi-data/csi.sock
I0202 18:16:49.575739       1 common.go:111] Probing CSI driver for readiness
I0202 18:16:49.575757       1 connection.go:183] GRPC call: /csi.v1.Identity/Probe
I0202 18:16:49.575767       1 connection.go:184] GRPC request: {}
I0202 18:16:49.584895       1 connection.go:186] GRPC response: {}
I0202 18:16:49.585018       1 connection.go:187] GRPC error: rpc error: code = Internal desc = TypeError: Cannot read properties of undefined (reading 'replace') TypeError: Cannot read properties of undefined (reading 'replace')
    at FreeNASApiDriver.getDetachedSnapshotParentDatasetName (/home/csi/app/src/driver/freenas/api.js:1992:43)
    at FreeNASApiDriver.Probe (/home/csi/app/src/driver/freenas/api.js:2135:14)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async requestHandlerProxy (/home/csi/app/bin/democratic-csi:217:18)
E0202 18:16:49.585063       1 csi-provisioner.go:197] CSI driver probe failed: rpc error: code = Internal desc = TypeError: Cannot read properties of undefined (reading 'replace') TypeError: Cannot read properties of undefined (reading 'replace')
    at FreeNASApiDriver.getDetachedSnapshotParentDatasetName (/home/csi/app/src/driver/freenas/api.js:1992:43)
    at FreeNASApiDriver.Probe (/home/csi/app/src/driver/freenas/api.js:2135:14)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async requestHandlerProxy (/home/csi/app/bin/democratic-csi:217:18)

I have no idea what exactly is it complaining about. Any help will be appreciated. I’m not sharing my TrueNAS setup as I think it’s not really relevant at this point. I think democratic-csi doesn’t even try to connect to it.