democratic-csi: TrueNAS Scale iscsi problem

Seems there is a problem with the chart and I’m running out of ideas about what could be wrong with it. I’ve downgraded to kube v1.23 as it was failing also with newer versions. But seems this one is getting closer to getting this work.

csiDriver:
  name: "org.democratic-csi.iscsi"
storageClasses:
- name: freenas-iscsi-csi
  defaultClass: true
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
  allowVolumeExpansion: true
  parameters:
    fsType: xfs
  mountOptions: []
  secrets:
    provisioner-secret:
    controller-publish-secret:
    node-stage-secret:
    node-publish-secret:
    controller-expand-secret:
driver:
  config: 
    driver: freenas-api-iscsi
    instance_id: truenas
    httpConnection:
      username: root
      password: *******
      protocol: https
      host: 192.168.3.71
      port: 443
      # apiKey: <key>
      allowInsecure: true
      apiVersion: 2
    zfs:
      datasetParentName: NVME_POOL/iscsi-truenas-ugly-zvol
      detachedSnapshotsDatasetParentName: SSD_POOL/truenas-ugly-snapshot-ssd-zvol
      zvolCompression:
      zvolDedup:
      zvolEnableReservation: false
      zvolBlocksize:
    iscsi:
      targetPortal:  "192.168.3.71:3260"
      interface: br0
      namePrefix: csi-
      nameSuffix: "-cluster"
      targetGroups:
        - targetGroupPortalGroup: 1
          targetGroupInitiatorGroup: 3
          targetGroupAuthType: None
          targetGroupAuthGroup: null
      extentInsecureTpc: true
      extentXenCompat: false
      extentDisablePhysicalBlocksize: true
      extentBlocksize: 4096
      extentRpm: "SSD"
      extentAvailThreshold: 0

controller pod is working fine, however, the scsi node is having issues with driver -registrar

ansible@lb:~/k8_cluster_setup$ k describe pod truenas-iscsi-democratic-csi-node-zlzrj -n d-csi Name: truenas-iscsi-democratic-csi-node-zlzrj Namespace: d-csi Priority: 0 Service Account: truenas-iscsi-democratic-csi-node-sa Node: cent1/192.168.3.120 Start Time: Mon, 15 May 2023 17:50:27 +0000 Labels: app.kubernetes.io/component=node-linux app.kubernetes.io/csi-role=node app.kubernetes.io/instance=truenas-iscsi app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=democratic-csi controller-revision-hash=54d67654b8 pod-template-generation=1 Annotations: checksum/configmap: e4197e4e8f57b38fae6c2051c9cf652bc5768e31ff16b9c1a2f5645bd6f3d0d2 checksum/secret: fb50eb9165f01cc8e218094fead91b51c6d4399d15514bc13920b75839be21c2 kubernetes.io/psp: 00-k0s-privileged Status: Running IP: 192.168.3.120 IPs: IP: 192.168.3.120 Controlled By: DaemonSet/truenas-iscsi-democratic-csi-node Containers: csi-driver: Container ID: containerd://d622cd2a0e1049fa734dec4ea516ee575633e628a6b80971f83d2484d4649c35 Image: docker.io/democraticcsi/democratic-csi:latest Image ID: docker.io/democraticcsi/democratic-csi@sha256:f7d38023197abf70ee3e025292dd2267b9b7eb4257045871208ae2d2f487d7cf Port: <none> Host Port: <none> Args: –csi-version=1.5.0 –csi-name=org.democratic-csi.iscsi –driver-config-file=/config/driver-config-file.yaml –log-level=info –csi-mode=node –server-socket=/csi-data/csi.sock.internal State: Running Started: Mon, 15 May 2023 17:50:30 +0000 Ready: True Restart Count: 0 Liveness: exec [bin/liveness-probe --csi-version=1.5.0 --csi-address=/csi-data/csi.sock.internal] delay=10s timeout=15s period=60s #success=1 #failure=3 Environment: CSI_NODE_ID: (v1:spec.nodeName) NODE_EXTRA_CA_CERTS: /tmp/certs/extra-ca-certs.crt Mounts: /config from config (rw) /csi-data from socket-dir (rw) /dev from dev-dir (rw) /etc/iscsi from iscsi-dir (rw) /etc/localtime from localtime (ro) /host from host-dir (rw) /lib/modules from modules-dir (ro) /run/udev from udev-data (rw) /sys from sys-dir (rw) /tmp/certs from extra-ca-certs (rw) /var/lib/iscsi from iscsi-info (rw) /var/lib/kubelet from kubelet-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h9xwr (ro) csi-proxy: Container ID: containerd://df899896dbc36216f1d6aac08d90e7a4ce8d2feccb9b3cfd86ab8502b9382cb6 Image: docker.io/democraticcsi/csi-grpc-proxy:v0.5.3 Image ID: docker.io/democraticcsi/csi-grpc-proxy@sha256:4d65ca1cf17d941a8df668b8fe2f1c0cfa512c8b0dbef3ff89a4cd405e076923 Port: <none> Host Port: <none> State: Running Started: Mon, 15 May 2023 17:50:30 +0000 Ready: True Restart Count: 0 Environment: BIND_TO: unix:///csi-data/csi.sock PROXY_TO: unix:///csi-data/csi.sock.internal Mounts: /csi-data from socket-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h9xwr (ro) driver-registrar: Container ID: containerd://715d74fb22d7d2afbb174fac07b35217cc36484432d069c18d9fd1546cf47e86 Image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1 Image ID: registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0103eee7c35e3e0b5cd8cdca9850dc71c793cdeb6669d8be7a89440da2d06ae4 Port: <none> Host Port: <none> Args: –v=5 –csi-address=/csi-data/csi.sock –kubelet-registration-path=/var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 15 May 2023 17:53:39 +0000 Finished: Mon, 15 May 2023 17:54:08 +0000 Ready: False Restart Count: 5 Liveness: exec [/csi-node-driver-registrar --kubelet-registration-path=/var/lib/kubelet/plugins/org.democratic-csi.iscsi/csi.sock --mode=kubelet-registration-probe] delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: KUBE_NODE_NAME: (v1:spec.nodeName) Mounts: /csi-data from socket-dir (rw) /registration from registration-dir (rw) /var/lib/kubelet from kubelet-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h9xwr (ro) cleanup: Container ID: containerd://f77ac1536bef736bfffccc74122a92b71b09f5d990734cc8496478a820b45d09 Image: docker.io/busybox:1.32.0 Image ID: docker.io/library/busybox@sha256:bde48e1751173b709090c2539fdf12d6ba64e88ec7a4301591227ce925f3c678 Port: <none> Host Port: <none> Command: /bin/sh -c – Args: while true; do sleep 2; done; State: Running Started: Mon, 15 May 2023 17:50:30 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /plugins from plugins-dir (rw) /registration from registration-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h9xwr (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: socket-dir: Type: HostPath (bare host directory volume) Path: /var/lib/kubelet/plugins/org.democratic-csi.iscsi HostPathType: DirectoryOrCreate plugins-dir: Type: HostPath (bare host directory volume) Path: /var/lib/kubelet/plugins HostPathType: Directory registration-dir: Type: HostPath (bare host directory volume) Path: /var/lib/kubelet/plugins_registry HostPathType: Directory kubelet-dir: Type: HostPath (bare host directory volume) Path: /var/lib/kubelet HostPathType: Directory iscsi-dir: Type: HostPath (bare host directory volume) Path: /etc/iscsi HostPathType: Directory iscsi-info: Type: HostPath (bare host directory volume) Path: /var/lib/iscsi HostPathType:
dev-dir: Type: HostPath (bare host directory volume) Path: /dev HostPathType: Directory modules-dir: Type: HostPath (bare host directory volume) Path: /lib/modules HostPathType:
localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType:
udev-data: Type: HostPath (bare host directory volume) Path: /run/udev HostPathType:
sys-dir: Type: HostPath (bare host directory volume) Path: /sys HostPathType: Directory host-dir: Type: HostPath (bare host directory volume) Path: / HostPathType: Directory config: Type: Secret (a volume populated by a Secret) SecretName: truenas-iscsi-democratic-csi-driver-config Optional: false extra-ca-certs: Type: ConfigMap (a volume populated by a ConfigMap) Name: truenas-iscsi-democratic-csi Optional: false kube-api-access-h9xwr: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/network-unavailable:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists node.kubernetes.io/pid-pressure:NoSchedule op=Exists node.kubernetes.io/unreachable:NoExecute op=Exists node.kubernetes.io/unschedulable:NoSchedule op=Exists Events: Type Reason Age From Message

---

Normal Scheduled 3m41s default-scheduler Successfully assigned d-csi/truenas-iscsi-democratic-csi-node-zlzrj to cent1 Normal Pulling 3m41s kubelet Pulling image “docker.io/democraticcsi/democratic-csi:latest” Normal Created 3m39s kubelet Created container cleanup Normal Pulled 3m39s kubelet Successfully pulled image “docker.io/democraticcsi/democratic-csi:latest” in 1.899412026s (1.899495973s including waiting) Normal Created 3m39s kubelet Created container csi-driver Normal Started 3m39s kubelet Started container csi-driver Normal Pulled 3m39s kubelet Container image “docker.io/democraticcsi/csi-grpc-proxy:v0.5.3” already present on machine Normal Created 3m39s kubelet Created container csi-proxy Normal Started 3m39s kubelet Started container csi-proxy Normal Started 3m39s kubelet Started container cleanup Normal Pulled 3m39s kubelet Container image “docker.io/busybox:1.32.0” already present on machine Warning Unhealthy 3m31s kubelet Liveness probe failed: F0515 17:50:38.279795 14 main.go:159] Kubelet plugin registration hasn’t succeeded yet, file=/var/lib/kubelet/plugins/org.democratic-csi.iscsi/registration doesn’t exist. goroutine 1 [running]: k8s.io/klog/v2.stacks(0x1) /workspace/vendor/k8s.io/klog/v2/klog.go:1038 +0x8a k8s.io/klog/v2.(*loggingT).output(0xf86600, 0x3, 0x0, 0xc000221880, 0x0, {0xc47a41, 0x1}, 0xc000284a10, 0x0) /workspace/vendor/k8s.io/klog/v2/klog.go:987 +0x5fd k8s.io/klog/v2.(*loggingT).printf(0xa63799, 0x4, 0x0, {0x0, 0x0}, {0xa8ac8d, 0x48}, {0xc000284a10, 0x1, 0x1}) /workspace/vendor/k8s.io/klog/v2/klog.go:753 +0x1c5 k8s.io/klog/v2.Fatalf(…) /workspace/vendor/k8s.io/klog/v2/klog.go:1532 main.main() /workspace/cmd/csi-node-driver-registrar/main.go:159 +0x48e

goroutine 8 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xc000091200) /workspace/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a created by k8s.io/klog/v2.init.0 /workspace/vendor/k8s.io/klog/v2/klog.go:420 +0xfb Warning Unhealthy 3m21s kubelet Liveness probe failed: F0515 17:50:48.287622 24 main.go:159] Kubelet plugin registration hasn’t succeeded yet, file=/var/lib/kubelet/plugins/org.democratic-csi.iscsi/registration doesn’t exist. goroutine 1 [running]: k8s.io/klog/v2.stacks(0x1) /workspace/vendor/k8s.io/klog/v2/klog.go:1038 +0x8a k8s.io/klog/v2.(*loggingT).output(0xf86600, 0x3, 0x0, 0xc00028ecb0, 0x0, {0xc47a41, 0x1}, 0xc00032ad90, 0x0) /workspace/vendor/k8s.io/klog/v2/klog.go:987 +0x5fd k8s.io/klog/v2.(*loggingT).printf(0xa63799, 0x4, 0x0, {0x0, 0x0}, {0xa8ac8d, 0x48}, {0xc00032ad90, 0x1, 0x1}) /workspace/vendor/k8s.io/klog/v2/klog.go:753 +0x1c5 k8s.io/klog/v2.Fatalf(…) /workspace/vendor/k8s.io/klog/v2/klog.go:1532 main.main() /workspace/cmd/csi-node-driver-registrar/main.go:159 +0x48e

goroutine 17 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xc000091200) /workspace/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a created by k8s.io/klog/v2.init.0 /workspace/vendor/k8s.io/klog/v2/klog.go:420 +0xfb Normal Started 3m11s (x2 over 3m39s) kubelet Started container driver-registrar Normal Created 3m11s (x2 over 3m39s) kubelet Created container driver-registrar Warning Unhealthy 3m11s kubelet Liveness probe failed: F0515 17:50:58.274645 34 main.go:159] Kubelet plugin registration hasn’t succeeded yet, file=/var/lib/kubelet/plugins/org.democratic-csi.iscsi/registration doesn’t exist. goroutine 1 [running]: k8s.io/klog/v2.stacks(0x1) /workspace/vendor/k8s.io/klog/v2/klog.go:1038 +0x8a k8s.io/klog/v2.(*loggingT).output(0xf86600, 0x3, 0x0, 0xc0000fd8f0, 0x0, {0xc47a41, 0x1}, 0xc00037afa0, 0x0) /workspace/vendor/k8s.io/klog/v2/klog.go:987 +0x5fd k8s.io/klog/v2.(*loggingT).printf(0xa63799, 0x4, 0x0, {0x0, 0x0}, {0xa8ac8d, 0x48}, {0xc00037afa0, 0x1, 0x1}) /workspace/vendor/k8s.io/klog/v2/klog.go:753 +0x1c5 k8s.io/klog/v2.Fatalf(…) /workspace/vendor/k8s.io/klog/v2/klog.go:1532 main.main() /workspace/cmd/csi-node-driver-registrar/main.go:159 +0x48e

goroutine 6 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xc000090060) /workspace/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a created by k8s.io/klog/v2.init.0 /workspace/vendor/k8s.io/klog/v2/klog.go:420 +0xfb Warning Unhealthy 3m1s kubelet Liveness probe failed: F0515 17:51:08.276632 18 main.go:159] Kubelet plugin registration hasn’t succeeded yet, file=/var/lib/kubelet/plugins/org.democratic-csi.iscsi/registration doesn’t exist. goroutine 1 [running]: k8s.io/klog/v2.stacks(0x1) /workspace/vendor/k8s.io/klog/v2/klog.go:1038 +0x8a k8s.io/klog/v2.(*loggingT).output(0xf86600, 0x3, 0x0, 0xc000242c40, 0x0, {0xc47a41, 0x1}, 0xc0002a2820, 0x0) /workspace/vendor/k8s.io/klog/v2/klog.go:987 +0x5fd k8s.io/klog/v2.(*loggingT).printf(0xa63799, 0x4, 0x0, {0x0, 0x0}, {0xa8ac8d, 0x48}, {0xc0002a2820, 0x1, 0x1}) /workspace/vendor/k8s.io/klog/v2/klog.go:753 +0x1c5 k8s.io/klog/v2.Fatalf(…) /workspace/vendor/k8s.io/klog/v2/klog.go:1532 main.main() /workspace/cmd/csi-node-driver-registrar/main.go:159 +0x48e

goroutine 19 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xc000090060) /workspace/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a created by k8s.io/klog/v2.init.0 /workspace/vendor/k8s.io/klog/v2/klog.go:420 +0xfb Warning Unhealthy 2m51s kubelet Liveness probe failed: F0515 17:51:18.292343 28 main.go:159] Kubelet plugin registration hasn’t succeeded yet, file=/var/lib/kubelet/plugins/org.democratic-csi.iscsi/registration doesn’t exist. goroutine 1 [running]: k8s.io/klog/v2.stacks(0x1) /workspace/vendor/k8s.io/klog/v2/klog.go:1038 +0x8a k8s.io/klog/v2.(*loggingT).output(0xf86600, 0x3, 0x0, 0xc000245880, 0x0, {0xc47a41, 0x1}, 0xc0002a8a00, 0x0) /workspace/vendor/k8s.io/klog/v2/klog.go:987 +0x5fd k8s.io/klog/v2.(*loggingT).printf(0xa63799, 0x4, 0x0, {0x0, 0x0}, {0xa8ac8d, 0x48}, {0xc0002a8a00, 0x1, 0x1}) /workspace/vendor/k8s.io/klog/v2/klog.go:753 +0x1c5 k8s.io/klog/v2.Fatalf(…) /workspace/vendor/k8s.io/klog/v2/klog.go:1532 main.main() /workspace/cmd/csi-node-driver-registrar/main.go:159 +0x48e

goroutine 20 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xc000137200) /workspace/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a created by k8s.io/klog/v2.init.0 /workspace/vendor/k8s.io/klog/v2/klog.go:420 +0xfb Normal Pulled 2m41s (x3 over 3m39s) kubelet Container image “registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1” already present on machine Normal Killing 2m41s (x2 over 3m11s) kubelet Container driver-registrar failed liveness probe, will be restarted Warning Unhealthy 2m41s kubelet Liveness probe failed: F0515 17:51:28.279059 38 main.go:159] Kubelet plugin registration hasn’t succeeded yet, file=/var/lib/kubelet/plugins/org.democratic-csi.iscsi/registration doesn’t exist. goroutine 1 [running]: k8s.io/klog/v2.stacks(0x1) /workspace/vendor/k8s.io/klog/v2/klog.go:1038 +0x8a k8s.io/klog/v2.(*loggingT).output(0xf86600, 0x3, 0x0, 0xc0002fccb0, 0x0, {0xc47a41, 0x1}, 0xc00037cd90, 0x0) /workspace/vendor/k8s.io/klog/v2/klog.go:987 +0x5fd k8s.io/klog/v2.(*loggingT).printf(0xa63799, 0x4, 0x0, {0x0, 0x0}, {0xa8ac8d, 0x48}, {0xc00037cd90, 0x1, 0x1}) /workspace/vendor/k8s.io/klog/v2/klog.go:753 +0x1c5 k8s.io/klog/v2.Fatalf(…) /workspace/vendor/k8s.io/klog/v2/klog.go:1532 main.main() /workspace/cmd/csi-node-driver-registrar/main.go:159 +0x48e

goroutine 33 [chan receive]: k8s.io/klog/v2.(*loggingT).flushDaemon(0xc000090060) /workspace/vendor/k8s.io/klog/v2/klog.go:1181 +0x6a created by k8s.io/klog/v2.init.0 /workspace/vendor/k8s.io/klog/v2/klog.go:420 +0xfb

on the node ansible@cent1:~$ ls -l /var/lib/kubelet/plugins/org.democratic-csi.iscsi/ total 0 srwxr-xr-x 1 root root 0 May 15 17:50 csi.sock srw------- 1 root root 0 May 15 17:50 csi.sock.internal

ansible@cent1:~$ ls -l /etc/iscsi/nodes total 0 ansible@cent1:~$

ansible@cent1:~$ sudo systemctl status open-iscsi ○ open-iscsi.service - Login to default iSCSI targets Loaded: loaded (/lib/systemd/system/open-iscsi.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Mon 2023-05-15 16:37:40 UTC; 1h 19min ago ├─ ConditionDirectoryNotEmpty=|/etc/iscsi/nodes was not met └─ ConditionDirectoryNotEmpty=|/sys/class/iscsi_session was not met Docs: man:iscsiadm(8) man:iscsid(8)

May 15 13:45:49 cent1 systemd[1]: Condition check resulted in Login to default iSCSI targets being skipped. May 15 16:30:37 cent1 systemd[1]: Condition check resulted in Login to default iSCSI targets being skipped. May 15 16:31:46 cent1 systemd[1]: Condition check resulted in Login to default iSCSI targets being skipped. May 15 16:37:40 cent1 systemd[1]: Condition check resulted in Login to default iSCSI targets being skipped.