democratic-csi: zfs-generic-iscsi | need clarification | problem targetcli + targets
Hello,
Im seeking your help regarding the setup of democratic-csi using zfs-generic-iscsi.
I notice that every time I provision a disk, this one create a new iscsi device in targetcli on the data server and this one fails to find the target when a pod tries to mount it.
on the DS Server ( Ubuntu 22.02 , using LVM + zfs pool + dataset ) I created a LV that I then used with zfs to create a pool and dataset :
root@ds:~# pvs
PV VG Fmt Attr PSize PFree
/dev/mapper/datavol data_vg lvm2 a-- 930.48g 0
root@ds:~# vgs
VG #PV #LV #SN Attr VSize VFree
data_vg 1 1 0 wz--n- 930.48g 0
root@ds:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data_lv data_vg twi-aot--- <930.25g 0.00 10.41
root@ds:~# zpool status
pool: tank
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
data_lv ONLINE 0 0 0
errors: No known data errors
root@ds:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 2.81M 899G 96K none
tank/k8s 288K 899G 96K none
tank/k8s/s 96K 899G 96K none
tank/k8s/v 96K 899G 96K none
Then I configured targetcli :
root@ds:~# targetcli
targetcli shell version 2.1.58
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> /iscsi create iqn.2023-03.com.ltd:lun
/> /iscsi/iqn.2023-03.com.ltd:lun/tpg1 set attribute authentication=1 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1
/> /iscsi/iqn.2023-03.com.ltd:lun/tpg1/acls create iqn.2023-03.com.ltd:client
/> /iscsi/iqn.2023-03.com.ltd:lun/tpg1/acls/iqn.2023-03.com.ltd:client set auth userid=k8s
/> /iscsi/iqn.2023-03.com.ltd:lun/tpg1/acls/iqn.2023-03.com.ltd:client set auth password=passtest
/> /iscsi/iqn.2023-03.com.ltd:lun/tpg1/acls/iqn.2023-03.com.ltd:client info
chap_password: passtest
chap_userid: k8s
wwns:
iqn.2023-03.com.ltd:client
/> cd /iscsi/iqn.2023-03.com.ltd:lun/tpg1/portals/
/iscsi/iqn.20.../tpg1/portals> ls
o- portals ............................................................................................................ [Portals: 1]
o- 0.0.0.0:3260 ............................................................................................................. [OK]
/iscsi/iqn.20.../tpg1/portals> delete ip_address=0.0.0.0 ip_port=3260
Deleted network portal 0.0.0.0:3260
/iscsi/iqn.20.../tpg1/portals> ls
o- portals ............................................................................................................ [Portals: 0]
/iscsi/iqn.20.../tpg1/portals> create ip_address=192.168.1.12 ip_port=3260
Using default IP port 3260
Created network portal 192.168.1.12:3260.
/iscsi/iqn.20.../tpg1/portals> ls
o- portals ............................................................................................................ [Portals: 1]
o- 192.168.1.12:3260 ........................................................................................................ [OK]
/iscsi/iqn.20.../tpg1/portals> cd /
/> ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 0]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 1]
| o- iqn.2023-03.com.ltd:lun ........................................................................................... [TPGs: 1]
| o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl]
| o- acls .......................................................................................................... [ACLs: 1]
| | o- iqn.2023-03.com.ltd:client ............................................................... [1-way auth, Mapped LUNs: 0]
| o- luns .......................................................................................................... [LUNs: 0]
| o- portals .................................................................................................... [Portals: 1]
| o- 192.168.1.12:3260 ................................................................................................ [OK]
o- loopback ......................................................................................................... [Targets: 0]
o- vhost ............................................................................................................ [Targets: 0]
o- xen-pvscsi ....................................................................................................... [Targets: 0]
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup/.
Configuration saved to /etc/target/saveconfig.json
Then I configured the k8s node by installing the package mentioned in the README ( restarted them , including the DS server )
Then I deployed with helm democratic-csi (and the snapshot controller) with the following parameters (zfs-generic-iscsi.yaml). I use root with ssh for the simplicity until I make it work, then I will work on sudo for security reason.
root@7a974fbef4e2:~# cat zfs-generic-iscsi.yaml
csiDriver:
# should be globally unique for a given cluster
name: "org.democratic-csi.iscsi"
storageClasses:
- name: zfs-generic-iscsi
defaultClass: false
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
# for block-based storage can be ext3, ext4, xfs
# for nfs should be nfs
fsType: ext4
secrets:
provisioner-secret:
controller-publish-secret:
node-stage-secret:
node-db.node.session.auth.authmethod: CHAP
node-db.node.session.auth.username: k8s
node-db.node.session.auth.password: passtest
# if true, volumes created from other snapshots will be
# zfs send/received instead of zfs cloned
# detachedVolumesFromSnapshots: "false"
# if true, volumes created from other volumes will be
# zfs send/received instead of zfs cloned
# detachedVolumesFromVolumes: "false"
volumeSnapshotClasses:
- name: zfs-generic-iscsi
# parameters:
# # if true, snapshots will be created with zfs send/receive
detachedSnapshots: "false"
# secrets:
# snapshotter-secret:
controller:
driver:
logLevel: debug
node:
driver:
logLevel: debug
driver:
config:
driver: zfs-generic-iscsi
sshConnection:
host: 192.168.1.12
port: 22
username: root
# use either password or key
password: "*********"
zfs:
#cli:
#sudoEnabled: true
datasetParentName: tank/k8s/v
detachedSnapshotsDatasetParentName: tank/k8s/s
zvolCompression:
zvolDedup:
zvolEnableReservation: false
zvolBlocksize:
iscsi:
targetPortal: "192.168.1.12:3260"
targetPortals: [192.168.1.12:3260"]
#targetPortals: []
# leave empty to omit usage of -I with iscsiadm
interface:
nameTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
#nameTemplate: "{{ parameters.[csi.storage.k8s.io/pvc/namespace] }}-{{ parameters.[csi.storage.k8s.io/pvc/name] }}"
#namePrefix: "csi-zfs-"
#nameSuffix: "-cluster"
shareStrategy: "targetCli"
shareStrategyTargetCli:
#sudoEnabled: true
basename: "iqn.2023-03.com.ltd:client"
tpg:
attributes:
# set to 1 to enable CHAP
authentication: 1
# this is required currently as we do not register all node iqns
# the effective outcome of this is, allow all iqns to connect
generate_node_acls: 1
cache_dynamic_acls: 1
# if generate_node_acls is 1 then must turn this off as well (assuming you want write ability)
demo_mode_write_protect: 0
auth:
# CHAP
userid: "k8s"
password: "passtest"
# mutual CHAP
#mutual_userid: "baz"
#mutual_password: "bar"
block:
attributes:
# set to 1 to enable Thin Provisioning Unmap
emulate_tpu: 1
All is good so far, all pods are up and logs of the csi-driver a clean ( no error at this point ).
my problems come when I provision / create disks from the k8s :
the disk resource deployed on K8s :
root@7a974fbef4e2:~# cat disk-perf-test-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-disk-pvc
spec:
storageClassName: zfs-generic-iscsi
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi # Adjust the size as needed
disk is provisonned succesfully :
root@7a974fbef4e2:~# kubectl get pvc -n democratic-csi
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
test-disk-pvc Bound pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 10Gi RWO zfs-generic-iscsi <unset> 48s
from k8s events :
democratic-csi 0s Normal ExternalProvisioning PersistentVolumeClaim/test-disk-pvc Waiting for a volume to be created either by the external provisioner 'org.democratic-csi.iscsi' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
democratic-csi 0s Normal Provisioning PersistentVolumeClaim/test-disk-pvc External provisioner is provisioning volume for claim "democratic-csi/test-disk-pvc"
democratic-csi 0s Normal ProvisioningSucceeded PersistentVolumeClaim/test-disk-pvc Successfully provisioned volume pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38
Then, when I go back on the DS server, I can see in targetcli that a new iscsi iqn.2023-03.com.ltd:client:test-disk-pvc device was created :by the CSI and its where I start to be confuse …
root@ds:~# targetcli ls
o- / ......................................................................................................................... [...]
o- backstores .............................................................................................................. [...]
| o- block .................................................................................................. [Storage Objects: 1]
| | o- test-disk-pvc .............. [/dev/zvol/tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 (10.0GiB) write-thru activated]
| | o- alua ................................................................................................... [ALUA Groups: 1]
| | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
| o- fileio ................................................................................................. [Storage Objects: 0]
| o- pscsi .................................................................................................. [Storage Objects: 0]
| o- ramdisk ................................................................................................ [Storage Objects: 0]
o- iscsi ............................................................................................................ [Targets: 2]
| o- iqn.2023-03.com.ltd:client:test-disk-pvc .......................................................................... [TPGs: 1]
| | o- tpg1 ..................................................................................... [gen-acls, tpg-auth, 1-way auth]
| | o- acls .......................................................................................................... [ACLs: 0]
| | o- luns .......................................................................................................... [LUNs: 1]
| | | o- lun0 ......... [block/test-disk-pvc (/dev/zvol/tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38) (default_tg_pt_gp)]
| | o- portals .................................................................................................... [Portals: 0]
| o- iqn.2023-03.com.ltd:lun ........................................................................................... [TPGs: 1]
| o- tpg1 ..................................................................................... [gen-acls, tpg-auth, 1-way auth]
| o- acls .......................................................................................................... [ACLs: 1]
| | o- iqn.2023-03.com.ltd:client ............................................................. [auth via tpg, Mapped LUNs: 0]
| o- luns .......................................................................................................... [LUNs: 0]
| o- portals .................................................................................................... [Portals: 1]
| o- 192.168.1.12:3260 ................................................................................................ [OK]
o- loopback ......................................................................................................... [Targets: 0]
o- vhost ............................................................................................................ [Targets: 0]
I can see it in lsblk (zd0) :
root@ds:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 465.3G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 2G 0 part /boot
└─sda3 8:3 0 463.2G 0 part
└─dm_crypt-0 252:0 0 463.2G 0 crypt
└─ubuntu--vg-ubuntu--lv 252:1 0 100G 0 lvm /
sdb 8:16 0 930.5G 0 disk
└─datavol 252:2 0 930.5G 0 crypt
├─data_vg-data_lv_tmeta 252:3 0 120M 0 lvm
│ └─data_vg-data_lv 252:5 0 930.2G 0 lvm
└─data_vg-data_lv_tdata 252:4 0 930.2G 0 lvm
└─data_vg-data_lv 252:5 0 930.2G 0 lvm
sr0 11:0 1 1024M 0 rom
zd0 230:0 0 10G 0 disk
and with zfs :
root@ds:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 2.91M 899G 96K none
tank/k8s 344K 899G 96K none
tank/k8s/s 96K 899G 96K none
tank/k8s/v 152K 899G 96K none
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 56K 899G 56K -
root@ds:~# zfs get all tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38
NAME PROPERTY VALUE SOURCE
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 type volume -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 creation Mon Mar 11 4:35 2024 -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 used 56K -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 available 899G -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 referenced 56K -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 compressratio 1.00x -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 reservation none default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 volsize 10G local
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 volblocksize 16K default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 checksum on default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 compression lz4 inherited from tank
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 readonly off default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 createtxg 18512 -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 copies 1 default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 refreservation none local
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 guid 4625150418645185089 -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 primarycache all default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 secondarycache all default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 usedbysnapshots 0B -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 usedbydataset 56K -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 usedbychildren 0B -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 usedbyrefreservation 0B -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 logbias latency default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 objsetid 282 -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 dedup off default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 mlslabel none default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 sync standard default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 refcompressratio 1.00x -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 written 56K -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 logicalused 28K -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 logicalreferenced 28K -
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 volmode default default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 snapshot_limit none default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 snapshot_count none default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 snapdev hidden default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 context none default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 fscontext none default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 defcontext none default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 rootcontext none default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 redundant_metadata all default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 encryption off default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 keylocation none default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 keyformat none default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 pbkdf2iters 0 default
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 democratic-csi:provision_success true local
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 democratic-csi:csi_share_volume_context {"node_attach_driver":"iscsi","portal":"192.168.1.12:3260","portals":"192.168.1.12:3260\"","interface":"","iqn":"iqn.2023-03.com.ltd:client:test-disk-pvc","lun":0} local
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 democratic-csi:iscsi_assets_name test-disk-pvc local
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 democratic-csi:managed_resource true local
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 democratic-csi:volume_context_provisioner_driver zfs-generic-iscsi local
tank/k8s/v/pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 democratic-csi:csi_volume_name pvc-dbfe2df4-6ddd-4934-9190-ec358bafec38 local
Maybe Im not using zfs the right way, but I would expect to have the block ( test-disk-pvc ) to be added in the lun of the existing iqn that I declared.
The problem is then, when I start a pod, on the DS server it’s complaining that it can not find the target of the newly created PVC:
root@7a974fbef4e2:~# cat disk-perf-test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: disk-perf-test-pod
spec:
containers:
- name: fio-container
image: ubuntu:latest
command: ["/bin/bash", "-c"]
args:
- apt-get update && apt-get install -y fio && fio --name=test --ioengine=sync --rw=randwrite --bs=4k --size=1G --numjobs=4 --time_based --runtime=30s
volumeMounts:
- name: disk-perf-volume
mountPath: /mnt # Adjust the mount path as needed
volumes:
- name: disk-perf-volume
persistentVolumeClaim:
claimName: test-disk-pvc
From journalctl -bxe -f on the DS server :
Mar 11 04:40:57 ds kernel: Unable to locate Target Portal Group on iqn.2023-03.com.ltd:client:test-disk-pvc
Mar 11 04:40:57 ds kernel: iSCSI Login negotiation failed.
Mar 11 04:40:58 ds kernel: CHAP user or password not set for Initiator ACL
Mar 11 04:40:58 ds kernel: Security negotiation failed.
Mar 11 04:40:58 ds kernel: iSCSI Login negotiation failed.
So im confused … I kind of understand the error, I guess it fails to authenticate because the new iscsi path is not configured accordingly but as it’s managed by the csi driver.
I was kind of expecting the block to be assign to the iscsi that I configured with targetcli. I also tried with no auth but Im still getting the error that it can not find iqn.2023-03.com.ltd:client:test-disk-pvc.
Does open-iscsi and iscsid are required on the DS node ?
Im not sure if it’s because I missed something in the config of the driver or in targetcli ?
Thanks
About this issue
- Original URL
- State: closed
- Created 4 months ago
- Comments: 16 (7 by maintainers)
ok.
I did some test with sudo enabled for both (
iscsiandzfs) likesnapshotandresize( im not sure what else I could try ) but it works.I just got an issue with a
hotsnapshot but the source pvc was heavily use so data loss was expected.If I have some time next week I will look at selinux (lol) , I will just leave it in
permissivemode and check what is call, it will be easier to identify what is executed under the hood bycsiuser.Thanks for your help.