democratic-csi: Unable to resize disk image to requested size (Ext4)

Yeah, excellent summary!

For configuring the driver to use 0% reserve you can try the following (helm values syntax):

driver:
  config:
    node:
      format:
        ext4:
          customOptions:
          - -m
          - 0

I would argue that this should be the default for ext4 in a containerized world, not having that set by default leads to the issue we figured out here.

I think a call is in order. Shout out to my github profile email and we’ll get something arranged.

Can you send the logs from the csi-driver container on a node pod on a relevant node (where the migration pod/process is running from)? My thinking is this (I think you are on to something with the filesystem calculation theory):

• migration agent is determining the size needed to migrate a VM and picks a size
• the size is being requested in the PVC creation process and democratic-csi is creating the volume as requested (adding padding to the size of the zvol as necessary)
• when the volume is actually put to use it’s formatted (assuming filesystem access type) with the side effect of ‘losing’ some available bytes (as reported by the node process vs the response from the controller)
• the importer is using the values reported by the node process based on the filesystem data not driven by the size of the volume as reported by the PV

There’s a lot of supposition here without a better understanding on my part about how the importer works and what it’s doing. If I had to venture a guess when working in filesystem mode, a vmdk/qcow images is being exported from esxi and then copied to the ext4/xfs/whatever filesystem of the PV/PVC. When it tries to resize the virtual disk it realizes there’s not enough space (on the filesystem) to handle that and therefore pukes. Again, just a crude guess at this point.

Another guess, if you run it using raw block access mode vs filesystem it would work (no fs overhead involved). Of note here is that when using block access mode I only return the total size and not the available size as I have no way to know when using raw bits like that (the size attribute of the block device from the lsblk -a -b -l -J -O <dev> command). For filesystem access mode I return relevant values as reported by the findmnt command (ie: filesystem data, not block device data) and additionally in certain circumstances (when available) also inode stats (df -i) in addition to byte stats.

Basically what I’m looking for in the node logs are the GetVolumeStats responses to see if the response values are corresponding to the errors in your screenshots.

The sizes of the zvols in the storage system (without fs overhead) should be:

{"host":"zfs-iscsi-democratic-csi-controller-b978479b6-csb2f","level":"info","message":"new request - driver: FreeNASSshDriver method: CreateVolume call: {\"metadata\":{\"user-agent\":[\"grpc-go/1.40.0\"],\"x-forwarded-host\":[\"localhost\"]},\"request\":{\"volume_capabilities\":[{\"access_mode\":{\"mode\":\"SINGLE_NODE_MULTI_WRITER\"},\"mount\":{\"mount_flags\":[],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"}],\"parameters\":{\"fsType\":\"ext4\",\"csi.storage.k8s.io/pvc/name\":\"bench2-vm-85624-v68jt\",\"csi.storage.k8s.io/pvc/namespace\":\"benchmark\",\"csi.storage.k8s.io/pv/name\":\"pvc-1d0b051c-fced-46ef-bb7a-1a99de64b129\"},\"secrets\":\"redacted\",\"name\":\"pvc-1d0b051c-fced-46ef-bb7a-1a99de64b129\",\"capacity_range\":{\"required_bytes\":\"72719023002\",\"limit_bytes\":\"0\"},\"volume_content_source\":null,\"accessibility_requirements\":null},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-10-17T02:05:20.268Z"}
{"host":"zfs-iscsi-democratic-csi-controller-b978479b6-csb2f","level":"info","message":"new response - driver: FreeNASSshDriver method: CreateVolume response: {\"volume\":{\"volume_id\":\"pvc-1d0b051c-fced-46ef-bb7a-1a99de64b129\",\"capacity_bytes\":72719024128,\"content_source\":null,\"volume_context\":{\"node_attach_driver\":\"iscsi\",\"portal\":\"172.16.1.120:3260\",\"portals\":\"\",\"interface\":\"\",\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-pvc-1d0b051c-fced-46ef-bb7a-1a99de64b129-cluster\",\"lun\":0,\"provisioner_driver\":\"freenas-iscsi\"}}}","service":"democratic-csi","timestamp":"2022-10-17T02:05:22.311Z"}

{"host":"zfs-iscsi-democratic-csi-controller-7cc68f9df-lsj8p","level":"info","message":"new request - driver: FreeNASSshDriver method: CreateVolume call: {\"metadata\":{\"x-forwarded-host\":[\"localhost\"],\"user-agent\":[\"grpc-go/1.40.0\"]},\"request\":{\"volume_capabilities\":[{\"access_mode\":{\"mode\":\"SINGLE_NODE_MULTI_WRITER\"},\"mount\":{\"mount_flags\":[],\"fs_type\":\"ext4\",\"volume_mount_group\":\"\"},\"access_type\":\"mount\"}],\"parameters\":{\"fsType\":\"ext4\",\"csi.storage.k8s.io/pvc/name\":\"bench3-vm-85624-j9l5k\",\"csi.storage.k8s.io/pvc/namespace\":\"benchmark\",\"csi.storage.k8s.io/pv/name\":\"pvc-99005402-b663-4189-bb96-5e682af92ea7\"},\"secrets\":\"redacted\",\"name\":\"pvc-99005402-b663-4189-bb96-5e682af92ea7\",\"capacity_range\":{\"required_bytes\":\"72719023002\",\"limit_bytes\":\"0\"},\"volume_content_source\":null,\"accessibility_requirements\":null},\"cancelled\":false}","service":"democratic-csi","timestamp":"2022-10-17T02:53:59.571Z"}
{"host":"zfs-iscsi-democratic-csi-controller-7cc68f9df-lsj8p","level":"info","message":"new response - driver: FreeNASSshDriver method: CreateVolume response: {\"volume\":{\"volume_id\":\"pvc-99005402-b663-4189-bb96-5e682af92ea7\",\"capacity_bytes\":72719024128,\"content_source\":null,\"volume_context\":{\"node_attach_driver\":\"iscsi\",\"portal\":\"172.16.1.119:3260\",\"portals\":\"\",\"interface\":\"\",\"iqn\":\"iqn.2005-10.org.freenas.ctl:csi-pvc-99005402-b663-4189-bb96-5e682af92ea7-cluster\",\"lun\":0,\"provisioner_driver\":\"freenas-iscsi\"}}}","service":"democratic-csi","timestamp":"2022-10-17T02:54:00.871Z"}

In short, I’m guessing it’s a discrepancy between the importer calculations for needed size, raw block/zvol size, and resulting fs size. Or said differently, the size for the requested PV/PVC by the importer may need to account for fs overhead when using filesystem access mode.

I am happy to arrange a call but I’m in Mountain Time USA and prefer to have it at a more ‘normal’ hour when I’m at my office 😃