rook: Bug: Disk usage not reclaimed when files are deleted

I have fresh installation of rook version 1.3.5 (tag release-1.3) and I had the same issue with mount options on rbd

Rook version

[root@rook-ceph-operator-868bf9558-jzd4v /]# rook version
rook: v1.3.5

rook-ceph-operator-config , i have used both ROOK_CSI_CEPH_IMAGE v2.1.0 and last v2.1.2

apiVersion: v1
data:
  CSI_ENABLE_SNAPSHOTTER: "true"
  CSI_FORCE_CEPHFS_KERNEL_CLIENT: "true"
  ROOK_CSI_ALLOW_UNSUPPORTED_VERSION: "false"
  ROOK_CSI_ATTACHER_IMAGE: quay.io/k8scsi/csi-attacher:v2.2.0
  ROOK_CSI_CEPH_IMAGE: quay.io/cephcsi/cephcsi:v2.1.0
  ROOK_CSI_ENABLE_CEPHFS: "true"
  ROOK_CSI_ENABLE_GRPC_METRICS: "true"
  ROOK_CSI_ENABLE_RBD: "true"
  ROOK_CSI_PROVISIONER_IMAGE: quay.io/k8scsi/csi-provisioner:v1.6.0
  ROOK_CSI_REGISTRAR_IMAGE: quay.io/k8scsi/csi-node-driver-registrar:v1.3.0
  ROOK_CSI_RESIZER_IMAGE: quay.io/k8scsi/csi-resizer:v0.5.0
  ROOK_CSI_SNAPSHOTTER_IMAGE: quay.io/k8scsi/csi-snapshotter:v1.2.2
  ROOK_OBC_WATCH_OPERATOR_NAMESPACE: "true"
kind: ConfigMap
metadata:
  creationTimestamp: "2020-07-24T07:26:43Z"
  name: rook-ceph-operator-config
  namespace: rook-ceph
  resourceVersion: "97681505"
  selfLink: /api/v1/namespaces/rook-ceph/configmaps/rook-ceph-operator-config
  uid: 05f05fa8-cd7f-11ea-8970-525400d9280f

My storage class

storageclasses.storage.k8s.io rook-ceph-block -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2020-07-24T13:15:13Z"
  name: rook-ceph-block
  resourceVersion: "97682453"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/rook-ceph-block
  uid: b528eb01-cdaf-11ea-8970-525400d9280f
mountOptions:
- discard
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  imageFeatures: layering
  imageFormat: "2"
  pool: replicapool
provisioner: rook-ceph.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

Pod and pvc

---
apiVersion: v1
kind: Pod
metadata:
  name: csirbd-demo-pod
spec:
  containers:
   - name: web-server
     image: nginx
     volumeMounts:
       - name: mypvc
         mountPath: /var/lib/www/html
  volumes:
   - name: mypvc
     persistentVolumeClaim:
       claimName: rbd-pvc
       readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-ceph-block

Inside pod

root@csirbd-demo-pod:/# mount | grep rbd
/dev/rbd0 on /var/lib/www/html type ext4 (rw,relatime,stripe=16)
root@csirbd-demo-pod:/# df -h /var/lib/www/html/
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd0       976M  2.6M  958M   1% /var/lib/www/html
root@csirbd-demo-pod:/# fstrim /var/lib/www/html/
fstrim: /var/lib/www/html/: FITRIM ioctl failed: Operation not permitted
root@csirbd-demo-pod:/# 

mountOptions - has no effect

@Madhu-1

Thanks for your response @mywkaa. I am running okd 4.5 cluster so k8s 1.18.3. I have finally upgraded to rook 1.4.4 which also upgraded ceph csi to 3.x. Also I have recreated my storage class with discard option as fstrim was not working. With all above I can see osd space is freeing up. Just not sure yet of performance impact of mounting with discard option.

This is expected. Ceph images are sparse, so if you create file storage increases. If you want to reclaim space try the fstrim command or mount the fs with discard (we might decide to do this). I think ceph-csi as fs options you can use while creating your storageclass, try to use mount_options and set discard.

@leseb I’ve just tried this on rook v1.1.0 on a cephfs StorageClass with the following mountOptions and it did successfully reclaim the space.

mountOptions:
- discard

However, I prefer rbd to cephfs so tried the same mountOptions for a rbd StorageClass but it did not work. Are you aware of any way to workaround this at the moment for rbd? Or any plan to support rbd reclaiming soon?

Same problems here on my test cluster, rook 1.3.11, ceph 15.2.5, csi 2.1.2.

I tried fstrim from host machine, which runs correctly and even said: 320.6 GiB (344195588096 bytes) trimmed However OSD is still showing as full- no change in space is reflected.

What are we missing?

What version of k8s are you using ? For me helped update to 1.18

@Madhu-1 @travisn I just checked the StorageClass YAMLs in rook’s repo and I have not seen the mountOptions (or similar) field in them.

I think we should add the mounOptions with discard flag as an example commented out, though I’m strongly inclined to have mountOptions: ["discard"] enabled by default on the example StorageClasses.

i was thinking its already there but it’s not present.Yes, we can add it.

You can specify mout option in storageclass, discard is just an example

https://github.com/ceph/ceph-csi/pull/848 should fix the issue for rbd.

Ping @Madhu-1 has this been addressed in the latest Ceph CSI release?