ceph-csi: PVC hangs in "Pending" status

Describe the bug

There are no meaningful error messages available. Just trying to deploy csi-rbd-provisioner.

Environment details

  • Image/version of Ceph CSI driver : https://github.com/ceph/ceph-csi/tree/master/deploy/rbd/kubernetes as is.
  • Helm chart version : -
  • Kernel version : 3.10.0-862.el7.x86_64
  • Mounter used for mounting PVC (for cephfs its fuse or kernel. for rbd its krbd or rbd-nbd) : -
  • Kubernetes cluster version : v1.18.2
  • Ceph cluster version : 14.2.11

Steps to reproduce

Steps are described here: https://docs.ceph.com/docs/master/rbd/rbd-kubernetes/

Actual results

PVC hangs in ‘Pending’ state

Expected behavior

Provisioned volume

Logs

If the issue is in PVC creation, deletion, cloning please attach complete logs of below containers.

  • csi-provisioner and csi-rbdplugin/csi-cephfsplugin container logs from the provisioner pod.

csi-provisioner

I0829 23:16:16.808022       1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume
I0829 23:16:16.808034       1 connection.go:183] GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-9fee2a2f-6c88-4ee4-9a67-393d05a4992b","parameters":{"clusterID":"<cluster_id>","pool":"k8s-csi-rbd"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Block":{}},"access_mode":{"mode":1}}]}
I0829 23:16:16.813020       1 connection.go:185] GRPC response: {}
I0829 23:16:16.813415       1 connection.go:186] GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-9fee2a2f-6c88-4ee4-9a67-393d05a4992b already exists
I0829 23:16:16.813452       1 controller.go:634] CreateVolume failed, supports topology = true, node selected false => may reschedule = false => state = Background: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-9fee2a2f-6c88-4ee4-9a67-393d05a4992b already exists

csi-rbdplugin

1 utils.go:163] ID: 8255 Req-ID: pvc-7273569d-541d-4b82-b9cc-9a194caa0d22 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-7273569d-541d-4b82-b9cc-9a194caa0d22 already exists

Additional context

Network connectivity is fine, native k8s provisioner works just fine.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 16 (2 by maintainers)

Most upvoted comments

Access to 3300 solved the issue, however, I’d like to see such things in the logs. Thank you, guys!

as ceph, as not returned any error we cannot log such things in cephcsi. As its working now closing this one