rook: Unable to delete `Retain`ed Ceph OBC

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior: When deleting an OBC the operator reports network connection errors to the rgw

2020-04-29 17:26:02.640976 E | op-bucket-prov: error getting policy for bucket "cmt-bucket-75ec3d7b-baec-4eec-b02d-31438723e038". RequestError: send request failed                                               
caused by: Get http://rook-ceph-rgw-cmt-ceph-object-store.rook-ceph:0/cmt-bucket-75ec3d7b-baec-4eec-b02d-31438723e038?policy=: dial tcp 10.3.55.80:0: connect: network is unreachable                             

E0429 17:26:02.641027       6 controller.go:190] error syncing 'rook-ceph/cmt-bucket': provisioner error revoking access to bucket RequestError: send request failed                                              
caused by: Get http://rook-ceph-rgw-cmt-ceph-object-store.rook-ceph:0/cmt-bucket-75ec3d7b-baec-4eec-b02d-31438723e038?policy=: dial tcp 10.3.55.80:0: connect: network is unreachable, requeuing

The connection is being attempted on port 0 instead of the port defined in the rgw service manifest, which is 80.

Expected behavior: The OBC is deleted and cleaned up.

How to reproduce it (minimal and precise): Create an OBC that has the StorageClass with a reclaim policy of Retain then try to delete it. Observe the errors. An OBC with a reclaim policy of Delete is removed correctly.

File(s) to submit: Content of the operator created rgw service:

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2020-04-29T17:21:41Z"
  labels:
    app: rook-ceph-rgw
    ceph_daemon_id: cmt-ceph-object-store
    rgw: cmt-ceph-object-store
    rook_cluster: rook-ceph
    rook_object_store: cmt-ceph-object-store
  name: rook-ceph-rgw-cmt-ceph-object-store
  namespace: rook-ceph
  ownerReferences:
  - apiVersion: ceph.rook.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: CephObjectStore
    name: cmt-ceph-object-store
    uid: d20d68aa-7331-418c-b19f-c54597a796e1
  resourceVersion: "160193732"
  selfLink: /api/v1/namespaces/rook-ceph/services/rook-ceph-rgw-cmt-ceph-object-store
  uid: 6b9c7cdf-bf1e-4ff0-b61d-429a2454dfc3
spec:
  clusterIP: 10.3.55.80
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: rook-ceph-rgw
    ceph_daemon_id: cmt-ceph-object-store
    rgw: cmt-ceph-object-store
    rook_cluster: rook-ceph
    rook_object_store: cmt-ceph-object-store
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

storageclass.yml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-retain-bucket
provisioner: ceph.rook.io/bucket
reclaimPolicy: Retain
parameters:
  objectStoreName: cmt-ceph-object-store
  objectStoreNamespace: rook-ceph

obc.yml

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: test-retain-bucket
  namespace: rook-ceph
spec:
  generateBucketName: cmt-bucket
  storageClassName: rook-ceph-retain-bucket

Environment:

• Rook version (use rook version inside of a Rook Pod): 1.3.2
• Storage backend version (e.g. for ceph do ceph -v): ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)
• Kubernetes version (use kubectl version): 1.17.3