rook: Applying cluster.yaml on v1.13.8: failed calling webhook "cephcluster-wh-rook-ceph-admission-controller-rook-ceph.rook.io": connect: connection refused

I’ve upgraded rook from v1.10.11 to v1.13.8 step by step (v1.10.11 -> v1.11.11 -> v1.12.11 -> v1.13.8). On https://rook.github.io/docs/rook/v1.13/Upgrade/rook-upgrade/ I’ve read that the admission controller is gone (which was enabled in my setup by ROOK_DISABLE_ADMISSION_CONTROLLER: "false"). So I changed this to ROOK_DISABLE_ADMISSION_CONTROLLER: "true" when still running v1.12.11. Upgrade to v1.13.8 went smoothly. Now I want to upgrade to Reef and try to apply the cluster.yaml. But this gives me:

rook $ kaf 04-cluster-prod.yaml
Error from server (InternalError): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"ceph.rook.io/v1\",\"kind\":\"CephCluster\",\"metadata\":{\"annotations\":{},\"name\":\"rook-ceph\",\"namespace\":\"rook-ceph\"},\"spec\":{\"annotations\":null,\"cephVersion\":{\"allowUnsupported\":false,\"image\":\"quay.io/ceph/ceph:v18.2.2\"},\"cleanupPolicy\":{\"allowUninstallWithVolumes\":false,\"confirmation\":\"\",\"sanitizeDisks\":{\"dataSource\":\"zero\",\"iteration\":1,\"method\":\"quick\"}},\"continueUpgradeAfterChecksEvenIfNotHealthy\":false,\"crashCollector\":{\"disable\":false},\"csi\":{\"cephfs\":null,\"readAffinity\":{\"enabled\":false}},\"dashboard\":{\"enabled\":true,\"ssl\":true},\"dataDirHostPath\":\"/var/lib/rook\",\"disruptionManagement\":{\"managePodBudgets\":true,\"osdMaintenanceTimeout\":30,\"pgHealthCheckTimeout\":0},\"healthCheck\":{\"daemonHealth\":{\"mon\":{\"disabled\":false,\"interval\":\"45s\"},\"osd\":{\"disabled\":false,\"interval\":\"60s\"},\"status\":{\"disabled\":false,\"interval\":\"60s\"}},\"livenessProbe\":{\"mgr\":{\"disabled\":false},\"mon\":{\"disabled\":false},\"osd\":{\"disabled\":false}},\"startupProbe\":{\"mgr\":{\"disabled\":false},\"mon\":{\"disabled\":false},\"osd\":{\"disabled\":false}}},\"labels\":null,\"logCollector\":{\"enabled\":true,\"maxLogSize\":\"500M\",\"periodicity\":\"daily\"},\"mgr\":{\"allowMultiplePerNode\":true,\"count\":2,\"modules\":null},\"mon\":{\"allowMultiplePerNode\":true,\"count\":3},\"monitoring\":{\"enabled\":false,\"metricsDisabled\":false},\"network\":{\"connections\":{\"compression\":{\"enabled\":false},\"encryption\":{\"enabled\":false},\"requireMsgr2\":false}},\"priorityClassNames\":{\"mgr\":\"system-cluster-critical\",\"mon\":\"system-node-critical\",\"osd\":\"system-node-critical\"},\"removeOSDsIfOutAndSafeToRemove\":false,\"resources\":null,\"skipUpgradeChecks\":false,\"storage\":{\"config\":null,\"nodes\":[{\"devices\":[{\"config\":{\"osdsPerDevice\":\"4\"},\"name\":\"nvme0n1\"},{\"config\":{\"osdsPerDevice\":\"4\"},\"name\":\"nvme1n1\"},{\"config\":{\"osdsPerDevice\":\"4\"},\"name\":\"nvme3n1\"}],\"name\":\"storage1.<redacted>\"},{\"devices\":[{\"config\":{\"osdsPerDevice\":\"4\"},\"name\":\"nvme0n1\"},{\"config\":{\"osdsPerDevice\":\"4\"},\"name\":\"nvme2n1\"},{\"config\":{\"osdsPerDevice\":\"4\"},\"name\":\"nvme3n1\"}],\"name\":\"storage2.<redacted>\"}],\"onlyApplyOSDPlacement\":false,\"useAllDevices\":false,\"useAllNodes\":false},\"waitTimeoutForHealthyOSDInMinutes\":10}}\n"}},"spec":{"cephVersion":{"image":"quay.io/ceph/ceph:v18.2.2"},"csi":{"cephfs":null,"readAffinity":{"enabled":false}},"mgr":{"modules":null}}}
to:
Resource: "ceph.rook.io/v1, Resource=cephclusters", GroupVersionKind: "ceph.rook.io/v1, Kind=CephCluster"
Name: "rook-ceph", Namespace: "rook-ceph"
for: "04-cluster-prod.yaml": error when patching "04-cluster-prod.yaml": Internal error occurred: failed calling webhook "cephcluster-wh-rook-ceph-admission-controller-rook-ceph.rook.io": failed to call webhook: Post "https://rook-ceph-admission-controller.rook-ceph.svc:443/validate-ceph-rook-io-v1-cephcluster?timeout=5s": dial tcp 10.99.221.127:443: connect: connection refused

Environment:

  • OS (e.g. from /etc/os-release): Ubuntu 20.04.6 LTS (Focal Fossa)
  • Kernel (e.g. uname -a): 5.15.0-105-generic
  • Cloud provider or hardware configuration:
  • Rook version (use rook version inside of a Rook Pod): v1.13.8
  • Storage backend version (e.g. for ceph do ceph -v): ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
  • Kubernetes version (use kubectl version): v1.29.2
  • Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
  • Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): HEALTH_OK

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 23 (11 by maintainers)

Most upvoted comments

Alright. I’m not into Go but I’ll figure it out. Thank you for your help!

Thank you. Take your free time! I’m not really blocked.

$ kubectl get validatingwebhookconfigurations -A
NAME                            WEBHOOKS   AGE
cert-manager-webhook            1          3y116d
ingress-nginx-admission         1          432d
metallb-webhook-configuration   7          432d
rook-ceph-webhook               5          2y3d

I see the issue you need to delete the rook-ceph-webhook (I forgot that webhooks are cluster based resouce) also here is the code https://github.com/rook/rook/blob/b32948c314d64f6b48e40f32d5df656b33d921d1/pkg/operator/ceph/webhook-config.go#L258-L282 that delete everything related to webhook in rook

https://github.com/rook/rook/blob/release-1.12/deploy/examples/operator.yaml#L509 it was there till 1.12 and in 1.13 we removed it.

validating webhook rook-ceph-webhook this is not a pod, this is kubernetes resource, try kubectl get validatingwebhook rook-ceph-webhook*