metallb: webhook 500 errors

environment

MetalLB v0.13.2 Kubernetes v1.24.2 CNI cilium v1.11.6 kube-proxy not deployed

The bug itself, as detailed as you can.

After the initial install it took multiple attempts to apply the CR’s to the cluster. The validating webhook was returning error 500 on the attempts and then finally accepted them. This greatly affects clusters managed through GitOps as reconciliations are failing.

k apply -f cluster/apps/networking/metallb/custom-resources/bgp-advertisement.yaml

Warning: resource bgpadvertisements/bgp-lb is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
Error from server (InternalError): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"metallb.io/v1beta1\",\"kind\":\"BGPAdvertisement\",\"metadata\":{\"annotations\":{},\"name\":\"bgp-lb\",\"namespace\":\"networking\"},\"spec\":{\"ipAddressPools\":[\"bgp-pool\"],\"peers\":[\"mikrotik\"]}}\n"}}}
to:
Resource: "metallb.io/v1beta1, Resource=bgpadvertisements", GroupVersionKind: "metallb.io/v1beta1, Kind=BGPAdvertisement"
Name: "bgp-lb", Namespace: "networking"
for: "cluster/apps/networking/metallb/custom-resources/bgp-advertisement.yaml": Internal error occurred: failed calling webhook "bgpadvertisementvalidationwebhook.metallb.io": failed to call webhook: Post "https://metallb-webhook-service.networking.svc:443/validate-metallb-io-v1beta1-bgpadvertisement?timeout=10s": proxy error from 127.0.0.1:6443 while dialing 172.20.0.186:9443, code 500: 500 Internal Server Error

controller errors with info logging

{"level":"error","ts":1657129516.5686264,"msg":"Reconciler error","controller":"cert-rotator","object":{"name":"webhook-server-cert","namespace":"networking"},"namespace":"networking","name":"webhook-server-cert","reconcileID":"1a97259f-b252-4af3-8ac0-c727359e2278","error":"Operation cannot be fulfilled on customresourcedefinitions.apiextensions.k8s.io \"bgppeers.metallb.io\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:234"}

controller errors with debug logging

{"action":"update","caller":"bgpadvertisement_webhook.go:70","level":"debug","name":"bgp-lb","namespace":"networking","ts":"2022-07-06T19:58:51Z","webhook":"bgpadvertisement"}
{"action":"update","caller":"bgppeer_webhook.go:60","level":"debug","name":"mikrotik","namespace":"networking","ts":"2022-07-06T19:58:51Z","webhook":"bgppeer"}
{"action":"update","caller":"ipaddresspool_webhook.go:64","level":"debug","name":"bgp-pool","namespace":"networking","ts":"2022-07-06T19:58:51Z","webhook":"ipaddresspool"}

logs flooded with this community error

E0706 19:58:53.147844       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.Community: unknown (get communities.metallb.io)
{"action":"update","caller":"bgppeer_webhook.go:60","level":"debug","name":"mikrotik","namespace":"networking","ts":"2022-07-06T19:59:15Z","webhook":"bgppeer"}
E0706 19:59:40.601150       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.Community: unknown (get communities.metallb.io)
E0706 20:00:14.243217       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.Community: unknown (get communities.metallb.io)
E0706 20:00:50.974510       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.Community: unknown (get communities.metallb.io)