test-infra: Test cluster `default` reports `unauthorized` error

Things should be fixed now. It seems that the root cause of this outage was the KES deployment getting stuck on some internal error that resulted in neither the pod crashing nor metrics indicating a failed secret sync (for which we already have an alert).

Please don’t switch everything to the community cluster: We’re still very very tight on GCP budget this year and that cluster has already had capacity issues of late. We don’t want to resolve them by increasing autoscaling capacity due to the tight budget (we’re still on track for at least 3.4M on 3M credits this year and actively working to cut costs).

There is an EKS cluster coming online that workloads could switch to in the near future. Hopefully we’ll have this resolved before then anyhow though.

Ah I see a lot of errors from the kubernetes-external-secrets deployment like the following. I think that could be the explanation for the kubeconfig going stale:

{"level":30,"message_time":"2023-03-13T19:25:54.105Z","pid":18,"hostname":"kubernetes-external-secrets-5f98c9ff97-ngs9k","payload":{},"msg":"starting poller for prow-monitoring/prometheus-alert-slack-post-testing-ops-secret-url"}
{"level":50,"message_time":"2023-03-13T19:25:54.106Z","pid":18,"hostname":"kubernetes-external-secrets-5f98c9ff97-ngs9k","payload":{"err":{"type":"TypeError","message":"Cannot read property 'get' of undefined","stack":"TypeError: Cannot read property 'get' of undefined\n    at Poller._scheduleNextPoll (/app/lib/poller.js:361:30)\n    at Poller.start (/app/lib/poller.js:415:10)\n    at Daemon._addPoller (/app/lib/daemon.js:59:43)\n    at Daemon.start (/app/lib/daemon.js:89:16)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)"}},"msg":"status check went boom for prow-monitoring/prometheus-alert-slack-post-testing-ops-secret-url"}

I’ve kicked over the pod and now it has synced the secret.

This cluster is part of the google infrastructure. I will advise to move the community-owned infrastructure by adding the cluster: k8s-infra-prow-build if this is critical. On-call folx are in PST. Will take some time before an intervention happening.