operator: Webhooks get context deadline exceeded

Pulling this in from Slack for tracking - After some time I’ve been unable to reliably resolve this. I am running the VM Operator on 6+ cluster but only getting this frequently on a few clusters, and I could not identify and correlation as to why.

After upgrading to v0.47.3 (helm chart 0.34.7) using the helm chart, I’m getting these errors from server-side apply:

Internal error occurred: failed calling webhook "vmagent.victoriametrics.com": failed to call webhook: Post "https://victoria-metrics-operator.monitoring.svc/validate-operator-victoriametrics-com-v1beta1-vmagent?timeout=10s": context deadline exceeded

I tried disabling the vmagent webhook, and then I started to get the same error with other resources like vmalert. There are no errors in the logs that seem directly related.

I’ve been able to confirm that we can connect to the service, it just seems to not respond to the webhook posts.

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 23 (8 by maintainers)

Commits related to this issue

Most upvoted comments

Thanks for detailed investigation!

It’s related to VictoriaMetrics issue https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6911

As an update, I haven’t seen any more alerts of dry run apply on VMAgent and other resources causing failed webhooks. The process that seems to work is to:

  • delete the existing validating webhook config
  • perform the helm upgrade

The bugfix was included into v0.54.0 release

However, issue should remain open until it will be fixed on golang

I think, we should remove relabeling configs validation from webhook checks until the issue will be fixed at golang.

I’m confused. @AndrewChubatiuk @f41gh7 does operator just parses those regexps from relabeling configs or does some matching?

Operator performs logic validation for relabel configs rules (aka dry-run). It reuses vmagent config parsing libs.