prometheus-operator: The CustomResourceDefinition "prometheuses.monitoring.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
What happened?
Did you expect to see something different?
How to reproduce it (as minimally and precisely as possible):
Environment
-
Prometheus Operator version:
Insert image tag or Git SHA here -
Kubernetes version information:
kubectl versionClient Version: version.Info{Major:“1”, Minor:“21”, GitVersion:“v1.21.0”, GitCommit:“cb303e613a121a29364f75cc67d3d580833a7479”, GitTreeState:“clean”, BuildDate:“2021-04-08T16:31:21Z”, GoVersion:“go1.16.1”, Compiler:“gc”, Platform:“linux/amd64”} Server Version: version.Info{Major:“1”, Minor:“18+”, GitVersion:“v1.18.20-eks-8c579e”, GitCommit:“8c579edfc914f013ff48b2a2b2c1308fdcacc53f”, GitTreeState:“clean”, BuildDate:“2021-07-31T01:34:13Z”, GoVersion:“go1.13.15”, Compiler:“gc”, Platform:“linux/amd64”} -
Kubernetes cluster kind:
Using EKS Cluster
- Manifests:
Used this guide to install
https://grafana.com/docs/grafana-cloud/quickstart/prometheus_operator/
Bundle yml : kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
- Prometheus Operator Logs:
insert Prometheus Operator logs relevant to the issue here
Anything else we need to know?:
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Comments: 18 (7 by maintainers)
Commits related to this issue
- Replace or create CRDs instead of applying them Noted workaround is described in https://github.com/prometheus-operator/prometheus-operator/issues/4355#issuecomment-957957151. The TLDR for the issue... — committed to aiyengar2/charts by aiyengar2 2 years ago
- Replace or create CRDs instead of applying them Noted workaround is described in https://github.com/prometheus-operator/prometheus-operator/issues/4355#issuecomment-957957151. The TLDR for the issue... — committed to aiyengar2/charts by aiyengar2 2 years ago
- Replace or create CRDs instead of applying them Noted workaround is described in https://github.com/prometheus-operator/prometheus-operator/issues/4355#issuecomment-957957151. The TLDR for the issue... — committed to aiyengar2/charts by aiyengar2 2 years ago
For ArgoCD since 2.5 version there is a solution https://blog.ediri.io/kube-prometheus-stack-and-argocd-25-server-side-apply-to-the-rescue
Can you try with
kubectl create. The issue is due to large annotation size which will be issue forkubectl applysince it stores last applied config.@paulfantom Is using
kubectl createinstead ofkubectl applyright now the best approach? We’re generating our monitoring stack based onkube-prometheusand applying those manifests in every pipeline run, this is now somehow blocked, because we would need to handle that edge-case differently and that just feels bad.It works with
create.It fails with
applyeven if the CRD didn’t previously exist.kubectl applywill storekubernetes.io/last-applied-configurationannotation even if we are creating a new resource 😃This is also important when you use a Delivery tool like argoCD for example.
@guusvw that’s kubernetes restriction (as mentioned in https://github.com/prometheus-operator/prometheus-operator/issues/4355#issuecomment-955899678) and using
kubectl createand/orkubectl replaceto apply CRDs is considered a valid workaround by kubernetes-sigs. FYI tools like argoCD have an internal logic to handle such cases by removing and creating object if needed. However if you use onlykubectlyou need to put that logic into your tooling. Something likekubectl replace -f crds/ || kubectl create -f crds/should do the trick.As for kube-prometheus, we can simplify things for you by moving CRDs to a separate directory in
manifests/if that would help. Just file an issue in the project first 😃https://github.com/prometheus-operator/prometheus-operator/pull/4937 is a quick hack. It would be good if you could test it.