rancher: Rancher-monitoring installation fails on fresh cluster with crd error
What kind of request is this (question/bug/enhancement/feature request): Bug
Steps to reproduce (least amount of steps as possible):
- Create a new Rancher cluster
- Install rancher-monitoring into the local cluster
This was reproduced on a fresh rke2 cluster.
or
- Create a new downstream cluster
- Install rancher-monitoring into the downstream cluster
Result: The job that creates the CRDs fails with:
rancher-monitoring-crd-create-w4bnw create-crds Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
rancher-monitoring-crd-create-w4bnw create-crds Name: "prometheuses.monitoring.coreos.com", Namespace: ""
rancher-monitoring-crd-create-w4bnw create-crds for: "/etc/config/crd-manifest.yaml": CustomResourceDefinition.apiextensions.k8s.io "prometheuses.monitoring.coreos.com" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
rancher-monitoring-crd-create-w4bnw create-crds Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
rancher-monitoring-crd-create-w4bnw create-crds Name: "alertmanagers.monitoring.coreos.com", Namespace: ""
rancher-monitoring-crd-create-w4bnw create-crds for: "/etc/config/crd-manifest.yaml": CustomResourceDefinition.apiextensions.k8s.io "alertmanagers.monitoring.coreos.com" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
Other details that may be helpful: Monitoring was deployed directly after the cluster was created, e.g. with terraform. So, it could be some kind of race condition. The issue started to appear first with Rancher 2.5.8.
Environment information
- Rancher version (
rancher/rancher/rancher/serverimage tag or shown bottom left in the UI): 2.5.8 - Monitoring version: 14.5.100
- Installation option (single install/HA): HA
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 1
- Comments: 15 (8 by maintainers)
Sorry I thought I had replied.
Yes I am guessing it is, I just seemed to see it more when Istio was part of the same installs rather than monitoring on its own.
An explanation for why this could be appearing sporadically in some clusters but not consistently reproducible in all setups could be because we do not currently wait for CRDs to be established before continuing to the next steps in our CRD chart.
This would explain this specific error since a fresh cluster with Rancher Monitoring V1 CRDs (which is based on apiextensions v1/beta1) requires the
set-preserve-unknown-fields-falseinitContainer to successfully update the CRD before the apiextensiosn v1 CRD is applied.We should re-test this issue to see if we can reproduce it after the fix in https://github.com/rancher/charts/pull/1203 is merged.