grafana-operator: [Bug] grafana operator 5.6.0 -> 5.6.1 upgrade issues openshift
Describe the bug We are unable to upgrade to grafana-operator.v.5.6.1 from grafana-operator.v.5.6.0
We have a development cluster where we have automatic upgrades for the operators enabled. This morning our cluster tried to upgrade grafana to 5.6.1 but encountered the following error:
install strategy failed: Deployment.apps "grafana-operator-controller-manager" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/managed-by":"olm", "app.kubernetes.io/name":"grafana-operator"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Version v.5.6.0
To Reproduce
- Install grafana operator v.5.6.0.
- Approve install-plan for v.5.6.1.
- Check the events of the v.5.6.1 operator csv.
Expected behavior Successful installation of v.5.6.1
Runtime (please complete the following information):
OS: Linux Grafana Operator Version v5.6.0 Environment: Openshift 4.12.26 Deployment type: Deployed via operatorhub
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 2
- Comments: 28
I verified it in my development environment. I deleted the
grafana-operator-controller-managerand the operator upgraded successfully to 5.6.1 and then to 5.6.3If you have the operator from community-operators catalog, you can try restarting (deleting) pods from
openshift-marketplace/community-operatrosdeployment. That usually does the trick for me.OLM isn’t making this easy haha. Lets give it some time maybe it still needs to process some stuff.
Not sure if its something on my end yet, will check further. But it now says that its up to date even though there are new versions.
That is a very good point @tkolo , the only thing that will happen is when the operator restarts it will reconcile any potential changes that could have been done while it was gone. This happens the same way when the operator is restarted. So it should be fine to do in a production env.
If someone wants a Quick And Dirty™ solution to resolve this, all you need to do is remove deployment
grafana-operator-controller-managerfromopenshift-operatorsnamespace. The operator will re-create it automatically and proceed with update to 5.6.1 normally. I’ve noticed no other side-effects in any grafana instance, then again, the cluster I tested it on is not a critical/production one so your mileage may vary.Thanks for the report @ginokok1996 , sorry about the issue.
This issue was introduced in https://github.com/grafana/grafana-operator/pull/1373 We could potentially do as suggested here: https://github.com/operator-framework/operator-lifecycle-manager/issues/952 to solve this issue.
We are working on a fix.