grafana-operator: [Bug] grafana operator 5.6.0 -> 5.6.1 upgrade issues openshift

Describe the bug We are unable to upgrade to grafana-operator.v.5.6.1 from grafana-operator.v.5.6.0

We have a development cluster where we have automatic upgrades for the operators enabled. This morning our cluster tried to upgrade grafana to 5.6.1 but encountered the following error:

install strategy failed: Deployment.apps "grafana-operator-controller-manager" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/managed-by":"olm", "app.kubernetes.io/name":"grafana-operator"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Version v.5.6.0

To Reproduce

  1. Install grafana operator v.5.6.0.
  2. Approve install-plan for v.5.6.1.
  3. Check the events of the v.5.6.1 operator csv.

Expected behavior Successful installation of v.5.6.1

Runtime (please complete the following information):

OS: Linux Grafana Operator Version v5.6.0 Environment: Openshift 4.12.26 Deployment type: Deployed via operatorhub

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 2
  • Comments: 28

Most upvoted comments

I verified it in my development environment. I deleted the grafana-operator-controller-manager and the operator upgraded successfully to 5.6.1 and then to 5.6.3

If you have the operator from community-operators catalog, you can try restarting (deleting) pods from openshift-marketplace/community-operatros deployment. That usually does the trick for me.

OLM isn’t making this easy haha. Lets give it some time maybe it still needs to process some stuff.

Screenshot_20240212_075257

Not sure if its something on my end yet, will check further. But it now says that its up to date even though there are new versions.

That is a very good point @tkolo , the only thing that will happen is when the operator restarts it will reconcile any potential changes that could have been done while it was gone. This happens the same way when the operator is restarted. So it should be fine to do in a production env.

If someone wants a Quick And Dirty™ solution to resolve this, all you need to do is remove deployment grafana-operator-controller-manager from openshift-operators namespace. The operator will re-create it automatically and proceed with update to 5.6.1 normally. I’ve noticed no other side-effects in any grafana instance, then again, the cluster I tested it on is not a critical/production one so your mileage may vary.

Thanks for the report @ginokok1996 , sorry about the issue.

This issue was introduced in https://github.com/grafana/grafana-operator/pull/1373 We could potentially do as suggested here: https://github.com/operator-framework/operator-lifecycle-manager/issues/952 to solve this issue.

We are working on a fix.