argocd-operator: Auto-upgrade to 0.0.15 on OpenShift 4.5 causing unexpected behaviour

Describe the bug An auto-update occurred on 25th May to version 0.0.15. After it, multiple things are happening:

  1. Sync started to fail with deployments.apps "myapp" is forbidden: User "system:serviceaccount:argocd:argocd-argocd-application-controller" cannot patch resource "deployments" in API group "apps" in the namespace "mynamespace"

From the error, I realised new RBAC resources have been created with an additional argocd- prefix. The namespace hosting ArgoCD is argocd, not sure if something has changed in this version. Service Accounts: image

  1. DEX server is not starting (already reported in https://github.com/argoproj-labs/argocd-operator/issues/322)

To Reproduce Steps to reproduce the behavior: Difficult to reproduce…

  1. OpenShift cluster (in my case v4.5) with a ArgoCD Operator version before 0.0.15
  2. Auto-Upgrade to 0.0.15

Expected behavior

ArgoCD should keep working as normal.

Screenshots

Additional context

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 8
  • Comments: 30

Most upvoted comments

@iam-veeramalla , with all respect, according to semver, breaking changes, should go under major version bump, not patch or minor version changes Documentation is a good start, but an automatic update is a no-go in such scenarios, as it is used in production systems, where production services are being maintained by argo.cd Please consider following semversion and please opt-in the levers for major version bumps to go manually

@iam-veeramalla , with all respect, according to semver, breaking changes, should go under major version bump, not patch or minor version changes Documentation is a good start, but an automatic update is a no-go in such scenarios, as it is used in production systems, where production services are being maintained by argo.cd Please consider following semversion and please opt-in the levers for major version bumps to go manually

Thanks @vyckou . Feedback acknowledged. We are currently working on getting a new release and we are also looking at the versioning and documentation.

@sbose78 @wtam2018 @bigkevmcd @chetan-rns Any idea if this new naming will be reverted or not? We’re trying to decide whether to rollback to 0.0.14 or roll forward and adjust to the new SA name, but it’s not clear if this is a permanent change or should be considered a bug (as it’s a breaking change) 😕

Hi @chrisob , Thanks for reaching out. This is a change that was introduced as part of the commit 3d1c589 . Unfortunately, This is a breaking change. I will create an issue to document the upgrade path with minimum or no issues.

cc: @wtam2018 @jannfis @sbose78

Hi @bortek , The new release would add a lot of enhancements so we need to perform some good testing. we have plans to release a new version in Sep or probably early October.

Will it work when upgrading from 0.0.14->0.0.XX in Openshift 4.6 , 4.7, 4.8 ?

The new release would have some breaking changes for sure but we will document the steps for a proper upgrade.

+1. It would help to have an upgrade doc similar to https://argoproj.github.io/argo-cd/operator-manual/upgrading/1.8-2.0/

@sbose78 @wtam2018 @bigkevmcd @chetan-rns Any idea if this new naming will be reverted or not? We’re trying to decide whether to rollback to 0.0.14 or roll forward and adjust to the new SA name, but it’s not clear if this is a permanent change or should be considered a bug (as it’s a breaking change) 😕

You need to create the Subscription yourself like this:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: argocd-operator
spec:
  channel: alpha
  installPlanApproval: Manual
  name: argocd-operator
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: argocd-operator.v0.0.14

@samuanv We just had too much issues due to the changed ServiceAccounts / RoleBindings. Those problems were exacerbated by our Gatekeeper policies preventing proper creation. Additionally users were also having problems with applications using Helm that worked before the update. We just decided to disable the automatic update, uninstall the operator, clean up all the mess and reinstall 0.0.14.

Thanks folks. As you pointed out, updating argocd-application-controller Cluster role binding to point to argocd-argocd-application-controller ServiceAccount fixed the issue. Just realised that there is a new rolebinding argocd-argocd-argocd-application-controller. This resource with 3x argocd is the winner 😄

We experience the same as described in OP regarding SA argocd-argocd-application-controller and need to updated the rolebindings to reflect the new Service Account, but is the creation of the new SA intended or a mistake and will be fixed?