argocd-operator: Auto-upgrade to 0.0.15 on OpenShift 4.5 causing unexpected behaviour
Describe the bug An auto-update occurred on 25th May to version 0.0.15. After it, multiple things are happening:
- Sync started to fail with
deployments.apps "myapp" is forbidden: User "system:serviceaccount:argocd:argocd-argocd-application-controller" cannot patch resource "deployments" in API group "apps" in the namespace "mynamespace"
From the error, I realised new RBAC resources have been created with an additional argocd-
prefix. The namespace hosting ArgoCD is argocd
, not sure if something has changed in this version.
Service Accounts:
- DEX server is not starting (already reported in https://github.com/argoproj-labs/argocd-operator/issues/322)
To Reproduce Steps to reproduce the behavior: Difficult to reproduce…
- OpenShift cluster (in my case v4.5) with a ArgoCD Operator version before 0.0.15
- Auto-Upgrade to 0.0.15
Expected behavior
ArgoCD should keep working as normal.
Screenshots
Additional context
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 8
- Comments: 30
@iam-veeramalla , with all respect, according to semver, breaking changes, should go under major version bump, not patch or minor version changes Documentation is a good start, but an automatic update is a no-go in such scenarios, as it is used in production systems, where production services are being maintained by argo.cd Please consider following semversion and please opt-in the levers for major version bumps to go manually
Thanks @vyckou . Feedback acknowledged. We are currently working on getting a new release and we are also looking at the versioning and documentation.
Hi @chrisob , Thanks for reaching out. This is a change that was introduced as part of the commit 3d1c589 . Unfortunately, This is a breaking change. I will create an issue to document the upgrade path with minimum or no issues.
cc: @wtam2018 @jannfis @sbose78
Hi @bortek , The new release would add a lot of enhancements so we need to perform some good testing. we have plans to release a new version in Sep or probably early October.
The new release would have some breaking changes for sure but we will document the steps for a proper upgrade.
+1. It would help to have an upgrade doc similar to https://argoproj.github.io/argo-cd/operator-manual/upgrading/1.8-2.0/
@sbose78 @wtam2018 @bigkevmcd @chetan-rns Any idea if this new naming will be reverted or not? We’re trying to decide whether to rollback to 0.0.14 or roll forward and adjust to the new SA name, but it’s not clear if this is a permanent change or should be considered a bug (as it’s a breaking change) 😕
You need to create the Subscription yourself like this:
@samuanv We just had too much issues due to the changed ServiceAccounts / RoleBindings. Those problems were exacerbated by our Gatekeeper policies preventing proper creation. Additionally users were also having problems with applications using Helm that worked before the update. We just decided to disable the automatic update, uninstall the operator, clean up all the mess and reinstall 0.0.14.
Thanks folks. As you pointed out, updating
argocd-application-controller
Cluster role binding to point toargocd-argocd-application-controller
ServiceAccount fixed the issue. Just realised that there is a new rolebindingargocd-argocd-argocd-application-controller
. This resource with 3xargocd
is the winner 😄We experience the same as described in OP regarding SA argocd-argocd-application-controller and need to updated the rolebindings to reflect the new Service Account, but is the creation of the new SA intended or a mistake and will be fixed?
Is it from this change? https://github.com/argoproj-labs/argocd-operator/commit/3d1c5897c53961fbee8a28f7a9cbbd7d591aad0a#diff-3296585f4de9b03d27051f38cd4384a5cbc24d646119172333fd5b152dbdc215R50