actions-runner-controller: ARC Finalizers Cause a deadlock when uninstalling AutoscalingRunnerSet in ArgoCD
Checks
- I’ve already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I’m sure my issue is not covered in the troubleshooting guide.
- I am using charts that are officially provided
Controller Version
0.9.0
Deployment Method
ArgoCD
Checks
- This isn’t a question or user support case (For Q&A and community support, go to Discussions).
- I’ve read the Changelog before submitting this issue and I’m sure it’s not due to any recently-introduced backward-incompatible changes
To Reproduce
1. Install the ARC AutoscalingRunnerSet as an ArgoCD app
2. Uninstall ArgoCD app
Describe the bug
The argocd app cannot be deleted because Argo tries to delete resources that are normally deleted by the ARC controller and normal deletion is blocked by the actions.github.com/cleanup-protection finalizer
including:
- AutoscalingRunnerSet
- kube mode service account
- kube mode role
- rs manager role
- kube mode role binding
- rs manager role binding
Describe the expected behavior
The chart should cleanly uninstall when i delete the argocd app
Additional Context
I suspect this may be resolvable by helm/argo annotations for the affected resources, I will test with a fork of the helm chart
Controller Logs
My employer's open source contribution policy forbids me from creating public Gists. I will provide redacted logs upon request via zip/tarball.
Runner Pod Logs
My employer's open source contribution policy forbids me from creating public Gists. I will provide redacted logs upon request via zip/tarball.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 9
- Comments: 17 (1 by maintainers)
Hey everyone,
After discussing this issue with the team, we decided not to apply annotations at this time. We are aware that many of you are using ArgoCD to manage ARC and scale sets, but we don’t have capacity to support it at this time. We would recommend that you maintain your own chart if it requires such modifications until we officially support it. Thank you @rteeling-evernorth, for raising the PR to fix it, and thank you for describing this problem so thoroughly! It will help other people apply changes to their own charts.
@nikola-jokic, @rteeling-evernorth ,
Since there is no intention no provide the capacity on ArgoCD. Could a possibility be around the support of annotations on all Kubernetes objects managed by the chart?
In the end, if we are able to set the annotations everywhere, it could be handled in the values, without requiring to fork the chart…
The problem here is that the argoCD annotations are not applied to everything - because this is ultimately an order of operations issue the ability to selectively apply the annotations to specific resources is needed. I agree a custom annotation value would be useful but it would not solve this issue.
Additionally, given the argo annotations are effectively inert if the resource is not actively being managed by argo, I dont see why this would have any adverse effect if installing via kubectl/helm/flux/whatever else. If we really want to split hairs, i can update PR to put the argo annotations on a feature flag if that would make the maintainers happy. Thoughts @nikola-jokic ?
Thanks for adding the appropriate labels. It’s on our radar now.
@ahatzz11 Had the same issue with having to manually resolve the finalizer deadlock. I’ve installed the controller in an entirely separate ArgoCD app, since the cardinality of the scale set controller to scale sets is 1:N. I assume GitHub is distributing the controller and runner set charts separately because of this. You should separate them into separate ArgoCD apps for the sake of ease of maintenance if nothing else.
Argo Sync waves are only relevant within the context of an Argo App, so as long as the two charts are managed by separate apps, you won’t see any conflict. Once this is done you just uninstall any app for the scale set, then uninstall the controller and you won’t have any deadlock issues.
Hope that helps!
Hello all, I’m also struggling here with ARC when deployed with ArgoCD. Some resources like AutoscalingListener and associated role and rolebinding inherit annotations and labels from AutoscalingRunnerSet template, and thus the ArgoCD
argocd.argoproj.io/instancelabel as well. These resources will then be considered by ArgoCD as part of the application and if self healing is activated, since they are not part of the Helm chart itself, ArgoCD will try to remove them. Starts an infinite looping battle with the ARC controller which wants to recreate the objects…As @AlexandreODelisle suggested, would be so nice to just expose Helm values to customize labels/annotations of these objects so that we can tell ArgoCD to ignore them in the process ! Looking forward for having this feature which would be so useful !
@nikola-jokic @Link- @rentziass
This issue has been sitting stale for a few months now. Would it be possible to get the associated PR on the next release?
This is the usual solution - giving the choice of annotations to the chart user. By my opinion - best solution for this kind of a problem. 👍
I was successfully able to fix the issue using argocd annotations
argocd.argoproj.io/sync-options: Delete=falseandargocd.argoproj.io/sync-wave: "1", I’ll open a PR