argo-cd: `kube-prometheus-stack` stuck in `OutOfSync`
Checklist:
- I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
- I’ve included steps to reproduce the bug.
- I’ve pasted the output of
argocd version
.
Describe the bug
Hi,
I am deploying the kube-prometheus-stack
helm chart using ArgoCD:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: monitoring
namespace: argocd
spec:
project: default
source:
chart: kube-prometheus-stack
repoURL: https://prometheus-community.github.io/helm-charts
targetRevision: 41.6.1
destination:
namespace: monitoring
server: https://kubernetes.default.svc
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- ServerSideApply=true
- CreateNamespace=true
It create all the resources but it stays in Current sync status: OutOfSync
due to the resource:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: monitoring-kube-prometheus-kubelet
...
If I click the resource, using the ArgoCD WebUI, Summary
and Diff
I get:
Expected behavior
Current sync status: Sync
Version
Argo CD: v2.5.0+b895da4
Build Date: 2022-10-25T14:40:01Z
Go Version: go1.18.7
Go Compiler: gc
Platform: linux/amd64
jsonnet: v0.18.0
kustomize: v4.5.7 2022-08-02T16:35:54Z
Helm: v3.10.1+g9f88ccb
kubectl: v0.24.2
Logs
From the argocd-application-controller-0
pod it shows:
time="2022-10-26T13:03:09Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'servicemonitor.monitoring.coreos.com/monitoring-kube-prometheus-kubelet serverside-applied'" application=argocd/monitoring kind=ServiceMonitor name=monitoring-kube-prometheus-kubelet namespace=monitoring phase=Sync syncId=00106-mKqpp
time="2022-10-26T13:03:14Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:41.6.1,Prune:true,DryRun:false,SyncStrategy:nil,Resources:[]SyncOperationResource{SyncOperationResource{Group:monitoring.coreos.com,Kind:ServiceMonitor,Name:monitoring-kube-prometheus-kubelet,Namespace:,},},Source:nil,Manifests:[],SyncOptions:[ServerSideApply=true CreateNamespace=true],} { true} [] {5 nil}}" application=argocd/monitoring
time="2022-10-26T13:03:14Z" level=info msg="Tasks (dry-run)" application=argocd/monitoring syncId=00107-rGhzV tasks="[Sync/0 resource monitoring.coreos.com/ServiceMonitor:monitoring/monitoring-kube-prometheus-kubelet obj->obj (,,)]"
time="2022-10-26T13:03:14Z" level=info msg="Applying resource ServiceMonitor/monitoring-kube-prometheus-kubelet in cluster: https://10.100.0.1:443, namespace: monitoring"
time="2022-10-26T13:03:14Z" level=info msg="Applying resource ServiceMonitor/monitoring-kube-prometheus-kubelet in cluster: https://10.100.0.1:443, namespace: monitoring"
time="2022-10-26T13:03:14Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'servicemonitor.monitoring.coreos.com/monitoring-kube-prometheus-kubelet serverside-applied'" application=argocd/monitoring kind=ServiceMonitor name=monitoring-kube-prometheus-kubelet namespace=monitoring phase=Sync syncId=00107-rGhzV
time="2022-10-26T13:03:19Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:41.6.1,Prune:true,DryRun:false,SyncStrategy:nil,Resources:[]SyncOperationResource{SyncOperationResource{Group:monitoring.coreos.com,Kind:ServiceMonitor,Name:monitoring-kube-prometheus-kubelet,Namespace:,},},Source:nil,Manifests:[],SyncOptions:[ServerSideApply=true CreateNamespace=true],} { true} [] {5 nil}}" application=argocd/monitoring
time="2022-10-26T13:03:19Z" level=info msg="Tasks (dry-run)" application=argocd/monitoring syncId=00108-awxqW tasks="[Sync/0 resource monitoring.coreos.com/ServiceMonitor:monitoring/monitoring-kube-prometheus-kubelet obj->obj (,,)]"
time="2022-10-26T13:03:19Z" level=info msg="Applying resource ServiceMonitor/monitoring-kube-prometheus-kubelet in cluster: https://10.100.0.1:443, namespace: monitoring"
time="2022-10-26T13:03:19Z" level=info msg="Applying resource ServiceMonitor/monitoring-kube-prometheus-kubelet in cluster: https://10.100.0.1:443, namespace: monitoring"
time="2022-10-26T13:03:19Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'servicemonitor.monitoring.coreos.com/monitoring-kube-prometheus-kubelet serverside-applied'" application=argocd/monitoring kind=ServiceMonitor name=monitoring-kube-prometheus-kubelet namespace=monitoring phase=Sync syncId=00108-awxqW
time="2022-10-26T13:03:24Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:41.6.1,Prune:true,DryRun:false,SyncStrategy:nil,Resources:[]SyncOperationResource{SyncOperationResource{Group:monitoring.coreos.com,Kind:ServiceMonitor,Name:monitoring-kube-prometheus-kubelet,Namespace:,},},Source:nil,Manifests:[],SyncOptions:[ServerSideApply=true CreateNamespace=true],} { true} [] {5 nil}}" application=argocd/monitoring
time="2022-10-26T13:03:24Z" level=info msg="Tasks (dry-run)" application=argocd/monitoring syncId=00109-rqkbn tasks="[Sync/0 resource monitoring.coreos.com/ServiceMonitor:monitoring/monitoring-kube-prometheus-kubelet obj->obj (,,)]"
time="2022-10-26T13:03:24Z" level=info msg="Applying resource ServiceMonitor/monitoring-kube-prometheus-kubelet in cluster: https://10.100.0.1:443, namespace: monitoring"
time="2022-10-26T13:03:24Z" level=info msg="Applying resource ServiceMonitor/monitoring-kube-prometheus-kubelet in cluster: https://10.100.0.1:443, namespace: monitoring"
time="2022-10-26T13:03:24Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'servicemonitor.monitoring.coreos.com/monitoring-kube-prometheus-kubelet serverside-applied'" application=argocd/monitoring kind=ServiceMonitor name=monitoring-kube-prometheus-kubelet namespace=monitoring phase=Sync syncId=00109-rqkbn
Thanks
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 14
- Comments: 21 (7 by maintainers)
Like mentioned above, there are a few approaches that can be used to address this issue in Argo CD:
Replace=true
annotationReplace=true
All the approaches above will fix the problem but require some amount of work to be done.
In Argo CD 2.5 you can now use ServerSideApply to avoid the error with big CRDs while syncing. However, Argo CD is unable to consider CRD default values during diff calculation which causes it to show resources out-of-sync when in fact they aren’t. To address this issue with the minimal amount of work users can leverage
ignoreDifferences
configuration.To deploy prometheus stack with Argo CD you can apply this
Application
resource:With this approach users don’t need to create an additional project to patch CRDs. Everything can be configured from within the
Application
resource. Note that every default value can be added to thejqPathExpression
list, like in the example above, and it will be ignored during diff calculation.Ideally Argo CD should be able to retrieve all schemas from the target cluster with the proper structure so it can be used to consider CRD default values during diff calculation. I created the following issue to track this enhancement (https://github.com/argoproj/argo-cd/issues/11139). Please vote for it if you want to see it implemented.
Closing this issue for now.
I installed it using this way:
Where
patchAnnotationTooLong.yml
contains:It fixes the annotation too long error
I have just reinstalled from scratch, unfortunately
ServerSideApply
is required.Yes I think so. At least I want to use ServerSideApply because it fixes other problems like being able to apply large CRDs etc.
@leoluz
Live manifest
Desired manifest
I also saw this old issue https://github.com/argoproj/argo-cd/issues/4126 related to what looks like the same problem.
Apart from that I also now see another issue using the loki helmchart in a ServiceMonitor - Probably same issue as with
kube-prometheus-stack
helm chart.Live for loki servicemonitor
Desired for loki servicemonitor
The ignoredifferences solution above worked for me except I had to specify something different to match:
@Cowboy-coder yes please… Just copy/paste the statefulset details in the new ticket from your previous comment: https://github.com/argoproj/argo-cd/issues/11074#issuecomment-1294634254
@Cowboy-coder
client-side-apply
diff is based on patches calculated with 3-way-diff using desired state, live state and last-applied-configuration annotation.Server-side-apply
diffs is a brand new implementation that uses the same library used by kubernetes while applying resources server side which leverages themanagedFields
to inspect and define fields ownership.@leoluz I have the same issue / background. Currently, syncing
kube-prometheus-stack
with Replace=true for the too big CRDs and testing SSA to resolve that hack.prometheus-operator-kubelet live manifest (after recreating it with ServerSideApply=true on 2.5.0)
Desired resource definition (generated locally via helm template; but synced via kustomize by ArgoCD)
I suspect the default value from the CRD plays a role here.