argo-cd: StatefulSet with volume claim causes diff with `ServerSideApply=true`
Checklist:
- I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
- I’ve included steps to reproduce the bug.
- I’ve pasted the output of
argocd version
.
Describe the bug
As described in https://github.com/argoproj/argo-cd/issues/11074 I get OutOfSync diff when installing loki using ServerSideApply. I haven’t double checked but Im pretty sure this is not a Loki-specific issue.
Live manifest
apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"write","app.kubernetes.io/instance":"loki","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"loki","app.kubernetes.io/part-of":"memberlist","app.kubernetes.io/version":"2.6.1","argocd.argoproj.io/instance":"logging","helm.sh/chart":"loki-3.2.0"},"name":"loki-write","namespace":"logging"},"spec":{"podManagementPolicy":"Parallel","replicas":3,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"write","app.kubernetes.io/instance":"loki","app.kubernetes.io/name":"loki"}},"serviceName":"loki-write-headless","template":{"metadata":{"annotations":{"checksum/config":"dc4356fb9c8ae2285982e39f348eaa3087a7bd09084224adb6915903fdf04574"},"labels":{"app.kubernetes.io/component":"write","app.kubernetes.io/instance":"loki","app.kubernetes.io/name":"loki","app.kubernetes.io/part-of":"memberlist"}},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app.kubernetes.io/component":"write","app.kubernetes.io/instance":"loki","app.kubernetes.io/name":"loki"}},"topologyKey":"kubernetes.io/hostname"}]}},"automountServiceAccountToken":true,"containers":[{"args":["-config.file=/etc/loki/config/config.yaml","-target=write"],"env":[{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"loki-s3"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"loki-s3"}}}],"image":"docker.io/grafana/loki:2.6.1","imagePullPolicy":"IfNotPresent","name":"write","ports":[{"containerPort":3100,"name":"http-metrics","protocol":"TCP"},{"containerPort":9095,"name":"grpc","protocol":"TCP"},{"containerPort":7946,"name":"http-memberlist","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"http-metrics"},"initialDelaySeconds":30,"timeoutSeconds":1},"resources":{},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true},"volumeMounts":[{"mountPath":"/etc/loki/config","name":"config"},{"mountPath":"/var/loki","name":"data"}]}],"securityContext":{"fsGroup":10001,"runAsGroup":10001,"runAsNonRoot":true,"runAsUser":10001},"serviceAccountName":"loki","terminationGracePeriodSeconds":300,"volumes":[{"configMap":{"name":"loki"},"name":"config"}]}},"updateStrategy":{"rollingUpdate":{"partition":0}},"volumeClaimTemplates":[{"metadata":{"name":"data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}},"storageClassName":"openebs-hostpath"}}]}}
creationTimestamp: '2022-10-26T09:13:13Z'
generation: 1
labels:
app.kubernetes.io/component: write
app.kubernetes.io/instance: loki
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: loki
app.kubernetes.io/part-of: memberlist
app.kubernetes.io/version: 2.6.1
argocd.argoproj.io/instance: logging
helm.sh/chart: loki-3.2.0
managedFields:
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:labels':
'f:app.kubernetes.io/component': {}
'f:app.kubernetes.io/instance': {}
'f:app.kubernetes.io/managed-by': {}
'f:app.kubernetes.io/name': {}
'f:app.kubernetes.io/part-of': {}
'f:app.kubernetes.io/version': {}
'f:argocd.argoproj.io/instance': {}
'f:helm.sh/chart': {}
'f:spec':
'f:podManagementPolicy': {}
'f:replicas': {}
'f:revisionHistoryLimit': {}
'f:selector': {}
'f:serviceName': {}
'f:template':
'f:metadata':
'f:annotations':
'f:checksum/config': {}
'f:labels':
'f:app.kubernetes.io/component': {}
'f:app.kubernetes.io/instance': {}
'f:app.kubernetes.io/name': {}
'f:app.kubernetes.io/part-of': {}
'f:spec':
'f:affinity':
'f:podAntiAffinity':
'f:requiredDuringSchedulingIgnoredDuringExecution': {}
'f:automountServiceAccountToken': {}
'f:containers':
'k:{"name":"write"}':
.: {}
'f:args': {}
'f:env':
'k:{"name":"AWS_ACCESS_KEY_ID"}':
.: {}
'f:name': {}
'f:valueFrom':
'f:secretKeyRef': {}
'k:{"name":"AWS_SECRET_ACCESS_KEY"}':
.: {}
'f:name': {}
'f:valueFrom':
'f:secretKeyRef': {}
'f:image': {}
'f:imagePullPolicy': {}
'f:name': {}
'f:ports':
'k:{"containerPort":3100,"protocol":"TCP"}':
.: {}
'f:containerPort': {}
'f:name': {}
'f:protocol': {}
'k:{"containerPort":7946,"protocol":"TCP"}':
.: {}
'f:containerPort': {}
'f:name': {}
'f:protocol': {}
'k:{"containerPort":9095,"protocol":"TCP"}':
.: {}
'f:containerPort': {}
'f:name': {}
'f:protocol': {}
'f:readinessProbe':
'f:httpGet':
'f:path': {}
'f:port': {}
'f:initialDelaySeconds': {}
'f:timeoutSeconds': {}
'f:resources': {}
'f:securityContext':
'f:allowPrivilegeEscalation': {}
'f:capabilities':
'f:drop': {}
'f:readOnlyRootFilesystem': {}
'f:volumeMounts':
'k:{"mountPath":"/etc/loki/config"}':
.: {}
'f:mountPath': {}
'f:name': {}
'k:{"mountPath":"/var/loki"}':
.: {}
'f:mountPath': {}
'f:name': {}
'f:securityContext':
'f:fsGroup': {}
'f:runAsGroup': {}
'f:runAsNonRoot': {}
'f:runAsUser': {}
'f:serviceAccountName': {}
'f:terminationGracePeriodSeconds': {}
'f:volumes':
'k:{"name":"config"}':
.: {}
'f:configMap':
'f:name': {}
'f:name': {}
'f:updateStrategy':
'f:rollingUpdate':
'f:partition': {}
'f:volumeClaimTemplates': {}
manager: argocd-controller
operation: Apply
time: '2022-10-28T07:36:52Z'
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations': {}
'f:labels':
.: {}
'f:app.kubernetes.io/component': {}
'f:app.kubernetes.io/managed-by': {}
'f:app.kubernetes.io/name': {}
'f:app.kubernetes.io/part-of': {}
'f:app.kubernetes.io/version': {}
'f:helm.sh/chart': {}
'f:spec':
'f:podManagementPolicy': {}
'f:replicas': {}
'f:revisionHistoryLimit': {}
'f:selector': {}
'f:serviceName': {}
'f:template':
'f:metadata':
'f:annotations':
.: {}
'f:checksum/config': {}
'f:labels':
.: {}
'f:app.kubernetes.io/component': {}
'f:app.kubernetes.io/instance': {}
'f:app.kubernetes.io/name': {}
'f:app.kubernetes.io/part-of': {}
'f:spec':
'f:affinity':
.: {}
'f:podAntiAffinity':
.: {}
'f:requiredDuringSchedulingIgnoredDuringExecution': {}
'f:automountServiceAccountToken': {}
'f:containers':
'k:{"name":"write"}':
.: {}
'f:args': {}
'f:env':
.: {}
'k:{"name":"AWS_ACCESS_KEY_ID"}':
.: {}
'f:name': {}
'f:valueFrom':
.: {}
'f:secretKeyRef': {}
'k:{"name":"AWS_SECRET_ACCESS_KEY"}':
.: {}
'f:name': {}
'f:valueFrom':
.: {}
'f:secretKeyRef': {}
'f:image': {}
'f:imagePullPolicy': {}
'f:name': {}
'f:ports':
.: {}
'k:{"containerPort":3100,"protocol":"TCP"}':
.: {}
'f:containerPort': {}
'f:name': {}
'f:protocol': {}
'k:{"containerPort":7946,"protocol":"TCP"}':
.: {}
'f:containerPort': {}
'f:name': {}
'f:protocol': {}
'k:{"containerPort":9095,"protocol":"TCP"}':
.: {}
'f:containerPort': {}
'f:name': {}
'f:protocol': {}
'f:readinessProbe':
.: {}
'f:failureThreshold': {}
'f:httpGet':
.: {}
'f:path': {}
'f:port': {}
'f:scheme': {}
'f:initialDelaySeconds': {}
'f:periodSeconds': {}
'f:successThreshold': {}
'f:timeoutSeconds': {}
'f:resources': {}
'f:securityContext':
.: {}
'f:allowPrivilegeEscalation': {}
'f:capabilities':
.: {}
'f:drop': {}
'f:readOnlyRootFilesystem': {}
'f:terminationMessagePath': {}
'f:terminationMessagePolicy': {}
'f:volumeMounts':
.: {}
'k:{"mountPath":"/etc/loki/config"}':
.: {}
'f:mountPath': {}
'f:name': {}
'k:{"mountPath":"/var/loki"}':
.: {}
'f:mountPath': {}
'f:name': {}
'f:dnsPolicy': {}
'f:restartPolicy': {}
'f:schedulerName': {}
'f:securityContext':
.: {}
'f:fsGroup': {}
'f:runAsGroup': {}
'f:runAsNonRoot': {}
'f:runAsUser': {}
'f:serviceAccount': {}
'f:serviceAccountName': {}
'f:terminationGracePeriodSeconds': {}
'f:volumes':
.: {}
'k:{"name":"config"}':
.: {}
'f:configMap':
.: {}
'f:defaultMode': {}
'f:name': {}
'f:name': {}
'f:updateStrategy':
'f:rollingUpdate':
.: {}
'f:partition': {}
'f:type': {}
manager: argocd-application-controller
operation: Update
time: '2022-10-26T09:13:13Z'
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
'f:status':
'f:availableReplicas': {}
'f:collisionCount': {}
'f:currentReplicas': {}
'f:currentRevision': {}
'f:observedGeneration': {}
'f:readyReplicas': {}
'f:replicas': {}
'f:updateRevision': {}
'f:updatedReplicas': {}
manager: kube-controller-manager
operation: Update
subresource: status
time: '2022-10-26T09:18:53Z'
- apiVersion: apps/v1
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:annotations':
'f:kubectl.kubernetes.io/last-applied-configuration': {}
'f:labels':
'f:app.kubernetes.io/instance': {}
'f:argocd.argoproj.io/instance': {}
manager: argocd-controller
operation: Update
time: '2022-10-28T07:19:13Z'
name: loki-write
namespace: logging
resourceVersion: '46346521'
uid: 159449f2-01c3-4ee1-8b91-2e1e90c1e9eb
spec:
podManagementPolicy: Parallel
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: write
app.kubernetes.io/instance: loki
app.kubernetes.io/name: loki
serviceName: loki-write-headless
template:
metadata:
annotations:
checksum/config: dc4356fb9c8ae2285982e39f348eaa3087a7bd09084224adb6915903fdf04574
creationTimestamp: null
labels:
app.kubernetes.io/component: write
app.kubernetes.io/instance: loki
app.kubernetes.io/name: loki
app.kubernetes.io/part-of: memberlist
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/component: write
app.kubernetes.io/instance: loki
app.kubernetes.io/name: loki
topologyKey: kubernetes.io/hostname
automountServiceAccountToken: true
containers:
- args:
- '-config.file=/etc/loki/config/config.yaml'
- '-target=write'
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: AWS_ACCESS_KEY_ID
name: loki-s3
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: AWS_SECRET_ACCESS_KEY
name: loki-s3
image: 'docker.io/grafana/loki:2.6.1'
imagePullPolicy: IfNotPresent
name: write
ports:
- containerPort: 3100
name: http-metrics
protocol: TCP
- containerPort: 9095
name: grpc
protocol: TCP
- containerPort: 7946
name: http-memberlist
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /ready
port: http-metrics
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/loki/config
name: config
- mountPath: /var/loki
name: data
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 10001
runAsGroup: 10001
runAsNonRoot: true
runAsUser: 10001
serviceAccount: loki
serviceAccountName: loki
terminationGracePeriodSeconds: 300
volumes:
- configMap:
defaultMode: 420
name: loki
name: config
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: openebs-hostpath
volumeMode: Filesystem
status:
phase: Pending
status:
availableReplicas: 3
collisionCount: 0
currentReplicas: 3
currentRevision: loki-write-68f4b7bcfc
observedGeneration: 1
readyReplicas: 3
replicas: 3
updateRevision: loki-write-68f4b7bcfc
updatedReplicas: 3
Desired manifest
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app.kubernetes.io/component: write
app.kubernetes.io/instance: loki
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: loki
app.kubernetes.io/part-of: memberlist
app.kubernetes.io/version: 2.6.1
argocd.argoproj.io/instance: logging
helm.sh/chart: loki-3.2.0
name: loki-write
namespace: logging
spec:
podManagementPolicy: Parallel
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: write
app.kubernetes.io/instance: loki
app.kubernetes.io/name: loki
serviceName: loki-write-headless
template:
metadata:
annotations:
checksum/config: dc4356fb9c8ae2285982e39f348eaa3087a7bd09084224adb6915903fdf04574
labels:
app.kubernetes.io/component: write
app.kubernetes.io/instance: loki
app.kubernetes.io/name: loki
app.kubernetes.io/part-of: memberlist
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/component: write
app.kubernetes.io/instance: loki
app.kubernetes.io/name: loki
topologyKey: kubernetes.io/hostname
automountServiceAccountToken: true
containers:
- args:
- '-config.file=/etc/loki/config/config.yaml'
- '-target=write'
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: AWS_ACCESS_KEY_ID
name: loki-s3
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: AWS_SECRET_ACCESS_KEY
name: loki-s3
image: 'docker.io/grafana/loki:2.6.1'
imagePullPolicy: IfNotPresent
name: write
ports:
- containerPort: 3100
name: http-metrics
protocol: TCP
- containerPort: 9095
name: grpc
protocol: TCP
- containerPort: 7946
name: http-memberlist
protocol: TCP
readinessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 30
timeoutSeconds: 1
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
volumeMounts:
- mountPath: /etc/loki/config
name: config
- mountPath: /var/loki
name: data
securityContext:
fsGroup: 10001
runAsGroup: 10001
runAsNonRoot: true
runAsUser: 10001
serviceAccountName: loki
terminationGracePeriodSeconds: 300
volumes:
- configMap:
name: loki
name: config
updateStrategy:
rollingUpdate:
partition: 0
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: openebs-hostpath
I also saw this old issue https://github.com/argoproj/argo-cd/issues/4126 related to what looks like the same problem.
To Reproduce
Install this helm chart: https://github.com/grafana/loki/tree/main/production/helm/loki
Expected behavior
No diff.
Screenshots
Version
argocd: v2.4.11+3d9e9f2.dirty
BuildDate: 2022-08-22T19:32:10Z
GitCommit: 3d9e9f2f95b7801b90377ecfc4073e5f0f07205b
GitTreeState: dirty
GoVersion: go1.19
Compiler: gc
Platform: darwin/amd64
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
argocd-server: v2.5.0+b895da4
Logs
Paste any relevant application logs here.
About this issue
- Original URL
- State: open
- Created 2 years ago
- Reactions: 70
- Comments: 23 (5 by maintainers)
Commits related to this issue
- Update: disable serversideapply Due to: - https://github.com/argoproj/argo-cd/issues/11143 - https://github.com/argoproj/argo-cd/issues/11074 Signed-off-by: Nicolas Lamirault <nicolas.lamirault@gmai... — committed to portefaix/portefaix-kubernetes by nlamirault 2 years ago
- fix(argocd): disable server-side apply Due to argoproj/argo-cd#11143 — committed to hur/homelab by hur a year ago
- fix: correct values to fix argo sync Argo would not sync properly unless the appropriate values were defined for the secret - Known issue doc'ed under 'https://github.com/argoproj/argo-cd/issues/11143... — committed to llajas/homelab by llajas a year ago
- fix: ignore diffs on statefulset.spec.volumeclaimtemplate This issue will be solved by argoproj/argo-cd#11143 — committed to ismailbay/k8s-dev-stack by ismailbay 6 months ago
Server-side diffing is still in Leo’s queue (he recently built a PoC), so a proper fix is on its way. But I don’t have an ETA yet.
I hit the same issue today with Loki.
Talked with Leo about this yesterday. He’s working through how to manage caching the diff.
Based on https://argo-cd.readthedocs.io/en/stable/user-guide/diffing And a post I saw on a related issue: https://github.com/argoproj/argo-cd/issues/11074#issuecomment-1298825859
This ignore differences block in your application definition for the kube prometheus stack (where you’re server side applying loki) will give you the shiny green checkmark:
For my use case, I am not too worried about Argo “thinking” there is a diff for these statefulsets because we mostly use default settings for the kube prometheus stack. We install it and forget about it.
Others of you may not want this workaround. Really the bug in how Argo is diffing needs addressing.
+1 I hit the same issue today on PVCs and on the ServiceMonitor CR from the Prometheus stack.
Hey @crenshaw-dev thanks for the update on that. Appreciate all your guys efforts.
@apelisse Tks for jumping in the discussion. In the current version of Argo CD Server-Side Apply, we don’t do a dry-run request. We tried to follow the same approach used by CSA aiming for efficiency and avoid sending that additional request to k8s API server. For that, I checked how K8s API server does the conversions and duplicated that logic in Argo CD controller. Unfortunately that is causing issues with default fields. I suspect that it is related with the OpenAPI document that we provide to the
managedfields.NewGVKParser()
constructor. I couldn’t find a way to retrieve an OpenAPI document that contains the definitions of schemas default values.Going forward, I believe that dry run is a better way to achieve more accurate diffs as (I believe) it will handle cases where the resource is configured with mutation webhooks. I have an Argo CD proposal to implement dryrun during diff calculation (https://github.com/argoproj/argo-cd/issues/11574). I am currently negotiating time to dedicate on the implementation.
@MohammedNoureldin Yes.
I have got a similar issue with loki, probably related?
for now I’m just patching statefulsets on kustomize/helm
Same issue with kuik
https://github.com/enix/kube-image-keeper/blob/eab1d909dd058a71154d1aaaf86d62b3f7bbceb4/helm/kube-image-keeper/templates/registry-statefulset.yaml#L108-L110