argo-cd: StatefulSet with volume claim causes diff with `ServerSideApply=true`

Checklist:

I’ve searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I’ve included steps to reproduce the bug.
I’ve pasted the output of argocd version.

Describe the bug

As described in https://github.com/argoproj/argo-cd/issues/11074 I get OutOfSync diff when installing loki using ServerSideApply. I haven’t double checked but Im pretty sure this is not a Loki-specific issue.

Live manifest

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"write","app.kubernetes.io/instance":"loki","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"loki","app.kubernetes.io/part-of":"memberlist","app.kubernetes.io/version":"2.6.1","argocd.argoproj.io/instance":"logging","helm.sh/chart":"loki-3.2.0"},"name":"loki-write","namespace":"logging"},"spec":{"podManagementPolicy":"Parallel","replicas":3,"revisionHistoryLimit":10,"selector":{"matchLabels":{"app.kubernetes.io/component":"write","app.kubernetes.io/instance":"loki","app.kubernetes.io/name":"loki"}},"serviceName":"loki-write-headless","template":{"metadata":{"annotations":{"checksum/config":"dc4356fb9c8ae2285982e39f348eaa3087a7bd09084224adb6915903fdf04574"},"labels":{"app.kubernetes.io/component":"write","app.kubernetes.io/instance":"loki","app.kubernetes.io/name":"loki","app.kubernetes.io/part-of":"memberlist"}},"spec":{"affinity":{"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"app.kubernetes.io/component":"write","app.kubernetes.io/instance":"loki","app.kubernetes.io/name":"loki"}},"topologyKey":"kubernetes.io/hostname"}]}},"automountServiceAccountToken":true,"containers":[{"args":["-config.file=/etc/loki/config/config.yaml","-target=write"],"env":[{"name":"AWS_ACCESS_KEY_ID","valueFrom":{"secretKeyRef":{"key":"AWS_ACCESS_KEY_ID","name":"loki-s3"}}},{"name":"AWS_SECRET_ACCESS_KEY","valueFrom":{"secretKeyRef":{"key":"AWS_SECRET_ACCESS_KEY","name":"loki-s3"}}}],"image":"docker.io/grafana/loki:2.6.1","imagePullPolicy":"IfNotPresent","name":"write","ports":[{"containerPort":3100,"name":"http-metrics","protocol":"TCP"},{"containerPort":9095,"name":"grpc","protocol":"TCP"},{"containerPort":7946,"name":"http-memberlist","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/ready","port":"http-metrics"},"initialDelaySeconds":30,"timeoutSeconds":1},"resources":{},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true},"volumeMounts":[{"mountPath":"/etc/loki/config","name":"config"},{"mountPath":"/var/loki","name":"data"}]}],"securityContext":{"fsGroup":10001,"runAsGroup":10001,"runAsNonRoot":true,"runAsUser":10001},"serviceAccountName":"loki","terminationGracePeriodSeconds":300,"volumes":[{"configMap":{"name":"loki"},"name":"config"}]}},"updateStrategy":{"rollingUpdate":{"partition":0}},"volumeClaimTemplates":[{"metadata":{"name":"data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}},"storageClassName":"openebs-hostpath"}}]}}
  creationTimestamp: '2022-10-26T09:13:13Z'
  generation: 1
  labels:
    app.kubernetes.io/component: write
    app.kubernetes.io/instance: loki
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: loki
    app.kubernetes.io/part-of: memberlist
    app.kubernetes.io/version: 2.6.1
    argocd.argoproj.io/instance: logging
    helm.sh/chart: loki-3.2.0
  managedFields:
    - apiVersion: apps/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            'f:app.kubernetes.io/component': {}
            'f:app.kubernetes.io/instance': {}
            'f:app.kubernetes.io/managed-by': {}
            'f:app.kubernetes.io/name': {}
            'f:app.kubernetes.io/part-of': {}
            'f:app.kubernetes.io/version': {}
            'f:argocd.argoproj.io/instance': {}
            'f:helm.sh/chart': {}
        'f:spec':
          'f:podManagementPolicy': {}
          'f:replicas': {}
          'f:revisionHistoryLimit': {}
          'f:selector': {}
          'f:serviceName': {}
          'f:template':
            'f:metadata':
              'f:annotations':
                'f:checksum/config': {}
              'f:labels':
                'f:app.kubernetes.io/component': {}
                'f:app.kubernetes.io/instance': {}
                'f:app.kubernetes.io/name': {}
                'f:app.kubernetes.io/part-of': {}
            'f:spec':
              'f:affinity':
                'f:podAntiAffinity':
                  'f:requiredDuringSchedulingIgnoredDuringExecution': {}
              'f:automountServiceAccountToken': {}
              'f:containers':
                'k:{"name":"write"}':
                  .: {}
                  'f:args': {}
                  'f:env':
                    'k:{"name":"AWS_ACCESS_KEY_ID"}':
                      .: {}
                      'f:name': {}
                      'f:valueFrom':
                        'f:secretKeyRef': {}
                    'k:{"name":"AWS_SECRET_ACCESS_KEY"}':
                      .: {}
                      'f:name': {}
                      'f:valueFrom':
                        'f:secretKeyRef': {}
                  'f:image': {}
                  'f:imagePullPolicy': {}
                  'f:name': {}
                  'f:ports':
                    'k:{"containerPort":3100,"protocol":"TCP"}':
                      .: {}
                      'f:containerPort': {}
                      'f:name': {}
                      'f:protocol': {}
                    'k:{"containerPort":7946,"protocol":"TCP"}':
                      .: {}
                      'f:containerPort': {}
                      'f:name': {}
                      'f:protocol': {}
                    'k:{"containerPort":9095,"protocol":"TCP"}':
                      .: {}
                      'f:containerPort': {}
                      'f:name': {}
                      'f:protocol': {}
                  'f:readinessProbe':
                    'f:httpGet':
                      'f:path': {}
                      'f:port': {}
                    'f:initialDelaySeconds': {}
                    'f:timeoutSeconds': {}
                  'f:resources': {}
                  'f:securityContext':
                    'f:allowPrivilegeEscalation': {}
                    'f:capabilities':
                      'f:drop': {}
                    'f:readOnlyRootFilesystem': {}
                  'f:volumeMounts':
                    'k:{"mountPath":"/etc/loki/config"}':
                      .: {}
                      'f:mountPath': {}
                      'f:name': {}
                    'k:{"mountPath":"/var/loki"}':
                      .: {}
                      'f:mountPath': {}
                      'f:name': {}
              'f:securityContext':
                'f:fsGroup': {}
                'f:runAsGroup': {}
                'f:runAsNonRoot': {}
                'f:runAsUser': {}
              'f:serviceAccountName': {}
              'f:terminationGracePeriodSeconds': {}
              'f:volumes':
                'k:{"name":"config"}':
                  .: {}
                  'f:configMap':
                    'f:name': {}
                  'f:name': {}
          'f:updateStrategy':
            'f:rollingUpdate':
              'f:partition': {}
          'f:volumeClaimTemplates': {}
      manager: argocd-controller
      operation: Apply
      time: '2022-10-28T07:36:52Z'
    - apiVersion: apps/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations': {}
          'f:labels':
            .: {}
            'f:app.kubernetes.io/component': {}
            'f:app.kubernetes.io/managed-by': {}
            'f:app.kubernetes.io/name': {}
            'f:app.kubernetes.io/part-of': {}
            'f:app.kubernetes.io/version': {}
            'f:helm.sh/chart': {}
        'f:spec':
          'f:podManagementPolicy': {}
          'f:replicas': {}
          'f:revisionHistoryLimit': {}
          'f:selector': {}
          'f:serviceName': {}
          'f:template':
            'f:metadata':
              'f:annotations':
                .: {}
                'f:checksum/config': {}
              'f:labels':
                .: {}
                'f:app.kubernetes.io/component': {}
                'f:app.kubernetes.io/instance': {}
                'f:app.kubernetes.io/name': {}
                'f:app.kubernetes.io/part-of': {}
            'f:spec':
              'f:affinity':
                .: {}
                'f:podAntiAffinity':
                  .: {}
                  'f:requiredDuringSchedulingIgnoredDuringExecution': {}
              'f:automountServiceAccountToken': {}
              'f:containers':
                'k:{"name":"write"}':
                  .: {}
                  'f:args': {}
                  'f:env':
                    .: {}
                    'k:{"name":"AWS_ACCESS_KEY_ID"}':
                      .: {}
                      'f:name': {}
                      'f:valueFrom':
                        .: {}
                        'f:secretKeyRef': {}
                    'k:{"name":"AWS_SECRET_ACCESS_KEY"}':
                      .: {}
                      'f:name': {}
                      'f:valueFrom':
                        .: {}
                        'f:secretKeyRef': {}
                  'f:image': {}
                  'f:imagePullPolicy': {}
                  'f:name': {}
                  'f:ports':
                    .: {}
                    'k:{"containerPort":3100,"protocol":"TCP"}':
                      .: {}
                      'f:containerPort': {}
                      'f:name': {}
                      'f:protocol': {}
                    'k:{"containerPort":7946,"protocol":"TCP"}':
                      .: {}
                      'f:containerPort': {}
                      'f:name': {}
                      'f:protocol': {}
                    'k:{"containerPort":9095,"protocol":"TCP"}':
                      .: {}
                      'f:containerPort': {}
                      'f:name': {}
                      'f:protocol': {}
                  'f:readinessProbe':
                    .: {}
                    'f:failureThreshold': {}
                    'f:httpGet':
                      .: {}
                      'f:path': {}
                      'f:port': {}
                      'f:scheme': {}
                    'f:initialDelaySeconds': {}
                    'f:periodSeconds': {}
                    'f:successThreshold': {}
                    'f:timeoutSeconds': {}
                  'f:resources': {}
                  'f:securityContext':
                    .: {}
                    'f:allowPrivilegeEscalation': {}
                    'f:capabilities':
                      .: {}
                      'f:drop': {}
                    'f:readOnlyRootFilesystem': {}
                  'f:terminationMessagePath': {}
                  'f:terminationMessagePolicy': {}
                  'f:volumeMounts':
                    .: {}
                    'k:{"mountPath":"/etc/loki/config"}':
                      .: {}
                      'f:mountPath': {}
                      'f:name': {}
                    'k:{"mountPath":"/var/loki"}':
                      .: {}
                      'f:mountPath': {}
                      'f:name': {}
              'f:dnsPolicy': {}
              'f:restartPolicy': {}
              'f:schedulerName': {}
              'f:securityContext':
                .: {}
                'f:fsGroup': {}
                'f:runAsGroup': {}
                'f:runAsNonRoot': {}
                'f:runAsUser': {}
              'f:serviceAccount': {}
              'f:serviceAccountName': {}
              'f:terminationGracePeriodSeconds': {}
              'f:volumes':
                .: {}
                'k:{"name":"config"}':
                  .: {}
                  'f:configMap':
                    .: {}
                    'f:defaultMode': {}
                    'f:name': {}
                  'f:name': {}
          'f:updateStrategy':
            'f:rollingUpdate':
              .: {}
              'f:partition': {}
            'f:type': {}
      manager: argocd-application-controller
      operation: Update
      time: '2022-10-26T09:13:13Z'
    - apiVersion: apps/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:availableReplicas': {}
          'f:collisionCount': {}
          'f:currentReplicas': {}
          'f:currentRevision': {}
          'f:observedGeneration': {}
          'f:readyReplicas': {}
          'f:replicas': {}
          'f:updateRevision': {}
          'f:updatedReplicas': {}
      manager: kube-controller-manager
      operation: Update
      subresource: status
      time: '2022-10-26T09:18:53Z'
    - apiVersion: apps/v1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
          'f:labels':
            'f:app.kubernetes.io/instance': {}
            'f:argocd.argoproj.io/instance': {}
      manager: argocd-controller
      operation: Update
      time: '2022-10-28T07:19:13Z'
  name: loki-write
  namespace: logging
  resourceVersion: '46346521'
  uid: 159449f2-01c3-4ee1-8b91-2e1e90c1e9eb
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: write
      app.kubernetes.io/instance: loki
      app.kubernetes.io/name: loki
  serviceName: loki-write-headless
  template:
    metadata:
      annotations:
        checksum/config: dc4356fb9c8ae2285982e39f348eaa3087a7bd09084224adb6915903fdf04574
      creationTimestamp: null
      labels:
        app.kubernetes.io/component: write
        app.kubernetes.io/instance: loki
        app.kubernetes.io/name: loki
        app.kubernetes.io/part-of: memberlist
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/component: write
                  app.kubernetes.io/instance: loki
                  app.kubernetes.io/name: loki
              topologyKey: kubernetes.io/hostname
      automountServiceAccountToken: true
      containers:
        - args:
            - '-config.file=/etc/loki/config/config.yaml'
            - '-target=write'
          env:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: AWS_ACCESS_KEY_ID
                  name: loki-s3
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: AWS_SECRET_ACCESS_KEY
                  name: loki-s3
          image: 'docker.io/grafana/loki:2.6.1'
          imagePullPolicy: IfNotPresent
          name: write
          ports:
            - containerPort: 3100
              name: http-metrics
              protocol: TCP
            - containerPort: 9095
              name: grpc
              protocol: TCP
            - containerPort: 7946
              name: http-memberlist
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /ready
              port: http-metrics
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/loki/config
              name: config
            - mountPath: /var/loki
              name: data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 10001
        runAsGroup: 10001
        runAsNonRoot: true
        runAsUser: 10001
      serviceAccount: loki
      serviceAccountName: loki
      terminationGracePeriodSeconds: 300
      volumes:
        - configMap:
            defaultMode: 420
            name: loki
          name: config
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: openebs-hostpath
        volumeMode: Filesystem
      status:
        phase: Pending
status:
  availableReplicas: 3
  collisionCount: 0
  currentReplicas: 3
  currentRevision: loki-write-68f4b7bcfc
  observedGeneration: 1
  readyReplicas: 3
  replicas: 3
  updateRevision: loki-write-68f4b7bcfc
  updatedReplicas: 3

Desired manifest

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/component: write
    app.kubernetes.io/instance: loki
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: loki
    app.kubernetes.io/part-of: memberlist
    app.kubernetes.io/version: 2.6.1
    argocd.argoproj.io/instance: logging
    helm.sh/chart: loki-3.2.0
  name: loki-write
  namespace: logging
spec:
  podManagementPolicy: Parallel
  replicas: 3
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: write
      app.kubernetes.io/instance: loki
      app.kubernetes.io/name: loki
  serviceName: loki-write-headless
  template:
    metadata:
      annotations:
        checksum/config: dc4356fb9c8ae2285982e39f348eaa3087a7bd09084224adb6915903fdf04574
      labels:
        app.kubernetes.io/component: write
        app.kubernetes.io/instance: loki
        app.kubernetes.io/name: loki
        app.kubernetes.io/part-of: memberlist
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/component: write
                  app.kubernetes.io/instance: loki
                  app.kubernetes.io/name: loki
              topologyKey: kubernetes.io/hostname
      automountServiceAccountToken: true
      containers:
        - args:
            - '-config.file=/etc/loki/config/config.yaml'
            - '-target=write'
          env:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: AWS_ACCESS_KEY_ID
                  name: loki-s3
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: AWS_SECRET_ACCESS_KEY
                  name: loki-s3
          image: 'docker.io/grafana/loki:2.6.1'
          imagePullPolicy: IfNotPresent
          name: write
          ports:
            - containerPort: 3100
              name: http-metrics
              protocol: TCP
            - containerPort: 9095
              name: grpc
              protocol: TCP
            - containerPort: 7946
              name: http-memberlist
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /ready
              port: http-metrics
            initialDelaySeconds: 30
            timeoutSeconds: 1
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
          volumeMounts:
            - mountPath: /etc/loki/config
              name: config
            - mountPath: /var/loki
              name: data
      securityContext:
        fsGroup: 10001
        runAsGroup: 10001
        runAsNonRoot: true
        runAsUser: 10001
      serviceAccountName: loki
      terminationGracePeriodSeconds: 300
      volumes:
        - configMap:
            name: loki
          name: config
  updateStrategy:
    rollingUpdate:
      partition: 0
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: openebs-hostpath

I also saw this old issue https://github.com/argoproj/argo-cd/issues/4126 related to what looks like the same problem.

To Reproduce

Install this helm chart: https://github.com/grafana/loki/tree/main/production/helm/loki

Expected behavior

No diff.

Screenshots

198033872-4ca9bfbb-654d-466e-bed0-41888c58d39c

Version

argocd: v2.4.11+3d9e9f2.dirty
  BuildDate: 2022-08-22T19:32:10Z
  GitCommit: 3d9e9f2f95b7801b90377ecfc4073e5f0f07205b
  GitTreeState: dirty
  GoVersion: go1.19
  Compiler: gc
  Platform: darwin/amd64
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
argocd-server: v2.5.0+b895da4

Logs

Paste any relevant application logs here.

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 70
Comments: 23 (5 by maintainers)

Commits related to this issue

Update: disable serversideapply Due to: - https://github.com/argoproj/argo-cd/issues/11143 - https://github.com/argoproj/argo-cd/issues/11074 Signed-off-by: Nicolas Lamirault <nicolas.lamirault@gmai... — committed to portefaix/portefaix-kubernetes by nlamirault 2 years ago
fix(argocd): disable server-side apply Due to argoproj/argo-cd#11143 — committed to hur/homelab by hur a year ago
fix: correct values to fix argo sync Argo would not sync properly unless the appropriate values were defined for the secret - Known issue doc'ed under 'https://github.com/argoproj/argo-cd/issues/11143... — committed to llajas/homelab by llajas a year ago
fix: ignore diffs on statefulset.spec.volumeclaimtemplate This issue will be solved by argoproj/argo-cd#11143 — committed to ismailbay/k8s-dev-stack by ismailbay 6 months ago

Most upvoted comments

Server-side diffing is still in Leo’s queue (he recently built a PoC), so a proper fix is on its way. But I don’t have an ETA yet.

+15

crenshaw-dev on Aug 3, 2023

I hit the same issue today with Loki.

+15

anthonyalayo on May 9, 2023

Talked with Leo about this yesterday. He’s working through how to manage caching the diff.

+13

crenshaw-dev on Oct 24, 2023

Based on https://argo-cd.readthedocs.io/en/stable/user-guide/diffing And a post I saw on a related issue: https://github.com/argoproj/argo-cd/issues/11074#issuecomment-1298825859

This ignore differences block in your application definition for the kube prometheus stack (where you’re server side applying loki) will give you the shiny green checkmark:

ignoreDifferences:
  - group: monitoring.coreos.com
    kind: ServiceMonitor
    jqPathExpressions:
      - '.spec.endpoints[]?.relabelings[]?.action'
  - group: apps
    kind: StatefulSet
    jqPathExpressions:
      - '.spec.volumeClaimTemplates[]?'

For my use case, I am not too worried about Argo “thinking” there is a diff for these statefulsets because we mostly use default settings for the kube prometheus stack. We install it and forget about it.

Others of you may not want this workaround. Really the bug in how Argo is diffing needs addressing.

sgsollie on Aug 3, 2023

+1 I hit the same issue today on PVCs and on the ServiceMonitor CR from the Prometheus stack.

alita1991 on Apr 24, 2023

Server-side diffing is still in Leo’s queue (he recently built a PoC), so a proper fix is on its way. But I don’t have an ETA yet.

Hey @crenshaw-dev thanks for the update on that. Appreciate all your guys efforts.

sgsollie on Aug 3, 2023

Server-side apply doesn’t ask for a conversion, but the apiserver does lots of conversions internally to operate on the statefulset. Without server-side apply, the object is not sent to the server, so nothing happens. With server-side apply, the conversion acts exactly as a default in that case (the conversion defaults these fields), so they end-up appearing in the dry-run output (I don’t know much about how any of this is working, but I suspect the diff is done through a dry-run request)

@apelisse Tks for jumping in the discussion. In the current version of Argo CD Server-Side Apply, we don’t do a dry-run request. We tried to follow the same approach used by CSA aiming for efficiency and avoid sending that additional request to k8s API server. For that, I checked how K8s API server does the conversions and duplicated that logic in Argo CD controller. Unfortunately that is causing issues with default fields. I suspect that it is related with the OpenAPI document that we provide to the managedfields.NewGVKParser() constructor. I couldn’t find a way to retrieve an OpenAPI document that contains the definitions of schemas default values.

Going forward, I believe that dry run is a better way to achieve more accurate diffs as (I believe) it will handle cases where the resource is configured with mutation webhooks. I have an Argo CD proposal to implement dryrun during diff calculation (https://github.com/argoproj/argo-cd/issues/11574). I am currently negotiating time to dedicate on the implementation.

leoluz on Feb 7, 2023

@MohammedNoureldin Yes.

sathieu on Oct 24, 2023

I have got a similar issue with loki, probably related?

MohammedNoureldin on Oct 23, 2023

for now I’m just patching statefulsets on kustomize/helm

  - target:
      kind: StatefulSet
      name: mimir-alertmanager
    patch: |-
      - op: remove
        path: /spec/volumeClaimTemplates
      - op: add
        path: /spec/volumeClaimTemplates
        value:
          - apiVersion: v1
            kind: PersistentVolumeClaim
            metadata:
              name: storage
            spec:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 1Gi
  - target:
      kind: StatefulSet
      name: mimir-compactor
    patch: |-
      - op: remove
        path: /spec/volumeClaimTemplates
      - op: add
        path: /spec/volumeClaimTemplates
        value:
          - apiVersion: v1
            kind: PersistentVolumeClaim
            metadata:
              name: storage
            spec:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 1Gi
  - target:
      kind: StatefulSet
      name: mimir-store-gateway
    patch: |-
      - op: remove
        path: /spec/volumeClaimTemplates
      - op: add
        path: /spec/volumeClaimTemplates
        value:
          - apiVersion: v1
            kind: PersistentVolumeClaim
            metadata:
              name: storage
            spec:
              accessModes:
                - ReadWriteOnce
              resources:
                requests:
                  storage: 1Gi

mtrin on Jun 21, 2023

Same issue with kuik

https://github.com/enix/kube-image-keeper/blob/eab1d909dd058a71154d1aaaf86d62b3f7bbceb4/helm/kube-image-keeper/templates/registry-statefulset.yaml#L108-L110

blackliner on Jun 15, 2023