flux2: Kustomize controller does not detect changes on a ressource

Describe the bug

Changes on a HelmRelease manifest from a Git repo are not applied by the kustomize controller nor being found by flux diff

Steps to reproduce

1. Have a git repo that is configured as a GitRepository source
2. Have a kusomization configured that creates a HelmRelease
3. Change the HelmRelease values section and remove a value from the list (in our case a multiline value)

Expected behavior

Changes are applied by the kustomize controller and the helm release is reconciled.

Screenshots and recordings

No response

OS / Distro

Linux

Flux version

v0.35.0

Flux check

► checking prerequisites ✔ Kubernetes 1.24.6 >=1.20.6-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.25.0 ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.26.0 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.22.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v0.29.0 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v0.27.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v0.30.0 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta1 ✔ buckets.source.toolkit.fluxcd.io/v1beta1 ✔ gitrepositories.source.toolkit.fluxcd.io/v1beta1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta1 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta1 ✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1 ✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1 ✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta1 ✔ receivers.notification.toolkit.fluxcd.io/v1beta1 ✔ all checks passed

Git provider

GitHub (Enterprise)

Container Registry provider

No response

Additional context

The change in the HelmRelease is a removal of a multiline yaml configuration from the values section.

~/k8s$ flux diff kustomization blueprint --path .
✓  Kustomization diffing...

flux diff shows no difference in the current configuration to the applied configuration even.

Currently configured ressource in the cluster:

$ kubectl get helmreleases.helm.toolkit.fluxcd.io  -n blueprint prometheus-msteams -o yaml
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  creationTimestamp: "2022-03-02T16:24:49Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 4
  labels:
    kustomize.toolkit.fluxcd.io/name: blueprint
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: prometheus-msteams
  namespace: blueprint
  resourceVersion: "223641439"
  uid: cb59fdb7-f414-4615-bc8b-a311858525a0
spec:
  chart:
    spec:
      chart: prometheus-msteams
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: prometheus-msteams
        namespace: blueprint
      version: 1.3.1
  dependsOn:
  - name: prometheus-operator
  install:
    remediation:
      retries: 3
  interval: 1m
  releaseName: prometheus-msteams
  values:
    customCardTemplate: '{{ define "teams.card" }} { "@type": "MessageCard", "@context":
      "http://schema.org/extensions", "themeColor": "{{- if eq .Status "resolved"
      -}}2DC72D {{- else if eq .Status "firing" -}} {{- if eq .CommonLabels.severity
      "critical" -}}8C1A1A {{- else if eq .CommonLabels.severity "warning" -}}FFA500
      {{- else -}}808080{{- end -}} {{- else -}}808080{{- end -}}", "summary": "{{-
      if eq .CommonAnnotations.summary "" -}} {{- if eq .CommonAnnotations.message
      "" -}} {{- js .CommonLabels.cluster | reReplaceAll "_" " " | reReplaceAll "-"
      " " | reReplaceAll `\''` "''" -}} {{- else -}} {{- js .CommonAnnotations.message
      | reReplaceAll "_" " " | reReplaceAll "-" " " | reReplaceAll `\''` "''" -}}
      {{- end -}} {{- else -}} {{- js .CommonAnnotations.summary | reReplaceAll "_"
      " " | reReplaceAll "-" " " | reReplaceAll `\''` "''" -}} {{- end -}}", "title":
      "Prometheus Alert ({{ .Status }})", "sections": [ {{$externalUrl := .ExternalURL}}
      {{- range $index, $alert := .Alerts }}{{- if $index }},{{- end }} { "activityTitle":
      "[{{ js $alert.Annotations.description |  reReplaceAll "_" " " | reReplaceAll
      `\''` "''" }}]({{ $externalUrl }})", "facts": [ {{- range $key, $value := $alert.Annotations
      }} { "name": "{{ $key }}", "value": "{{ js $value | reReplaceAll "_" " " | reReplaceAll
      `\''` "''" }}" }, {{- end -}} {{$c := counter}}{{ range $key, $value := $alert.Labels
      }}{{if call $c}},{{ end }} { "name": "{{ $key }}", "value": "{{ js $value |
      reReplaceAll "_" " " | reReplaceAll `\''` "''" }}" } {{- end }} ], "markdown":
      true } {{- end }} ] } {{ end }}'
    metrics:
      serviceMonitor:
        enabled: true
        scrapeInterval: 30s
    replicaCount: 2
    resources:
      limits:
        cpu: 30m
  valuesFrom:
  - kind: ConfigMap
    name: prometheus-msteams-config-values
    optional: true
status:
  conditions:
  - lastTransitionTime: "2023-01-31T08:18:01Z"
    message: Release reconciliation succeeded
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2022-10-05T12:01:05Z"
    message: Helm upgrade succeeded
    reason: UpgradeSucceeded
    status: "True"
    type: Released
  helmChart: blueprint/blueprint-prometheus-msteams
  lastAppliedRevision: 1.3.1
  lastAttemptedRevision: 1.3.1
  lastAttemptedValuesChecksum: 4c81287381ac4d31719d9a83a6711baed5b92daf
  lastReleaseRevision: 3
  observedGeneration: 4

Configuration in the GitRepo:

$ kubectl kustomize . > /tmp/resource.yaml
$ grep -A 26 -B 9 "chart: prometheus-msteams" /tmp/resource.yaml 
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: prometheus-msteams
  namespace: blueprint
spec:
  chart:
    spec:
      chart: prometheus-msteams
      sourceRef:
        kind: HelmRepository
        name: prometheus-msteams
        namespace: blueprint
      version: 1.3.1
  dependsOn:
  - name: prometheus-operator
  install:
    remediation:
      retries: 3
  interval: 1m
  releaseName: prometheus-msteams
  values:
    metrics:
      serviceMonitor:
        enabled: true
        scrapeInterval: 30s
    replicaCount: 2
    resources:
      limits:
        cpu: 30m
  valuesFrom:
  - kind: ConfigMap
    name: prometheus-msteams-config-values
    optional: true
---

As you can see the value “customCardTemplate” is no longer present. However the kustomize controller does not identify any change here.

Code of Conduct

• I agree to follow this project’s Code of Conduct