helm: Nested `null` values don't remove keys as expected

Still exists in v3.9.2

Still exists in 3.11.1 Reproducible only with subcharts

We also hit this issue (similar to one described by @vbuciuc) with not working “null” for values override when subchart is listed as a dependency in requirements.yaml with 3.2.x. Previous versions (3.1.x) work as expected.

Raised #9136

I am experiencing the same problem with 3.4.2 when using subcharts (as described by @BohdanKalytka above). In my case I want to overwrite the securityContext when using the ElasticSearch helm chart in an OpenShift cluster. Is this issue going to be reopened, or should we create a new one?

Hi,

I’m using helm 2.15.0 and I’ve still met this issue.

Given a subchart with values:

securityContext:
  runAsUser: 65534
  fsGroup: 65534

Where the values are used with toYaml

And in a parent chart values:

sub:
  securityContext:
  runAsUser: null

Then the actual output is:

securityContext:
  runAsUser: 65534
  fsGroup: 65534

While it should have been:

securityContext:
  fsGroup: 65534

The issue is still present in 3.4.2, but only in certain situations. I have a chart with multiple sub-charts. When putting default into sub chart values null override does not work. If you move the same values into parent chart values it works as expected. This is reproducible only with sub-charts. When you put it in a single (flat) chart everything works fine.

After moving from helm v2.16.1 to v3.0.2. I have had this issue with annoations unsetting and cpu limits.

I have ran into the exact same issue using 3.0.1, it might be good to reopen this issue ?

Still exist in

HELM_VERSION="v3.6.3"

Can someone reopen the issue and fix it please ?

I just encounter this issue in 2.14.1 and 3.0.2. Any update on this?

still exists in 3.12.3

@bacongobbler @technosophos I’m running into this issue as well with Helm 3.3.4, can you please re-open;

I can confirm that it worked in 3.1.2 and it stopped working after 3.2.x as @paleg mentions. For now, it appears that @tuzla0autopilot4 's workaround of setting it to a non-map value like false will produce the intended behavior but does give output warnings when that is done.

Still seeing this problem with helm 2.16.3 but only when there’s a requirements.yaml listing the subchart as dependency.

boo is the subchart and foo is the parent chart:

vibu@item-ax35755:~/work$ cat boo/values.yaml 
object:
  fromSubchart:
    hello: from boo
vibu@item-ax35755:~/work$ cat foo/values.yaml 
boo:
  object:
    fromParent:
      hello: from foo
    fromSubchart: null
vibu@item-ax35755:~/work$ cat boo/templates/test.yaml 
{{ toYaml .Values.object }}
vibu@item-ax35755:~/work$ cat foo/requirements.yaml 
dependencies:
- name: boo
  repository: file://../boo
  version: 0.1.0

vibu@item-ax35755:~/work/foo$ helm version -c
Client: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}
vibu@item-ax35755:~/work/foo$ helm dep up
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.
Saving 1 charts
Deleting outdated charts

vibu@item-ax35755:~/work/foo$ helm template .
---
# Source: foo/charts/boo/templates/test.yaml
fromParent:
  hello: from foo
fromSubchart:
  hello: from boo

  
vibu@item-ax35755:~/work/foo$ mv requirements.yaml{,.bak}
vibu@item-ax35755:~/work/foo$ helm template .
---
# Source: foo/charts/boo/templates/test.yaml
fromParent:
  hello: from foo

Both charts here:

helm_5184.zip

In fact, I just upgraded 2.14.0 => 2.14.2. Not only does the nulled key still exist, but it also contains the previous value. The previous behavior just made it null, so I believe this has actually regressed.

I’m also encountering this exact issue with the filebeat chart.

@cdenneen You can get around file output specifically with config.output.file.enabled=false.

I’m encountering an issue where I want to use the new inputs key for filebeat but cannot remove the prospectors key.

Existing values:

config:
  filebeat.prospectors:
    - type: log
      enabled: true
      paths:
        - /var/log/*.log
        - /var/log/messages
        - /var/log/syslog

My override: --set config.filebeat.prospectors=null

Result: (config is used to set a Secret value)

filebeat:
  prospectors: null

still exists in 3.12.3

While I was unable to override a default subcharted value with foo: nil, I was able to override with foo: "".

Please give #7743 a try. It looks like the commits made to Helm 2 were not ported over to Helm 3, which is why many users would be seeing this behaviour over there.

I have set values to null (and left them null in my values files) and ignore the long list of table warnings which is OK for now but I’d really just like to do a one-time cleanup of the old incorrect values stored in kubernetes. Until this value deletion bug is fixed, is there a manual way to edit the values on an existing deployment with no downtime?

@bacongobbler , while I haven’t explictly tested 2.14.0, what @sgillespie matches the behaviour I referred to in https://github.com/helm/helm/issues/5184#issuecomment-517059748. Sadly I think it’s a case of the unit tests passing, but the end to end failing - as the individual components are being used in series, which makes removing a key in an earlier stage makes it have no effect in later stages (and is why the original values are coming through).

I agree it’s a regression, though backing it out is also a regression for anyone relying on the new behaviour.

I’ve had a quick go at a relatively minor patch which I think will reduce the impact (and fixes the test I added above) in #6146 - I’m sorry we got this wrong the in last attempt.

stable/filebeat

With null or nil I’m seeing the same thing… the key isn’t removed… it’s overwritten as null/nil but not removed and when you are replacing it with different key it causes problem:

---values.yml in the Chart
output.file:
  path: /var/log/foo.log

---my overrides
output.elasticsearch:
   hosts:
      - 'http://localhost:9200'
output.file: null

or

---my overrides
output:
  elasticsearch:
     host:
      - 'http://localhost:9200'
  file: null

or

---my overrides
output:
  elasticsearch:
     hosts:
      - 'http://localhost:9200'
output.file: null

» k get secret filebeat -o jsonpath="{.data.filebeat\.yml}" | base64 -D
    filebeat.config:
      modules:
        path: ${path.config}/modules.d/*.yml
        reload.enabled: false
      prospectors:
        path: ${path.config}/prospectors.d/*.yml
        reload.enabled: false
    filebeat.prospectors:
    - enabled: true
      fields:
        apenv: dev
        app: kubernetes
        log_category: kubernetes
      fields_under_root: true
      paths:
      - /var/log/*.log
      - /var/log/messages
      - /var/log/syslog
      type: log
    - containers.ids:
      - '*'
      fields:
        apenv: dev
        app: kubernetes
        log_category: kubernetes
      fields_under_root: true
      processors:
      - add_kubernetes_metadata:
          in_cluster: true
      - drop_event:
          when:
            equals:
              kubernetes.container.name: filebeat
      type: docker
    http.enabled: true
    http.port: 5066
    output:
      elasticsearch:
        hosts:
        - http://localhost9200
    output.file: null
    processors:
    - add_cloud_metadata: null

pod errors as:

Exiting: error unpacking config data: more than one namespace configured accessing 'output' (source:'filebeat.yml')

repro: pyroscope-folder.zip fsGroup and runAsUser null values are not removed

helm upgrade release-1 pyroscope-folder  --dry-run  --install > run1   
helm upgrade release-1 pyroscope-folder --values pyroscope-folder/values.yaml --dry-run  --install > run2 

diff -u run1 run2
--- run1        2023-09-12 18:26:31.238211712 +0700
+++ run2        2023-09-12 18:27:28.071648302 +0700
@@ -1,6 +1,6 @@
 Release "release-1" does not exist. Installing it now.
 NAME: release-1
-LAST DEPLOYED: Tue Sep 12 18:26:26 2023
+LAST DEPLOYED: Tue Sep 12 18:26:51 2023
 NAMESPACE: default
 STATUS: pending-install
 REVISION: 1
@@ -194,9 +194,7 @@
     spec:
       serviceAccountName: release-1-pyroscope
       securityContext:
-        fsGroup: 10001
         runAsNonRoot: true
-        runAsUser: 10001
       containers:
         - name: "pyroscope"
           securityContext:

Running into the same issue with liveness and readiness probes. Deleting Default Key from the template guide does not work.

This might’ve been fixed in https://github.com/helm/helm/pull/6080. That PR fixes the case where coalesceDeps is called twice. Can someone confirm with master?

If so, can #6146 be closed, or is it attempting to fix a separate issue? I’m trying to grasp my head around the problem space here but it’s unclear what these PRs are attempting to solve. Sorry.