helm: Nested `null` values do not remove keys from sub-charts

This issue is such pain in the ass. Please fix it asap.

I went back through all the Helm 3 versions to confirm the behavior in each version. It wasn’t clear to me if this was a one-off change, or a series of fixes and regressions with varying semantics. I didn’t check RC, beta or alpha releases.

Test

Confirm removal of values in a subchart defined in Chart.yaml dependencies from inside a parent chart.

Specifically, the expected behavior tested is the ability to override a map or scalar referenced in a subchart with a null value in the parent chart.

Summary

It seems to have been a problem since Helm 3.2.0 - which folks have previously suggested.

Notes

In early versions (v3.0.0-v3.1.3) Helm would behave as expected when replacing maps with null, although it would render a warning coalesce.go:196: warning: cannot overwrite table with non table for <VALUE>. Replacing scalars with a null value worked correctly with no errors.

In versions since v3.2.0, removing neither maps nor scalars works as expected. Using other falsy scalar values works as expected, but isn’t always appropriate depending on the specific subchart.

Example: Child chart

# templates/deployment.yaml
{{- if .Values.securityContext }}
securityContext: {{- toYaml .Values.securityContext | nindent 2 }}
{{- end }}

# values.yaml
securityContext:
  runAsUser: 999

Example: Parent chart

# Chart.yaml
dependencies:
  - name: subchart
    version: "0.1.0"

# values.yaml
subchart:
  securityContext: ~

Outcome

Version	Behavior
v3.7.2	Unexpected
v3.7.1	Unexpected
v3.7.0	Unexpected
v3.6.3	Unexpected
v3.6.2	Unexpected
v3.6.1	Unexpected
v3.6.0	Unexpected
v3.5.4	Unexpected
v3.5.3	Unexpected
v3.5.2	Unexpected
v3.5.1	Unexpected
v3.5.0	Unexpected
v3.4.2	Unexpected
v3.4.1	Unexpected
v3.4.0	Unexpected
v3.3.4	Unexpected
v3.3.3	Unexpected
v3.3.2	Unexpected
v3.3.1	Unexpected
v3.3.0	Unexpected
v3.2.4	Unexpected
v3.2.3	Unexpected
v3.2.2	Unexpected
v3.2.1	Unexpected
v3.2.0	Unexpected
v3.1.3	Expected
v3.1.2	Expected
v3.1.1	Expected
v3.1.0	Expected
v3.0.3	Expected
v3.0.2	Expected
v3.0.1	Expected
v3.0.0	Expected

Script

for version in helm-*amd64.tar.gz; do 
  rm -rf darwin-amd64 2>/dev/null; 
  tar zxf $version; 
  darwin-amd64/helm version; 
  darwin-amd64/helm template PARENT | grep securityContext && echo Unexpected; 
 done

@joejulian - I believe https://github.com/helm/helm/pull/11440 should fix this bug - looks like an bug where null keys are preemptively removed before subcharts are processed which causes the subcharts to merge their keys back into the parent chart again.

This is quite critical… Just thinking about the massive number of workarounds and pseudo properties to control what’s rendered just because it doesn’t work

@pmorch sorry, it’s been a sec since I’ve been working on this. As I recall, the subchart inheritance specifically is the part that doesn’t work. If you instead manually set a field to null either with a --set flag or a -f flag in the helm install command, it does work.

If you want to write your values.yaml as if this bug doesn’t exist, and then just include -f values.yaml in all your helm install commands, that would be the workaround.

Just hit this one myself using Helm 3.7.0. The issue definitely still exists, and the above (annoying) workaround does work. It seems like a pretty basic use case.

This is still present in 3.7.1. Using false works in some contexts.

argo-cd:
  redis:
    securityContext: false

Still the same problem in Helm 3.7.0

Assigning false is not a workaround, it just passes the false value through to the resource. I guess that might work sometimes, but otherwise it’ll just fail ex: got "boolean", expected "map"

Helm v3.9.3

It would be nice if we see a fix for this issue that also exists in helm 3.9.0

This wokraround worked for me luckly…

# values.yaml
subchart:
  securityContext: false 

and …

it doen`t work if the subchart has a schema json… 😦

That workaround does not work in most of the cases we need (like trying to set null to securityContext in bitnami charts because of openshift).

For the general pattern like this, using false works, although Helm complains about replacing a table with a boolean.

      {{- if .Values.redis.securityContext }}
      securityContext: {{- toYaml .Values.redis.securityContext | nindent 8 }}
      {{- end }}

Helm says it “cannot” replace the table, but ultimately does.

warning: cannot overwrite table with non table …

Some Workarounds

1. Use false instead of ~, null or empty
2. Remove dependencies from Chart.yaml after installing them into charts directory (although you may then have issues with sub-chart dependency conditions)
3. Explicitly reference parent chart values.yaml e.g. helm install NAME CHART -f CHART/values.yaml (even though it should be redundant)
4. Use --set subchart.nested.value=false(although 3 is probably generally better)

Same problem with helm 3.6.0. Any update on this matter?

Looking at this.

For what it’s worth I’ve found a possible workaround, or at least another bug but it helped me get past this.

Basically use 2 values files.

Given a default values.yaml in a dependent chart of:

# values.yaml
podDisruptionBudget:
  enabled: false
  minAvailable: 1
  # maxUnavailable: 1

Overriding that with another values file like this, did NOT achieve the null override, and the default 1 was used.

# dev-values.yaml
podDisruptionBudget:
  enabled: true
  minAvailable: null
  maxUnavailable: 1

But when I had a 2nd values override file passed into helm it worked.

# dev-further-override-values.yaml
podDisruptionBudget:
  enabled: true
  minAvailable: null
  maxUnavailable: 1

No idea why this behavior would be any different, and this is with ArgoCD v2.4.3 which i believe used Helm v3.8.1.

How many years to wait for given PR to be merged?!

Any updates on this issue?

I’m trying to install an application via ArgoCD and I’m listing the application’s chart as a dependency i.e. subchart. As expected, ArgoCD doesn’t translate the livenessProbe: {} present in the subchart’s values because of this issue.

@mattfarina In case you need an easy reproducible case for this bug to be labeled as confirmed:

I’m having the same issue with the Bitnami Redis chart when trying to set the podDisruptionBudget.maxUnavailable key. For this to work you need to be able to unset the minAvailable key as it’s otherwise not a valid PDB spec.

So with a parent chart like this:

Chart.yaml

apiVersion: v2
name: redis
version: 0.0.1
dependencies:
  - name: redis
    version: 10.7.16
    repository: https://charts.bitnami.com/bitnami

values.yaml

redis:
  podDisruptionBudget:
    enabled: true
    minAvailable: null    # THIS DOES NOT WORK
    maxUnavailable: 1

Output:

% helm template redis . -s charts/redis/templates/pdb.yaml | kubectl apply --dry-run=server -f -
The PodDisruptionBudget "redis" is invalid: spec: Invalid value: policy.PodDisruptionBudgetSpec{MinAvailable:(*intstr.IntOrString)(0xc038134280), Selector:(*v1.LabelSelector)(0xc0381342a0), MaxUnavailable:(*intstr.IntOrString)(0xc0381342e0)}: minAvailable and maxUnavailable cannot be both set

Versions used:

% helm version
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.15.6"}
% kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"archive", BuildDate:"2020-11-25T13:19:56Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.15-gke.6000", GitCommit:"b02f5ea6726390a4b19d06fa9022981750af2bbc", GitTreeState:"clean", BuildDate:"2020-11-18T09:16:22Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

@RytisLT is this available to vanilla Helm or do we need to use Flux?

@pmorch Sorry if my comment was unclear–not in the subchart itself, in the helm command when you apply the chart, you can specify a values.yaml file. According to this documentation, the -f flag specifies a file, e.g.:

helm install -f myvalues.yaml myredis ./redis

I’m seeing this issue in helm 3.5.4 as well. My current workaround is to reassert the values.yaml file in my helm command with -f ./values.yaml, but this is a bit unwieldy.

Confirmed. I am getting the same issue with my helm chart https://github.com/kintoproj/kinto-helm Right now, the only workaround I found is to pass it as a --set parameter. Pretty annoying.

@clayvan could you please write the exact steps to reproduce the workaround you mentioned?

I was not able to reproduce it, neither with Helm v3.8.1 nor v3.9.2 (latest).

That behavior actually could be something related to ArgoCD itself not Helm.

for those using flux to sync helmreleases there is no workaround unfortunately at the moment

I’ve managed to work-around the issue using postRederers. I’ve removed unwanted volumeClaimTemplates like this:

postRenderers:
  - kustomize:
      patchesJson6902:
        - target:
            group: apps
            version: v1
            kind: StatefulSet
            name: filerun-v1-mariadb
          patch:
            - op: remove
              path: /spec/volumeClaimTemplates

@giselleserate-okta I’m sorry, I still don’t get it. The entire problem is that a field: null in myvalues.yaml doesn’t work. How is your command a workaround? What do you put in myvalues.yaml and how does that remove a field that is set in the ./redis chart?

for those using flux to sync helmreleases there is no workaround unfortunately at the moment

Reproduction here: https://github.com/karolinepauls/helm-unset-dep-value-bug-repro - you can use it to confirm the issue, since it’s currently unconfirmed.

Relates to https://github.com/helm/helm/issues/1966.

I believe there are 2 similar issues preventing this from working, modifiying the chart object makes calls to CoalesceValues non-idempotent:

1. See here - The call to dependencies.processImportValues is overwriting the null values in the subchart so subsequent calls to CoalesceValues do not remove the default key.

2. See here - The reference to the value is copied directly to the values map, subsequent modifications of this object are applied to the chart, so subsequent calls to CoalesceValues do not remove the default key