helm: Helm 3 - upgrade nginx - spec.clusterIP: Invalid value: "": field is immutable

I have the same Problem, even without setting the service type or clusterIP with helm v3.0.0-rc.2 if i use the --force option with the helm update --install command. Without the --force it works fine

@zen4ever nailed the issue in https://github.com/helm/helm/issues/6378#issuecomment-532766512. I’ll try to explain it in more detail…

As others have pointed out, the issue arises when a chart defines a clusterIP with an empty string. When the Service is installed, Kubernetes populates this field with the clusterIP it assigned to the Service.

When helm upgrade is invoked, the chart asked for the clusterIP to be removed, hence why the error message is spec.clusterIP: Invalid value: "": field is immutable.

This happens because of the following behaviour:

1. On install, the chart specified it wanted the clusterIP to be an empty string
2. Kubernetes auto-assigned the Service a clusterIP. We’ll use 172.17.0.1 for this example
3. On helm upgrade, the chart wants the clusterIP to be an empty string (or in @zen4ever’s case above, it is omitted)

When generating the three-way patch, it sees that the old state was "", live state is currently at "172.17.0.1", and proposed state is "". Helm detected that the user requested to change the clusterIP from “172.17.0.1” to “”, so it supplied a patch.

In Helm 2, it ignored the live state, so it saw no change (old state: clusterIP: "" to new state: clusterIP: ""), and no patch was generated, bypassing this behaviour.

My recommendation would be to change the template output. If no clusterIP is being provided as a value, then don’t set the value to an empty string… Omit the field entirely.

e.g. in the case of stable/nginx-ingress:

spec:
{{- if not .Values.controller.service.omitClusterIP }}
  clusterIP: "{{ .Values.controller.service.clusterIP }}"
{{- end }}

Should be changed to:

spec:
{{- if not .Values.controller.service.omitClusterIP }}
  {{ with .Values.controller.service.clusterIP }}clusterIP: {{ quote . }}{{ end }}
{{- end }}

This is also why --set controller.service.omitClusterIP=true works in this case.

TL;DR don’t do this in your Service templates:

clusterIP: ""

Otherwise, Helm will try to change the service’s clusterIP from an auto-generated IP address to the empty string, hence the error message.

Hope this helps!

We’re running into the same issue:

apiVersion: v1
kind: Service
{{ include "mde.metadata" $ }}
spec:
  ports:
  - name: {{ include "mde.portName" $ | quote }}
    port: {{ include "mde.port" $ }}
    protocol: TCP
    targetPort: {{ include "mde.port" $ }}
  selector:
    app: {{ include "mde.name" $ }}
  sessionAffinity: None
  type: ClusterIP

spec.clusterIP is not part of the Service template, yet with Helm 3.0.2 and a helm upgrade ... --force --install call, we’re also seeing:

Error: UPGRADE FAILED: failed to replace object: Service “dummy” is invalid: spec.clusterIP: Invalid value: “”: field is immutable

Please re-open.

Hey @bacongobbler , we faced with the same issue during migrate helm v2 release to helm v3. We use type: ClusterIP in Service but omit ClusterIP at all and we get:

Error: UPGRADE FAILED: failed to replace object: Service "dummy" is invalid: spec.clusterIP: Invalid value: "": field is immutable

We don’t have spec.clusterIP: in our helm template but we got this Error after migrate release via helm 2to3

Service template:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}
  labels:
    app: {{ .Values.image.name }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "-" }}
    cluster: {{ default "unknown" .Values.cluster }}
    region: {{ default "unknown" .Values.region }}
    datacenter: {{ default "unknown" .Values.datacenter }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  type: ClusterIP
  ports:
    - port: {{ .Values.service.port }}
      targetPort: {{ .Values.service.port }}
      protocol: TCP
      name: http
  selector:
    app: {{ .Values.image.name }}
    release: {{ .Release.Name }}

We have the same problem. We don’t define clusterIP at all in our chart and it is not present in the final template. However, we still get the same error and only with --force flag.

Please reopen. Issue is not fixed. This “hack” suggested by nasseemkullah is not appropriate. Don’t ask people to jump on heads. Just fix it. Very poor migration path. Helm sucks.

Hey all, please be aware that the related issue in Kubernetes was accepted as a bug

FYI @bacongobbler

https://github.com/kubernetes/kubernetes/issues/91459

I think this is issue with helm with --force option during upgrade. Helm is trying to recreate service but it also replace spec.clusterIP so it throw error. I can confirm this using my own custom chart. Error: UPGRADE FAILED: failed to replace object: Service "litespeed" is invalid: spec.clusterIP: Invalid value: "": field is immutable

I believe this is an issue with the nginx-ingress chart, not helm3. By default, the chart will always try to pass controller.service.clusterIP = "" and defaultBackend.service.clusterIP = "" unless you set controller.service.omitClusterIP=true and defaultBackend.service.omitClusterIP=true.

link to sources: https://github.com/helm/charts/blob/master/stable/nginx-ingress/values.yaml#L321 https://github.com/helm/charts/blob/master/stable/nginx-ingress/templates/controller-service.yaml#L22

workaround:

$ helm upgrade --install ingress-test stable/nginx-ingress --set controller.service.omitClusterIP=true --set defaultBackend.service.omitClusterIP=true

According to my understanding this a problem with Kubernetes, because “forcefully overwriting” does not behave the same way as “deleting and recreating again”. Is there any upstream bug?

On the other hand, Helm is also misleading, because --force is described as “force resource updates through a replacement strategy”. While in reality it does not do any replacement, it just attempts to forcefully overwrite resources (it would be better to name the flag --force-overwrite). Forceful replacement would look like deleting and recreating again (there could be a flag --force-recreate). Of course, --force-recreate could be a bit dangerous to use for some resources, but it would always succeed.

Anyway, Helm could implement a fallback workaround for such type of issues. If the current behavior (described as --force-overwrite) fails and detects an immutable field error, it should delete and recreate the resource (as --force-recreate).

@johannges, I was just about to post the same. 👍

Messing around with this some more, I’ve observed that:

• helm upgrade ... --force --install - results in The Service “dummy” is invalid:spec.clusterIP: Invalid value: “”: field is immutable
• helm template ... | kubectl apply -f - - works
• helm template ... | kubectl replace -f - - results in The Service “dummy” is invalid:spec.clusterIP: Invalid value: “”: field is immutable
• helm template ... | kubectl replace --force -f - - works

kubectl version - 1.14.6 helm version - 3.0.2

same here- we dont define ClusterIP anywhere but still see the error

Hey @bacongobbler , we faced with the same issue during migrate helm v2 release to helm v3. We use type: ClusterIP in Service but omit ClusterIP at all and we get:

Error: UPGRADE FAILED: failed to replace object: Service "dummy" is invalid: spec.clusterIP: Invalid value: "": field is immutable

We don’t have spec.clusterIP: in our helm template but we got this Error after migrate release via helm 2to3

Service template:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}
  labels:
    app: {{ .Values.image.name }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "-" }}
    cluster: {{ default "unknown" .Values.cluster }}
    region: {{ default "unknown" .Values.region }}
    datacenter: {{ default "unknown" .Values.datacenter }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  type: ClusterIP
  ports:
    - port: {{ .Values.service.port }}
      targetPort: {{ .Values.service.port }}
      protocol: TCP
      name: http
  selector:
    app: {{ .Values.image.name }}
    release: {{ .Release.Name }}

@bacongobbler I’ve got to this thread after checking https://github.com/helm/helm/issues/7956

As with all previous commenters: we don’t have “clusterIP” in templates at all, but error is still present with latest Helm if --force flag is used.

Helm version: 3.4.1

“helm -n kube-system get manifest CHART_NAME | grep clusterIP” shows no results.

Error:

field is immutable && failed to replace object: Service "SERVICE_NAME" is invalid: spec.clusterIP: Invalid value: "": field is immutable

@antonakv what a way to start the year 😃 I think in general we are playing with fire when providing clusterIP as a configurable value in a chart, and cannot totally blame one tool/person/PR in particular. If clusterIP needs to be a configurable value, by default it should not be in the rendered template, that’s the idea of my commenting out in the values files as per https://github.com/helm/charts/blob/270172836fd8cf56d787cf7d04d938856de0c794/stable/nginx-ingress/values.yaml#L236

This, if I’m not mistaken, should prevent anby future headaches for those that install the chart as of that change. But for those of us (myself included) who had installed it prior, and then migrated to helm3, I’m afraid hardcording the current clusterIP values in our values files OR uninstalling and reinstalling the chart (causes downtime!) are the only options I see.

Opinions are my own, I am not paid to work on helm, just an end user like you. Those who are paid to work on this full time may be able to provide more insight.

Happy new year and good luck! Don’t give up on helm, together we can make it better.

As far as Kubernetes is concerned: yes.

I have the same Problem, even without setting the service type or clusterIP with helm v3.0.0-rc.2 if i use the --force option with the helm update --install command. Without the --force it works fine

Cool! I inspired from your answer, that i have to comment force: .. line in helmfile yaml :

helmDefaults:
  tillerless: true
  verify: false
  wait: true
  timeout: 600
  # force: true <---- THI ONE IS COMMENTED

It works 🎉

Looks like Microsoft is mentor of project. I see style. 😃

As a temporary solution if you’re trying to get this to work for now while this issue gets resolved I found if I did the following I was able to perform an update:

1. Get the clusterIP values for the controller and the default-backend via: kubectl get svc | grep ingress
2. Add the following overwrites to your existing helm values file:

     controller:
       service:
         clusterIP: <cluster-ip-address-for-controller>
     defaultBackend:
       service:
         clusterIP: <cluster-ip-address-for-default-backend>
   

3. Perform the update.

I’ve tested this for a cluster I’m running and it didn’t require any recreation.

@davidfernandezm did you ever find a solution for this? I’m seeing the same on my end and my services are defined exactly as yours are. No option for clusterIP is being set, and yet Helm still fails on an upgrade.

@zen4ever nailed the issue in #6378 (comment). I’ll try to explain it in more detail…

As others have pointed out, the issue arises when a chart defines a clusterIP with an empty string. When the Service is installed, Kubernetes populates this field with the clusterIP it assigned to the Service.

When helm upgrade is invoked, the chart asked for the clusterIP to be removed, hence why the error message is spec.clusterIP: Invalid value: "": field is immutable.

This happens because of the following behaviour:

1. On install, the chart specified it wanted the clusterIP to be an empty string
2. Kubernetes auto-assigned the Service a clusterIP. We’ll use 172.17.0.1 for this example
3. On helm upgrade, the chart wants the clusterIP to be an empty string (or in @zen4ever’s case above, it is omitted)

When generating the three-way patch, it sees that the old state was "", live state is currently at "172.17.0.1", and proposed state is "". Helm detected that the user requested to change the clusterIP from “172.17.0.1” to “”, so it supplied a patch.

In Helm 2, it ignored the live state, so it saw no change (old state: clusterIP: "" to new state: clusterIP: ""), and no patch was generated, bypassing this behaviour.

My recommendation would be to change the template output. If no clusterIP is being provided as a value, then don’t set the value to an empty string… Omit the field entirely.

e.g. in the case of stable/nginx-ingress:

spec:
{{- if not .Values.controller.service.omitClusterIP }}
  clusterIP: "{{ .Values.controller.service.clusterIP }}"
{{- end }}

Should be changed to:

spec:
{{- if not .Values.controller.service.omitClusterIP }}
  {{ with .Values.controller.service.clusterIP }}clusterIP: {{ quote . }}{{ end }}
{{- end }}

Hi @bacongobbler, I think since if no value is provided we will still wind up with clusterIP: "" … better would be value clusterIP: "" completely commented out in the values file. This omits it from rendered manifests when set and should save future headaches. However if using helm3 and current helm state has clusterIP: "" set, one needs to hardcode the clusterIP addresses in values files.

This is also why --set controller.service.omitClusterIP=true works in this case.

TL;DR don’t do this in your Service templates:

clusterIP: ""

Otherwise, Helm will try to change the service’s clusterIP from an auto-generated IP address to the empty string, hence the error message.

Hope this helps!

@bacongobbler I am really sorry if I miss something here, maybe I simply don’t know enough about Helm internals.

“My recommendation would be to change the template output. If no clusterIP is being provided as a value, then don’t set the value to an empty string… Omit the field entirely.”

So what is the solution? Does that mean that “–force” flag can’t be used at all if clusterIP field is not set to some static value?

If anyone is experiencing symptoms that are not a result of the explanation provided in https://github.com/helm/helm/issues/6378#issuecomment-557746499, can you please open a new issue with your findings and how we can reproduce it on our end?

The issue raised by the OP was because of the scenario provided above, where a chart set the ClusterIP to an empty string on install. It is entirely possible that there are other scenarios where this particular case can crop up, such as others have mentioned with the use of the --force flag. Those cases should be discussed separately, as the diagnosis and solution may differ than the advice provided earlier.

Thank you!

I have the same Problem, even without setting the service type or clusterIP with helm v3.0.0-rc.2 if i use the --force option with the helm update --install command. Without the --force it works fine

Best solution, it works for me, thank you!

@antonakv this issue duplicate of 7956 @bacongobbler more info

@mssachan see #7082 and the draft proposal in #7431 for your use case. That proposal aims to implement kubectl replace —force, which would be similar to Helm 2’s helm install —force‘s behaviour.

@juan131 @Ronsevet: remove --force The meaning changed.

In case anyone ends up here using helm v3 via terraform, since you cant directly tell it not to use --force i had success manually deleting the chart using helm delete then re-running terraform. this sucks but it does work.

edit: the whole error: ( “nginx-ingress-singleton-controller” is the release name i set. it has no specific meaning )

Error: cannot patch "nginx-ingress-singleton-controller" with kind Service: Service "nginx-ingress-singleton-controller" is invalid: spec.clusterIP: Invalid value:
"": field is immutable && cannot patch "nginx-ingress-singleton-default-backend" with kind Service: Service "nginx-ingress-singleton-default-backend" is invalid: sp
ec.clusterIP: Invalid value: "": field is immutable

  on .terraform/modules/app_dev/nginx-ingress.tf line 1, in resource "helm_release" "nginx_ingress":
   1: resource "helm_release" "nginx_ingress" {

I was encountering this error as well for existing deployed kafka and redis chart releases. Removing --force did indeed resolve this.

Now I’m getting a new error from the redis release: Error: UPGRADE FAILED: release redis failed, and has been rolled back due to atomic being set: cannot patch "redis-master" with kind StatefulSet: StatefulSet.apps "redis-master" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

Agreed with @bacongobbler that this looks related to the Helm v3 three-way merge patch strategy that’s likely resulting in passing in fields (even with the same values as before) to the update/patch that Kubernetes considers immutable/unchangeable after first creation.