kubernetes: applying a service twice with clusterIP set to blank returns an error

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:

  1. Create a small file service.yml with a Service in it, and clusterIP: ""
  2. kubectl apply -f service.yml - the service is created
  3. kubectl apply -f service.yml - get an error

Then error is the classic:

F0906 19:32:48.426166   87883 helpers.go:119] The Service "myservice" is invalid: spec.clusterIP: Invalid value: "": field is immutable

I did check the annotation kubectl.kubernetes.io/last-applied-configuration to see that it has the clusterIP: "", which it does. Nonetheless, it treats it as mutated.

In short: kubectl apply -f service.yml done twice should always work, since the file has not changed.

What you expected to happen:

The service to be unchanged.

How to reproduce it (as minimally and precisely as possible):

See steps above.

Anything else we need to know?:

It also occurs when the first apply does not set clusterIP and the second one sets it to "" or vice-versa, see #65272 . This is more basic, though.

Environment:

  • Kubernetes version (use kubectl version): tried with client 1.9.x through 1.11.x, server 1.9.x through 1.10.x
  • Cloud provider or hardware configuration: tried bare metal, on EC2, and docker for mac
  • OS (e.g. from /etc/os-release): linuxkit (docker for Mac), CoreOS on bare metal and EC2

About this issue

  • Original URL
  • State: closed
  • Created 6 years ago
  • Reactions: 1
  • Comments: 20 (15 by maintainers)

Most upvoted comments

The behavior that makes that acceptable on create and not on update is part of the service rest handler, which allocates cluster IP on creation of the service and does not permit clearing it via update.

Understood from the technical perspective.

From a usage perspective (the important part), it creates hurdles and makes automation difficult. At the least, I think we should deprecate the clusterIP: "" if you cannot use it twice.

This also creates a problem using the Kubernetes API with something like Google Cloud Deployment Manager (GCDM). Minimum steps to demonstrate the problem:

Create a minimal type provider for the GKE API

Create the file '/type-provider-minimal.jinja`:

resources:
- name: api-v1-tp
  type: deploymentmanager.v2beta.typeProvider
  properties:
    options:
      validationOptions:
        schemaValidation: IGNORE_WITH_WARNINGS
      inputMappings:
      - fieldName: Authorization
        location: HEADER
        value: >
          $.concat("Bearer ", $.googleOauth2AccessToken())
    descriptorUrl: https://{{ properties["k8s_endpoint"] }}/swaggerapi/api/v1
- name: apis-apps-v1-tp
  type: deploymentmanager.v2beta.typeProvider
  properties:
    options:
      validationOptions:
        schemaValidation: IGNORE_WITH_WARNINGS
      inputMappings:
      - fieldName: Authorization
        location: HEADER
        value: >
          $.concat("Bearer ", $.googleOauth2AccessToken())
    descriptorUrl: https://{{ properties["k8s_endpoint"] }}/swaggerapi/apis/apps/v1

and deploy it with GCDM:

 gcloud deployment-manager deployments create csh-type-provider-minimal --template=type-provider-minimal.jinja --properties=k8s_endpoint:XXX.XXX.XXX.XXX

(here XXX.XXX.XXX.XXX is the IP address of our Kubernetes master).

Create a service

Create the file test-svc:

{% set APIS_APPS_V1_CLUSTER_TYPE = env['project'] + '/apis-apps-v1-tp' %}

{% set SERVICE_COLLECTION = '/api/v1/namespaces/{namespace}/services' %}

{% set DEPLOYMENT_COLLECTION = '/apis/apps/v1/namespaces/{namespace}/deployments' %} 

resources:
- name: test-svc
  type: {{ API_V1_CLUSTER_TYPE }}:{{ SERVICE_COLLECTION }}
  properties:
    apiVersion: v1
    kind: Service
    namespace: default
    metadata:
      name: test-svc
    spec:
      ports:
      - name: port0
        port: 8090
        protocol: TCP
        targetPort: 8080
      selector:
        app: test-app
- name: test-deployment
  type: {{ APIS_APPS_V1_CLUSTER_TYPE }}:{{ DEPLOYMENT_COLLECTION }}
  properties:
    apiVersion: apps/v1
    kind: Deployment
    namespace: default
    metadata:
      name: test-deployment
      labels:
        app: test-app
    spec:
      selector:
        matchLabels:
          app: test-app
      template:
        metadata:
          labels:
            app: test-app
        spec:
          containers:
          - image: gcr.io/hello-minikube-zero-install/hello-node
            name: test-app

and deploy it:

gcloud deployment-manager deployments create csh-test --template=test.jinja

Run kubectl exec -ti <pod-name> /bin/bash to log into a container in the default namespace and then run curl http://test-svc:8090/ to check that Hello World is displayed. So far, so good.

Updating the service

Now modify the service, e.g. copy the file test.jinja to test-modified.jinja and add an extra port to test-svc:

    spec:
      ports:
        ...
      - name: port1
        port: 8091
        protocol: TCP
        targetPort: 8080

Now update the service:

gcloud deployment-manager deployments update csh-test --template=forGoogle/test-modified.jinja
The fingerprint of the deployment is 23G7_WkD_3I4q00Ntp_aSg==
Waiting for update [operation-1549451161859-58137b619779e-cb2b19a1-71726f37]...failed.                                                                                                                                  
ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1549451161859-58137b619779e-cb2b19a1-71726f37]: errors:
- code: RESOURCE_ERROR
  location: /deployments/csh-test/resources/test-svc
  message: '{"ResourceType":"ol-stag/api-v1-tp:/api/v1/namespaces/{namespace}/services","ResourceErrorCode":"422","ResourceErrorMessage":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Service
    \"test-svc\" is invalid: [metadata.resourceVersion: Invalid value: \"\": must
    be specified for an update, spec.clusterIP: Invalid value: \"\": field is immutable]","reason":"Invalid","details":{"name":"test-svc","kind":"Service","causes":[{"reason":"FieldValueInvalid","message":"Invalid
    value: \"\": must be specified for an update","field":"metadata.resourceVersion"},{"reason":"FieldValueInvalid","message":"Invalid
    value: \"\": must be specified for an update","field":"metadata.resourceVersion"},{"reason":"FieldValueInvalid","message":"Invalid
    value: \"\": field is immutable","field":"spec.clusterIP"}]},"code":422,"statusMessage":"{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"Service
    \\\"test-svc\\\" is invalid: [metadata.resourceVersion: Invalid value: \\\"\\\":
    must be specified for an update, spec.clusterIP: Invalid value: \\\"\\\": field
    is immutable]\",\"reason\":\"Invalid\",\"details\":{\"name\":\"test-svc\",\"kind\":\"Service\",\"causes\":[{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid
    value: \\\"\\\": must be specified for an update\",\"field\":\"metadata.resourceVersion\"},{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid
    value: \\\"\\\": must be specified for an update\",\"field\":\"metadata.resourceVersion\"},{\"reason\":\"FieldValueInvalid\",\"message\":\"Invalid
    value: \\\"\\\": field is immutable\",\"field\":\"spec.clusterIP\"}]},\"code\":422}\n","requestPath":"https://35.231.142.169:443/api/v1/namespaces/default/services/test-svc","httpMethod":"PUT"}}'

This was tested against 1.11.6-gke.2 on the Kubernetes master.

@liggitt ‘s suggestion to omit clusterIP from the manifest doesn’t directly apply here, since the null clusterIP is presumably being inserted by GCDM, presumably in response to how it interprets Kubernetes’ Swagger API definition?