skaffold: Error from server (Invalid): error when applying patch: Job.batch "migration-elasticsearch" is invalid:

Information

Skaffold latest version fails to apply a job. The previous version (v0.40.0) works with no issue. When deploying to GKE (v1.13.11-gke.5), I get the following error -

- Error from server (Invalid): error when applying patch:
 - {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"},\"name\":\"migration-elasticsearch\",\"namespace\":\"com-test\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"job\":\"migration-elasticsearch\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"cloud.google.com/gke-nodepool\",\"operator\":\"In\",\"values\":[\"default-pool\"]}]}]}}},\"containers\":[{\"envFrom\":[{\"configMapRef\":{\"name\":\"common-config\"}},{\"configMapRef\":{\"name\":\"elasticsearch-config\"}}],\"image\":\"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-459-g54d307d8@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"migration-elasticsearch\",\"resources\":{\"limits\":{\"cpu\":\"150m\",\"memory\":\"100Mi\"},\"requests\":{\"cpu\":\"75m\",\"memory\":\"50Mi\"}}}],\"initContainers\":[{\"command\":[\"sh\",\"-c\",\"while ! nc -zv -w 1 elasticsearch 9200; do echo \\\"waiting for elasticsearch to be available\\\"; sleep 2; done;\"],\"image\":\"busybox:1.30.0\",\"name\":\"check-postgres\"}],\"restartPolicy\":\"Never\"}}}}\n"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"migration-elasticsearch"}],"containers":[{"image":"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-459-g54d307d8@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1","name":"migration-elasticsearch"}]}}}}
 - to:
 - Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
 - Name: "migration-elasticsearch", Namespace: "com-test"
 - Object: &{map["apiVersion":"batch/v1" "kind":"Job" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"},\"name\":\"migration-elasticsearch\",\"namespace\":\"com-test\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"job\":\"migration-elasticsearch\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"cloud.google.com/gke-nodepool\",\"operator\":\"In\",\"values\":[\"default-pool\"]}]}]}}},\"containers\":[{\"envFrom\":[{\"configMapRef\":{\"name\":\"common-config\"}},{\"configMapRef\":{\"name\":\"elasticsearch-config\"}}],\"image\":\"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-433-gd7d9c5ab@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"migration-elasticsearch\",\"resources\":{\"limits\":{\"cpu\":\"150m\",\"memory\":\"100Mi\"},\"requests\":{\"cpu\":\"75m\",\"memory\":\"50Mi\"}}}],\"initContainers\":[{\"command\":[\"sh\",\"-c\",\"while ! nc -zv -w 1 elasticsearch 9200; do echo \\\"waiting for elasticsearch to be available\\\"; sleep 2; done;\"],\"image\":\"busybox:1.30.0\",\"name\":\"check-postgres\"}],\"restartPolicy\":\"Never\"}}}}\n"] "creationTimestamp":"2019-11-04T15:26:34Z" "labels":map["app.kubernetes.io/managed-by":"skaffold-v0.41.0" "skaffold.dev/builder":"google-cloud-build" "skaffold.dev/cleanup":"true" "skaffold.dev/deployer":"kubectl" "skaffold.dev/profiles":"com-test" "skaffold.dev/run-id":"com" "skaffold.dev/tag-policy":"git-commit" "skaffold.dev/tail":"true"] "name":"migration-elasticsearch" "namespace":"com-test" "resourceVersion":"32010771" "selfLink":"/apis/batch/v1/namespaces/com-test/jobs/migration-elasticsearch" "uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f"] "spec":map["backoffLimit":'\x06' "completions":'\x01' "parallelism":'\x01' "selector":map["matchLabels":map["controller-uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f"]] "template":map["metadata":map["creationTimestamp":<nil> "labels":map["app.kubernetes.io/managed-by":"skaffold-v0.41.0" "controller-uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f" "job":"migration-elasticsearch" "job-name":"migration-elasticsearch" "skaffold.dev/builder":"google-cloud-build" "skaffold.dev/cleanup":"true" "skaffold.dev/deployer":"kubectl" "skaffold.dev/profiles":"com-test" "skaffold.dev/run-id":"com" "skaffold.dev/tag-policy":"git-commit" "skaffold.dev/tail":"true"]] "spec":map["affinity":map["nodeAffinity":map["requiredDuringSchedulingIgnoredDuringExecution":map["nodeSelectorTerms":[map["matchExpressions":[map["key":"cloud.google.com/gke-nodepool" "operator":"In" "values":["default-pool"]]]]]]]] "containers":[map["envFrom":[map["configMapRef":map["name":"common-config"]] map["configMapRef":map["name":"elasticsearch-config"]]] "image":"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-433-gd7d9c5ab@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1" "imagePullPolicy":"IfNotPresent" "name":"migration-elasticsearch" "resources":map["limits":map["cpu":"150m" "memory":"100Mi"] "requests":map["cpu":"75m" "memory":"50Mi"]] "terminationMessagePath":"/dev/termination-log" "terminationMessagePolicy":"File"]] "dnsPolicy":"ClusterFirst" "initContainers":[map["command":["sh" "-c" "while ! nc -zv -w 1 elasticsearch 9200; do echo \"waiting for elasticsearch to be available\"; sleep 2; done;"] "image":"busybox:1.30.0" "imagePullPolicy":"IfNotPresent" "name":"check-postgres" "resources":map[] "terminationMessagePath":"/dev/termination-log" "terminationMessagePolicy":"File"]] "restartPolicy":"Never" "schedulerName":"default-scheduler" "securityContext":map[] "terminationGracePeriodSeconds":'\x1e']]] "status":map["completionTime":"2019-11-04T15:26:39Z" "conditions":[map["lastProbeTime":"2019-11-04T15:26:39Z" "lastTransitionTime":"2019-11-04T15:26:39Z" "status":"True" "type":"Complete"]] "startTime":"2019-11-04T15:26:34Z" "succeeded":'\x01']]}
 - for: "STDIN": Job.batch "migration-elasticsearch" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"skaffold.dev/tail":"true", "skaffold.dev/run-id":"com", "app.kubernetes.io/managed-by":"skaffold-v0.41.0", "job":"migration-elasticsearch", "skaffold.dev/profiles":"com-test", "skaffold.dev/builder":"google-cloud-build", "skaffold.dev/deployer":"kubectl", "skaffold.dev/tag-policy":"git-commit", "skaffold.dev/cleanup":"true", "controller-uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f", "job-name":"migration-elasticsearch"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container{core.Container{Name:"check-postgres", Image:"busybox:1.30.0", Command:[]string{"sh", "-c", "while ! nc -zv -w 1 elasticsearch 9200; do echo \"waiting for elasticsearch to be available\"; sleep 2; done;"}, Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, Containers:[]core.Container{core.Container{Name:"migration-elasticsearch", Image:"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-459-g54d307d8@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource{core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0xc01d569ee0), SecretRef:(*core.SecretEnvSource)(nil)}, core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0xc01d569f00), SecretRef:(*core.SecretEnvSource)(nil)}}, Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{"memory":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100Mi", Format:"BinarySI"}, "cpu":resource.Quantity{i:resource.int64Amount{value:150, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"150m", Format:"DecimalSI"}}, Requests:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:75, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"75m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:52428800, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"50Mi", Format:"BinarySI"}}}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc00ef2a8e8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc001ba89a0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*core.Affinity)(0xc01d569fe0), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil)}}: field is immutable
  • Skaffold version: v0.41.0
  • Operating system: MacOS
  • Contents of skaffold.yaml:
apiVersion: skaffold/v1beta17
kind: Config
build:
  local:
    push: false
  artifacts:

    # db migrations
    - image: eu.gcr.io/com/migration-postgres
      context: db/postgres
    - image: eu.gcr.io/com/migration-elasticsearch
      context: db/elasticsearch

profiles:
  - name: local-dev
    build:
      local:
        push: false
    activation:
      - kubeContext: minikube
    deploy:
      kubectl:
        manifests:
          - k8s/local/**
  - name: com-test
    build:
      googleCloudBuild:
        projectId: com
        timeout: "900s"
        machineType: N1_HIGHCPU_8
    deploy:
      kubectl:
        manifests:
          - k8s/gke/com-test/**

Steps to reproduce the behavior

skaffold run -l skaffold.dev/run-id=com -p com-test -v debug

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 3
  • Comments: 20 (6 by maintainers)

Most upvoted comments

hey all, sorry for the delay on this one. I was able to reproduce with the repo that @chinmaya-n provided (thank you!), and it looks like this was broken by https://github.com/GoogleContainerTools/skaffold/pull/3086. we’ll need to add back force applying to make this work again for jobs.

I just experienced a similar error when deploying a job. This only happens if the resource already exists in the K8 cluster and the current run is trying to override the existing ones. If you run this after removing the existing objects using skaffold delete, it deploys fine.