skaffold: Error from server (Invalid): error when applying patch: Job.batch "migration-elasticsearch" is invalid:
Information
Skaffold latest version fails to apply a job. The previous version (v0.40.0) works with no issue. When deploying to GKE (v1.13.11-gke.5), I get the following error -
- Error from server (Invalid): error when applying patch:
- {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"},\"name\":\"migration-elasticsearch\",\"namespace\":\"com-test\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"job\":\"migration-elasticsearch\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"cloud.google.com/gke-nodepool\",\"operator\":\"In\",\"values\":[\"default-pool\"]}]}]}}},\"containers\":[{\"envFrom\":[{\"configMapRef\":{\"name\":\"common-config\"}},{\"configMapRef\":{\"name\":\"elasticsearch-config\"}}],\"image\":\"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-459-g54d307d8@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"migration-elasticsearch\",\"resources\":{\"limits\":{\"cpu\":\"150m\",\"memory\":\"100Mi\"},\"requests\":{\"cpu\":\"75m\",\"memory\":\"50Mi\"}}}],\"initContainers\":[{\"command\":[\"sh\",\"-c\",\"while ! nc -zv -w 1 elasticsearch 9200; do echo \\\"waiting for elasticsearch to be available\\\"; sleep 2; done;\"],\"image\":\"busybox:1.30.0\",\"name\":\"check-postgres\"}],\"restartPolicy\":\"Never\"}}}}\n"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"migration-elasticsearch"}],"containers":[{"image":"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-459-g54d307d8@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1","name":"migration-elasticsearch"}]}}}}
- to:
- Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
- Name: "migration-elasticsearch", Namespace: "com-test"
- Object: &{map["apiVersion":"batch/v1" "kind":"Job" "metadata":map["annotations":map["kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"},\"name\":\"migration-elasticsearch\",\"namespace\":\"com-test\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/managed-by\":\"skaffold-v0.41.0\",\"job\":\"migration-elasticsearch\",\"skaffold.dev/builder\":\"google-cloud-build\",\"skaffold.dev/cleanup\":\"true\",\"skaffold.dev/deployer\":\"kubectl\",\"skaffold.dev/profiles\":\"com-test\",\"skaffold.dev/run-id\":\"com\",\"skaffold.dev/tag-policy\":\"git-commit\",\"skaffold.dev/tail\":\"true\"}},\"spec\":{\"affinity\":{\"nodeAffinity\":{\"requiredDuringSchedulingIgnoredDuringExecution\":{\"nodeSelectorTerms\":[{\"matchExpressions\":[{\"key\":\"cloud.google.com/gke-nodepool\",\"operator\":\"In\",\"values\":[\"default-pool\"]}]}]}}},\"containers\":[{\"envFrom\":[{\"configMapRef\":{\"name\":\"common-config\"}},{\"configMapRef\":{\"name\":\"elasticsearch-config\"}}],\"image\":\"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-433-gd7d9c5ab@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"migration-elasticsearch\",\"resources\":{\"limits\":{\"cpu\":\"150m\",\"memory\":\"100Mi\"},\"requests\":{\"cpu\":\"75m\",\"memory\":\"50Mi\"}}}],\"initContainers\":[{\"command\":[\"sh\",\"-c\",\"while ! nc -zv -w 1 elasticsearch 9200; do echo \\\"waiting for elasticsearch to be available\\\"; sleep 2; done;\"],\"image\":\"busybox:1.30.0\",\"name\":\"check-postgres\"}],\"restartPolicy\":\"Never\"}}}}\n"] "creationTimestamp":"2019-11-04T15:26:34Z" "labels":map["app.kubernetes.io/managed-by":"skaffold-v0.41.0" "skaffold.dev/builder":"google-cloud-build" "skaffold.dev/cleanup":"true" "skaffold.dev/deployer":"kubectl" "skaffold.dev/profiles":"com-test" "skaffold.dev/run-id":"com" "skaffold.dev/tag-policy":"git-commit" "skaffold.dev/tail":"true"] "name":"migration-elasticsearch" "namespace":"com-test" "resourceVersion":"32010771" "selfLink":"/apis/batch/v1/namespaces/com-test/jobs/migration-elasticsearch" "uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f"] "spec":map["backoffLimit":'\x06' "completions":'\x01' "parallelism":'\x01' "selector":map["matchLabels":map["controller-uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f"]] "template":map["metadata":map["creationTimestamp":<nil> "labels":map["app.kubernetes.io/managed-by":"skaffold-v0.41.0" "controller-uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f" "job":"migration-elasticsearch" "job-name":"migration-elasticsearch" "skaffold.dev/builder":"google-cloud-build" "skaffold.dev/cleanup":"true" "skaffold.dev/deployer":"kubectl" "skaffold.dev/profiles":"com-test" "skaffold.dev/run-id":"com" "skaffold.dev/tag-policy":"git-commit" "skaffold.dev/tail":"true"]] "spec":map["affinity":map["nodeAffinity":map["requiredDuringSchedulingIgnoredDuringExecution":map["nodeSelectorTerms":[map["matchExpressions":[map["key":"cloud.google.com/gke-nodepool" "operator":"In" "values":["default-pool"]]]]]]]] "containers":[map["envFrom":[map["configMapRef":map["name":"common-config"]] map["configMapRef":map["name":"elasticsearch-config"]]] "image":"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-433-gd7d9c5ab@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1" "imagePullPolicy":"IfNotPresent" "name":"migration-elasticsearch" "resources":map["limits":map["cpu":"150m" "memory":"100Mi"] "requests":map["cpu":"75m" "memory":"50Mi"]] "terminationMessagePath":"/dev/termination-log" "terminationMessagePolicy":"File"]] "dnsPolicy":"ClusterFirst" "initContainers":[map["command":["sh" "-c" "while ! nc -zv -w 1 elasticsearch 9200; do echo \"waiting for elasticsearch to be available\"; sleep 2; done;"] "image":"busybox:1.30.0" "imagePullPolicy":"IfNotPresent" "name":"check-postgres" "resources":map[] "terminationMessagePath":"/dev/termination-log" "terminationMessagePolicy":"File"]] "restartPolicy":"Never" "schedulerName":"default-scheduler" "securityContext":map[] "terminationGracePeriodSeconds":'\x1e']]] "status":map["completionTime":"2019-11-04T15:26:39Z" "conditions":[map["lastProbeTime":"2019-11-04T15:26:39Z" "lastTransitionTime":"2019-11-04T15:26:39Z" "status":"True" "type":"Complete"]] "startTime":"2019-11-04T15:26:34Z" "succeeded":'\x01']]}
- for: "STDIN": Job.batch "migration-elasticsearch" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"skaffold.dev/tail":"true", "skaffold.dev/run-id":"com", "app.kubernetes.io/managed-by":"skaffold-v0.41.0", "job":"migration-elasticsearch", "skaffold.dev/profiles":"com-test", "skaffold.dev/builder":"google-cloud-build", "skaffold.dev/deployer":"kubectl", "skaffold.dev/tag-policy":"git-commit", "skaffold.dev/cleanup":"true", "controller-uid":"7c291e1f-ff17-11e9-b9a4-42010a84025f", "job-name":"migration-elasticsearch"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container{core.Container{Name:"check-postgres", Image:"busybox:1.30.0", Command:[]string{"sh", "-c", "while ! nc -zv -w 1 elasticsearch 9200; do echo \"waiting for elasticsearch to be available\"; sleep 2; done;"}, Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, Containers:[]core.Container{core.Container{Name:"migration-elasticsearch", Image:"eu.gcr.io/com/migration-elasticsearch:rel-1.38.0.0-459-g54d307d8@sha256:9f58df2249b61110d74475a60ecee84ac28a2abd5bc2d0f321c298640e36d7e1", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource{core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0xc01d569ee0), SecretRef:(*core.SecretEnvSource)(nil)}, core.EnvFromSource{Prefix:"", ConfigMapRef:(*core.ConfigMapEnvSource)(0xc01d569f00), SecretRef:(*core.SecretEnvSource)(nil)}}, Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList{"memory":resource.Quantity{i:resource.int64Amount{value:104857600, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100Mi", Format:"BinarySI"}, "cpu":resource.Quantity{i:resource.int64Amount{value:150, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"150m", Format:"DecimalSI"}}, Requests:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:75, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"75m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:52428800, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"50Mi", Format:"BinarySI"}}}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc00ef2a8e8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc001ba89a0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*core.Affinity)(0xc01d569fe0), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil)}}: field is immutable
- Skaffold version: v0.41.0
- Operating system: MacOS
- Contents of skaffold.yaml:
apiVersion: skaffold/v1beta17
kind: Config
build:
local:
push: false
artifacts:
# db migrations
- image: eu.gcr.io/com/migration-postgres
context: db/postgres
- image: eu.gcr.io/com/migration-elasticsearch
context: db/elasticsearch
profiles:
- name: local-dev
build:
local:
push: false
activation:
- kubeContext: minikube
deploy:
kubectl:
manifests:
- k8s/local/**
- name: com-test
build:
googleCloudBuild:
projectId: com
timeout: "900s"
machineType: N1_HIGHCPU_8
deploy:
kubectl:
manifests:
- k8s/gke/com-test/**
Steps to reproduce the behavior
skaffold run -l skaffold.dev/run-id=com -p com-test -v debug
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 3
- Comments: 20 (6 by maintainers)
hey all, sorry for the delay on this one. I was able to reproduce with the repo that @chinmaya-n provided (thank you!), and it looks like this was broken by https://github.com/GoogleContainerTools/skaffold/pull/3086. we’ll need to add back force applying to make this work again for jobs.
I just experienced a similar error when deploying a
job. This only happens if the resource already exists in the K8 cluster and the current run is trying to override the existing ones. If you run this after removing the existing objects usingskaffold delete, it deploys fine.