kubernetes: 'kubectl delete job ' hangs for ten minutes, then times out

Running

kubectl delete job <job name>

or

kubectl delete -f <job yaml definition>.yaml

causes kubectl to hang for approximately 10 minutes.

(absent waiting 10 minutes, ctrl+c is required to kill the process)

After 10 minutes, kubectl reports:

F0714 16:26:29.350802   10368 helpers.go:108] error: timed out waiting for the condition

This bug has caused major problems as jobs that have run for months have suddenly ceased to operate normally.

In essence, the kubectl binary just hangs up and does nothing unless verbosity is increased.

This occurs in:

$kubectl version
Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:19:19Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Background

I have a scheduled task that performs:

kubectl delete -f job.yaml
kubectl create -f job.yaml

I did this to schedule tasks that need to run periodically and close. Rather than having huge quantities of dead containers stacking up, we opted to use jobs instead.

Debug information

Adding verbosity 9:

kubectl delete job sproket_maker-3 --v=9
I0714 15:00:19.009628    7749 loader.go:327] Config loaded from file /home/dstynchula/.kube/config
I0714 15:00:19.010519    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/api
I0714 15:00:19.327875    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/api 200 OK in 317 milliseconds
I0714 15:00:19.327888    7749 round_trippers.go:324] Response Headers:
I0714 15:00:19.327894    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:16 GMT
I0714 15:00:19.327897    7749 round_trippers.go:327]     Content-Length: 135
I0714 15:00:19.327900    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:19.327928    7749 request.go:891] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]}
I0714 15:00:19.328086    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/apis
I0714 15:00:19.398573    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis 200 OK in 70 milliseconds
I0714 15:00:19.398585    7749 round_trippers.go:324] Response Headers:
I0714 15:00:19.398588    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:19.398591    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:16 GMT
I0714 15:00:19.398594    7749 round_trippers.go:327]     Content-Length: 1649
I0714 15:00:19.398623    7749 request.go:891] Response Body: {"kind":"APIGroupList","groups":[{"name":"apps","versions":[{"groupVersion":"apps/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"apps/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v1","version":"v1"}],"preferredVersion":{"groupVersion":"autoscaling/v1","version":"v1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"},{"groupVersion":"batch/v2alpha1","version":"v2alpha1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"policy","versions":[{"groupVersion":"policy/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"policy/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"rbac.authorization.k8s.io","versions":[{"groupVersion":"rbac.authorization.k8s.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"rbac.authorization.k8s.io/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]}]}
I0714 15:00:19.399553    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/api
I0714 15:00:19.469695    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/api 200 OK in 70 milliseconds
I0714 15:00:19.469706    7749 round_trippers.go:324] Response Headers:
I0714 15:00:19.469710    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:19.469713    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:16 GMT
I0714 15:00:19.469715    7749 round_trippers.go:327]     Content-Length: 135
I0714 15:00:19.469740    7749 request.go:891] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]}
I0714 15:00:19.469849    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/apis
I0714 15:00:19.540326    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis 200 OK in 70 milliseconds
I0714 15:00:19.540337    7749 round_trippers.go:324] Response Headers:
I0714 15:00:19.540340    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:19.540343    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:17 GMT
I0714 15:00:19.540346    7749 round_trippers.go:327]     Content-Length: 1649
I0714 15:00:19.540372    7749 request.go:891] Response Body: {"kind":"APIGroupList","groups":[{"name":"apps","versions":[{"groupVersion":"apps/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"apps/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v1","version":"v1"}],"preferredVersion":{"groupVersion":"autoscaling/v1","version":"v1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"},{"groupVersion":"batch/v2alpha1","version":"v2alpha1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"policy","versions":[{"groupVersion":"policy/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"policy/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"rbac.authorization.k8s.io","versions":[{"groupVersion":"rbac.authorization.k8s.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"rbac.authorization.k8s.io/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]}]}
I0714 15:00:19.541964    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/api
I0714 15:00:19.612162    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/api 200 OK in 70 milliseconds
I0714 15:00:19.612176    7749 round_trippers.go:324] Response Headers:
I0714 15:00:19.612180    7749 round_trippers.go:327]     Content-Length: 135
I0714 15:00:19.612183    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:19.612187    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:17 GMT
I0714 15:00:19.612213    7749 request.go:891] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]}
I0714 15:00:19.612330    7749 round_trippers.go:299] curl -k -v -XGET  -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" -H "Accept: application/json, */*" https://api.internal.prod.kubernetes.local/apis
I0714 15:00:19.682754    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis 200 OK in 70 milliseconds
I0714 15:00:19.682773    7749 round_trippers.go:324] Response Headers:
I0714 15:00:19.682780    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:19.682786    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:17 GMT
I0714 15:00:19.682791    7749 round_trippers.go:327]     Content-Length: 1649
I0714 15:00:19.682827    7749 request.go:891] Response Body: {"kind":"APIGroupList","groups":[{"name":"apps","versions":[{"groupVersion":"apps/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"apps/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v1","version":"v1"}],"preferredVersion":{"groupVersion":"autoscaling/v1","version":"v1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"},{"groupVersion":"batch/v2alpha1","version":"v2alpha1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"policy","versions":[{"groupVersion":"policy/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"policy/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]},{"name":"rbac.authorization.k8s.io","versions":[{"groupVersion":"rbac.authorization.k8s.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"rbac.authorization.k8s.io/v1alpha1","version":"v1alpha1"},"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.30.91.140:443"}]}]}
I0714 15:00:19.683728    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3
I0714 15:00:19.754895    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 OK in 71 milliseconds
I0714 15:00:19.754914    7749 round_trippers.go:324] Response Headers:
I0714 15:00:19.754921    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:19.754926    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:17 GMT
I0714 15:00:19.754933    7749 round_trippers.go:327]     Content-Length: 1448
I0714 15:00:19.754965    7749 request.go:891] Response Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"sproket_maker-3","namespace":"datascience","selfLink":"/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3","uid":"9c8a4148-49a2-11e6-a9db-1284d5eeaed3","resourceVersion":"1345707","creationTimestamp":"2016-07-14T09:09:04Z","labels":{"app":"sproket_maker-3"}},"spec":{"parallelism":0,"completions":1,"selector":{"matchLabels":{"app":"sproket_maker-3"}},"manualSelector":true,"template":{"metadata":{"name":"sproket_maker-3","creationTimestamp":null,"labels":{"app":"sproket_maker-3"}},"spec":{"containers":[{"name":"sproket_maker-3","image":"<image name>","command":["/bin/bash","-c"],"args":["echo '<ip> <hostname>' \u003e\u003e /etc/hosts \u0026\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000"],"ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"nodetype":"datascience-scheduled"},"securityContext":{},"imagePullSecrets":[{"name":"dockerhub"}]}}},"status":{"conditions":[{"type":"Complete","status":"True","lastProbeTime":"2016-07-14T09:09:09Z","lastTransitionTime":"2016-07-14T09:09:09Z"}],"startTime":"2016-07-14T09:09:04Z","completionTime":"2016-07-14T09:09:09Z","active":1,"succeeded":1}}
I0714 15:00:20.755424    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3
I0714 15:00:20.827376    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 OK in 71 milliseconds
I0714 15:00:20.827392    7749 round_trippers.go:324] Response Headers:
I0714 15:00:20.827396    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:20.827399    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:18 GMT
I0714 15:00:20.827402    7749 round_trippers.go:327]     Content-Length: 1448
I0714 15:00:20.827441    7749 request.go:891] Response Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"sproket_maker-3","namespace":"datascience","selfLink":"/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3","uid":"9c8a4148-49a2-11e6-a9db-1284d5eeaed3","resourceVersion":"1345707","creationTimestamp":"2016-07-14T09:09:04Z","labels":{"app":"sproket_maker-3"}},"spec":{"parallelism":0,"completions":1,"selector":{"matchLabels":{"app":"sproket_maker-3"}},"manualSelector":true,"template":{"metadata":{"name":"sproket_maker-3","creationTimestamp":null,"labels":{"app":"sproket_maker-3"}},"spec":{"containers":[{"name":"sproket_maker-3","image":"company/sproket_maker","command":["/bin/bash","-c"],"args":["echo '<ip> <name>' \u003e\u003e /etc/hosts \u0026\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000"],"ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"nodetype":"datascience-scheduled"},"securityContext":{},"imagePullSecrets":[{"name":"dockerhub"}]}}},"status":{"conditions":[{"type":"Complete","status":"True","lastProbeTime":"2016-07-14T09:09:09Z","lastTransitionTime":"2016-07-14T09:09:09Z"}],"startTime":"2016-07-14T09:09:04Z","completionTime":"2016-07-14T09:09:09Z","active":1,"succeeded":1}}
I0714 15:00:20.829210    7749 request.go:558] Request Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"sproket_maker-3","namespace":"datascience","selfLink":"/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3","uid":"9c8a4148-49a2-11e6-a9db-1284d5eeaed3","resourceVersion":"1345707","creationTimestamp":"2016-07-14T09:09:04Z","labels":{"app":"sproket_maker-3"}},"spec":{"parallelism":0,"completions":1,"selector":{"matchLabels":{"app":"sproket_maker-3"}},"manualSelector":true,"template":{"metadata":{"name":"sproket_maker-3","creationTimestamp":null,"labels":{"app":"sproket_maker-3"}},"spec":{"containers":[{"name":"sproket_maker-3","image":"company/sproket_maker","command":["/bin/bash","-c"],"args":["echo '<ip> <name>' \u003e\u003e /etc/hosts \u0026\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000"],"ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"nodetype":"datascience-scheduled"},"securityContext":{},"imagePullSecrets":[{"name":"dockerhub"}]}}},"status":{"conditions":[{"type":"Complete","status":"True","lastProbeTime":"2016-07-14T09:09:09Z","lastTransitionTime":"2016-07-14T09:09:09Z"}],"startTime":"2016-07-14T09:09:04Z","completionTime":"2016-07-14T09:09:09Z","active":1,"succeeded":1}}
I0714 15:00:20.829248    7749 round_trippers.go:299] curl -k -v -XPUT  -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3
I0714 15:00:20.900850    7749 round_trippers.go:318] PUT https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 OK in 71 milliseconds
I0714 15:00:20.900874    7749 round_trippers.go:324] Response Headers:
I0714 15:00:20.900877    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:20.900880    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:18 GMT
I0714 15:00:20.900884    7749 round_trippers.go:327]     Content-Length: 1448
I0714 15:00:20.900927    7749 request.go:891] Response Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"sproket_maker-3","namespace":"datascience","selfLink":"/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3","uid":"9c8a4148-49a2-11e6-a9db-1284d5eeaed3","resourceVersion":"1345707","creationTimestamp":"2016-07-14T09:09:04Z","labels":{"app":"sproket_maker-3"}},"spec":{"parallelism":0,"completions":1,"selector":{"matchLabels":{"app":"sproket_maker-3"}},"manualSelector":true,"template":{"metadata":{"name":"sproket_maker-3","creationTimestamp":null,"labels":{"app":"sproket_maker-3"}},"spec":{"containers":[{"name":"sproket_maker-3","image":"company/sproket_maker","command":["/bin/bash","-c"],"args":["echo '<ip> <name>' \u003e\u003e /etc/hosts \u0026\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000"],"ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"nodetype":"datascience-scheduled"},"securityContext":{},"imagePullSecrets":[{"name":"dockerhub"}]}}},"status":{"conditions":[{"type":"Complete","status":"True","lastProbeTime":"2016-07-14T09:09:09Z","lastTransitionTime":"2016-07-14T09:09:09Z"}],"startTime":"2016-07-14T09:09:04Z","completionTime":"2016-07-14T09:09:09Z","active":1,"succeeded":1}}
I0714 15:00:20.901220    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3
I0714 15:00:20.971869    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 OK in 70 milliseconds
I0714 15:00:20.971886    7749 round_trippers.go:324] Response Headers:
I0714 15:00:20.971889    7749 round_trippers.go:327]     Content-Length: 1448
I0714 15:00:20.971892    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:20.971897    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:18 GMT
I0714 15:00:20.971935    7749 request.go:891] Response Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"sproket_maker-3","namespace":"datascience","selfLink":"/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3","uid":"9c8a4148-49a2-11e6-a9db-1284d5eeaed3","resourceVersion":"1345707","creationTimestamp":"2016-07-14T09:09:04Z","labels":{"app":"sproket_maker-3"}},"spec":{"parallelism":0,"completions":1,"selector":{"matchLabels":{"app":"sproket_maker-3"}},"manualSelector":true,"template":{"metadata":{"name":"sproket_maker-3","creationTimestamp":null,"labels":{"app":"sproket_maker-3"}},"spec":{"containers":[{"name":"sproket_maker-3","image":"company/sproket_maker","command":["/bin/bash","-c"],"args":["echo '<ip> <name>' \u003e\u003e /etc/hosts \u0026\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000"],"ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"nodetype":"datascience-scheduled"},"securityContext":{},"imagePullSecrets":[{"name":"dockerhub"}]}}},"status":{"conditions":[{"type":"Complete","status":"True","lastProbeTime":"2016-07-14T09:09:09Z","lastTransitionTime":"2016-07-14T09:09:09Z"}],"startTime":"2016-07-14T09:09:04Z","completionTime":"2016-07-14T09:09:09Z","active":1,"succeeded":1}}
I0714 15:00:21.972360    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3
I0714 15:00:22.043372    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 OK in 70 milliseconds
I0714 15:00:22.043389    7749 round_trippers.go:324] Response Headers:
I0714 15:00:22.043395    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:22.043398    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:19 GMT
I0714 15:00:22.043402    7749 round_trippers.go:327]     Content-Length: 1448
I0714 15:00:22.043436    7749 request.go:891] Response Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"sproket_maker-3","namespace":"datascience","selfLink":"/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3","uid":"9c8a4148-49a2-11e6-a9db-1284d5eeaed3","resourceVersion":"1345707","creationTimestamp":"2016-07-14T09:09:04Z","labels":{"app":"sproket_maker-3"}},"spec":{"parallelism":0,"completions":1,"selector":{"matchLabels":{"app":"sproket_maker-3"}},"manualSelector":true,"template":{"metadata":{"name":"sproket_maker-3","creationTimestamp":null,"labels":{"app":"sproket_maker-3"}},"spec":{"containers":[{"name":"sproket_maker-3","image":"company/sproket_maker","command":["/bin/bash","-c"],"args":["echo '<ip> <name>' \u003e\u003e /etc/hosts \u0026\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000"],"ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"nodetype":"datascience-scheduled"},"securityContext":{},"imagePullSecrets":[{"name":"dockerhub"}]}}},"status":{"conditions":[{"type":"Complete","status":"True","lastProbeTime":"2016-07-14T09:09:09Z","lastTransitionTime":"2016-07-14T09:09:09Z"}],"startTime":"2016-07-14T09:09:04Z","completionTime":"2016-07-14T09:09:09Z","active":1,"succeeded":1}}
I0714 15:00:22.972359    7749 round_trippers.go:299] curl -k -v -XGET  -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" -H "Accept: application/json, */*" https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3
I0714 15:00:23.043554    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 OK in 71 milliseconds
I0714 15:00:23.043570    7749 round_trippers.go:324] Response Headers:
I0714 15:00:23.043574    7749 round_trippers.go:327]     Content-Length: 1448
I0714 15:00:23.043577    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:23.043580    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:20 GMT
I0714 15:00:23.043615    7749 request.go:891] Response Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"sproket_maker-3","namespace":"datascience","selfLink":"/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3","uid":"9c8a4148-49a2-11e6-a9db-1284d5eeaed3","resourceVersion":"1345707","creationTimestamp":"2016-07-14T09:09:04Z","labels":{"app":"sproket_maker-3"}},"spec":{"parallelism":0,"completions":1,"selector":{"matchLabels":{"app":"sproket_maker-3"}},"manualSelector":true,"template":{"metadata":{"name":"sproket_maker-3","creationTimestamp":null,"labels":{"app":"sproket_maker-3"}},"spec":{"containers":[{"name":"sproket_maker-3","image":"company/sproket_maker","command":["/bin/bash","-c"],"args":["echo <ip> <name>' \u003e\u003e /etc/hosts \u0026\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000"],"ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"nodetype":"datascience-scheduled"},"securityContext":{},"imagePullSecrets":[{"name":"dockerhub"}]}}},"status":{"conditions":[{"type":"Complete","status":"True","lastProbeTime":"2016-07-14T09:09:09Z","lastTransitionTime":"2016-07-14T09:09:09Z"}],"startTime":"2016-07-14T09:09:04Z","completionTime":"2016-07-14T09:09:09Z","active":1,"succeeded":1}}
I0714 15:00:23.972322    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3
I0714 15:00:24.043488    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 OK in 71 milliseconds
I0714 15:00:24.043506    7749 round_trippers.go:324] Response Headers:
I0714 15:00:24.043512    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:21 GMT
I0714 15:00:24.043517    7749 round_trippers.go:327]     Content-Length: 1448
I0714 15:00:24.043522    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:24.043559    7749 request.go:891] Response Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"sproket_maker-3","namespace":"datascience","selfLink":"/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3","uid":"9c8a4148-49a2-11e6-a9db-1284d5eeaed3","resourceVersion":"1345707","creationTimestamp":"2016-07-14T09:09:04Z","labels":{"app":"sproket_maker-3"}},"spec":{"parallelism":0,"completions":1,"selector":{"matchLabels":{"app":"sproket_maker-3"}},"manualSelector":true,"template":{"metadata":{"name":"sproket_maker-3","creationTimestamp":null,"labels":{"app":"sproket_maker-3"}},"spec":{"containers":[{"name":"sproket_maker-3","image":"company/sproket_maker","command":["/bin/bash","-c"],"args":["echo '<ip> <name>' \u003e\u003e /etc/hosts \u0026\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000"],"ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"nodetype":"datascience-scheduled"},"securityContext":{},"imagePullSecrets":[{"name":"dockerhub"}]}}},"status":{"conditions":[{"type":"Complete","status":"True","lastProbeTime":"2016-07-14T09:09:09Z","lastTransitionTime":"2016-07-14T09:09:09Z"}],"startTime":"2016-07-14T09:09:04Z","completionTime":"2016-07-14T09:09:09Z","active":1,"succeeded":1}}
I0714 15:00:24.972316    7749 round_trippers.go:299] curl -k -v -XGET  -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" -H "Accept: application/json, */*" https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3
I0714 15:00:25.043459    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 OK in 71 milliseconds
I0714 15:00:25.043480    7749 round_trippers.go:324] Response Headers:
I0714 15:00:25.043483    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:25.043487    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:22 GMT
I0714 15:00:25.043489    7749 round_trippers.go:327]     Content-Length: 1448
I0714 15:00:25.043525    7749 request.go:891] Response Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"sproket_maker-3","namespace":"datascience","selfLink":"/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3","uid":"9c8a4148-49a2-11e6-a9db-1284d5eeaed3","resourceVersion":"1345707","creationTimestamp":"2016-07-14T09:09:04Z","labels":{"app":"sproket_maker-3"}},"spec":{"parallelism":0,"completions":1,"selector":{"matchLabels":{"app":"sproket_maker-3"}},"manualSelector":true,"template":{"metadata":{"name":"sproket_maker-3","creationTimestamp":null,"labels":{"app":"sproket_maker-3"}},"spec":{"containers":[{"name":"sproket_maker-3","image":"company/sproket_maker","command":["/bin/bash","-c"],"args":["echo '<ip> <name>' \u003e\u003e /etc/hosts \u0026\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000"],"ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"nodetype":"datascience-scheduled"},"securityContext":{},"imagePullSecrets":[{"name":"dockerhub"}]}}},"status":{"conditions":[{"type":"Complete","status":"True","lastProbeTime":"2016-07-14T09:09:09Z","lastTransitionTime":"2016-07-14T09:09:09Z"}],"startTime":"2016-07-14T09:09:04Z","completionTime":"2016-07-14T09:09:09Z","active":1,"succeeded":1}}
I0714 15:00:25.972362    7749 round_trippers.go:299] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.3.0 (linux/amd64) kubernetes/2831379" https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3
I0714 15:00:26.049541    7749 round_trippers.go:318] GET https://api.internal.prod.kubernetes.local/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3 200 OK in 77 milliseconds
I0714 15:00:26.049561    7749 round_trippers.go:324] Response Headers:
I0714 15:00:26.049568    7749 round_trippers.go:327]     Content-Type: application/json
I0714 15:00:26.049573    7749 round_trippers.go:327]     Date: Thu, 14 Jul 2016 22:00:23 GMT
I0714 15:00:26.049579    7749 round_trippers.go:327]     Content-Length: 1448
I0714 15:00:26.049615    7749 request.go:891] Response Body: {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"sproket_maker-3","namespace":"datascience","selfLink":"/apis/batch/v1/namespaces/datascience/jobs/sproket_maker-3","uid":"9c8a4148-49a2-11e6-a9db-1284d5eeaed3","resourceVersion":"1345707","creationTimestamp":"2016-07-14T09:09:04Z","labels":{"app":"sproket_maker-3"}},"spec":{"parallelism":0,"completions":1,"selector":{"matchLabels":{"app":"sproket_maker-3"}},"manualSelector":true,"template":{"metadata":{"name":"sproket_maker-3","creationTimestamp":null,"labels":{"app":"sproket_maker-3"}},"spec":{"containers":[{"name":"sproket_maker-3","image":"company/sproket_maker","command":["/bin/bash","-c"],"args":["echo '<ip> <name>' \u003e\u003e /etc/hosts \u0026\u0026 start.sh python /var/www/sites/cron/company-ml-master/sproket_maker.py --shards 10 --shard-index 3 --limit 20000"],"ports":[{"containerPort":80,"protocol":"TCP"}],"resources":{},"terminationMessagePath":"/dev/termination-log","imagePullPolicy":"Always"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","nodeSelector":{"nodetype":"datascience-scheduled"},"securityContext":{},"imagePullSecrets":[{"name":"dockerhub"}]}}},"status":{"conditions":[{"type":"Complete","status":"True","lastProbeTime":"2016-07-14T09:09:09Z","lastTransitionTime":"2016-07-14T09:09:09Z"}],"startTime":"2016-07-14T09:09:04Z","completionTime":"2016-07-14T09:09:09Z","active":1,"succeeded":1}}