kubernetes: RunContainerError: "runContainer: operation timeout: context deadline exceeded"
Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+coreos.0", GitCommit:"cc65f5321f9230bf9a3fa171155c1213d6e3480e", GitTreeState:"clean", BuildDate:"2016-12-14T04:08:28Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
OS Version: CoreOS stable (1185.5.0)
From the kubelet logs:
Dec 20 15:40:30 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper[1401]: I1220 15:40:30.077807 1401 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/75207ba5-c6c6-11e6-8758-06099f2a458d-default-token-3nexu" (spec.Name: "default-token-3nexu") pod "75207ba5-c6c6-11e6-8758-06099f2a458d" (UID: "75207ba5-c6c6-11e6-8758-06099f2a458d").
Dec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper[1401]: E1220 15:40:31.616962 1401 docker_manager.go:2324] container start failed: RunContainerError: runContainer: operation timeout: context deadline exceeded
Dec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper[1401]: E1220 15:40:31.617011 1401 pod_workers.go:184] Error syncing pod 3d09fff2-c6c7-11e6-8758-06099f2a458d, skipping: failed to "StartContainer" for "prometheus" with RunContainerError: "runContainer: operation timeout: context deadline exceeded"
Dec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper[1401]: I1220 15:40:31.880783 1401 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/3d09fff2-c6c7-11e6-8758-06099f2a458d-default-token-0shxu" (spec.Name: "default-token-0shxu") pod "3d09fff2-c6c7-11e6-8758-06099f2a458d" (UID: "3d09fff2-c6c7-11e6-8758-06099f2a458d").
Dec 20 15:40:31 ip-10-66-21-87.eu-west-1.compute.internal kubelet-wrapper[1401]: I1220 15:40:31.881996 1401 operation_executor.go:917] MountVolume.SetUp succeeded for volume "kubernetes.io/configmap/3d09fff2-c6c7-11e6-8758-06099f2a458d-config-volume" (spec.Name: "config-volume") pod "3d09fff2-c6c7-11e6-8758-06099f2a458d" (UID: "3d09fff2-c6c7-11e6-8758-06099f2a458d").
docker ps:
8c2ea1120e4a prom/prometheus:v1.2.1 "/bin/prometheus -sto" 10 seconds ago Created k8s_prometheus.edf8dc49_prometheus-227300114-v7z1h_system_3d09fff2-c6c7-11e6-8758-06099f2a458d_af434bd8
No logs on the container:
core@ip-10-66-21-87 ~ $ docker logs 8c2ea1120e4a
core@ip-10-66-21-87 ~ $
docker inspect:
[
{
"Id": "8c2ea1120e4a8b41ca2a0c4ef9afca5d19d790c633e1b965434a0ec9fca9cd04",
"Created": "2016-12-20T15:42:32.840639703Z",
"Path": "/bin/prometheus",
"Args": [
"-storage.local.retention=336h",
"-storage.local.memory-chunks=262144",
"-config.file=/etc/prometheus/prometheus.yml",
"-alertmanager.url=http://alertmanager:9093",
"-web.external-url=https://prometheus.dev.uw.systems"
],
"State": {
"Status": "created",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "0001-01-01T00:00:00Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:18eb3e2914849e994e7a22ad84e1ce3c7d41a51eec2ce68584728f36181900bc",
"ResolvConfPath": "",
"HostnamePath": "",
"HostsPath": "",
"LogPath": "",
"Name": "/k8s_prometheus.edf8dc49_prometheus-227300114-v7z1h_system_3d09fff2-c6c7-11e6-8758-06099f2a458d_af434bd8",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "system_u:object_r:svirt_lxc_file_t:s0:c241,c498",
"ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c241,c498",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": null,
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "",
"PortBindings": null,
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Dns": null,
"DnsOptions": null,
"DnsSearch": null,
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": false,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": null,
"StorageOpt": null,
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 0,
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": null,
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": 0,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"BlkioIOps": 0,
"BlkioBps": 0,
"SandboxSize": 0
},
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/var/lib/docker/overlay/a84216d9ef3e8348e9296b791fa044617d02ede93add2eaf4564effa5c31644d/root",
"MergedDir": "/var/lib/docker/overlay/473a3409ce77a1846d1260b943d8861844c1b22a23cc32743587771efd2fcf68/merged",
"UpperDir": "/var/lib/docker/overlay/473a3409ce77a1846d1260b943d8861844c1b22a23cc32743587771efd2fcf68/upper",
"WorkDir": "/var/lib/docker/overlay/473a3409ce77a1846d1260b943d8861844c1b22a23cc32743587771efd2fcf68/work"
}
},
"Mounts": [],
"Config": {
"Hostname": "8c2ea1120e4a",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"9090/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PROMETHEUS_PORT_9090_TCP_PROTO=tcp",
"ALERTMANAGER_PORT_9093_TCP=tcp://10.3.0.53:9093",
"ALERTMANAGER_PORT_9093_TCP_PORT=9093",
"GITLAB_PORT_80_TCP_ADDR=10.3.0.28",
"EXTERNAL_INGRESS_PROXY_PORT_80_TCP_PORT=80",
"PROM_PUSHGATEWAY_PORT_80_TCP_ADDR=10.3.0.251",
"INTERNAL_INGRESS_PROXY_PORT_8080_TCP_PROTO=tcp",
"PROMETHEUS_SERVICE_PORT=9090",
"GITLAB_PORT_80_TCP_PROTO=tcp",
"EXTERNAL_INGRESS_PROXY_SERVICE_PORT_HTTP_ADMIN=8080",
"EXTERNAL_INGRESS_PROXY_PORT_8080_TCP_PROTO=tcp",
"INTERNAL_INGRESS_PROXY_SERVICE_PORT_HTTP=80",
"INTERNAL_INGRESS_PROXY_SERVICE_PORT_HTTP_ADMIN=8080",
"GITHUB_SSHKEY_PROVIDER_PORT_80_TCP_PROTO=tcp",
"PROM_PUSHGATEWAY_SERVICE_PORT_PROMETHEUS=80",
"PROM_PUSHGATEWAY_PORT=tcp://10.3.0.251:80",
"GITLAB_SERVICE_PORT_HTTPS=443",
"GITLAB_PORT_443_TCP_PROTO=tcp",
"GITHUB_SSHKEY_PROVIDER_PORT_80_TCP_PORT=80",
"PROMETHEUS_SERVICE_HOST=10.3.0.110",
"PROMETHEUS_PORT_9090_TCP_PORT=9090",
"GITHUB_SSHKEY_PROVIDER_PORT_6379_TCP_PORT=6379",
"PROMETHEUS_SERVICE_PORT_PROMETHEUS=9090",
"PROMETHEUS_PORT_9090_TCP_ADDR=10.3.0.110",
"GITHUB_SSHKEY_PROVIDER_PORT_80_TCP=tcp://10.3.0.44:80",
"GITHUB_SSHKEY_PROVIDER_PORT_6379_TCP_PROTO=tcp",
"BLACKBOX_EXPORTER_SERVICE_PORT_HTTP=80",
"GRAFANA_PORT_3000_TCP=tcp://10.3.0.113:3000",
"GITLAB_PORT_443_TCP=tcp://10.3.0.28:443",
"GITLAB_PORT_22_TCP_PROTO=tcp",
"PROM_PUSHGATEWAY_PORT_80_TCP_PORT=80",
"GITLAB_SERVICE_HOST=10.3.0.28",
"GITLAB_PORT_22_TCP=tcp://10.3.0.28:22",
"GITHUB_SSHKEY_PROVIDER_SERVICE_PORT_REDIS=6379",
"GRAFANA_SERVICE_HOST=10.3.0.113",
"GITHUB_SSHKEY_PROVIDER_PORT_80_TCP_ADDR=10.3.0.44",
"EXTERNAL_INGRESS_PROXY_PORT_8080_TCP_ADDR=10.3.0.161",
"KUBERNETES_PORT_443_TCP_ADDR=10.3.0.1",
"GITLAB_PORT_443_TCP_ADDR=10.3.0.28",
"INTERNAL_INGRESS_PROXY_SERVICE_PORT=80",
"INTERNAL_INGRESS_PROXY_PORT_8080_TCP=tcp://10.3.0.176:8080",
"PROM_PUSHGATEWAY_SERVICE_HOST=10.3.0.251",
"KUBERNETES_PORT_443_TCP_PORT=443",
"KUBERNETES_SERVICE_PORT=443",
"GITLAB_SERVICE_PORT=80",
"BLACKBOX_EXPORTER_SERVICE_PORT=80",
"GRAFANA_PORT_3000_TCP_ADDR=10.3.0.113",
"GITHUB_SSHKEY_PROVIDER_PORT_6379_TCP=tcp://10.3.0.44:6379",
"EXTERNAL_INGRESS_PROXY_SERVICE_HOST=10.3.0.161",
"EXTERNAL_INGRESS_PROXY_SERVICE_PORT_HTTP=80",
"KUBERNETES_PORT_443_TCP=tcp://10.3.0.1:443",
"GITLAB_PORT_22_TCP_PORT=22",
"ALERTMANAGER_PORT_9093_TCP_PROTO=tcp",
"INTERNAL_INGRESS_PROXY_PORT_8080_TCP_PORT=8080",
"PROMETHEUS_PORT_9090_TCP=tcp://10.3.0.110:9090",
"KUBERNETES_PORT_443_TCP_PROTO=tcp",
"EXTERNAL_INGRESS_PROXY_PORT_80_TCP=tcp://10.3.0.161:80",
"ALERTMANAGER_SERVICE_HOST=10.3.0.53",
"BLACKBOX_EXPORTER_PORT_80_TCP_PORT=80",
"PROM_PUSHGATEWAY_SERVICE_PORT=80",
"PROMETHEUS_PORT=tcp://10.3.0.110:9090",
"KUBERNETES_SERVICE_HOST=10.3.0.1",
"EXTERNAL_INGRESS_PROXY_SERVICE_PORT=80",
"EXTERNAL_INGRESS_PROXY_PORT=tcp://10.3.0.161:80",
"GRAFANA_PORT_3000_TCP_PORT=3000",
"PROM_PUSHGATEWAY_PORT_80_TCP=tcp://10.3.0.251:80",
"INTERNAL_INGRESS_PROXY_PORT_80_TCP_PORT=80",
"INTERNAL_INGRESS_PROXY_PORT_8080_TCP_ADDR=10.3.0.176",
"PROM_PUSHGATEWAY_PORT_80_TCP_PROTO=tcp",
"BLACKBOX_EXPORTER_SERVICE_HOST=10.3.0.3",
"BLACKBOX_EXPORTER_PORT_80_TCP=tcp://10.3.0.3:80",
"ALERTMANAGER_PORT=tcp://10.3.0.53:9093",
"GRAFANA_PORT=tcp://10.3.0.113:3000",
"INTERNAL_INGRESS_PROXY_PORT=tcp://10.3.0.176:80",
"EXTERNAL_INGRESS_PROXY_PORT_80_TCP_PROTO=tcp",
"EXTERNAL_INGRESS_PROXY_PORT_8080_TCP_PORT=8080",
"EXTERNAL_INGRESS_PROXY_PORT_80_TCP_ADDR=10.3.0.161",
"GRAFANA_SERVICE_PORT=3000",
"ALERTMANAGER_PORT_9093_TCP_ADDR=10.3.0.53",
"GITLAB_PORT_443_TCP_PORT=443",
"GITLAB_PORT_22_TCP_ADDR=10.3.0.28",
"GITHUB_SSHKEY_PROVIDER_PORT=tcp://10.3.0.44:80",
"INTERNAL_INGRESS_PROXY_SERVICE_HOST=10.3.0.176",
"KUBERNETES_SERVICE_PORT_HTTPS=443",
"KUBERNETES_PORT=tcp://10.3.0.1:443",
"BLACKBOX_EXPORTER_PORT=tcp://10.3.0.3:80",
"ALERTMANAGER_SERVICE_PORT_ALERTMANAGER=9093",
"GITLAB_SERVICE_PORT_SSH=22",
"GITLAB_PORT=tcp://10.3.0.28:80",
"GITHUB_SSHKEY_PROVIDER_SERVICE_HOST=10.3.0.44",
"GITHUB_SSHKEY_PROVIDER_PORT_6379_TCP_ADDR=10.3.0.44",
"INTERNAL_INGRESS_PROXY_PORT_80_TCP=tcp://10.3.0.176:80",
"GITLAB_PORT_80_TCP_PORT=80",
"GITHUB_SSHKEY_PROVIDER_SERVICE_PORT=80",
"EXTERNAL_INGRESS_PROXY_PORT_8080_TCP=tcp://10.3.0.161:8080",
"GRAFANA_PORT_3000_TCP_PROTO=tcp",
"INTERNAL_INGRESS_PROXY_PORT_80_TCP_ADDR=10.3.0.176",
"INTERNAL_INGRESS_PROXY_PORT_80_TCP_PROTO=tcp",
"GITLAB_SERVICE_PORT_HTTP=80",
"GITLAB_PORT_80_TCP=tcp://10.3.0.28:80",
"BLACKBOX_EXPORTER_PORT_80_TCP_PROTO=tcp",
"BLACKBOX_EXPORTER_PORT_80_TCP_ADDR=10.3.0.3",
"ALERTMANAGER_SERVICE_PORT=9093",
"GITHUB_SSHKEY_PROVIDER_SERVICE_PORT_WEB=80",
"GRAFANA_SERVICE_PORT_GRAFANA=3000",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": [
"-storage.local.retention=336h",
"-storage.local.memory-chunks=262144",
"-config.file=/etc/prometheus/prometheus.yml",
"-alertmanager.url=http://alertmanager:9093",
"-web.external-url=https://prometheus.dev.uw.systems"
],
"Image": "prom/prometheus:v1.2.1",
"Volumes": {
"/prometheus": {}
},
"WorkingDir": "/prometheus",
"Entrypoint": [
"/bin/prometheus"
],
"OnBuild": null,
"Labels": {
"io.kubernetes.container.hash": "edf8dc49",
"io.kubernetes.container.name": "prometheus",
"io.kubernetes.container.ports": "[{\"name\":\"web\",\"containerPort\":9090,\"protocol\":\"TCP\"}]",
"io.kubernetes.container.restartCount": "11",
"io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
"io.kubernetes.pod.name": "prometheus-227300114-v7z1h",
"io.kubernetes.pod.namespace": "system",
"io.kubernetes.pod.terminationGracePeriod": "30",
"io.kubernetes.pod.uid": "3d09fff2-c6c7-11e6-8758-06099f2a458d"
}
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": null,
"SandboxKey": "",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": null
}
}
]
kubectl describe pod:
Name: prometheus-227300114-v7z1h
Namespace: system
Node: ip-10-66-21-87.eu-west-1.compute.internal/10.66.21.87
Start Time: Tue, 20 Dec 2016 15:16:10 +0000
Labels: app=prometheus
pod-template-hash=227300114
Status: Pending
IP: 10.2.56.5
Controllers: ReplicaSet/prometheus-227300114
Containers:
prometheus:
Container ID: docker://8c2ea1120e4a8b41ca2a0c4ef9afca5d19d790c633e1b965434a0ec9fca9cd04
Image: prom/prometheus:v1.2.1
Image ID: docker://sha256:18eb3e2914849e994e7a22ad84e1ce3c7d41a51eec2ce68584728f36181900bc
Port: 9090/TCP
Args:
-storage.local.retention=336h
-storage.local.memory-chunks=262144
-config.file=/etc/prometheus/prometheus.yml
-alertmanager.url=http://alertmanager:9093
-web.external-url=https://prometheus.dev.uw.systems
Limits:
cpu: 1500m
memory: 6000Mi
Requests:
cpu: 250m
memory: 2000Mi
State: Waiting
Reason: RunContainerError
Ready: False
Restart Count: 11
Volume Mounts:
/etc/kubernetes/ssl from ssl (ro)
/etc/prometheus from config-volume (rw)
/prometheus from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-0shxu (ro)
Environment Variables: <none>
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: prometheus
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: prometheus-ebs-pvc
ReadOnly: false
ssl:
Type: HostPath (bare host directory volume)
Path: /etc/kubernetes/ssl
default-token-0shxu:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-0shxu
QoS Class: Burstable
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
29m 29m 1 {default-scheduler } Normal Scheduled Successfully assigned prometheus-227300114-v7z1h to ip-10-66-21-87.eu-west-1.compute.internal
29m 27m 8 {controller-manager } Warning FailedMount Failed to attach volume "pvc-64b14071-c5db-11e6-8758-06099f2a458d" on node "ip-10-66-21-87.eu-west-1.compute.internal" with: Error attaching EBS volume "vol-02ff0c2158d03e018" to instance "i-01fb7ff93bad20e38": VolumeInUse: vol-02ff0c2158d03e018 is already attached to an instance
status code: 400, request id:
27m 27m 1 {kubelet ip-10-66-21-87.eu-west-1.compute.internal} Warning FailedMount Unable to mount volumes for pod "prometheus-227300114-v7z1h_system(3d09fff2-c6c7-11e6-8758-06099f2a458d)": timeout expired waiting for volumes to attach/mount for pod "prometheus-227300114-v7z1h"/"system". list of unattached/unmounted volumes=[data]
27m 27m 1 {kubelet ip-10-66-21-87.eu-west-1.compute.internal} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "prometheus-227300114-v7z1h"/"system". list of unattached/unmounted volumes=[data]
24m 24m 1 {kubelet ip-10-66-21-87.eu-west-1.compute.internal} spec.containers{prometheus} Normal Pulling pulling image "prom/prometheus:v1.2.1"
24m 24m 1 {kubelet ip-10-66-21-87.eu-west-1.compute.internal} spec.containers{prometheus} Normal Pulled Successfully pulled image "prom/prometheus:v1.2.1"
22m 43s 12 {kubelet ip-10-66-21-87.eu-west-1.compute.internal} spec.containers{prometheus} Warning Failed Failed to create docker container "prometheus" of pod "prometheus-227300114-v7z1h_system(3d09fff2-c6c7-11e6-8758-06099f2a458d)" with error: operation timeout: context deadline exceeded
22m 43s 12 {kubelet ip-10-66-21-87.eu-west-1.compute.internal} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "prometheus" with RunContainerError: "runContainer: operation timeout: context deadline exceeded"
22m 42s 12 {kubelet ip-10-66-21-87.eu-west-1.compute.internal} spec.containers{prometheus} Normal Pulled Container image "prom/prometheus:v1.2.1" already present on machine
About this issue
- Original URL
- State: closed
- Created 8 years ago
- Reactions: 2
- Comments: 18 (11 by maintainers)
So we can make sure it is due to EBS volume.
I can see a very suspicious ERROR:
VolumeInUse: vol-02ff0c2158d03e018 is already attached to an instance. Can you check if your EBS volume has already attached a different VM other than the right node?It seems the root cause is:
Could you please check what happens to your EBS volume: “vol-02ff0c2158d03e018”?
btw, EBS volume requires AWS cloud provider, are you using AWS?