kubernetes: SandboxChanged Pod sandbox changed, it will be killed and re-created.

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

Uncomment only one, leave it on its own line:

/kind bug /kind feature

kind bug

What happened: when creating the deploy,the pod status was always ContainerCreating ,when l use kubectl descirbe the pod ,it’s show like this:

FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  19h		1s		5169	kubelet, slave-140			Normal		SandboxChanged	Pod sandbox changed, it will be killed and re-created.
  19h		0s		5170	kubelet, slave-140			Warning		FailedSync	Error syncing pod

What you expected to happen:

normal,it’s should recreate a new sandbox successful , and the pod should be running normal.

but when l login into the node, l use the commad ** docker ps -a | grep podname ** , l found the 2 pause exit container. l think this is the reason to course the bug.

How to reproduce it (as minimally and precisely as possible): l have no method to reproduce it. in my pro evn , happens very often, when l upgrade k8s to v1.7.6, but before cluster of v1.4.1 , l was not meeted this.

some time,when was use the command dokcer rm $(docker pa -aq) to clean the no running conatienrs , l may reproduce it. but it’s not always reproduce it

Anything else we need to know?: in my k8s cluster , every node , l foud is still has many many exist pause container, l don’t kown why?

it’s nomarl ,when a po was delete or not exist , the pod’s pause container and the real container was delete by the kubelet.

so l want konw the reason why so many exit pause container was still on the node.

Environment:

Kubernetes version (use kubectl version): v1.7.6
Cloud provider or hardware configuration:
OS (e.g. from /etc/os-release): NAME=“CentOS Linux” VERSION=“7 (Core)” ID=“centos” ID_LIKE=“rhel fedora” VERSION_ID=“7” PRETTY_NAME=“CentOS Linux 7 (Core)” ANSI_COLOR=“0;31” CPE_NAME=“cpe:/o:centos:centos:7” HOME_URL=“https://www.centos.org/” BUG_REPORT_URL=“https://bugs.centos.org/”

CENTOS_MANTISBT_PROJECT=“CentOS-7” CENTOS_MANTISBT_PROJECT_VERSION=“7” REDHAT_SUPPORT_PRODUCT=“centos” REDHAT_SUPPORT_PRODUCT_VERSION=“7”

Kernel (e.g. uname -a): 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Install tools: manual
Others:

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 3
Comments: 23 (11 by maintainers)

Most upvoted comments

Funnily enough, this exact error message is shown when you set containers.resources.{requests,limits}.memory to Xm instead of XM, where X is an integer.

Events:
  Type     Reason                  Age                 From                                              Message
  ----     ------                  ----                ----                                              -------
  Normal   SandboxChanged          37s (x12 over 61s)  kubelet, gke-staging-n1-standard-8-84c26351-j084  Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  36s (x13 over 61s)  kubelet, gke-staging-n1-standard-8-84c26351-j084  Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "fluentd-7vfcd": Error response from daemon: oci runtime error: container_linux.go:247: starting
 container process caused "process_linux.go:286: decoding sync type from init pipe caused \"read parent: connection reset by peer\""

+24

ervinb on Mar 5, 2019

@feiskyer l know ， l was viewed the code of the syncPod and teardownPod , when the teardown pod to call and relase the pod network by use cin plugin, when is return err, the syncPod method was return , waiting for the next interval sycPod , so the pod’s new sandbox nerver to be create, and the the pod is hang ContainerCreating.

now , l must need to rm the exit Pause container in my cluster nodes. do you have some good method to resolve this problem?

l forget it : all the po of the podStates is ContainerCreating, was created by the statefulset, the pod of deployment created, didn’t occur this.

huangjiasingle on Dec 11, 2017

Was the issue opened, is there a reference?

george-angel on May 9, 2018

@StephenLU0422 you problem coused by the mutil sandbox container in you node. please restart the kubelet, is’t will be ok.

huangjiasingle on Jan 14, 2018