kubeedge: can't deploy nginx-deployment sample succeed

What happened: When I finished setup KubeEdge, then I test the sample nginx-deployment using kubectl apply -f $GOPATH/src/github.com/kubeedge/kubeedge/build/deployment.yaml, the status is Pending (I wait for 8 min, it’s still pending), then I delete it, the status is Terminating(I wait for a long time but it is still Terminating). See below for more.

What you expected to happen: nginx-deployment is running well.

How to reproduce it (as minimally and precisely as possible): follow the doc.

Anything else we need to know?:

Environment:

  • KubeEdge version(e.g. cloudcore/edgecore --version):
./cloudcore --version
KubeEdge v1.2.1-dirty

CloudSide Environment:

  • Hardware configuration (e.g. lscpu): Intel(R) Core(TM) i3-4150 CPU @ 3.50GHz
  • OS (e.g. cat /etc/os-release): Ubuntu 16.04 LTS x86_64
  • Kernel (e.g. uname -a): ubuntu 4.4.0-174
  • Go version (e.g. go version): go1.12.14 linux/amd64
  • Others:

EdgeSide Environment: the same as CloudSide

  1. checking the system after setup KubeEdge, the node status is:
# kubectl get node
NAME         STATUS   ROLES    AGE   VERSION
edge-node    Ready    edge     31m   v1.17.1-kubeedge-v1.2.0
edge-node2   Ready    edge     30m   v1.17.1-kubeedge-v1.2.0
ubuntu       Ready    master   68m   v1.17.4

the node is ready, and the crds is as follows:

# kubectl get crds
NAME                                           CREATED AT
clusterobjectsyncs.reliablesyncs.kubeedge.io   2020-03-17T06:45:08Z
devicemodels.devices.kubeedge.io               2020-03-17T06:44:50Z
devices.devices.kubeedge.io                    2020-03-17T06:44:55Z
objectsyncs.reliablesyncs.kubeedge.io          2020-03-17T06:45:08Z
  1. deploy nginx:
kubectl apply -f $GOPATH/src/github.com/kubeedge/kubeedge/build/deployment.yaml

but it is pending.

  1. after some time, it still pending:
# kubectl get pod
NAME                                           READY   STATUS        RESTARTS   AGE
nginx-deployment-77698bff7d-jdm8k              0/1     Pending       0          12m
  1. check the information:
# kubectl describe pod nginx-deployment-77698bff7d-jdm8k
Name:           nginx-deployment-77698bff7d-jdm8k
Namespace:      default
Priority:       0
Node:           edge-node2/
Labels:         app=nginx
                pod-template-hash=77698bff7d
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/nginx-deployment-77698bff7d
Containers:
  nginx:
    Image:        nginx:1.15.12
    Port:         80/TCP
    Host Port:    80/TCP
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-gb4kq (ro)
Conditions:
  Type           Status
  PodScheduled   True 
Volumes:
  default-token-gb4kq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-gb4kq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  13m   default-scheduler  Successfully assigned default/nginx-deployment-77698bff7d-jdm8k to edge-node2

The only I know from the information is the deployment is assigned to the node, nothing else.

  1. try to get the log but refused:
# kubectl logs nginx-deployment-77698bff7d-jdm8k
Error from server: Get https://192.168.0.153:10250/containerLogs/default/nginx-deployment-77698bff7d-jdm8k/nginx: dial tcp 192.168.0.153:10250: connect: connection refused
  1. finaly I delete it.
# kubectl delete -f $GOPATH/src/github.com/kubeedge/kubeedge/build/deployment.yaml
deployment.apps "nginx-deployment" deleted

it is terminating:

# kubectl get pod
NAME                                           READY   STATUS        RESTARTS   AGE
nginx-deployment-77698bff7d-jdm8k              0/1     Terminating   0          20m

the pod information:

# kubectl get pod nginx-deployment-77698bff7d-jdm8k -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2020-03-17T07:12:58Z"
  deletionGracePeriodSeconds: 30
  deletionTimestamp: "2020-03-17T07:33:56Z"
  generateName: nginx-deployment-77698bff7d-
  labels:
    app: nginx
    pod-template-hash: 77698bff7d
  name: nginx-deployment-77698bff7d-jdm8k
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: nginx-deployment-77698bff7d
    uid: c3764762-55a3-41f4-a5b8-11706470f261
  resourceVersion: "11060"
  selfLink: /api/v1/namespaces/default/pods/nginx-deployment-77698bff7d-jdm8k
  uid: f37c9c28-d6c5-44a8-8743-f3d4fc5da89a
spec:
  containers:
  - image: nginx:1.15.12
    imagePullPolicy: IfNotPresent
    name: nginx
    ports:
    - containerPort: 80
      hostPort: 80
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-gb4kq
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: edge-node2
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-gb4kq
    secret:
      defaultMode: 420
      secretName: default-token-gb4kq
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-03-17T07:12:58Z"
    status: "True"
    type: PodScheduled
  phase: Pending
  qosClass: BestEffort

Is there sth I miss?

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 2
  • Comments: 17 (11 by maintainers)

Most upvoted comments

In my case, my edge node could not pull sandbox image (ex : kubeedge/pause:3.1). After change to connected repository, pending and terminating issued solved.

I have same issue too. In my case, edge part is raspberry pi (3 b+). I crossed-compile for arm architecutre. but nginx is still peding.

hi @fisherxu I checked #1406 . I tried deploy several time today and yesterday. My issue may be a little different. I delete pod forced, but the new pod is still pendding. I think the actually problem my be error: tls: use of closed connection, ie, the cloud side can’t write message to edge side, or vice versa.

I checked the logs on edge side, then found the following error (matched the tmie 15:12:59 on cloud side)

I0317 15:12:54.408802  100838 process.go:449] node connection event occur: cloud_disconnected
I0317 15:12:54.408828  100838 eventbus.go:88] Success in pubMQTT with topic: node/connection
E0317 15:12:59.323218  100838 process.go:130] failed to send message: tls: use of closed connection
E0317 15:12:59.323255  100838 process.go:196] websocket write error: failed to send message, error: tls: use of closed connection

it seems the error is error: tls: use of closed connection`. but the connection is ok, because I see the nodes is ready.