kubeedge: imitator.go:222] failed to unmarshal message content to unstructured obj: Object 'Kind' is missing in '{"metadata":{...

Same Problem in v1.12.2

May 30 12:10:52 edge.kubeedge edgecore[8418]: E0530 12:10:52.621760 8418 imitator.go:235] failed to unmarshal message content to unstructured obj: Object ‘Kind’ is missing in '{“metadata”:

edgecore journal: Feb 13 13:41:44 node3 edgecore[696]: E0213 13:41:44.889527 696 imitator.go:235] failed to unmarshal message content to unstructured obj: Object ‘Kind’ is missing in ‘{“metadata”:{“name”:“node3”,“creationTimestamp”:null,“labels”:{“beta.kubernetes.io/arch”:“amd64”,“beta.kubernetes.io/os”:“linux”,“kubernetes.io/arch”:“amd64”,“kubernetes.io/hostname”:“node3”,“kubernetes.io/os”:“linux”,“node-role.kubernetes.io/agent”:“”,“node-role.kubernetes.io/edge”:“”},“annotations”:{“volumes.kubernetes.io/controller-managed-attach-detach”:“true”}},“spec”:{},“status”:{“capacity”:{“cpu”:“2”,“ephemeral-storage”:“29751268Ki”,“hugepages-2Mi”:“0”,“memory”:“2030768Ki”,“pods”:“110”},“allocatable”:{“cpu”:“2”,“ephemeral-storage”:“27418768544”,“hugepages-2Mi”:“0”,“memory”:“1928368Ki”,“pods”:“110”},“conditions”:[{“type”:“MemoryPressure”,“status”:“False”,“lastHeartbeatTime”:“2023-02-13T13:41:44Z”,“lastTransitionTime”:“2023-02-13T13:41:44Z”,“reason”:“KubeletHasSufficientMemory”,“message”:“kubelet has sufficient memory available”},{“type”:“DiskPressure”,“status”:“False”,“lastHeartbeatTime”:“2023-02-13T13:41:44Z”,“lastTransitionTime”:“2023-02-13T13:41:44Z”,“reason”:“KubeletHasNoDiskPressure”,“message”:“kubelet has no disk pressure”},{“type”:“PIDPressure”,“status”:“False”,“lastHeartbeatTime”:“2023-02-13T13:41:44Z”,“lastTransitionTime”:“2023-02-13T13:41:44Z”,“reason”:“KubeletHasSufficientPID”,“message”:“kubelet has sufficient PID available”},{“type”:“Ready”,“status”:“False”,“lastHeartbeatTime”:“2023-02-13T13:41:44Z”,“lastTransitionTime”:“2023-02-13T13:41:44Z”,“reason”:“KubeletNotReady”,“message”:“PLEG is not healthy: pleg has yet to be successful”}],“addresses”:[{“type”:“InternalIP”,“address”:“192.168.0.203”},{“type”:“Hostname”,“address”:“node3”}],“daemonEndpoints”:{“kubeletEndpoint”:{“Port”:10350}},“nodeInfo”:{“machineID”:“f0b0fc81fa174d229e625821ee9c3192”,“systemUUID”:“09a8a270-d4ed-ce4a-878d-0bf50257e16d”,“bootID”:“ca817b86-d22a-4a6d-bcd0-78b1fb67862a”,“kernelVersion”:“5.4.0-139-generic”,“osImage”:“Ubuntu 20.04.5 LTS”,“containerRuntimeVersion”:“docker://20.10.12”,“kubeletVersion”:“v1.23.15-kubeedge-v1.13.0”,“kubeProxyVersion”:“v0.0.0-master+$Format:%H$”,“operatingSystem”:“linux”,“architecture”:“amd64”},“images”:[{“names”:[“kubeedge/installation-package@sha256:780cbe1d64aad905b75cf877c5023868c8eef36c5936b5bec3bdeaef2ecb897f”,“kubeedge/installation-package:v1.13.0”],“sizeBytes”:208755554},{“names”:[“registry.aliyuncs.com/google_containers/kube-proxy@sha256:239f53a4e58bdf281a1ea120fb93161daee2d0e9da7ad03a2ddcb28a83f7325a”,“registry.aliyuncs.com/google_containers/kube-proxy:v1.23.16”],“sizeBytes”:110832791},{“names”:[“nginx@sha256:e8ab8d42e0c34c104ac60b43ba60b19af08e19a0e6d50396bdfd4cef0347ba83”,“nginx:1.15.4”],“sizeBytes”:109083698},{“names”:[“kubeedge/edgemesh-agent@sha256:3a7fee53b98f1be02a15da03e4fda263608d05dbdfdfd0f1bff89d11892db83d”,“kubeedge/edgemesh-agent:v1.13.0”],“sizeBytes”:73737840},{“names”:[“flannel/flannel@sha256:08e150f603cc4759a48e986fc26c0f531af1395d1bd48981226f86f45521b01c”,“flannel/flannel:v0.20.2”],“sizeBytes”:59604122},{“names”:[“eclipse-mosquitto@sha256:fe2ab4877478b3d848819ec6a54ee54bc1e6a6ece1812b90528fa043737c85ea”,“eclipse-mosquitto:1.6.15”],“sizeBytes”:11564549},{“names”:[“flannel/flannel-cni-plugin@sha256:bf4b62b131666d040f35a327d906ee5a3418280b68a88d9b9c7e828057210443”,“flannel/flannel-cni-plugin:v1.1.2”],“sizeBytes”:7966554},{“names”:[“kubeedge/pause@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db”,“kubeedge/pause:3.6”],“sizeBytes”:682696}]}}’

I have a similar issue with version 1.12.0

The node is registered but never becomes ready

On the edge side I am seeing:

Jan 29 21:03:30 edge5 edgecore[83810]: E0129 21:03:30.043669   83810 kubelet.go:1224] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.044095   83810 server.go:173] "Starting to listen read-only" address="127.0.0.1" port=10350
Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.053140   83810 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.054932   83810 server.go:389] "Adding debug handlers to kubelet server"
Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.062325   83810 volume_manager.go:291] "Starting Kubelet Volume Manager"
Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.073228   83810 desired_state_of_world_populator.go:146] "Desired state populator starts to run"

Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.524784   83810 kubelet_node_status.go:69] "Attempting to register node" node="edge5"
Jan 29 21:03:30 edge5 edgecore[83810]: E0129 21:03:30.540756   83810 imitator.go:235] failed to unmarshal message content to unstructured obj: Object 'Kind' is missing in '{"metadata":{"name":"edge5","creationTimestamp":null,"labels":{"beta.kubernetes.io/arch":"arm64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"arm64","kubernetes.io/hostname":"edge5","kubernetes.io/os":"linux","node-role.kubernetes.io/agent":"","node-role.kubernetes.io/edge":""},"annotations":{"volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{},"status":{"capacity":{"cpu":"4","hugepages-1Gi":"0","hugepages-2Mi":"0","hugepages-32Mi":"0","hugepages-64Ki":"0","memory":"926260Ki","pods":"110"},"allocatable":{"cpu":"4","hugepages-1Gi":"0","hugepages-2Mi":"0","hugepages-32Mi":"0","hugepages-64Ki":"0","memory":"823860Ki","pods":"110"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2023-01-29T21:03:30Z","lastTransitionTime":"2023-01-29T21:03:30Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2023-01-29T21:03:30Z","lastTransitionTime":"2023-01-29T21:03:30Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2023-01-29T21:03:30Z","lastTransitionTime":"2023-01-29T21:03:30Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"False","lastHeartbeatTime":"2023-01-29T21:03:30Z","lastTransitionTime":"2023-01-29T21:03:30Z","reason":"KubeletNotReady","message":"[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful, missing node capacity for resources: ephemeral-storage]"}],"addresses":[{"type":"InternalIP","address":"192.168.60.38"},{"type":"Hostname","address":"edge5"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10350}},"nodeInfo":{"machineID":"526f2d7f62924c7ab708d9d111b51df0","systemUUID":"526f2d7f62924c7ab708d9d111b51df0","bootID":"a7a81426-9a36-45c0-a2db-d023e76799a9","kernelVersion":"5.19.0-1004-raspi","osImage":"Ubuntu 22.10","containerRuntimeVersion":"docker://20.10.23","kubeletVersion":"v1.22.6-kubeedge-v1.12.1","kubeProxyVersion":"v0.0.0-master+$Format:%H$","operatingSystem":"linux","architecture":"arm64"},"images":[{"names":["kubeedge/installation-package@sha256:edc2cb87fbc56061c951bbd9f03dc15197bdcf4584d9da2ec4f1f1eec04ab0d9","kubeedge/installation-package:v1.12.1"],"sizeBytes":192770920},{"names":["eclipse-mosquitto@sha256:fe2ab4877478b3d848819ec6a54ee54bc1e6a6ece1812b90528fa043737c85ea","eclipse-mosquitto:1.6.15"],"sizeBytes":11225541},{"names":["kubeedge/pause@sha256:d22591b61e9c2b52aecbf07106d5db313c4f178e404d660b32517b18fcbf0144","kubeedge/pause:3.1"],"sizeBytes":524888}]}}'
Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.593468   83810 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.620059   83810 kubelet_node_status.go:72] "Successfully registered node" node="edge5"
Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.941447   83810 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv6
Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.947230   83810 status_manager.go:158] "Starting to sync pod status with apiserver"
Jan 29 21:03:30 edge5 edgecore[83810]: I0129 21:03:30.949075   83810 kubelet.go:1807] "Starting kubelet main sync loop"
Jan 29 21:03:30 edge5 edgecore[83810]: E0129 21:03:30.951037   83810 kubelet.go:1831] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Jan 29 21:03:31 edge5 edgecore[83810]: E0129 21:03:31.055466   83810 kubelet.go:1831] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Jan 29 21:03:31 edge5 edgecore[83810]: E0129 21:03:31.256032   83810 kubelet.go:1831] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Jan 29 21:03:31 edge5 edgecore[83810]: E0129 21:03:31.657191   83810 kubelet.go:1831] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Jan 29 21:03:32 edge5 edgecore[83810]: E0129 21:03:32.458539   83810 kubelet.go:1831] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Jan 29 21:03:34 edge5 edgecore[83810]: E0129 21:03:34.059359   83810 kubelet.go:1831] "Skipping pod synchronization" err="container runtime status check may not have completed yet"

On cloudcore I am seeing:

 49 E0129 21:02:52.367630       1 ws.go:107] failed to read message, error: websocket: close 1006 (abnormal closure): unexpected EOF
 50 E0129 21:02:52.368090       1 message_handler.go:166] projectID e632aba927ea4ac2b575ec1603d56f10 node edge5 read message err
 51 I0129 21:02:52.368270       1 message_handler.go:139] edge node edge5 for project e632aba927ea4ac2b575ec1603d56f10 disConnected
 52 E0129 21:02:52.368381       1 node_session.go:194] syncAckMessage err: AckMessageQueue for node edge5 has shutdown
 53 W0129 21:02:52.368510       1 upstream.go:218] parse message: 4aa6a3d4-d88a-42f6-99a8-3316f77f1765 resource type with error, message resource: node/edge5, err: resource type not found
 54 E0129 21:02:52.368523       1 node_session.go:218] syncNoAckMessage err: NoAckMessageQueue for node edge5 has shutdown
 55 W0129 21:03:03.877997       1 upstream.go:218] parse message: b302e76b-5c1c-40c6-9947-f9cffde350f3 resource type with error, message resource: node/edge5, err: resource type not found
 56 I0129 21:03:03.878093       1 message_handler.go:122] edge node edge5 for project e632aba927ea4ac2b575ec1603d56f10 connected
 57 I0129 21:03:03.878464       1 node_session.go:136] Start session for edge node edge5
 58 I0129 21:03:03.891998       1 upstream.go:89] Dispatch message: 6ccf5808-537d-4a0a-9365-2d1f94a4b7fa
 59 I0129 21:03:03.892098       1 upstream.go:96] Message: 6ccf5808-537d-4a0a-9365-2d1f94a4b7fa, resource type is: membership/detail
 60 E0129 21:03:04.867955       1 ws.go:107] failed to read message, error: websocket: close 1006 (abnormal closure): unexpected EOF
 61 E0129 21:03:04.868515       1 message_handler.go:166] projectID e632aba927ea4ac2b575ec1603d56f10 node edge5 read message err
 62 I0129 21:03:04.868696       1 message_handler.go:139] edge node edge5 for project e632aba927ea4ac2b575ec1603d56f10 disConnected
 63 W0129 21:03:04.868943       1 upstream.go:218] parse message: 5aaf4f34-9b10-4171-9901-58bb03b80c60 resource type with error, message resource: node/edge5, err: resource type not found
 64 E0129 21:03:04.869076       1 node_session.go:218] syncNoAckMessage err: NoAckMessageQueue for node edge5 has shutdown
 65 E0129 21:03:04.869151       1 node_session.go:194] syncAckMessage err: AckMessageQueue for node edge5 has shutdown
 66 W0129 21:03:16.535665       1 upstream.go:218] parse message: 170aed64-6177-4016-afb7-5101fba6922d resource type with error, message resource: node/edge5, err: resource type not found
 67 I0129 21:03:16.535744       1 message_handler.go:122] edge node edge5 for project e632aba927ea4ac2b575ec1603d56f10 connected
 68 I0129 21:03:16.536469       1 node_session.go:136] Start session for edge node edge5
 69 I0129 21:03:16.565298       1 upstream.go:89] Dispatch message: 7d23c4dd-6932-4a83-905d-69849bd33e23
 70 I0129 21:03:16.565385       1 upstream.go:96] Message: 7d23c4dd-6932-4a83-905d-69849bd33e23, resource type is: membership/detail
 71 E0129 21:03:17.632038       1 ws.go:107] failed to read message, error: websocket: close 1006 (abnormal closure): unexpected EOF
 72 E0129 21:03:17.632587       1 message_handler.go:166] projectID e632aba927ea4ac2b575ec1603d56f10 node edge5 read message err
 73 I0129 21:03:17.633256       1 message_handler.go:139] edge node edge5 for project e632aba927ea4ac2b575ec1603d56f10 disConnected
 74 E0129 21:03:17.633465       1 node_session.go:218] syncNoAckMessage err: NoAckMessageQueue for node edge5 has shutdown
 75 W0129 21:03:17.633745       1 upstream.go:218] parse message: af580cdc-1f7e-4e76-b30a-a79851221d9c resource type with error, message resource: node/edge5, err: resource type not found
 76 E0129 21:03:17.633292       1 node_session.go:194] syncAckMessage err: AckMessageQueue for node edge5 has shutdown
 77 W0129 21:03:28.910772       1 upstream.go:218] parse message: 675e75a7-cd55-472a-888a-4f9fd1a34a54 resource type with error, message resource: node/edge5, err: resource type not found
 78 I0129 21:03:28.910813       1 message_handler.go:122] edge node edge5 for project e632aba927ea4ac2b575ec1603d56f10 connected
 79 I0129 21:03:28.911508       1 node_session.go:136] Start session for edge node edge5
 80 I0129 21:03:28.926545       1 upstream.go:89] Dispatch message: 9c387d15-e128-41fb-9b6c-d12443b90e0d
 81 I0129 21:03:28.926648       1 upstream.go:96] Message: 9c387d15-e128-41fb-9b6c-d12443b90e0d, resource type is: membership/detail
 82 E0129 21:03:30.080310       1 upstream.go:1228] Query lease %s failed, error: %vedge5lease.coordination.k8s.io "edge5" not found