edgemesh: EdgeMesh-agent deployment failure on Raspberry Pi 4 (ARM based CPU)

What happened:

edgemesh-agent cannot be deployed on Raspberry Pi 4.

pi@raspberrypi:~ $ docker ps -a

CONTAINER ID   IMAGE                     COMMAND            CREATED         STATUS                     PORTS     NAMES

abbba6511322   kubeedge/edgemesh-agent   "edgemesh-agent"   2 minutes ago   Exited (1) 2 minutes ago             k8s_edgemesh-agent_edgemesh-agent-edge-7bbsm_kubeedge_1dce335a-e6a7-4b91-b2b3-e11d97c395b8_92

a78a927b7ec2   kubeedge/pause-arm:3.1    "/pause"           8 hours ago     Up 8 hours                           k8s_POD_edgemesh-agent-edge-7bbsm_kubeedge_1dce335a-e6a7-4b91-b2b3-e11d97c395b8_0

Check docker container through docker logs

pi@raspberrypi:~ $ docker logs abbba6511322

standard_init_linux.go:228: exec user process caused: exec format error

What you expected to happen:

EdgeMesh deployment complete without any error

root@charliecloud-ASUSPRO-D340MC-C-S340MC:/home/charlie-cloud# kubectl get all -n kubeedge -o wide

NAME                                   READY   STATUS    RESTARTS   AGE     IP              NODE                NOMINATED NODE   READINESS GATES
pod/edgemesh-agent-edge-7bbsm          1/1     Running     57         5h10m   192.168.0.86    raspberrypi         <none>           <none>
pod/edgemesh-agent-edge-rbcrv          1/1     Running   0          6h6m    192.168.0.206   default-edge-node   <none>           <none>
pod/edgemesh-server-7bdbcfc7b5-hf4ds   1/1     Running   0          36s     <none>          cloud-node          <none>           <none>

NAME                                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE    CONTAINERS       IMAGES                           SELECTOR
daemonset.apps/edgemesh-agent-cloud   0         0         0       0            0           <none>          6h7m   edgemesh-agent   kubeedge/edgemesh-agent:latest   k8s-app=kubeedge,kubeedge=edgemesh-agent
daemonset.apps/edgemesh-agent-edge    2         2         2       2            2           <none>          6h6m   edgemesh-agent   kubeedge/edgemesh-agent:latest   k8s-app=kubeedge,kubeedge=edgemesh-agent

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS        IMAGES                            SELECTOR
deployment.apps/edgemesh-server   1/1     1            1           6h44m   edgemesh-server   kubeedge/edgemesh-server:latest   k8s-app=kubeedge,kubeedge=edgemesh-server

NAME                                         DESIRED   CURRENT   READY   AGE     CONTAINERS        IMAGES                            SELECTOR
replicaset.apps/edgemesh-server-7bdbcfc7b5   1         1         1       6h44m   edgemesh-server   kubeedge/edgemesh-server:latest   k8s-app=kubeedge,kubeedge=edgemesh-server,pod-template-hash=7bdbcfc7b5

How to reproduce it (as minimally and precisely as possible):

Deploy a k8s cluster through kubeadm
Deploy a KubeEdge cluster (Binary Method)
- master node: k8s master + KubeEdge cloud
- edge node 1: KubeEdge edge node (intel CPU PC)
- edge node 2: KubeEdge edge node (arm CPU Raspberry Pi 4)
Deploy EdgeMesh through kubectl apply -f by the Official Document

Anything else we need to know?:

command:

kubectl get all -n kubeedge -o wide

root@charliecloud-ASUSPRO-D340MC-C-S340MC:/home/charlie-cloud# kubectl get all -n kubeedge -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP              NODE                NOMINATED NODE   READINESS GATES
pod/edgemesh-agent-edge-7bbsm          0/1     Error     57         5h10m   192.168.0.86    raspberrypi         <none>           <none>
pod/edgemesh-agent-edge-rbcrv          1/1     Running   0          6h6m    192.168.0.206   default-edge-node   <none>           <none>
pod/edgemesh-server-7bdbcfc7b5-hf4ds   0/1     Pending   0          36s     <none>          cloud-node          <none>           <none>

NAME                                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE    CONTAINERS       IMAGES                           SELECTOR
daemonset.apps/edgemesh-agent-cloud   0         0         0       0            0           <none>          6h7m   edgemesh-agent   kubeedge/edgemesh-agent:latest   k8s-app=kubeedge,kubeedge=edgemesh-agent
daemonset.apps/edgemesh-agent-edge    2         2         1       2            1           <none>          6h6m   edgemesh-agent   kubeedge/edgemesh-agent:latest   k8s-app=kubeedge,kubeedge=edgemesh-agent

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS        IMAGES                            SELECTOR
deployment.apps/edgemesh-server   0/1     1            0           6h44m   edgemesh-server   kubeedge/edgemesh-server:latest   k8s-app=kubeedge,kubeedge=edgemesh-server

NAME                                         DESIRED   CURRENT   READY   AGE     CONTAINERS        IMAGES                            SELECTOR
replicaset.apps/edgemesh-server-7bdbcfc7b5   1         1         0       6h44m   edgemesh-server   kubeedge/edgemesh-server:latest   k8s-app=kubeedge,kubeedge=edgemesh-server,pod-template-hash=7bdbcfc7b5

Environment:

EdgeMesh version: main branch

root@charliecloud-ASUSPRO-D340MC-C-S340MC:/home/charlie-cloud/Downloads/edgemesh# git checkout

M	build/server/edgemesh/05-configmap.yaml
Your branch is up to date with 'origin/main'.

(2021/09/17 18:46)

Kubernetes version (use kubectl version):

root@charliecloud-ASUSPRO-D340MC-C-S340MC:/home/charlie-cloud/Downloads/edgemesh# kubectl version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:07:41Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.14", GitCommit:"0fd2b5afdfe3134d6e1531365fdb37dd11f54d1c", GitTreeState:"clean", BuildDate:"2021-08-11T18:02:17Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

KubeEdge version(e.g. cloudcore --version and edgecore --version):

cloudcore on k8s master node

root@charliecloud-ASUSPRO-D340MC-C-S340MC:/home/charlie-cloud/Downloads/edgemesh# cloudcore --version

KubeEdge v1.8.0-beta.0.81+149aa721b82ae9

The intel CPU based PC

charlie-edge@charlieedge-CM1745:~$ edgecore --version

KubeEdge v1.8.0-beta.0.81+149aa721b82ae9

The ARM CPU based Raspberry Pi 4

pi@raspberrypi:~ $ edgecore --version

KubeEdge v1.8.0-beta.0.91+29905c6704b6e9

Cloud nodes Environment:

Hardware configuration (e.g. lscpu):

root@charliecloud-ASUSPRO-D340MC-C-S340MC:/home/charlie-cloud/Downloads/edgemesh# lscpu

  Architecture:                    x86_64
  CPU op-mode(s):                  32-bit, 64-bit
  Byte Order:                      Little Endian
  Address sizes:                   39 bits physical, 48 bits virtual
  CPU(s):                          6
  On-line CPU(s) list:             0-5
  Thread(s) per core:              1
  Core(s) per socket:              6
  Socket(s):                       1
  NUMA node(s):                    1
  Vendor ID:                       GenuineIntel
  CPU family:                      6
  Model:                           158
  Model name:                      Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz
  Stepping:                        10
  CPU MHz:                         887.468
  CPU max MHz:                     4100.0000
  CPU min MHz:                     800.0000
  BogoMIPS:                        6000.00
  Virtualization:                  VT-x
  L1d cache:                       192 KiB
  L1i cache:                       192 KiB
  L2 cache:                        1.5 MiB
  L3 cache:                        9 MiB
  NUMA node0 CPU(s):               0-5
  Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
  Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT disabled
  Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT disabled
  Vulnerability Meltdown:          Mitigation; PTI
  Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling
  Vulnerability Srbds:             Mitigation; Microcode
  Vulnerability Tsx async abort:   Mitigation; Clear CPU buffers; SMT disabled
  Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constan
                                  t_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16
                                  xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd
                                  ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt in
                                  tel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d

OS (e.g. cat /etc/os-release):

root@charliecloud-ASUSPRO-D340MC-C-S340MC:/home/charlie-cloud/Downloads/edgemesh# cat /etc/os-release

  NAME="Ubuntu"
  VERSION="20.04.3 LTS (Focal Fossa)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 20.04.3 LTS"
  VERSION_ID="20.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=focal
  UBUNTU_CODENAME=focal

Kernel (e.g. uname -a):

root@charliecloud-ASUSPRO-D340MC-C-S340MC:/home/charlie-cloud/Downloads/edgemesh# uname -a

Linux charliecloud-ASUSPRO-D340MC-C-S340MC 5.11.0-34-generic #36~20.04.1-Ubuntu SMP Fri Aug 27 08:06:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Go version (e.g. go version):

root@charliecloud-ASUSPRO-D340MC-C-S340MC:/home/charlie-cloud/Downloads/edgemesh# go version

go version go1.17 linux/amd64

Others:

Edge nodes Environment:

PC (intel CPU)

edgecore version (e.g. edgecore --version):

charlie-edge@charlieedge-CM1745:~$ edgecore --version

KubeEdge v1.8.0-beta.0.81+149aa721b82ae9

Hardware configuration (e.g. lscpu):

charlie-edge@charlieedge-CM1745:~$ lscpu

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              2
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      21
Model:                           16
Model name:                      AMD A8-5500 APU with Radeon(tm) HD Graphics
Stepping:                        1
Frequency boost:                 enabled
CPU MHz:                         1400.000
CPU max MHz:                     3200.0000
CPU min MHz:                     1400.0000
BogoMIPS:                        6400.57
Virtualization:                  AMD-V
L1d cache:                       32 KiB
L1i cache:                       128 KiB
L2 cache:                        4 MiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled v
                                ia prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user
                                pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, STIBP disabled,
                                RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtr
                                r pge mca cmov pat pse36 clflush mmx fxsr sse s
                                se2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtsc
                                p lm constant_tsc rep_good nopl nonstop_tsc cpu
                                id extd_apicid aperfmperf pni pclmulqdq monitor
                                ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave 
                                avx f16c lahf_lm cmp_legacy svm extapic cr8_leg
                                acy abm sse4a misalignsse 3dnowprefetch osvw ib
                                s xop skinit wdt lwp fma4 tce nodeid_msr tbm to
                                poext perfctr_core perfctr_nb cpb hw_pstate ssb
                                d vmmcall bmi1 arat npt lbrv svm_lock nrip_save
                                tsc_scale vmcb_clean flushbyasid decodeassists
                                pausefilter pfthreshold

OS (e.g. cat /etc/os-release):

charlie-edge@charlieedge-CM1745:~$ cat /etc/os-release

NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Kernel (e.g. uname -a):

charlie-edge@charlieedge-CM1745:~$ uname -a

Linux charlieedge-CM1745 5.11.0-34-generic #36~20.04.1-Ubuntu SMP Fri Aug 27 08:06:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Go version (e.g. go version):

charlie-edge@charlieedge-CM1745:~$ go version

go version go1.17 linux/amd64

Others:

Raspberry Pi (arm CPU)

edgecore version (e.g. edgecore --version):

pi@raspberrypi:~ $ edgecore --version

KubeEdge v1.8.0-beta.0.91+29905c6704b6e9

Hardware configuration (e.g. lscpu):

pi@raspberrypi:~ $ lscpu

Architecture:        armv7l
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
Vendor ID:           ARM
Model:               3
Model name:          Cortex-A72
Stepping:            r0p3
CPU max MHz:         1500.0000
CPU min MHz:         600.0000
BogoMIPS:            108.00
Flags:               half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32

OS (e.g. cat /etc/os-release):

pi@raspberrypi:~ $ cat /etc/os-release

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Kernel (e.g. uname -a):

pi@raspberrypi:~ $ uname -a

Linux raspberrypi 5.10.60-v7l+ #1449 SMP Wed Aug 25 15:00:44 BST 2021 armv7l GNU/Linux

Go version (e.g. go version):

pi@raspberrypi:~ $ go version

go version go1.17 linux/arm

Others:

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 21 (10 by maintainers)

Most upvoted comments

https://github.com/kubeedge/edgemesh/issues/138, we have support the multi architecure, you can see the Makefile for detail, run make docker-cross-build to build the images

khalid-huang on Oct 28, 2021

Because I do not have a arm64 arch machine, so I dont not test the Dockerfile yet. If we would like to using the x86 marchine to build a arm64 image, we will need qemu. And we need some time to do the job. If you have a arm64 arch machine, you can use the follow dockerfile to build the image

FROM arm64v8/alpine:3.9

RUN apk update && apk --no-cache add iptables

COPY ./edgemesh-agent /usr/local/bin/edgemesh-agent

ENTRYPOINT ["edgemesh-agent"]

khalid-huang on Oct 13, 2021

Maybe we can refer to https://github.com/kubeedge/kubeedge/blob/master/Makefile#L166 to support multi arch for EdgeMesh. But we need some time to do this work. If you just need an arm version for EdgeMesh, may be you can try CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o agent-arm ./agent/cmd/edgemesh-agent/agent.go to build edgemesh-agent binary.

khalid-huang on Sep 18, 2021