WSL: Failed to retrieve the list of available network interfaces when launching Spark in win10 Bash

  • A brief description I followed the official tutorial for Spark : http://spark.apache.org/docs/1.6.2/quick-start.html When I launched Spark through pyspark command, it reported the error as follows: Python 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2] on linux2 Type “help”, “copyright”, “credits” or “license” for more information. 16/12/08 12:41:09 INFO spark.SparkContext: Running Spark version 1.6.2 16/12/08 12:41:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 16/12/08 12:41:10 INFO spark.SecurityManager: Changing view acls to: root 16/12/08 12:41:10 INFO spark.SecurityManager: Changing modify acls to: root 16/12/08 12:41:10 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 16/12/08 12:41:10 WARN util.NetUtil: Failed to retrieve the list of available network interfaces java.net.SocketException: Cannot assign requested address (IOCTL SIOCGIFNETMASK failed) at java.net.NetworkInterface.getAll(Native Method) at java.net.NetworkInterface.getNetworkInterfaces(NetworkInterface.java:343) at io.netty.util.NetUtil.<clinit>(NetUtil.java:156) at io.netty.channel.socket.DefaultServerSocketChannelConfig.<init>(DefaultServerSocketChannelConfig.java:39) at io.netty.channel.socket.nio.NioServerSocketChannel$NioServerSocketChannelConfig.<init>(NioServerSocketChannel.java:189) at io.netty.channel.socket.nio.NioServerSocketChannel$NioServerSocketChannelConfig.<init>(NioServerSocketChannel.java:187) at io.netty.channel.socket.nio.NioServerSocketChannel.<init>(NioServerSocketChannel.java:85) at io.netty.channel.socket.nio.NioServerSocketChannel.<init>(NioServerSocketChannel.java:70) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at io.netty.bootstrap.AbstractBootstrap$BootstrapChannelFactory.newChannel(AbstractBootstrap.java:453) at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:306) at io.netty.bootstrap.AbstractBootstrap.doBind(AbstractBootstrap.java:271) at io.netty.bootstrap.AbstractBootstrap.bind(AbstractBootstrap.java:267) at org.apache.spark.network.server.TransportServer.init(TransportServer.java:129) at org.apache.spark.network.server.TransportServer.<init>(TransportServer.java:74) at org.apache.spark.network.TransportContext.createServer(TransportContext.java:103) at org.apache.spark.rpc.netty.NettyRpcEnv.startServer(NettyRpcEnv.scala:125) at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:450) at org.apache.spark.rpc.netty.NettyRpcEnvFactory$$anonfun$4.apply(NettyRpcEnv.scala:449) at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2024) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2015) at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:454) at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:53) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:253) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:193) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:288) at org.apache.spark.SparkContext.<init>(SparkContext.scala:457) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) at py4j.Gateway.invoke(Gateway.java:214) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68) at py4j.GatewayConnection.run(GatewayConnection.java:209) at java.lang.Thread.run(Thread.java:745) 16/12/08 12:41:10 WARN util.NetUtil: Failed to find the loopback interface 16/12/08 12:41:10 INFO util.Utils: Successfully started service ‘sparkDriver’ on port 61992. 16/12/08 12:41:10 INFO slf4j.Slf4jLogger: Slf4jLogger started 16/12/08 12:41:11 INFO Remoting: Starting remoting 16/12/08 12:41:11 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@127.0.1.1:61993] 16/12/08 12:41:11 INFO util.Utils: Successfully started service ‘sparkDriverActorSystem’ on port 61993. 16/12/08 12:41:11 INFO spark.SparkEnv: Registering MapOutputTracker 16/12/08 12:41:11 INFO spark.SparkEnv: Registering BlockManagerMaster 16/12/08 12:41:11 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-77a9c925-853e-436a-bf01-5f26e797f68c 16/12/08 12:41:11 INFO storage.MemoryStore: MemoryStore started with capacity 511.1 MB 16/12/08 12:41:11 INFO spark.SparkEnv: Registering OutputCommitCoordinator 16/12/08 12:41:11 INFO server.Server: jetty-8.y.z-SNAPSHOT 16/12/08 12:41:11 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 16/12/08 12:41:11 INFO util.Utils: Successfully started service ‘SparkUI’ on port 4040. 16/12/08 12:41:11 INFO ui.SparkUI: Started SparkUI at http://127.0.1.1:4040 16/12/08 12:41:11 INFO executor.Executor: Starting executor ID driver on host localhost 16/12/08 12:41:11 INFO util.Utils: Successfully started service ‘org.apache.spark.network.netty.NettyBlockTransferService’ on port 61997. 16/12/08 12:41:11 INFO netty.NettyBlockTransferService: Server created on 61997 16/12/08 12:41:11 INFO storage.BlockManagerMaster: Trying to register BlockManager 16/12/08 12:41:11 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:61997 with 511.1 MB RAM, BlockManagerId(driver, localhost, 61997) 16/12/08 12:41:11 INFO storage.BlockManagerMaster: Registered BlockManager Welcome to ____ __ / / ___ / / \ / _ / _ `/ __/ '/ / / ._/_,// //_\ version 1.6.2 //

Using Python version 2.7.6 (default, Jun 22 2015 17:58:13) SparkContext available as sc, HiveContext available as sqlContext.

---------------it ends----------------- So I conducted a experiment on ubuntu on virtualBox in the same config, Results as follows: Python 2.7.12 (default, Jul 1 2016, 15:12:24) [GCC 5.4.0 20160609] on linux2 Type “help”, “copyright”, “credits” or “license” for more information. Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties 16/12/08 20:06:30 INFO SparkContext: Running Spark version 1.6.2 16/12/08 20:06:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable 16/12/08 20:06:38 WARN Utils: Your hostname, hiocde resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface eth0) 16/12/08 20:06:38 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 16/12/08 20:06:38 INFO SecurityManager: Changing view acls to: hiocde 16/12/08 20:06:38 INFO SecurityManager: Changing modify acls to: hiocde 16/12/08 20:06:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hiocde); users with modify permissions: Set(hiocde) 16/12/08 20:06:44 INFO Utils: Successfully started service ‘sparkDriver’ on port 44831. 16/12/08 20:06:47 INFO Slf4jLogger: Slf4jLogger started 16/12/08 20:06:47 INFO Remoting: Starting remoting 16/12/08 20:06:49 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.0.2.15:45304] 16/12/08 20:06:49 INFO Utils: Successfully started service ‘sparkDriverActorSystem’ on port 45304. 16/12/08 20:06:50 INFO SparkEnv: Registering MapOutputTracker 16/12/08 20:06:50 INFO SparkEnv: Registering BlockManagerMaster 16/12/08 20:06:50 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-74bcf03b-e04c-4456-a8b6-b146bbc27aa4 16/12/08 20:06:51 INFO MemoryStore: MemoryStore started with capacity 511.1 MB 16/12/08 20:06:51 INFO SparkEnv: Registering OutputCommitCoordinator 16/12/08 20:06:54 INFO Utils: Successfully started service ‘SparkUI’ on port 4040. 16/12/08 20:06:54 INFO SparkUI: Started SparkUI at http://10.0.2.15:4040 16/12/08 20:06:55 INFO Executor: Starting executor ID driver on host localhost 16/12/08 20:06:55 INFO Utils: Successfully started service ‘org.apache.spark.network.netty.NettyBlockTransferService’ on port 33630. 16/12/08 20:06:55 INFO NettyBlockTransferService: Server created on 33630 16/12/08 20:06:55 INFO BlockManagerMaster: Trying to register BlockManager 16/12/08 20:06:55 INFO BlockManagerMasterEndpoint: Registering block manager localhost:33630 with 511.1 MB RAM, BlockManagerId(driver, localhost, 33630) 16/12/08 20:06:55 INFO BlockManagerMaster: Registered BlockManager Welcome to ____ __ / / ___ / / \ / _ / _ `/ __/ '/ / / ._/_,// //_\ version 1.6.2 //

Using Python version 2.7.12 (default, Jul 1 2016 15:12:24) SparkContext available as sc, HiveContext available as sqlContext.

---------------it ends----------------- As above, Ubuntu not reported the error.

I searched it and found this in stackoverflow: http://stackoverflow.com/questions/17601130/installed-spark-built-against-right-hadoop-version-getting-cannot-assigned-re So I changed /etc/hosts and added a line: 127.0.1.1 my-host-name; and setting the environmental variable SPARK_LOCAL_IP=my-host-name. But the try was in vain, the error still existed!

I really hope I can run Spark on win10 Bash not on VM, So any idea can solve this problem or this function has not implemented yet like the issue said: https://github.com/Microsoft/BashOnWindows/issues/69 ? thanks.

  • Expected results

  • Actual results (with terminal output if applicable)

  • Your Windows build number 14393.447

  • Steps / All commands required to reproduce the error from a brand new installation

  1. pyspark //reported the error, but I can enter the python-shell

  2. textFile = sc.textFile(“README.md”) //normal in python-shell

  3. textFile.count() /*** reported the error again and couldn’t give number of items in RDD. Some report as follows: 16/12/08 13:33:56 ERROR : Cannot assign requested address (IOCTL SIOCGIFNETMASK failed) java.net.SocketException: Cannot assign requested address (IOCTL SIOCGIFNETMASK failed) at java.net.NetworkInterface.getAll(Native Method) at java.net.NetworkInterface.getNetworkInterfaces(NetworkInterface.java:343) … ***/

  • Strace of the failing command

  • Required packages and commands to install

See our contributing instructions for assistance.

About this issue

  • Original URL
  • State: closed
  • Created 10 years ago
  • Comments: 17 (2 by maintainers)

Most upvoted comments

Thanks @fpqc for chiming in 😃