spark: [BUG]: Trying to follow the "Getting Started" guide step by step

Describe the bug I was following the Getting Started guide step by step. When I execute the following:

C:\Users\j.shaer\source\repos\HelloSpark\HelloSpark\bin\Debug\netcoreapp2.1>spark-submit 
`--class org.apache.spark.deploy.DotnetRunner ` --master local ` microsoft-spark-2.4.x-0.1.0.jar ` HelloSpark

I get this:

Exception in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR file:/C:/Users/j.shaer/source/repos/HelloSpark/HelloSpark/bin/Debug/netcoreapp2.1/%60--class
        at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
        at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:221)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:116)
        at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$3.<init>(SparkSubmit.scala:911)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:911)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 26 (10 by maintainers)

Most upvoted comments

One temporary work-around to avoid seeing the spark temporary files error is to add the following two lines in the log4j.properties file of your spark installation:

log4j.logger.org.apache.spark.util.ShutdownHookManager=OFF
log4j.logger.org.apache.spark.SparkEnv=ERROR

Source: Stack Overflow

amgadmadkour on Nov 11, 2020

Just to help others - in your spark directory there is a conf directory - add those two lines to the “log4j.properties” file. If there is no “log4j.properties” there should be a “log4j.properties.template” - copy the .template and remove the “.template” then add those lines at the top and the error will be hidden

GoEddie on Apr 25, 2019

It appears that you are using Spark 2.4.2, which is not supported yet. Can you please try either 2.4.0, 2.4.1 or 2.3.*? More intfo #43.

imback82 on Apr 25, 2019

I got the same issue and found out it is because I am using Spark 2.4.2. After I change to Spark 2.4.1, it is working. Please refer to my following post about the details with some examples, i.e. read/write parquet files; read from HDFS/Hive using C#. .NET for Apache Spark Preview with Examples

FahaoTang on Apr 26, 2019

@jalchr I am closing this as a duplicate of #48. Thanks for reporting!

imback82 on Apr 26, 2019

Great! Thanks!

marcoparenzan on Apr 25, 2019