spark-on-k8s-operator: javaOptions and spark.driver.extraJavaOptions are ignored

operator 3.1.1

javaOptions: "-DXXX=test"

and

"spark.driver.extraJavaOptions" : "-DXXX=test"

are ignored!

I don’t see them in the log under /opt/spark/bin/spark-submit

and in my java code:

System.out.println("XXX=" + System.getProperty("XXX"));

I see: XXX=null

can someone check?

About this issue

Original URL
State: open
Created 2 years ago
Comments: 16 (3 by maintainers)

Most upvoted comments

@yevsh Sorry for the late reply. Looks like in the executor pods the extra java options are passed through the environment vairables SPARK_JAVA_OPT_* (SPARK_JAVA_OPT_0, SPARK_JAVA_OPT_1, etc). The difference being that the executor process is initiated via java command while the driver is submitted using spark-submit. Login to your executor pod and check these environment variables.

dragan-vujovic on Nov 11, 2022