flintrock: Spark 3.0.0 incompatibilities

Hi, I’m using Flintrock to deploy a Scala/Spark application to a simple EC2 cluster (master + 1 slave), but I’m having some problems. In particular, I’m using the following custom Flintrock configuration to launch the cluster:

provider: ec2

services:
  spark:
    version: 3.0.0

launch:
  num-slaves: 1
  install-spark: True

providers:
  ec2:
    key-name: my-key-pair
    identity-file: my-key-pair.pem
    instance-type: m4.large
    region: us-east-1
    ami: ami-00b882ac5193044e4
    user: ec2-user
    instance-profile-name: ec2-s3

debug: true

As you can see, I’m using Spark 3.0.0. Once I have packaged my Scala application (using sbt) I perform a Flintrock’s copy-file operation. Then, when I want to run the JAR using spark-submit, I have issues:

• If I run spark-submit from my local machine using option --master spark://<ec2-master-name>:7077 and --deploy-mode cluster, I get a Java error saying Cannot run program "/usr/local/opt/openjdk@11/bin/java" (in directory "/home/ec2-user/spark/work/driver-20200718133018-0005")
• If I run spark-submit from inside the master cluster (using either Flintrock’s run-command --master-only or login commands), the job gets correctly submitted, but the spark UI is not reporting it (at <ec2-master-name>:8080). Instead, the application execution can be viewed at <ec2-master-name>:4040, even though in the Executors page only the master seems to be active.

I can’t wrap my head around this problem. I don’t know if it is some kind of Flintrock’s compatibility issue with Spark 3.0.0, or I’m doing something wrong. I apologize if this is the wrong place where to post this kind of doubts.

However, my environment is the following:

• Flintrock version: 1.0.0
• Python version: 3.8.4
• OS: macOS Catalina 10.15.5
• SBT version: 1.3.13
• Scala version: 2.12.10
• OpenJDK version: 1.8