spark-on-k8s-operator: Config Maps and Volumes are not getting mounted

Our GKE cluster is running on Kubernetes version v1.21.14. Pods were running all good until yesterday, now Configmaps and Volumes are not getting Mounted.

Deployment Mode: Helm Chart

Helm Chart Version: 1.1.0

Image: v1beta2-1.2.3-3.1.1

Kubernetes Version: 1.21.14

Helm command to install: helm install spark-operator --namespace *** --set image.tag= v1beta2-1.2.3-3.1.1 --set webhook.enable=true -f values.yaml

Spark operator pod starts successfully after webhook-init pod get completed.

But my application pod using Spark Operator is unable to come up due to below error:

Events: │ │ Type Reason Age From Message │ │ ---- ------ ---- ---- ------- │ │ Normal Scheduled 49m default-scheduler Successfully assigned pod/re-driver to gke │ │ w--taints-6656b326-49of │ │ Warning FailedMount 49m kubelet MountVolume.SetUp failed for volume "spark-conf-volume-driver" : configmap "spark-drv-8c0f12839ca69805-conf-map" not found │ │ Warning FailedMount 27m (x3 over 40m) kubelet Unable to attach or mount volumes: unmounted volumes=[re-checkpoint], unattached volumes=[spark-conf-volume-driver kube-api-acce │ │ s-lsflz re-checkpoint app-conf-vol cert-secret-volume spark-local-dir-1[]: timed out waiting for the condition │ │ Warning FailedMount 20m (x2 over 45m) kubelet Unable to attach or mount volumes: unmounted volumes=[re-checkpoint], unattached volumes=[kube-api-access-lsflz re-checkpoint app-conf-vol cert-secret-volume spark-local-dir-1 spark-conf-volume-driver[]: timed out waiting for the condition

About this issue

Original URL
State: open
Created 2 years ago
Reactions: 1
Comments: 17 (3 by maintainers)

Most upvoted comments

My issue was the webhook port… It for some reason doesn’t run on the default port anymore. So I had to update the port to 443 based on the docs here, even though I’m on EKS instead of GKE

JunaidChaudry on Jun 20, 2023

Hi All,

Experienced this when we migrated to the helm chart installation of the spark operator, our volumes were mounted correctly via configmaps, but kubernetes was erroring out…

Make sure you have the following settings enabled on the helm chart below…

webhook:
  # -- Enable webhook server
  enable: true
  namespaceSelector: "spark-webhook-enabled=true"

Then label the spark namespace (or target namespace for your spark jobs) with:

spark-webhook-enabled=true

We found that was enough to get it working.

Application side

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: ScheduledSparkApplication
metadata:
 name: example-test
 namespace: spark
spec:
 schedule: "31 12 * * *"
 concurrencyPolicy: Allow
 template:
   timeToLiveSeconds: 1200
   type: Python
   arguments:
     - --config-file-name=/opt/spark/work-dir/config/config.ini
   sparkConf:
     spark.kubernetes.decommission.script: "/opt/decom.sh"
     .  . .
   hadoopConf:
     fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
     . . .
   mode: cluster
   imagePullPolicy: Always
   mainApplicationFile: local:///opt/spark/work-dir/run.py
   sparkVersion: "3.2.1"
   restartPolicy:
       type: Never
   driver:
     cores: 1
     coreLimit: "500m"
     memory: "500m"
     labels:
       version: 3.2.1
     serviceAccount: job-tole
     volumeMounts:
       - name: "config"
         mountPath: "/opt/spark/work-dir/config"
   executor:
     cores: 1
     instances: 1
     memory: "500m"
     labels:
       version: 3.2.1
     volumeMounts:
       - name: "config"
         mountPath: "/opt/spark/work-dir/config"
   volumes:
     - name: "config"
       configMap:
         name: "config"
         items:
           - key: "config.ini"
             path: "config.ini"

Please note that: If you don’t use the helm chart you still need to enable the webhook, otherwise the spark-operator won’t be able to create the right configmaps and volume mounts on the driver and executor pods when they are spawned.

GaryLouisStewart on Mar 15, 2023