spark-on-k8s-operator: Job never comes up and fails with driver pod not found

Sometimes the creation request succeeds and then the job never comes up and doing a describe on the spark application shows

SparkApplicationFailed               6m54s  spark-operator  SparkApplication log-validation failed: Driver Pod not found

We’ve seen this happen about 25% of the time. Any advice ?

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 38 (19 by maintainers)

Commits related to this issue

Fixed the spec update issue in #795 — committed to liyinan926/spark-on-k8s-operator by liyinan926 4 years ago
Fixed the spec update issue in #795 (#804) — committed to GoogleCloudPlatform/spark-on-k8s-operator by liyinan926 4 years ago
Revert "Fixed the spec update issue in #795 (#804)" This reverts commit 1687c0647c11a5c64d6eeb808522fae75051ad6a. — committed to GoogleCloudPlatform/spark-on-k8s-operator by liyinan926 4 years ago
Revert "Fixed the spec update issue in #795 (#804)" (#805) This reverts commit 1687c0647c11a5c64d6eeb808522fae75051ad6a. — committed to GoogleCloudPlatform/spark-on-k8s-operator by liyinan926 4 years ago
Fixed the spec update issue in #795 (#804) — committed to nokia/spark-on-k8s-operator by liyinan926 4 years ago
Revert "Fixed the spec update issue in #795 (#804)" (#805) This reverts commit 1687c0647c11a5c64d6eeb808522fae75051ad6a. — committed to nokia/spark-on-k8s-operator by liyinan926 4 years ago
Fixed the spec update issue in #795 (#804) — committed to nokia/spark-on-k8s-operator by liyinan926 4 years ago
Revert "Fixed the spec update issue in #795 (#804)" (#805) This reverts commit 1687c0647c11a5c64d6eeb808522fae75051ad6a. — committed to nokia/spark-on-k8s-operator by liyinan926 4 years ago

Most upvoted comments

Still seeing the same issue.

  Type     Reason                               Age   From            Message
  ----     ------                               ----  ----            -------
  Normal   SparkApplicationAdded                58s   spark-operator  SparkApplication logs-batch was added, enqueuing it for submission
  Normal   SparkApplicationSubmitted            53s   spark-operator  SparkApplication logs-batch was submitted successfully
  Normal   SparkApplicationSpecUpdateProcessed  53s   spark-operator  Successfully processed spec update for SparkApplication logs-batch
  Warning  SparkApplicationFailed               52s   spark-operator  SparkApplication logs-batch failed: Driver Pod not found

sb2nov on Feb 11, 2020