spark-on-k8s-operator: spec.driver.coreRequest 0.5 not working

Hi, this is the issue following the previous question. it looks setting spec.driver.coreRequest to 0.5 is not working. I am using Spark 3.0. I inspected related yaml.

Environment:
sparkoperator-0.8.2     
v1beta2-1.2.0-3.0.0

when I issue the following command, I get kubectl describe sparkapplications

...
Spec:
  Driver:
    Core Limit:     1200m
    Cores:          1
    Cores Request:  0.5
    Labels:
      Version:  3.0.1
    Memory:     512m
    ...
  Executor:
    Cores:          1
    Cores Request:  0.5
...

when I inspect driver, I get:

...
Restart Count:  0
    Limits:
      cpu:     1200m
      memory:  896Mi
    Requests:
      cpu:     1
      memory:  896Mi
...

when I inspect executor, I get:

...
ports:
    - containerPort: 7079
      name: blockmanager
      protocol: TCP
    resources:
      limits:
        memory: 1408Mi
      requests:
        cpu: "1"
        memory: 1408Mi
...

So it looks coreRequest was applied in spark application, but it did not get applied to driver app and executor app. I set the instances of executor to be 3, but one of them is in Pending status. Could you please help? Thanks.

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 15 (5 by maintainers)

Most upvoted comments

@kz33 super! Just tried. worked like a charm! Thanks a lot @kz33 .

By the way, spark-operator should reject coresRequest and not display it in kubectl describe sparkapplications? which is misleading. @liyinan926

weicheng113 on Nov 23, 2020

@weicheng113 I am curious, why is it Cores Request instead of Core Request? Can you share your yaml? Mine:
spec:
  driver:
    coreRequest: "0.5"
    coreLimit: "1200m"
  executor:
    coreRequest: "0.5"
this is coreRequest，not coresRequest

@kz33 Good catch, that might be the problem. Let me try that out. Thanks for the spark-operator log output.

cwei-bgl on Nov 23, 2020