pipelines: Cannot use resource request based on PipelineParam

We are trying to build a generic pipeline that will have some steps with configurable resources. This is needed because based on what is being run it might need more or less cpu, memory and etc.

Consider the pipeline described bellow:

@dsl.pipeline(
    name='Generic Pipe'
)
def pipeline_definition(memory_request: str = '8G', cpu_request: str = '4'):
    op = ContainerOp(...)

    op.set_memory_request(memory_request)\
      .set_cpu_request(cpu_request)

This pipeline will fail to match the expected resource Regex and would not be successfully uploaded. This is what would be generated on the yaml file:

 resources:
    requests:
        cpu: '{{inputs.parameters.cpu-request}}'
        memory: '{{inputs.parameters.memory-request}}'

And here is the error:

kfp_server_api.rest.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.15.6', 'Date': 'Mon, 26 Aug 2019 21:46:03 GMT', 'Content-Type': 'text/plain; charset=utf-8', 'Content-Length': '601', 'Connection': 'keep-alive', 'x-powered-by': 'Express', 'x-envoy-upstream-service-time': '55'})
HTTP response body: {"error_message":"Error creating pipeline: Create pipeline failed: Failed to get parameters from the workflow: InvalidInputError: Failed to parse the parameter.: error unmarshaling JSON: while decoding JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'","error_details":"Error creating pipeline: Create pipeline failed: Failed to get parameters from the workflow: InvalidInputError: Failed to parse the parameter.: error unmarshaling JSON: while decoding JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'"}

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Reactions: 25
  • Comments: 21 (14 by maintainers)

Commits related to this issue

Most upvoted comments

/reopen

This would still be a useful feature to have.

Argo was upgraded to 2.7.5 in https://github.com/kubeflow/pipelines/pull/3537 so now Argo’s pod spec patching (https://github.com/argoproj/argo/pull/1687) is available to implement a fix in the Pipelines SDK.

@chensun

I don’t think this will be super complicated to fix. I guess it’s currently blocked by this allowlist here. We can take a look when we have some cycle.

I’m getting blocked by this too. We’re also trying to configure GPU dynamically, but that’s running into problems because add_node_selector_constraint can’t take PipelineParams: I was hoping I could use a previous parse step to build the node selector and then apply it using parse_task.outputs.

Would that be covered by this ticket, or need a new one, or is there a workaround I’m missing?

Sorry, I misread the issue’s description. Please ignore my comment above

Hello,

This issue is related to argoproj/argo#703 and it is not fixed yet.