pipelines: Cannot use resource request based on PipelineParam
We are trying to build a generic pipeline that will have some steps with configurable resources. This is needed because based on what is being run it might need more or less cpu, memory and etc.
Consider the pipeline described bellow:
@dsl.pipeline(
name='Generic Pipe'
)
def pipeline_definition(memory_request: str = '8G', cpu_request: str = '4'):
op = ContainerOp(...)
op.set_memory_request(memory_request)\
.set_cpu_request(cpu_request)
This pipeline will fail to match the expected resource Regex and would not be successfully uploaded. This is what would be generated on the yaml file:
resources:
requests:
cpu: '{{inputs.parameters.cpu-request}}'
memory: '{{inputs.parameters.memory-request}}'
And here is the error:
kfp_server_api.rest.ApiException: (500)
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Server': 'nginx/1.15.6', 'Date': 'Mon, 26 Aug 2019 21:46:03 GMT', 'Content-Type': 'text/plain; charset=utf-8', 'Content-Length': '601', 'Connection': 'keep-alive', 'x-powered-by': 'Express', 'x-envoy-upstream-service-time': '55'})
HTTP response body: {"error_message":"Error creating pipeline: Create pipeline failed: Failed to get parameters from the workflow: InvalidInputError: Failed to parse the parameter.: error unmarshaling JSON: while decoding JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'","error_details":"Error creating pipeline: Create pipeline failed: Failed to get parameters from the workflow: InvalidInputError: Failed to parse the parameter.: error unmarshaling JSON: while decoding JSON: quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'"}
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 25
- Comments: 21 (14 by maintainers)
Commits related to this issue
- feat(sdk): add runtime resource requests. Fixes #1956 (#5447) * added resource request at runtime * fixed things * Update to use read only parameter insteadt * added test case and better example ... — committed to chauhang/pipelines by NikeNano 3 years ago
- Split framework struct (#1956) * rename framework struct SupportedModelTypes -> SupportedModelFormats ServingRuntimeFramework -> SupportedModelFormat Framework -> ModelFormat Signed-off-by: Suresh N... — committed to magdalenakuhn17/pipelines by Suresh-Nakkeran 3 years ago
/reopen
This would still be a useful feature to have.
Argo was upgraded to 2.7.5 in https://github.com/kubeflow/pipelines/pull/3537 so now Argo’s pod spec patching (https://github.com/argoproj/argo/pull/1687) is available to implement a fix in the Pipelines SDK.
@chensun
I don’t think this will be super complicated to fix. I guess it’s currently blocked by this allowlist here. We can take a look when we have some cycle.
I’m getting blocked by this too. We’re also trying to configure GPU dynamically, but that’s running into problems because
add_node_selector_constraintcan’t take PipelineParams: I was hoping I could use a previous parse step to build the node selector and then apply it usingparse_task.outputs.Would that be covered by this ticket, or need a new one, or is there a workaround I’m missing?
Sorry, I misread the issue’s description. Please ignore my comment above
Hello,
This issue is related to argoproj/argo#703 and it is not fixed yet.