argo-workflows: Cannot specify key for input artifact (without full artifact location)

Tested on 2.9.0-rc3

The official example works:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifactory-repository-ref-
spec:
  entrypoint: main
  artifactRepositoryRef:
    key: minio
  templates:
    - name: main
      container:
        image: docker/whalesay:latest
        command: [sh, -c]
        args: ["cowsay hello world | tee /tmp/hello_world.txt"]
      outputs:
        artifacts:
          - name: hello_world
            path: /tmp/hello_world.txt

When switching to input it fails:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifactory-repository-ref-
spec:
  entrypoint: main
  artifactRepositoryRef:
    key: minio
  templates:
    - name: main
      container:
        image: docker/whalesay:latest
        command: [sh, -c]
        args: ["cowsay hello world | tee /tmp/hello_world.txt"]
      inputs:
        artifacts:
          - name: hello_world
            path: /tmp/hello_world.txt

with

Failed to submit workflow: templates.entrypoint.steps[0].main templates.main-template inputs.artifacts.hello_world was not supplied

Another important point IMO. I don’t see any ways to specify the key during workflow creation (the location of your object within S3/bucket). My understanding is that artifactRepositoryRef can be used to setup default repositories and then can be reused within workflows specifying the location of the folder or file we want to use as inputs or outputs. Was that designed for that purpose?

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 12
  • Comments: 16 (12 by maintainers)

Most upvoted comments

Do 👍 to show interest.

@hadim I’m going to recategorize this as an “enhancement”. We should do more work in this area, and I’d like to asses interest.

Kubeflow Pipelines (KFP) try to separate cluster config (like artifact repository) from workflow config. If this feature is supported from argo, we could let each user manage its artifact repository in their own namespace while keeping the workflow definitions shareable.

This is one of the areas KFP hasn’t been able to achieve multi-tenancy separation: https://github.com/kubeflow/pipelines/issues/1223#issuecomment-656507073.

Some recent discussion: https://kubeflow.slack.com/archives/CE10KS9M4/p1602516358147900

This issue isn’t really to do with artifactRepositoryRef, it’s not supported at all. I’m going to rename this issue to reflect this.

I think you must specify the key within the bucket, as well as bucket, endpoint etc. Not great I agree, but that is how it is today.