argo-workflows: argo run failed: Permission denied

Summary

What happened/what you expected to happen?

run argo job failed: bash: /argo/staging/script: Permission denied

Diagnostics

What Kubernetes provider are you using?

docker desktop for mac.

(base) ➜  kubectl version 
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

What version of Argo Workflows are you running?

(base) ➜  argo version
argo: v3.1.5
  BuildDate: 2021-08-04T07:03:32Z
  GitCommit: 3dbee0ec368f3ea8c31f49c8b1a4617cc32bcce9
  GitTreeState: clean
  GitTag: v3.1.5
  GoVersion: go1.15.7
  Compiler: gc
  Platform: darwin/amd64

Are you pasting thousands of log lines? That’s too much information.

$ argo -n argo submit workflow.yaml 
$ kubectl -n argo logs seldon-batch-process-2052519094  -c main
bash: /argo/staging/script: Permission denied

Message from the maintainers:

here is my workflow yaml file:

# Source: seldon-batch-workflow/templates/workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: seldon-batch-process
  namespace: argo
spec:
  entrypoint: seldon-batch-process
  volumeClaimTemplates:
  - metadata:
      name: "seldon-pvc"
      ownerReferences:
      - apiVersion: argoproj.io/v1alpha1
        blockOwnerDeletion: true
        kind: Workflow
        name: "{{workflow.name}}"
        uid: "{{workflow.uid}}"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: "2Mi"
  volumes:
  - name: rclone-config
    secret:
      secretName: "rclone-config-secret"
  templates:
  - name: seldon-batch-process
    steps:
    - - name: create-seldon-resource
        template: create-seldon-resource-template
    - - name: wait-seldon-resource
        template: wait-seldon-resource-template
    - - name: download-object-store
        template: download-object-store-template
    - - name: process-batch-inputs
        template: process-batch-inputs-template
    - - name: upload-object-store
        template: upload-object-store-template
    - - name: delete-seldon-resource
        template: delete-seldon-resource-template

  - name: create-seldon-resource-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    resource:
      action: create
      manifest: |
        apiVersion: machinelearning.seldon.io/v1
        kind: SeldonDeployment
        metadata:
          name: "sklearn"
          namespace: argo
          ownerReferences:
          - apiVersion: argoproj.io/v1alpha1
            blockOwnerDeletion: true
            kind: Workflow
            name: "{{workflow.name}}"
            uid: "{{workflow.uid}}"
        spec:
          name: "sklearn"
          predictors:
            - componentSpecs:
              - spec:
                  containers:
                  - name: classifier
                    env:
                    - name: GUNICORN_THREADS
                      value: "10"
                    - name: GUNICORN_WORKERS
                      value: "1"
                    resources:
                      requests:
                        cpu: 5m
                        memory: 10Mi
                      limits:
                        cpu: 50m
                        memory: 100Mi
              graph:
                children: []
                implementation: SKLEARN_SERVER
                modelUri: gs://seldon-models/sklearn/iris
                name: classifier
              name: default
              replicas: 1

  - name: wait-seldon-resource-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    script:
      image: bitnami/kubectl:1.17
      command: [bash]
      source: |
        sleep 5
        kubectl rollout status \
            deploy/$(kubectl get deploy -l seldon-deployment-id="sklearn" -o jsonpath='{.items[0].metadata.name}')

  - name: download-object-store-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    script:
      image: rclone/rclone:1.53
      volumeMounts:
      - name: "seldon-pvc"
        mountPath: /assets
      - name: rclone-config
        mountPath: /config/rclone
        readOnly: true
      command: [sh]
      source: |
        rclone copyto cluster-minio:data/input-data.txt /assets/input-data.txt

  - name: process-batch-inputs-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    script:
      image: seldonio/seldon-core-s2i-python37:1.5.0
      volumeMounts:
      - name: "seldon-pvc"
        mountPath: /assets
      command: [bash]
      source: |
        seldon-batch-processor \
            --deployment-name "sklearn" \
            --benchmark \
            --namespace "argo" \
            --host "istio-ingressgateway.istio-system.svc.cluster.local" \
            --workers "2" \
            --data-type "data" \
            --payload-type "ndarray" \
            --retries "3" \
            --input-data-path "/assets/input-data.txt" \
            --output-data-path "/assets/output-data.txt"

  - name: upload-object-store-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    script:
      image: rclone/rclone:1.53
      volumeMounts:
      - name: "seldon-pvc"
        mountPath: /assets
      - name: rclone-config
        mountPath: /config/rclone
        readOnly: true
      command: [sh]
      source: |
        rclone copyto /assets/output-data.txt cluster-minio:data/output-data-{{workflow.uid}}.txt


  - name: delete-seldon-resource-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    script:
      image: bitnami/kubectl:1.17
      command: [bash]
      source: |
        sleep 5
        kubectl delete seldondeployments -n argo  sklearn

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 3
  • Comments: 20 (11 by maintainers)

Most upvoted comments

Maybe write a workflow that runs as non root and uses script? That could be an e2e test.

@lianxmfor You need to run wait-seldon-resource-template and delete-seldon-resource-template as a root user, you can add below two lines to wait-seldon-resource-template and delete-seldon-resource-template

securityContext:
    runAsUser: 0

Complete workflow.yaml


---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  {{- if eq .Values.workflow.useNameAsGenerateName false }}
  name: {{ .Values.workflow.name }}
  {{- else }}
  generateName: {{ .Values.workflow.name }}
  {{- end }}
  namespace: {{ .Values.workflow.namespace }}
spec:
  entrypoint: seldon-batch-process
  volumeClaimTemplates:
  - metadata:
      name: "{{ .Values.pvc.name }}"
      ownerReferences:
      - apiVersion: argoproj.io/v1alpha1
        blockOwnerDeletion: true
        kind: Workflow
        name: "{{`{{workflow.name}}`}}"
        uid: "{{`{{workflow.uid}}`}}"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: "{{ .Values.pvc.storage }}"
  volumes:
  - name: rclone-config
    secret:
      secretName: "{{ .Values.rclone.configSecretName }}"
  templates:
  - name: seldon-batch-process
    steps:
    - - name: create-seldon-resource
        template: create-seldon-resource-template
    - - name: wait-seldon-resource
        template: wait-seldon-resource-template
    - - name: download-object-store
        template: download-object-store-template
    - - name: process-batch-inputs
        template: process-batch-inputs-template
    - - name: upload-object-store
        template: upload-object-store-template
    - - name: delete-seldon-resource
        template: delete-seldon-resource-template

  - name: create-seldon-resource-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    resource:
      action: create
      manifest: |
        apiVersion: machinelearning.seldon.io/v1
        kind: SeldonDeployment
        metadata:
          name: "{{ .Values.seldonDeployment.name }}"
          namespace: {{ .Values.workflow.namespace }}
          ownerReferences:
          - apiVersion: argoproj.io/v1alpha1
            blockOwnerDeletion: true
            kind: Workflow
            name: "{{`{{workflow.name}}`}}"
            uid: "{{`{{workflow.uid}}`}}"
        spec:
          name: "{{ .Values.seldonDeployment.name }}"
          predictors:
            - componentSpecs:
              - spec:
                  containers:
                  - name: classifier
                    env:
                    - name: GUNICORN_THREADS
                      value: "{{ .Values.seldonDeployment.serverThreads }}"
                    - name: GUNICORN_WORKERS
                      value: "{{ .Values.seldonDeployment.serverWorkers }}"
                    resources:
                      requests:
                        cpu: {{ .Values.seldonDeployment.requests.cpu }}
                        memory: {{ .Values.seldonDeployment.requests.memory }}
                      limits:
                        cpu: {{ .Values.seldonDeployment.limits.cpu }}
                        memory: {{ .Values.seldonDeployment.limits.memory }}
              graph:
                children: []
                implementation: {{ .Values.seldonDeployment.server }}
                modelUri: {{ .Values.seldonDeployment.modelUri }}
                name: classifier
              name: default
              replicas: {{ .Values.seldonDeployment.replicas }}
  - name: wait-seldon-resource-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    securityContext: 
        runAsUser: 0
    script:
      image: bitnami/kubectl:1.17
      command: [bash]
      source: |
        sleep {{ .Values.seldonDeployment.waitTime }}
        kubectl rollout status \
            deploy/$(kubectl get deploy -l seldon-deployment-id="{{ .Values.seldonDeployment.name }}" -o jsonpath='{.items[0].metadata.name}')
  - name: download-object-store-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    script:
      image: {{ .Values.rclone.image }}
      volumeMounts:
      - name: "{{ .Values.pvc.name }}"
        mountPath: /assets
      - name: rclone-config
        mountPath: /config/rclone
        readOnly: true
      command: [sh]
      source: |
        rclone copyto {{ .Values.rclone.inputDataPath }} /assets/input-data.txt
  - name: process-batch-inputs-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    script:
      image: {{ .Values.seldonDeployment.image }}
      volumeMounts:
      - name: "{{ .Values.pvc.name }}"
        mountPath: /assets
      command: [bash]
      source: |
        seldon-batch-processor \
            --deployment-name "{{ .Values.seldonDeployment.name }}" \
            {{- if eq .Values.batchWorker.enableBenchmark true }}
            --benchmark \
            {{- end }}
            --namespace "{{ .Values.workflow.namespace }}" \
            --host "{{ .Values.batchWorker.host }}" \
            --workers "{{ .Values.batchWorker.workers }}" \
            --data-type "{{ .Values.batchWorker.dataType }}" \
            --payload-type "{{ .Values.batchWorker.payloadType }}" \
            --retries "{{ .Values.batchWorker.retries }}" \
            --input-data-path "/assets/input-data.txt" \
            --output-data-path "/assets/output-data.txt"
  - name: upload-object-store-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    script:
      image: {{ .Values.rclone.image }}
      volumeMounts:
      - name: "{{ .Values.pvc.name }}"
        mountPath: /assets
      - name: rclone-config
        mountPath: /config/rclone
        readOnly: true
      command: [sh]
      source: |
        rclone copyto /assets/output-data.txt {{ .Values.rclone.outputDataPath }}
  - name: delete-seldon-resource-template
    metadata:
      annotations:
        sidecar.istio.io/inject: "false"
    securityContext: 
        runAsUser: 0
    script:
      image: bitnami/kubectl:1.17
      command: [bash]
      source: |
        sleep {{ .Values.seldonDeployment.waitTime }}
        kubectl delete seldondeployments -n {{ .Values.workflow.namespace }}  {{ .Values.seldonDeployment.name }}

I know enough to revert the change but not enough to easily add a test. Let me see what I can come up with.

I’m facing same issue too. version 2.12.9

Oh, never mind. I fixed my issue after change [sh, -c] -> [sh]