argo-workflows: argo run failed: Permission denied
Summary
What happened/what you expected to happen?
run argo job failed: bash: /argo/staging/script: Permission denied
Diagnostics
What Kubernetes provider are you using?
docker desktop for mac.
(base) ➜ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:59:11Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
What version of Argo Workflows are you running?
(base) ➜ argo version
argo: v3.1.5
BuildDate: 2021-08-04T07:03:32Z
GitCommit: 3dbee0ec368f3ea8c31f49c8b1a4617cc32bcce9
GitTreeState: clean
GitTag: v3.1.5
GoVersion: go1.15.7
Compiler: gc
Platform: darwin/amd64
Are you pasting thousands of log lines? That’s too much information.
$ argo -n argo submit workflow.yaml
$ kubectl -n argo logs seldon-batch-process-2052519094 -c main
bash: /argo/staging/script: Permission denied
Message from the maintainers:
here is my workflow yaml file:
# Source: seldon-batch-workflow/templates/workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: seldon-batch-process
namespace: argo
spec:
entrypoint: seldon-batch-process
volumeClaimTemplates:
- metadata:
name: "seldon-pvc"
ownerReferences:
- apiVersion: argoproj.io/v1alpha1
blockOwnerDeletion: true
kind: Workflow
name: "{{workflow.name}}"
uid: "{{workflow.uid}}"
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: "2Mi"
volumes:
- name: rclone-config
secret:
secretName: "rclone-config-secret"
templates:
- name: seldon-batch-process
steps:
- - name: create-seldon-resource
template: create-seldon-resource-template
- - name: wait-seldon-resource
template: wait-seldon-resource-template
- - name: download-object-store
template: download-object-store-template
- - name: process-batch-inputs
template: process-batch-inputs-template
- - name: upload-object-store
template: upload-object-store-template
- - name: delete-seldon-resource
template: delete-seldon-resource-template
- name: create-seldon-resource-template
metadata:
annotations:
sidecar.istio.io/inject: "false"
resource:
action: create
manifest: |
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
name: "sklearn"
namespace: argo
ownerReferences:
- apiVersion: argoproj.io/v1alpha1
blockOwnerDeletion: true
kind: Workflow
name: "{{workflow.name}}"
uid: "{{workflow.uid}}"
spec:
name: "sklearn"
predictors:
- componentSpecs:
- spec:
containers:
- name: classifier
env:
- name: GUNICORN_THREADS
value: "10"
- name: GUNICORN_WORKERS
value: "1"
resources:
requests:
cpu: 5m
memory: 10Mi
limits:
cpu: 50m
memory: 100Mi
graph:
children: []
implementation: SKLEARN_SERVER
modelUri: gs://seldon-models/sklearn/iris
name: classifier
name: default
replicas: 1
- name: wait-seldon-resource-template
metadata:
annotations:
sidecar.istio.io/inject: "false"
script:
image: bitnami/kubectl:1.17
command: [bash]
source: |
sleep 5
kubectl rollout status \
deploy/$(kubectl get deploy -l seldon-deployment-id="sklearn" -o jsonpath='{.items[0].metadata.name}')
- name: download-object-store-template
metadata:
annotations:
sidecar.istio.io/inject: "false"
script:
image: rclone/rclone:1.53
volumeMounts:
- name: "seldon-pvc"
mountPath: /assets
- name: rclone-config
mountPath: /config/rclone
readOnly: true
command: [sh]
source: |
rclone copyto cluster-minio:data/input-data.txt /assets/input-data.txt
- name: process-batch-inputs-template
metadata:
annotations:
sidecar.istio.io/inject: "false"
script:
image: seldonio/seldon-core-s2i-python37:1.5.0
volumeMounts:
- name: "seldon-pvc"
mountPath: /assets
command: [bash]
source: |
seldon-batch-processor \
--deployment-name "sklearn" \
--benchmark \
--namespace "argo" \
--host "istio-ingressgateway.istio-system.svc.cluster.local" \
--workers "2" \
--data-type "data" \
--payload-type "ndarray" \
--retries "3" \
--input-data-path "/assets/input-data.txt" \
--output-data-path "/assets/output-data.txt"
- name: upload-object-store-template
metadata:
annotations:
sidecar.istio.io/inject: "false"
script:
image: rclone/rclone:1.53
volumeMounts:
- name: "seldon-pvc"
mountPath: /assets
- name: rclone-config
mountPath: /config/rclone
readOnly: true
command: [sh]
source: |
rclone copyto /assets/output-data.txt cluster-minio:data/output-data-{{workflow.uid}}.txt
- name: delete-seldon-resource-template
metadata:
annotations:
sidecar.istio.io/inject: "false"
script:
image: bitnami/kubectl:1.17
command: [bash]
source: |
sleep 5
kubectl delete seldondeployments -n argo sklearn
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 3
- Comments: 20 (11 by maintainers)
Maybe write a workflow that runs as non root and uses script? That could be an e2e test.
@lianxmfor You need to run wait-seldon-resource-template and delete-seldon-resource-template as a root user, you can add below two lines to wait-seldon-resource-template and delete-seldon-resource-template
Complete workflow.yaml
I know enough to revert the change but not enough to easily add a test. Let me see what I can come up with.
I’m facing same issue too. version 2.12.9
Oh, never mind. I fixed my issue after change
[sh, -c]
->[sh]