argo-workflows: Workflow is running even if a ValidatingWebhookConfiguration rejects the pods

Summary

I have a validating webhooks which rejects all pods in specific namespaces. I submit a workflow and the pod is not created because it is rejected by the webhook (as it is expected). However, the workflow is still Running. I would expect the workflow to be in the Error state. I have created a repo with some instructions, if you want to reproduce it in a kind cluster: https://github.com/karabill/argo_reject_operator_problem. Note that this is happening only in argo v2.12.* (I tried v2.12.9 and a couple of other v2.12.*). I was not able to reproduce it in v2.11.8.

Diagnostics

What Kubernetes provider are you using? AWS or kind

What version of Argo Workflows are you running?

• v2.12.9

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"argoproj.io/v1alpha1","kind":"Workflow","metadata":{"annotations":{},"name":"myworkflow85","namespace":"test-workflow"},"spec":{"entrypoint":"start","templates":[{"container":{"args":["echo 'it should not run'"],"command":["/bin/sh","-c"],"image":"alpine:3.7"},"name":"start"}]}}
  creationTimestamp: "2021-03-01T16:01:50Z"
  generation: 2
  labels:
    workflows.argoproj.io/phase: Running
  name: myworkflow85
  namespace: test-workflow
  resourceVersion: "1568"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/test-workflow/workflows/myworkflow85
  uid: 63c1aa45-501a-4a6a-9f69-ddd865084596
spec:
  arguments: {}
  entrypoint: start
  templates:
  - arguments: {}
    container:
      args:
      - echo 'it should not run'
      command:
      - /bin/sh
      - -c
      image: alpine:3.7
      name: ""
      resources: {}
    inputs: {}
    metadata: {}
    name: start
    outputs: {}
status:
  finishedAt: null
  nodes:
    myworkflow85:
      displayName: myworkflow85
      finishedAt: "2021-03-01T16:01:50Z"
      id: myworkflow85
      message: 'admission webhook "pod.validation.webhook" denied the request: Pod
        is not allowed'
      name: myworkflow85
      phase: Error
      progress: 1/1
      startedAt: "2021-03-01T16:01:50Z"
      templateName: start
      templateScope: local/myworkflow85
      type: Pod
  phase: Running
  progress: 1/1
  startedAt: "2021-03-01T16:01:50Z"

time="2021-03-01T16:01:50.893Z" level=info msg="Processing workflow" namespace=test-workflow workflow=myworkflow85
time="2021-03-01T16:01:50.893Z" level=info msg="Updated phase  -> Running" namespace=test-workflow workflow=myworkflow85
time="2021-03-01T16:01:50.893Z" level=info msg="Pod node myworkflow85 initialized Pending" namespace=test-workflow workflow=myworkflow85
time="2021-03-01T16:01:50.899Z" level=info msg="Failed to create pod myworkflow85 (myworkflow85): admission webhook \"pod.validation.webhook\" denied the request: Pod is not allowed" namespace=test-workflow workflow=myworkflow85
time="2021-03-01T16:01:50.899Z" level=error msg="Mark error node" error="admission webhook \"pod.validation.webhook\" denied the request: Pod is not allowed" namespace=test-workflow nodeName=myworkflow85 workflow=myworkflow85
time="2021-03-01T16:01:50.899Z" level=info msg="node myworkflow85 phase Pending -> Error" namespace=test-workflow workflow=myworkflow85
time="2021-03-01T16:01:50.899Z" level=info msg="node myworkflow85 message: admission webhook \"pod.validation.webhook\" denied the request: Pod is not allowed" namespace=test-workflow workflow=myworkflow85
time="2021-03-01T16:01:50.900Z" level=info msg="node myworkflow85 finished: 2021-03-01 16:01:50.900014662 +0000 UTC" namespace=test-workflow workflow=myworkflow85
time="2021-03-01T16:01:50.900Z" level=error msg="error in entry template execution" error="admission webhook \"pod.validation.webhook\" denied the request: Pod is not allowed" namespace=test-workflow workflow=myworkflow85
time="2021-03-01T16:01:50.908Z" level=info msg="Workflow update successful" namespace=test-workflow phase=Running resourceVersion=1568 workflow=myworkflow85

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.