argo-workflows: Windows-based pod runs indefinitely

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I’d like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

When I submit a workflow with a single task that runs in a windows/servercore:ltsc2022 container, the pod is stuck in Running phase indefinitely, although the command in the container succeeds.

Before we updated from v3.4.8, we used ltsc2019 images without problems. The Windows nodes were upgraded as well, they have the label node.kubernetes.io/windows-build=10.0.20348.

(Below snippet is an extraction from our workflow with modifications to hide sensitive data.)

Version

v3.4.11

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don’t enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: test
  namespace: dev
spec:
  templates:
    - name: whoami-template
      nodeSelector:
        kubernetes.io/os: windows
      metadata:
        annotations:
          sidecar.istio.io/inject: 'false'
      container:
        name: whoami-container
        image: >-
          mcr.microsoft.com/windows/servercore:ltsc2022
        command:
          - whoami
        resources: {}
        securityContext:
          capabilities:
            drop:
              - ALL
          runAsUser: 10001
          runAsNonRoot: true
          allowPrivilegeEscalation: false
          seccompProfile:
            type: RuntimeDefault
      tolerations:
        - key: windows
          operator: Exists
          effect: NoSchedule
    - name: main
      dag:
        tasks:
          - name: whoami
            template: whoami-template
  entrypoint: main
  arguments: {}
  serviceAccountName: workflow-execution
  podGC:
    strategy: OnWorkflowCompletion
  securityContext:
    seccompProfile:
      type: RuntimeDefault

Logs from the workflow controller

time="2023-09-13T12:38:16.903Z" level=info msg="Processing workflow" namespace=dev workflow=test
time="2023-09-13T12:38:16.909Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2023-09-13T12:38:16.909Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2023-09-13T12:38:16.909Z" level=info msg="Updated phase  -> Running" namespace=dev workflow=test
time="2023-09-13T12:38:16.909Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=dev workflow=test
time="2023-09-13T12:38:16.909Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=dev workflow=test
time="2023-09-13T12:38:16.909Z" level=info msg="DAG node test initialized Running" namespace=dev workflow=test
time="2023-09-13T12:38:16.909Z" level=warning msg="was unable to obtain the node for test-2372842584, taskName whoami"
time="2023-09-13T12:38:16.909Z" level=warning msg="was unable to obtain the node for test-2372842584, taskName whoami"
time="2023-09-13T12:38:16.909Z" level=info msg="All of node test.whoami dependencies [] completed" namespace=dev workflow=test
time="2023-09-13T12:38:16.909Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=dev workflow=test
time="2023-09-13T12:38:16.909Z" level=info msg="Pod node test-2372842584 initialized Pending" namespace=dev workflow=test
time="2023-09-13T12:38:16.984Z" level=info msg="Created pod: test.whoami (test-whoami-template-2372842584)" namespace=dev workflow=test
time="2023-09-13T12:38:16.984Z" level=info msg="TaskSet Reconciliation" namespace=dev workflow=test
time="2023-09-13T12:38:16.984Z" level=info msg=reconcileAgentPod namespace=dev workflow=test
time="2023-09-13T12:38:16.984Z" level=info msg="Workflow to be dehydrated" Workflow Size=2213
time="2023-09-13T12:38:17.002Z" level=info msg="Workflow update successful" namespace=dev phase=Running resourceVersion=93379873 workflow=test
time="2023-09-13T12:38:26.904Z" level=info msg="Processing workflow" namespace=dev workflow=test
time="2023-09-13T12:38:26.904Z" level=info msg="Task-result reconciliation" namespace=dev numObjs=0 workflow=test
time="2023-09-13T12:38:26.904Z" level=info msg="node changed" namespace=dev new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=test-2372842584 old.message= old.phase=Pending old.progress=0/1 workflow=test
time="2023-09-13T12:38:26.905Z" level=info msg="TaskSet Reconciliation" namespace=dev workflow=test
time="2023-09-13T12:38:26.905Z" level=info msg=reconcileAgentPod namespace=dev workflow=test
time="2023-09-13T12:38:26.905Z" level=info msg="Workflow to be dehydrated" Workflow Size=2498
time="2023-09-13T12:38:26.928Z" level=info msg="Workflow update successful" namespace=dev phase=Running resourceVersion=93379981 workflow=test
time="2023-09-13T12:38:36.929Z" level=info msg="Processing workflow" namespace=dev workflow=test
time="2023-09-13T12:38:36.929Z" level=info msg="Task-result reconciliation" namespace=dev numObjs=0 workflow=test
time="2023-09-13T12:38:36.929Z" level=info msg="node unchanged" namespace=dev nodeID=test-2372842584 workflow=test
time="2023-09-13T12:38:36.930Z" level=info msg="TaskSet Reconciliation" namespace=dev workflow=test
time="2023-09-13T12:38:36.930Z" level=info msg=reconcileAgentPod namespace=dev workflow=test
time="2023-09-13T12:38:48.121Z" level=info msg="Processing workflow" namespace=dev workflow=test
time="2023-09-13T12:38:48.121Z" level=info msg="Task-result reconciliation" namespace=dev numObjs=0 workflow=test
time="2023-09-13T12:38:48.121Z" level=info msg="node changed" namespace=dev new.message= new.phase=Running new.progress=0/1 nodeID=test-2372842584 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=test
time="2023-09-13T12:38:48.121Z" level=info msg="TaskSet Reconciliation" namespace=dev workflow=test
time="2023-09-13T12:38:48.121Z" level=info msg=reconcileAgentPod namespace=dev workflow=test
time="2023-09-13T12:38:48.121Z" level=info msg="Workflow to be dehydrated" Workflow Size=2482
time="2023-09-13T12:38:48.141Z" level=info msg="Workflow update successful" namespace=dev phase=Running resourceVersion=93380211 workflow=test
time="2023-09-13T12:38:56.430Z" level=info msg="Received Workflow Controller config map dev/argo-workflows-configmap-c9btt7bg2m update"
time="2023-09-13T12:38:56.435Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  azure:\n    accountKeySecret:\n      key: storage-account-access-key\n      name: storage-account-credentials\n    container: workflow-artifacts\n    endpoint: https://*******************.blob.core.windows.net\ninitialDelay: 0s\nmetricsConfig: {}\nnamespaceParallelism: 5\nnodeEvents: {}\nparallelism: 10\npodSpecLogStrategy: {}\nresourceRateLimit:\n  burst: 1\n  limit: 10\nsso:\n  clientId:\n    key: \"\"\n  clientSecret:\n    key: \"\"\n  issuer: \"\"\n  redirectUrl: \"\"\n  sessionExpiry: 0s\ntelemetryConfig: {}\n"
time="2023-09-13T12:38:56.435Z" level=info msg="Persistence configuration disabled"
time="2023-09-13T12:38:56.435Z" level=info executorImage="quay.io/argoproj/argoexec:v3.4.11" executorImagePullPolicy= managedNamespace=dev
time="2023-09-13T12:38:58.142Z" level=info msg="Processing workflow" namespace=dev workflow=test
time="2023-09-13T12:38:58.142Z" level=info msg="Task-result reconciliation" namespace=dev numObjs=0 workflow=test
time="2023-09-13T12:38:58.142Z" level=info msg="node unchanged" namespace=dev nodeID=test-2372842584 workflow=test
time="2023-09-13T12:38:58.142Z" level=info msg="TaskSet Reconciliation" namespace=dev workflow=test
time="2023-09-13T12:38:58.142Z" level=info msg=reconcileAgentPod namespace=dev workflow=test
time="2023-09-13T12:38:59.505Z" level=info msg="Received Workflow Controller config map dev/argo-workflows-configmap-c9btt7bg2m update"
time="2023-09-13T12:38:59.509Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  azure:\n    accountKeySecret:\n      key: storage-account-access-key\n      name: storage-account-credentials\n    container: workflow-artifacts\n    endpoint: https://********************.blob.core.windows.net\ninitialDelay: 0s\nmetricsConfig: {}\nnamespaceParallelism: 5\nnodeEvents: {}\nparallelism: 10\npodSpecLogStrategy: {}\nresourceRateLimit:\n  burst: 1\n  limit: 10\nsso:\n  clientId:\n    key: \"\"\n  clientSecret:\n    key: \"\"\n  issuer: \"\"\n  redirectUrl: \"\"\n  sessionExpiry: 0s\ntelemetryConfig: {}\n"
time="2023-09-13T12:38:59.509Z" level=info msg="Persistence configuration disabled"
time="2023-09-13T12:38:59.509Z" level=info executorImage="quay.io/argoproj/argoexec:v3.4.11" executorImagePullPolicy= managedNamespace=dev

Logs from in your workflow’s wait container

time="2023-09-13T12:38:35.970Z" level=info msg="Starting Workflow Executor" version=latest+v3.4.11.dirty
time="2023-09-13T12:38:36.124Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2023-09-13T12:38:36.124Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=dev podName=test-whoami-template-2372842584 template="{\"name\":\"whoami-template\",\"inputs\":{},\"outputs\":{},\"nodeSelector\":{\"kubernetes.io/os\":\"windows\"},\"metadata\":{\"annotations\":{\"sidecar.istio.io/inject\":\"false\"}},\"container\":{\"name\":\"whoami-container\",\"image\":\"mcr.microsoft.com/windows/servercore:ltsc2022\",\"command\":[\"whoami\"],\"resources\":{},\"securityContext\":{\"capabilities\":{\"drop\":[\"ALL\"]},\"runAsUser\":10001,\"runAsNonRoot\":true,\"allowPrivilegeEscalation\":false,\"seccompProfile\":{\"type\":\"RuntimeDefault\"}}},\"archiveLocation\":{\"archiveLogs\":true,\"azure\":{\"endpoint\":\"https://****************.blob.core.windows.net\",\"container\":\"workflow-artifacts\",\"accountKeySecret\":{\"name\":\"storage-account-credentials\",\"key\":\"storage-account-access-key\"},\"blob\":\"test/test-whoami-template-2372842584\"}},\"tolerations\":[{\"key\":\"windows\",\"operator\":\"Exists\",\"effect\":\"NoSchedule\"}]}" version="&Version{Version:latest+v3.4.11.dirty,BuildDate:2023-09-08T00:11:10Z,GitCommit:v3.4.11,GitTag:unknown,GitTreeState:,GoVersion:go1.20,Compiler:gc,Platform:windows/amd64,}"
time="2023-09-13T12:38:36.124Z" level=info msg="Starting deadline monitor"