argo-workflows: Pod logs no longer displayed in UI with POD_NAMES v2 since v3.2.2

Summary

What happened/what you expected to happen? Logs of running or completed pods shall be displayed in the Argo UI when I select a node and then click “Main logs”. This is no longer working since v3.2.2, if we use POD_NAMES = v2 (which we want to use). The UI displays “Waiting for data” for some time, later it shows an empty log window although the real log is not empty (visible with kubectl log…)

What version is it broken in? v3.2.2 (and still broken in v3.2.3)

What version was it working in? v3.2.1

Diagnostics

When using POD_NAMES = v1 it’s still working as expected.

I assume the UI tries to get logs under wrong name (v1 name) although pods are created with v2 name (?)

What Kubernetes provider are you using? BareMetal

What executor are you running? Emissary

# argo workflows v3.2.2
# base is the cluster install: https://github.com/argoproj/argo-workflows/blob/master/manifests/install.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: clusterworkflowtemplates.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: ClusterWorkflowTemplate
    listKind: ClusterWorkflowTemplateList
    plural: clusterworkflowtemplates
    shortNames:
    - clusterwftmpl
    - cwft
    singular: clusterworkflowtemplate
  scope: Cluster
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: cronworkflows.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: CronWorkflow
    listKind: CronWorkflowList
    plural: cronworkflows
    shortNames:
    - cwf
    - cronwf
    singular: cronworkflow
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
          status:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: workfloweventbindings.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: WorkflowEventBinding
    listKind: WorkflowEventBindingList
    plural: workfloweventbindings
    shortNames:
    - wfeb
    singular: workfloweventbinding
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: workflows.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: Workflow
    listKind: WorkflowList
    plural: workflows
    shortNames:
    - wf
    singular: workflow
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Status of the workflow
      jsonPath: .status.phase
      name: Status
      type: string
    - description: When the workflow was started
      format: date-time
      jsonPath: .status.startedAt
      name: Age
      type: date
    name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
          status:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
    subresources: {}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: workflowtasksets.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: WorkflowTaskSet
    listKind: WorkflowTaskSetList
    plural: workflowtasksets
    shortNames:
    - wfts
    singular: workflowtaskset
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
          status:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: workflowtemplates.argoproj.io
spec:
  group: argoproj.io
  names:
    kind: WorkflowTemplate
    listKind: WorkflowTemplateList
    plural: workflowtemplates
    shortNames:
    - wftmpl
    singular: workflowtemplate
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            type: object
            x-kubernetes-map-type: atomic
            x-kubernetes-preserve-unknown-fields: true
        required:
        - metadata
        - spec
        type: object
    served: true
    storage: true
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo-server
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo-role
rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - create
  - get
  - update
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
  name: argo-aggregate-to-admin
rules:
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  - workflows/finalizers
  - workfloweventbindings
  - workfloweventbindings/finalizers
  - workflowtemplates
  - workflowtemplates/finalizers
  - cronworkflows
  - cronworkflows/finalizers
  - clusterworkflowtemplates
  - clusterworkflowtemplates/finalizers
  - workflowtasksets
  - workflowtasksets/finalizers
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
  name: argo-aggregate-to-edit
rules:
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  - workflows/finalizers
  - workfloweventbindings
  - workfloweventbindings/finalizers
  - workflowtemplates
  - workflowtemplates/finalizers
  - cronworkflows
  - cronworkflows/finalizers
  - clusterworkflowtemplates
  - clusterworkflowtemplates/finalizers
  verbs:
  - create
  - delete
  - deletecollection
  - get
  - list
  - patch
  - update
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: argo-aggregate-to-view
rules:
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  - workflows/finalizers
  - workfloweventbindings
  - workfloweventbindings/finalizers
  - workflowtemplates
  - workflowtemplates/finalizers
  - cronworkflows
  - cronworkflows/finalizers
  - clusterworkflowtemplates
  - clusterworkflowtemplates/finalizers
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-cluster-role
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - pods/exec
  verbs:
  - create
  - get
  - list
  - watch
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - ""
  resources:
  - persistentvolumeclaims
  verbs:
  - create
  - delete
  - get
- apiGroups:
  - argoproj.io
  resources:
  - workflows
  - workflows/finalizers
  - workflowtasksets
  - workflowtasksets/finalizers
  verbs:
  - get
  - list
  - watch
  - update
  - patch
  - delete
  - create
- apiGroups:
  - argoproj.io
  resources:
  - workflowtemplates
  - workflowtemplates/finalizers
  - clusterworkflowtemplates
  - clusterworkflowtemplates/finalizers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - get
  - list
- apiGroups:
  - argoproj.io
  resources:
  - cronworkflows
  - cronworkflows/finalizers
  verbs:
  - get
  - list
  - watch
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - create
  - get
  - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-server-cluster-role
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - watch
  - list
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - create
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  - pods/exec
  - pods/log
  verbs:
  - get
  - list
  - watch
  - delete
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - watch
  - create
  - patch
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - argoproj.io
  resources:
  - eventsources
  - sensors
  - workflows
  - workfloweventbindings
  - workflowtemplates
  - cronworkflows
  - clusterworkflowtemplates
  verbs:
  - create
  - get
  - list
  - watch
  - update
  - patch
  - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argo-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argo-role
subjects:
- kind: ServiceAccount
  name: argo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argo-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argo-cluster-role
subjects:
- kind: ServiceAccount
  name: argo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: argo-server-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argo-server-cluster-role
subjects:
- kind: ServiceAccount
  name: argo-server
# ---
# apiVersion: v1
# kind: ConfigMap
# metadata:
#   name: workflow-controller-configmap
---
apiVersion: v1
kind: Service
metadata:
  name: argo-server
spec:
  ports:
  - name: web
    port: 2746
    targetPort: 2746
  selector:
    app: argo-server
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: workflow-controller
  name: workflow-controller-metrics
spec:
  ports:
  - name: metrics
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app: workflow-controller
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-server
spec:
  selector:
    matchLabels:
      app: argo-server
  template:
    metadata:
      labels:
        app: argo-server
    spec:
      containers:
      - args:
        - server
        - --auth-mode
        - server
        - --auth-mode
        - client
        - --secure=false
        - --basehref=/argo/
        image: quay.io/argoproj/argocli:v3.2.2
        name: argo-server
        env:
          - name: POD_NAMES
            value: "v2"
        ports:
        - containerPort: 2746
          name: web
        readinessProbe:
          httpGet:
            path: /
            port: 2746
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 20
        securityContext:
          capabilities:
            drop:
            - ALL
        volumeMounts:
        - mountPath: /tmp
          name: tmp
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
      serviceAccountName: argo-server
      volumes:
      - emptyDir: {}
        name: tmp
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: workflow-controller
spec:
  selector:
    matchLabels:
      app: workflow-controller
  template:
    metadata:
      labels:
        app: workflow-controller
    spec:
      containers:
      - args:
        - --configmap
        - workflow-controller-configmap
        - --executor-image
        - quay.io/argoproj/argoexec:v3.2.2
        command:
        - workflow-controller
        env:
        - name: LEADER_ELECTION_IDENTITY
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMES
          value: "v2"
        image: quay.io/argoproj/workflow-controller:v3.2.2
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 6060
          initialDelaySeconds: 90
          periodSeconds: 60
          timeoutSeconds: 30
        name: workflow-controller
        ports:
        - containerPort: 9090
          name: metrics
        - containerPort: 6060
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
      nodeSelector:
        kubernetes.io/os: linux
      securityContext:
        runAsNonRoot: true
      serviceAccountName: argo

# Logs from the workflow controller:
time="2021-11-15T15:20:26.550Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-analyze-and-inlineprocessing-1043567331/labelPodCompleted

C:\g>kubectl logs -n argo-workflows deploy/workflow-controller| grep wf-read-jlhhk
time="2021-11-15T15:19:56.437Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.482Z" level=info msg="Updated phase  -> Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.483Z" level=info msg="Steps node wf-read-jlhhk initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.483Z" level=info msg="StepGroup node wf-read-jlhhk-3260892249 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.483Z" level=info msg="Pod node wf-read-jlhhk-3807626384 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.511Z" level=info msg="Created pod: wf-read-jlhhk[0].CreateContextFolder (wf-read-jlhhk-createcontextfolder-3807626384)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.511Z" level=info msg="Workflow step group node wf-read-jlhhk-3260892249 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.511Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.511Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.536Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13591 workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.438Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.438Z" level=info msg="Updating node wf-read-jlhhk-3807626384 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.438Z" level=info msg="Setting node wf-read-jlhhk-3807626384 outputs: {\"parameters\":[{\"name\":\"contextfolderdirname\",\"value\":\"99b9a7393e792b03\",\"valueFrom\":{\"path\":\"/tmp/jsonResult.txt\"}}]}" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.438Z" level=info msg="Updating node wf-read-jlhhk-3807626384 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.439Z" level=info msg="Step group node wf-read-jlhhk-3260892249 successful" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.439Z" level=info msg="node wf-read-jlhhk-3260892249 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.439Z" level=info msg="node wf-read-jlhhk-3260892249 finished: 2021-11-15 15:20:06.439223391 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.439Z" level=info msg="StepGroup node wf-read-jlhhk-2120161252 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.439Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.440Z" level=info msg="DAG node wf-read-jlhhk-2838756226 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.440Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.ShowArguments dependencies [] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.441Z" level=info msg="Pod node wf-read-jlhhk-194545767 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.451Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.ShowArguments (wf-read-jlhhk-show-message-194545767)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.451Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.WadoRetrieve dependencies [] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.452Z" level=info msg="Pod node wf-read-jlhhk-123707277 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.483Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.WadoRetrieve (wf-read-jlhhk-wadoretrieve-123707277)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.483Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.483Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.483Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.503Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13649 workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.510Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-createcontextfolder-3807626384/labelPodCompleted
time="2021-11-15T15:20:16.452Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.453Z" level=info msg="Updating node wf-read-jlhhk-194545767 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.453Z" level=info msg="Updating node wf-read-jlhhk-194545767 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.453Z" level=info msg="Updating node wf-read-jlhhk-123707277 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.453Z" level=info msg="Updating node wf-read-jlhhk-123707277 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.453Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.457Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.PrintEnv dependencies [ShowArguments] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.459Z" level=info msg="Pod node wf-read-jlhhk-3548498998 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.469Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.PrintEnv (wf-read-jlhhk-printenv-3548498998)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.469Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.AnalyzeAndInlineProcessing dependencies [WadoRetrieve] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.471Z" level=info msg="Pod node wf-read-jlhhk-1043567331 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.482Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.AnalyzeAndInlineProcessing (wf-read-jlhhk-analyze-and-inlineprocessing-1043567331)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.482Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.482Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.482Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.502Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13705 workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.508Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-show-message-194545767/labelPodCompleted
time="2021-11-15T15:20:16.508Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-wadoretrieve-123707277/labelPodCompleted
time="2021-11-15T15:20:26.471Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="Updating node wf-read-jlhhk-3548498998 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="Updating node wf-read-jlhhk-3548498998 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="Updating node wf-read-jlhhk-1043567331 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="Setting node wf-read-jlhhk-1043567331 outputs: {\"parameters\":[{\"name\":\"processing-list\",\"value\":\"[{\\\"name\\\":\\\"syngo.processingsimulator\\\",\\\"tag\\\":\\\"Sim\\\",\\\"series\\\":\\\"\\\"}]\",\"valueFrom\":{\"path\":\"/tmp/processing-list.json\"}}]}" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="Updating node wf-read-jlhhk-1043567331 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.473Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.UpdateStudyLocator dependencies [AnalyzeAndInlineProcessing] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.475Z" level=info msg="Pod node wf-read-jlhhk-4025525804 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.484Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.UpdateStudyLocator (wf-read-jlhhk-updatestudylocator-4025525804)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.486Z" level=info msg="TaskGroup node wf-read-jlhhk-670985907 initialized Running (message: )" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.486Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.StartProcessingJobs(0:name:syngo.processingsimulator,series:,tag:Sim) dependencies [AnalyzeAndInlineProcessing] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.487Z" level=info msg="Steps node wf-read-jlhhk-3436236554 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.487Z" level=info msg="StepGroup node wf-read-jlhhk-3462332412 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.488Z" level=info msg="Pod node wf-read-jlhhk-2071656461 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.518Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.StartProcessingJobs(0:name:syngo.processingsimulator,series:,tag:Sim)[0].StartProcessing (wf-read-jlhhk-start-processing-job-2071656461)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.519Z" level=info msg="Workflow step group node wf-read-jlhhk-3462332412 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.519Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.519Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.519Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.542Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13783 workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.550Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-printenv-3548498998/labelPodCompleted
time="2021-11-15T15:20:26.550Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-analyze-and-inlineprocessing-1043567331/labelPodCompleted
time="2021-11-15T15:20:36.486Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.486Z" level=info msg="Updating node wf-read-jlhhk-4025525804 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.486Z" level=info msg="Updating node wf-read-jlhhk-4025525804 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.486Z" level=info msg="Updating node wf-read-jlhhk-2071656461 status Pending -> Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.486Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.489Z" level=info msg="Workflow step group node wf-read-jlhhk-3462332412 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.489Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.489Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.489Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.502Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13846 workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.508Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-updatestudylocator-4025525804/labelPodCompleted

C:\g>kubectl logs -n argo-workflows deploy/workflow-controller| grep wf-read-jlhhk
time="2021-11-15T15:19:56.437Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.482Z" level=info msg="Updated phase  -> Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.483Z" level=info msg="Steps node wf-read-jlhhk initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.483Z" level=info msg="StepGroup node wf-read-jlhhk-3260892249 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.483Z" level=info msg="Pod node wf-read-jlhhk-3807626384 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.511Z" level=info msg="Created pod: wf-read-jlhhk[0].CreateContextFolder (wf-read-jlhhk-createcontextfolder-3807626384)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.511Z" level=info msg="Workflow step group node wf-read-jlhhk-3260892249 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.511Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.511Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:19:56.536Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13591 workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.438Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.438Z" level=info msg="Updating node wf-read-jlhhk-3807626384 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.438Z" level=info msg="Setting node wf-read-jlhhk-3807626384 outputs: {\"parameters\":[{\"name\":\"contextfolderdirname\",\"value\":\"99b9a7393e792b03\",\"valueFrom\":{\"path\":\"/tmp/jsonResult.txt\"}}]}" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.438Z" level=info msg="Updating node wf-read-jlhhk-3807626384 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.439Z" level=info msg="Step group node wf-read-jlhhk-3260892249 successful" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.439Z" level=info msg="node wf-read-jlhhk-3260892249 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.439Z" level=info msg="node wf-read-jlhhk-3260892249 finished: 2021-11-15 15:20:06.439223391 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.439Z" level=info msg="StepGroup node wf-read-jlhhk-2120161252 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.439Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.440Z" level=info msg="DAG node wf-read-jlhhk-2838756226 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.440Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.ShowArguments dependencies [] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.441Z" level=info msg="Pod node wf-read-jlhhk-194545767 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.451Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.ShowArguments (wf-read-jlhhk-show-message-194545767)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.451Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.WadoRetrieve dependencies [] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.452Z" level=info msg="Pod node wf-read-jlhhk-123707277 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.483Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.WadoRetrieve (wf-read-jlhhk-wadoretrieve-123707277)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.483Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.483Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.483Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.503Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13649 workflow=wf-read-jlhhk
time="2021-11-15T15:20:06.510Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-createcontextfolder-3807626384/labelPodCompleted
time="2021-11-15T15:20:16.452Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.453Z" level=info msg="Updating node wf-read-jlhhk-194545767 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.453Z" level=info msg="Updating node wf-read-jlhhk-194545767 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.453Z" level=info msg="Updating node wf-read-jlhhk-123707277 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.453Z" level=info msg="Updating node wf-read-jlhhk-123707277 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.453Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.457Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.PrintEnv dependencies [ShowArguments] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.459Z" level=info msg="Pod node wf-read-jlhhk-3548498998 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.469Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.PrintEnv (wf-read-jlhhk-printenv-3548498998)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.469Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.AnalyzeAndInlineProcessing dependencies [WadoRetrieve] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.471Z" level=info msg="Pod node wf-read-jlhhk-1043567331 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.482Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.AnalyzeAndInlineProcessing (wf-read-jlhhk-analyze-and-inlineprocessing-1043567331)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.482Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.482Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.482Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.502Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13705 workflow=wf-read-jlhhk
time="2021-11-15T15:20:16.508Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-show-message-194545767/labelPodCompleted
time="2021-11-15T15:20:16.508Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-wadoretrieve-123707277/labelPodCompleted
time="2021-11-15T15:20:26.471Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="Updating node wf-read-jlhhk-3548498998 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="Updating node wf-read-jlhhk-3548498998 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="Updating node wf-read-jlhhk-1043567331 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="Setting node wf-read-jlhhk-1043567331 outputs: {\"parameters\":[{\"name\":\"processing-list\",\"value\":\"[{\\\"name\\\":\\\"syngo.processingsimulator\\\",\\\"tag\\\":\\\"Sim\\\",\\\"series\\\":\\\"\\\"}]\",\"valueFrom\":{\"path\":\"/tmp/processing-list.json\"}}]}" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="Updating node wf-read-jlhhk-1043567331 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.472Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.473Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.UpdateStudyLocator dependencies [AnalyzeAndInlineProcessing] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.475Z" level=info msg="Pod node wf-read-jlhhk-4025525804 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.484Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.UpdateStudyLocator (wf-read-jlhhk-updatestudylocator-4025525804)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.486Z" level=info msg="TaskGroup node wf-read-jlhhk-670985907 initialized Running (message: )" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.486Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.StartProcessingJobs(0:name:syngo.processingsimulator,series:,tag:Sim) dependencies [AnalyzeAndInlineProcessing] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.487Z" level=info msg="Steps node wf-read-jlhhk-3436236554 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.487Z" level=info msg="StepGroup node wf-read-jlhhk-3462332412 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.488Z" level=info msg="Pod node wf-read-jlhhk-2071656461 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.518Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.StartProcessingJobs(0:name:syngo.processingsimulator,series:,tag:Sim)[0].StartProcessing (wf-read-jlhhk-start-processing-job-2071656461)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.519Z" level=info msg="Workflow step group node wf-read-jlhhk-3462332412 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.519Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.519Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.519Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.542Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13783 workflow=wf-read-jlhhk
time="2021-11-15T15:20:26.550Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-printenv-3548498998/labelPodCompleted
time="2021-11-15T15:20:26.550Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-analyze-and-inlineprocessing-1043567331/labelPodCompleted
time="2021-11-15T15:20:36.486Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.486Z" level=info msg="Updating node wf-read-jlhhk-4025525804 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.486Z" level=info msg="Updating node wf-read-jlhhk-4025525804 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.486Z" level=info msg="Updating node wf-read-jlhhk-2071656461 status Pending -> Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.486Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.489Z" level=info msg="Workflow step group node wf-read-jlhhk-3462332412 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.489Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.489Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.489Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.502Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13846 workflow=wf-read-jlhhk
time="2021-11-15T15:20:36.508Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-updatestudylocator-4025525804/labelPodCompleted
time="2021-11-15T15:20:46.521Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.522Z" level=info msg="Updating node wf-read-jlhhk-2071656461 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.522Z" level=info msg="Updating node wf-read-jlhhk-2071656461 status Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.522Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.524Z" level=info msg="Step group node wf-read-jlhhk-3462332412 successful" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.524Z" level=info msg="node wf-read-jlhhk-3462332412 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.524Z" level=info msg="node wf-read-jlhhk-3462332412 finished: 2021-11-15 15:20:46.524810933 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.524Z" level=info msg="StepGroup node wf-read-jlhhk-308096113 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.525Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-2071656461 are [wf-read-jlhhk-2071656461]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.527Z" level=info msg="Pod node wf-read-jlhhk-3192613456 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.536Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.StartProcessingJobs(0:name:syngo.processingsimulator,series:,tag:Sim)[1].Publish (wf-read-jlhhk-publish-results-from-folder-3192613456)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.536Z" level=info msg="Workflow step group node wf-read-jlhhk-308096113 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.536Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.536Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.536Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.557Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13879 workflow=wf-read-jlhhk
time="2021-11-15T15:20:46.565Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-start-processing-job-2071656461/labelPodCompleted
time="2021-11-15T15:20:56.538Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.538Z" level=info msg="Updating node wf-read-jlhhk-3192613456 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.538Z" level=info msg="Updating node wf-read-jlhhk-3192613456 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.538Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.541Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-2071656461 are [wf-read-jlhhk-2071656461]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.541Z" level=info msg="Step group node wf-read-jlhhk-308096113 successful" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.541Z" level=info msg="node wf-read-jlhhk-308096113 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.541Z" level=info msg="node wf-read-jlhhk-308096113 finished: 2021-11-15 15:20:56.541379435 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.542Z" level=info msg="Outbound nodes of wf-read-jlhhk-3192613456 is [wf-read-jlhhk-3192613456]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.542Z" level=info msg="Outbound nodes of wf-read-jlhhk-3436236554 is [wf-read-jlhhk-3192613456]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.542Z" level=info msg="node wf-read-jlhhk-3436236554 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.542Z" level=info msg="node wf-read-jlhhk-3436236554 finished: 2021-11-15 15:20:56.542219497 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.542Z" level=info msg="Checking daemoned children of wf-read-jlhhk-3436236554" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.542Z" level=info msg="node wf-read-jlhhk-670985907 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.542Z" level=info msg="node wf-read-jlhhk-670985907 finished: 2021-11-15 15:20:56.542242496 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.542Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.LoadProcessingResults dependencies [StartProcessingJobs] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.544Z" level=info msg="Pod node wf-read-jlhhk-2470001139 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.559Z" level=info msg="Created pod: wf-read-jlhhk[1].mainwork.LoadProcessingResults (wf-read-jlhhk-analyze-and-inlineprocessing-2470001139)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.559Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.559Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.559Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.597Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13917 workflow=wf-read-jlhhk
time="2021-11-15T15:20:56.603Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-publish-results-from-folder-3192613456/labelPodCompleted
time="2021-11-15T15:21:06.562Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.562Z" level=info msg="Updating node wf-read-jlhhk-2470001139 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.562Z" level=info msg="Setting node wf-read-jlhhk-2470001139 outputs: {\"parameters\":[{\"name\":\"processing-list\",\"value\":\"[{\\\"name\\\":\\\"syngo.processingsimulator\\\",\\\"tag\\\":\\\"Sim\\\",\\\"series\\\":\\\"\\\"}]\",\"valueFrom\":{\"path\":\"/tmp/processing-list.json\"}}]}" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.562Z" level=info msg="Updating node wf-read-jlhhk-2470001139 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.563Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.565Z" level=info msg="All of node wf-read-jlhhk[1].mainwork.Wait6sec dependencies [LoadProcessingResults] completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.567Z" level=info msg="Suspend node wf-read-jlhhk-3429905478 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.567Z" level=info msg="node wf-read-jlhhk[1].mainwork.Wait6sec suspended" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.567Z" level=info msg="node wf-read-jlhhk-3429905478 phase Pending -> Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.567Z" level=info msg="Workflow step group node wf-read-jlhhk-2120161252 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.567Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.567Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.588Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=13994 workflow=wf-read-jlhhk
time="2021-11-15T15:21:06.594Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-analyze-and-inlineprocessing-2470001139/labelPodCompleted
time="2021-11-15T15:21:12.568Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.569Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-3807626384 are [wf-read-jlhhk-3807626384]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.570Z" level=info msg="node wf-read-jlhhk[1].mainwork.Wait6sec suspended" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.570Z" level=info msg="auto resuming node wf-read-jlhhk[1].mainwork.Wait6sec" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.570Z" level=info msg="node wf-read-jlhhk-3429905478 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.570Z" level=info msg="node wf-read-jlhhk-3429905478 finished: 2021-11-15 15:21:12.570769869 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.572Z" level=info msg="Outbound nodes of wf-read-jlhhk-2838756226 set to [wf-read-jlhhk-3548498998 wf-read-jlhhk-4025525804 wf-read-jlhhk-3429905478]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.572Z" level=info msg="node wf-read-jlhhk-2838756226 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.572Z" level=info msg="node wf-read-jlhhk-2838756226 finished: 2021-11-15 15:21:12.572526391 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.572Z" level=info msg="Checking daemoned children of wf-read-jlhhk-2838756226" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.572Z" level=info msg="Step group node wf-read-jlhhk-2120161252 successful" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.572Z" level=info msg="node wf-read-jlhhk-2120161252 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.572Z" level=info msg="node wf-read-jlhhk-2120161252 finished: 2021-11-15 15:21:12.572564589 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.574Z" level=info msg="Outbound nodes of wf-read-jlhhk-2838756226 is [wf-read-jlhhk-3548498998 wf-read-jlhhk-4025525804 wf-read-jlhhk-3429905478]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.574Z" level=info msg="Outbound nodes of wf-read-jlhhk is [wf-read-jlhhk-3548498998 wf-read-jlhhk-4025525804 wf-read-jlhhk-3429905478]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.574Z" level=info msg="node wf-read-jlhhk phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.574Z" level=info msg="node wf-read-jlhhk finished: 2021-11-15 15:21:12.574944682 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.574Z" level=info msg="Checking daemoned children of wf-read-jlhhk" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.574Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.575Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.575Z" level=info msg="Running OnExit handler: exit-handler" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.575Z" level=info msg="Steps node wf-read-jlhhk-1017822540 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.575Z" level=info msg="StepGroup node wf-read-jlhhk-527365286 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.575Z" level=info msg="Pod node wf-read-jlhhk-2862962021 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.585Z" level=info msg="Created pod: wf-read-jlhhk.onExit[0].DestroyContextFolder (wf-read-jlhhk-destroycontextfolder-2862962021)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.585Z" level=info msg="Workflow step group node wf-read-jlhhk-527365286 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:12.632Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=14011 workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.587Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.587Z" level=info msg="Updating node wf-read-jlhhk-2862962021 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.587Z" level=info msg="Updating node wf-read-jlhhk-2862962021 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.587Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.587Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.587Z" level=info msg="Running OnExit handler: exit-handler" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.588Z" level=info msg="Step group node wf-read-jlhhk-527365286 successful" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.588Z" level=info msg="node wf-read-jlhhk-527365286 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.588Z" level=info msg="node wf-read-jlhhk-527365286 finished: 2021-11-15 15:21:22.588331227 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.588Z" level=info msg="StepGroup node wf-read-jlhhk-460107715 initialized Running" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.588Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-2862962021 are [wf-read-jlhhk-2862962021]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.588Z" level=info msg="Pod node wf-read-jlhhk-2873462450 initialized Pending" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.600Z" level=info msg="Created pod: wf-read-jlhhk.onExit[1].workflow-succeeded-notification (wf-read-jlhhk-workflow-succeeded-notification-2873462450)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.600Z" level=info msg="Skipping wf-read-jlhhk.onExit[1].workflow-failed-notification: when 'Succeeded != Succeeded' evaluated false" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.600Z" level=info msg="Skipped node wf-read-jlhhk-980737568 initialized Skipped (message: when 'Succeeded != Succeeded' evaluated false)" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.600Z" level=info msg="Workflow step group node wf-read-jlhhk-460107715 not yet completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.622Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Running resourceVersion=14053 workflow=wf-read-jlhhk
time="2021-11-15T15:21:22.635Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-destroycontextfolder-2862962021/labelPodCompleted
time="2021-11-15T15:21:32.599Z" level=info msg="Processing workflow" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.600Z" level=info msg="Updating node wf-read-jlhhk-2873462450 exit code 0" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.600Z" level=info msg="Updating node wf-read-jlhhk-2873462450 status Pending -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.600Z" level=info msg="TaskSet Reconciliation" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.600Z" level=info msg=reconcileAgentPod namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.600Z" level=info msg="Running OnExit handler: exit-handler" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.600Z" level=info msg="SG Outbound nodes of wf-read-jlhhk-2862962021 are [wf-read-jlhhk-2862962021]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.600Z" level=info msg="Step group node wf-read-jlhhk-460107715 successful" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.600Z" level=info msg="node wf-read-jlhhk-460107715 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.600Z" level=info msg="node wf-read-jlhhk-460107715 finished: 2021-11-15 15:21:32.600984205 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.601Z" level=info msg="Outbound nodes of wf-read-jlhhk-2873462450 is [wf-read-jlhhk-2873462450]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.601Z" level=info msg="Outbound nodes of wf-read-jlhhk-980737568 is [wf-read-jlhhk-980737568]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.601Z" level=info msg="Outbound nodes of wf-read-jlhhk-1017822540 is [wf-read-jlhhk-2873462450 wf-read-jlhhk-980737568]" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.601Z" level=info msg="node wf-read-jlhhk-1017822540 phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.601Z" level=info msg="node wf-read-jlhhk-1017822540 finished: 2021-11-15 15:21:32.601077601 +0000 UTC" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.601Z" level=info msg="Checking daemoned children of wf-read-jlhhk-1017822540" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.601Z" level=info msg="Updated phase Running -> Succeeded" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.601Z" level=info msg="Marking workflow completed" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.601Z" level=info msg="Marking workflow as pending archiving" namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.601Z" level=info msg="Checking daemoned children of " namespace=kondor-workflow workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.620Z" level=info msg="Workflow update successful" namespace=kondor-workflow phase=Succeeded resourceVersion=14090 workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.624Z" level=info msg="archiving workflow" namespace=kondor-workflow uid=5907921b-0a35-4849-805d-83d4e30a8770 workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.627Z" level=info msg="cleaning up pod" action=labelPodCompleted key=kondor-workflow/wf-read-jlhhk-workflow-succeeded-notification-2873462450/labelPodCompleted
time="2021-11-15T15:21:32.675Z" level=info msg="archiving workflow" namespace=kondor-workflow uid=5907921b-0a35-4849-805d-83d4e30a8770 workflow=wf-read-jlhhk
time="2021-11-15T15:21:32.679Z" level=info msg="Queueing Succeeded workflow kondor-workflow/wf-read-jlhhk for delete in 5m0s"
time="2021-11-15T15:21:32.680Z" level=info msg="Queueing Succeeded workflow kondor-workflow/wf-read-jlhhk for delete in 5m0s"```

---
<!-- Issue Author: Don't delete this message to encourage other users to support your issue! -->
**Message from the maintainers**:

Impacted by this regression? Give it a 👍. We prioritise the issues with the most 👍.