argo-workflows: artifact passing: "x509: certificate signed by unknown authority"

Hey! I am using argo workflows together with minio as a base for various machine learning automation setups. I love it, thanks for your efforts! I did however encounter I problem when switching to the new minio-operator version, that I can’t seem to figure out (in a non-hacky way). Hope you can help me out here.

Summary

What happened/what you expected to happen? Minio-operator has been around for a while now, in favour of its (now deprecated) former version. In order to use the new buckets for artifact passing, one needs to make use of https. In our case, the certificates used to do so, are self-signed wich causes issues when artifacts need to be uploaded. After the .tar.gz has been created, the upload fails with error message “… x509: certificate signed by unknown authority”. I’m curious how to configure things right here. Do I really need to mount the related .ca file to ANY workflow using artifacts itself, in order to use minio with https (and a self-signed) cert?

  • Have you double-checked your configuration? Maybe 30% of issues are wrong configuration.

I’ve double-checked my config. To my best knowledge, there is no option that allows me to ignore the insecure authority. Please verify if you like:

containerRuntimeExecutor: docker
artifactRepository:
  archiveLogs: false
  s3:
    endpoint: minio.minio-ssd.svc.cluster.local:443
    bucket: test
    insecure: false
    useSDKCreds: false
    accessKeySecret:
      name: minio-key
      key: access_key_id
    secretKeySecret:
      name: minio-secret
      key: secret_access_key

I’ve also tried setting the kubeletInsecure Option, but I assume, that would require me to use kubelet as workflows executor?

Diagnostics

What Kubernetes provider are you using? On-premise, 10-node cluster, each node is a Centos-7 virtual machine, Kubernetes Version 1.20.8

What version of Argo Workflows are you running? The version used for argo-cli and argo-workflow-controller was v2.12.5.

What executor are you running? Docker/K8SAPI/Kubelet/PNS/Emissary Docker was used as an executor (I have no other possibility).

Did this work in a previous version? I.e. is it a regression? I just switched to the minio operator version and thus can’t confirm for later versions.

Workflow YAML:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  creationTimestamp: "2021-08-31T14:53:20Z"
  generateName: artifact-passing-
  generation: 4
  labels:
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Failed
    workflows.argoproj.io/resubmitted-from-workflow: artifact-passing-pj8v7
  name: artifact-passing-5bs6q
  namespace: twin-pipelines
  resourceVersion: "660135225"
  uid: f28ff596-914e-4d97-b55f-3de083bd455d
spec:
  arguments: {}
  entrypoint: artifact-example
  serviceAccountName: pipeline-runner
  templates:
  - arguments: {}
    inputs: {}
    metadata: {}
    name: artifact-example
    outputs: {}
    steps:
    - - arguments: {}
        name: generate-artifact
        template: whalesay
    - - arguments:
          artifacts:
          - from: '{{steps.generate-artifact.outputs.artifacts.hello-art}}'
            name: message
        name: consume-artifact
        template: print-message
  - arguments: {}
    container:
      args:
      - sleep 1; cowsay hello world | tee /tmp/hello_world.txt
      command:
      - sh
      - -c
      image: docker/whalesay:latest
      name: ""
      resources: {}
    inputs: {}
    metadata: {}
    name: whalesay
    outputs:
      artifacts:
      - name: hello-art
        path: /tmp/hello_world.txt
  - arguments: {}
    container:
      args:
      - cat /tmp/message
      command:
      - sh
      - -c
      image: alpine:latest
      name: ""
      resources: {}
    inputs:
      artifacts:
      - name: message
        path: /tmp/message
    metadata: {}
    name: print-message
    outputs: {}
status:
  conditions:
  - status: "True"
    type: Completed
  finishedAt: "2021-08-31T14:54:07Z"
  message: child 'artifact-passing-5bs6q-1898326195' failed
  nodes:
    artifact-passing-5bs6q:
      children:
      - artifact-passing-5bs6q-2310228769
      displayName: artifact-passing-5bs6q
      finishedAt: "2021-08-31T14:54:07Z"
      id: artifact-passing-5bs6q
      message: child 'artifact-passing-5bs6q-1898326195' failed
      name: artifact-passing-5bs6q
      outboundNodes:
      - artifact-passing-5bs6q-1898326195
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 36
        memory: 36
      startedAt: "2021-08-31T14:53:20Z"
      templateName: artifact-example
      templateScope: local/artifact-passing-5bs6q
      type: Steps
    artifact-passing-5bs6q-1898326195:
      boundaryID: artifact-passing-5bs6q
      displayName: generate-artifact
      finishedAt: "2021-08-31T14:53:57Z"
      hostNodeName: cuda01-p100
      id: artifact-passing-5bs6q-1898326195
      message: 'failed to save outputs: timed out waiting for the condition'
      name: artifact-passing-5bs6q[0].generate-artifact
      phase: Error
      progress: 1/1
      resourcesDuration:
        cpu: 36
        memory: 36
      startedAt: "2021-08-31T14:53:20Z"
      templateName: whalesay
      templateScope: local/artifact-passing-5bs6q
      type: Pod
    artifact-passing-5bs6q-2310228769:
      boundaryID: artifact-passing-5bs6q
      children:
      - artifact-passing-5bs6q-1898326195
      displayName: '[0]'
      finishedAt: "2021-08-31T14:54:07Z"
      id: artifact-passing-5bs6q-2310228769
      message: child 'artifact-passing-5bs6q-1898326195' failed
      name: artifact-passing-5bs6q[0]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 36
        memory: 36
      startedAt: "2021-08-31T14:53:20Z"
      templateName: artifact-example
      templateScope: local/artifact-passing-5bs6q
      type: StepGroup
  phase: Failed
  progress: 1/1
  resourcesDuration:
    cpu: 36
    memory: 36
  startedAt: "2021-08-31T14:53:20Z"

Logs from the workflow controller:

time="2021-08-31T12:55:45.257Z" level=info msg="config map" name=argo-workflow-controller-configmap
time="2021-08-31T12:55:45.283Z" level=info msg="Workers: workflow: 32, pod: 32"
time="2021-08-31T12:55:45.286Z" level=info msg="Persistence disabled - so archived workflow GC disabled - you must restart the controller if you enable this"
time="2021-08-31T12:55:45.290Z" level=info msg="Starting workflow TTL controller (workflowTTLWorkers 4)"
time="2021-08-31T12:55:45.490Z" level=info msg="Started workflow TTL worker"
time="2021-08-31T12:55:54.474Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:55:54.476Z" level=info msg="Pod node artifact-passing-fnjrp-7432736 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:55:54.499Z" level=info msg="Created pod: artifact-passing-fnjrp[0].generate-artifact (artifact-passing-fnjrp-7432736)" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:55:54.500Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:55:54.537Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660077457 workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:04.503Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:04.505Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:04.520Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660077545 workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:14.525Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:14.527Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.430Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.430Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-fnjrp-7432736 templateName=whalesay workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="Step group node artifact-passing-fnjrp-4055596050 deemed failed: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp-4055596050 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp-4055596050 message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp-4055596050 finished: 2021-08-31 12:56:40.432283488 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="step group artifact-passing-fnjrp-4055596050 was unsuccessful: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="Outbound nodes of artifact-passing-fnjrp-7432736 is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="Outbound nodes of artifact-passing-fnjrp is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp finished: 2021-08-31 12:56:40.432517288 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="Checking daemoned children of artifact-passing-fnjrp" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="Updated message  -> child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.432Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T12:56:40.451Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660077841 workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:06.330Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:06.332Z" level=info msg="Pod node artifact-passing-fnjrp-7432736 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:06.345Z" level=info msg="Created pod: artifact-passing-fnjrp[0].generate-artifact (artifact-passing-fnjrp-7432736)" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:06.345Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:06.359Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660079525 workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:16.348Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:16.349Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:16.366Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660079624 workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:26.371Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:26.373Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.415Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.415Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-fnjrp-7432736 templateName=whalesay workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="Step group node artifact-passing-fnjrp-4055596050 deemed failed: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp-4055596050 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp-4055596050 message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp-4055596050 finished: 2021-08-31 13:00:52.417418597 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="step group artifact-passing-fnjrp-4055596050 was unsuccessful: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="Outbound nodes of artifact-passing-fnjrp-7432736 is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="Outbound nodes of artifact-passing-fnjrp is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp finished: 2021-08-31 13:00:52.417652801 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="Checking daemoned children of artifact-passing-fnjrp" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="Updated message  -> child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.417Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:00:52.432Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660079916 workflow=artifact-passing-fnjrp
time="2021-08-31T13:59:56.715Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:59:56.716Z" level=info msg="Pod node artifact-passing-fnjrp-7432736 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:59:56.728Z" level=info msg="Created pod: artifact-passing-fnjrp[0].generate-artifact (artifact-passing-fnjrp-7432736)" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:59:56.728Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T13:59:56.747Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660108615 workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:06.733Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:06.735Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:06.754Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660108704 workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.721Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.721Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-fnjrp-7432736 templateName=whalesay workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="Step group node artifact-passing-fnjrp-4055596050 deemed failed: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp-4055596050 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp-4055596050 message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp-4055596050 finished: 2021-08-31 14:00:42.722724904 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="step group artifact-passing-fnjrp-4055596050 was unsuccessful: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="Outbound nodes of artifact-passing-fnjrp-7432736 is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="Outbound nodes of artifact-passing-fnjrp is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp finished: 2021-08-31 14:00:42.722943298 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.722Z" level=info msg="Checking daemoned children of artifact-passing-fnjrp" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.723Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.723Z" level=info msg="Updated message  -> child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.723Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.723Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:00:42.741Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660109020 workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:03.243Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:03.244Z" level=info msg="Pod node artifact-passing-fnjrp-7432736 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:03.254Z" level=info msg="Created pod: artifact-passing-fnjrp[0].generate-artifact (artifact-passing-fnjrp-7432736)" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:03.254Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:03.271Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660115546 workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:13.260Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:13.261Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:13.279Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660115636 workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:23.286Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:23.288Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.409Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.410Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-fnjrp-7432736 templateName=whalesay workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="Step group node artifact-passing-fnjrp-4055596050 deemed failed: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp-4055596050 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp-4055596050 message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp-4055596050 finished: 2021-08-31 14:14:49.411222907 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="step group artifact-passing-fnjrp-4055596050 was unsuccessful: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="Outbound nodes of artifact-passing-fnjrp-7432736 is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="Outbound nodes of artifact-passing-fnjrp is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp finished: 2021-08-31 14:14:49.411365931 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="Checking daemoned children of artifact-passing-fnjrp" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="Updated message  -> child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.411Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-fnjrp
time="2021-08-31T14:14:49.429Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660115925 workflow=artifact-passing-fnjrp
time="2021-08-31T14:16:55.295Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:16:55.295Z" level=info msg="Updated phase  -> Running" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:16:55.298Z" level=info msg="Steps node artifact-passing-djjm8 initialized Running" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:16:55.298Z" level=info msg="StepGroup node artifact-passing-djjm8-2975015209 initialized Running" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:16:55.299Z" level=info msg="Pod node artifact-passing-djjm8-187514843 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:16:55.310Z" level=info msg="Created pod: artifact-passing-djjm8[0].generate-artifact (artifact-passing-djjm8-187514843)" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:16:55.310Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:16:55.339Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660116982 workflow=artifact-passing-djjm8
time="2021-08-31T14:17:05.314Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:05.316Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:05.331Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660117073 workflow=artifact-passing-djjm8
time="2021-08-31T14:17:15.335Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:15.337Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.406Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.406Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-djjm8-187514843 templateName=whalesay workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.407Z" level=info msg="Step group node artifact-passing-djjm8-2975015209 deemed failed: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8-2975015209 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8-2975015209 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8-2975015209 finished: 2021-08-31 14:17:41.40810134 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="step group artifact-passing-djjm8-2975015209 was unsuccessful: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="Outbound nodes of artifact-passing-djjm8-187514843 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="Outbound nodes of artifact-passing-djjm8 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8 finished: 2021-08-31 14:17:41.408331268 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="Checking daemoned children of artifact-passing-djjm8" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="Updated message  -> child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.408Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:17:41.422Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660117382 workflow=artifact-passing-djjm8
time="2021-08-31T14:33:13.468Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:13.469Z" level=info msg="Pod node artifact-passing-djjm8-187514843 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:13.479Z" level=info msg="Created pod: artifact-passing-djjm8[0].generate-artifact (artifact-passing-djjm8-187514843)" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:13.480Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:13.498Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660124874 workflow=artifact-passing-djjm8
time="2021-08-31T14:33:23.485Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:23.487Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:23.505Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660124966 workflow=artifact-passing-djjm8
time="2021-08-31T14:33:33.511Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:33.513Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.838Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.838Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-djjm8-187514843 templateName=whalesay workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="Step group node artifact-passing-djjm8-2975015209 deemed failed: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8-2975015209 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8-2975015209 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8-2975015209 finished: 2021-08-31 14:33:59.84042281 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="step group artifact-passing-djjm8-2975015209 was unsuccessful: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="Outbound nodes of artifact-passing-djjm8-187514843 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="Outbound nodes of artifact-passing-djjm8 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8 finished: 2021-08-31 14:33:59.840687014 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="Checking daemoned children of artifact-passing-djjm8" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="Updated message  -> child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.840Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:33:59.864Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660125279 workflow=artifact-passing-djjm8
time="2021-08-31T14:45:00.006Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:00.007Z" level=info msg="Pod node artifact-passing-djjm8-187514843 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:00.017Z" level=info msg="Created pod: artifact-passing-djjm8[0].generate-artifact (artifact-passing-djjm8-187514843)" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:00.017Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:00.035Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660130668 workflow=artifact-passing-djjm8
time="2021-08-31T14:45:10.023Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:10.025Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:10.047Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660130760 workflow=artifact-passing-djjm8
time="2021-08-31T14:45:20.052Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:20.054Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.161Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.162Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-djjm8-187514843 templateName=whalesay workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="Step group node artifact-passing-djjm8-2975015209 deemed failed: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8-2975015209 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8-2975015209 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8-2975015209 finished: 2021-08-31 14:45:46.163355789 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="step group artifact-passing-djjm8-2975015209 was unsuccessful: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="Outbound nodes of artifact-passing-djjm8-187514843 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="Outbound nodes of artifact-passing-djjm8 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8 finished: 2021-08-31 14:45:46.163518793 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="Checking daemoned children of artifact-passing-djjm8" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="Updated message  -> child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.163Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-djjm8
time="2021-08-31T14:45:46.201Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660131066 workflow=artifact-passing-djjm8
time="2021-08-31T14:48:31.364Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:31.364Z" level=info msg="Updated phase  -> Running" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:31.366Z" level=info msg="Steps node artifact-passing-pj8v7 initialized Running" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:31.366Z" level=info msg="StepGroup node artifact-passing-pj8v7-1355320965 initialized Running" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:31.367Z" level=info msg="Pod node artifact-passing-pj8v7-854535231 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:31.375Z" level=info msg="Created pod: artifact-passing-pj8v7[0].generate-artifact (artifact-passing-pj8v7-854535231)" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:31.375Z" level=info msg="Workflow step group node artifact-passing-pj8v7-1355320965 not yet completed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:31.396Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660132426 workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:41.380Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:41.382Z" level=info msg="Workflow step group node artifact-passing-pj8v7-1355320965 not yet completed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:41.395Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660132519 workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:51.400Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:48:51.402Z" level=info msg="Workflow step group node artifact-passing-pj8v7-1355320965 not yet completed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.530Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.530Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-pj8v7-854535231 templateName=whalesay workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="Step group node artifact-passing-pj8v7-1355320965 deemed failed: child 'artifact-passing-pj8v7-854535231' failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7-1355320965 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7-1355320965 message: child 'artifact-passing-pj8v7-854535231' failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7-1355320965 finished: 2021-08-31 14:49:17.532468701 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="step group artifact-passing-pj8v7-1355320965 was unsuccessful: child 'artifact-passing-pj8v7-854535231' failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="Outbound nodes of artifact-passing-pj8v7-854535231 is [artifact-passing-pj8v7-854535231]" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="Outbound nodes of artifact-passing-pj8v7 is [artifact-passing-pj8v7-854535231]" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7 message: child 'artifact-passing-pj8v7-854535231' failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7 finished: 2021-08-31 14:49:17.532738572 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="Checking daemoned children of artifact-passing-pj8v7" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="Updated message  -> child 'artifact-passing-pj8v7-854535231' failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.532Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-pj8v7
time="2021-08-31T14:49:17.548Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660132831 workflow=artifact-passing-pj8v7
time="2021-08-31T14:53:20.830Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:20.830Z" level=info msg="Updated phase  -> Running" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:20.833Z" level=info msg="Steps node artifact-passing-5bs6q initialized Running" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:20.833Z" level=info msg="StepGroup node artifact-passing-5bs6q-2310228769 initialized Running" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:20.833Z" level=info msg="Pod node artifact-passing-5bs6q-1898326195 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:20.892Z" level=info msg="Created pod: artifact-passing-5bs6q[0].generate-artifact (artifact-passing-5bs6q-1898326195)" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:20.892Z" level=info msg="Workflow step group node artifact-passing-5bs6q-2310228769 not yet completed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:20.911Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660134823 workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:30.899Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:30.901Z" level=info msg="Workflow step group node artifact-passing-5bs6q-2310228769 not yet completed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:30.915Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660134912 workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:40.920Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:53:40.921Z" level=info msg="Workflow step group node artifact-passing-5bs6q-2310228769 not yet completed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.060Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.060Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-5bs6q-1898326195 templateName=whalesay workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.061Z" level=info msg="Step group node artifact-passing-5bs6q-2310228769 deemed failed: child 'artifact-passing-5bs6q-1898326195' failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q-2310228769 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q-2310228769 message: child 'artifact-passing-5bs6q-1898326195' failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q-2310228769 finished: 2021-08-31 14:54:07.062100084 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="step group artifact-passing-5bs6q-2310228769 was unsuccessful: child 'artifact-passing-5bs6q-1898326195' failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="Outbound nodes of artifact-passing-5bs6q-1898326195 is [artifact-passing-5bs6q-1898326195]" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="Outbound nodes of artifact-passing-5bs6q is [artifact-passing-5bs6q-1898326195]" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q message: child 'artifact-passing-5bs6q-1898326195' failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q finished: 2021-08-31 14:54:07.06231562 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="Checking daemoned children of artifact-passing-5bs6q" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="Updated message  -> child 'artifact-passing-5bs6q-1898326195' failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.062Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-5bs6q
time="2021-08-31T14:54:07.077Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660135225 workflow=artifact-passing-5bs6q

The workflow’s pods that are problematic:

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      cni.projectcalico.org/podIP: ""
      cni.projectcalico.org/podIPs: ""
      workflows.argoproj.io/node-message: timed out waiting for the condition
      workflows.argoproj.io/node-name: artifact-passing-5bs6q[0].generate-artifact
      workflows.argoproj.io/template: '{"name":"whalesay","arguments":{},"inputs":{},"outputs":{"artifacts":[{"name":"hello-art","path":"/tmp/hello_world.txt"}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["sleep
        1; cowsay hello world | tee /tmp/hello_world.txt"],"resources":{}},"archiveLocation":{"archiveLogs":false,"s3":{"endpoint":"minio.minio-ssd.svc.cluster.local:443","bucket":"test","insecure":false,"accessKeySecret":{"name":"minio-key","key":"access_key_id"},"secretKeySecret":{"name":"minio-secret","key":"secret_access_key"},"key":"artifact-passing-5bs6q/artifact-passing-5bs6q-1898326195"}}}'
    creationTimestamp: "2021-08-31T14:53:20Z"
    labels:
      workflows.argoproj.io/completed: "true"
      workflows.argoproj.io/workflow: artifact-passing-5bs6q
    name: artifact-passing-5bs6q-1898326195
    namespace: twin-pipelines
    ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Workflow
      name: artifact-passing-5bs6q
      uid: f28ff596-914e-4d97-b55f-3de083bd455d
    resourceVersion: "660135229"
    uid: 166a07d6-6c0f-4a52-965f-66d5e956eba4
  spec:
    containers:
    - command:
      - argoexec
      - wait
      env:
      - name: ARGO_POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      image: argoproj/argoexec:v2.12.5
      imagePullPolicy: IfNotPresent
      name: wait
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /argo/podmetadata
        name: podmetadata
      - mountPath: /var/run/docker.sock
        name: docker-sock
        readOnly: true
      - mountPath: /argo/secret/minio-secret
        name: minio-secret
        readOnly: true
      - mountPath: /argo/secret/minio-key
        name: minio-key
        readOnly: true
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: pipeline-runner-token-xmwtk
        readOnly: true
    - args:
      - sleep 1; cowsay hello world | tee /tmp/hello_world.txt
      command:
      - sh
      - -c
      image: docker/whalesay:latest
      imagePullPolicy: Always
      name: main
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: pipeline-runner-token-xmwtk
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: cuda01-p100
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Never
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: pipeline-runner
    serviceAccountName: pipeline-runner
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - downwardAPI:
        defaultMode: 420
        items:
        - fieldRef:
            apiVersion: v1
            fieldPath: metadata.annotations
          path: annotations
      name: podmetadata
    - hostPath:
        path: /var/run/docker.sock
        type: Socket
      name: docker-sock
    - name: minio-secret
      secret:
        defaultMode: 420
        items:
        - key: secret_access_key
          path: secret_access_key
        secretName: minio-secret
    - name: minio-key
      secret:
        defaultMode: 420
        items:
        - key: access_key_id
          path: access_key_id
        secretName: minio-key
    - name: pipeline-runner-token-xmwtk
      secret:
        defaultMode: 420
        secretName: pipeline-runner-token-xmwtk
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2021-08-31T14:53:20Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2021-08-31T14:53:20Z"
      message: 'containers with unready status: [wait main]'
      reason: ContainersNotReady
      status: "False"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2021-08-31T14:53:20Z"
      message: 'containers with unready status: [wait main]'
      reason: ContainersNotReady
      status: "False"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2021-08-31T14:53:20Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: docker://b09c5c6b844c40a348effe10b123d4ccdd5b2fba2f703701404d94d3aaa4d713
      image: docker/whalesay:latest
      imageID: docker-pullable://docker/whalesay@sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b
      lastState: {}
      name: main
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: docker://b09c5c6b844c40a348effe10b123d4ccdd5b2fba2f703701404d94d3aaa4d713
          exitCode: 0
          finishedAt: "2021-08-31T14:53:25Z"
          reason: Completed
          startedAt: "2021-08-31T14:53:24Z"
    - containerID: docker://8005ed619702ef75e17413e419ce0504c97d7f0ee43d96552213d7a22a90483d
      image: argoproj/argoexec:v2.12.5
      imageID: docker-pullable://argoproj/argoexec@sha256:1841ef4c44e582f66af82091fd15eeb493faddb359a54fb75087c993b5404e0b
      lastState: {}
      name: wait
      ready: false
      restartCount: 0
      started: false
      state:
        terminated:
          containerID: docker://8005ed619702ef75e17413e419ce0504c97d7f0ee43d96552213d7a22a90483d
          exitCode: 1
          finishedAt: "2021-08-31T14:53:57Z"
          message: timed out waiting for the condition
          reason: Error
          startedAt: "2021-08-31T14:53:22Z"
    hostIP: 172.18.60.241
    phase: Failed
    podIP: 10.42.6.16
    podIPs:
    - ip: 10.42.6.16
    qosClass: BestEffort
    startTime: "2021-08-31T14:53:20Z"
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Logs from in your workflow’s wait container:

time="2021-08-31T14:53:40.563Z" level=info msg="Saving from /tmp/argo/outputs/artifacts/hello-art.tgz to s3 (endpoint: minio.minio-ssd.svc.cluster.local:443, bucket: test, key: artifact-passing-5bs6q/artifact-passing-5bs6q-1898326195/hello-art.tgz)"
time="2021-08-31T14:53:40.576Z" level=warning msg="Failed to put file: Get https://minio.minio-ssd.svc.cluster.local/test/?location=: x509: certificate signed by unknown authority"
time="2021-08-31T14:53:57.022Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/hello-art.tgz, key: artifact-passing-5bs6q/artifact-passing-5bs6q-1898326195/hello-art.tgz"
time="2021-08-31T14:53:57.022Z" level=info msg="Creating minio client minio.minio-ssd.svc.cluster.local:443 using static credentials"
time="2021-08-31T14:53:57.022Z" level=info msg="Saving from /tmp/argo/outputs/artifacts/hello-art.tgz to s3 (endpoint: minio.minio-ssd.svc.cluster.local:443, bucket: test, key: artifact-passing-5bs6q/artifact-passing-5bs6q-1898326195/hello-art.tgz)"
time="2021-08-31T14:53:57.036Z" level=warning msg="Failed to put file: Get https://minio.minio-ssd.svc.cluster.local/test/?location=: x509: certificate signed by unknown authority"
time="2021-08-31T14:53:57.036Z" level=error msg="executor error: timed out waiting for the condition"
time="2021-08-31T14:53:57.036Z" level=info msg="Killing sidecars"
time="2021-08-31T14:53:57.041Z" level=info msg="Alloc=8616 TotalAlloc=24264 Sys=70848 NumGC=7 Goroutines=7"
time="2021-08-31T14:53:57.052Z" level=fatal msg="timed out waiting for the condition"

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 15
  • Comments: 34 (7 by maintainers)

Most upvoted comments

I just tested this environment variable inside the workflow-controller-configmap (Note that the variable name is SSL_CERT_FILE, not SSL_CERT_DIR - it failed with SSL_CERT_DIR) and it works!:

data:
  executor:
    env:
      - name: SSL_CERT_FILE
        value: /run/secrets/kubernetes.io/serviceaccount/ca.crt

Now I am getting the same error, but from the argo-server deployment logs, when I click on “…logs from the artifacts.”:

time="2021-12-14T20:25:15.079Z" level=info msg="Creating minio client minio.minio-tenant.svc.cluster.local:443 using static credentials"
time="2021-12-14T20:25:15.079Z" level=info msg="Getting from s3 (endpoint: minio.minio-tenant.svc.cluster.local:443, bucket: argo-workflow, key: pod-hppzw/pod-4165558653/main.log) to /tmp/artifact884264937"

In the browser it shows:

failed to get file: Get "https://minio.minio-tenant.svc.cluster.local/argo-workflow/?location=": x509: certificate signed by unknown authority

I’ll attempt to use the same environment variable in the argo-server container to see what happens. Will keep you posted.

+4
cotjoey on Dec 14, 2021

Similar to the issue mentioned by @edevash , we have external generated Certs and rootca that needs to be available at execution time and not at build time.

We are planning to use the following workaround done on top of workflow/controller/workflowpod.go,

  1. Add an additional mount volume(/usr/local/share/ca-certificates). This volume will be pointed to a secret file which contains external generated CA cert.
  2. Update both init and wait containers to run update-ca-certificates before executing argoexec [init or wait].

We have validated this in our setup and it seems to work. Though, not straightforward, this helps mount CA certs at execution time.

+3
rmathagiarun on Jun 27, 2022

This is what I did to fix the other issue (I am on Red Hat 8):

I inspected the Dockerfile of the argocli (https://hub.docker.com/r/argoproj/argocli/dockerfile) and found that it was picking up the certificates from /etc/ssl/certs in the container.

So I mounted a hostPath volume from the host (/etc/pki/tls/certs/ca-bundle.crt) to /etc/ssl/certs in the container.

Before that I had to add the Kubernetes CA to /etc/pki/tls/certs/ca-bundle.crt, the ones in /run/secrets/kubernetes.io/serviceaccount/ca.crt

The configuration looks like this in my argo-server deployment YAML:

  volumeMounts:
        - mountPath: /etc/ssl/certs/ca-bundle.crt
          name: host-file
          readOnly: true
volumes:
      - name: host-file
        hostPath:
          path: /etc/pki/tls/certs/ca-bundle.crt
          type: File
+3
cotjoey on Dec 17, 2021

Solution from https://github.com/argoproj/argo-workflows/issues/6635#issuecomment-1127604348 does not with newer versions. Starting on v3.4.0, the argoproj/argoexec image is based on gcr.io/distroless/static. Hence, running update-ca-certificates is not possible anymore.

Just in case it’s helpful for others, this is how we had to do it very recently:

ARG BASE_IMAGE=<acr-endpoint>/argoproj/argoexec:v3.4.9

FROM ${BASE_IMAGE}
COPY CA.cer /etc/ssl/certs/ca.crt
ENV SSL_CERT_FILE=/etc/ssl/certs/ca.crt

Our k8s clusters do not have internet access, thus using our own cert makes sense to us.

+1
cmontemuino on Aug 11, 2023

we have same issue in 3.4.2

+1
jinnjwu on Oct 31, 2022

Hi, it seems like this solution would fit in case the minio certificate has been signed by the kube CA, is that correct?

However if you’re luck (like me) and your S3 SSL cert has been signed by some other root ca, I don’t see any alternative to rebuilding your own version of the executor image as suggested in the previous comment: https://github.com/argoproj/argo-workflows/issues/6635#issuecomment-974487984

One possible feature that could avoid a rebuild would be to add more customization options to the executor container, such as additional volumeMounts to add custom certificates. Currently I don’t think this is supported.

EDIT: It looks like specifying custom resources on the executor is now supported: https://github.com/argoproj/argo-workflows/blob/92b3ef27af7a7e6b930045e95072a47c8745b1d3/docs/workflow-controller-configmap.yaml#L214-L237

+1
mxm-tr on May 9, 2022

One work around for this issue is to add an environment variable to the argoexec to point to kubernetes certificate authority which is injected into the container. This works because go looks for this environment variable to find SSL certs.

I did this by building a custom container for argoexec with this Dockerfile.

FROM quay.io/argoproj/argoexec:latest
ENV SSL_CERT_FILE=/run/secrets/kubernetes.io/serviceaccount/ca.crt

Then tagging this container and pushing it to our local repository. To use this, you can edit your worflow-controller deployment replacing the executor with the newly built executor.

...
spec:
  template:
    spec:
      containers:
      - args:
        - --configmap
        - workflow-controller-configmap
        - --executor-image
        - myspecial/argoexec
        - --namespaced
...

This will let Argo Workflows successfully pass artifacts back and forth with Minio (using the newer version of Minio Operator which disallows insecure connections) over TLS.

+1
jhart99 on Nov 19, 2021
Read more comments on GitHub
← argo-workflows: Artifact input file permission problems
argo-workflows: artifactGC not working with IRSA →