argo-workflows: ArchiveLogs not working when upgrading to v3

Summary

Archiving logs in GCS bucket broke by just upgrading version from 2.12.11 to 3.0.0/3.0.1

Diagnostics

What Kubernetes provider are you using? GKE

What version of Argo Workflows are you running? 3.0.0/3.0.1

By just upgrading version I can confirm that even with archiveLogs enabled, workflow-controller does not store logs in GCS anymore.

This is my config:

apiVersion: v1
kind: ConfigMap
data:
  config: |
    containerRuntimeExecutor: k8sapi
    artifactRepository:
      archiveLogs: true
      gcs:
        bucket: <redacted>
        keyFormat: '{{workflow.namespace}}/{{workflow.name}}/{{pod.name}}'
        serviceAccountKeySecret:
          key: <redacted>
          name: <redacted>

I have tried both versions with and without config key, in both cases I can see my controller reading the config:

time="2021-04-09T13:29:39.779Z" level=info msg="Configuration:\nartifactRepository:\n  archiveLogs: true\n  gcs:\n <truncated>

I am unsure what logs are necessary to debug this, but I am available to run tests and collect logs upon request.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.