argo-workflows: ClusterWorkflowTemplate podMetadata are ignored with `workflowTemplateRef `

Summary

I expected that creating a Workflow from a ClusterWorkflowTemplate, the resulting pods would inherit labels from podMetadata. Those labels are mainly used to match specific NetworkPolicies granting access to internet and other services.

Argo Workflows version: 3.1.3

Diagnostics

Either a workflow that reproduces the bug, or paste you whole workflow YAML, including status, something like:

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: go-pipeline
spec:
  serviceAccountName: cicd-pipeline
  activeDeadlineSeconds: 3600
  ttlStrategy:
    secondsAfterCompletion: 604800
  podMetadata:
    labels:
      domain: ci
      component: ci-executor-pod
      provider: http
...

Kubernetes provider: GKE

executor: PNS

# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep go-pipeline-***

time="2021-10-28T16:17:01.700Z" level=info msg="Processing workflow" namespace=cicd-pipelines workflow=go-pipeline-59vhk
time="2021-10-28T16:17:01.737Z" level=info msg="Updated phase  -> Running" namespace=cicd-pipelines workflow=go-pipeline-59vhk
time="2021-10-28T16:17:01.737Z" level=info msg="Creating pvc go-pipeline-59vhk-workdir" namespace=cicd-pipelines workflow=go-pipeline-59vhk
time="2021-10-28T16:17:01.746Z" level=info msg="DAG node go-pipeline-59vhk initialized Running" namespace=cicd-pipelines workflow=go-pipeline-59vhk
time="2021-10-28T16:17:01.747Z" level=info msg="All of node go-pipeline-59vhk.clone-code dependencies [] completed" namespace=cicd-pipelines workflow=go-pipeline-59vhk
time="2021-10-28T16:17:01.747Z" level=info msg="Pod node go-pipeline-59vhk-2560901167 initialized Pending" namespace=cicd-pipelines workflow=go-pipeline-59vhk
time="2021-10-28T16:17:01.781Z" level=info msg="Created pod: go-pipeline-59vhk.clone-code (go-pipeline-59vhk-2560901167)" namespace=cicd-pipelines workflow=go-pipeline-59vhk
time="2021-10-28T16:17:01.860Z" level=info msg="Workflow update successful" namespace=cicd-pipelines phase=Running resourceVersion=91309602 workflow=go-pipeline-59vhk
time="2021-10-28T16:18:01.778Z" level=info msg="Processing workflow" namespace=cicd-pipelines workflow=go-pipeline-59vhk
time="2021-10-28T16:18:01.779Z" level=info msg="Updating node go-pipeline-59vhk-2560901167 status Pending -> Running" namespace=cicd-pipelines workflow=go-pipeline-59vhk
time="2021-10-28T16:18:01.795Z" level=info msg="Workflow update successful" namespace=cicd-pipelines phase=Running resourceVersion=91310073 workflow=go-pipeline-59vhk


kubectl get pods --show-labels

NAME                                                     READY   STATUS      RESTARTS   AGE   LABELS
go-pipeline-59vhk-2641655615                             0/2     Pending     0          4s    workflows.argoproj.io/completed=false,workflows.argoproj.io/workflow=go-pipeline-59vhk

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

About this issue

Original URL
State: closed
Created 3 years ago
Reactions: 4
Comments: 27 (10 by maintainers)

Commits related to this issue

[argo] Add k8s labels to tempalte.metadata instead of spec.podMetadata Argo does not propagate the spec.podLabels to the pods. Only the sections under templates from the WorkflowTemplate are propagat... — committed to jhasm/metaflow by jhasm 8 months ago

Most upvoted comments

@sarabala1979 I have the same issue with WorkflowTemplate using the latest version v3.2.3 , I created a workflowTemplate with labels in podMetadata, this labels should be applied to the workflow pod. But they were ignored.

Diagnostics

Here is a WorkflowTemplate and a Workflow to reproduce the bug.

WorkflowTemplate

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate                  
metadata:
  name: hello-world-template    
  namespace: argo
spec:
  entrypoint: main
  podMetadata:
    labels:
        app: whalesay
        tier: demo          
  templates:
  - name: main              
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["hello world"]
      resources:                
        limits:
          memory: 32Mi
          cpu: 100m

Workflow

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
  namespace: argo
spec:
  entrypoint: main
  templates:
  - name: main
    steps:
    - - name: main
        templateRef:
            name: hello-world-template
            template: main

What Kubernetes provider are you using? I’m running Argo on a Gke cluster with following version of k8s : Kubernetes v1.21.5

What executor are you running? Emissary

# description of the created pod  on k8s
kubectl -n argo describe pod hello-world-rwrdj-3091738535
Name:         hello-world-rwrdj-3091738535
Namespace:    argo
Priority:     0
Node:         gke-main-cluster-default-202103121115-5d220df4-qdja/10.100.0.13
Start Time:   Tue, 09 Nov 2021 14:59:45 +0100
Labels:       workflows.argoproj.io/completed=false
              workflows.argoproj.io/workflow=hello-world-rwrdj
Annotations:  cni.projectcalico.org/containerID: a87c79ed09509eb5e5007c0a1b0188b394d6b64a0845a06068bf99ebc801a927
              cni.projectcalico.org/podIP: 10.92.9.174/32
              cni.projectcalico.org/podIPs: 10.92.9.174/32
              workflows.argoproj.io/node-id: hello-world-rwrdj-3091738535
              workflows.argoproj.io/node-name: hello-world-rwrdj[0].main
...
...

I was expecting the following labels to be added to the pod labels

    labels:
        app: whalesay
        tier: demo

amaloun on Nov 9, 2021