kyverno: [BUG] Openshift DeploymentConfig resources being stripped of fields

Software version numbers

  • Kubernetes version: OCP 4.6.21 (based on Kubernetes v1.19.0)
  • Kyverno version: 1.3.4

Describe the bug DeploymentConfig resources in Openshift have many fields removed. DeploymentConfig resources are very similar to Deployments, but with a few extra features (more info here).

When using a policy that behaves correctly with Deployment, StatefulSet or DaemonSet, the DeploymentConfig has many fields removed.

To Reproduce Steps to reproduce the behavior:

  1. Apply policy to DeploymentConfig kinds
  2. Create a DeploymentConfig
  3. DeploymentConfig resource rejected by Openshift for not having necessary fields

Expected behavior Additional fields added that match the policy, similar to Deployment, StatefulSet or DaemonSet resources.

Additional context

The policy in question is: -

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: set-service-labels
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
spec:
  background: false
  rules:
    - name: set-service-labels-pods
      match:
        resources:
          kinds:
          - Pod
      exclude:
        resources:
          namespaces:
            - "kube*"
            - "openshift*"
            - "kube-*"
            - "openshift-*"
      preconditions:
        any:
          - key: "{{ request.operation }}"
            operator: Equals
            value: "CREATE"
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              +(service): "{{ request.object.metadata.labels.app }}"
          spec:
            containers:
              - (name): "*"
                env:
                  - name: "SERVICE"
                    value: "{{ request.object.metadata.labels.app }}"
    - name: set-service-labels-deployments-and-sets
      match:
        resources:
          kinds:
          - Deployment
          - DeploymentConfig
          - DaemonSet
          - StatefulSet
      exclude:
        resources:
          namespaces:
            - "kube*"
            - "openshift*"
            - "kube-*"
            - "openshift-*"
      preconditions:
        any:
          - key: "{{ request.operation }}"
            operator: Equals
            value: "CREATE"
      mutate:
        patchStrategicMerge:
          spec:
            template:
              metadata:
                labels:
                  +(service): "{{ request.object.spec.template.metadata.labels.app }}"
              spec:
                containers:
                  - (name): "*"
                    env:
                      - name: "SERVICE"
                        value: "{{ request.object.spec.template.metadata.labels.app }}"

Supplying a standard Deployment, I see the following: -

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example
  namespace: default
spec:
  selector:
    matchLabels:
      app: hello-openshift
  replicas: 3
  template:
    metadata:
      labels:
        app: hello-openshift
    spec:
      containers:
        - name: hello-openshift
           image: openshift/hello-openshift
           ports:
             - containerPort: 8080

Outcome (output snipped): -

kind: Pod
apiVersion: v1
[...]
metadata:
  labels:
    app: hello-openshift
    service: hello-openshift
spec:
  containers:
    - name: hello-openshift
      image: openshift/hello-openshift
      ports:
       - containerPort: 8080
      env:
       - name: "SERVICE"
         value: "hello-openshift"

When supplying a DeploymentConfig (notice the similarity to a Deployment): -

deployment_config.yaml

apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
  name: example
  namespace: default
spec:
  selector:
    app: hello-openshift
  replicas: 3
  template:
    metadata:
      labels:
        app: hello-openshift
    spec:
      containers:
        - name: hello-openshift
           image: openshift/hello-openshift
           ports:
             - containerPort: 8080

The response is below: -

$ oc apply -f deployment_config.yaml
The DeploymentConfig "example" is invalid: spec.template.spec.containers[0].image: Required value

Debug Logging

I turned on debug logging, and found the following, which is very worrying: -

I0422 12:12:06.371948       1 strategicMergePatch.go:84] EngineMutate "msg"="generating JSON patches from patched resource" "kind"="DeploymentConfig" "name"="example" "namespace"="default" "policy"="set-service-labels" "rule"="set-service-labels-deployments-and-sets" "patchedResource"={"apiVersion":"apps.openshift.io/v1","kind":"DeploymentConfig","metadata":{"managedFields":[{"apiVersion":"apps.openshift.io/v1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:replicas":{},"f:selector":{".":{},"f:app":{}},"f:strategy":{"f:activeDeadlineSeconds":{},"f:rollingParams":{".":{},"f:intervalSeconds":{},"f:maxSurge":{},"f:maxUnavailable":{},"f:timeoutSeconds":{},"f:updatePeriodSeconds":{}},"f:type":{}},"f:template":{".":{},"f:metadata":{".":{},"f:creationTimestamp":{},"f:labels":{".":{},"f:app":{}}},"f:spec":{".":{},"f:containers":{".":{},"k:{\"name\":\"hello-openshift\"}":{".":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8080,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:protocol":{}}},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}},"f:triggers":{}}},"manager":"Mozilla","operation":"Update","time":"2021-04-22T12:12:03Z"}],"name":"example","namespace":"default"},"spec":{"replicas":3,"selector":{"app":"hello-openshift"},"strategy":{"activeDeadlineSeconds":21600,"resources":{},"rollingParams":{"intervalSeconds":1,"maxSurge":"25%","maxUnavailable":"25%","timeoutSeconds":600,"updatePeriodSeconds":1},"type":"Rolling"},"template":{"metadata":{"labels":{"app":"hello-openshift","tags.datadoghq.com/service":"hello-openshift"}},"spec":{"containers":[{"env":[{"name":"SERVICE","value":"hello-openshift"}],"name":"hello-openshift"}],"dnsPolicy":"ClusterFirst","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"terminationGracePeriodSeconds":30}},"test":false,"triggers":[{"type":"ConfigChange"}]},"status":{"availableReplicas":0,"latestVersion":0,"observedGeneration":0,"replicas":0,"unavailableReplicas":0,"updatedReplicas":0}}
I0422 12:12:06.372306       1 strategicMergePatch.go:96] EngineMutate "msg"="generated patch" "kind"="DeploymentConfig" "name"="example" "namespace"="default" "policy"="set-service-labels" "rule"="set-service-labels-deployments-and-sets" "patch"="{\"op\":\"add\",\"path\":\"/spec/template/spec/containers/0/env\",\"value\":[{\"name\":\"SERVICE\",\"value\":\"hello-openshift\"}]}"
I0422 12:12:06.372348       1 strategicMergePatch.go:96] EngineMutate "msg"="generated patch" "kind"="DeploymentConfig" "name"="example" "namespace"="default" "policy"="set-service-labels" "rule"="set-service-labels-deployments-and-sets" "patch"="{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/0/terminationMessagePolicy\"}"
I0422 12:12:06.372380       1 strategicMergePatch.go:96] EngineMutate "msg"="generated patch" "kind"="DeploymentConfig" "name"="example" "namespace"="default" "policy"="set-service-labels" "rule"="set-service-labels-deployments-and-sets" "patch"="{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/0/image\"}"
I0422 12:12:06.372411       1 strategicMergePatch.go:96] EngineMutate "msg"="generated patch" "kind"="DeploymentConfig" "name"="example" "namespace"="default" "policy"="set-service-labels" "rule"="set-service-labels-deployments-and-sets" "patch"="{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/0/imagePullPolicy\"}"
I0422 12:12:06.372445       1 strategicMergePatch.go:96] EngineMutate "msg"="generated patch" "kind"="DeploymentConfig" "name"="example" "namespace"="default" "policy"="set-service-labels" "rule"="set-service-labels-deployments-and-sets" "patch"="{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/0/ports\"}"
I0422 12:12:06.372478       1 strategicMergePatch.go:96] EngineMutate "msg"="generated patch" "kind"="DeploymentConfig" "name"="example" "namespace"="default" "policy"="set-service-labels" "rule"="set-service-labels-deployments-and-sets" "patch"="{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/0/resources\"}"
I0422 12:12:06.372508       1 strategicMergePatch.go:96] EngineMutate "msg"="generated patch" "kind"="DeploymentConfig" "name"="example" "namespace"="default" "policy"="set-service-labels" "rule"="set-service-labels-deployments-and-sets" "patch"="{\"op\":\"remove\",\"path\":\"/spec/template/spec/containers/0/terminationMessagePath\"}"
I0422 12:12:06.372554       1 strategicMergePatch.go:96] EngineMutate "msg"="generated patch" "kind"="DeploymentConfig" "name"="example" "namespace"="default" "policy"="set-service-labels" "rule"="set-service-labels-deployments-and-sets" "patch"="{\"op\":\"add\",\"path\":\"/spec/template/metadata/labels/tags.datadoghq.com~1service\",\"value\":\"hello-openshift\"}"

This implies that nearly all of the fields are being removed, specifically: -

			"spec": {
				"containers": [{
					"env": [{
						"name": "SERVICE",
						"value": "hello-openshift"
					}],
					"name": "hello-openshift"
				}],

Do you know why a DeploymentConfig would be treated differently to a Deployment?

I inspected the AdmissionRequest coming in from the DeploymentConfig (using my own tool) and it appears to be no different than a standard Deployment request. Let me know if you would like the output from that too (it is the full JSON admission request object)

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 20 (10 by maintainers)

Most upvoted comments

Latest update - I have a version of this working!

The below works as expected: -

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: set-service-labels
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
    description: "Get the value from the app label, create the service tag, and expose it to the container as SERVICE"
spec:
  background: false
  rules:
    - name: set-service-labels-pods
      match:
        resources:
          kinds:
          - Pod
      exclude:
        resources:
          namespaces:
            - "kube*"
            - "openshift*"
            - "voltron*"
            - "kube-*"
            - "openshift-*"
            - "voltron-*"
      preconditions:
        all:
          - key: "{{ request.operation }}"
            operator: Equals
            value: "CREATE"
          - key: "SERVICE"
            operator: NotIn
            value: "{{ request.object.spec.containers[].env[].name }}"
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              +(service): "{{ request.object.metadata.labels.app }}"
          spec:
            containers:
              - (name): "*"
                env:
                  - name: "SERVICE"
                    value: "{{ request.object.metadata.labels.app }}"
    - name: set-service-labels-deployments-and-sets
      match:
        resources:
          kinds:
          - Deployment
          - DaemonSet
          - StatefulSet
      exclude:
        resources:
          namespaces:
            - "kube*"
            - "openshift*"
            - "voltron*"
            - "kube-*"
            - "openshift-*"
            - "voltron-*"
      preconditions:
        all:
          - key: "{{ request.operation }}"
            operator: Equals
            value: "CREATE"
          - key: "SERVICE"
            operator: NotIn
            value: "{{ request.object.spec.template.spec.containers[].env[].name }}"
      mutate:
        patchStrategicMerge:
          spec:
            template:
              metadata:
                labels:
                  +(service): "{{ request.object.spec.template.metadata.labels.app }}"
              spec:
                containers:
                  - (name): "*"
                    env:
                      - name: "SERVICE"
                        value: "{{ request.object.spec.template.metadata.labels.app }}"
    - name: set-service-labels-deploymentconfig
      match:
        resources:
          kinds:
          - DeploymentConfig
      exclude:
        resources:
          namespaces:
            - "kube*"
            - "openshift*"
            - "kube-*"
            - "openshift-*"
      preconditions:
        all:
          - key: "{{ request.operation }}"
            operator: Equals
            value: "CREATE"
          - key: "SERVICE"
            operator: NotIn
            value: "{{ request.object.spec.template.spec.containers[].env[].name }}"
      mutate:
        patchesJson6902: |-
         - op: add
           path: /spec/template/metadata/labels/service
           value: "{{ request.object.spec.template.metadata.labels.app }}"
         - op: add
           path: /spec/template/spec/containers/0/env/-1
           value: {"name": "SERVICE","value": {{ request.object.spec.template.metadata.labels.app }} }

This seems to do what we need.

I suppose the limitation we now have is that the DeploymentConfig rule only takes effect on the first container. This will be fine in a lot of cases, but it won’t always be the case.

Is there a way we can apply this to all container definitions, rather than just the first?

+1
stuh84 on May 4, 2021
Read more comments on GitHub
← kyverno: [BUG] Mutating `image` with custom JMESPath function `regex_replace_all` overwrites all instances
kyverno: [BUG] OwnerRefInvalidNamespace warnings on webhook configurations →