kyverno: Delete generated resource when triggering resource is deleted

Adding my 2 cents here as well 😄 following the conversation with @chipzoller & @JimBugwadia on Slack (use case described here: https://kubernetes.slack.com/archives/CLGR9BJU9/p1627952458037000 the long an very interesting discussion followed here: https://kubernetes.slack.com/archives/CLGR9BJU9/p1627952499037900)

We manage several on-prem Kubernetes clusters and expose LoadBalancer Services using MetalLB, to protect the hosts we use Calico GlobalNetworkPolicies.

Our goals:

1. When a LoadBalancer type Service is provisioned and is allocated a Public/External ingress IP a GlobalNetworkPolicy resource should be generated to allow access from outside the cluster to the assigned LoadBalancer Ingress IP and Service ports.
2. When the LoadBalancer Service is updated, the GlobalNetworkPolicy should be updated to reflect the change
3. When the LoadBalancer Service is deleted, the associated GlobalNetworkPolicy should be deleted as well

Kyverno is able to perfectly implement (1) & (2) above, however (3) turned out to be more challenging and after several attempts and great ideas from the team here was deemed impossible.

I think a similar requirement for managing the full life-cycle of generated resources (i.e. including their eventual deletion/garbage collection) is going to prove to be useful in many use cases.

Hope this helps clarify what we’re trying to achieve, happy to provide more details.

---

I figured I’d also share the generate policy I managed to cook up with the help of the awesome folks of this project in case someone encounters a similar need:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-prednat-gnp-for-lb-svc
  annotations:
    policies.kyverno.io/title: Pre DNAT Global Network Policy for LoadBalancer Service
    policies.kyverno.io/category: Network Security
    policies.kyverno.io/subject: GlobalNetworkPolicy
    policies.kyverno.io/description: >-
      By default, Ingress communication to the External IP of a LoadBalancer type
      service is denied, this policy will generate a Calico Global Network Policy
      to allow communication to the assigned LoadBalancer IP and ports specified
      in the Service resource. 
spec:
  background: true
  validationFailureAction: audit
  rules:
  - name: lb-prednat-gnp
    match:
      resources:
        kinds:
        - Service
    preconditions:
      all:
      - key: "{{request.object.spec.type}}"
        operator: Equals
        value: LoadBalancer
      - key: "{{request.object.status.loadBalancer.ingress[0].ip}}"
        operator: NotEquals
        value: ""
    generate:
      kind: GlobalNetworkPolicy
      apiVersion: crd.projectcalico.org/v1
      name: "lb-prednat-{{request.object.metadata.namespace}}-{{request.object.metadata.name}}"
      synchronize: true
      data:
        spec:
          order: 1
          selector: nodepool == 'workloads'
          preDNAT: true
          applyOnForward: true
          ingress:
          - action: Allow
            protocol: "{{request.object.spec.ports[0].protocol}}"
            destination:
              nets:
              - "{{request.object.status.loadBalancer.ingress[0].ip}}"
              ports: "{{request.object.spec.ports[*].port}}"

I intend to expand it to support loadBalancerSourceRanges to dynamically define the source block in the GlobalNetworkPolicy, will update here when it’s working.

Regardless if this gets implemented or not (though 🤞🏼), Kyverno is great tool and the team behind it is amazing and super helpful! so thanks for all your hard work, cheers! ❤️

We are discussing how to reintroduce this behavior in possibly a separate field. No commitments or proposals as of yet.

Hi @dkulchinsky, we are planning this in coming release, we had discussed about this feature with team internally, have to come up with the proposal.

@dkulchinsky not started to look into this yet, engaged in other issues, i will discuss with team and come up with a proposal soon.

@prateekpandey14 the other issue with ownerReferences is that it only works for resources that are namespace scoped and in the same namespace

We have a use case where we generate a cluster-scoped resource from a namespaced resource