cilium: Cilium unable to remove taints on Azure AKS

I think I managed to get it working using https://github.com/kubemod/kubemod, here’s a brief overview how, to point anyone in the right direction:

1. While the mentioned project has a helm chart (https://github.com/kubemod/kubemod-helm) it is currently incomplete. It allows you to set tolerations, but it only does so for operator itself, author neglected to consider that cert generating Job also needs tolerations set. Since my starting point is AKS cluster with only system node pool, with CriticalAddonsOnly=true:NoSchedule taint set, I had to move to a different installation method. Luckily there are also Kubernetes manifests being provided in the kubemod repo (https://raw.githubusercontent.com/kubemod/kubemod/v0.15.0/bundle.yaml). I downloaded those and updated Job, CronJob and Deployment with this toleration:

tolerations:
- effect: NoSchedule
  key: CriticalAddonsOnly
  operator: Exists

1. If you take a look at MutatingWebhookConfiguration in the manifests bundle you will see that controller intercepts pretty much all k8s objects you might want to change. I did not need this so I removed everything and just left this:

operations:
- CREATE
resources:
- nodes

After a new node joins the cluster it will get a taint applied to it by the controller, after Cilium agent is installed it will remove the taint and pods can be scheduled on that node. There’s not much point in reapplying a taint on each UPDATE so I removed it from list of operations. 3) Finally you need a ModRule object that adds the correct taint, it looks something like this:

apiVersion: api.kubemod.io/v1beta1
kind: ModRule
metadata:
  name: add-cilium-taint
  namespace: kubemod-system
spec:
  type: Patch
  match:
    - select: '$.kind'
      matchValue: Node
    ## make this rule idempotent
    - select: '$.spec.taints[*].key'
      matchValue: node.cilium.io/agent-not-ready
      negate: true
  patch:
    - op: add
      path: /spec/taints/-1
      value: |-
        effect: NoSchedule
        key: node.cilium.io/agent-not-ready
        value: "true"

Apply that and check the logs of kubemod operator, it should look something like this: {"level":"info","ts":"2022-05-12 14:21:53.168Z","logger":"modrule-webhook","msg":"Applying ModRule patch","request uid":"06a40d51-141e-4cdc-9f65-dacc16837958" ,"namespace":"","resource":"nodes/aks-d16asv5-36614584-vmss000000","operation":"CREATE","patch":[{"op":"add","path":"/spec/taints/2","value":{"effect":"NoSchedule","key":"node.cilium.io/agent-not-ready","value":"true"}}]}