kubernetes: PodDisruptionBudget updates are forbidden

https://github.com/kubernetes/kubernetes/pull/69867 allows this update, and will be included in 1.15

Either way, willPodDisruptionBudget be updatable?

FYI, I get the same error even if the update changes nothing. How do I update my kubernetes cluster in a CD pipeline with a PodDisruptionBudget in it? Normally, I can do kubectl apply -f somefile.yml, and it intelligently handles services, deployments, statefulsets, etc. But if there is a disruption budget in there, boom.

@kow3ns After reading this thread, I’m convinced we need an answer for mutating PDBs through kubectl apply (and thus Helm). For things like Pods, it’s ok that you can’t mutate them because we have things like Deployment that allow you to declaratively transition to a new Pod template. There’s no such thing for PDB, so we are essentially preventing users from managing PDBs declaratively. They would have to imperatively delete the old one when they make any change.

Operationally you have a fair point that updating maxUnavailable should be uncommon, but making it immutable results in poor UX for the times when it’s necessary (e.g. even if there’s only one value that makes sense, maybe you used the wrong value by mistake and need to fix it). @foxish seems to think there’s no semantic reason we can’t make PDB mutable, but even if there were some reason that all changes needed to occur by deleting/recreating the PDB, then I’d argue we need to provide a “PDB Deployment” controller that contains a mutable PDB template and does the delete/recreate for you.

Our CI/CD pipeline is using helm and we hit this issue as well.

Error: UPGRADE FAILED: PodDisruptionBudget.policy "my-release" is invalid: spec: Forbidden: updates to poddisruptionbudget spec are forbidden.

I don’t know how to avoid this problem other than removing the PodDisruptionBudget object.

EDIT: As a workaround (using Helm) I can enclose the object definition like this:

{{- if .Release.IsInstall -}}
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
...
{{- end -}}

I have a solution that seems to be working well for me:

Similar to the {{- if .Release.IsInstall -}} workaround linked before, but with the ability to change on subsequent upgrades.

If you leverage helm hooks to create the Pod Disruption Budget, you can delete and recreate it on every helm install/upgrade.

Just add these annotations to the PDB like below in your chart:

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: {{ template "app.name" . }}
  release: {{ .Release.Name }}
  annotations:
    helm.sh/hook: post-upgrade, post-install
    "helm.sh/hook-delete-policy": "before-hook-creation"
spec:
  minAvailable: {{ default (sub .Values.replicaCount 1 ) .Values.PodDisruptionBudget.minAvailable }}

Pinging @kubernetes/sig-apps-feature-requests please, we’re hacking on top of helm charts, creating separate imperative workflows and hooks instead of declarative kubernetes resource management, just to remove and recreating PDBs every time. Any action on this would be very helpful for consumers please…

For those having this issue with helm, you can use the --force argument during an upgrade to force a deletion/re-creation of the resource if apply fails. It isn’t specific to PDB’s, though.

https://github.com/helm/helm/issues/2914

This breaks upgrades for lots of helm charts that only have minAvailable, eg:

https://github.com/helm/charts/blob/master/stable/nginx-ingress/templates/controller-poddisruptionbudget.yaml

kubernetes version v1.9.8

Error: UPGRADE FAILED: PodDisruptionBudget.policy "nginx-ingress" is invalid: spec: Forbidden: updates to poddisruptionbudget spec are forbidden.

I agree, it would be really nice to be able to apply these files after creation for CD workflows.

On the linked thread it seems like here was chosen as the place to continue discussion on this topic. Are there any further updates to this discussion elsewhere, or any issues implementing changes?

To add a further discussion point - is there any current reason that PDBs should not be mutable?

@foxish’s comment:

It appears that there was a rationale behind immutability of the spec initially when the plan included a time component as well (allowing N evictions over M hours). There isn’t a reason that maxUnav/minAv shouldn’t be mutable that I can think of.

Seems to imply that this was made immutable for a potential feature that does not yet exist. Whereas the current immutable implementation raises very real issues with kubectl apply and helm usage now as noted in this thread. I’m perhaps over simplifying but I believe these should be considered bugs in the implementation without a controller as @enisoc mentions. That said is it not better to make this a mutable resource rather than work to add another resource/controller. This would somewhat depend on if the other feature has been planned or dropped?

While I agree this restriction is tedious for helm charts, the fact that helm does not detect that the change will fail is also a problem: rather than rejecting the upgrade, helm installs the upgrade, only to have it fail, and then forces you to roll it back. If there are going to be immutable fields, then there should be a way for helm to determine that those fields are immutable prior to actually starting the upgrade. Ideally, a helm diff would even flag these.