kubernetes: kubectl apply --prune --all fails to remove existing resources
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
I ran
kubectl apply --all --namespace default --prune -f base/ --recursive
It applied all my new yaml in base/ but did not remove existing resources from the cluster due to an error:
error: error pruning nonNamespaced object /v1, Kind=Namespace: namespaces “kube-system” is forbidden: this namespace may not be deleted
What you expected to happen:
I expected all the yaml in base/ to get applied to my cluster, and all other resources currently running on the cluster that were not represented in base/ to be removed.
From the error message it is unclear what resource preventing this from succeeding, and why it was included given I attempted to scope the command to the default namespace (not kube-system
).
How to reproduce it (as minimally and precisely as possible):
-
Start a clean cluster
-
Deploy nginx
kubectl run nginx --image=nginx
-
Create a simple service
# myservice.yaml apiVersion: v1 kind: Service metadata: name: myservice spec: ports: - name: http port: 80 targetPort: http selector: app: myservice type: ClusterIP
-
Deploy myservice with the intent to prune nginx
kubectl apply --all --namespace default --prune -f myservice.yaml
-
See unexpected error
error: error pruning nonNamespaced object /v1, Kind=Namespace: namespaces “kube-system” is forbidden: this namespace may not be deleted
Anything else we need to know?:
I really want a way to deploy a known set of configs to a cluster and delete everything else.
Environment:
- Kubernetes version (use
kubectl version
): Client Version: version.Info{Major:“1”, Minor:“11”, GitVersion:“v1.11.1”, GitCommit:“b1b29978270dc22fecc592ac55d903350454310a”, GitTreeState:“clean”, BuildDate:“2018-07-17T18:53:20Z”, GoVersion:“go1.10.3”, Compiler:“gc”, Platform:“darwin/amd64”} Server Version: version.Info{Major:“1”, Minor:“9+”, GitVersion:“v1.9.7-gke.3”, GitCommit:“9b5b719c5f295c99de68ffb5b63101b0e0175376”, GitTreeState:“clean”, BuildDate:“2018-05-31T18:32:23Z”, GoVersion:“go1.9.3b4”, Compiler:“gc”, Platform:“linux/amd64”} - Cloud provider or hardware configuration: Google Kubernetes Engine
- OS (e.g. from /etc/os-release): macOS
- Kernel (e.g.
uname -a
): Darwin Nicks-MacBook-Pro.local 17.6.0 Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64 x86_64
About this issue
- Original URL
- State: open
- Created 6 years ago
- Reactions: 19
- Comments: 45 (4 by maintainers)
I believe I read in some other issue that there are (still?) some plans to improve the whole apply/prune handling. However, for this specific issue, there is some kind of workaround by using prune-whitelist.
The command from the initial description
could be rewritten like this:
However, this approach has the following drawbacks:
So if you really need to make it work right now, I think this workaround could be OK. For the long run, hopefully the apply/prune mechanism will be improved to better handle cases like this.
seems I have the same problem. save this to a file named
a.yaml
And run
kubectl apply --prune --dry-run=client --all -n sandbox -f a.yaml
, the resultsI have no idea why it want to prune the
namespace/cattle-system
./remove-lifecycle stale
Unrelated to the original problem related to namespaces, but this is one of the first issues that comes up on Google when looking for why only certain resource types are pruned (because of the default whitelist) - so just in case it’s helpful for anyone else:
We came up with a one-liner to generate a
--prune-whitelist
argument list based on available resources in the cluster:kubectl api-resources -o wide > /tmp/api-resources; grep 'delete' /tmp/api-resources | cut -c$(grep -b -o APIVERSION /tmp/api-resources | awk -F ':' '{ print $1 }')- | awk '{ print $1"/"$3 }' | sed 's#^v1/#core/v1/#' | xargs -I {} echo "--prune-whitelist="{} | xargs
Short explanation:
grep
only those that are deletablecut
the columns untilAPIVERSION
(asSHORTNAMES
can be empty,grep
andawk
to find the column number)awk
togetherAPIVERSION
/KIND
sed
ifAPIVERSION
isv1
, make itcore/v1
xargs
generate list of arguments from linesIt’s not very performant and probably has some other issues, but for our use case it works fine.
It is probably because one of the resources in a.yaml is referencing
cattle-system
namespace. The namespace will be kept in track and used for pruning:https://github.com/kubernetes/kubernetes/blob/e22e9b4f836a401c61d1967300e853c9de0ffb36/staging/src/k8s.io/kubectl/pkg/cmd/apply/apply.go#L645
Alas, I’m out of ideas, sorry…