kubernetes: Federated deployments cannot be deleted

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug /sig cli /sig federation @kubernetes/sig-federation

What happened: I have a federated cluster composed of one host cluster and one joined cluster running in AWS. They were created following the steps in the documentation, and I am able to see via the kubernetes dashboard that all components are running correctly.

To test the federated setup, I used a simple nginx deployment also from the docs, stored in deployment.proxy.yml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: proxy
spec:
  replicas: 5 # tells deployment to run 2 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

I then execute kubectl create --context=federation -f deployment.proxy.yml. Assuming I don’t change apiVersion, the deployment runs fine and almost immediately reaches the desired count (if I use apiVersion: apps/v1beta2 or any other API version than extensions/ with my federated setup, I get errors and it won’t start). I then go to tear down the deployment by executing kubectl delete deployment --context=federation proxy.

However, kubectl reports error: timed out waiting for the condition. The deployment instances spin down to zero, but the deployment won’t disappear:

NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
proxy     0         0         0            0           1h

I can delete them via the dashboard, or by switching to joined cluster’s context and using kubctl delete deployment proxy, but even after kubectll reports they are deleted successfully, they reappear and still cannot be deleted from the federation context.

What you expected to happen: The deployment to be deleted. This works fine if I’m not using my federated context.

How to reproduce it (as minimally and precisely as possible): Create a federated setup via the docs, launch the above job, and then try to delete it.

Environment:

  • Kubernetes version:
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1",
  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release): Ubuntu 16.04

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Reactions: 2
  • Comments: 16 (5 by maintainers)

Most upvoted comments

This is because that moving federation controllers to sync controller failed to update Status.ObservedGeneration (e.g. https://github.com/kubernetes/kubernetes/blob/release-1.8/federation/pkg/federatedtypes/deployment.go#L55) which is used by kubectl for it to proceed the next step. This bug applies to ReplicaSet, too

Bug still persists on k8s 1.9.0, On-Premise. I am able to delete the deployment when executing the following command(s):

kubectl delete deploy [name] --cascade=false --context federation
kubectl delete deploy [name] --context [cluster]

Only deleting the deployment on the [cluster] does not work as the federation controller automatically recreates the deployment. That’s why it works when you first delete the deployment on the federation with --cascade=false