kubernetes: delete ingress object hangs

What happened:

delete ingress object hangs even with --force and --grace-period=0

$ k delete ingress ingress-2048 -n game-2048  --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
ingress.extensions "ingress-2048" force deleted
.........

no events in game-2048 namespace

$ k get event -n game-2048
No resources found in game-2048 namespace.

What you expected to happen: deleted

How to reproduce it (as minimally and precisely as possible): it might be hard to reproduce

Anything else we need to know?:

on a eks cluster, the ingress is created from this doc, after delete aws load balancer controller, i can’t delete ingress object

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T18:49:28Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-a84824", GitCommit:"a84824a1c7b79853d0c57143ea42c9de1788dbba", GitTreeState:"clean", BuildDate:"2020-08-05T01:08:51Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 5
  • Comments: 21 (4 by maintainers)

Most upvoted comments

yes, there is a finalizer, after read some docs and know what exactly finalizer is.

λ  k get ingress ingress-2048 -n game-2048 -o yaml | grep -A 5 finalizers
  finalizers:
  - ingress.k8s.aws/resources
  generation: 4
  name: ingress-2048
  namespace: game-2048
  resourceVersion: "43226"

i remember i deleted the alb resource in aws management console before, so it always failed. after kubectl edit and remove finalizers, i can delete it

so i’ll close it.

thanks for your help !

Use following command to patch and delete the ingress-2048

kubectl patch ingress $Ingressname -n $namespace -p ‘{“metadata”:{“finalizers”:[]}}’ --type=merge

Use following command to patch and delete the ingress-2048

kubectl patch ingress $Ingressname -n $namespace > -p ‘{“metadata”:{“finalizers”:[]}}’ --type=merge

The correct command is without stdout sign:

kubectl patch ingress $Ingressname -n $namespace -p '{"metadata":{"finalizers":[]}}' --type=merge

Does the load balancer add a finalizer to the ingress object? (you can check with kubectl get ingress $name -o yaml). If so, the object will not be removed until that finalizer is removed.

You can make delete return without waiting by adding -wait false, but for the object to actually be deleted the finalizer must be removed.

@jmmen1 try

  1. delete the ns
  2. Delete the ing object
  3. apply the controller again
  4. Wait for a few seconds/minutes
  5. check if ing object has been removed or not.

It fails on the following error because of admission controller Error from server (InternalError): Internal error occurred: failed calling webhook "vingress.elbv2.k8s.aws": the server could not find the requested resource

run: kubectl delete ValidatingWebhookConfiguration aws-load-balancer-webhook

and then try to delete the ingress.

HI, I am I am facing the same issue. The ALB was deleted, the ingress object is still alive:

$ kubectl patch ingress argocd-server -n argocd -p '{"metadata":{"finalizers":[]}}' --type=merge

Here the output of the above command:

Error from server (InternalError): Internal error occurred: failed calling webhook "vingress.elbv2.k8s.aws": the server could not find the requested resource

The Logs:

{"level":"debug","ts":44444444,"logger":"secrets-manager","msg":"Monitoring secrets","groupID":"argocd/argocd-server","secrets":null} {"level":"error","ts":44444444,"logger":"controller-runtime.manager.controller.ingress","msg":"Reconciler error","name":"argocd-server","namespace":"argocd","error":"Internal error occurred: failed calling webhook \"vingress.elbv2.k8s.aws\": the server could not find the requested resource"}

Someone, can give any tips ?

kubectl delete ValidatingWebhookConfiguration aws-load-balancer-webhook

tnx. This is life saver.

Hello, I faced the same issue, deleting the below sections from the ingress resource made me able to delete it. Make sure that aws-load-balancer-controller and aws-webhook installed successfully in the cluster

  finalizers:
  - ingress.k8s.aws/resources
-------------------------------
        f:metadata:
        f:finalizers:
          .: {}
          v:"ingress.k8s.aws/resources": {}
kubectl patch ingress $Ingressname -n $namespace -p '{"metadata":{"finalizers":[]}}' --type=merge

This worked for me too… I was struggling with this dangling ingress… Thanks…!!

Use following command to patch and delete the ingress-2048 kubectl patch ingress $Ingressname -n $namespace > -p ‘{“metadata”:{“finalizers”:[]}}’ --type=merge

The correct command is without stdout sign:

kubectl patch ingress $Ingressname -n $namespace -p '{"metadata":{"finalizers":[]}}' --type=merge

thank u so much!!!

I have also ended up in this situation, using the aws-load-balancer-controller. Sadly, deleting and then recreating the controller still causes kubectl to hang when deleting the ingress.

Anyone else have more ideas for me to try?

update: I was able to delete the resource, by reinstalling alb-ingress-controller again and then my k8 controller was able to successfully remove the ingress object. then I removed the alb-ingress-controlled again. 😄

I used:

kubectl edit ingress my-ingress

and then I removed the finalizers but still the ingress is not deleted. This is in AWS EKS.

...
 managedFields:
  - apiVersion: networking.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:loadBalancer:
          f:ingress: {}
    manager: controller
    operation: Update
...

Do I need to do anything else for the changes to be applied?