kubernetes-ingress-controller: Upstream IP's not updated

Summary

For some reasons Kong Ingress controller has stopped updating the upstream IP’s when we for example deploy with kubectl patch. For new pods it doesn’t even create the upstream it seems.

Kong Ingress controller version 0.9.0

Kong or Kong Enterprise version 2.5.0

Kubernetes version

v1.17.5

Environment

Cloud provider or hardware configuration: Bare Metal with Rancher (v2.4.5)
OS (e.g. from /etc/os-release): Ubuntu 18.04
Kernel (e.g. uname -a): inux pre-production1 5.3.0-53-generic #47~18.04.1-Ubuntu SMP Thu May 7 13:10:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Install tools: Helm 3.x

What happened

Upstreams and upstream IP’s are not created

Expected behavior

Upstream and upstream IP’s should be created trough Kong Ingress

Steps To Reproduce

Apply kubectl patch with a newer docker image to an existing pod

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 22 (4 by maintainers)

Most upvoted comments

@l4nd0r , @phlegx We were having frequent issues with upstream target IPs not being updated, forcing us to restart kong. In our case it was targets with weight=0 in postgres, created by kong 2.1. The way kong deletes target was changed on kong 2.2 and migration job left some old targets with weight=0 . See my comment on https://github.com/Kong/kubernetes-ingress-controller/issues/1025#issuecomment-803002692

The solution is just to remove those old targets with weight=0 from database.

Don’t know if it is same thing here, but just to add some info.

cpcostaneves on Mar 22, 2021

Just a short heads up, I’m a co worker of @phlegx and we tried to further look into the problem (I’m not that experienced with kubernetes and kong-ingress-controller yet, so forgive me if I miss something obvious).

On deployment (CI) we execute following kubectl command:

11:18:11  + kubectl --kubeconfig /root/.kube/config_pre-production patch deployment --namespace bms bms-api --patch spec:
11:18:11   template:
11:18:11    spec:
11:18:11     containers:
11:18:11     - name: bms-api
11:18:11       image: xxx
11:18:12  deployment.apps/bms-api patched

We tried a few things to further hunt down the problem:

Scaling down Kong to 1 instance
Scaling down the services being deployed to 1 instance
Setting --v flag mentioned at https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/troubleshooting.md#debug

After the last point it seemed to be fixed (also removed the flag again and upscaled kong and related services) so seemingly recreating kong with its ingress-controller container fixed the situation for us. Unfortunately we can’t provide further details on the specific problem we had.

tak1n on Jul 15, 2020