ingress-nginx: nginx-ingress-controller not updating configuration

Problem

I’m experiencing problems with nginx-ingress-controller not updating the nginx configuration even though it logs the ingress events.

I’m using gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.6

Debug info

When viewing the logs for the ingress controller I see the following:

I0530 11:17:20.584359       9 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kube-lego", Name:"kube-lego-nginx", UID:"3497760c-3fa6-11e7-91bb-42010af0017f", APIVersion:"extensions", ResourceVersion:"10209024", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress kube-lego/kube-lego-nginx
I0530 11:17:20.899245       9 status.go:302] updating Ingress default/assets status to [{IP } {IP2 } {IP3 }]
I0530 11:17:20.904150       9 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"assets", UID:"8c295151-4529-11e7-ba87-42010af0017f", APIVersion:"extensions", ResourceVersion:"10209025", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/assets

But the ingress controller is not logging any configuration update.

Normally I see this being logged: I0529 16:33:32.984297 9 controller.go:427] ingress backend successfully reloaded...

However that’s not being logged in this case.

when looking at the generated config with the following command: kubectl exec -it --namespace kube-system nginx-ingress-controller-7d7wg cat /etc/nginx/nginx.conf

The configuration does not contain the expected changes from the ingress definition.

Attempted fixes

deleting the ingress and applying it again

This resulted in this line being logged. Still no configuration change

Logs after deleting:

I0530 11:37:08.544328       9 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"assets", UID:"8c295151-4529-11e7-ba87-42010af0017f", APIVersion:"extensions", ResourceVersion:"10210789", FieldPath:""}): type: 'Normal' reason: 'DELETE' Ingress default/assets

After re-applying:

I0530 11:39:45.247382       9 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"assets", UID:"ada80611-452c-11e7-ba87-42010af0017f", APIVersion:"extensions", ResourceVersion:"10211021", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/assets
I0530 11:39:50.900844       9 status.go:302] updating Ingress default/assets status to [{35.187.174.58 } {35.189.211.19 } {35.189.238.220 }]
I0530 11:39:50.905120       9 event.go:218] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"assets", UID:"ada80611-452c-11e7-ba87-42010af0017f", APIVersion:"extensions", ResourceVersion:"10211032", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/assets

Still no changes to the generated nginx configuration.

About this issue

Original URL
State: closed
Created 7 years ago
Reactions: 7
Comments: 19 (10 by maintainers)

Commits related to this issue

[muellerpublicde] update nginx-ingress image. see https://github.com/kubernetes/ingress/issues/794 — committed to devent/sscontrol-osgi by devent 7 years ago

Most upvoted comments

@2color please check if the image quay.io/aledbf/nginx-ingress-controller:0.130 solves the issue

aledbf on May 30, 2017

@stibi yes. Please open a new issue if you see this in 0.9-beta.10 (post the logs after the update of the configmap)

aledbf on Jul 10, 2017

@jordanjennings after we fixed all this 😃 https://github.com/kubernetes/ingress/projects/3

aledbf on Jun 7, 2017

Hi guys, is the fix part of the beta 10 image? I have just deployed the beta10 image on my cluster and I’m having the exact same problem as described here.

stibi on Jul 10, 2017

Thanks @aledbf

2color on Jun 8, 2017

I am seeing a similar issue, I upgraded from my patched beta.5 from #696 to beta.7 and I am seeing 502s after deploys.

In particular, I can verify that after a Deployment has spun up a new ReplicaSet matching the appropriate Service selector, Endpoints have been populated for the new PodIPs, but the upstream block in the nginx.conf continues to have the PodIP of the previous ReplicaSet’s pod(s) even though those Pods have terminated

Rolling back fixes; I believe that deleting the nginx-ingress pods also has replacements come up with an initially valid config but not track new changes

Of likely relevance, I am still working against 1.5 apiserver and kubelets.

donaldguy on May 30, 2017