ingress-nginx: Updating proxy-ssl-secret update does not update proxy_ssl_trusted_certificate in nginx.conf

NGINX Ingress controller version: 0.32.0

Kubernetes version (use kubectl version):

$ kc version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.10", GitCommit:"059c666b8d0cce7219d2958e6ecc3198072de9bc", GitTreeState:"clean", BuildDate:"2020-04-03T15:17:29Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: Azure
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools: Helm 3
  • Others:

What happened:

I am unable to get the nginx ingress controller to update proxy_ssl_trusted_certificate in nginx.conf when the proxy-ssl-secret is updated. As a result, nginx ingress continues to use an outdated trusted certificate. I am updating the proxy-ssl-secret annotation and using kubectl apply to update the ingress resource. From the logs, although the nginx ingress controller receives the update, the proxy_ssl_trusted_certificate in nginx.conf is not getting updated. The only way to force this update is by deleting the ingress config using kubectl delete and applying the updated config using kubectl apply.

I am using a secret called root-cert living in the test namespace and exposing the myapp-v1 service using ingress.

Logs from the ingress controller: On CREATE

I0527 18:09:19.773239       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"myapp-ns", Name:"myapp-v1", UID:"1b4c5667-da40-4ce8-8db1-ce2b0cad6246", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"6673993", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress myapp-ns/myapp-v1
I0527 18:09:19.773590       6 backend_ssl.go:66] Adding Secret "test/root-cert" to the local store
I0527 18:09:23.107313       6 controller.go:139] Configuration changes detected, backend reload required.
I0527 18:09:23.217382       6 controller.go:155] Backend successfully reloaded.

On UPDATE:

I0527 18:09:47.697082       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"myapp-ns", Name:"myapp-v1", UID:"1b4c5667-da40-4ce8-8db1-ce2b0cad6246", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"6674055", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress myapp-ns/myapp-v1
I0527 18:09:55.154230       6 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"myapp-ns", Name:"myapp-v1", UID:"1b4c5667-da40-4ce8-8db1-ce2b0cad6246", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"6674071", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress myapp-ns/myapp-v1

It is clear from the logs that the secret corresponding to proxy-ssl-secret is not being updated.

What you expected to happen:

I expect proxy_ssl_trusted_certificate in nginx.conf to be updated with the certificate stored in the new secret added using the proxy-ssl-secret annotation on the ingress resource.

How to reproduce it:

  1. Install nginx ingress controller
  2. Add a trusted certificate to authenticate the backend server using the proxy-ssl-secret annotation, and use kubectl apply to apply the ingress configuration.
  3. Make a note of the proxy_ssl_trusted_certificate in nginx.conf
  4. Update the proxy-ssl-secret annotation with a new secret and run kubectl apply
  5. Verify if proxy_ssl_trusted_certificate in nginx.conf is updated

Anything else we need to know:

/kind bug

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Reactions: 2
  • Comments: 30 (18 by maintainers)

Most upvoted comments

@ckittel , I was also experiencing that problem on 2 AKS clusters with OSM - so this was the reason for creating the fix

still waiting for PR review : https://github.com/kubernetes/ingress-nginx/pull/8066/commits/a7f446cc4a98bdf813a54128c933214ce49da722

it is just sending signal to nginx to reload cert-store , just a few lines of code, in this part where there is a problem

@shashankram I am sorry but version 0.32 is not officially supported anymore, please update to a newer version. @bmv126 could you please provide the output of the commands requested by @longwuyuan in this comments: https://github.com/kubernetes/ingress-nginx/issues/5608#issuecomment-935190470. Thank you

/kind bug

SSL related issues are rather complicated and need very very precise and high amount of simplification while describing the problem.

Your original message does not include really any helpful step-by-step detailed process to reproduce the problem, if someone wants to triage. Your original message does not elaborate a lot with a show of the helm/kubectl commands and outputs like ;

  • helm ls -A
  • kubectl -n <controllernamespace> get get all,secret -A -o wide
  • kubectl -n test get all,secret,ing -o wide
  • kubectl describe <objectname> <objectnamespace> # for related objects like secrets,configmaps,ingresses etc
  • There have been 17 months of fixes and release since the original post so the relevance of the information here needs validation and triaging
  • You could install the current release and provide a detailed step by step process to reproduce the problem

Hence there is very less data for someone to try to understand the problem and help solve it /remove-kind bug /kind support /triage needs-information