ingress-nginx: TLS Certificate not updated when Secret is updated

NGINX Ingress controller version: 0.30.0

Kubernetes version (use kubectl version): Server Version: version.Info{Major:“1”, Minor:“16”, GitVersion:“v1.16.3”, GitCommit:“b3cbbae08ec52a7fc73d334838e18d17e8512749”, GitTreeState:“clean”, BuildDate:“2019-11-13T11:13:49Z”, GoVersion:“go1.12.12”, Compiler:“gc”, Platform:“linux/amd64”}

Environment: Kubernetes / Docker

Cloud provider or hardware configuration: Giant Swarm cluster on AWS
OS (e.g. from /etc/os-release): Alpine Linux
Kernel (e.g. uname -a): Linux ca09dc795922 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64 GNU/Linux
Install tools: helm
Others:

What happened:

ingress-nginx was happily serving an endpoint with a TLS certificate stored in a Secret specified by Ingress .spec.tls[0].secretName.
Secret values were replaced with new certificate data.
ingress-nginx continued to use old certificate
After many hours (>12) the new certificate is used

What you expected to happen:

Expected the new certificate to be used, as normally happens.

I can only presume that the lua certificate cache is not being updated for some reason.

How to reproduce it:

We have only seen this once, despite performing the same operation (updating Secret manually and via cert-manager) in many environments. I’m not confident that it can be reproduced and I suspect that restarting the ingress Pods will cause use of the correct certificate.

Anything else we need to know:

The certificates are different but valid. Serial numbers, CN and SAN are different but each are valid for the configured tls hosts. kubectl get secret the-secret-name -n default -o json | jq -r '.data["tls.crt"]' | base64 -d | head kubectl ingress-nginx certs --host redacted.host.name | head
The list of TLS hosts has been modified.
For new hosts in the ingress, the new certificate is used. For old hosts, the old certificate is used.
This is not about using the default certificate.

/kind bug

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 9
Comments: 30 (20 by maintainers)

Most upvoted comments

Thank you for the reminder. We have some issues updating to the latest ingress. I suggest that we close this issue as it can no longer be reproduced and we will re-engage if we see this again. Many thanks to the testers and for your ongoing effort with this project!

hobti01 on Oct 13, 2021

@hobti01 friendly ping

rikatz on Oct 12, 2021

@hobti01 we are having some issues reproducing this in v1.0.0 (maybe we fixed without knowing!)

Can you please check if this still occurs with newer versions? Thanks!!

rikatz on Sep 8, 2021

@KuberDriver there should not be any difference in behaviour, but if you have time for a test with multiple instances it would be more representative of production deployments.

Thank you for your testing efforts ❤️

hobti01 on Sep 2, 2021

Finally fixed by updating all secrets with the same hostname. logs of nginx pod

Adding location "/path1(/|$)(.*)" for server "my.testing.com" with upstream "namespace1-api-service-80" (Ingress "namespace1/ingress")
Adding location "/path2(/|$)(.*)" for server "my.testing.com" with upstream "namespace2-api-service-80" (Ingress "namespace2/ingress")

code in nginx pod lua_package_path “/etc/nginx/lua/?.lua;;”;

function _M.get_pem_cert_key(hostname)
  local uid = certificate_servers:get(hostname)
  if not uid then
    return nil
  end

  return certificate_data:get(uid)
end

so maybe it stores and retrieves certs by a host as a key that configured in ingresses of all namespaces. If it has other certs for the same hostname, it won’t work properly until those old certs get updated.

nathsingchan on Dec 4, 2020