ingress-nginx: ingress-nginx intermittently serves the default certificate instead of a configured tls certificate for rules without a host

NGINX Ingress controller version:

Installed with the Digital Ocean NGINX Ingress Controller 1-Click App. Looks to be NGINX 0.44.0 through the 3.23.0 Helm chart:

$ kubectl get deployment ingress-nginx-controller --namespace=ingress-nginx -o jsonpath='{.metadata.labels}' {"app.kubernetes.io/component":"controller","app.kubernetes.io/instance":"ingress-nginx","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/version":"0.44.0","helm.sh/chart":"ingress-nginx-3.23.0"}

Kubernetes version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"archive", BuildDate:"2021-03-30T00:00:00Z", GoVersion:"go1.16", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: Digital Ocean
  • OS (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
  • Kernel (e.g. uname -a): Linux pool-o1y0v82td-8wjdr 4.19.0-11-amd64 #1 SMP Debian 4.19.146-1 (2020-09-17) x86_64 GNU/Linux
  • Install tools: N/A
  • Others: N/A

What happened:

As detailed in cert-manager issue #4012, cert-manager is creating the certificate and ingress-nginx seems to be picking it up, but the default fake certificate is still being served:

$ curl -vk https://example.myplaceonline.com/ 2>&1 | grep -e subject: -e issuer: -e Hello
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
<html><head><title>HTTP Hello World</title></head><body><h1>Hello from helloworldweb-849f6d4b9f-ppcr6</h1></body></html>

cert-manager created the certificate:

$ kubectl describe certificate ingress1-cert --namespace=testns1
[...]
Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Issuing    2m11s  cert-manager  Issuing certificate as Secret does not exist
  Normal  Generated  2m10s  cert-manager  Stored new private key in temporary Secret resource "ingress1-cert-q9hgc"
  Normal  Requested  2m10s  cert-manager  Created new CertificateRequest resource "ingress1-cert-xmtml"
  Normal  Issuing    63s    cert-manager  The certificate has been successfully issued

Describing the ingress shows the TLS certificate:

$ kubectl describe ingress ingress1 --namespace=testns1
Name:             ingress1
Namespace:        testns1
Address:          143.198.245.101
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  ingress1-cert terminates example.myplaceonline.com
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /(.*)   helloworldweb:80 (10.244.0.47:80)
Annotations:  cert-manager.io/issuer: letsencrypt-staging-issuer
              nginx.ingress.kubernetes.io/rewrite-target: /$1
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    19m (x3 over 20m)  nginx-ingress-controller  Scheduled for sync

Running the ingress with --v=5 debug logging shows the following “ssl” related entries which suggests it found the certificate but might not be applying it?

I0522 17:21:30.359718       6 backend_ssl.go:41] "Syncing Secret" name="testns1/ingress1-cert"
I0522 17:21:30.360392       6 ssl.go:112] "parsing ssl certificate extensions"
I0522 17:21:30.360571       6 backend_ssl.go:145] "Configuring Secret \"testns1/ingress1-cert\" for TLS encryption (CN: [example.myplaceonline.com])"
I0522 17:21:30.360884       6 nginx.go:342] "Event received" type=UPDATE object="&Ingress{ObjectMeta:{ingress1  testns1  6f15804a-3817-4470-aa5e-f55f39d92688 159480 2 2021-05-22 02:00:46 +0000 UTC <nil> <nil> map[] map[cert-manager.io/issuer:letsencrypt-staging-issuer nginx.ingress.kubernetes.io/rewrite-target:/$1] [] []  [{kubectl-create Update networking.k8s.io/v1 2021-05-22 02:00:46 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:nginx.ingress.kubernetes.io/rewrite-target\":{}}},\"f:spec\":{\"f:rules\":{}}}} {kubectl-edit Update networking.k8s.io/v1 2021-05-22 02:07:01 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\"f:cert-manager.io/issuer\":{}}},\"f:spec\":{\"f:tls\":{}}}} {nginx-ingress-controller Update networking.k8s.io/v1beta1 2021-05-22 17:21:30 +0000 UTC FieldsV1 {\"f:status\":{\"f:loadBalancer\":{\"f:ingress\":{}}}}}]},Spec:IngressSpec{Backend:nil,TLS:[]IngressTLS{IngressTLS{Hosts:[example.myplaceonline.com],SecretName:ingress1-cert,},},Rules:[]IngressRule{IngressRule{Host:,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/(.*),Backend:IngressBackend{ServiceName:helloworldweb,ServicePort:{0 80 },Resource:nil,},PathType:*Prefix,},},},},},},IngressClassName:nil,},Status:IngressStatus{LoadBalancer:{[{143.198.245.101  []}]},},}"
I0522 17:21:30.361196       6 queue.go:87] "queuing" item="&Ingress{ObjectMeta:{ingress1  testns1  6f15804a-3817-4470-aa5e-f55f39d92688 159480 2 2021-05-22 02:00:46 +0000 UTC <nil> <nil> map[] map[cert-manager.io/issuer:letsencrypt-staging-issuer nginx.ingress.kubernetes.io/rewrite-target:/$1] [] []  [{kubectl-create Update networking.k8s.io/v1 2021-05-22 02:00:46 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:nginx.ingress.kubernetes.io/rewrite-target\":{}}},\"f:spec\":{\"f:rules\":{}}}} {kubectl-edit Update networking.k8s.io/v1 2021-05-22 02:07:01 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\"f:cert-manager.io/issuer\":{}}},\"f:spec\":{\"f:tls\":{}}}} {nginx-ingress-controller Update networking.k8s.io/v1beta1 2021-05-22 17:21:30 +0000 UTC FieldsV1 {\"f:status\":{\"f:loadBalancer\":{\"f:ingress\":{}}}}}]},Spec:IngressSpec{Backend:nil,TLS:[]IngressTLS{IngressTLS{Hosts:[example.myplaceonline.com],SecretName:ingress1-cert,},},Rules:[]IngressRule{IngressRule{Host:,IngressRuleValue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/(.*),Backend:IngressBackend{ServiceName:helloworldweb,ServicePort:{0 80 },Resource:nil,},PathType:*Prefix,},},},},},},IngressClassName:nil,},Status:IngressStatus{LoadBalancer:{[{143.198.245.101  []}]},},}"
I0522 17:21:30.361945       6 queue.go:128] "syncing" key="testns1/ingress1"

I reviewed /etc/nginx/nginx.conf but it seems the certificates are handled by a Lua module and I’m not sure how to dive into that.

How to reproduce it:

Reproduction steps detailed in cert-manager issue #4012.

Anything else we need to know: N/A

/kind bug

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 36 (33 by maintainers)

Commits related to this issue

Most upvoted comments

another log message needed is the below sequence of steps and the related logs ;

  • start a terminal to follow the ingress controller logs kubectl -n <ingresscontrollernamespace> logs -f <ingresscontrollerpodname>
  • edit the ingress in question and remove the tls section
  • check if the config was reloaded in the controllerpod logs
  • edit the ingress and add the tls section again
  • check if the ingresscontrollerpod showed reload of config
  • Copy paste the log messages (related to this test) in this issue
+1
longwuyuan on Jun 5, 2021
Read more comments on GitHub
← ingress-nginx: ingress-nginx-controller pod fails to start on s390x
ingress-nginx: ingress-nginx returns 503 during nginx process reload →