ingress-nginx: Nginx doesn't support AEAD-CHACHA20-POLY1305-SHA256 cipher

What happened:

When integrating NGINX with the pinniped (https://pinniped.dev) supervisor, I receive a 502 bad gateway when doing HTTPS to the backend. The logs from the ingress-controller pod shows:

2022/12/04 12:57:24 [error] 102#102: *291951 SSL_do_handshake() failed (SSL: error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:SSL alert number 70) while SSL handshaking to upstream, client: 127.0.0.1, server: pinniped-supervisor.blog.tremolo.dev, request: "GET / HTTP/2.0", upstream: "https://10.42.0.10:10250/", host: "pinniped-supervisor.blog.tremolo.dev"
2022/12/04 12:57:24 [error] 102#102: *291951 SSL_do_handshake() failed (SSL: error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:SSL alert number 70) while SSL handshaking to upstream, client: 127.0.0.1, server: pinniped-supervisor.blog.tremolo.dev, request: "GET / HTTP/2.0", upstream: "https://10.42.2.8:10250/", host: "pinniped-supervisor.blog.tremolo.dev"
2022/12/04 12:57:24 [error] 102#102: *291951 SSL_do_handshake() failed (SSL: error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:SSL alert number 70) while SSL handshaking to upstream, client: 127.0.0.1, server: pinniped-supervisor.blog.tremolo.dev, request: "GET / HTTP/2.0", upstream: "https://10.42.0.10:10250/", host: "pinniped-supervisor.blog.tremolo.dev"
127.0.0.1 - - [04/Dec/2022:12:57:24 +0000] "GET / HTTP/2.0" 502 150 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:107.0) Gecko/20100101 Firefox/107.0" 17 0.005 [pinniped-supervisor-pinniped-supervisor-api-443] [] 10.42.0.10:10250, 10.42.2.8:10250, 10.42.0.10:10250 0, 0, 0 0.003, 0.001, 0.001 502, 502, 502 64ae7a27872728869f931bf68ce46404

What you expected to happen:

HTTPS to the supervisor

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.5.1
  Build:         d003aae913cc25f375deb74f898c7f3c65c06f05
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6

-------------------------------------------------------------------------------

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:28:30Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6+k3s1", GitCommit:"418c3fa858b69b12b9cefbcff0526f666a6236b9", GitTreeState:"clean", BuildDate:"2022-04-28T22:16:18Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

  • Cloud provider or hardware configuration: Civo
  • OS (e.g. from /etc/os-release): N/A
  • Kernel (e.g. uname -a): N/A
  • Install tools: Civo
  • Basic cluster related info:
    • kubectl version
    • kubectl get nodes -o wide
NAME                                                      STATUS   ROLES    AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
k3s-civo-auth-pinniped-6502-9ec1d5-node-pool-33cd-h3zf4   Ready    <none>   36h   v1.23.6+k3s1   192.168.1.5   <none>        Alpine Linux v3.16   5.15.78-0-virt   containerd://1.5.11-k3s2
k3s-civo-auth-pinniped-6502-9ec1d5-node-pool-33cd-xyoyo   Ready    <none>   36h   v1.23.6+k3s1   192.168.1.4   <none>        Alpine Linux v3.16   5.15.78-0-virt   containerd://1.5.11-k3s2
k3s-civo-auth-pinniped-6502-9ec1d5-node-pool-33cd-kkxy6   Ready    <none>   36h   v1.23.6+k3s1   192.168.1.3   <none>        Alpine Linux v3.16   5.15.78-0-virt   containerd://1.5.11-k3s2

  • How was the ingress-nginx-controller installed: Civo marketplace

  • Current State of the controller:

    • kubectl describe ingressclasses
Name:         nginx
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.5.1
Annotations:  <none>
Controller:   k8s.io/ingress-nginx
Events:       <none>

ingress object:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: https
  name: pinniped-supervisor
  namespace: pinniped-supervisor
spec:
  rules:
  - host: pinniped-supervisor.blog.tremolo.dev
    http:
      paths:
      - backend:
          service:
            name: pinniped-supervisor-api
            port:
              number: 443
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - pinniped-supervisor-api.blog.tremolo.dev
    secretName: tls-pinniped-supervisor-api-doesnotexist
  • Others:

Here’s the certificate and openssl s_client from pinnped:

CONNECTED(00000005)
depth=0
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0
verify error:num=21:unable to verify the first certificate
verify return:1
write W BLOCK
---
Certificate chain
 0 s:
   i:/CN=Pinniped Supervisor Aggregation CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBsjCCAVigAwIBAgIQPUSXuHk1E2n7YHkSZ1/GMTAKBggqhkjOPQQDAjAtMSsw
KQYDVQQDEyJQaW5uaXBlZCBTdXBlcnZpc29yIEFnZ3JlZ2F0aW9uIENBMB4XDTIy
MTIwMzAxMDE1OVoXDTIzMTIwMzAxMDY1OVowADBZMBMGByqGSM49AgEGCCqGSM49
AwEHA0IABLUB2pmk9pHrwcEKsYyXw8qIT73889TdkYBahzbRp+E0TlQN8Vc0Oktr
vrsDfWZv2OrjZTn3fBfL7MZzqoSxAHejgYYwgYMwEwYDVR0lBAwwCgYIKwYBBQUH
AwEwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBQ7JmbAfML1UU84CC/2sNCuNmJ4
8zA9BgNVHREBAf8EMzAxgi9waW5uaXBlZC1zdXBlcnZpc29yLWFwaS5waW5uaXBl
ZC1zdXBlcnZpc29yLnN2YzAKBggqhkjOPQQDAgNIADBFAiEAvQEb7XU2lFja4/V4
h6TwDtNG6aS6G99mAHO98s+ZHdICIDh3AUJxdpp89Ulu5udbU0N1qxCxGu5SeozU
ds5trUc5
-----END CERTIFICATE-----
subject=
issuer=/CN=Pinniped Supervisor Aggregation CA
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 952 bytes and written 381 bytes
---
New, TLSv1/SSLv3, Cipher is AEAD-CHACHA20-POLY1305-SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : AEAD-CHACHA20-POLY1305-SHA256
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Start Time: 1670121814
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

How to reproduce this issue:

  1. Deploy ingress-nginx w/LoadBalancer
  2. Deploy pinniped supervisor - kubectl apply -f https://get.pinniped.dev/v0.20.0/install-pinniped-supervisor.yaml
  3. Create the above Ingress rule
  4. Access the host

Anything else we need to know:

I tried to get nginx to recognize the AEAD-CHACHA20-POLY1305-SHA256 cipher by setting:

ssl-ciphers: ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:AEAD-CHACHA20-POLY1305-SHA256

in the ingress-nginx-controller ConfigMap, but didn’t help

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16 (8 by maintainers)

Most upvoted comments

Thanks @bmv126 !!! that did the trick. I tried nginx.ingress.kubernetes.io/proxy-ssl-protocols: "TLSv1.3" but that didn’t make any change to the result.

+2
mlbiam on Dec 5, 2022

Hi @mlbiam

From the errors in description, the issue is because of using invalid tls version when communicating to backend pod.

2022/12/04 12:57:24 [error] 102#102: *291951 SSL_do_handshake() failed (SSL: error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:SSL alert number 70) while SSL handshaking to upstream, client: 127.0.0.1, server: pinniped-supervisor.blog.tremolo.dev, request: "GET / HTTP/2.0", upstream: "https://10.42.0.10:10250/", host: "pinniped-supervisor.blog.tremolo.dev"

As below is set in ingress resource, communication to backend happens via tls

    nginx.ingress.kubernetes.io/backend-protocol: https

By default nginx uses older tls v1 version to communicate, as per http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_protocols

Default: | proxy_ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

You can update the ingress resource to use supported backend tls versions: I think below should work.

    nginx.ingress.kubernetes.io/configuration-snippet: |
           proxy_ssl_protocols TLSv1.2 TLSv1.3;
+2
bmv126 on Dec 5, 2022

@mlbiam Yes. That is because the tmpl which generates the nginx.conf expects CA file to be provided and does not render only the ssl-protocols.

https://github.com/kubernetes/ingress-nginx/blob/3474c33e15d809ba401b38891a2ed3c4080b751e/rootfs/etc/nginx/template/nginx.tmpl#L1000

This logic can be improved to separate the proxy-protocols from the proxy-ssl-verify requirements.

+1
bmv126 on Dec 5, 2022

I see that you need to get this working, but I don’t see real clear descriptive data on what is the problem with the ingress-nginx-controller that needs to be fixed.

ingress-nginx --> piniped using HTTPS fails, user sees a 502 from nginx and the log messages from the beginning of this issue appear in the nginx logs.

2022/12/04 12:57:24 [error] 102#102: *291951 SSL_do_handshake() failed (SSL: error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:SSL alert number 70) while SSL handshaking to upstream, client: 127.0.0.1, server: pinniped-supervisor.blog.tremolo.dev, request: "GET / HTTP/2.0", upstream: "https://10.42.0.10:10250/", host: "pinniped-supervisor.blog.tremolo.dev"
2022/12/04 12:57:24 [error] 102#102: *291951 SSL_do_handshake() failed (SSL: error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:SSL alert number 70) while SSL handshaking to upstream, client: 127.0.0.1, server: pinniped-supervisor.blog.tremolo.dev, request: "GET / HTTP/2.0", upstream: "https://10.42.2.8:10250/", host: "pinniped-supervisor.blog.tremolo.dev"
2022/12/04 12:57:24 [error] 102#102: *291951 SSL_do_handshake() failed (SSL: error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:SSL alert number 70) while SSL handshaking to upstream, client: 127.0.0.1, server: pinniped-supervisor.blog.tremolo.dev, request: "GET / HTTP/2.0", upstream: "https://10.42.0.10:10250/", host: "pinniped-supervisor.blog.tremolo.dev"
127.0.0.1 - - [04/Dec/2022:12:57:24 +0000] "GET / HTTP/2.0" 502 150 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:107.0) Gecko/20100101 Firefox/107.0" 17 0.005 [pinniped-supervisor-pinniped-supervisor-api-443] [] 10.42.0.10:10250, 10.42.2.8:10250, 10.42.0.10:10250 0, 0, 0 0.003, 0.001, 0.001 502, 502, 502 64ae7a27872728869f931bf68ce46404

If your app pod generated a self-signed cert, regardless of you telling the ingress-controller, to use a secret of type tls, that contains that self-signed cert, generated by the app pod, to authenticate to the app pod, I do not know how that will work when the ingress-nginx-controller knows CAs like letsencrypt/digicert/blah/blah but not the CA in that app pod

This is a very confusing statement. I’ve been working with ingress-nginx for five years and have never had to explicitly trust a certificate. Regardless of if the cert is self signed or the CA is self signed. The documentation is pretty clear on this too, that you need to explicitly enable ca verification.

I am not going to test this because the very concept of a self-signed cert in this or any other backend pod without the ingress-nginx-controller knowing the CA will fail. There could be ways to add the CA to the ingess-nginx-controller pod etc etc but I have no idea of all that

Again, based on my own experience of five years with multiple production deployments and the ingress-nginx docs, this is not a correct statement. So unless this is a new, breaking change something is very off on these statements.

+1
mlbiam on Dec 5, 2022

I see that you need to get this working, but I don’t see real clear descriptive data on what is the problem with the ingress-nginx-controller that needs to be fixed.

Other related info I can think of, I have already stated. I can put a certificate from letsencrypt in a pod made from nginx:alpine and it works just fine.

  • If your app pod generated a self-signed cert, regardless of you telling the ingress-controller, to use a secret of type tls, that contains that self-signed cert, generated by the app pod, to authenticate to the app pod, I do not know how that will work when the ingress-nginx-controller knows CAs like letsencrypt/digicert/blah/blah but not the CA in that app pod
  • I am not going to test this because the very concept of a self-signed cert in this or any other backend pod without the ingress-nginx-controller knowing the CA will fail. There could be ways to add the CA to the ingess-nginx-controller pod etc etc but I have no idea of all that
  • I am not sure if we have established that neither the annotation nor the configMap is working to set a cipher
+1
longwuyuan on Dec 5, 2022

I think the problem is not described well.

  • Can you confirm that you have a self-signed cert in that pod for sure. If yes, then the ingress-nginx-controller code can do nothing about this other than honoring the use of the annotation for ssl-passthrough . I already saw that you want to terminate on the ingress-nginx-controller so once you have TLS to the app from outside the cluster, you could consider plain-text traffic from ingress-nginx-controller to the pod (if the app offers that)
  • If you want TLS terminated at the ingress-nginx-controller and then you want to top that with HTTPS to the backend pod, then its not a mystery or confusion to ack that you need a well known CA for the cert in the pod. Simple reason being if you self-sign a cert that gets presented in response to the backend-protocol: HTTPS, then neither the ingress-controller-pod nor any other pod nor any other host has your CA cert inside it to recognize your self-signed cert

/assign

+1
longwuyuan on Dec 4, 2022

And if your backend app requires to terminate TLS then you can try the ssl-passthrough annotation as well. Terminating on the ingress-controller and then using backend-protocol HTTPS means that the pinniped pod or any other pod expecting TLS connections should have a cert with a well known CA as well. Self-signed cert in the backend pod means the CA will not be known outside that pod

+1
longwuyuan on Dec 4, 2022
Read more comments on GitHub
← ingress-nginx: Nginx Controller serving ingress objects where ingressclass annotation does not match controller class
ingress-nginx: nginx /healthz endpoint breaks when using an ingress with nginx.ingress.kubernetes.io/rewrite-target without a host →