ingress-nginx: auth-tls-pass-certificate-to-upstream does not work with https

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: 0.21.0

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"435f92c719f279a3a67808c80521ea17d5715c66", GitTreeState:"clean", BuildDate:"2018-11-27T01:14:37Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

What happened:

When adding auth-tls-pass-certificate-to-upstream: true to an ingress resource, the client certificate passed to the ingress controller is not forwarded to the backend pod.

What you expected to happen:

The backend pod should receive the client certificate.

How to reproduce it (as minimally and precisely as possible):

Start an https server expecting mtls
Create an ingress resource, such as the following, that points to the mtls server’s service

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: mtls-sample
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "https"
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
spec:
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: mtls-svc
              servicePort: 443

Send a request to the ingress controller, such as the following.

# curl -L --cert 4_client/certs/localhost.cert.pem --key 4_client/private/localhost.key.pem https://172.17.0.11:443/hello -k

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.15.6</center>
</body>
</html>

View the pod logs (assuming go app) to verify missing client cert.

2018/12/04 17:42:29 http: TLS handshake error from 172.17.0.11:41922: tls: client didn't provide a certificate

Send same curl directly to pod service and verify mtls succeeds.

# curl -L --cert 4_client/certs/localhost.cert.pem --key 4_client/private/localhost.key.pem https://10.102.202.95:443/hello -k

Hello World/

Anything else we need to know:

Unless i’m misunderstanding the annotation, I’d expect the client cert to be passed on to the upstream pod.

About this issue

Original URL
State: open
Created 6 years ago
Comments: 36 (9 by maintainers)

Most upvoted comments

Again, still interested! 😃

/remove-lifecycle stale

baurmatt on Oct 14, 2019

Any update on this? 😃 I really like to see this implemented!

/remove-lifecycle rotten

baurmatt on Apr 16, 2019

Can’t wait to get this! 😉

/remove-lifecycle stale

baurmatt on Jan 13, 2020

Still interested! 😃

/remove-lifecycle stale

baurmatt on Jul 15, 2019

Following annotation configuration works

annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/auth-tls-verify-client: “on” nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-root-cert nginx.ingress.kubernetes.io/configuration-snippet: | proxy_ssl_name “hostname”; nginx.ingress.kubernetes.io/ingress.allow-http: “false” nginx.ingress.kubernetes.io/backend-protocol: “HTTPS” nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: “true” nginx.ingress.kubernetes.io/proxy-ssl-secret: default/ingress-cert-key-with-ca-root-cert nginx.ingress.kubernetes.io/proxy-ssl-verify: “on” nginx.ingress.kubernetes.io/proxy-ssl-verify-depth: “1”

As per the documentation: 
[](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#backend-certificate-authentication)

nginx.ingress.kubernetes.io/proxy-ssl-secret: secretName: Specifies a Secret with the certificate tls.crt, key tls.key in PEM format used for authentication to a proxied HTTPS server. It should also contain trusted CA certificates ca.crt in PEM format used to verify the certificate of the proxied HTTPS server. This annotation expects the Secret name in the form “namespace/secretName”.

vijaysimhajoshi on Mar 10, 2022

Same issue.

asettouf on Nov 18, 2021

/remove-lifecycle stale /lifecycle frozen

rikatz on Aug 6, 2021

/kind feature /remove-lifecycle stale /assign

rikatz on May 4, 2021

/triage accepted

cmluciano on Jan 12, 2021

@johnrosso Are you still interested in submitting a PR for this feature ?

cmluciano on Jan 12, 2021

@rdoering Here are the annotations that worked for me:

annotations:
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
    nginx.ingress.kubernetes.io/auth-tls-secret: default/{{ .Values.ingress.caTls.caSecretName }}
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;
      proxy_ssl_name "mydemo.demo.com";
    kubernetes.io/ingress.allow-http: "false"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/proxy-ssl-secret: default/{{ .Values.ingress.caTls.caSecretName }}
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-verify-depth: "1"
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /

IoakeimSamarasGR on Jul 6, 2020

Can you elaborate on the meaning of global value @aledbf ?

Sorry, we use a configmap that configures global values https://github.com/kubernetes/ingress-nginx/blob/master/internal/ingress/controller/config/config.go#L431 with defaults https://github.com/kubernetes/ingress-nginx/blob/master/internal/ingress/controller/config/config.go#L586

aledbf on Dec 4, 2018

this should be a global value instead of a new annotation,

I can take a stab at this. Can you elaborate on the meaning of global value @aledbf ?

this does not means the header will be compatible with envoy

100% understood.

joshrosso on Dec 4, 2018

I suggest we provide the functionality to specify the header key for which the client cert will be forwarded in. Something such as:

That makes sense. We already do this for another header https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#forwarded-for-header That said, this should be a global value instead of a new annotation, at least as a first step.

Edit: just in case, this does not means the header will be compatible with envoy (http://nginx.org/en/docs/http/ngx_http_ssl_module.html != https://www.envoyproxy.io/docs/envoy/latest/configuration/http_conn_man/headers#x-forwarded-client-cert)

aledbf on Dec 4, 2018

Alright, after digging deeper, I’m finding the issue to be more around standardization of passing client certificates in headers rather than my initial theory, the nginx-ingress-controller not passing the client cert.

I’ve found nginx is passing the client cert to the backend pod in the Ssl-client-certificate header.

It seems for projects like Envoy, there’s been lots of discussion around how to accomplish this, in their case they went with x-forwarded-client-cert. And some suggestions on stackoverflow suggest using headers like x-ssl-cert.

I suggest we provide the functionality to specify the header key for which the client cert will be forwarded in. Something such as:

nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream-header: "x-forwarded-client-cert"

To ensure compatibility with upstream servers / pods.

Let me know your thoughts.

joshrosso on Dec 4, 2018