istio: Ingressgateway TLSmode=SIMPLE doesn't support both token auth and client cert auth since 1.10.0

Bug Description

We are facing unexpected behavior for tlsmode=SIMPLE and tlsmode=MUTUAL when we upgrade from 1.9.9 to 1.10.0. Below is our gateway.yaml. We are using tlsmode=SIMPLE since our gateway will accept both token auth and client cert auth. Also we set caCertificates to validate client cert if has. But when we upgrade from 1.9.9 to 1.10.0, we found our client cert is not honored by gateway from curl-1.10.0-simple.log, which is different from curl-1.9.9-simple.log.

By dumping ingressgateway proxy-config from ingressgateway using below command, I found 1.9.9 with tlsmode=simple has require_client_certificate=false and have set validation_context from caCertficates we set, while 1.10.0 with tlsmode=simple only has require_client_certificate=false and don’t have any validation_context config.

podname=$(k get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}')
k exec -ti $podname -n istio-system -- curl http://localhost:15000/config_dump > $podname.proxyconfig.json

I think it’s related to this commit #31391, which is first released in istio 1.10.0. After going through the conversations in #31391 and #31984, I think this should be a regression. tlsmode=SIMPLE should also have ability to optional check client cert and should set validation_context if user specify caCertificates, which should be equal to VerifyClientCertIfGiven in go TLS handshake options.

# gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingressgateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      tls:
        mode: SIMPLE
        minProtocolVersion: TLSV1_2
        serverCertificate: /etc/ingressgateway-https-certs/tls.crt
        privateKey: /etc/ingressgateway-https-certs/tls.key
        caCertificates: /etc/ca-certs/ca.pem
#curl-1.10.0-simple.log
$ curl --cert-type p12 --cert client-cert.pfx https://xxx -vvv
*   Trying xx...
* TCP_NODELAY set
* Connected to xxx port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
#curl-1.9.9-simple.log
$ curl --cert-type p12 --cert client-cert.pfx https://xxx -vvv
*   Trying xxxxx...
* TCP_NODELAY set
* Connected to xxx port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):

Version

istio version: We are upgrading istio from 1.9.9 to 1.10.0

$ k version --short
Client Version: v1.22.2
Server Version: v1.20.9

Additional Information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 16 (8 by maintainers)

Most upvoted comments

Ok if this is actually sending a “Certificate Request” bit then I think we need a new mode to make this explicit. Thanks for the research.

For optional TLS negotiation contract, here are some investigation result to answer above questions

It is valid, afaik, for tls clients to send the client cert even if it’s not requested?

Regarding to TLS RFC (https://www.ietf.org/rfc/rfc5246.txt, section 7.4.4 and 7.4.6 for optional/required mTLS), if server side doesn’t send “Certificate Request” message to client, client won’t send “Client Certificate” to Server. ( see “7.4.6 Client Certificate”) The expected negotiation steps are :

  1. Server side first send “Certificate Request” message to client side with the CA list
  2. client side then send “Client Certificate” message back with the client cert, server side decide whether to fail if no client cert provided (require client cert) or continue (client cert is optional)

Thus for optional mtls, server side need to send CA list to client.

The reason why we did that change is #31391 (review) and #31984 (comment)

I am also of the same understanding that Envoy does not request a cert unless we set require_client_cert to true. Quick read of envoy code confirms the same https://github.com/envoyproxy/envoy/blob/bb95af848c939cfe5b5ee33c5b1770558077e64e/source/extensions/transport_sockets/tls/cert_validator/default_validator.cc#L426

Are you sure client cert is requested and validated for SIMPLE?

For how envoy handle optional mTLS logic, one correctness is

  1. SSL_CTX_set_client_CA_list maps to the Server send “Certificate Request” message to client “SSL_CTX_set_client_CA_list() sets the list of CAs sent to the client when requesting a client certificate for ctx.” https://linux.die.net/man/3/ssl_ctx_set_client_ca_list

  2. SSL_CTX_set_verify is to set local flag in server SSL lib whether require client cert when receiving response from client “SSL_CTX_set_verify() sets the verification flags for ctx to be mode and specifies the verify_callback function to be used.” (https://linux.die.net/man/3/ssl_ctx_set_verify

Thus in Envoy code, if CA list is specified in server config, it will be always send to client no matter ‘require_client_cert’ is specified or not. https://github.com/envoyproxy/envoy/blob/bb95af848c939cfe5b5ee33c5b1770558077e64e/source/extensions/transport_sockets/tls/cert_validator/default_validator.cc#L424


If we can agree on the expected SSL contract for optional TLS, back to Istio app model, I somehow agree with @howardjohn this is implicit, we can consider to expose require_client_cert flag in Istio model (to align with Envoy model, actually also similar with Jetty model https://www.programcreek.com/java-api-examples/?class=org.eclipse.jetty.util.ssl.SslContextFactory&method=setTrustStorePath) or even define a new type besides SIMPLE and MTLS.

As a simple workaround, we can always use following EnvoyFilter to add combined_validation_context or validation_context to SIMPLE mode:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-validation-context-to-simple-mode
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: FILTER_CHAIN
    match:
      context: GATEWAY
      listener:
        portNumber: 8443
    patch:
      operation: MERGE
      value:
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
            common_tls_context:
              combined_validation_context:
                default_validation_context: {}
                validation_context_sds_secret_config:
                  name: kubernetes://ssl-certificate-cacert
                  sds_config:
                    ads: {}
                    resource_api_version: V3