istio: Ingressgateway TLSmode=SIMPLE doesn't support both token auth and client cert auth since 1.10.0
Bug Description
We are facing unexpected behavior for tlsmode=SIMPLE
and tlsmode=MUTUAL
when we upgrade from 1.9.9 to 1.10.0.
Below is our gateway.yaml
. We are using tlsmode=SIMPLE
since our gateway will accept both token auth and client cert auth. Also we set caCertificates
to validate client cert if has.
But when we upgrade from 1.9.9 to 1.10.0, we found our client cert is not honored by gateway from curl-1.10.0-simple.log
, which is different from curl-1.9.9-simple.log
.
By dumping ingressgateway proxy-config from ingressgateway using below command, I found 1.9.9 with tlsmode=simple
has require_client_certificate=false
and have set validation_context
from caCertficates
we set, while 1.10.0 with tlsmode=simple
only has require_client_certificate=false
and don’t have any validation_context
config.
podname=$(k get pods -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}')
k exec -ti $podname -n istio-system -- curl http://localhost:15000/config_dump > $podname.proxyconfig.json
I think it’s related to this commit #31391, which is first released in istio 1.10.0.
After going through the conversations in #31391 and #31984, I think this should be a regression. tlsmode=SIMPLE
should also have ability to optional check client cert and should set validation_context
if user specify caCertificates
, which should be equal to VerifyClientCertIfGiven
in go TLS handshake options.
# gateway.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: ingressgateway
namespace: default
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
minProtocolVersion: TLSV1_2
serverCertificate: /etc/ingressgateway-https-certs/tls.crt
privateKey: /etc/ingressgateway-https-certs/tls.key
caCertificates: /etc/ca-certs/ca.pem
#curl-1.10.0-simple.log
$ curl --cert-type p12 --cert client-cert.pfx https://xxx -vvv
* Trying xx...
* TCP_NODELAY set
* Connected to xxx port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
#curl-1.9.9-simple.log
$ curl --cert-type p12 --cert client-cert.pfx https://xxx -vvv
* Trying xxxxx...
* TCP_NODELAY set
* Connected to xxx port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
Version
istio version: We are upgrading istio from 1.9.9 to 1.10.0
$ k version --short
Client Version: v1.22.2
Server Version: v1.20.9
Additional Information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 16 (8 by maintainers)
Ok if this is actually sending a “Certificate Request” bit then I think we need a new mode to make this explicit. Thanks for the research.
For optional TLS negotiation contract, here are some investigation result to answer above questions
Regarding to TLS RFC (https://www.ietf.org/rfc/rfc5246.txt, section 7.4.4 and 7.4.6 for optional/required mTLS), if server side doesn’t send “Certificate Request” message to client, client won’t send “Client Certificate” to Server. ( see “7.4.6 Client Certificate”) The expected negotiation steps are :
Thus for optional mtls, server side need to send CA list to client.
For how envoy handle optional mTLS logic, one correctness is
SSL_CTX_set_client_CA_list maps to the Server send “Certificate Request” message to client “SSL_CTX_set_client_CA_list() sets the list of CAs sent to the client when requesting a client certificate for ctx.” https://linux.die.net/man/3/ssl_ctx_set_client_ca_list
SSL_CTX_set_verify is to set local flag in server SSL lib whether require client cert when receiving response from client “SSL_CTX_set_verify() sets the verification flags for ctx to be mode and specifies the verify_callback function to be used.” (https://linux.die.net/man/3/ssl_ctx_set_verify
Thus in Envoy code, if CA list is specified in server config, it will be always send to client no matter ‘require_client_cert’ is specified or not. https://github.com/envoyproxy/envoy/blob/bb95af848c939cfe5b5ee33c5b1770558077e64e/source/extensions/transport_sockets/tls/cert_validator/default_validator.cc#L424
If we can agree on the expected SSL contract for optional TLS, back to Istio app model, I somehow agree with @howardjohn this is implicit, we can consider to expose require_client_cert flag in Istio model (to align with Envoy model, actually also similar with Jetty model https://www.programcreek.com/java-api-examples/?class=org.eclipse.jetty.util.ssl.SslContextFactory&method=setTrustStorePath) or even define a new type besides SIMPLE and MTLS.
As a simple workaround, we can always use following
EnvoyFilter
to addcombined_validation_context
orvalidation_context
to SIMPLE mode: