istio: File path based Gateway TLS setting is not working when using SPIRE as workload identity issuer.

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

Hypothesis

When SPIRE is configured to provide workload identity it replaces the default sds-cluster. However for ingress gateway, the source of the TLS certs is also moved to sds-cluster. The default SDS client, when SPIRE workload api socket is not mount, understand the specific resource name file-cert:<key>~<cert>. However, once it is replaced by SPIRE, the above resource name is passed onto the spire-agent, which is not able to resolve it.

This breaks the file based TLS SIMPLE mode configuration when SPIRE is the workload identity issuer.

Context I set up my mesh to use SPIRE issuing workload identity. Traffic between proxies, including ingress gateway, are mTLS based on SPIRE issued identity. Workload socket are mounted from spire-agent through CSI.

Error case I’m trying to set up SIMPLE mode TLS at the ingress gateway. Following is the Gateway resource:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: main-gateway
  namespace: mesh-config
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - "*"
      port:
        name: http
        number: 8080
        protocol: HTTP
    - hosts:
        - "romeo.example.net"
      port:
        name: https
        number: 8443
        protocol: HTTPS
      tls:
        mode: SIMPLE
        privateKey: /run/secrets/gateway-tls/tls.key
        serverCertificate: /run/secrets/gateway-tls/tls.cert

Running this configuration, I fail to curl over TLS. (I replaced the host name in logs below to avoid expose internal details)

The gateway open up at 8443, but my NLB maps 443 to 8443.

❯ curl -k https://romeo.example.net/httpbin/hostname
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to romeo.example.net:443

❯ openssl s_client -connect romeo.example.net:443
CONNECTED(00000006)
00D329DB01000000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:304:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 345 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

❯ istioctl proxy-config secret istio-ingressgateway-6bd97dcb68-22rc5.istio-system
RESOURCE NAME                                                                    TYPE           STATUS      VALID CERT     SERIAL NUMBER                                NOT AFTER                NOT BEFORE
file-cert:/run/secrets/gateway-tls/tls.cert~/run/secrets/gateway-tls/tls.key                    WARMING     false
...

In ingress gateway log stream, I saw

2023-10-20T02:30:35.254103Z     warning envoy config external/envoy/source/common/config/grpc_stream.h:153      StreamSecrets gRPC config stream to sds-grpc closed: 3, workload is not authorized for the requested identities ["file-cert:/run/secrets/gateway-tls/tls.cert~/run/secrets/gateway-tls/tls.key"]      thread=18

It seems to me that the proxy attempted to fetch identity from spire-agent for unknown reason.

I dump the config on ingress gateway, the TLS config doesn’t seem to be right

            "tls_certificate_sds_secret_configs": [
             {
              "name": "file-cert:/run/secrets/gateway-tls/tls.cert~/run/secrets/gateway-tls/tls.key",
              "sds_config": {
               "api_config_source": {
                "api_type": "GRPC",
                "grpc_services": [
                 {
                  "envoy_grpc": {
                   "cluster_name": "sds-grpc"
                  }
                 }
                ],
                "set_node_on_first_message_only": true,
                "transport_api_version": "V3"
               },
               "resource_api_version": "V3"
              }
             }
            ]

Alternative

Using the same cert and key, I place them in Kubernetes secret, the TLS works correctly.

      tls:
        mode: SIMPLE
        credentialName: romeo

That indicate the keys and cert are correct.

Restriction

I don’t wish to use Kubernetes secret to store the key and cert as it is not safe. My design is to use a sidecar mount in memory medium with the gateway and rotate the cert and key in memory.

Version

❯ istioctl version
client version: 1.19.0
control plane version: 1.18.2
data plane version: 1.18.2 (6 proxies)

Additional Information

No response