ztunnel: Invalid peer certificate on ztunnel, when using subordinate AWS private CA

I’m testing ztunnel on 1.22 release. I have 2 book example apps deployed, one with sidecar and one with ambient. The sidecar one is working, but with the ambient one I get io error: invalid peer certificate: UnknownIssuer. We’re using an AWS Private CA.

ztunnel logs:


ztunnel-9tkd9 2024-05-16T09:40:56.360787Z    debug    h2::codec::framed_write:xds{id=6}:Connection{peer=Client}    send    frame=Data { stream_id: StreamId(1) }
ztunnel-9tkd9 2024-05-16T09:40:56.361244Z    debug    h2::codec::framed_read:xds{id=6}:Connection{peer=Client}    received    frame=WindowUpdate { stream_id: StreamId(0), size_increment: 87 }
ztunnel-9tkd9 2024-05-16T09:40:56.361257Z    debug    h2::codec::framed_read:xds{id=6}:Connection{peer=Client}    received    frame=Ping { ack: false, payload: [2, 4, 16, 16, 9, 14, 7, 7] }
ztunnel-9tkd9 2024-05-16T09:40:56.361262Z    debug    h2::codec::framed_write:xds{id=6}:Connection{peer=Client}    send    frame=Ping { ack: true, payload: [2, 4, 16, 16, 9, 14, 7, 7] }
ztunnel-9tkd9 2024-05-16T09:41:04.301540Z    debug    proxy::outbound:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    outbound spawn START    dur=24.91µs id=d7c00270b56591b9a13ae9f3343722c2
ztunnel-9tkd9 2024-05-16T09:41:04.301568Z    debug    state:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    fetch workload    addr=/10.61.66.245
ztunnel-9tkd9 2024-05-16T09:41:04.301577Z    debug    state:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    fetch address    network_addr.address=172.20.185.75
ztunnel-9tkd9 2024-05-16T09:41:04.301582Z    debug    state:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    fetch address    network_addr.address=172.20.185.75
ztunnel-9tkd9 2024-05-16T09:41:04.301591Z    debug    proxy::outbound:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    request from sleep-5775c4fd7b-5kgww to 172.20.185.75:9080 via 10.61.73.159:15008 type Direct    
ztunnel-dxgrd 2024-05-16T09:41:04.302120Z    debug    state:proxy{uid=6eb1a9cb-6fa0-474b-8e64-829e12ee4087}    fetch workload    addr=/10.61.73.159
ztunnel-dxgrd 2024-05-16T09:41:04.302145Z    debug    proxy::inbound:proxy{uid=6eb1a9cb-6fa0-474b-8e64-829e12ee4087}    fetching cert    destination=10.61.73.159:15008 identity=spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage
ztunnel-dxgrd 2024-05-16T09:41:04.302255Z    debug    rustls::server::hs:proxy{uid=6eb1a9cb-6fa0-474b-8e64-829e12ee4087}    decided upon suite TLS13_AES_256_GCM_SHA384    
ztunnel-9tkd9 2024-05-16T09:41:04.301603Z    debug    access    connection opened    src.addr=10.61.66.245:40580 src.workload=sleep-5775c4fd7b-5kgww src.namespace=istio-ambient-test src.identity="spiffe://cluster.local/ns/istio-ambient-test/sa/sleep" dst.addr=10.61.73.159:15008 dst.hbone_addr=10.61.73.159:9080 dst.service=productpage.istio-ambient-test.svc.cluster.local dst.workload=productpage-v1-69bb8f79bd-2xwqg dst.namespace=istio-ambient-test dst.identity="spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage" direction="outbound"
ztunnel-9tkd9 2024-05-16T09:41:04.301625Z    debug    proxy::outbound:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    proxy to 10.61.73.159:9080 using HBONE via 10.61.73.159:15008 type Direct    
ztunnel-9tkd9 2024-05-16T09:41:04.301651Z    debug    proxy::pool:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    checkout - found mutex for pool key ConnectionMeta { key: 167944548397613624, id: 1 }, waiting for writelock    
ztunnel-9tkd9 2024-05-16T09:41:04.301664Z    debug    proxy::pool:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    new connection needed for 10.61.66.245(spiffe://cluster.local/ns/istio-ambient-test/sa/sleep)->10.61.73.159:15008[spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage]    
ztunnel-9tkd9 2024-05-16T09:41:04.301687Z    debug    proxy::pool:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    spawning new pool conn for 10.61.66.245(spiffe://cluster.local/ns/istio-ambient-test/sa/sleep)->10.61.73.159:15008[spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage]    
ztunnel-9tkd9 2024-05-16T09:41:04.302033Z    debug    rustls::client::hs:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    No cached session for IpAddress(V4(Ipv4Addr([10, 61, 73, 159])))    
ztunnel-9tkd9 2024-05-16T09:41:04.302093Z    debug    rustls::client::hs:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    Not resuming any session    
ztunnel-9tkd9 2024-05-16T09:41:04.302584Z    debug    rustls::client::hs:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    Using ciphersuite TLS13_AES_256_GCM_SHA384    
ztunnel-9tkd9 2024-05-16T09:41:04.302599Z    debug    rustls::client::tls13:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    Not resuming    
ztunnel-9tkd9 2024-05-16T09:41:04.302685Z    debug    rustls::client::tls13:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    TLS1.3 encrypted extensions: [Protocols([ProtocolName(6832)])]    
ztunnel-9tkd9 2024-05-16T09:41:04.302693Z    debug    rustls::client::hs:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    ALPN protocol is Some(b"h2")    
ztunnel-dxgrd 2024-05-16T09:41:04.302377Z    debug    rustls::server::hs:proxy{uid=6eb1a9cb-6fa0-474b-8e64-829e12ee4087}    Chosen ALPN protocol [104, 50]    
ztunnel-dxgrd 2024-05-16T09:41:04.302899Z    warn    hyper_util:proxy{uid=6eb1a9cb-6fa0-474b-8e64-829e12ee4087}    TLS handshake error: tls handshake error: Custom { kind: InvalidData, error: AlertReceived(UnknownCA) }    
ztunnel-9tkd9 2024-05-16T09:41:04.302712Z    debug    rustls::client::tls13:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    Got CertificateRequest CertificateRequestPayloadTls13 { context: , extensions: [SignatureAlgorithms([ECDSA_NISTP384_SHA384, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA512, RSA_PSS_SHA384, RSA_PSS_SHA256, RSA_PKCS1_SHA512, RSA_PKCS1_SHA384, RSA_PKCS1_SHA256]), AuthorityNames([DistinguishedName(30183116301406035504030c0d53756d757020526f6f74204341)])] }    
ztunnel-9tkd9 2024-05-16T09:41:04.302722Z    debug    rustls::client::common:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    Attempting client auth    
ztunnel-9tkd9 2024-05-16T09:41:04.302878Z    error    access    connection complete    src.addr=10.61.66.245:40580 src.workload=sleep-5775c4fd7b-5kgww src.namespace=istio-ambient-test src.identity="spiffe://cluster.local/ns/istio-ambient-test/sa/sleep" dst.addr=10.61.73.159:15008 dst.hbone_addr=10.61.73.159:9080 dst.service=productpage.istio-ambient-test.svc.cluster.local dst.workload=productpage-v1-69bb8f79bd-2xwqg dst.namespace=istio-ambient-test dst.identity="spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage" direction="outbound" bytes_sent=0 bytes_recv=0 duration="1ms" error="io error: invalid peer certificate: UnknownIssuer"
ztunnel-9tkd9 2024-05-16T09:41:04.302911Z    debug    proxy::outbound:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2}    outbound spawn DONE    dur=1.396622ms id=d7c00270b56591b9a13ae9f3343722c2

istioctl x ztunnel-config certificates ztunnel-9tkd9.istio-system

CERTIFICATE NAME                                                     TYPE             STATUS        VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-details     Leaf             Available     true           6c32f522c1420149a4a01e10a16f3ed0     2024-05-17T09:33:51Z     2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-details     Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-details     Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-ratings     Leaf             Available     true           fa1f3e702a01cc1f79aba803615d709f     2024-05-17T09:33:51Z     2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-ratings     Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-ratings     Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews     Leaf             Available     true           2d5c38cf86cd0a9685937c3042acb61f     2024-05-17T09:33:51Z     2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews     Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews     Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-ambient-test/sa/sleep                Leaf             Available     true           199c73a2d76dffea34d91a77f02eaa94     2024-05-17T09:33:51Z     2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-ambient-test/sa/sleep                Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/sleep                Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews     Leaf             Available     true           e633d757240e179488dcf8bb5cbe760d     2024-05-17T09:33:51Z     2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews     Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews     Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/sleep                Leaf             Available     true           39c56714e107e9a9f36a5ac72eac55d3     2024-05-17T09:33:51Z     2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/sleep                Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/sleep                Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/keda/sa/keda-operator                      Leaf             Available     true           8502805087ea3df591b70f758775cecf     2024-05-17T09:33:51Z     2024-05-16T09:31:51Z
spiffe://cluster.local/ns/keda/sa/keda-operator                      Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/keda/sa/keda-operator                      Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z

istioctl x ztunnel-config certificates ztunnel-dxgrd.istio-system

CERTIFICATE NAME                                                         TYPE             STATUS        VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage     Leaf             Available     true           3e6698d905495a93832e7651eedc42b9     2024-05-17T09:33:59Z     2024-05-16T09:31:59Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage     Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage     Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews         Leaf             Available     true           38410b2bc9fb024ce0c535ab07d2f9a8     2024-05-17T09:33:59Z     2024-05-16T09:31:59Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews         Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews         Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-details         Leaf             Available     true           5ea8f4c328f6a4176b496e3119d7cb16     2024-05-17T09:33:59Z     2024-05-16T09:31:59Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-details         Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-details         Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-ratings         Leaf             Available     true           7eee5b3bfedb2438552ae03ca246706b     2024-05-17T09:33:59Z     2024-05-16T09:31:59Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-ratings         Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-ratings         Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews         Leaf             Available     true           354295f084eea9e4de053fc9c7cf8fea     2024-05-17T09:33:59Z     2024-05-16T09:31:59Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews         Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews         Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z
spiffe://cluster.local/ns/kuberhealthy/sa/default                        Leaf             Available     true           bf0a6a25a3a1b13f6006d43f229a5297     2024-05-17T09:34:27Z     2024-05-16T09:32:27Z
spiffe://cluster.local/ns/kuberhealthy/sa/default                        Intermediate     Available     true           56efcac223e5ec5f449ebdb4510a4301     2024-06-03T12:16:36Z     2024-05-04T11:16:36Z
spiffe://cluster.local/ns/kuberhealthy/sa/default                        Root             Available     true           d2c1a62c1fe185ee9c3ff8a48c62e5c5     2032-01-04T14:45:42Z     2022-01-04T13:45:42Z

If I run ambient locally, with the certificate created by Istio, there’s no intermediate certificate.

CERTIFICATE NAME                                          TYPE     STATUS        VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
spiffe://cluster.local/ns/default/sa/bookinfo-reviews     Leaf     Available     true           6665474b276008f25ec290a1ef1a1416     2024-05-17T12:53:39Z     2024-05-16T12:51:39Z
spiffe://cluster.local/ns/default/sa/bookinfo-reviews     Root     Available     true           af70b9815698da5b404e336633c3770      2034-05-14T12:48:25Z     2024-05-16T12:48:25Z

Update: There’s an issue with zTunnel when using AWS PCA and cert-manager, with a subordinate CA.

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Reactions: 6
  • Comments: 35 (16 by maintainers)

Most upvoted comments

Ah, so were you using the plugin CA feature of Istio?