ztunnel: Invalid peer certificate on ztunnel, when using subordinate AWS private CA
I’m testing ztunnel on 1.22 release.
I have 2 book example apps deployed, one with sidecar and one with ambient.
The sidecar one is working, but with the ambient one I get io error: invalid peer certificate: UnknownIssuer.
We’re using an AWS Private CA.
ztunnel logs:
ztunnel-9tkd9 2024-05-16T09:40:56.360787Z debug h2::codec::framed_write:xds{id=6}:Connection{peer=Client} send frame=Data { stream_id: StreamId(1) }
ztunnel-9tkd9 2024-05-16T09:40:56.361244Z debug h2::codec::framed_read:xds{id=6}:Connection{peer=Client} received frame=WindowUpdate { stream_id: StreamId(0), size_increment: 87 }
ztunnel-9tkd9 2024-05-16T09:40:56.361257Z debug h2::codec::framed_read:xds{id=6}:Connection{peer=Client} received frame=Ping { ack: false, payload: [2, 4, 16, 16, 9, 14, 7, 7] }
ztunnel-9tkd9 2024-05-16T09:40:56.361262Z debug h2::codec::framed_write:xds{id=6}:Connection{peer=Client} send frame=Ping { ack: true, payload: [2, 4, 16, 16, 9, 14, 7, 7] }
ztunnel-9tkd9 2024-05-16T09:41:04.301540Z debug proxy::outbound:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} outbound spawn START dur=24.91µs id=d7c00270b56591b9a13ae9f3343722c2
ztunnel-9tkd9 2024-05-16T09:41:04.301568Z debug state:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} fetch workload addr=/10.61.66.245
ztunnel-9tkd9 2024-05-16T09:41:04.301577Z debug state:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} fetch address network_addr.address=172.20.185.75
ztunnel-9tkd9 2024-05-16T09:41:04.301582Z debug state:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} fetch address network_addr.address=172.20.185.75
ztunnel-9tkd9 2024-05-16T09:41:04.301591Z debug proxy::outbound:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} request from sleep-5775c4fd7b-5kgww to 172.20.185.75:9080 via 10.61.73.159:15008 type Direct
ztunnel-dxgrd 2024-05-16T09:41:04.302120Z debug state:proxy{uid=6eb1a9cb-6fa0-474b-8e64-829e12ee4087} fetch workload addr=/10.61.73.159
ztunnel-dxgrd 2024-05-16T09:41:04.302145Z debug proxy::inbound:proxy{uid=6eb1a9cb-6fa0-474b-8e64-829e12ee4087} fetching cert destination=10.61.73.159:15008 identity=spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage
ztunnel-dxgrd 2024-05-16T09:41:04.302255Z debug rustls::server::hs:proxy{uid=6eb1a9cb-6fa0-474b-8e64-829e12ee4087} decided upon suite TLS13_AES_256_GCM_SHA384
ztunnel-9tkd9 2024-05-16T09:41:04.301603Z debug access connection opened src.addr=10.61.66.245:40580 src.workload=sleep-5775c4fd7b-5kgww src.namespace=istio-ambient-test src.identity="spiffe://cluster.local/ns/istio-ambient-test/sa/sleep" dst.addr=10.61.73.159:15008 dst.hbone_addr=10.61.73.159:9080 dst.service=productpage.istio-ambient-test.svc.cluster.local dst.workload=productpage-v1-69bb8f79bd-2xwqg dst.namespace=istio-ambient-test dst.identity="spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage" direction="outbound"
ztunnel-9tkd9 2024-05-16T09:41:04.301625Z debug proxy::outbound:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} proxy to 10.61.73.159:9080 using HBONE via 10.61.73.159:15008 type Direct
ztunnel-9tkd9 2024-05-16T09:41:04.301651Z debug proxy::pool:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} checkout - found mutex for pool key ConnectionMeta { key: 167944548397613624, id: 1 }, waiting for writelock
ztunnel-9tkd9 2024-05-16T09:41:04.301664Z debug proxy::pool:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} new connection needed for 10.61.66.245(spiffe://cluster.local/ns/istio-ambient-test/sa/sleep)->10.61.73.159:15008[spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage]
ztunnel-9tkd9 2024-05-16T09:41:04.301687Z debug proxy::pool:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} spawning new pool conn for 10.61.66.245(spiffe://cluster.local/ns/istio-ambient-test/sa/sleep)->10.61.73.159:15008[spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage]
ztunnel-9tkd9 2024-05-16T09:41:04.302033Z debug rustls::client::hs:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} No cached session for IpAddress(V4(Ipv4Addr([10, 61, 73, 159])))
ztunnel-9tkd9 2024-05-16T09:41:04.302093Z debug rustls::client::hs:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} Not resuming any session
ztunnel-9tkd9 2024-05-16T09:41:04.302584Z debug rustls::client::hs:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} Using ciphersuite TLS13_AES_256_GCM_SHA384
ztunnel-9tkd9 2024-05-16T09:41:04.302599Z debug rustls::client::tls13:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} Not resuming
ztunnel-9tkd9 2024-05-16T09:41:04.302685Z debug rustls::client::tls13:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} TLS1.3 encrypted extensions: [Protocols([ProtocolName(6832)])]
ztunnel-9tkd9 2024-05-16T09:41:04.302693Z debug rustls::client::hs:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} ALPN protocol is Some(b"h2")
ztunnel-dxgrd 2024-05-16T09:41:04.302377Z debug rustls::server::hs:proxy{uid=6eb1a9cb-6fa0-474b-8e64-829e12ee4087} Chosen ALPN protocol [104, 50]
ztunnel-dxgrd 2024-05-16T09:41:04.302899Z warn hyper_util:proxy{uid=6eb1a9cb-6fa0-474b-8e64-829e12ee4087} TLS handshake error: tls handshake error: Custom { kind: InvalidData, error: AlertReceived(UnknownCA) }
ztunnel-9tkd9 2024-05-16T09:41:04.302712Z debug rustls::client::tls13:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} Got CertificateRequest CertificateRequestPayloadTls13 { context: , extensions: [SignatureAlgorithms([ECDSA_NISTP384_SHA384, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA512, RSA_PSS_SHA384, RSA_PSS_SHA256, RSA_PKCS1_SHA512, RSA_PKCS1_SHA384, RSA_PKCS1_SHA256]), AuthorityNames([DistinguishedName(30183116301406035504030c0d53756d757020526f6f74204341)])] }
ztunnel-9tkd9 2024-05-16T09:41:04.302722Z debug rustls::client::common:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} Attempting client auth
ztunnel-9tkd9 2024-05-16T09:41:04.302878Z error access connection complete src.addr=10.61.66.245:40580 src.workload=sleep-5775c4fd7b-5kgww src.namespace=istio-ambient-test src.identity="spiffe://cluster.local/ns/istio-ambient-test/sa/sleep" dst.addr=10.61.73.159:15008 dst.hbone_addr=10.61.73.159:9080 dst.service=productpage.istio-ambient-test.svc.cluster.local dst.workload=productpage-v1-69bb8f79bd-2xwqg dst.namespace=istio-ambient-test dst.identity="spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage" direction="outbound" bytes_sent=0 bytes_recv=0 duration="1ms" error="io error: invalid peer certificate: UnknownIssuer"
ztunnel-9tkd9 2024-05-16T09:41:04.302911Z debug proxy::outbound:proxy{uid=95ac874c-9921-4b5f-86ec-165767321851}:outbound{id=d7c00270b56591b9a13ae9f3343722c2} outbound spawn DONE dur=1.396622ms id=d7c00270b56591b9a13ae9f3343722c2
istioctl x ztunnel-config certificates ztunnel-9tkd9.istio-system
CERTIFICATE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-details Leaf Available true 6c32f522c1420149a4a01e10a16f3ed0 2024-05-17T09:33:51Z 2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-details Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-details Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-ratings Leaf Available true fa1f3e702a01cc1f79aba803615d709f 2024-05-17T09:33:51Z 2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-ratings Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-ratings Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews Leaf Available true 2d5c38cf86cd0a9685937c3042acb61f 2024-05-17T09:33:51Z 2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-ambient-test/sa/sleep Leaf Available true 199c73a2d76dffea34d91a77f02eaa94 2024-05-17T09:33:51Z 2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-ambient-test/sa/sleep Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/sleep Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews Leaf Available true e633d757240e179488dcf8bb5cbe760d 2024-05-17T09:33:51Z 2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/sleep Leaf Available true 39c56714e107e9a9f36a5ac72eac55d3 2024-05-17T09:33:51Z 2024-05-16T09:31:51Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/sleep Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/sleep Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/keda/sa/keda-operator Leaf Available true 8502805087ea3df591b70f758775cecf 2024-05-17T09:33:51Z 2024-05-16T09:31:51Z
spiffe://cluster.local/ns/keda/sa/keda-operator Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/keda/sa/keda-operator Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
istioctl x ztunnel-config certificates ztunnel-dxgrd.istio-system
CERTIFICATE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage Leaf Available true 3e6698d905495a93832e7651eedc42b9 2024-05-17T09:33:59Z 2024-05-16T09:31:59Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-productpage Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews Leaf Available true 38410b2bc9fb024ce0c535ab07d2f9a8 2024-05-17T09:33:59Z 2024-05-16T09:31:59Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-ambient-test/sa/bookinfo-reviews Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-details Leaf Available true 5ea8f4c328f6a4176b496e3119d7cb16 2024-05-17T09:33:59Z 2024-05-16T09:31:59Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-details Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-details Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-ratings Leaf Available true 7eee5b3bfedb2438552ae03ca246706b 2024-05-17T09:33:59Z 2024-05-16T09:31:59Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-ratings Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-ratings Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews Leaf Available true 354295f084eea9e4de053fc9c7cf8fea 2024-05-17T09:33:59Z 2024-05-16T09:31:59Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/istio-sidecar-test/sa/bookinfo-reviews Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
spiffe://cluster.local/ns/kuberhealthy/sa/default Leaf Available true bf0a6a25a3a1b13f6006d43f229a5297 2024-05-17T09:34:27Z 2024-05-16T09:32:27Z
spiffe://cluster.local/ns/kuberhealthy/sa/default Intermediate Available true 56efcac223e5ec5f449ebdb4510a4301 2024-06-03T12:16:36Z 2024-05-04T11:16:36Z
spiffe://cluster.local/ns/kuberhealthy/sa/default Root Available true d2c1a62c1fe185ee9c3ff8a48c62e5c5 2032-01-04T14:45:42Z 2022-01-04T13:45:42Z
If I run ambient locally, with the certificate created by Istio, there’s no intermediate certificate.
CERTIFICATE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
spiffe://cluster.local/ns/default/sa/bookinfo-reviews Leaf Available true 6665474b276008f25ec290a1ef1a1416 2024-05-17T12:53:39Z 2024-05-16T12:51:39Z
spiffe://cluster.local/ns/default/sa/bookinfo-reviews Root Available true af70b9815698da5b404e336633c3770 2034-05-14T12:48:25Z 2024-05-16T12:48:25Z
Update: There’s an issue with zTunnel when using AWS PCA and cert-manager, with a subordinate CA.
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 6
- Comments: 35 (16 by maintainers)
Ah, so were you using the plugin CA feature of Istio?