spring-cloud-kubernetes: Config map not loaded on Istio enabled namespace
Hi,
We are using spring-cloud-kubernetes, particularly the config part of it, to load the application configuration from a configuration map, and it does work great on the bare cluster. However, we are implementing Istio of top of it.
We already tried creating destination rules, etc, but nothing seems to be working. Is there a known direction we should look into? Maybe some missing configuration on our side that someone would know? For note, we are running on Azure AKS.
P.S: We did go over the existing issues, and there is quite a lot of them that are similar, but the error message seems to be a different one and the solutions in the other issues did not help us.
Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
Full error log:
2019-04-24 16:43:52.010 WARN 1 --- [ restartedMain] o.s.cloud.kubernetes.StandardPodUtils : Failed to get pod with name:[sample-service-5585c68b6b-m6cbd]. You should look into this if things aren't working as you expect. Are you missing serviceaccount permissions?
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [Pod] with name: [sample-service-5585c68b6b-m6cbd] in namespace: [sample] failed.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[kubernetes-client-4.1.0.jar:na]
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[kubernetes-client-4.1.0.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:221) ~[kubernetes-client-4.1.0.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:177) ~[kubernetes-client-4.1.0.jar:na]
at org.springframework.cloud.kubernetes.StandardPodUtils.internalGetPod(StandardPodUtils.java:62) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
at org.springframework.cloud.kubernetes.StandardPodUtils.lambda$new$0(StandardPodUtils.java:46) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
at org.springframework.cloud.kubernetes.LazilyInstantiate.swapper(LazilyInstantiate.java:41) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
at org.springframework.cloud.kubernetes.LazilyInstantiate.lambda$new$0(LazilyInstantiate.java:34) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
at org.springframework.cloud.kubernetes.LazilyInstantiate.get(LazilyInstantiate.java:29) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
at org.springframework.cloud.kubernetes.StandardPodUtils.isInsideKubernetes(StandardPodUtils.java:56) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
at org.springframework.cloud.kubernetes.profile.KubernetesProfileApplicationListener.addKubernetesProfile(KubernetesProfileApplicationListener.java:52) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
at org.springframework.cloud.kubernetes.profile.KubernetesApplicationContextInitializer.initialize(KubernetesApplicationContextInitializer.java:57) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:649) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:373) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:139) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.cloud.bootstrap.BootstrapApplicationListener.bootstrapServiceContext(BootstrapApplicationListener.java:191) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:105) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:71) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:75) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:347) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:306) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
at com.example.SampleServiceApplication.main(SampleServiceApplication.java:31) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.1.4.RELEASE.jar:2.1.4.RELEASE]
Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:439) ~[na:na]
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:184) ~[na:na]
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) ~[na:na]
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) ~[na:na]
at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:281) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:251) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:151) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:195) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119) ~[kubernetes-client-4.1.0.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:66) ~[kubernetes-client-4.1.0.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:107) ~[kubernetes-client-4.1.0.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[okhttp-3.8.1.jar:na]
at okhttp3.RealCall.execute(RealCall.java:69) ~[okhttp-3.8.1.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379) ~[kubernetes-client-4.1.0.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344) ~[kubernetes-client-4.1.0.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:313) ~[kubernetes-client-4.1.0.jar:na]
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:296) ~[kubernetes-client-4.1.0.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:794) ~[kubernetes-client-4.1.0.jar:na]
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:210) ~[kubernetes-client-4.1.0.jar:na]
... 32 common frames omitted
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 15 (9 by maintainers)
@pistantan can you try applying a rule like this: https://github.com/snowdrop/istio-routing-example/blob/master/rules/load-balancing-rule.yml#L14 (using the proper host for the Kubernetes API server of course). What that does is explicitly turn off Mutual TLS for traffic to a specific host.