spring-cloud-kubernetes: Config map not loaded on Istio enabled namespace

Hi,

We are using spring-cloud-kubernetes, particularly the config part of it, to load the application configuration from a configuration map, and it does work great on the bare cluster. However, we are implementing Istio of top of it.

We already tried creating destination rules, etc, but nothing seems to be working. Is there a known direction we should look into? Maybe some missing configuration on our side that someone would know? For note, we are running on Azure AKS.

P.S: We did go over the existing issues, and there is quite a lot of them that are similar, but the error message seems to be a different one and the solutions in the other issues did not help us.

Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message

Full error log:

2019-04-24 16:43:52.010  WARN 1 --- [  restartedMain] o.s.cloud.kubernetes.StandardPodUtils    : Failed to get pod with name:[sample-service-5585c68b6b-m6cbd]. You should look into this if things aren't working as you expect. Are you missing serviceaccount permissions?

io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Pod]  with name: [sample-service-5585c68b6b-m6cbd]  in namespace: [sample]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[kubernetes-client-4.1.0.jar:na]
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) ~[kubernetes-client-4.1.0.jar:na]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:221) ~[kubernetes-client-4.1.0.jar:na]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:177) ~[kubernetes-client-4.1.0.jar:na]
	at org.springframework.cloud.kubernetes.StandardPodUtils.internalGetPod(StandardPodUtils.java:62) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
	at org.springframework.cloud.kubernetes.StandardPodUtils.lambda$new$0(StandardPodUtils.java:46) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
	at org.springframework.cloud.kubernetes.LazilyInstantiate.swapper(LazilyInstantiate.java:41) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
	at org.springframework.cloud.kubernetes.LazilyInstantiate.lambda$new$0(LazilyInstantiate.java:34) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
	at org.springframework.cloud.kubernetes.LazilyInstantiate.get(LazilyInstantiate.java:29) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
	at org.springframework.cloud.kubernetes.StandardPodUtils.isInsideKubernetes(StandardPodUtils.java:56) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
	at org.springframework.cloud.kubernetes.profile.KubernetesProfileApplicationListener.addKubernetesProfile(KubernetesProfileApplicationListener.java:52) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
	at org.springframework.cloud.kubernetes.profile.KubernetesApplicationContextInitializer.initialize(KubernetesApplicationContextInitializer.java:57) ~[spring-cloud-kubernetes-core-1.0.0.RELEASE.jar:1.0.0.RELEASE]
	at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:649) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
	at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:373) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
	at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:139) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
	at org.springframework.cloud.bootstrap.BootstrapApplicationListener.bootstrapServiceContext(BootstrapApplicationListener.java:191) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
	at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:105) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
	at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:71) ~[spring-cloud-context-2.1.0.RELEASE.jar:2.1.0.RELEASE]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
	at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:75) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
	at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
	at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:347) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:306) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) ~[spring-boot-2.1.4.RELEASE.jar:2.1.4.RELEASE]
	at com.example.SampleServiceApplication.main(SampleServiceApplication.java:31) ~[classes/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.1.4.RELEASE.jar:2.1.4.RELEASE]
Caused by: javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
	at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(SSLSocketInputRecord.java:439) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:184) ~[na:na]
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) ~[na:na]
	at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:281) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:251) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:151) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:195) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
	at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119) ~[kubernetes-client-4.1.0.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
	at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:66) ~[kubernetes-client-4.1.0.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
	at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:107) ~[kubernetes-client-4.1.0.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[okhttp-3.8.1.jar:na]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[okhttp-3.8.1.jar:na]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[okhttp-3.8.1.jar:na]
	at okhttp3.RealCall.execute(RealCall.java:69) ~[okhttp-3.8.1.jar:na]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379) ~[kubernetes-client-4.1.0.jar:na]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344) ~[kubernetes-client-4.1.0.jar:na]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:313) ~[kubernetes-client-4.1.0.jar:na]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:296) ~[kubernetes-client-4.1.0.jar:na]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:794) ~[kubernetes-client-4.1.0.jar:na]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:210) ~[kubernetes-client-4.1.0.jar:na]
	... 32 common frames omitted

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (9 by maintainers)

Most upvoted comments

@pistantan can you try applying a rule like this: https://github.com/snowdrop/istio-routing-example/blob/master/rules/load-balancing-rule.yml#L14 (using the proper host for the Kubernetes API server of course). What that does is explicitly turn off Mutual TLS for traffic to a specific host.