spinnaker: Clouddriver (integrated with Kubernetes) keeps throwing exceptions on Cache Refresh

*Clouddriver keeps throwing exceptions periodically on refreshing cache *

Cloud Provider : AWS

Environment:

Deployed AWS Spinnaker image to manage deployments in kubernetes

Component Version spinnaker-gate: 3.15.0 spinnaker 0.78.0 spinnaker-echo 1.132.0 spinnaker-font50 1.87.0 spinnaker-igor 1.64.0 spinnaker-orca 1.363.0 spinnaker-deck 2.1032.0 clouddriver version: 1.562.0

kubernetes Version: 1.5.2

Feature Area Clouddriver

Description

Cloud driver keeps throwing exceptions in log looks like it’s not able to refresh cache. Please note that I was able to create Load Balancers successfully but I was not able to create/destroy server groups

Please let me know how to enable debug logging so I can share debug logs as well.

Steps to Reproduce

1. After deploy and configure to integrate with Kubernetes
2. I created application
3. I tried creating Server Group and observed that dialog showing spinning (Force Cache Refreshing) and It never gets finished.

Additional Details Screen Shot:

I have been getting following exception frequently Log snippet

2017-03-18 08:23:33.023  WARN 30000 --- [ecutionAction-1] c.n.s.c.cache.LoggingInstrumentation     : com.netflix.spinnaker.clouddriver.kubernetes.provider.KubernetesProvider:pritesh-spinnaker/KubernetesServerGroupCachingAgent[1/1] completed

java.lang.IllegalArgumentException: null
	at sun.reflect.GeneratedMethodAccessor1039.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at groovy.lang.MetaClassImpl.setProperty(MetaClassImpl.java:2705)
	at groovy.lang.MetaClassImpl.setProperty(MetaClassImpl.java:3747)
	at com.netflix.spinnaker.clouddriver.kubernetes.deploy.description.servergroup.KubernetesTcpSocketAction.setProperty(DeployKubernetesAtomicOperationDescription.groovy)
	at org.codehaus.groovy.runtime.InvokerHelper.setProperty(InvokerHelper.java:195)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.setProperty(ScriptBytecodeAdapter.java:484)
	at com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiConverter.fromTcpSocketAction(KubernetesApiConverter.groovy:739)
	at com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiConverter$fromTcpSocketAction$0.callStatic(Unknown Source)
	at com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiConverter.fromProbe(KubernetesApiConverter.groovy:711)
	at com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiConverter$fromProbe.callStatic(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56)
	at com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiConverter$fromProbe.callStatic(Unknown Source)
	at com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiConverter.fromContainer(KubernetesApiConverter.groovy:522)
	at sun.reflect.GeneratedMethodAccessor677.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:384)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:69)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:166)
	at com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiConverter$_fromReplicationController_closure17.doCall(KubernetesApiConverter.groovy:666)
	at sun.reflect.GeneratedMethodAccessor676.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
	at groovy.lang.Closure.call(Closure.java:414)
	at groovy.lang.Closure.call(Closure.java:430)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3170)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.collect(DefaultGroovyMethods.java:3140)
	at org.codehaus.groovy.runtime.dgm$66.invoke(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callSafe(AbstractCallSite.java:87)
	at com.netflix.spinnaker.clouddriver.kubernetes.api.KubernetesApiConverter.fromReplicationController(KubernetesApiConverter.groovy:665)
	at com.netflix.spinnaker.clouddriver.kubernetes.model.KubernetesServerGroup.<init>(KubernetesServerGroup.groovy:142)
	at sun.reflect.GeneratedConstructorAccessor180.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:105)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:271)
	at com.netflix.spinnaker.clouddriver.kubernetes.provider.agent.KubernetesServerGroupCachingAgent$_buildCacheResult_closure27.doCall(KubernetesServerGroupCachingAgent.groovy:361)
	at sun.reflect.GeneratedMethodAccessor779.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
	at groovy.lang.Closure.call(Closure.java:414)
	at groovy.lang.Closure.call(Closure.java:430)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.with(DefaultGroovyMethods.java:242)
	at org.codehaus.groovy.runtime.dgm$757.invoke(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoMetaMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:251)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:71)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at com.netflix.spinnaker.clouddriver.kubernetes.provider.agent.KubernetesServerGroupCachingAgent.buildCacheResult(KubernetesServerGroupCachingAgent.groovy:357)
	at sun.reflect.GeneratedMethodAccessor1032.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:190)
	at com.netflix.spinnaker.clouddriver.kubernetes.provider.agent.KubernetesServerGroupCachingAgent.loadData(KubernetesServerGroupCachingAgent.groovy:237)
	at com.netflix.spinnaker.cats.agent.CachingAgent$CacheExecution.executeAgentWithoutStore(CachingAgent.java:66)
	at com.netflix.spinnaker.cats.agent.CachingAgent$CacheExecution.executeAgent(CachingAgent.java:59)
	at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler$AgentExecutionAction.execute(ClusteredAgentScheduler.java:193)
	at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler$AgentJob.run(ClusteredAgentScheduler.java:167)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)