skywalking: Skywalking oap NullPointerException

Please answer these questions before submitting your issue.

  • Why do you submit this issue?
  • Question or discussion
  • [ x] Bug
  • Requirement
  • Feature or performance improvement

Bug

  • Which version of SkyWalking, OS and JRE? K8s 1.18.6 Skywalking 8.1

  • What happened? I’ve installed skywalking on K8s following below steps:

git clone https://github.com/apache/skywalking-kubernetes.git
kubectl create namespace demo
cd chart
helm repo add elastic https://helm.elastic.co
helm dep up skywalking
helm install skywalking skywalking -n demo

It works fine for a few hours, and then the skywalking oap pod keeps crashing(maybe after reboot the k8s VM?):

skywalking-oap-59dc695b5f-qfgwp    0/1     CrashLoopBackOff   9          18m

log shows

2020-10-14 04:13:07,072 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4030 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentDispatcher is added into DefaultScopeDefine 12.
2020-10-14 04:13:07,072 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4030 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.service.ServiceTrafficDispatcher is added into DefaultScopeDefine 1.
2020-10-14 04:13:07,417 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -4375 [main] INFO  [] - start the provider default in agent-analyzer module.
2020-10-14 04:13:07,790 - org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager -4748 [pool-3-thread-1] ERROR [] - null
java.lang.NullPointerException: null
	at org.apache.skywalking.oap.server.cluster.plugin.kubernetes.NamespacedPodListInformer.listPods(NamespacedPodListInformer.java:96) ~[cluster-kubernetes-plugin-8.1.0.jar:8.1.0]
	at org.apache.skywalking.oap.server.cluster.plugin.kubernetes.KubernetesCoordinator.queryRemoteNodes(KubernetesCoordinator.java:59) ~[cluster-kubernetes-plugin-8.1.0.jar:8.1.0]
	at org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager.refresh(RemoteClientManager.java:128) ~[server-core-8.1.0.jar:8.1.0]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2020-10-14 04:13:08,413 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -5371 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.EndpointRelationDispatcher is added into DefaultScopeDefine 6.
2020-10-14 04:13:08,413 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -5371 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.ServiceRelationDispatcher is added into DefaultScopeDefine 4

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 20 (8 by maintainers)

Most upvoted comments

I meet the same problem with the version 8.1.0, Could you tell me how to solved it ?

@kezhenxu94 I think this us only a master branch issue as you talked about. This issue is talking about 8.1.

Could you confirm?

The NPE exists since v8.1.0, and I also encountered this in the ALS test case, it occurs in a relatively low possibility and can be recovered by Kubernetes’ restarting, the other issue that I mentioned by DM is “broke pipe”, which is only reproducible in the master branch, they’re different problems