skywalking: Skywalking oap NullPointerException
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- Question or discussion
- [ x] Bug
- Requirement
- Feature or performance improvement
Bug
-
Which version of SkyWalking, OS and JRE? K8s 1.18.6 Skywalking 8.1
-
What happened? I’ve installed skywalking on K8s following below steps:
git clone https://github.com/apache/skywalking-kubernetes.git
kubectl create namespace demo
cd chart
helm repo add elastic https://helm.elastic.co
helm dep up skywalking
helm install skywalking skywalking -n demo
It works fine for a few hours, and then the skywalking oap pod keeps crashing(maybe after reboot the k8s VM?):
skywalking-oap-59dc695b5f-qfgwp 0/1 CrashLoopBackOff 9 18m
log shows
2020-10-14 04:13:07,072 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4030 [main] INFO [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.segment.SegmentDispatcher is added into DefaultScopeDefine 12.
2020-10-14 04:13:07,072 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -4030 [main] INFO [] - Dispatcher org.apache.skywalking.oap.server.core.analysis.manual.service.ServiceTrafficDispatcher is added into DefaultScopeDefine 1.
2020-10-14 04:13:07,417 - org.apache.skywalking.oap.server.library.module.BootstrapFlow -4375 [main] INFO [] - start the provider default in agent-analyzer module.
2020-10-14 04:13:07,790 - org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager -4748 [pool-3-thread-1] ERROR [] - null
java.lang.NullPointerException: null
at org.apache.skywalking.oap.server.cluster.plugin.kubernetes.NamespacedPodListInformer.listPods(NamespacedPodListInformer.java:96) ~[cluster-kubernetes-plugin-8.1.0.jar:8.1.0]
at org.apache.skywalking.oap.server.cluster.plugin.kubernetes.KubernetesCoordinator.queryRemoteNodes(KubernetesCoordinator.java:59) ~[cluster-kubernetes-plugin-8.1.0.jar:8.1.0]
at org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager.refresh(RemoteClientManager.java:128) ~[server-core-8.1.0.jar:8.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2020-10-14 04:13:08,413 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -5371 [main] INFO [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.EndpointRelationDispatcher is added into DefaultScopeDefine 6.
2020-10-14 04:13:08,413 - org.apache.skywalking.oap.server.core.analysis.DispatcherManager -5371 [main] INFO [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.ServiceRelationDispatcher is added into DefaultScopeDefine 4
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 20 (8 by maintainers)
I meet the same problem with the version 8.1.0, Could you tell me how to solved it ?
The NPE exists since v8.1.0, and I also encountered this in the ALS test case, it occurs in a relatively low possibility and can be recovered by Kubernetes’ restarting, the other issue that I mentioned by DM is “broke pipe”, which is only reproducible in the master branch, they’re different problems