pulsar: failed to get Partitioned metadata : Policies not found for mytenant/mynamespace namespace

Describe the bug We have 4 pulsar cluster running on kubernetes with geo-replication. All worked well. Before going to prod, we did some load testing. And during one of them (last one at time of writing), pulsar brokers started to loop on exceptions like:

13:30:24.053 [pulsar-io-21-3] WARN  org.apache.pulsar.client.impl.BinaryProtoLookupService - [non-persistent://mytenant/mynamespace/mytopic-98f088294f0c7a509ab4b1a5412b79308a75d50e] failed to get Partitioned metadata : Policies not found for session/user_events namespace
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: Policies not found for mytenant/mynamespace

Note : we have a lot of non-persistent / not partitioned topics in that namespace (23k). Too many?

To Reproduce Well, difficult. We got this “by chance” during a load test and I cannot find out what caused it

During my investigation so far I tried many things, including:

pulsar-admin namespaces policies mytenant/mynamespace

that returns information without any problem. But I really want to understand what happened and not only restart the cluster and start from fresh.

Everything looks fine (according to logs):

zookeeper
global zookeeper (used for geo replication)
bookkeeper (I checked even if we do not persist those topics)

Only brokers in all regions do log a lot (well, maybe not that much per topic, but with so many topics it is quite a lot).

Any clue on how to investigate / fix it (in case it happens in prod once we do that move) ?

About this issue

Original URL
State: closed
Created 5 years ago
Comments: 15 (5 by maintainers)

Most upvoted comments

Getting the issue again with all kind of topics :

non partitioned global topics
partitioned global topics
local (i.e. only one cluster) non partitioned topics
local (i.e. only one cluster) partitioned topics

And for the latter, I checked that all 3 zk instances have the same content for /namespace/<mytenant>/<mynamespace>

When I do a pulsar-admin topics list tenant/namespace I get a response HTTP 500 Internal Server Error.

Checking the logs I can see on one broker (the one handling the request):

17:25:53.474 [pulsar-web-30-3] INFO  org.eclipse.jetty.server.RequestLog - 10.11.1.122 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/persistent/tenant/namespace HTTP/1.1" 200 675 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.516 [pulsar-web-30-1] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0x15555555_0x2aaaaaaa
17:25:53.516 [pulsar-web-30-4] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0x55555554_0x6aaaaaa9
17:25:53.516 [pulsar-web-30-1] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x15555555_0x2aaaaaaa HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.516 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x55555554_0x6aaaaaa9 HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.516 [pulsar-web-30-7] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x95555553_0xaaaaaaa8 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 1
17:25:53.518 [pulsar-web-30-3] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xd5555552_0xeaaaaaa7 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.519 [pulsar-web-30-6] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0x3fffffff_0x55555554
17:25:53.519 [pulsar-web-30-6] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x3fffffff_0x55555554 HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.519 [pulsar-web-30-4] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0xbffffffd_0xd5555552
17:25:53.520 [pulsar-web-30-1] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x00000000_0x15555555 HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 1
17:25:53.520 [pulsar-web-30-8] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0x2aaaaaaa_0x3fffffff
17:25:53.520 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xbffffffd_0xd5555552 HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 3
17:25:53.520 [pulsar-web-30-8] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x2aaaaaaa_0x3fffffff HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.521 [pulsar-web-30-6] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x6aaaaaa9_0x7ffffffe HTTP/1.1" 307 0 "-" "Pulsar-Java-v2.4.1" 1
17:25:53.521 [pulsar-web-30-7] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0x7ffffffe_0x95555553
17:25:53.522 [pulsar-web-30-7] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x7ffffffe_0x95555553 HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 4
17:25:53.522 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xeaaaaaa7_0xffffffff HTTP/1.1" 200 57 "-" "Pulsar-Java-v2.4.1" 1
17:25:53.524 [pulsar-web-30-6] INFO  org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Namespace bundle is not owned by any broker tenant/namespace/0xaaaaaaa8_0xbffffffd
17:25:53.524 [pulsar-web-30-6] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xaaaaaaa8_0xbffffffd HTTP/1.1" 204 0 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.529 [pulsar-web-30-5] ERROR org.apache.pulsar.broker.admin.v2.NonPersistentTopics - [null] Failed to get list of topics under namespace tenant/namespace
java.util.concurrent.ExecutionException: org.apache.pulsar.client.admin.PulsarAdminException$NotFoundException: Tenant does not exist
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_212]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[?:1.8.0_212]
	at org.apache.pulsar.broker.admin.v2.NonPersistentTopics.getList(NonPersistentTopics.java:184) ~[org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_212]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_212]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_212]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_212]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) ~[org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:316) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:298) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:268) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) [org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) [org.glassfish.jersey.core-jersey-server-2.27.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) [org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:53) [org.apache.pulsar-pulsar-broker-2.4.1.jar:2.4.1]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1340) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) [org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1242) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.Server.handle(Server.java:503) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.apache.pulsar.client.admin.PulsarAdminException$NotFoundException: Tenant does not exist
	at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:214) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.apache.pulsar.client.admin.internal.TopicsImpl$5.failed(TopicsImpl.java:159) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.glassfish.jersey.client.JerseyInvocation$4.failed(JerseyInvocation.java:1030) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$4.completed(JerseyInvocation.java:1017) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:227) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:85) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:243) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:234) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.asynchttpclient.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:66) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:113) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:142) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	... 2 more
Caused by: javax.ws.rs.NotFoundException: HTTP 404 Not Found
	at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1086) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:99) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$4.completed(JerseyInvocation.java:1017) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:227) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:85) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:268) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:312) ~[org.glassfish.jersey.core-jersey-common-2.27.jar:?]
	at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.27.jar:?]
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:243) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$3.onCompleted(AsyncHttpConnector.java:234) ~[org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:2.4.1]
	at org.asynchttpclient.AsyncCompletionHandler.onCompleted(AsyncCompletionHandler.java:66) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:113) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:142) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.7.0.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:591) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:508) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
	... 2 more
17:25:53.532 [pulsar-web-30-5] INFO  org.eclipse.jetty.server.RequestLog - 10.11.1.122 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace HTTP/1.1" 500 5406 "-" "Pulsar-Java-v2.4.1" 21

On other 2 brokers I get:

17:25:53.522 [pulsar-web-30-3] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x95555553_0xaaaaaaa8?authoritative=true HTTP/1.1" 404 34 "-" "Pulsar-Java-v2.4.1" 3
17:25:53.525 [pulsar-web-30-5] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x6aaaaaa9_0x7ffffffe?authoritative=true HTTP/1.1" 404 34 "-" "Pulsar-Java-v2.4.1" 2

and

17:25:53.521 [pulsar-web-30-4] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0xd5555552_0xeaaaaaa7?authoritative=true HTTP/1.1" 200 113 "-" "Pulsar-Java-v2.4.1" 2
17:25:53.524 [pulsar-web-30-1] INFO  org.eclipse.jetty.server.RequestLog - 10.11.0.50 - - [22/Oct/2019:17:25:53 +0000] "GET /admin/v2/non-persistent/tenant/namespace/0x00000000_0x15555555?authoritative=true HTTP/1.1" 200 170 "-" "Pulsar-Java-v2.4.1" 2

On zookeeper, on all 3 nodes, if I check the content of /namespace/tenant/namespace I get:

[zk: localhost:2181(CONNECTED) 0] ls /namespace/tenant/namespace
[0xd5555552_0xeaaaaaa7, 0x6aaaaaa9_0x7ffffffe, 0xeaaaaaa7_0xffffffff, 0x95555553_0xaaaaaaa8, 0x00000000_0x15555555]
[zk: localhost:2181(CONNECTED) 1] get /namespace/tenant/namespace/0xd5555552_0xeaaaaaa7
{"nativeUrl":"pulsar://10.11.0.111:6650","httpUrl":"http://10.11.0.111:8080","disabled":false}
[zk: localhost:2181(CONNECTED) 2] get /namespace/tenant/namespace/0x6aaaaaa9_0x7ffffffe
{"nativeUrl":"pulsar://10.11.3.88:6650","httpUrl":"http://10.11.3.88:8080","disabled":false}
[zk: localhost:2181(CONNECTED) 3] get /namespace/tenant/namespace/0xeaaaaaa7_0xffffffff
{"nativeUrl":"pulsar://10.11.0.50:6650","httpUrl":"http://10.11.0.50:8080","disabled":false}
[zk: localhost:2181(CONNECTED) 4] get /namespace/tenant/namespace/0x95555553_0xaaaaaaa8
{"nativeUrl":"pulsar://10.11.3.88:6650","httpUrl":"http://10.11.3.88:8080","disabled":false}
[zk: localhost:2181(CONNECTED) 5] get /namespace/tenant/namespace/0x00000000_0x15555555
{"nativeUrl":"pulsar://10.11.0.111:6650","httpUrl":"http://10.11.0.111:8080","disabled":false}
[zk: localhost:2181(CONNECTED) 5] get /admin/local-policies/tenant/namespace
{"bundles":{"boundaries":["0x00000000","0x15555555","0x2aaaaaaa","0x3fffffff","0x55555554","0x6aaaaaa9","0x7ffffffe","0x95555553","0xaaaaaaa8","0xbffffffd","0xd5555552","0xeaaaaaa7","0xffffffff"],"numBundles":12}}

So I indeed have some bundles that have “disappeared” from brokers. But I got no broker crash or zookeeper issue whatsoever.

But I do not see any error in zookeeper logs apart maybe things like:

14:21:52.112 [ProcessThread(sid:2 cport:-1):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x200d5ed81570008 type:delete cxid:0x3032c zxid:0x100008631 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0000 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0000
14:21:52.415 [ProcessThread(sid:2 cport:-1):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x200d5ed81570007 type:delete cxid:0x244de zxid:0x100008634 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0000 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0000
14:22:52.112 [ProcessThread(sid:2 cport:-1):] INFO  org.apache.zookeeper.server.PrepRequestProcessor - Got user-level KeeperException when processing sessionid:0x200d5ed81570008 type:delete cxid:0x3034d zxid:0x100008636 txntype:-1 reqpath:n/a Error Path:/ledgers/00/0000 Error:KeeperErrorCode = Directory not empty for /ledgers/00/0000

but those are info logs.

Still a mystery for me … But learning everytime, so now going to rebuild the cluster and check a few more things before using it to ensure it is all ok

f-ld on Oct 22, 2019