pulsar: Unable to consume messages, need to re-subscribe client
This is similar to issue 2013. We have several shared subscriptions of partitioned topics and we found some cases where a subscription’s client is not receiving messages, but it’s not related to a specific topic partition.
In this case, if we re-subscribe the client (PulsarClient.subscribeAsync), messages start to be delivered as expected.
Client logs this warn:
[log_time:16:17:42.377] [thread:pulsar-timer-6-1] [level:WARN ] [logger:UnAckedMessageTracker] - [ConsumerBase{subscription='picking-consumer-prod.fbm-wms-picking', consumerName='i-08e97d991d75e9d07-10.73.125.156', topic='persistent://fury/global/wms-wave-feed-prod.fbm-wms-picking/wms-wave-feed-prod.fbm-wms-picking'}] 1 messages have timed-out
And we don’t see any error from brokers.
Any idea? Any log or metric that may be useful to debug this problem?
System configuration
Pulsar version (brokers): 1.20.0 Pulsar clients version: 1.22.0 Bookkeeper version: 4.7.1
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 16 (8 by maintainers)
Hello! We started using Pulsar instead of Kafka in our system, and currently facing the same issue, when few subscriptions (like 1 - 3 of 30) stop receiving messages, and they get stuck in a backlog, until either a subscriber, or pulsar proxy restarted.
We are using version 2.4.2, with 3 brokers in a cluster. Issue appears pretty often, like 3 - 6 times in a day. Stats also attached. Hope this issue can be fixed or at least hope there can be any workaround except of force deleting kube pods with subscriptions to make them reconnect. We used Kafka to handle some critical parts of our system, unfortunately we can’t rely on Pulsar without having simplest consumer - producer scenario work well. Thank you!
topic-stats-internal.txt topic-stats.txt
@sijie I am seeing the same issue on 3.0.0, unable to figure out the root cause of it. How can we fix it without restarting the broker pod? Can we get this issue re-opened?