spring-pulsar: PulsarListener does not recover after it has startup failure

I’m hitting an issue where the consumption randomly stops working. I haven’t seen any pattern to it. I’m using spring-pulsar 1.1.0.

I’ve spent the past 4 hours trying to figure out why data wasn’t being saved to the database. It turned out that the spring-pulsar consumers weren’t receiving data. I added more logging to the @PulsarListener annotated method to log out the object after the first line of code. It still didn’t log out. I added more logging to the @PulsarListener annotated method, and it started working. I restarted the container (with no code changes), and it stopped logging. I restarted again with no code changes, and it started logging again.

The methods are consuming an Avro topic. The data is being produced, and I verified with the pulsar-admin that new messages are arriving. There are five different methods in that class that are @PulsarListener listener annotated. With this bug, some methods will receive messages and others won’t. No exceptions are being logged, except some initial exclusive consumer exceptions while the other container is being terminated that stop being logged after startup. The methods are using both topicPattern and topics. The process is running in a K8s container.

Is there any known limitation or bug that I’m hitting? It makes no sense that the methods stop receiving data with no code changes.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 16

Commits related to this issue

Most upvoted comments

The workaround I’m doing now is operational. I’m waiting for the other container to stop before starting the new one.

Thanks for picking this up. If you go the retry route without some kind of notify/exit on failure, that won’t fully solve the problem. Is there a way for it to continue retrying in the background in a while loop so that there isn’t a state where the consumer doesn’t connect again?