cf-java-client: Firehose netty leaks with reactor 3.1.2+0.7.2 Fails with reactor 3.1.3+0.7.3

I’ve been getting some Netty leak messages when connecting to the firehose. I tried to upgrade to reactor 3.1.3+0.7.3. With that version I no longer get leak messages but the firehose fails work anymore after a handful of reconnects.

I’ve provided a sample application. Change the reactor reactor versions to see the different problems.

Using 3.1.2+0.7.2 I usually see a netty leak message after ~40-200 reconnects. Using 3.1.3+0.7.3 I usually stop getting data with my new firehose connection after ~20 reconnects.

DefaultConnectionContext context = DefaultConnectionContext.builder().apiHost("some.api").build();

TokenProvider tokenProvider = ClientCredentialsGrantTokenProvider.builder().clientId("doppler").clientSecret("secret").build();
ReactorDopplerClient doppler = ReactorDopplerClient.builder()
		.tokenProvider(tokenProvider)
		.connectionContext(context)
		.build();

long noMessagetimeout = 30000;
while(true) {
	long timeout = System.currentTimeMillis()+noMessagetimeout;
	AtomicBoolean messageSent = new AtomicBoolean(false);
	Disposable disposable = doppler.firehose(FirehoseRequest.builder().subscriptionId("subscription").build()).subscribe(envelope -> messageSent.set(true));
	while(!disposable.isDisposed()) {
		Thread.sleep(100); //Don't peg the processor
		if(messageSent.get()) {
			System.out.println("Resetting the firehose.");
			disposable.dispose();
			continue;
		}
		if(System.currentTimeMillis() > timeout) {
			System.out.println("Failed to get anything from firehose in "+noMessagetimeout+"ms.  Exiting.");
			return;
		}
	}
}

About this issue

Original URL
State: closed
Created 6 years ago
Comments: 24 (24 by maintainers)

Most upvoted comments

Great. Thanks for the input. I’m going to leave this open until we get the Reactor releases and do the version upgrades.

nebhale on Feb 15, 2018

I stopped it at 80,000 loops this morning and never saw that issue again. So I think you can safely go to production with it, but I’ll keep trying to track down the root cause.

nebhale on Feb 15, 2018

@youngm For what it’s worth, a bit more idiomatic version of your test would be:

AtomicInteger counter = new AtomicInteger();

for (; ; ) {
    doppler.firehose(FirehoseRequest.builder()
        .subscriptionId("subscription")
        .build())
        .doOnSubscribe(s -> System.out.printf("Connecting to the firehose: %d%n", counter.getAndIncrement()))
        .blockFirst(Duration.ofSeconds(30));
}

And if you look closely, the actually number is 17 attempts to connect. Something tells me that the fact there there are 16 successful attempts and then number 17 blocks isn’t a coincidence. Guessing it’s a thread or connection thing not being cleaned up properly.

(oh, and I’ve reproduced the issue)

nebhale on Feb 14, 2018