spring-data-neo4j: Unable to acquire connection from pool within configured maximum time

This is fixed with driver 4.4.10 and 5.3.1 now. Thanks to everyone involved investigating and fixing it. Until there are new SDN releases, the following upgrade paths should work:

• SDN 7.0.x / 7.1.x: Go to Java-Driver 5.3.1
• SDN 6.3.x: Go to Java-Driver 4.4.10

🎁 🎄

We have a fix ready to be implemented. This will be available in the 5.4 Java driver which is planned for release in early January. It will also be back ported to the LTS 4.4 driver as a patch release, which will be available sooner if all things go smoothly.

This has been now addressed with the upgrade to driver 4.4.10 in 6.3.x and 6.2.x. Waiting for a 5.3 patch or the before mentioned 5.4. Thanks.

Another workaround is using explicit concatMap instead of flat map and avoiding parallel transactions this way. We are investigating this.

Thanks to awesome VMWare folks we have this down to be reproducible with this

package org.springframework.data.neo4j.integration.x;

import java.time.Duration;
import java.util.Arrays;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.TimeUnit;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.neo4j.driver.AuthTokens;
import org.neo4j.driver.Config;
import org.neo4j.driver.Driver;
import org.neo4j.driver.GraphDatabase;
import org.neo4j.driver.Session;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.neo4j.config.AbstractNeo4jConfig;
import org.springframework.data.neo4j.config.AbstractReactiveNeo4jConfig;
import org.springframework.data.neo4j.core.DatabaseSelectionProvider;
import org.springframework.data.neo4j.core.ReactiveDatabaseSelectionProvider;
import org.springframework.data.neo4j.core.transaction.Neo4jBookmarkManager;
import org.springframework.data.neo4j.core.transaction.Neo4jTransactionManager;
import org.springframework.data.neo4j.repository.config.EnableNeo4jRepositories;
import org.springframework.data.neo4j.repository.config.EnableReactiveNeo4jRepositories;
import org.springframework.data.neo4j.test.BookmarkCapture;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import org.springframework.test.context.junit.jupiter.SpringJUnitConfig;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.ReactiveTransactionManager;
import org.springframework.transaction.annotation.EnableTransactionManagement;

import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.test.StepVerifier;

@ExtendWith(SpringExtension.class)
@SpringJUnitConfig(FooTest.Config.class)
public class FooTest {

	@Test
	void f(@Autowired MovieRepository movieRepository, @Autowired Driver driver) {
		UUID id = UUID.randomUUID();
		Flux
				.range(1, 5)
				// .doOnCancel(() -> System.out.println("doOnCancel (outer)"))
				.flatMap(
						i -> movieRepository.findById(id)
								.switchIfEmpty(Mono.error(new RuntimeException()))
								//.doOnCancel(() -> System.out.println("doOnCancel"))
				)
				//.collectList()
				//.as(TransactionalOperator.create(transactionManager)::transactional)
				.then()
				.as(StepVerifier::create)
				.verifyError();
		System.out.println("---- complete");
		try (Session session = driver.session()) {
			System.out.println(session.run("RETURN 1").single().get(0));
		}
	}


	@Configuration
	@EnableTransactionManagement
	@EnableReactiveNeo4jRepositories(considerNestedRepositories = true)
	static class Config extends AbstractReactiveNeo4jConfig {

		@Bean
		public Driver driver() {
			org.neo4j.driver.Config config = org.neo4j.driver.Config.builder()
					.withMaxConnectionPoolSize(2)
					.withConnectionAcquisitionTimeout(20, TimeUnit.SECONDS)
					.withLeakedSessionsLogging()
					.build();
			return
					GraphDatabase.driver("bolt://localhost:7687", AuthTokens.basic("neo4j", "secret"), config);
		}
	}

}

Hello @michael-simons, thank you for your reply. Yes I see that just a simple findById works fine. I have to add more details of my particular case. I have a more complex structure of Flux methods invocation in my project, so i guess it makes the difference.

My method consumes a list of objects and creates a Flux to iterate over the list, than in flatMap there is another repository call which returns an existing node and then I do a call to get a user which is not found and an exception is thrown.

I easily reproduce it with such an example

@GetMapping("/{id}")
fun doSomethingByUser(@PathVariable userId: UUID): Mono<Void> = userService.doSomething(userId)

fun doSomething(id: UUID): Mono<Void> {
        return Flux.fromIterable(listOf(1, 2, 3, 4, 5, 7, 8, 9, 10))
            .flatMap {
                nodeRepository.findById(nodeIdWhichExists)
                    .then(findUserById(id))
            }
            .collectList().then()
    }

private fun findUserById(id: UUID): Mono<User> {
        return userRepository.findById(id)
            .switchIfEmpty(Mono.error(NotFoundException("User $id not found")))
    }

collectList is important here, since I do some mapping in the project after that

As the result the requests are stuck and I see “org.neo4j.driver.exceptions.ClientException: Unable to acquire connection from the pool within configured maximum time of 60000ms”

I used your apach bench command to reproduce ab -n 20000 -c 10 localhost:8080/6aeb4340-5712-4678-ba94-fd001a56b43d

Hello @RomanRomanenkov thanks for using our module. I tried to recreate your scenario, please see attached project

sdn2632.zip

It basically recreates what you have

@RestController
public class MovieController {

	private final MovieRepository movieRepository;

	public MovieController(MovieRepository movieRepository) {
		this.movieRepository = movieRepository;
	}

	@GetMapping("/{id}")
	public Mono<Movie> findById(@PathVariable UUID id) {
		return movieRepository.findById(id)
			.switchIfEmpty(Mono.error(new SomeException()));
	}

	static class SomeException extends RuntimeException {

	}

	@ExceptionHandler
	public ResponseEntity<String> handle(SomeException ex) {
		return ResponseEntity.internalServerError().body("Schade");
	}

}

with minimal config

spring.neo4j.pool.log-leaked-sessions=true

spring.data.neo4j.repositories.type=reactive

spring.neo4j.authentication.username=neo4j
spring.neo4j.authentication.password=secret

You’ll need the tx manager:

package com.example.sdn2632;

import org.neo4j.driver.Driver;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.neo4j.core.ReactiveDatabaseSelectionProvider;
import org.springframework.data.neo4j.core.ReactiveUserSelectionProvider;
import org.springframework.data.neo4j.core.transaction.ReactiveNeo4jTransactionManager;
import org.springframework.data.neo4j.repository.config.ReactiveNeo4jRepositoryConfigurationExtension;
import org.springframework.transaction.ReactiveTransactionManager;

@Configuration
public class Neo4jConfig {
	@Bean(ReactiveNeo4jRepositoryConfigurationExtension.DEFAULT_TRANSACTION_MANAGER_BEAN_NAME)
	public ReactiveTransactionManager reactiveTransactionManager(
		Driver driver,
		ReactiveDatabaseSelectionProvider databaseSelectionProvider
	) {
		return ReactiveNeo4jTransactionManager.with(driver)
			.withDatabaseSelectionProvider(databaseSelectionProvider)
			.withUserSelectionProvider(ReactiveUserSelectionProvider.getDefaultSelectionProvider())
			.build();
	}
}

With apache bench used like this

ab -n 20000 -c 10 localhost:8080/6aeb4340-5712-4678-ba94-fd001a56b43d

I See the connections via call dbms.listConnections(); spike to 100, which is the drivers default.

All requests succeed, regardless whether I use an id that exists or not.

The connections will be there for an hour, which is also the default.

Configuring

spring.neo4j.pool.max-connection-lifetime=30s

you will see them going away in the database again.

No logs about leaked sessions.

Please share as much details as you have, thanks.