quarkus: JDBC connection does not reset

Describe the bug

  1. Having up the application, with a sqs collector, started at booting via ManagedExecutor:
 void onStart(@Observes(during = AFTER_COMPLETION) StartupEvent ev) {
        managedExecutor.execute(this::startProcessing);
    }

Somewhere down the line, I do some DB selects / inserts 2. Reset the database. Reproduced in aws cloud, but even in local, with docker postgres db container. At this step end, databse is up and ready. Accepting new connections. 3. Got stack trace

Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:350)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164)
	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
	at io.agroal.pool.wrapper.PreparedStatementWrapper.executeQuery(PreparedStatementWrapper.java:78)
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57)
	... 54 more
Caused by: java.io.EOFException
	at org.postgresql.core.PGStream.receiveChar(PGStream.java:445)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2057)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
	... 60 more

  1. App still up, receives a new sqs event, in method from step 1
  2. App fails on any db interaction with following:
aused by: org.hibernate.exception.JDBCConnectionException: could not prepare statement
	at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:112)
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:186)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:151)
	at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:2103)
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2040)
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2018)
	at org.hibernate.loader.Loader.doQuery(Loader.java:948)
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
	at org.hibernate.loader.Loader.doList(Loader.java:2849)
	at org.hibernate.loader.Loader.doList(Loader.java:2831)
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2663)
	at org.hibernate.loader.Loader.list(Loader.java:2658)
	at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506)
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
	at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219)
	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414)
	at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625)
	at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593)
	... 42 more
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
	at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:877)
	at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:1734)
	at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:428)
	at io.agroal.pool.wrapper.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:659)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149)
	at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:176)
	... 58 more

Expected behavior

Database connection is restored after DB is up again.

Actual behavior

Infinite failure. It recovers just after the app is restarted.

Output of java -version

openjdk 11.0.11 2021-04-20

Quarkus version or git rev

1.13.7

Build tool (ie. output of mvnw --version or gradlew --version)

maven

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 2
  • Comments: 21 (10 by maintainers)

Most upvoted comments

@maxandersen Hi mate (Anthony writing), Since:

  • the symptom appeared exactly starting 1.13.7
  • the problem is not happening with another app built on plain vert.x + c3p0 pooling hitting the same db cluster

I’m pretty sure there is something really tricky and specific. In my case, the db is an amazon prod cluster and the apps are deployed on openshift. I indeed suspect some specific amazon config but we do not administrate the db, so we didn’t manage to get any clue.

I managed to reduce this “pseudo” leak, I’ll paste my settings tomorrow. I’m also in touch with Luis and I send him info as soon as I have.

From Reddit thread highlighting the issue:

I was able to reproduce the issue, but it doesn’t seem very serious to me.

  1. ⁠Start Quarkus application, send a few requests.
  2. ⁠Stop Postgres.
  3. ⁠Send requests, observe “PSQLException: This connection has been closed.” errors.
  4. ⁠Start Postgres.
  5. ⁠Send requests, keep observing connection closed errors for about 30 seconds.
  6. ⁠Normal behavior resumes.

My application configuration doesn’t have anything special regarding database connection validation or maximum age.

In my opinion the only scenario where this is an issue is when database connections are frequently interrupted.