quarkus: JDBC connection does not reset
Describe the bug
- Having up the application, with a sqs collector, started at booting via
ManagedExecutor:
void onStart(@Observes(during = AFTER_COMPLETION) StartupEvent ev) {
managedExecutor.execute(this::startProcessing);
}
Somewhere down the line, I do some DB selects / inserts 2. Reset the database. Reproduced in aws cloud, but even in local, with docker postgres db container. At this step end, databse is up and ready. Accepting new connections. 3. Got stack trace
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:350)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
at io.agroal.pool.wrapper.PreparedStatementWrapper.executeQuery(PreparedStatementWrapper.java:78)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:57)
... 54 more
Caused by: java.io.EOFException
at org.postgresql.core.PGStream.receiveChar(PGStream.java:445)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2057)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
... 60 more
- App still up, receives a new sqs event, in method from step 1
- App fails on any db interaction with following:
aused by: org.hibernate.exception.JDBCConnectionException: could not prepare statement
at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:112)
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:186)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:151)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:2103)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2040)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2018)
at org.hibernate.loader.Loader.doQuery(Loader.java:948)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
at org.hibernate.loader.Loader.doList(Loader.java:2849)
at org.hibernate.loader.Loader.doList(Loader.java:2831)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2663)
at org.hibernate.loader.Loader.list(Loader.java:2658)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414)
at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625)
at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593)
... 42 more
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:877)
at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:1734)
at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:428)
at io.agroal.pool.wrapper.ConnectionWrapper.prepareStatement(ConnectionWrapper.java:659)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:149)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:176)
... 58 more
Expected behavior
Database connection is restored after DB is up again.
Actual behavior
Infinite failure. It recovers just after the app is restarted.
Output of java -version
openjdk 11.0.11 2021-04-20
Quarkus version or git rev
1.13.7
Build tool (ie. output of mvnw --version or gradlew --version)
maven
About this issue
- Original URL
- State: open
- Created 3 years ago
- Reactions: 2
- Comments: 21 (10 by maintainers)
@maxandersen Hi mate (Anthony writing), Since:
I’m pretty sure there is something really tricky and specific. In my case, the db is an amazon prod cluster and the apps are deployed on openshift. I indeed suspect some specific amazon config but we do not administrate the db, so we didn’t manage to get any clue.
I managed to reduce this “pseudo” leak, I’ll paste my settings tomorrow. I’m also in touch with Luis and I send him info as soon as I have.
From Reddit thread highlighting the issue:
I was able to reproduce the issue, but it doesn’t seem very serious to me.
My application configuration doesn’t have anything special regarding database connection validation or maximum age.
In my opinion the only scenario where this is an issue is when database connections are frequently interrupted.