Connector: PooledDataSourceConnection sometimes not returning Connections back to the ConnectionPool

Bug Report

Describe the Bug

We have observed that after some time the PolicyDefinitionApiController won’t return Policies anymore. When the getAllPolicies method of the controller is called, the EDC waits forever.

After investigating the issue we found that all connections of the GenericObjectPool where taken and not returned back. The EDC gets stuck at this line: https://github.com/eclipse-edc/Connector/blob/c2bc86f0e4eafbf925e145b69ef5de9d3cd0a359/extensions/common/sql/sql-pool/sql-pool-apache-commons/src/main/java/org/eclipse/edc/sql/pool/commons/CommonsConnectionPool.java#L66 and never reaches the catch-block, as all Connections are already taken and not returned to the GenericObjectPool.

Expected Behavior

PooledDataSourceConnection always returning Connections or CommonsConnectionPool releasing Connections after some time.

Observed Behavior

Threads waiting forever for a new Connection.

Steps to Reproduce

To this point we weren’t able to reproduce the issue locally.

Context Information

Postgres and EDC hosted in Azure
Using default connection pool settings
- minIdleConnections: 1
- maxIdleConnections: 4
- maxTotalConnections: 8

Detailed Description

getStatsString from the GenericObjectPool: activeTimes=[68, 51, 49, 48, 51, 48, 572, 131, 53, 51, 46, 49, 53, 119, 127, 116, 125, 56, 54, 56, 47, 48, 47, 46, 47, 47, 49, 52, 49, 47, 50, 47, 49, 49, 50, 48, 48, 46, 47, 48, 51, 47, 48, 47, 47, 95434970, 112, 123, 108, 129, 122, 127, 48, 49, 47, 49, 51, 47, 46, 51, 48, 107, 122, 110, 124, 50, 48, 46, 47, 111, 124], blockWhenExhausted=true, borrowedCount=79, closed=false, createdCount=8, destroyedByBorrowValidationCount=0, destroyedByEvictorCount=0, evictorShutdownTimeoutDuration=PT10S, fairness=false, idleTimes=[30, 142598, 183484, 630982, 183245, 611874, 41764, 15, 16, 30, 9872, 259748, 99866, 10415, 27686, 15, 15, 31, 16, 15, 31, 551836, 10083, 215851, 1572925, 21025, 37442, 14427, 1106441, 18414, 11314608, 4449, 20980, 1334669, 2607, 4832836, 3785, 2872, 57479555, 2087, 1741692, 5692, 2429842, 4907689, 3674, 1777048, 5342593, 5662, 15, 15, 15, 40565, 15, 15, 31, 15, 15, 29, 583269, 334157, 4496, 1496561, 6809, 85024, 436618, 2243, 12166, 15, 14, 30, 15, 15, 30, 5279537, 55971689, 66331, 33003, 15, 15], lifo=true, maxBorrowWaitDuration=PT0.466S, maxTotal=8, maxWaitDuration=PT-0.001S, minEvictableIdleDuration=PT30M, numTestsPerEvictionRun=3, returnedCount=71, softMinEvictableIdleDuration=PT-0.001S, testOnBorrow=true, testOnCreate=true, testOnReturn=true, testWhileIdle=false, durationBetweenEvictionRuns=PT-0.001S, waitTimes=[466, 15, 15, 16, 16, 15, 16, 15, 15, 383, 15, 15, 15, 16, 15, 15, 15, 464, 16, 15, 415, 20, 17, 15, 16, 15, 15, 16, 15, 16, 15, 15, 16, 15, 15, 16, 15, 15, 15, 15, 15, 16, 15, 15, 16, 16, 16, 15, 14, 16, 15, 15, 16, 15, 450, 16, 16, 423, 17, 15, 15, 20, 15, 15, 16, 15, 15, 14, 14, 450, 15, 15, 414, 16, 16, 15, 16, 15, 15], createdCount=8, makeObjectCount=0, maxIdle=4, minIdle=1

Possible Implementation

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 15 (10 by maintainers)

Most upvoted comments

In the transaction framework, the underlying pooled connection will be wrapped. With local transactions, this is done with ConnectionWrapper, while in JTA transactions, the transaction manager implementation will do this. In both cases, calling close() on the connection wrapper will do nothing. When the transaction completes, the transaction manager (e.g. LocalTransactionContext) will free the underlying pooled connection.

With local transactions, as long as the Strem is opened from within a transaction block and LocalDataSourceRegistry is used, the connection will be closed. I don’t think the DoorKeeper is needed to manage connections. Should we discuss?

jimmarino on Dec 6, 2022

@efiege in the meantime, can you please try and reproduce this against M7 or the latest repository version without external extensions (e.g. “flyway”)?

jimmarino on Dec 6, 2022