requests: Our use of urllib3's ConnectionPools is not threadsafe.
We use a PoolManager to provide us with connection pools from urllib3. However, we don’t actually run our requests through it: instead we get a suitable ConnectionPool and then use urlopen on that instead.
This can lead to bugs when using threads and hitting a large number of hosts from one Session object. Each host creates a new ConnectionPool in the PoolManager: if too many are created, the least recently used one is evicted.
If that eviction happens in between us getting hold of the ConnectionPool and us actually trying to send a request, we can try to send on a closed connection pool. That’s pretty lame. You can see the discussion on IRC that led to this issue here.
We can either try to make that section thread safe, or try to reconnect when we hit closed pool errors. Thoughts?
About this issue
- Original URL
- State: closed
- Created 10 years ago
- Reactions: 2
- Comments: 27 (17 by maintainers)
I’m sorry if this is the wrong place for the question, but this appears to be the only official place that such a thing is discussed that I can find.
Is requests.Session thread-safe or not?
The frontpage says that requests is thread safe, but does not mention anything about Session specifically.
I understand that urllib3 connection pool is thread safe. Is Session?
@pior It’s my belief that that should be safe.
@Lukasa If the issue is limited to the stdlib cookie jar, then using a requests.Session across multiple threads should be fine for APIs (without cookies).
Is this correct?