boto3: Exceptions coming from boto3/botocore when running boto3.client('sts') too many times simultaneously

So finally sessions and resources can be shared across threads or not? Immediately after I start using them as described in the docs I hit KeyError: 'credential_provider' or similar. My threads do upload only so I guess I’m on safe side but I keep finding statements contradicting the docs.

So finally sessions and resources can be shared across threads or not? Immediately after I start using them as described in the docs I hit KeyError: 'credential_provider' or similar. My threads do upload only so I guess I’m on safe side but I keep finding statements contradicting the docs.

@zgoda-mobica , Same issue here. I had ThreadPoolExecutor and each thread was invoking boto3.client('lambda').invoke() without using Session() which throws KeyError: 'credential_provider' error.

So now using following in thread invoked method…

session = boto3.session.Session()
lambda_client = session.client('lambda')
lambda_response = lambda_client.invoke(...)
....

and it seems to be ok.

Workaround for this in the mean time: Put a boto3.client('sts') call immediately before the for loop.

What is the reason that the “DEFAULT_SESSION” was introduced at the first place?

Creating a Session is a very resource intensive process as it has to load and deserialize several models before it can create clients. If this was required every time a client was created we’d see unacceptable performance impact. It’s a rare use-case customers actually need individual sessions, so we avoid this by creating a single instance by default for reuse. If this is undesired behavior customers are free to instantiate their own Session as you’re showing.

With the code you’ve proposed above, it will have the same performance as creating one client currently. However, we see creating a client take ~15x longer once we start creating many clients. This is because we are currently amortizing that Session cost over each client creation by reusing a single DEFAULT_SESSION instance.

Performance Impact Example

import timeit
import boto3

CLIENT_COUNT = 1_000

global_time = timeit.timeit(
    'boto3.client("s3")',
    setup='import boto3',
    number=CLIENT_COUNT
)
print(f"Average time per client with Global Session: {global_time/CLIENT_COUNT} sec")

unique_time = timeit.timeit(
    'boto3.session.Session().client("s3")',
    setup='import boto3',
    number=CLIENT_COUNT
)
print(f"Average time per client with Unique Session: {unique_time/CLIENT_COUNT} sec")

We’ll see results over 1000 runs come out as seen below:

$ python perf_test.py
Average time per client with Global Session: 0.001492504167 sec
Average time per client with Unique Session: 0.022752786 sec

As you can see this quickly becomes cost prohibitive for time sensitive applications. This is why the current documentation instructs users to pass clients to threads rather than create them in the thread.

There is also an outstanding unanswered question in botocore about thread safety. Would be great to know what’s the actual correct approach should be.

You’re correct about this behavior. Historically we’ve said that session methods aren’t thread safe, but once you’ve created a client or resource, we guarantee that those calls are thread safe (http://boto3.readthedocs.io/en/latest/guide/resources.html#multithreading-multiprocessing).

While we can investigate changing that stance, I suspect there will be more work than just the component logic, though that’s a good start and one of the most common content points in sessions.

I documented the issue based on my own understanding. https://medium.com/@life-is-short-so-enjoy-it/aws-boto3-misunderstanding-about-thread-safe-a7261d7391fd

The documentation on multithreading was updated in this PR: https://github.com/boto/boto3/pull/2848.

Here are links to the multithreading documentation for Clients, Resources, and Sessions. Does that help clarify things?

The documentation says that it is “recommended” to create a resource per thread but the example code below that recommendation creates a resource AND a session per thread. IOW, it is ambiguous if sessions can be shared between threads or not. Further down, the documentation states that resources are NOT thread-safe. That would imply that it’s not merely “recommended” to create a resource per thread, but rather that it’s a requirement to do so.

I’m confused.