boto3: KeyError: 'endpoint_resolver'

Hi James, Yes it is - weird it hasn’t been raised before. would you recommend this as a solution? https://geekpete.com/blog/multithreading-boto3/

I changed s3 = boto3.resource(‘s3’) to below and it work for me

session = boto3.session.Session()
s3 = session.resource('s3')

Is this function being called in multiple threads? The only way I see that happening is if boto3.client is being invoked in multiple threads (client creation is not thread safe).

@cpury: I ran into the same issue as well. I wasn’t comfortable creating a global client due to this documentation, so I tried until I get a client, e.g.

    client = None
    while not client:
        try:
            client = boto3.client('lambda')
        except:
            client = None

This seems to work.

A simple solution based on what I read above, if you want to use threads and not multiprocessing, - works fine.

cond = True
while cond:
    try:
        s3 = boto3.client('s3', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)
        cond = False
    except:
        cond = True

Thanks for your answer, it works for me 😃

Ideal fix would be

session = boto3.session.Session()
awslambda = session.client('lambda')

(note that its not using sesion.resource() which is only applicable for small number of resources)

All the clients created using boto3.client()/boto3.resource() share default global session and thus aren’t thread safe. https://github.com/boto/boto3/blob/c9399569448890ea78f5ba8e8ae8a55b8fddc3ce/boto3/__init__.py#L85-L100

I observe similar behavior on v1.9.16, but the error is KeyError: 'credential_provider' and it depends on the number of workers in a ThreadPool. For me the error seems to happen only when the number of threads is more than half (n/2+1) of tasks.

Making client global also “fixes” the error.

#1592 seems to be relevant.

A simple solution based on what I read above, if you want to use threads and not multiprocessing, - works fine.

cond = True
while cond:
    try:
        s3 = boto3.client('s3', aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY)
        cond = False
    except:
        cond = True

Wow, super weird! Inspired by your code, I changed mine to use a global lambda client object that’s shared among all threads. AKA, do exactly the opposite of what’s recommended here in this ticket. And that fixes it!

So for anyone running into the same issue: Just use a global lambda object. And if anyone of the boto3 team wants to explain how this makes sense, please do, I’m curious 😃

We’ve run into this recently with python 2.7 and bot 1.10.16.

We think we likely need to work around it with uwsgi lazy loading setting toggle. but still looking through it.

I’m using uWSGI lazy-apps=true and still getting this issue. Running single process, multiple threads enabled, but not explicitly invoked in Django 2.2, Python 3.8.

@vinczente Yes, each thread should have its own session.

I’m running into this now. I have a Flask application sitting behind Apache. When multiple clients are accessing a resource in the Flask app, that request triggers a boto3.client(‘s3’).list_objects call, where this error comes up. My app is using “boto3”:“1.7.4” and “botocore”:“1.10.6”.