boto3: Import failure on `from boto3.s3.transfer import TransferConfig`

Describe the bug

Hi,

Since Boto3 v1.33.0 we see a failure to import the following from our code application:

from boto3.s3.transfer import TransferConfig

Resulting in the following redacted Traceback:

/usr/local/lib/python3.10/dist-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/usr/local/lib/python3.10/dist-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
    ???
<frozen importlib._bootstrap>:1027: in _find_and_load
    ???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:688: in _load_unlocked
    ???
/usr/local/lib/python3.10/dist-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
src/tests/test_s3bucket.py:2: in <module>
    from XXX.utils import s3_obj_helper
src/XXX/utils/s3_obj_helper.py:7: in <module>
    from boto3.s3.transfer import TransferConfig
/home/psrcode/.local/lib/python3.10/site-packages/boto3/s3/transfer.py:146: in <module>
    from boto3.crt import create_crt_transfer_manager
/home/psrcode/.local/lib/python3.10/site-packages/boto3/crt.py:26: in <module>
    from s3transfer.crt import (
/home/psrcode/.local/lib/python3.10/site-packages/s3transfer/crt.py:28: in <module>
    from awscrt.s3 import (
E   ImportError: cannot import name 'S3ResponseError' from 'awscrt.s3' (/usr/local/lib/python3.10/dist-packages/awscrt/s3.py)

The same import is working fine on 1.29.7.

Looking at the change log not much seems to be related apart from this: * feature:``s3``: Boto3 will now opt into using the awscrt on select EC2 instance types for s3 transfers.

For now we will freeze our version locally to 1.29.7.

Cheers

Expected Behavior

The import should work.

Current Behavior

The import is broken.

Reproduction Steps

from boto3.s3.transfer import TransferConfig

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.33.0/1

Environment details (OS name and version, etc.)

Ubuntu 22.04.2

About this issue

Original URL
State: closed
Created 7 months ago
Reactions: 2
Comments: 21 (9 by maintainers)

Most upvoted comments

That’s correct, AL2023 adding the CRT to the default Python setup was not a known integration point before yesterday. A quick note of clarity that boto3 should always be used in a virtualenv to avoid unintentional interactions with the default Python installation, but we’re in the process of releasing a change to fix this now.

We expect this issue to be resolved in 1.33.3.

nateprewitt on Nov 29, 2023

The awscrt is an optional dependency that Boto3 will use to optimize functionality if it’s present. It’s not (and currently can’t be) a required dependency for Boto3. It should be kept up to date across AWS products that use it. The main other culprits would be the AWSCLI v2 and the AWS IoT SDK for Python v2.

As far as knowing, this feature has been in place for ~2 years and is intended to be a no-hassle opt-in. That does however make it harder to track for dependency management. If you know you’re installing boto3 in conjunction with one of the other tools above, you can change your requirement to boto3[crt] which will ensure you always have the minimum required awscrt version.

We’re discussing options to try to minimize (or ideally eliminate) these kinds of import issues with version mismatches and the awscrt going forward.

nateprewitt on Nov 28, 2023

Great to hear, @hsec. I’m going to put this in auto-resolve since the original issue is no longer reproducible and we haven’t heard back on further issues in the last 24 hours.

nateprewitt on Nov 30, 2023

Good morning. I uninstalled boto3 (1.33.3) and installed version 1.33.4 and all is good again. Thank you again for your patience and rapid response.

hsec on Nov 30, 2023

Thanks for letting us know @lisenet. We found the CRT copy on AL2023 was outdated yesterday and are working on a fix to help mitigate that case. We’ll provide an update here once that’s available.

nateprewitt on Nov 29, 2023