botocore: botocore 1.28.0 and higher fail to generate valid S3 presigned URLs

Describe the bug

Since upgrading the botocore 1.28.0 presign requests to S3 generate links like this:

https://s3.amazonaws.com/tiendeo.data.temporary.lab/DataReporting/Process.CommercialReport.CommercialReport/4D370376-2F63-48DF-B150-DA51507BC73F/C35C5CEA-2D51-42ED-A857-F3F86FE6178E.png?AWSAccessKeyId=AKIAJY4KR3BOOATC5GEA&Signature=AFIjb6Idelu5UdQwRac4as6BHNA%3D&Expires=1666883364

instead of this:

https://s3.eu-west-1.amazonaws.com/tiendeo.data.temporary.lab/DataReporting/Process.CommercialReport.CommercialReport/4D370376-2F63-48DF-B150-DA51507BC73F/C35C5CEA-2D51-42ED-A857-F3F86FE6178E.png?AWSAccessKeyId=AKIAJY4KR3BOOATC5GEA&Signature=AFIjb6Idelu5UdQwRac4as6BHNA%3D&Expires=1666883364

As they do not include the region the link fails with the following response:

<Error>
<Code>PermanentRedirect</Code>
<Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
<Endpoint>tiendeo.data.temporary.lab.s3.amazonaws.com</Endpoint>
<Bucket>tiendeo.data.temporary.lab</Bucket>
<RequestId>WKMQ1WVE9XTWNXR2</RequestId>
<HostId>FIefmbEnBZZd4Mrt5SP0IDVC5r6eTPRRzIkFwSSAF+xujig0IVj33FIRQOoRkt8eVCPb24OP7+M=</HostId>
</Error>

Expected Behavior

The presinged link should be valid.

Current Behavior

The presinged link are invalid.

Reproduction Steps

$ pip install -U botocore awscli
$ aws s3 presign s3://bucket/key.png
# invalid link
$ pip install -U botocore==1.27.96
$ aws s3 presign s3://bucket/key.png
# valid link

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.28.2

Environment details (OS name and version, etc.)

Tested on macOS, Linux and Windows

About this issue

Original URL
State: closed
Created 2 years ago
Reactions: 1
Comments: 16 (7 by maintainers)

Most upvoted comments

Hi @pvieito, @benedikt-bartscher, thanks for reporting this. We’ve identified the regression and it appears to be localized specifically to bucket names that aren’t valid host labels as defined in RFC 1123. The issue with the first reported bucket is specifically the use of . in the bucket name. These names require different formatting for the presigned URL which is no longer occurring.

We have a fix in the works that we’re currently validating and intend to release as soon as possible. We’ll provide an update here once that’s finished. As a short term mitigation, you may need to continue using 1.24.96 until this patch is available. Thanks for your patience!

nateprewitt on Oct 27, 2022

Can 1.28.4 be released to resolve this? Also hitting this issue.

slimm609 on Oct 28, 2022