botocore: botocore 1.9.13 resulting in 400s on s3 HeadObject if region is wrong

Problem

Upon upgrading to awcli 1.14.60 we started to get:

fatal error: An error occurred (400) when calling the HeadObject operation: Bad Request

when invoking aws s3 cp. It turns out that if our default region was not set correctly (i.e. it didn’t match the region the file was actually stored in) this happens. We can fix by specifying --region but that means we have to know the underlying region of each s3 key.

Our current fix is to pin awscli==1.14.59 which uses botocore 1.9.12 and avoids all errors. This is happening across all our local boxes and CircleCi builds.

Suspicion is that https://github.com/boto/botocore/pull/1409 might have something to do with it.

Repro

Assume your_file.txt is in us-west-2 aws s3 cp s3://your_bucket/your_file.txt --region us-east-1 will fail with above error aws s3 cp s3://your_bucket/your_file.txt --region us-west-2 will succeed

Note: We use sse liberally so I’m not sure if this repros on a key without encryption.

System info

$ venv/bin/aws --version
aws-cli/1.14.60 Python/3.6.2 Darwin/17.4.0 botocore/1.9.13