boto3: New region ap-east-1 causes issues with multiple services

@j4v I agree we need to add a workaround for users that want to programmatically make API calls to all available regions but have not opted into every region (e.g. ap-east-1). I think it would make sense to have an opt-out flag in the get_available_regions method call that removes any opt-in regions from the returned list. So something like:

import boto3.session
# Does not list ap-east-1 because it is opt-in
boto3.session.Session().get_available_regions('s3', allow_opt_in_regions=False)
# However if you do not specify allow_opt_in_regions, you get all of the regions
boto3.session.Session().get_available_regions('s3')

I think this direction makes sense because:

1. We cannot know if a specific account has opted into a region. The get_available_regions() does not make any HTTP request and there is not even an API call to list all of the regions that is available to a user. So we cannot add the functionality of excluding a region that is opt-in without a new parameter or new method on the session.

2. There is already a allow_non_regional parameter to exclude regions that are just special endpoints. So there is a precedence for having a filter parameter in the method.

3. We would make the allow_opt_in_regions default to True because it would be a breaking change for users that have already opted into ap-east-1 and expect the get_available_regions() calls to return it to start filtering it out.

The only issue is that we do not have the upstream metadata yet in our endpoints.json to programmatically account for each new opt-in region. So we will have to go get that added in order to programmatically account for any new opt-in region.

Let us know what you think.

@kyleknap I like that idea a lot, it would be really helpful for my use case.

It’s probably on the AWS side of development and not the boto side, but it would be nice if we could determine whether opt-in regions were enabled or not programmatically as well without the need for new APIs or permissions. Maybe something like STS GetCallerIdentity to a disabled opt-in region returns “region disabled” or something, rather than a somewhat-cryptic error message. I’m not sure the best route for that, more-so just throwing out ideas.

Hi team, i am still facing the same issue for the same region, do we have any Solution on how we can get creds for opt-in regions

It’s too bad that the ec2:DescribeRegions permission is required to find out if regions are enabled or not for the entire account

Also, ignoring how impossibly unlikely it is (I hope), what if a region comes out that doesn’t support EC2?

I’d really like AWS to just implement real, consistent error messages that tell you if your access is denied because the region is disabled