amplify-cli: Amplify ElasticSearch Streaming function breaks after AWS upgrades your Lambda runtime

Describe the bug If you are getting the following error in your Amplify ElasticSearch streaming function (DdbToEsFn)

Unable to import module 'python_streaming_function': cannot import name 'BotocoreHTTPSession'

It is because AWS Lambda is updating the default boto3 and botocore SDKs for the Python 2.7, Python 3.6 and Python 3.7 runtimes. This update changes the version of boto3 from 1.7.74 to 1.9.221 for the Python 2.7 and Python 3.6 runtimes and from 1.9.42 to 1.9.221 for the Python 3.7 runtime. This causes problems with BotocoreHTTPSession due to changes made to it and can no longer be imported.

To reproduce your AWS account must be using the latest Python run time AWS is upgrading to or replicate the above environment on your local machine.

To Reproduce Steps to reproduce the behavior:

Assuming your account has the latest runtime, use any version 3.8 or lower of the cli
Make sure one of your schema has @searchable
Execute amplify --push
Execute any mutation against the schema with @searchable in it
Inspect logs of the DdbToEsFn to see the error

Expected behavior Not to cause errors

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 4
Comments: 31 (6 by maintainers)

Most upvoted comments

@keithmacklin @houmark @tluolamo We dug into the issue and after talking to the Lambda team, we understood the reason for the change and errors. We’re working on a fix in the CLI that will enable you to re-deploy your function as a part of the ampliy push command to avoid seeing the Lambda errors. For the time being, here’s an explanation and the workaround provided by the Lambda team to prevent your app from seeing errors:

AWS Lambda is updating the default boto3 and botocore SDKs for the Python 2.7, Python 3.6 and Python 3.7 runtimes. This update changes the version of boto3 from 1.7.74 to 1.9.221 for the Python 2.7 and Python 3.6 runtimes and from 1.9.42 to 1.9.221 for the Python 3.7 runtime. The update contains SDK enhancements and updated APIs. One notable example of an internal class that underwent change is the BotocoreHTTPSessions class which is no longer supported starting with botocore version 1.11.0.

Since the CLI used the BotocoreHTTPSessions class, that’s the reason why you’re seeing these errors.

As a workaround, you can update your DdbToEsFn-xxxxxxfunction in the AWS Console to use a Lambda layer that contains the previous version of the SDK. For your convenience, the AWS Lambda team already published these layers for you to use. You can find additional information on Lambda layers out here - https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

Here are the list of lambda layer ARN’s which you can use based on the region you are in:_

ap-northeast-1: arn:aws:lambda:ap-northeast-1:249908578461:layer:AWSLambda-Python-AWS-SDK:1
us-east-1: arn:aws:lambda:us-east-1:668099181075:layer:AWSLambda-Python-AWS-SDK:1
ap-southeast-1: arn:aws:lambda:ap-southeast-1:468957933125:layer:AWSLambda-Python-AWS-SDK:1
eu-west-1: arn:aws:lambda:eu-west-1:399891621064:layer:AWSLambda-Python-AWS-SDK:1
us-west-1: arn:aws:lambda:us-west-1:325793726646:layer:AWSLambda-Python-AWS-SDK:1
ap-east-1: arn:aws:lambda:ap-east-1:118857876118:layer:AWSLambda-Python-AWS-SDK:1
ap-northeast-2: arn:aws:lambda:ap-northeast-2:296580773974:layer:AWSLambda-Python-AWS-SDK:1
ap-northeast-3: arn:aws:lambda:ap-northeast-3:961244031340:layer:AWSLambda-Python-AWS-SDK:1
ap-south-1:631267018583: arn:aws:lambda:ap-south-1:631267018583:layer:AWSLambda-Python-AWS-SDK:1
ap-southeast-2: arn:aws:lambda:ap-southeast-2:817496625479:layer:AWSLambda-Python-AWS-SDK:1
ca-central-1: arn:aws:lambda:ca-central-1:778625758767:layer:AWSLambda-Python-AWS-SDK:1
eu-central-1: arn:aws:lambda:eu-central-1:292169987271:layer:AWSLambda-Python-AWS-SDK:1
eu-north-1: arn:aws:lambda:eu-north-1:642425348156:layer:AWSLambda-Python-AWS-SDK:1
eu-west-2: arn:aws:lambda:eu-west-2:142628438157:layer:AWSLambda-Python-AWS-SDK:1
eu-west-3: arn:aws:lambda:eu-west-3:959311844005:layer:AWSLambda-Python-AWS-SDK:1
sa-east-1: arn:aws:lambda:sa-east-1:640010853179:layer:AWSLambda-Python-AWS-SDK:1us-us-east-2: arn:aws:lambda:us-east-2:259788987135:layer:AWSLambda-Python-AWS-SDK:1
us-west-2: arn:aws:lambda:us-west-2:420165488524:layer:AWSLambda-Python-AWS-SDK:1
cn-north-1: arn:aws-cn:lambda:cn-north-1:683298794825:layer:AWSLambda-Python-AWS-SDK:1
cn-northwest-1: arn:aws-cn:lambda:cn-northwest-1:382066503313:layer:AWSLambda-Python-AWS-SDK:1
us-gov-west-1:: arn:aws-us-gov:lambda:us-gov-west-1:556739011827:layer:AWSLambda-Python-AWS-SDK:1
us-gov-east-1: arn:aws-us-gov:lambda:us-gov-east-1:138526772879:layer:AWSLambda-Python-AWS-SDK:1

kaustavghosh06 on Sep 24, 2019

This was a planned change it seems, but it was so badly communicated, that nobody inside the AWS Amplify team was aware and had a chance to push out a fix. And even if they did, then this will require a push by every client using Elasticsearch on Amplify as far as I understand. Knowing AWS they will just release this in 3.8.x or 4.x and not care about 3.0.0, which means we are forced to upgrade to that version and potentially face other issues with multi auth etc.

The AWS and especially the AWS Amplify team needs to rethink their strategies on releases and keep 1-2 versions back which they do critical bugfixes on, even after a new version has recently been released. We used to jump on a new version right away, and especially before we were in production, but after having learned the hard way more than once, I am now holding back until I can see that the new versions seems mostly smooth. Also having no release notes since 3.0.0 doesn’t make me feel like even attempting the update.

I’ve applied the layer and it seems to work, but little by little we have more and more and more patches on top of the main setup.

houmark on Sep 20, 2019

Decided I would try to update amplify to get the benefit of the layer… now I cannot even push. Awesome. Just trading one bug for another.

Inaccessible host: xxxxxx-20190814102923-deployment.s3.amazonaws.com'. This service may not be available in the us-east-1’ region.

mormsbee on Oct 22, 2019