amplify-cli: Unable to stream models to ElasticSearch
What is the current behavior?
When I create a mutation, the lambda function that streams dynamodb models to elasticsearch fails and logs Unable to import module 'python_streaming_function': cannot import name 'BotocoreHTTPSession'.
I haven’t edited any of the code as it is automatically generated by the GraphQL Transform
What is the expected behavior?
I should be able to search my models in AWS elasticsearch
Which versions and which environment (browser, react-native, nodejs) / OS are affected by this issue? Did this work in previous versions?
"aws-amplify": "^1.1.40",
"aws-amplify-react-native": "^2.1.19",
"aws-appsync": "^2.0.0",
"aws-appsync-react": "^2.0.0",
"react": "^16.9.0",
"react-apollo": "2.5.8",
"react-native": "^0.60.5",
aws-cli/1.16.243 Python/3.6.0 Windows/10 botocore/1.12.233
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 3
- Comments: 23 (6 by maintainers)
+1 We have started seeing this issue in an extremely similar timeframe.
Has the version of Botocore made available to AWS Lambda changed? Alternatively, have the amplify team made any changes to the underlying zipfile housing this Streaming code?
Please treat this as a high severity issue. As Amplify does not offer a way to “refresh” entries in ES, the only remedy to missing ES records at them moment is a full table rewrite.
@usmansbk We dug into the issue and after talking to the Lambda team, we understood the reason for the change and errors. We’re working on a fix in the CLI that will enable you to re-deploy your function as a part of the ampliy push command to avoid seeing the Lambda errors. For the time being, here’s an explanation and the workaround provided by the Lambda team to prevent your app from seeing errors:
AWS Lambda is updating the default boto3 and botocore SDKs for the Python 2.7, Python 3.6 and Python 3.7 runtimes. This update changes the version of boto3 from 1.7.74 to 1.9.221 for the Python 2.7 and Python 3.6 runtimes and from 1.9.42 to 1.9.221 for the Python 3.7 runtime. The update contains SDK enhancements and updated APIs. One notable example of an internal class that underwent change is the BotocoreHTTPSessions class which is no longer supported starting with botocore version 1.11.0.
Since the CLI used the
BotocoreHTTPSessions class, that’s the reason why you’re seeing these errors.As a workaround, you can update your
DdbToEsFn-xxxxxxfunction in the AWS Console to use a Lambda layer that contains the previous version of the SDK. For your convenience, the AWS Lambda team already published these layers for you to use. You can find additional information on Lambda layers out here - https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.htmlHere are the list of lambda layer ARN’s which you can use based on the region you are in:_
@sacrampton
Open the generated lambda function configuration in the
AWS console -> Lambda, which is typically namedDdbToEsFn-XXXXXX.Select layers
Then click the
Add a layerbutton and select theARNoption.Paste one of the ARNs above based on your region
@kaustavghosh06 Why is it amplify CLI, JS, Cognito and Lambda teams are working out of sync? There has been hundreds of issues referring one another in the github itself. I do not want to highlight the feedback I get from the AWS support. To make the amplify a complete framework to serverless cloud problems you guys should work in sync.
Back to the issue, this issue has halted our ES indexing for the last 1 week and we got complaints from users that the search is no more working. Unfortunately this happened at our production env. In future, what level of guarantee we’ll get to make sure that this doesn’t happen again?
@kaustavghosh06 Thanks for the workaround! I hope the AWS team will not do breaking changes like this one!
@mikeRChambers610 Bear with me I haven’t used appsync or amplify in over a year but I will try to help out. These are the arns you should be using. This problem should be resolved by now seeing how this issue is well over a year old but it appears that it isn’t so this is how i fixed it. you should be using a lambda function to stream your models from your dynamo into elastic search. everything should be inside the same region. There was some sort of bug introduced into the built in streaming lambda SDKs which is why you need to swap out whatever amplify is using by default with an older version provided below.
What you need to do is
I would like to note that the error you are getting is due to the fact the index you’re trying to access doesn’t exist. This is most likely due to the fact that you can’t (or you at least couldn’t back when I was working with amplify last year) retroactively add already existing indices to your ES. If the index already exists inside the DB and the streaming functionality hasn’t been set up yet, next time you go to add something to that index it will fail because as far as ES is concerned it doesn’t exist. The easiest way to get around this is start from scratch with a fresh dynamo and ES instance. If that is not a possibility I’m sure you could write some sort of lambda that will go through your dynamo and add any missing indexes into your ES instance.
Good luck. It’s stuff like this that made me ditch amplify completely. This kind of stuff should work flawlessly out of the box. There’s way to much overhead and confusing configurations, not to mention pretty nasty bugs within the CLI back when I tried to use it. I’m hoping it matures someday but when I came across this issue over a year ago, this project was no where near ready to be used in a high volume production environment.
@kaustavghosh06 I am still seeing this issue in lamda function.
I think
lambdashould warned developers who using build-in libraries from AWS, this is not only affecting Amplify + Elasticsearch users, but everyone who used deprecated library.In the meantime, what Amplify can do is to specify version for each dependencies.
@babu-upcomer thats rough.
I’m in beta right now and have a similar story (luckily didn’t have more than a few dozen people’s experience hurt). It kind of undermines my trust in AWS in general to have something like this happen.
I am also seeing this problem - I created a new type with searchable and no index is created in ElasticSearch.
The instructions for the work around are a bit opaque to me - assuming you have just used standard amplify/appsync to sync DynamoDB and ElasticSearch, what are the specific set of steps/commands we need to execute to “unbreak” this whilst we are waiting for the fix?
This issue should be fixed from the Lambda service side as they rolled-back their changes - causing this behavior. Please feel free to comment on this thread if you’re still seeing this issue.