amplify-cli: @searchable directive does not index existing documents

Do you want to request a feature or report a bug? Bug

What is the current behavior? When adding @searchable directive to an existing model (one with entries already in Dynamo), the ElasticSearch service does not index existing documents. It will only index new documents added after the @searchable directive is added.

Steps to reproduce the issue:

Create a model in /amplify/backend/api/schema.graphql
Use amplify push to deploy CloudFormation template and publish API
Use AppSync GraphQL explorer to create new documents for that model
Go back to schema.graphql file and add @searchable directive
Use amplify push to update resources

What is the expected behavior? Elasticsearch service should index all existing documents for a given model, and not just new ones.

Additional environment details

OS version (ie Windows 10 build X, macOS Sierra 10.12.6, etc.): Ubuntu 18, Node v.8.9.3
Output of amplify --version: 0.1.16
Did this work in previous versions?: N?

About this issue

Original URL
State: closed
Created 6 years ago
Reactions: 25
Comments: 15 (3 by maintainers)

Commits related to this issue

fix(graphql-transform-tutorial): Update transform tutorial docs to include note about adding @searchable to models with existing data (response to issue #98) — committed to ev-dev/amplify-cli by deleted user 6 years ago

Most upvoted comments

It would be even better if the CLI did this easily via something like: amplify api reindex

Delete Existing indexes
Repush all @searchable fields via the above lambda
Cleanup

+15

GeorgeBellTMH on Jun 5, 2020

+1. I am running into this too. Is there a workaround in the meantime like using https://www.npmjs.com/package/dynamodb-to-elasticsearch to backfill the data?

+13

hisham on Nov 30, 2018

Thanks @brene the script works beautifully! For anyone looking to put together the script by brene, you can find the Lambda function ARN in AWS > Services > Lambda. There really is a lambda generated called DdbToEsFn to my surprise as I didn’t know since it was auto-generated. Also, you can find the Event source ARN in AWS > DynamoDB > <Your Table> > Overview Tab > Stream details

clodal on Jun 5, 2020

Hi everyone - we’ve added a section in our documentation explaining how to backfill existing data https://docs.amplify.aws/cli/graphql-transformer/directives#backfill-your-elasticsearch-index-from-your-dynamodb-table

renebrandel on Apr 24, 2020

@malcomm The fix is fairly simple. Create a migration Lamda function. Query all documents (or query a batch and re-run). Make it update one “migration” field in each document. DynamoDB will then sync the entire document to Elasticsearch. You can then delete the “migration” field or keep it around for a future DB migration.

houmark on Feb 13, 2020

This is a feature request. Backfilling data is a goal in the future but is not immediately in scope.

mikeparisstuff on Sep 6, 2018

+1 enterprise must-have.

monaye on Jul 11, 2019