scout-extended: [Bug] scout:reimport moves index too soon, multiple race conditions

If you are using a queue to reimport into algolia that isn’t FIFO eg AWS SQS, there is a very high chance of hitting a race condition where you move the temp index to the real index without pushing all the data into the new index. I’ve seen examples of 5-10% of the records missing.

Also depending on how long the reimport takes and how frequently you are updating records it seems like there is a high chance the reimport will import stale data by using this approach timeline eg:

start reimport
records gets pushed into new temp index
record gets updated and pushed into old index
temp index gets moved to main index.

You now have stale data in the newly moved index

Unsure is this is related but you seem to reference config('scout.synchronous', false) in the code but I can’t seem to figure out where or how that is defined ?

About this issue

Original URL
State: open
Created 5 years ago
Comments: 20 (7 by maintainers)

Most upvoted comments

I agree on the problem, gonna see if I can fix it in the upcoming days 😃

tomcoonen on Mar 20, 2020

Well, maybe you/we can work in a pull request to get this fixed. Just to be clear, the problem is:

Only when the user is using queues:

1. we copy settings to temporary index
2. we queue jobs of indexation
3. we move the temporary index to production without waiting for the queued jobs to be complete.

nunomaduro on Mar 20, 2020

I’m taking this issues @nunomaduro

TomKlotzPro on Apr 18, 2019