meilisearch: Duplicate documents in the search route

Hi there,

I’m running into kind of the same issue.

We noticed, that meili was returning wrong search documents while doing geo searches. (Vienna results popped up in london results). So we didn’t have duplicate entries in the search, but completely different results to what was requested via filters on meilisearch.

I was using v0.29.1 updated to v0.29.2 and recreated the index (only updating didn’t resolve the issue). The search documents are correct now, if we encounter any problems when doing document updates/additions/deletes and the search results seem odd, I’ll keep you posted.

Some details on the index and meili (if relevant):

Version: v0.29.1 (where the issue popped up initially) Special search functionality features in usage: distinct attribute, geosearch

This PR will be fixed by https://github.com/meilisearch/milli/pull/690.

My hope is that it’s the last time we reopen this issue 🙈

@thijndehaas Thank you for all the information 😄 We have reproduced and found a fix for the bug! (PR: https://github.com/meilisearch/milli/pull/750 ). My hope is that we can release meilisearch v0.30.5 with this fix within the next couple of days. We will keep you updated about the status of the release.

I’m seeing the same issue still with 0.29.2. I did the following when upgrading from 0.29.0 to 0.29.2

• Deleted all our documents in all indexes while still on 0.29.0
• Created a dump
• Setup 0.29.2 and imported the dump from 0.29.0 (this had only 2 empty indexes and our API keys)
• Reindexed all our data to the already created indexes.

This happened a few days later (Upgraded 2022-11-23, issue occured again at 2022-11-30)

Currently running a test towards our test instance with mass update/delete of documents at random and duplicate ID:s to see if I can reproduce this.

Next test will be to delete the indexes and recreate them and see if the issue comes up again.

1. I never dump the indexes, I have never looked into it because I can re-index all documents pretty fast after an update.
2. This time all 1.157 tasks have completed successfully.
3. When I found the duplicates I created a new index and swapped it. When I search in the old (corrupt) index the duplicates are gone now. I’m not sure how they are fixed. When indexes are swapped, are they re-created in the background or is some memory cleared?
4. The data in the documents seems correct compared to the current data.

A minute ago my colleague found another websites where the problem is back. I will try to debug the corrupt documents there before I swap all data.

Update:

• Restarting Meilisearch does not fix the duplicate documents
• I have swapped the indexes in the other website and somehow the problem is not fixed there.
• For your third question, the documentAdditionOrUpdate does not fix the problem. documentDeletion removes only one of the documents. When I index the document after removing the duplicate problem is back.

we are also experiencing this issue at freshline.io

we are seeing both duplicate search results with the same primary key and we’re seeing incorrect results being returned. in our case this has caused products from Customer A’s store to show up on Customer B’s store—which is obviously a big problem

we recently upgraded from 0.26 to 0.29.2. and we batch all of our meilisearch requests, so sometimes we are sending a large number of updates at once

Reopening as this issue is always present, even in the latest release, v0.30.0. The only way it seems to reproduce is by replacing many documents i.e. 9000 tasks. Note that the update or replace operation is internally removing and reindexing the documents. Therefore this issue can also be triggered with document deletion.

@thijndehaas sees duplicated documents with the same EAN, a typical issue we thought we fixed by closing this issue just before. You can see the discussion on this Slack thread.

It can seem like an isolated case but another user is experiencing the same issue. @0x15f doesn’t report duplicated documents (maybe because he uses only the filtering features on this index), but he reports the Missing key in the documents database error when sending documents deletion operations, a typical issue we thought we fixed by closing this issue just before. He is also experiencing An unexpected crash that occurred when processing the task but doesn’t have the logs, he will run again the v0.30.0 and try to provide the logs. We also know that he is running Meilisearch on an xfs file system.

This is the same user that is related to #3163, you can find a link to the corrupted data.ms file there.

Hey @qbx2,

There was a bug in our internal implementation of the soft-deletion system, this system must be invisible to the user, it is a system that speed-up the deletion and replacement of documents. A document must be replaced or updated when its document id matches one of the documents already in the database.