mongoengine: switch_collection method is not thread safe

Hi guys,

We just run into an issue when archiving data from one collection into another using switch_collection in multi threaded environment (sample code we used below):

def archive(self):
        self.switch_collection(Item._meta['archive_collection'])
        self.save()
        self.switch_collection(Item._meta['collection'])
        self.delete()

The scenario is a following: 1st thread runs archive, executes self.switch_collection(Item._meta['archive_collection']) 2nd thread wants to query different document and doesn’t find it, because 1st thread has switched collection.

It would be a good idea to make switch_collection (and preferably switch_db) method thread safe. What do you think?

About this issue

Original URL
State: open
Created 10 years ago
Reactions: 6
Comments: 15 (4 by maintainers)

Most upvoted comments

6 years passed and the bug which is “just a little bit” critical unless you do a lot of concurrency testing is still open 😦

upcFrost on Jan 8, 2021

My approach to resolve this problem was to stop using switch_collection altogether and instead dynamically generate one class per collection using inheritance. See here for more details.

I think something similar can be done to switch_collection itself, by having it dynamically generate a new class per collection requested (if it hasn’t already generated it).

samuelfekete on Dec 17, 2018

Why not completely review this or maybe even remove this feature? I know that this sound bald but beyond the thread issue the concept of a “switch_db” is already dangerous, doesn’t looks like an streamlined code.

Ideally one should use simply use connA and connB to reference different collections, because in fact they are different connections

rturk on Jan 6, 2015